Puppeteer Web Scraping

What is Puppeteer Web Scraping?

Puppeteer Web Scraping is an automated web data extraction tool based on Puppeteer, designed for handling complex websites that require full browser rendering and JavaScript execution. With CoreClaw, you can scrape dynamic pages, SPA applications, and infinite scroll pages without writing code, enabling scenarios such as dynamic content collection, SPA data extraction, and interactive page scraping.

• ✅ Full Browser Rendering - Uses real browser engine to perfectly render dynamic content
• ✅ JavaScript Execution - Automatically executes page JavaScript to capture dynamically generated content
• ✅ Smart Link Discovery - Intelligently discovers and tracks page links using CSS selectors
• ✅ Custom Page Functions - Write custom JavaScript functions for flexible data extraction
• ✅ Infinite Scroll Support - Automatically scrolls to load more content for infinite scroll pages
• ✅ Automatic Banner Closing - Automatically identifies and closes Cookie banners and other distracting elements
• ✅ URL Filtering - Supports Glob patterns and regex for URL filtering
• ✅ Depth Control - Configurable crawling depth for precise control over scraping scope

What Data Can You Extract?

How to Use Puppeteer Web Scraping?

CoreClaw Puppeteer Web Scraping handles browser startup, page loading, JavaScript execution, link discovery, and data extraction in the background. In just a few minutes, you can extract data through these steps:

1. Create a free CoreClaw account with your email
2. Open the Puppeteer Web Scraping dashboard
3. Enter the list of start URLs
4. Configure scraping parameters (depth, link selector, URL filtering, etc.)
5. Write custom page functions (optional, for extracting specific data)
6. Configure advanced options (infinite scroll, Cookie banner handling, etc.)
7. Click "Start" and let our cloud servers handle the scraping work
8. Download the cleaned dataset in JSON format

➡️ Input

Main Parameter Description

Usage Examples

Example 1: Basic Scraping

• Start URL: https://example.com
• Max Depth: 1
• Max Pages: 10
• Result: Scrapes start page and its first-level link pages, extracts basic data

Example 2: Scraping SPA Applications

• Start URL: https://spa.example.com
• Page Function: Extract dynamically generated product list
• Wait Selector: .product-list
• Result: Waits for dynamic content to load before extracting data

Example 3: Infinite Scroll Pages

• Start URL: https://news.example.com
• Infinite Scroll: true
• Max Scroll Times: 10
• Scroll Delay: 3000 milliseconds
• Result: Automatically scrolls to load and extract all news content

Example 4: Custom Data Extraction

• Start URL: https://blog.example.com
• Page Function: Extract article title, author, and publish date
• Link Selector: a.article-link
• Result: Extracts detailed metadata from blog articles

Example 5: URL Filtering

• Start URL: https://example.com
• URL Pattern: **/article/**
• Regex Pattern: ^https://example\.com/article/\d+$
• Result: Only scrapes article pages matching the pattern

⬅️ Output

For your convenience, output results are displayed in tables and tabs. You can download results in JSON format.

Output Description

Each scraped page will output the following data:

Basic Fields

• url - Page URL
• title - Page title
• depth - Crawling depth (starting from 0)
• statusCode - HTTP status code

Link Information

• linksFound - Number of links found on this page
• links - List of discovered links (optional)

Custom Data

• Custom data extracted via pageFunction

Other Information

• error - Error message (if any)
• timestamp - Scraping timestamp

Example Data:

1{
2  "url": "https://example.com/page",
3  "title": "Page Title",
4  "depth": 1,
5  "statusCode": 200,
6  "linksFound": 45,
7  "links": [
8    "https://example.com/page2",
9    "https://example.com/page3"
10  ],
11  "customData": {
12    "author": "Author Name",
13    "publishDate": "2024-01-01"
14  },
15  "error": "",
16  "timestamp": "2024-01-01T00:00:00.000Z"
17}

FAQ

What's the Difference Between Full Browser Scraping and Regular Scraping?

Use regular scrapers for static websites. Use full browser scrapers for dynamic websites and SPA applications.

How to Write Page Functions?

Page functions are custom JavaScript functions for extracting specific data from pages. Use page.evaluate() to execute code in the browser context for better performance.

How to Handle Infinite Scroll Pages?

Start with a smaller scrollMaxPages for testing, then increase after confirming the effect.

How Are Cookie Banners Automatically Closed?

The tool has built-in automatic Cookie banner closing functionality.

For special banners, use waitForSelector to wait for main content to load.

How to Set Crawling Depth?

• The greater the depth, the more pages grow exponentially
• Recommend using with maxPages to prevent scraping too many pages
• For large websites, recommend depth not exceeding 3

Recommendation: In most scenarios, depth 1-2 is sufficient.

What Are the Use Cases?

• Dynamic Page Scraping - Scrape dynamic content requiring JavaScript rendering
• SPA Data Extraction - Extract data from React, Vue, Angular and other single-page applications
• Infinite Scroll Content - Scrape social media, news lists and other infinite scroll pages
• Interactive Pages - Pages requiring clicks, scrolling and other interactions to get content
• E-commerce Product Collection - Scrape product information, prices, and reviews from e-commerce sites
• Social Media Monitoring - Monitor social media posts and updates
• News Aggregation - Aggregate latest news from major news websites
• Data Backup - Back up website content to prevent data loss

How to Handle Anti-Scraping Mechanisms?

Automatic Handling:

• Uses real browser, not easily identified
• Automatically handles Cookie and Session
• Supports proxy rotation (if configured)

Recommendation: Set scraping parameters reasonably and comply with website terms of use.

Is Scraping Data Legal?

Our full browser web scraper only publicly accesses websites and extracts visible content. Users should comply with the terms of service and usage agreements of target websites when using scraped data. We recommend using it only for legitimate business analysis and research purposes.