Web Scraper Tool

What is a Web Scraper Tool?

A web scraper tool is an automated website crawling utility designed to bulk scrape pages, extract structured data (titles, descriptions, keywords, headings, images, links, etc.), and automatically discover relevant links. With CoreClaw, you can obtain structured web data with zero code, empowering data collection, SEO analysis, competitive research, and content monitoring.

• ✅ Automatic Link Discovery - Automatically finds and crawls links within the same domain
• ✅ Depth Control - Configurable crawling depth (starting page depth is 0)
• ✅ Page Limits - Set the maximum number of pages per crawl to manage costs
• ✅ Resource Management - Option to block media/CSS to accelerate crawling speed
• ✅ jQuery Injection - Optional injection of the jQuery library for easier data extraction
• ✅ SSL Support - Ignores SSL errors from self-signed certificates
• ✅ Remote Browsers - Connects to fingerprint browser pools via CDP
• ✅ Flexible Configuration - Supports custom timeouts, wait conditions, and data extraction logic

What data can you extract from websites?

How to use the Web Scraper Tool?

CoreClaw Web Scraper Tool handles proxy rotation, task scheduling, data standardization, and final delivery for you in the background. In just a few minutes, you can get your data by following these steps:

1. Create a free CoreClaw account using your email
2. Open the Web Scraper Tool control panel
3. Enter the starting URL and set parameters (depth, page limits, resource control, etc.)
4. Configure advanced options (jQuery injection, SSL settings, debug logs, etc.)
5. Click "Start" and let our cloud servers handle the crawling process
6. Download the cleaned dataset in JSON or CSV format

➡️ Input

Key Parameter Descriptions

Usage Examples

Example 1: Basic Web Crawling

• Starting URL: https://example.com
• Max Depth: 1
• Max Pages: 10
• Result: Crawls the starting page and its first-level links, extracting titles, descriptions, keywords, etc., from all pages.

Example 2: Deep Crawling

• Starting URL: https://blog.example.com
• Max Depth: 3
• Max Pages: 50
• jQuery Injection: true
• Result: Recursively crawls pages up to 3 levels deep, injecting jQuery to facilitate data extraction.

Example 3: Fast Crawling (Ignore Resources)

• Starting URL: https://example.com
• Download Media: false
• Download CSS: false
• Timeout: 30s
• Result: Crawls only page text content, ignoring images, videos, and CSS for a significant speed boost.

⬅️ Output

For your convenience, output results are displayed in tables and tabs. You can choose to download the results in CSV/JSON format.

Output Content Description

Each crawled page will output the following data:

Basic Fields

• url - Page URL
• depth - Crawling Depth
• title - Page Title
• description - Page Description
• keywords - Keywords

Structured Fields

• h1 - Main Heading
• h2List - List of subheadings
• textLength - Text length
• imageCount - Number of images
• linkCount - Number of links

JSON Example :

{
  "url": "https://example.com/page",
  "depth": 1,
  "title": "Page Title",
  "description": "Page description text",
  "keywords": "keyword1, keyword2",
  "h1": "Main Heading",
  "h2List": ["Subheading 1", "Subheading 2"],
  "textLength": 5000,
  "imageCount": 12,
  "linkCount": 45
}

FAQ

How is the crawling depth calculated?

Crawling depth is calculated starting from the initial page:

• Depth 0: Only the starting page is crawled.
• Depth 1: Crawls the starting page + first-level linked pages.
• Depth 2-10: Incremental depth, supporting up to 10 levels.

How can I control the number of pages crawled?

You can control the volume using two parameters:

• maxPagesPerCrawl: Sets the maximum number of pages for each crawl.
• maxCrawlingDepth: Limits how deep the crawler goes.

What is the purpose of jQuery injection?

The jQuery injection feature allows you to:

• Easily extract specific data from the page.
• Use jQuery selectors to find elements.
• Simplify DOM manipulation and data extraction logic.

How are SSL certificate errors handled?

By default, the tool ignores SSL certificate errors (ignoreSslErrors: true).

• ✅ Self-signed certificates - Automatically ignored.
• ✅ Expired certificates - Automatically ignored.
• ✅ Incomplete certificate chains - Automatically ignored.

What are the options for the waitUntil parameter?

The following navigation completion conditions are supported:

• load - Page is fully loaded (including all resources).
• domcontentloaded - DOM is fully loaded (does not wait for images, etc.).
• networkidle0 - At least 0 network connections for 500ms.
• networkidle2 - At most 2 network connections for 500ms (Default).

How can I speed up the crawling process?

Use the following methods to boost performance:

• Set downloadMedia: false - Skip images and videos.
• Set downloadCss: false - Skip CSS stylesheets.
• Reduce pageLoadTimeoutSecs - Shorten the timeout duration.
• Use domcontentloaded as the wait condition.

What are the common use cases?

• SEO Audits - Check titles, descriptions, heading structures, and counts for images and links.
• Competitive Analysis - Extract product info, monitor pricing and descriptions, and track content changes.
• Content Aggregation - Collect news articles, blog posts, and product listings from multiple sources.
• Data Collection - Bulk extract structured data from websites.
• Content Monitoring - Track changes in content and structure across multiple pages.

How can I customize the data extraction logic?

If you need to extract specific page data (such as prices, authors, dates, etc.), you can do so through custom data extraction logic. CoreClaw offers flexible configuration options, supporting customized extraction fields and rules based on your requirements.