Webpage Content Extractor

What is a Web Content Extractor?

A web content extractor is an automated data extraction tool specifically designed for bulk scraping page content from websites, supporting multiple output formats and intelligent content cleaning. With CoreClaw, you can obtain structured web content with zero code, facilitating content aggregation, SEO analysis, AI knowledge base construction, and data mining.

• ✅ Multiple Output Formats - Supports Markdown, HTML, and plain text
• ✅ Smart Link Discovery - Automatically detects internal and external links
• ✅ Depth Control - Configurable crawling depth (0-10 levels)
• ✅ CSS Selectors - Extract specific page regions
• ✅ Smart Waits - Waits for selectors, dynamic content, network idle
• ✅ Content Cleaning - Automatically removes navigation, normalizes whitespace
• ✅ Pattern Matching - Use regular expressions to include/exclude URLs
• ✅ Batch Processing - Supports concurrent scraping of multiple sites, efficiently building content libraries

What data can you extract from websites?

How to Scrape Web Content?

CoreClaw Web Content Extractor handles proxy rotation, task scheduling, data standardization, and final delivery for you in the background. In just a few minutes, you can get your data by following these steps:

1. Create a free CoreClaw account using your email
2. Open the Web Content Extractor control panel
3. Enter the starting URL and set parameters (max pages, depth, extraction mode, etc.)
4. Select extraction mode and CSS selectors (optional)
5. Click "Start" and let our cloud servers handle the data scraping
6. Download the dataset in JSON or CSV format

➡️ Input

Key Parameter Descriptions

Usage Examples

Example 1: Basic Crawling

• Starting URL: https://example.com
• Max Pages: 50
• Max Depth: 2
• Extraction Mode: Markdown
• Result: Extracts Markdown content from 50 pages

Example 2: Extract Specific Area

• Starting URL: https://blog.example.com
• CSS Selector: article
• Wait for Selector: .content
• Extraction Mode: Markdown
• Result: Extracts content only from the article area

Example 3: Discover Links Only

• Starting URL: https://example.com
• Crawl Mode: discover_only
• Include Links: true
• Result: Discovers all links, no content extraction

⬅️ Output

For your convenience, the output results are displayed in tables and tabs. You can choose to download the results in CSV/JSON format.

Output Content Description

Basic Fields

• url - Page URL
• title - Page Title
• statusCode - HTTP Status Code
• depth - Crawling Depth

Content Fields (Corresponding fields returned based on extraction mode)

• markdown - Markdown formatted content
• html - HTML formatted content
• text - Plain text content

Auxiliary Fields

• excerpt - Content preview (300 characters)
• links_internal - Discovered internal links
• links_external - Discovered external links

Example Data:

{
  "url": "https://example.com/page",
  "title": "Page Title",
  "markdown": "# Page Title\n\nHere is the page content...",
  "excerpt": "Here is a preview of the page content, the first 300 characters...",
  "links_internal": [
    "https://example.com/about",
    "https://example.com/contact"
  ],
  "links_external": [
    "https://external.com/link"
  ],
  "depth": 1,
  "statusCode": 200
}

FAQ

Which output formats are supported?

• Markdown - Suitable for documentation, knowledge bases, RAG applications
• HTML - Preserves original formatting and structure
• Plain Text - Suitable for text analysis and data mining

How to control crawling depth?

Use the maxDepth parameter to control crawling depth:

• 0: Only scrape the starting page
• 1: Scrape the starting page + first-level links
• 2-10: Increment depth, up to 10 levels

Can I extract only specific areas of a page?

Yes. Use the cssSelector parameter to specify the page area to extract:

• article - Extracts article content
• .content - Extracts content with the specified class name
• #main - Extracts content with the specified ID

How to filter unwanted URLs?

Use the following two methods to filter:

• includePatterns - Only crawl URLs matching the regex pattern
• excludePatterns - Exclude URLs matching the regex pattern

What are the limits for crawling depth and time?

Limits for different parameters:

• maxPages: 1-10000 pages
• maxDepth: 0-10 levels
• concurrency: 1-50 concurrent tasks
• requestTimeoutSecs: 5-600 seconds Recommendation: Set parameters reasonably based on website scale and server performance.

How to handle dynamically loaded content?

Use smart wait parameters:

• waitForSelector - Wait for a specific element to appear
• waitUntil - Control loading strategy (domcontentloaded/load/networkidle)

What does the content cleaning feature do?

The content cleaning feature automatically:

• Removes irrelevant content such as navigation bars, footers, and advertisements
• Normalizes whitespace characters
• Generates a content summary (excerpt)
• Improves content readability

What application scenarios can it be used for?

• RAG Workflow - Extract content for AI knowledge bases
• Content Aggregation - Collect articles and blog posts
• SEO Analysis - Analyze page structure and content
• Data Mining - Extract structured data from websites
• Document Generation - Create documents from web pages