RAG Web Browser

What is a RAG Web Browser?

A RAG web browser is an automated content extraction tool designed to provide real-time web search and scraping capabilities for Retrieval-Augmented Generation (RAG) pipelines and AI applications. With CoreClaw, you can obtain structured web content with zero code, empowering AI chatbots, knowledge base construction, content aggregation, and data mining.

• ✅ Google Search Integration - Intelligent multi-page search result extraction to find relevant content
• ✅ Dual Scraping Modes - Supports raw-http (fast HTTP requests) and browser-playwright (full browser rendering)
• ✅ Multi-Format Output - Supports Markdown, Plain Text, and HTML formats
• ✅ Concurrent Processing - Configurable parallelism for simultaneous multi-page scraping to boost efficiency
• ✅ Smart Content Handling - Automatic cookie popup removal, element filtering, and readability extraction
• ✅ Performance Monitoring - Built-in timing metrics and debug mode for real-time performance tracking
• ✅ Dynamic Content Support - Scrapes dynamic content by waiting for JavaScript rendering
• ✅ Production Ready - Optimized for the CoreClaw cloud environment for stability and reliability

What data can you extract?

How to use the RAG Web Browser?

CoreClaw RAG Web Browser handles proxy rotation, task scheduling, concurrency control, and data standardization for you in the background. In just a few minutes, you can get your data by following these steps:

1. Create a free CoreClaw account using your email
2. Open the RAG Web Browser control panel
3. Enter your search query or a direct target URL
4. Configure parameters (max results, output format, scraping mode, concurrency, etc.)
5. Set advanced options (timeouts, retry attempts, content handling, etc.)
6. Click "Start" and let our cloud servers handle the scraping
7. Download the cleaned dataset in JSON format

➡️ Input

Key Parameter Descriptions

Usage Examples

Example 1: Scraping based on Google Search

• Query: Latest developments in AI
• Max Results: 5
• Output Format: Markdown
• Scraping Mode: raw-http
• Result: Extracts Markdown content from the top 5 search result pages.

Example 2: Direct Scraping of a Specific URL

• Query: https://example.com/article
• Output Format: text
• Scraping Mode: browser-playwright
• Dynamic Content Wait: 15 seconds
• Result: Scrapes plain text from the specified page after waiting for JavaScript rendering.

Example 3: Concurrent Multi-Page Scraping

• Query: Machine learning tutorials
• Max Results: 10
• Concurrency: 5
• Remove Cookie Warnings: true
• Result: Concurrently scrapes 10 pages with automatic cookie popup removal, significantly increasing speed.

Example 4: Readability-Optimized Extraction

• Query: https://example.com
• HTML Transformer: readableText
• Remove Elements: .advertisement, .sidebar
• Result: Extracts the main content of the page while automatically removing ads and sidebar elements.

⬅️ Output

For your convenience, output results are displayed in tables and tabs. You can choose to download the results in JSON format.

Output Content Description

Each scraped page will output the following data:

Crawl Information (crawl)

• httpStatusCode - HTTP status code
• httpStatusMessage - HTTP status message
• loadedAt - Timestamp of when the page was loaded
• requestStatus - Status of the request
• uniqueKey - Unique identifier

Debug Information (debug)

• timeMeasures - Timing metrics
• totalTimeMs - Total time elapsed (ms)
• urlsScraped - Number of URLs scraped

Search Result (searchResult)

• title - Page title
• description - Page description
• url - Page URL

Metadata (metadata)

• title - Page title
• url - Page URL
• description - Page description
• languageCode - Language code

Content Output

• markdown - Markdown formatted content
• text - Plain text content
• html - Raw HTML content

JSON Example:

{
  "url": "https://example.com",
  "crawl": {
    "httpStatusCode": 200,
    "httpStatusMessage": "OK",
    "loadedAt": "2024-01-01T00:00:00.000Z",
    "requestStatus": "handled",
    "uniqueKey": "abc123",
    "debug": {
      "timeMeasures": [],
      "totalTimeMs": 1654,
      "urlsScraped": 3
    }
  },
  "searchResult": {
    "title": "Page Title",
    "description": "Page description",
    "url": "https://example.com"
  },
  "metadata": {
    "title": "Page Title",
    "url": "https://example.com",
    "description": "Page description",
    "languageCode": "en"
  },
  "markdown": "# Page Content...",
  "text": "Page content...",
  "html": "<html>...</html>"
}

FAQ

What is the difference between raw-http and browser-playwright modes?

raw-http mode:

• Fast - Uses Cheerio for parsing without browser rendering
• Low Cost - Minimal resource consumption
• Best for - Static web pages, SEO-friendly sites

browser-playwright mode:

• Powerful - Full browser rendering with JavaScript execution
• Accurate - Perfectly renders dynamic content
• Best for - Single Page Applications (SPAs), login-required pages, heavy JavaScript sites

How do I choose an output format?

Markdown - Recommended Format

• Ideal for LLM processing
• Preserves structure (headings, lists, links)

Plain Text

• Pure text, small file size
• Best for keyword extraction and text analysis
• Easy to search and process

HTML

• Preserves full original structure
• Best for scenarios requiring exact replication
• Supports custom post-processing

Recommendation: Use Markdown for RAG applications, Plain Text for text analysis, and HTML for exact structure requirements.

How should I set the concurrency?

The desiredConcurrency parameter controls the number of pages scraped simultaneously:

Recommendation: Start with 3 and adjust based on the target site's response.

How do I handle dynamic content?

For dynamic content requiring JavaScript rendering:

1. Use browser-playwright mode

   • Enables full browser rendering
   • Automatically waits for JavaScript execution

2. Set dynamic content wait time

   • Use the dynamicContentWaitSecs parameter
   • Default is 10 seconds; can be adjusted to 30-60 seconds as needed

3. Verify content loading

   • Enable debugMode to see loading details
   • Adjust wait time based on debug info

How do I remove unwanted content?

Filter content using the following methods:

Remove Cookie Warnings

• Set removeCookieWarnings: true
• Automatically removes common cookie consent popups

Custom Element Filtering

• Use the removeElementsCssSelector parameter
• Specify elements to remove using CSS selectors
• Example: .advertisement, .sidebar, .footer

Readability Extraction

• Set htmlTransformer: "readableText"
• Automatically extracts the main content and removes irrelevant elements

What are the common use cases?

• AI Chatbots - Providing real-time web search capabilities for LLMs to answer user queries
• RAG Workflows - Extracting and processing web content for knowledge base construction
• Content Aggregation - Collecting articles, documentation, and research materials
• SEO Analysis - Extracting metadata and content from competitor websites
• Data Mining - Extracting structured data from web pages
• Knowledge Graphs - Crawling related pages to build domain-specific knowledge bases