Shopify Product Scraper

What is Shopify Product Scraper？

Shopify Product Scraper crawls product data from any Shopify store via its JSON API, sitemap discovery, and browser-based anti-bot fallback.Extracts title, price, description, SKU, variants, images, inventory status, and metadata.

Shopify Product Scraper Features

• Zero-config discovery: Provide a store URL, the worker parses robots.txt to find product sitemaps automatically
• Dual data source: Tries /products.json bulk API first (fast), falls back to individual product endpoints via browser (robust)
• Browser fingerprint integration: Uses the platform-hosted remote browser to bypass WAF / Cloudflare when direct HTTP is blocked
• Sitemap fallback: Tries 5 common sitemap paths when robots.txt doesn't declare one
• Locale-aware URL normalization: Strips Shopify locale prefixes (/es-US/products/) for correct JSON API access
• JSON+HTML mode: Optionally fetch HTML pages first, then JSON — preserving raw HTML for custom parsing
• Extendable output: Inject a custom JavaScript function to transform, filter, or enrich each product row
• Scraper lifecycle hooks: Inject hooks at PRE/POST navigation, URL filtering, and RUN/FINISH stages
• Concurrent crawling: Configurable parallelism (1-20) with automatic retry on failure
• All variants expanded: Each variant becomes a separate output row with its own SKU, price, option attributes, and images

How to Get Quick Start

Input

{
    "startUrl": [{ "url": "https://www.gymshark.com" }],
    "maxRequestsPerCrawl": 0,
    "maxConcurrency": 20,
    "checkForBanner": true,
    "fetchHtml": false,
    "debugLog": false,
    "extendOutputFunction": "async ({ data, item, product, images, fns, name, request, variants, context, customData, input, platform }) => {
  return item;
}",
    "extendScraperFunction": "async ({ fns, customData, platform, label }) => {
 
}",
    "customData": {}
}

Output fields

Usage Examples

Example 1: Basic scrape — crawl all products from one store

{
    "startUrl": [{ "url": "https://www.gymshark.com" }],
    "maxRequestsPerCrawl": 0,
    "maxConcurrency": 20
}

Example 2: Multiple stores in parallel

The startUrl array controls how the platform splits work across concurrent subtasks (via the b field).

{
    "startUrl": [
        { "url": "https://www.gymshark.com" },
        { "url": "https://www.spanx.com" },
        { "url": "https://www.nativecos.com" }
    ],
    "maxRequestsPerCrawl": 0,
    "maxConcurrency": 20
}

Example 3: Custom output mapping — price threshold and discount calculation

Use extendOutputFunction to transform or reject each row. Return null to skip.

async ({ data, item, fns, input, platform }) => {
    // Filter out low-price items
    if (item.price < 10) return null;

    // Compute discount percentage
    const comparePrice = data.product?.variants?.[0]?.compare_at_price;
    if (comparePrice && item.price) {
        item.additional.discount_pct = Math.round((1 - item.price / comparePrice) * 100);
    }

    // Exclude gift cards
    if (item.product_type === 'Gift Card') return null;

    return item;
}

Example 4: Scraper lifecycle — inject custom sitemap URLs

Use extendScraperFunction to hook into different stages of the crawl lifecycle.

async ({ fns, customData, platform, label }) => {
    if (label === 'SETUP') {
        // Access the request queue to inject additional URLs
        const extraSitemap = customData.extraSitemapUrl;
        if (extraSitemap) {
            // The runner passes { requestQueue } at SETUP stage
        }
    }

    if (label === 'FILTER_SITEMAP_URL') {
        // this.url is the product/sitemap URL being evaluated
        // this.filter(false) excludes it from the crawl
    }

    if (label === 'PRENAVIGATION') {
        // this.crawlingContext.request — modify headers before each request
    }

    if (label === 'FINISHED') {
        // this.crawler — access crawler stats, persist state
    }
}

Example 5: HTML mode — parse embedded JSON-LD from product pages

Enable fetchHtml to get the full HTML page alongside the JSON API response. The HTML body is available in request.userData.body inside the output function.

{
    "startUrl": [{ "url": "https://www.colourpop.com" }],
    "maxRequestsPerCrawl": 20,
    "fetchHtml": true
}

In extendOutputFunction:

async ({ data, item, request, context }) => {
    const htmlBody = request.userData.body;
    if (htmlBody) {
        // Parse JSON-LD or meta tags from the HTML
        // context.$ is cheerio loaded from the HTML page
    }
    return item;
}

Example 6: Debug mode — inspect failed JSON responses

Set debugLog: true and failed responses (missing product title) are saved to storage for inspection.

{
    "startUrl": [{ "url": "https://www.kith.com" }],
    "maxRequestsPerCrawl": 5,
    "debugLog": true
}

Input Reference

Known Limitations

• WAF-protected stores: Stores with aggressive WAF (Cloudflare, Akamai) may return challenge pages instead of product data. These appear in output with titles like "Verifying your connection..." and empty product fields. Filter them in extendOutputFunction:
  javascriptCopy

  if (!item.sku || item.title === 'Verifying your connection...') return null;

• Browser-only mode: When products.json is blocked, all requests go through the browser, which is slower (~1 req/sec per concurrent browser). With 5 concurrent browsers, expect ~5 products/sec.
• Currency detection: Always outputs "USD" — multi-currency stores need custom parsing via extendOutputFunction.

Proxy & Network

On CoreClaw, all outbound HTTP requests go through the platform's SOCKS5 proxy. The proxy address is read from PROXY_AUTH and PROXY_DOMAIN environment variables (set automatically by the platform).

The browser is connected via WebSocket CDP (ChromeWs env var + PROXY_AUTH auth). Both are platform-injected — no manual configuration needed.

FAQ

What types of Shopify stores are supported?

All online stores built on the Shopify platform can be scraped, regardless of theme or language version. The tool automatically detects and handles localized URLs.

Can I scrape non-Shopify sites?

This tool is designed specifically for Shopify. Non-Shopify sites may be attempted, but data structures may not be compatible. The checkForBanner parameter verifies whether robots.txt contains Shopify identifiers.

What's the optimal concurrency setting?

The default of 20 concurrent requests works well for most scenarios. For stores with strict rate limiting, we recommend reducing it to 5-10.