Product Hunt Scraper

What is Product Hunt Scraper？

Product Hunt Scraper is a CoreClaw Worker for discovering trending products by keyword. It is designed for market research, competitor tracking, lead generation, product newsletter workflows, and AI/startup trend monitoring.

The worker is adapted from the open-source ph_ai_tracker / ProductHunt-Scraper project, but removes long-running services and local persistence so it fits CoreClaw's one-shot Worker model.

Key Features

• Search Product Hunt by multiple commercial keywords, such as AI agents, developer tools, sales automation, and data analytics.
• Uses search_terms as the CoreClaw task-splitting field.
• Recommended auto strategy:
  • Uses Product Hunt API first when a token is provided.
  • Uses bounded Product Hunt Atom feed and product-page site search fallbacks; browser/HTTP page scraping are only used when explicitly selected.
• Automatically uses CoreClaw's SOCKS5 proxy through PROXY_DOMAIN and PROXY_AUTH.
• Streams one clean structured row per product.
• Emits a structured failure row if all providers fail, instead of crashing silently.

Project Structure

producthunt-scraper/
├── main.py              # CoreClaw Worker entry
├── scraper.py           # Product Hunt providers: browser, HTTP scraper, API, feed, search fallback
├── input_schema.json    # CoreClaw input form
├── output_schema.json   # CoreClaw output table schema
├── requirements.txt     # Python dependencies
├── README.md            # English documentation
├── README_CN.md         # Chinese documentation
├── sdk.py               # CoreClaw SDK
├── sdk_pb2.py
└── sdk_pb2_grpc.py

Input Parameters

Recommended Input

{
  "search_terms": [
    { "string": "AI agents" },
    { "string": "developer tools" },
    { "string": "sales automation" }
  ],
  "limit": 20,
  "strategy": "auto",
  "api_token": "",
  "recent_days": 30,
  "max_enrich": 0,
  "timeout_seconds": 45
}

CoreClaw may split search_terms and pass a single subtask as {"string": "AI agents"}. This Worker explicitly supports that flattened input shape.

Output Fields

Strategy Guide

CoreClaw Network Requirement

CoreClaw cloud does not reliably allow direct outbound access. The Worker is designed to use platform-provided network access:

PROXY_DOMAIN=<platform proxy endpoint>
PROXY_AUTH=<username:password>
ChromeWs=<remote fingerprint browser endpoint>

The Worker automatically builds:

socks5://<PROXY_AUTH>@<PROXY_DOMAIN>
ws://<PROXY_AUTH>@<ChromeWs>

Do not hard-code proxy credentials.

Local Verification

Syntax check:

python -m py_compile main.py scraper.py

The actual main.py entry requires CoreClaw's SDK gRPC service, so run full end-to-end tests on CoreClaw.

Implementation Notes

Removed from the original project:

• SQLite persistence
• HTTP API server
• scheduler / cron mode
• AI tagging
• Docker service mode

Kept and adapted:

• Product Hunt structured product model
• __NEXT_DATA__ parser
• DOM fallback parser
• Product Hunt GraphQL API, public Atom feed, and site-search fallback paths
• CoreClaw proxy and browser runtime support

FAQ

What is this Product Hunt Scraper for?

It scrapes trending products from Product Hunt by keywords for market research, competitor tracking, lead generation, and startup trend monitoring.

Why am I getting 403 / access blocked errors?

Switch to browser strategy; it uses CoreClaw’s fingerprint browser + proxy.

What does search_terms do?

It’s your keyword list; CoreClaw automatically splits tasks by this field.