Amazon search scraping

Amazon Search CoreClaw Worker

The worker keeps the original flow:

1. Normalize site, keyword, and page_num.
2. Fetch and parse Amazon search pages.
3. Follow search-page next links.
4. Deduplicate product URLs by ASIN.
5. Fetch product detail pages asynchronously with a semaphore.
6. Parse and align search-page and detail-page records.
7. Push one CoreClaw result containing search_page and detail_page.

Input

{
  "site": "us",
  "keyword": "apple",
  "page_num": 2
}

site defaults to us. Invalid page_num values fall back to 1.
The Worker exposes only site, keyword, and page_num. Search/detail
requests always use the internal constants MAX_RETRY = 3 and
DETAIL_CONCURRENCY = 10; they are not read from CoreClaw input. Detail
pages are fetched and parsed in batches of 10. Each batch must finish parsing
before the next batch starts, and each response body is released immediately
after parsing to limit memory usage.
The task timeout is fixed internally at 10800 seconds. AMAZON_MAX_PAGE_NUM
and AMAZON_REQUEST_TIMEOUT can be supplied as runtime environment variables.
Proxy configuration follows the LinkedIn workers: HTTPS_PROXY takes
priority over HTTP_PROXY; otherwise PROXY_DOMAIN, PROXY_AUTH, and
PROXY_SCHEME are used to build the proxy URL.