SHEIN Product Scraper (Keyword/Category-Driven)

SHEIN Product Discovery

Overview

This project discovers SHEIN products from three different input modes:

• shein_products_by-keyword
• shein_products_by-category-url
• shein_products_by-category-id

The worker opens the corresponding SHEIN page, attempts to pass SHEIN risk verification automatically, parses the product grid, and returns a structured product list.

Typical use cases:

• Product discovery by search term
• Product discovery by category landing page
• Product discovery by category ID
• Listing monitoring
• Price tracking
• Competitor research
• Trend and badge analysis

Discovery Modes

1. shein_products_by-keyword

Use a keyword and build the SHEIN search URL as:

https://{country}.shein.com/pdsearch/{keyword}/?sort=...&page=...&limit=...

2. shein_products_by-category-url

Use a full category URL directly, then append the common query parameters:

{category_url}?sort=...&page=...&limit=...

3. shein_products_by-category-id

Use a category ID and build the category page URL as:

https://{country}.shein.com/{category_id}.html?sort=...&page=...&limit=...

Workflow

1. Read type and value from the input payload.
2. Build the target SHEIN URL according to the selected mode.
3. Connect to a remote Chromium instance through ChromeWs.
4. Open the SHEIN page and attempt to pass SHEIN verification automatically when required.
5. Wait until the SHEIN product page is actually ready.
6. Parse product cards from the page.
7. Return a normalized product array.

Runtime Requirements

The worker depends on:

• playwright
• selectolax
• httpx
• grpcio
• protobuf

Environment variables used at runtime:

• ChromeWs: required, remote Chromium CDP websocket host
• PROXY_AUTH: optional, authentication prefix added to the websocket URL

If ChromeWs is missing, the worker returns a failure result.

Input Parameters

The input schema is defined in input_schema.json.

Request Example

{
  "type": "shein_products_by-keyword",
  "value": "dress",
  "country": "us",
  "sort": "recommend",
  "page": 1,
  "limit": 20,
  "flow_retry_num": 3
}

Parameters

Supported Countries

Sort Options

The current code maps sort values as follows:

Output Structure

The output schema is defined in output_schema.json.

Response Example

{
  "type": "shein_products_by-keyword",
  "url": "https://us.shein.com/pdsearch/dress/?source=sort&sourceStatus=4&page=1&force_suggest=1&limit=20",
  "code": 1,
  "count": 20,
  "products": [
    {
      "goods_id": "123456789",
      "product_url": "https://us.shein.com/example-p-123456789.html",
      "title": "Mock Neck Bodycon Dress",
      "main_image": "https://img.ltwebstatic.com/...",
      "price": 7.51,
      "price_usd": 7.51,
      "currency": "USD",
      "original_price": 17.18,
      "original_price_usd": 17.18,
      "discount_percent": 56,
      "rating": 4.5,
      "reviews_count": 1300,
      "position": 1,
      "sold_count": 1500,
      "is_local": true,
      "is_trending": false,
      "free_shipping": true,
      "quick_ship": true,
      "badges": [
        "Bestseller",
        "#1"
      ],
      "color_count": 12
    }
  ],
  "error": "",
  "error_code": ""
}

Top-Level Response Fields

Product Fields

Each item in products represents one product card from the SHEIN page.

Notes About Field Availability

• Not every product card exposes every field.
• price_usd and original_price_usd depend on whether SHEIN provides US price attributes for the card.
• original_price, discount_percent, rating, reviews_count, sold_count, and color_count may be null when the card does not show the corresponding element.
• badges may be an empty array.

Error Codes

Important Implementation Notes

• shein_products_by-category-url uses the input URL directly, then appends the worker query parameters.
• shein_products_by-category-id builds the category page as https://{country}.shein.com/{category_id}.html.
• Relative product detail links are still normalized with https://us.shein.com in the current parser implementation.
• The actual returned fields are determined by main.py.