Google Search Results (SERP) Scraper API

Google Search By Keyword

Learn More

What does this scraper return?

It queries Google SERP by keyword and outputs one row per organic result. Public search-level fields are repeated on every row. Organic-result fields change row by row.

How is the final row count determined?

The final output row count is capped by max_results. The scraper estimates the required page count at about 9 organic results per page, starts those page requests concurrently, then continues requesting later pages until it has streamed the requested number of unique rows or no more useful results are found.

Which fields are shared across all rows?

Search-level fields such as keyword, current_page, google_domain, country, language, geo_location, safe_search, results_count_collected, results_total_text, related_queries, and people_also_ask are repeated on each organic-result row.

Which fields are row-specific?

position is rewritten as the continuous output rank after pagination and de-duplication. Other organic-result fields such as title, source_name, display_url, url, clean_url, root_domain, redirect_url, snippet, highlighted_terms, image_alt, has_image, and favicon_url come from the current item in the organic list.

How are proxy and runtime identity handled?

The script reads PROXY_AUTH and PROXY_DOMAIN, builds socks5://{PROXY_AUTH}@{PROXY_DOMAIN}, and sends it as input_proxy.

It also extracts runtime values from PROXY_AUTH when frontend input does not override them:

• task_id: from taskId-...
• user_name: current default account segment
• user_id: current configured default user id unless explicitly provided

Result

This is the shape of one returned row. The full result is still a row list at the platform level because the SDK uses table-style push_data.

[
  {
    "keyword": "OpenAI",
    "current_page": 1,
    "start_position": 0,
    "google_domain": "https://www.google.com/",
    "country": "us",
    "language": "en",
    "geo_location": "w+CAIQICINVW5pdGVkIFN0YXRlcw",
    "safe_search": "off",
    "exclude_autocorrected_results": false,
    "results_filtering": "0",
    "scraped_at": "2026-05-06T18:00:00+08:00",
    "search_url": "https://www.google.com/search?q=OpenAI&gl=us&hl=en",
    "results_count_collected": 2,
    "results_total_text": "1630000000",
    "related_queries": [
      {
        "block_position": 1,
        "topic_title": "OpenAI API",
        "related_search_url": "https://www.google.com/search?q=OpenAI+API",
        "item_position": 1
      }
    ],
    "position": 1,
    "title": "OpenAI",
    "source_name": "OpenAI",
    "display_url": "https://openai.com",
    "url": "https://openai.com?ref=google",
    "clean_url": "https://openai.com",
    "root_domain": "openai.com",
    "redirect_url": "https://www.google.com/url?url=https://openai.com",
    "snippet": "OpenAI official site",
    "highlighted_terms": "OpenAI",
    "image_alt": "",
    "has_image": false,
    "favicon_url": "https://openai.com/favicon.ico",
    "people_also_ask": [
      {
        "position": 1,
        "question": "What is OpenAI?",
        "answer": "AI research company",
        "source_url": "https://openai.com/about"
      }
    ]
  }
]

Dictionary

Nested Arrays

related_queries

people_also_ask

Input

Default Behavior

• domain=https://www.google.com/
• gl=us
• hl=en
• safe=off
• nfpr=0
• filter=0
• max_results=10

Hidden internal defaults still include uule, num=10, render_js, device, retry logic, 5 parallel probes per offset, a second same-offset probe batch when all 5 probes fail, and a lightweight pool capped at 50 in-flight probe requests.

Structure

• request_builder.py: input parsing, normalization, validation, and request payload construction
• google_serp_client.py: spider interface HTTP request execution with retry support
• pagination_runner.py: concurrent internal pagination, unique-row tracking, and ordered streaming output
• response_mapper.py: interface response extraction and row-level output mapping
• main.py: task orchestration and CoreSDK integration
• output_schema.json: output field schema aligned with the current row format

Test

python -m unittest discover -s tests -v

The integration tests start a local mock CoreSDK gRPC server and a local mock spider interface server, then verify payload construction, retry behavior, and row-level output mapping.