2026-09-17, Engineering · SERP API · Rank Tracking

SERP API reliability at scale: retries, idempotency, billing and completeness

A reliable SERP pipeline is one that can prove every planned request for a run ended as a stored, valid result, or knows exactly which did not and why. Retries and idempotency are the mechanisms; completeness is the measure. The pieces below use this API’s async tasks: a ledger of planned work, submission that is safe to repeat, a retry budget per failure class, billing rules you can reconcile, data-quality checks that catch “successful” empty pages, and alerts. Success rates are yours to measure on your own keyword set.

Why request success is the wrong metric

Most SERP monitoring starts with HTTP status codes. At scale, the failures that hurt are invisible to them:

1. Never submitted. A scheduler crashed after building 40,000 of 60,000 tasks.
2. Submitted, never received. A webhook endpoint returned 500 through its retry window during a deploy.
3. Received, not stored. A worker threw on an unexpected field and dropped the message.
4. Stored, not valid. The result parsed, but the organic list is empty for a keyword that always has ten results.
5. Valid, wrong target. The task ran for the wrong market because a config change swapped country codes.

Each of these reports 100% HTTP success. A completeness metric catches all five:

completeness(run) = tasks with a valid stored result ÷ tasks planned for the run

The ledger: plan before you submit

Write the plan down before touching the API. One row per planned task:

create table serp_ledger (
  run_id        text not null,              -- e.g. '2026-09-17'
  task_key      text not null,              -- idempotency key, see below
  task_type     text not null,              -- GOOGLE, GOOGLE_NEWS, CHATGPT, ...
  payload       jsonb not null,
  task_id       uuid,                       -- set when the API admits it
  state         text not null default 'planned',
                                            -- planned | queued | completed | failed | invalid | stored
  attempts      int not null default 0,
  last_error    text,
  credits       int,
  updated_at    timestamptz not null default now(),
  primary key (run_id, task_key)
);
create index on serp_ledger (run_id, state);

The ledger turns every question about a run into a query: how many are still queued, which failed and with what, which came back invalid, what the run cost.

Idempotent submission

Build keys from meaning

An idempotency key identifies what a task is, not the attempt:

def task_key(run_id: str, surface: str, keyword_id: str, market: str, device: str, attempt: int = 0) -> str:
    base = f"{surface}:{run_id}:{keyword_id}:{market}:{device}"
    return base if attempt == 0 else f"{base}:r{attempt}"

Keys are unique across your account. Creating a task with a key that exists answers 409 RESOURCE_CONFLICT (in a batch, the item fails with RESOURCE_ALREADY_EXISTS) and creates nothing. So a scheduler that crashes halfway can resubmit the whole run; everything already admitted is refused, everything missing is created. The :r1 suffix is only for deliberate resubmission of a task that ran and failed, covered below. Idempotency keys explained goes deeper.

A useful property: a request whose every task is refused, for example for INSUFFICIENT_CREDITS, creates nothing and claims no key, so the same keys can be sent again after a top-up.

Submit in batches, record every item

POST /v1/async/task/batch takes 1 to 500 tasks. Each is validated and admitted on its own; results[] answers each by index in input order.

import os
import time
import random

import requests

API = "https://api.answerline.dev"
S = requests.Session()
S.headers["Authorization"] = f"Bearer {os.environ['ANSWERLINE_API_KEY']}"

def post_batch(chunk: list[dict]) -> dict:
    for attempt in range(6):
        try:
            resp = S.post(f"{API}/v1/async/task/batch", json=chunk, timeout=60)
        except requests.ConnectionError:
            resp = None  # safe to resend: every task carries an idempotency key
        if resp is not None and resp.status_code == 200:
            return resp.json()
        if resp is not None and resp.status_code not in (429, 500, 502, 503, 504):
            resp.raise_for_status()
        wait = int(resp.headers.get("Retry-After", 0)) if resp is not None else 0
        time.sleep(max(wait, random.uniform(0, min(60, 2 ** attempt))))
    raise RuntimeError("batch not accepted after retries")

def submit(db, run_id: str, rows: list[dict]) -> None:
    for i in range(0, len(rows), 500):
        chunk = rows[i:i + 500]
        body = post_batch([
            {"taskType": r["task_type"], "payload": r["payload"], "idempotencyKey": r["task_key"],
             "webhook": {"url": "https://collector.example.com/hooks/serp"}}
            for r in chunk
        ])
        for item in body["results"]:
            row = chunk[item["index"]]
            if item["success"]:
                db.mark_queued(run_id, row["task_key"], item["task"]["id"], item["credits"]["creditsToCharge"])
            elif item["error"]["code"] == "RESOURCE_ALREADY_EXISTS":
                db.mark_exists(run_id, row["task_key"])       # admitted on an earlier attempt
            else:
                db.mark_failed(run_id, row["task_key"], item["error"]["code"])

Two things to handle deliberately:

• RESOURCE_ALREADY_EXISTS with no stored task id. This happens when an earlier submission was admitted but its response was lost. Keep the row as queued and let the webhook fill in the id when the result arrives, since the delivery body carries task.idempotencyKey.
• QUEUE_LIMIT_EXCEEDED. A 429 that rejects the whole batch. Its details include queuedCount, maxQueueSize and remainingCapacity; wait for the queue to drain before resending, rather than retrying on a timer.

A retry budget per failure class

Retrying everything is how pipelines melt down during an upstream incident. Classify, then set a budget per class.

The source for the HTTP rows is the error codes table; API errors and retries has full wrappers in Python and TypeScript.

Task-level retries are budgeted

Per the async docs, transient failures are retried automatically within five minutes of a task’s first start. When a task still ends FAILED, it is not charged, and resubmitting is your decision. Set the budget per run, not per task:

• Resubmit FAILED tasks at most twice (:r1, :r2).
• Stop resubmitting when more than a set share of the run has failed, because that is an incident, not noise. Alert instead.
• Spread resubmissions over the remaining run window rather than firing them all at once.

Deadlines you can plan around

Every task finishes within documented bounds (async docs):

• 72 hours from creation to start. A task still QUEUED then moves to FAILED with 502 EXTERNAL_SERVICE_ERROR and details.cause QUEUE_WAIT_EXCEEDED, uncharged.
• Five minutes from first start to finish. A task that runs out of time fails with Maximum retries exceeded, uncharged.
• A start can be taken back. A poller can see PROCESSING then QUEUED again. Treat only COMPLETED and FAILED as final.

These bounds mean a run always converges: after the deadline every ledger row is either final or was never admitted. Set your run’s completeness deadline accordingly, and size submissions so your concurrency can drain them well inside it.

Collection: webhooks plus reconciliation

Receive results by webhook, and never rely on webhooks alone.

1. Verify, store, acknowledge. Verify Webhook-Signature on the raw body, write the body to an inbox table keyed by task.id, answer 2xx. A delivery needs a 2xx within 15 seconds; otherwise it is retried, up to 10 attempts, the last about 21 to 43 minutes after the first.
2. Deduplicate on task.id. Deliveries can repeat and arrive in any order. Webhook-Id is not signed; task.id is.
3. Process from the inbox into the ledger and your results store.
4. Reconcile. On a timer, select ledger rows still queued whose expected finish has passed, and fetch each with GET /v1/async/task/{taskId}. Ingest what is final.
5. Redeliver after outages. The dashboard’s delivery log lets owners redeliver delivered or dead deliveries once an endpoint is fixed; reconciliation covers the same gap without manual steps.

Webhook receiver design builds the inbox and worker; sync, async or webhooks explains when polling is simpler.

Billing on failure: reconcile it

This API’s rules (credits docs):

• An accepted request reserves its maximum possible cost, shown as credits.creditsToCharge.
• On success it is charged the actual cost, which never exceeds the reservation, shown as credits.creditsCharged and in the X-Credits-Charged header on monitor responses.
• A request or task that fails is charged nothing, and its reservation is released.
• Tasks cleared from the queue with DELETE /v1/async/queue were never charged.

Other vendors bill differently, some charging for specific 4xx responses; cheapest SERP API lists several policies with sources. When you run more than one provider, store the charge per task so a failure-heavy day shows up as a cost anomaly, not just a quality one.

A daily reconciliation query:

select run_id,
       count(*) filter (where state = 'stored')                as stored,
       count(*) filter (where state = 'failed')                as failed,
       sum(credits) filter (where state = 'stored')            as credits_charged,
       sum(credits) filter (where state <> 'stored')           as credits_on_non_stored
from serp_ledger
group by run_id
order by run_id desc
limit 14;

credits_on_non_stored should be zero with this API’s rules once rows record creditsCharged from the result body. If it is not, you are recording reservations as charges, or dropping results after paying for them (the invalid state).

GET /v1/credits returns remaining, perCycle and cycleResetsAt; compare the balance before and after a run with the ledger’s sum.

Data-quality checks: “successful” is not “valid”

A COMPLETED task means the page was read. It does not mean the page is what you wanted. Validate before a row counts toward completeness.

Structural checks

def validate_google(result: dict, expect_organic: int = 5) -> list[str]:
    problems = []
    organic = result.get("organicResults", [])
    if len(organic) < expect_organic:
        problems.append(f"organic_count={len(organic)}")
    positions = [r.get("position") for r in organic]
    if positions != sorted(positions):
        problems.append("positions_not_monotonic")
    if any(not r.get("link") for r in organic):
        problems.append("missing_link")
    return problems

Set expect_organic per keyword from history rather than globally: some queries genuinely return few results.

Distribution checks per run

Single results can be legitimately odd. Runs should not be:

• Median organic count across the run versus the trailing seven runs.
• Share of results with aioverview present, with localResults, with ads, compared with recent runs for the same keyword set.
• Share of results where your tracked domain appears at all.

A sudden shift across thousands of keywords is more often a collection or parsing change than a genuine SERP change. Hold the run’s reporting and investigate before publishing movements to users. SERP feature change tracking separates feature changes from noise.

Canary keywords

Include a small fixed set of keywords with stable, well-understood SERPs in every run, in every market. If canaries change, suspect the pipeline first.

Target checks

Store the request payload next to the result and assert that what you asked for is what you planned: country, location, device, hl. Configuration drift shows up here, not in HTTP codes.

Throughput and backpressure

• Concurrency slots bound running jobs. Async tasks take a slot only when they start and otherwise wait in your queue; synchronous calls fail at once with CONCURRENT_LIMIT_EXCEEDED when slots are busy. For scheduled work, use tasks.
• Priority (1 to 10, default 1, higher first) orders your own queue. Give the interactive or client-facing subset a higher priority than bulk backfills.
• Queue capacity rejects submissions that would overflow it. Watch GET /v1/async/status for queuedTasks, processingTasks, priorityBreakdown and concurrency.
• Abort cleanly. DELETE /v1/async/queue removes every task still QUEUED (uncharged) and leaves running and finished tasks alone; it is safe to repeat. Mark the affected ledger rows so completeness reflects the abort.

Rate limits lists the headers to read for throttling before you hit a limit.

What to monitor and alert on

task.latencyMs runs from first start to final outcome and excludes queue time, so trend queue wait separately from createdAt to completion.

Send your own X-Request-Id on API calls (1 to 128 characters) so a failing request can be traced in your logs and quoted to support.

Checklist

1. Plan every task into a ledger before submitting.
2. Build idempotency keys from meaning; add a suffix only for deliberate resubmission.
3. Submit in batches of up to 500 and record every item’s outcome.
4. Classify failures and give each class a retry rule and a budget.
5. Receive by signed webhook, dedupe on task.id, and reconcile by polling.
6. Record creditsCharged per task and reconcile against the balance.
7. Validate results structurally, per run and with canaries before counting them.
8. Alert on completeness and its components, not on HTTP success.

For warehouse loading of the results, see SERP data in BigQuery; for the volume side, enterprise rank tracking. Start with the async tasks docs.

Keep reading