2026-09-17, Rank Tracking · Engineering · SERP API

Enterprise rank tracking: collecting millions of keywords reliably

Enterprise rank tracking is a scheduling and data-engineering problem more than a scraping one. At hundreds of thousands to millions of keywords, the questions become: how do you spread collection so it finishes on time, how do you avoid paying twice for the same SERP, how do you keep a queue flowing without overflowing it, where do billions of rows live, and how do you know a run is correct before customers see it. The sections below answer each on this API’s async tasks, with formulas to size your own system from your own measurements.

The shape of the workload

Write the matrix down first:

checks per month = keywords × markets × devices × runs per month

Every multiplier is a product decision. A platform with 400,000 customer keywords, an average of 1.5 markets, desktop and mobile, checked daily, needs 400,000 × 1.5 × 2 × 30 = 36 million checks a month. The same platform checking the long tail weekly might need a third of that. The biggest savings come before any engineering.

Step 1: tier the schedule

Not every keyword deserves a daily check.

Promote and demote automatically: a tail keyword whose position changed by more than a threshold moves to hot for a week. This keeps coverage where change happens.

Spread runs over the day and week

Do not submit a week’s work at midnight on Monday. Assign each keyword a stable slot:

import hashlib

def slot(keyword_id: str, slots: int) -> int:
    return int(hashlib.sha256(keyword_id.encode()).hexdigest(), 16) % slots

# weekly tier: 7 day-slots; daily tier: 24 hour-slots
day = slot("kw-839201", 7)
hour = slot("kw-839201", 24)

A stable hash keeps each keyword on the same day and hour, so its history has even spacing, and load is flat.

Step 2: collect each SERP once

Platforms with many customers track the same queries. A collection key identifies a unique SERP observation:

(normalised query, country, hl, location, device, pages, include flags, date)

Normalise the query (trim, collapse whitespace, lowercase if your product treats case as equal) and deduplicate on the collection key before submission. One task serves every project that tracks that key; results are fanned out by join, not by extra requests.

-- tracked keywords across all tenants for today's slot
select md5(concat_ws('|', lower(trim(query)), country, coalesce(hl, ''), coalesce(location, ''), device,
                     pages::text, include_aio::text, current_date::text)) as collection_key,
       min(query) as query, country, hl, location, device, pages, include_aio,
       array_agg(project_keyword_id) as subscribers
from tracked_keywords
where schedule_day = extract(isodow from current_date) - 1
group by 1, 3, 4, 5, 6, 7, 8;

The collection key doubles as the basis of the idempotency key.

Step 3: submit through a queue feeder

The limits that shape submission

From the async and rate limits docs:

• POST /v1/async/task/batch takes 1 to 500 tasks per request, each admitted independently.
• Requests are counted per account in a one-second window; X-RateLimit-Limit and X-RateLimit-Remaining report yours.
• Async submissions are checked against queue capacity. A batch that would overflow it is rejected whole with 429 QUEUE_LIMIT_EXCEEDED, whose details include queuedCount, batchSize, maxQueueSize and remainingCapacity.
• Concurrency slots bound how many tasks run at once; async tasks wait in the queue for a slot.
• A task has 72 hours from creation to start, else it fails uncharged with QUEUE_WAIT_EXCEEDED.
• priority (1 to 10, default 1) orders your own queue; higher runs first, equal priorities in submission order.

Feed, don’t dump

A feeder keeps the queue at a working depth and never overflows it:

import os
import time

import requests

API = "https://api.answerline.dev"
S = requests.Session()
S.headers["Authorization"] = f"Bearer {os.environ['ANSWERLINE_API_KEY']}"
TARGET_QUEUED = 20_000   # your choice: enough buffer, far below capacity

def queued() -> int:
    return S.get(f"{API}/v1/async/status", timeout=30).json()["queuedTasks"]

def feed(pending: "Iterator[list[dict]]", record) -> None:
    for batch in pending:                     # batches of <= 500 tasks, highest priority first
        while queued() > TARGET_QUEUED:
            time.sleep(15)
        resp = S.post(f"{API}/v1/async/task/batch", json=batch, timeout=60)
        if resp.status_code == 429:
            wait = int(resp.headers.get("Retry-After", 5))
            time.sleep(wait)
            resp = S.post(f"{API}/v1/async/task/batch", json=batch, timeout=60)  # same keys: safe to resend
        resp.raise_for_status()
        record(batch, resp.json()["results"])

In production, wrap the retry in the fuller policy from API errors and retries, and read details.remainingCapacity from a QUEUE_LIMIT_EXCEEDED response to size the next batch.

Idempotency keys at scale

Build keys from the collection key and run:

def idem_key(collection_key: str, attempt: int = 0) -> str:
    return f"serp:{collection_key}" + (f":r{attempt}" if attempt else "")

The collection key already includes the date, so re-running a crashed feeder re-submits nothing that was admitted: those items return RESOURCE_ALREADY_EXISTS. Only deliberate resubmissions of failed tasks get a suffix.

Priorities

Use a small number of levels with clear meaning, for example 8 for customer-triggered refreshes, 5 for hot, 3 for core, 1 for tail and deep. Priorities only order your own queue, so a large backfill at priority 1 never delays a customer refresh at 8.

Step 4: size throughput from your own measurements

Daily capacity follows from concurrency and task duration:

tasks per day ≈ concurrency × 86,400 ÷ D

where D is your mean seconds per task from first start to completion. Measure D, do not assume it: every completed task reports task.latencyMs (first start to final outcome, excluding queue time), and monitor responses carry X-Latency-Ms. Duration depends on engine, pages, AI Overview and market, so measure per task profile.

Then plan:

1. Compute daily checks from the tiered schedule.
2. Divide by capacity from the formula. Keep utilisation well below 100% to leave room for retries and slow days.
3. If a day does not fit, move tiers, not deadlines: shift core to fewer runs or tail to biweekly, or move to a plan with more concurrency (listed on /docs/rate-limits).

The 72-hour start bound is generous for weekly schedules and irrelevant for daily ones if the day fits. If you find tasks approaching it, the schedule does not fit the concurrency.

Step 5: collect results

At this volume, polling task by task is wasteful. Use webhooks:

1. Add webhook: {"url": ...} to every task.
2. Run several receiver instances behind a load balancer. Each verifies Webhook-Signature on the raw body, writes the body to durable storage keyed by task.id, and answers 2xx within 15 seconds.
3. Process asynchronously from that storage.
4. Reconcile: for tasks whose results have not arrived after their expected finish, fetch GET /v1/async/task/{taskId}.

Webhook receiver design covers the receiver; SERP API reliability at scale covers the ledger that makes reconciliation cheap.

Step 6: storage layout

Two layers

1. Raw layer. The full webhook body per task, compressed, in object storage, path-partitioned by date: raw/google/2026/09/17/<task_id>.json.gz. Keep the request payload beside it. The raw layer lets you re-parse history when you add a field or fix an extraction bug, without paying to collect again.
2. Modeled layer. Extracted rows in a columnar warehouse:

Partition by observed_date, cluster by country and collection_key (or domain for competitor queries). SERP data in BigQuery has DDL and load code for one warehouse.

Volume estimate

Organic rows dominate: about 10 per page per observation. 36 million observations a month at one page is roughly 360 million organic rows a month, plus features. Estimate from your own observed average row counts and bytes, and set retention per layer: modeled rows for as long as customers chart history, raw bodies for as long as you might re-parse.

Positions: define them once

Decide and document whether “position” counts only organic results (as organicResults[].position does) or includes features. Store organic position as collected and compute any “absolute” position in a view, so the definition can change without rewriting history.

Step 7: data quality before customers see it

At millions of rows, a parsing change or a misconfigured market shows up as millions of false movements. Gate publication on checks:

1. Completeness. Share of planned collection keys with a stored, valid observation. Publish per tier only above your threshold; show gaps as gaps, never interpolate silently.
2. Structural validity. Organic count within the keyword’s historical range, positions increasing, links present.
3. Distribution stability. Per market and device, compare today’s median organic count, AI Overview rate, local pack rate and ads rate with the trailing week. A shift across thousands of keywords at once is a pipeline signal until proven otherwise.
4. Canaries. A fixed set of keywords with well-understood SERPs in every market and slot.
5. Movement sanity. Share of keywords whose best position changed by more than N compared with normal days. Hold the run for review when it spikes.
6. Target assertions. The stored payload’s country, location, device and hl match the plan.
7. Duplicates. One observation per collection key per day; duplicates signal a key-construction bug.

Failed tasks are not charged, and resubmitting them under a new key suffix within the run window closes most completeness gaps. Budget resubmissions per run; a large failure share is an incident to alert on, not something to retry through.

Cost at scale

Credit costs: Google Search async task 3 credits for one page; +2 per extra page; +2 for include.aioverview; synchronous calls +2 (avoid them for bulk work).

Shared collection reduces the observation count before these multipliers apply; measure your overlap rate across tenants, because on multi-tenant platforms it is often the largest single saving. Plans and enterprise options are on /pricing, and cost planning walks through budgeting.

Operating it

• Dashboards: completeness per tier and market, queue depth from GET /v1/async/status, inbox lag, failure codes, credits from GET /v1/credits against the month’s plan.
• Runbooks: a stalled queue (check concurrency in use, then engine status), a failure spike (pause the feeder, keep collecting results, resubmit after recovery), a parsing regression (hold publication, re-parse from the raw layer).
• Abort: DELETE /v1/async/queue removes every task still queued, which was never charged, and leaves running tasks alone. Record the abort so completeness reflects it.
• Tracing: send your own X-Request-Id per submission so logs join across systems.

Pitfalls

• Dumping the week at once. Feed the queue; spread slots.
• Paying per tenant for the same SERP. Deduplicate on a collection key.
• Random idempotency keys. Re-running a feeder then creates duplicates.
• Assumed task duration. Measure latencyMs per task profile before promising schedules.
• No raw layer. Every extraction fix then requires paid re-collection.
• Publishing before validating. One bad run becomes millions of false alerts.
• Interpolating gaps. Show them; they are information about the pipeline.

See the rank tracking use case and the async tasks docs to start.

Keep reading