2026-09-15, Google · Tutorials

Track Google AI Overviews with async batches

A handful of keywords fit in a script that calls Google Search synchronously. A tracker with hundreds or thousands of keywords across markets does not: each synchronous call holds a connection open, and polling that many tasks means thousands of requests. The version built here uses keyword batches of up to 500 GOOGLE tasks with the AI Overview requested, signed webhook deliveries, and a storage model for trending AI Overview citations.

If you are still deciding between synchronous calls, polling and webhooks, read sync calls, async tasks or webhooks first. For the metrics themselves (presence rate, citation rate, organic rank vs citation), see monitoring Google AI Overviews.

What you are tracking

AI Overview tracking records, for each keyword, market and day, whether Google showed an AI Overview and which sources it cited. The unit of work is one GOOGLE task whose payload asks for the overview. Everything else, from alerts to share-of-citation reports, is a query over those stored observations.

Step 1: design the payload

{
  "query": "project management software",
  "country": "US",
  "include": { "aioverview": { "markdown": true } }
}

The decisions that matter:

A market is a country, or a city with location.

Step 2: derive keys and build batches

Each task is one keyword in one market on one day, and that identity is its idempotency key. Hash the keyword text so keys stay short and free of spaces and punctuation, and keep the readable query in your own keyword table:

import { createHash } from "node:crypto";
import { Client, MAX_BATCH, type TaskInput } from "@answerline/sdk";

const client = new Client({ apiKey: process.env.ANSWERLINE_API_KEY! });

type Market = { country: string };

const kwId = (query: string) => createHash("sha256").update(query.trim().toLowerCase()).digest("hex").slice(0, 16);

function buildTasks(keywords: string[], markets: Market[], day: string, webhookUrl: string): TaskInput[] {
  return keywords.flatMap((query) =>
    markets.map((m) => ({
      taskType: "GOOGLE" as const,
      payload: { query, ...m, include: { aioverview: { markdown: true } } },
      idempotencyKey: `aio:${kwId(query)}:${m.country}:${day}`,
      webhook: { url: webhookUrl },
    })),
  );
}

export async function submitDay(keywords: string[], markets: Market[], day: string) {
  const tasks = buildTasks(keywords, markets, day, "https://your-app.example/hooks/aioverview");
  const created: Array<{ key: string; taskId: string }> = [];
  for (let i = 0; i < tasks.length; i += MAX_BATCH) {
    const chunk = tasks.slice(i, i + MAX_BATCH);
    const batch = await client.createBatch(chunk);
    for (const item of batch.results) {
      const key = chunk[item.index].idempotencyKey!;
      if (item.success) created.push({ key, taskId: item.task.id });
      else if (item.error.code === "INSUFFICIENT_CREDITS") throw new Error(`balance ran out at ${key}`);
      else if (item.error.code !== "RESOURCE_ALREADY_EXISTS") console.warn(key, item.error.code, item.error.message);
    }
    await saveCreated(created); // persist per chunk: the reconciliation pass in step 5 needs these ids
  }
  return created;
}

createBatch refuses fewer than 1 or more than 500 tasks before sending, so the loop chunks. Each item in results answers by its index in input order: a VALIDATION_ERROR on one keyword (for example an unsupported country) does not cost you the rest of the batch, and RESOURCE_ALREADY_EXISTS means a previous run already created that task, which is what makes the whole function safe to re-run. Only exceeding your pending-task limit rejects a whole batch, with 429 QUEUE_LIMIT_EXCEEDED and details.remainingCapacity.

The same submission in Python with requests, for teams whose scheduler is Python:

import hashlib
import os
import requests

API = "https://api.answerline.dev"
session = requests.Session()
session.headers["Authorization"] = f"Bearer {os.environ['ANSWERLINE_API_KEY']}"

def kw_id(query: str) -> str:
    return hashlib.sha256(query.strip().lower().encode()).hexdigest()[:16]

def build_tasks(keywords: list, markets: list, day: str, webhook_url: str) -> list:
    return [
        {
            "taskType": "GOOGLE",
            "payload": {"query": q, **m, "include": {"aioverview": {"markdown": True}}},
            "idempotencyKey": f"aio:{kw_id(q)}:{m['country']}:{day}",
            "webhook": {"url": webhook_url},
        }
        for q in keywords
        for m in markets
    ]

def submit(tasks: list) -> list:
    created = []
    for i in range(0, len(tasks), 500):
        chunk = tasks[i:i + 500]
        res = session.post(f"{API}/v1/async/task/batch", json=chunk, timeout=120)
        res.raise_for_status()
        body = res.json()
        for item in body["results"]:
            key = chunk[item["index"]]["idempotencyKey"]
            if item["success"]:
                created.append((key, item["task"]["id"]))
            elif item["error"]["code"] == "INSUFFICIENT_CREDITS":
                raise RuntimeError(f"balance ran out at {key}")
        print(body["summary"])  # {"total": ..., "succeeded": ..., "failed": ...}
    return created

Step 3: receive deliveries

Each finished task is POSTed to the webhook with the same JSON GET /v1/async/task/{taskId} returns. Verify the signature on the raw bytes, skip dashboard test deliveries, store by task.id, and answer within 15 seconds:

import express from "express";
import { verifyWebhook } from "@answerline/sdk";

const app = express();

app.post("/hooks/aioverview", express.raw({ type: "application/json" }), async (req, res) => {
  if (!(await verifyWebhook(req.body, req.get("webhook-signature"), process.env.WEBHOOK_SECRET!))) {
    return res.sendStatus(400);
  }
  const delivery = JSON.parse(req.body.toString("utf8"));
  if (delivery.test) return res.sendStatus(204);
  // insert ... on conflict (task_id) do nothing: deliveries can repeat and arrive out of order
  await rawDeliveries.insertIgnore(delivery.task.id, delivery.task.idempotencyKey, req.body);
  res.sendStatus(204);
});

Store the raw body and parse it in a separate job. Parsing an AI Overview with its organic results, chips and PAA items is more work than you want inside a 15-second budget, and keeping the raw JSON lets you re-derive tables when you change your parsing rules. The webhooks guide covers secret rotation, the retry schedule and the delivery log; verifying webhook signatures has a Python verifier if your receiver is not in TypeScript.

Step 4: parse into citation rows

A completed delivery’s response.result carries organicResults and aioverview side by side. aioverview is null when Google showed no AI Overview for that request; otherwise it can hold:

• text, and markdown when you asked for it.
• sources[]: the source rail, each with position, url, label, description, and redirectLink when Google served the link as a redirect.
• citationPills[]: inline chips. One entry per (chip, source) pair; entries of the same chip share a 1-based citationPillId and carry per-source label, url, domain, and position (the source’s place in sources). Omitted when the answer has no chips.
• relatedLinks[]: a chip’s “View related links” items that are not in sources, grouped by the same citationPillId, with no position. Omitted when no chip has them.
• videos[] and ads[]: videos and sponsored ads Google placed inside the overview.

A parser that produces one observation row per task and one citation row per cited source:

type Observation = {
  key: string; hasOverview: boolean; organicRank: number | null;
  citations: Array<{ domain: string; url: string; position: number; chips: number[] }>;
  relatedDomains: string[]; adsInOverview: number;
};

const hostOf = (url: string) => new URL(url).hostname.toLowerCase();
const onDomain = (url: string, domain: string) => hostOf(url) === domain || hostOf(url).endsWith(`.${domain}`);

export function parse(delivery: any, ourDomain: string): Observation | null {
  if (delivery.task.status !== "COMPLETED") return null; // FAILED: not charged, keep it as a coverage gap
  const { organicResults = [], aioverview } = delivery.response.result;
  const ours = organicResults.find((r: any) => onDomain(r.link, ourDomain));
  const base = { key: delivery.task.idempotencyKey, organicRank: ours?.position ?? null };
  if (!aioverview) return { ...base, hasOverview: false, citations: [], relatedDomains: [], adsInOverview: 0 };

  const chipsBySource = new Map<number, number[]>();
  for (const pill of aioverview.citationPills ?? []) {
    chipsBySource.set(pill.position, [...(chipsBySource.get(pill.position) ?? []), pill.citationPillId]);
  }
  return {
    ...base,
    hasOverview: true,
    citations: (aioverview.sources ?? []).map((s: any) => ({
      domain: hostOf(s.url).replace(/^www\./, ""), url: s.url, position: s.position, chips: chipsBySource.get(s.position) ?? [],
    })),
    relatedDomains: [...new Set((aioverview.relatedLinks ?? []).map((l: any) => l.domain))],
    adsInOverview: (aioverview.ads ?? []).length,
  };
}

chips distinguishes a source that is only in the rail from one Google attached to a sentence in the answer, which matters for reporting. Keep relatedDomains separate from citations: related links are not in the source rail, and the overview’s markdown can contain more URLs than sources and citationPills because related links appear inline too.

Step 5: reconcile missed deliveries

A delivery gets up to 10 attempts over roughly 21 to 43 minutes, then it is marked dead. After each run, compare the task ids saved in step 2 with the ids you stored in step 3, and read the difference with client.getTask(id) (or GET /v1/async/task/{taskId}). File the ones whose task.status is COMPLETED or FAILED through the same parser, and leave the rest for the next pass. Only those two statuses are final; a task can move from PROCESSING back to QUEUED.

Step 6: trend it

With observation and citation rows in place, the useful queries are simple:

• Overview presence per keyword and market over the last 30 runs: how often hasOverview is true.
• Citation share per domain: citations of that domain divided by observations with an overview, per keyword cluster.
• Chip share: the same, counting only citations with at least one chip.
• Rank vs citation quadrants: ranked on page one and cited, ranked but not cited, cited but not ranked, neither. “Cited but not ranked” is where the overview sends visibility to pages organic tracking would never show you.
• Change events: a keyword whose overview appeared or disappeared, or whose cited domain set changed between runs. Require the change to persist for two runs before alerting, because a single observation can flip back.

For the wider picture of how other SERP blocks shift around the overview, see tracking SERP feature changes.

Cost and scheduling

A GOOGLE task is $0.00103 on every tier, include.aioverview adds $0.00071 once, and each extra page adds $0.00071.

Pages are a maximum: the charge depends on the pages Google returned and never exceeds the reserved creditsToCharge. Failed tasks are charged nothing. Run the first row weekly and it is about $4.49 a month; daily, about $31.32. A split that keeps cost in check is money keywords daily and the long tail weekly. Monitoring cadence and cost planning help you choose, and tier prices are on the pricing page.

Trigger submitDay from any scheduler. Tasks wait in the queue and run as capacity frees up, so a large batch takes longer instead of failing.

Pitfalls

1. Forgetting include.aioverview. Without it you get no overview; the parser above would record every keyword as “no overview”.
2. Treating null as a gap. It is a data point: no overview was shown for that request.
3. Keys without the country. US and GB runs of the same keyword collide and the second is silently skipped as RESOURCE_ALREADY_EXISTS.
4. Counting related links as citations. They are not in the source rail.
5. Counting URLs in markdown. It includes related links; use sources and citationPills.
6. Parsing inside the webhook handler. Store, answer 2xx, parse later.

See the Google Search engine page for the full field list, and the async tasks docs for batch details.

Keep reading