Keyword Research API

Who it's for

SEO platforms building keyword databases, content teams expanding topic clusters, and tools scoring keywords by SERP features.

What you get

• People Also Ask questions per keyword, which map the subtopics around it
• Related searches for expansion at the tail
• Which SERP features each keyword triggers, for prioritization
• AI Overview presence as a signal for which queries are shifting to AI answers

Try it

curl -X POST "https://api.answerline.dev/v1/monitor/google" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"email deliverability","country":"US","include":{"aioverview":{"markdown":true}}}'

Sync or queue it with POST /v1/async/task. Full field list on the Google Search page and in the API reference.

Engines for this workload

Keyword research from the result page

For any seed keyword, the result page lists the questions Google groups around it (peopleAlsoAsk), the searches it suggests next (relatedSearches), the pages it considers relevant (organicResults), the features it triggers (local pack, shopping cards, knowledge graph, AI Overview) and, with include.aioverview, the sources its AI summary cites.

That is enough to discover subtopics, cluster keywords by intent, and prioritize by the kind of page that can win. It says nothing about demand: the API does not estimate volume, so join that from the source you already use.

Metrics

Architecture

seeds ─> frontier ─> batch (≤500) ─> webhook ─> parse ─┬─> keywords, edges, features
             ▲                                          │
             └──── new children (depth < max, budget ok) ┘
                                                        ▼
                                         clustering job (SERP similarity)

Expansion is a breadth-first crawl whose frontier is fed by webhook results.

Seed and expansion design

1. Seeds: 10 to 50 head terms per topic, written the way users search.
2. Children of a keyword: its peopleAlsoAsk[].question and relatedSearches[].query values.
3. Depth: the seed is depth 0. Cap at 2 and review a sample of the deepest keywords before allowing more.
4. Relevance filter: expand a child only if it shares a content token with the seeds or passes an embedding-similarity threshold. Without it, related searches wander into neighboring topics.
5. Normalization: lowercase, collapse whitespace, strip a trailing ?. Key on the normalized form and keep the original text for display.
6. Budget: fix the maximum number of tasks before starting and stop enqueueing when it is reached.

Market and language

country sets the result geography and is required. PAA questions and related searches follow it, so a German keyword set needs country: "DE", crawled separately from the English one.

Schedule

Run discovery crawls on demand or monthly. Re-check the feature profile and SERP similarity of an established keyword set monthly or quarterly to catch intent shifts, such as a cluster starting to show AI Overviews or shopping cards.

Idempotency

Each crawl level is a set of GOOGLE tasks submitted through POST /v1/async/task/batch. The webhook handler parses each result and adds children to the frontier, and a small loop submits the frontier.

The idempotency key depends only on the crawl and the keyword:

kw:{crawl_id}:{normalized_keyword_hash}

A keyword discovered from two parents is submitted once; the second submission returns RESOURCE_ALREADY_EXISTS. The crawl can be restarted at any point. See async tasks and webhooks.

Storage schema

create table crawls (
  crawl_id     text primary key,
  country      text not null,
  max_depth    int  not null,
  max_tasks    int  not null,
  with_aio     boolean not null
);

create table keywords (
  crawl_id     text references crawls,
  norm         text not null,
  display      text not null,
  depth        int  not null,
  task_id      uuid unique,
  status       text not null default 'PENDING',   -- PENDING, QUEUED, COMPLETED, FAILED, SKIPPED
  primary key (crawl_id, norm)
);

create table edges (
  crawl_id  text,
  parent    text,
  child     text,
  via       text not null,                         -- 'paa' or 'related'
  primary key (crawl_id, parent, child, via)
);

create table serp_profile (
  task_id       uuid primary key,
  top_urls      text[] not null,
  paa_count     int not null,
  paa_aio_count int not null,
  related_count int not null,
  has_aio       boolean not null,
  aio_domains   text[] not null default '{}',
  has_local     boolean not null,
  has_shopping  boolean not null,
  has_kg        boolean not null,
  has_forum_module boolean not null,
  ads_count     int not null
);

The keywords primary key (crawl_id, norm) deduplicates the crawl itself; serp_profile.task_id absorbs repeated webhook deliveries.

Change detection

For recurring checks of a fixed keyword set, compare each keyword’s serp_profile with the previous one: AI Overview gained or lost, shopping or local features appearing, and top-10 Jaccard similarity against the previous run below a threshold. When many keywords in a cluster change together, intent has shifted and the page targeting that cluster needs a review. SERP features change tracking covers this pattern.

End-to-end Python

Frontier submission

import hashlib, os, requests, psycopg

API = "https://api.answerline.dev"
HEADERS = {"Authorization": f"Bearer {os.environ['API_KEY']}"}
HOOK = {"url": "https://hooks.example.com/keywords"}
db = psycopg.connect(os.environ["DATABASE_URL"], autocommit=True)

def norm(text: str) -> str:
    return " ".join(text.lower().split()).rstrip("? ")

def add_keyword(crawl_id, display, depth, parent=None, via=None):
    n = norm(display)
    db.execute("insert into keywords (crawl_id, norm, display, depth) values (%s,%s,%s,%s) on conflict do nothing",
               (crawl_id, n, display, depth))
    if parent:
        db.execute("insert into edges values (%s,%s,%s,%s) on conflict do nothing", (crawl_id, parent, n, via))

def submit_frontier(crawl_id):
    crawl = db.execute("select country, max_tasks, with_aio from crawls where crawl_id = %s", (crawl_id,)).fetchone()
    country, max_tasks, with_aio = crawl
    used = db.execute("select count(*) from keywords where crawl_id = %s and task_id is not null", (crawl_id,)).fetchone()[0]
    pending = db.execute("select norm, display from keywords where crawl_id = %s and status = 'PENDING' "
                         "order by depth limit %s", (crawl_id, max(0, max_tasks - used))).fetchall()
    for i in range(0, len(pending), 500):
        chunk = pending[i:i + 500]
        tasks = []
        for n, display in chunk:
            payload = {"query": display, "country": country}
            if with_aio:
                payload["include"] = {"aioverview": {"markdown": False}}
            tasks.append({"taskType": "GOOGLE", "payload": payload, "webhook": HOOK,
                          "idempotencyKey": f"kw:{crawl_id}:{hashlib.sha1(n.encode()).hexdigest()}"})
        res = requests.post(f"{API}/v1/async/task/batch", json=tasks, headers=HEADERS, timeout=60)
        res.raise_for_status()
        for item in res.json()["results"]:
            n = chunk[item["index"]][0]
            if item["success"]:
                db.execute("update keywords set task_id = %s, status = 'QUEUED' where crawl_id = %s and norm = %s",
                           (item["task"]["id"], crawl_id, n))
            elif item["error"]["code"] == "VALIDATION_ERROR":
                db.execute("update keywords set status = 'SKIPPED' where crawl_id = %s and norm = %s", (crawl_id, n))
    if used + len(pending) >= max_tasks:
        db.execute("update keywords set status = 'SKIPPED' where crawl_id = %s and status = 'PENDING'", (crawl_id,))

Run submit_frontier on a timer (once a minute is plenty) until no keyword is PENDING or QUEUED. Items refused with INSUFFICIENT_CREDITS stay PENDING and go out on a later pass.

Parse and expand

The receiver verifies Webhook-Signature, drops test deliveries, and passes the body to expand. Signature verification is shown in the AI visibility tracking guide and in webhooks.

from urllib.parse import urlsplit

FORUMS = {"reddit.com", "quora.com", "stackexchange.com", "stackoverflow.com"}
STOPWORDS = {"the", "a", "for", "to", "of", "and", "in", "how", "what", "is"}

def host(url):
    return (urlsplit(url or "").hostname or "").removeprefix("www.")

def expand(d):
    task = d["task"]
    row = db.execute("update keywords set status = %s where task_id = %s and status = 'QUEUED' "
                     "returning crawl_id, norm, depth", (task["status"], task["id"])).fetchone()
    if not row or task["status"] != "COMPLETED":
        return          # duplicate delivery, or failed task
    crawl_id, parent, depth = row
    r = d["response"]["result"]

    organic = sorted(r.get("organicResults", []), key=lambda o: (o.get("page", 1), o.get("position", 0)))
    top_urls = [o["link"] for o in organic[:10]]
    paa = r.get("peopleAlsoAsk", [])
    related = r.get("relatedSearches", [])
    aio = r.get("aioverview")
    db.execute("insert into serp_profile values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) on conflict do nothing", (
        task["id"], top_urls, len(paa), sum(1 for q in paa if q.get("type") == "AIOVERVIEW"), len(related),
        bool(aio), sorted({host(s["url"]) for s in (aio or {}).get("sources", [])}),
        "localResults" in r, bool(r.get("shoppingCards")), "knowledgeGraph" in r,
        "peopleAreSaying" in r or any(host(u) in FORUMS for u in top_urls), len(r.get("ads", [])),
    ))

    max_depth = db.execute("select max_depth from crawls where crawl_id = %s", (crawl_id,)).fetchone()[0]
    if depth >= max_depth:
        return
    seeds = db.execute("select string_agg(norm, ' ') from keywords where crawl_id = %s and depth = 0",
                       (crawl_id,)).fetchone()[0]
    seed_tokens = set(seeds.split()) - STOPWORDS
    children = [(q["question"], "paa") for q in paa] + [(s["query"], "related") for s in related]
    for text, via in children:
        if set(norm(text).split()) & seed_tokens:
            add_keyword(crawl_id, text, depth + 1, parent, via)

The status guard (status = 'QUEUED') makes expand idempotent: a second delivery finds the keyword already final and returns before inserting anything.

Cluster by SERP similarity

from itertools import combinations

def clusters(crawl_id, threshold=0.3, top_n=10):
    rows = db.execute("""select k.display, p.top_urls from keywords k join serp_profile p using (task_id)
                         where k.crawl_id = %s""", (crawl_id,)).fetchall()
    urls = {kw: set(u[:top_n]) for kw, u in rows if u}
    parent = {kw: kw for kw in urls}

    def find(x):
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x

    for a, b in combinations(urls, 2):
        inter = len(urls[a] & urls[b])
        if inter and inter / len(urls[a] | urls[b]) >= threshold:
            parent[find(a)] = find(b)

    groups = {}
    for kw in urls:
        groups.setdefault(find(kw), []).append(kw)
    return sorted(groups.values(), key=len, reverse=True)

Pairwise comparison is quadratic, which is fine for a few thousand keywords. Beyond that, build an inverted index from URL to keywords and compare only keywords that share a URL. Single-linkage union-find can chain loosely related keywords together; if clusters look too broad, raise the threshold or require a minimum shared-URL count. Tune the threshold on a few dozen keyword pairs you have labeled as same or different intent.

Cluster-level metrics

-- After writing clusters to keyword_clusters(crawl_id, cluster_id, norm)
select c.cluster_id,
       count(*)                          as keywords,
       avg(p.has_aio::int)               as aio_presence_rate,
       avg(p.has_shopping::int)          as shopping_rate,
       avg(p.has_forum_module::int)      as forum_presence,
       avg(p.ads_count)                  as ad_density,
       sum(p.paa_count)                  as paa_questions
from keyword_clusters c
join keywords k using (crawl_id, norm)
join serp_profile p using (task_id)
where c.crawl_id = 'crm-2026-09'
group by c.cluster_id
order by keywords desc;

Crawl health is the number of children each keyword links to, per depth. When it falls, the topic is saturating:

select k.depth, count(*) as tasks, count(e.child)::numeric / count(*) as linked_children_per_task
from keywords k
left join edges e on e.crawl_id = k.crawl_id and e.parent = k.norm
where k.crawl_id = 'crm-2026-09' and k.status = 'COMPLETED'
group by k.depth order by k.depth;

Cost planning

At the Starter tier, a Google Search task costs $0.00103 for one page, $0.00071 more per extra page, and $0.00071 more when include.aioverview is set and Google shows an overview (include.paaAioverview is free). Google News tasks cost $0.0002. A failed task is not charged. Higher tiers, reached by 30-day spend, cost less. See balance and pricing.

Worked example: a content team researches one topic from 30 seeds, depth 2, capped at 2,000 keywords.

Clustering needs only page 1, so extra pages rarely pay off for research. Request the AI Overview in the crawl only if AI presence drives prioritization; otherwise add it in the recheck for the keywords you keep.

A new account’s 100 free requests work on Google Search too: 100 keywords on page 1, with or without the AI Overview, since each free request covers a whole request. That is enough to test the crawl on one seed before topping up. Tiers are on pricing.

Pitfalls

Treating structure as demand. A question in People Also Ask shows Google associates it with the topic, not how many people ask it. Join a volume source before ranking content ideas.

Unfiltered related searches. Related searches drift quickly toward adjacent topics and brands. Apply the relevance filter at every depth.

Duplicates that are not identical strings. “crm for small business” and “small business crm” normalize differently. SERP similarity catches them: near-identical top-10 sets mean one page can target both.

Comparing across markets. A cluster built on US results does not transfer to the UK or to another language. Crawl each market separately.

PAA items without answers. snippet, title and link exist for LINK items; markdown and sources exist for AIOVERVIEW items only when include.paaAioverview is true. Read type before reading fields.

Budget overruns from re-crawls. Keys scoped to the crawl id stop duplicates within a crawl, but a new crawl id re-runs everything. Reuse profiles from a recent crawl when the keyword is unchanged.

Missing features read as zeros. localResults, shoppingCards, knowledgeGraph and peopleAreSaying are omitted when absent; aioverview is null. Test for presence.

Which engines fit

People Also Ask keyword research walks through a smaller version of this crawl. The Google Search engine page lists every request field.

Related: SERP keyword data vs search volume data and People Also Ask keyword research.

More use cases