2026-09-17, Keyword Research · SERP API · Google

Find your competitors' keywords from live SERP data

To find a competitor’s keywords from SERP data, you invert the usual keyword-tool workflow. Instead of asking a database “what does billfox.com rank for”, you build a large candidate set around your market, run every candidate through Google, and record where each competitor’s domain appears in organicResults and ads. The output is exact and current for the keywords you checked, in the market and device you chose. It is not exhaustive, and it has no demand numbers attached.

This API returns no search volume and no keyword difficulty, only what the results page showed. Prioritising the gaps you find needs a volume source, covered at the end.

The method in six steps

1. Seed the candidate set from your own terms, the competitor’s positioning and their brand name.
2. Expand it with relatedSearches[].query and peopleAlsoAsk[].question.
3. Run every candidate and store organic and ad placements per domain.
4. Probe competitor topics with site: queries to find pages and themes you missed.
5. Read ads to see which candidates they pay for.
6. Build an overlap and gap matrix, then join volume and prioritise.

Placeholder names: you are ledgerly.com; rivals are billfox.com and invoicepro.io.

Step 1: seeds

Good seeds come from four places:

Aim for 30 to 100 seeds per product area. Normalise (lowercase, collapse whitespace) and give each an ID.

Step 2: expansion with related searches and People Also Ask

Every Google Search response carries two expansion sources:

• relatedSearches[] with query: searches Google suggests next.
• peopleAlsoAsk[] with question: questions Google groups around the query.

One expansion level is usually enough for competitor keyword discovery, because the goal is a wide candidate set to test, not a deep topic map. Filter children for relevance (share at least one content token with a seed, or pass an embedding threshold), or related searches drift into neighbouring categories. For deeper crawls with clustering, see keyword research and People Also Ask keyword research.

Because the seed run already returns organic results and ads, seeds cost nothing extra in step 3: the same response feeds both expansion and competitor matching.

Step 3: run candidates and match domains

Submit all candidates as async tasks. The same code submits seeds and expanded keywords.

import hashlib, os, requests

API = "https://api.answerline.dev"
HEADERS = {"Authorization": f"Bearer {os.environ['API_KEY']}"}

def submit(candidates, country="US", run="ckw-2026-09"):
    ids = {}
    for i in range(0, len(candidates), 500):
        chunk = candidates[i:i + 500]
        tasks = [{
            "taskType": "GOOGLE",
            "payload": {"query": kw, "country": country, "pages": 2},
            "idempotencyKey": f"{run}:{country}:{hashlib.sha1(kw.encode()).hexdigest()[:16]}",
            "webhook": {"url": "https://seo.example.com/hooks/ckw"},
        } for kw in chunk]
        res = requests.post(f"{API}/v1/async/task/batch", json=tasks, headers=HEADERS, timeout=60)
        res.raise_for_status()
        for item in res.json()["results"]:
            if item["success"]:
                ids[item["task"]["id"]] = chunk[item["index"]]
    return ids

pages: 2 checks the top 20, which catches competitor pages sitting just off page one, the most actionable gaps. Each extra page adds 2 credits; use pages: 1 if you only care about the top 10.

When a result arrives (by webhook, verified as in verifying webhook signatures), reduce it to per-domain rows:

from urllib.parse import urlsplit

COMPETITORS = {
    "you": {"ledgerly.com"},
    "billfox": {"billfox.com", "help.billfox.com"},
    "invoicepro": {"invoicepro.io"},
}

def host(url):
    return (urlsplit(url or "").hostname or "").removeprefix("www.")

def owner(domain):
    for name, domains in COMPETITORS.items():
        if domain in domains or any(domain.endswith("." + d) for d in domains):
            return name
    return None

def reduce(keyword, result):
    organic = sorted(result.get("organicResults", []), key=lambda o: (o.get("page", 1), o.get("position", 0)))
    best = {}
    for rank, o in enumerate(organic, 1):
        name = owner(host(o["link"]))
        if name and name not in best:
            best[name] = {"rank": rank, "url": o["link"]}
    advertisers = {owner(a.get("domain") or host(a.get("url"))) for a in result.get("ads", [])
                   if a.get("type") == "RESULT"} - {None}
    children = [s["query"] for s in result.get("relatedSearches", [])]
    children += [q["question"] for q in result.get("peopleAlsoAsk", [])]
    return {"keyword": keyword, "best": best, "advertisers": sorted(advertisers), "children": children}

best holds each tracked competitor’s best organic rank and ranking URL for the keyword. Store every other domain too if you want to discover competitors you did not list; the most frequent unknown domains across the candidate set are usually worth adding.

Step 4: site: queries to probe a competitor’s topics

Google interprets operators in the query text, and Google’s own help page documents site: for restricting results to a site, quotes for exact matches and - to exclude a word (checked 2026-09-17). That makes a query like this useful:

curl -X POST https://api.answerline.dev/v1/monitor/google \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "query": "site:billfox.com recurring invoices", "country": "US" }'

Read organicResults[].title and organicResults[].link. Titles of a competitor’s pages on a topic are a direct source of keyword candidates: feature names, integrations, template pages, glossary terms, comparison pages you did not know existed. Useful probes:

Treat these as a sample. A site: search returns what Google chooses to show for that query, not an export of the competitor’s indexed pages, and it says nothing about which keywords those pages rank for. Feed the titles back into step 3 as candidates, where rank is measured on the real query.

Step 5: ads show paid keywords

For every candidate you already have ads[]. Text ads (type RESULT) carry domain, blockPosition, title and description. A competitor’s domain appearing there means their campaigns showed an ad for that search in that market at that moment; whether they bid on the exact keyword or a broader match is not visible.

The advertisers on a page can differ between searches, so one sample under-counts. For the candidates where paid competition matters (brand terms, high-intent category terms), run two or three extra samples on different days and compute:

ad presence rate = samples with their ad ÷ samples taken

Keywords with a high ad presence rate and no organic ranking for that competitor are keywords they pay for because they cannot rank. Those are often good organic targets for you: commercial intent is confirmed by someone’s budget, and the organic slot is not theirs.

Step 6: overlap and gap matrix

With best and advertisers per keyword, build one row per keyword:

(Illustrative rows.) Classes:

def classify(row, you="you", rival="billfox"):
    y, r = row["best"].get(you, {}).get("rank"), row["best"].get(rival, {}).get("rank")
    if r and r <= 10 and not y:
        return "gap"
    if r and r <= 10 and y and y > 10:
        return "near_gap"
    if y and y <= 10 and (not r or r > y):
        return "strength" if not r or r > 10 else "shared_ahead"
    if y and r and y <= 10 and r < y:
        return "shared_behind"
    if rival in row["advertisers"] and not r:
        return "paid_only"
    return "untouched"

Overlap between you and a rival = keywords where both rank ≤ 10 ÷ keywords where either does. It tells you how directly you compete in search, which is often different from how directly you compete in sales.

Adding volume and difficulty

Result pages tell you who ranks; they do not tell you whether anyone searches. Before prioritising the gap list, join a demand source:

• Google Keyword Planner. Google’s help page says it shows “estimates on the number of searches a keyword gets each month” and average cost data, and that you must complete Google Ads account setup, including billing information, to access features such as getting keyword ideas (Google Ads Help, checked 2026-09-17). Upload your gap and near-gap keywords to get estimates.
• Google Search Console. The Performance report shows clicks, impressions, CTR and average position for your own site in Google Search (Search Console Help, checked 2026-09-17). It covers only queries where your site already appears, which makes it the right check for “near gap” and “shared, behind” rows, not for true gaps.
• Commercial keyword databases. SEO suites sell volume and difficulty estimates and their own competitor keyword lists. SpyFu’s homepage, for example, says “Download their SEO keywords”, and Ahrefs lists a $29-per-month Starter plan described as helping you “spy on competitors” (SpyFu, Ahrefs pricing, checked 2026-09-17). Their lists are a good source of additional candidates for step 3, where you then measure positions yourself, in your markets. More tools are compared in competitive intelligence tools by job.

A simple priority score once volume is joined:

priority = volume × intent_weight × gap_weight

with gap_weight highest for near gaps (a page exists) and paid-only keywords (intent proven by spend), and intent_weight set per keyword group. Keep keyword difficulty, if you use one, as a separate column from the vendor that produced it; it is a model, not an observation.

Cost

A Google Search async task costs 3 credits for one page and 2 more per extra page. Synchronous calls add 2.

Worked example for two competitors in one market:

Re-running the same candidate set monthly to watch the gap close costs the 6,400 credits of the first two rows again, or 3,840 at one page. A free account’s 500 monthly credits cover the 80-seed run and a handful of probes, enough to validate the matching code. See pricing.

Pitfalls

• Treating “not found” as “doesn’t rank”. It means not within the pages you requested, for that market and device.
• Subdomains and country sites. List all of a competitor’s hosts, or their help centre and regional sites look like unknown domains.
• One market for all conclusions. A gap in the US can be a strength in the UK. Run each country separately.
• Reading site: results as rankings. Probes find pages and themes; ranks come from the real queries.
• One ad sample. Use presence rates across samples.
• Prioritising without volume. A long gap list with no demand data sends writers after keywords nobody searches.

To keep watching these keywords after the analysis, see competitor SEO tracking. The request and response fields are on the Google Search engine page.

Keep reading