2026-09-17, ChatGPT · Brand Monitoring · Tutorials

ChatGPT visibility tracker: metrics, prompt sets, sampling and a working pipeline

A ChatGPT visibility tracker runs a fixed set of buyer prompts through ChatGPT on a schedule, several times per prompt and market, and reduces each answer to a few facts: was the brand mentioned, was its site cited and where, did its products appear. Those rates, over time and against competitors, are the visibility metric. You can buy this as a platform (Peec AI, Otterly.ai, Profound, Scrunch, AthenaHQ, Semrush, Ahrefs Brand Radar) or build it on an answer API. The method comes first here, since it determines whether either approach gives numbers you can trust; then a Python pipeline on AnswerLine; then the off-the-shelf options.

Third-party facts are from the vendors’ own pages, checked 2026-09-17, with links.

Options

Metrics

Define these before collecting anything. Each maps to one part of the ChatGPT answer.

To keep them comparable over time:

1. Only completed answers go in the denominator. A failed collection is not an answer without you.
2. Mentions and citations stay separate. A brand can be recommended without a link and linked without a recommendation. See mentions vs citations.
3. The competitor set is fixed and versioned, because share of voice changes meaning whenever the set changes. The share of voice framework covers the arithmetic.

Building the prompt set

The prompt set decides what “visibility” means for you.

1. Start from buyer questions rather than keywords. Sales calls, support tickets, site search and People Also Ask are better sources than a keyword list; People Also Ask for prompt research shows one method.
2. Cover the funnel: category questions (“best payroll software for small business”), comparisons (“Gusto vs ADP”), problems (“how to run payroll for contractors in two states”), and brand questions (“is Acme Payroll good for restaurants”).
3. Tag every prompt with topic, funnel stage and whether it names a brand. Unbranded prompts measure discovery and branded prompts measure reputation, so report them separately.
4. Write prompts the way people type: short, conversational, sometimes with context (“for a 20-person agency”). Do not name your brand in unbranded prompts.
5. Size by topic. Ten to twenty prompts per topic gives a rate that no single phrasing dominates.
6. Keep a stable core. Changing prompts breaks trend lines, so add new prompts as a separate cohort and fold them in after a few weeks.

Prompt set design goes deeper on balance and size.

Sampling: how many runs you need

ChatGPT’s answers vary between runs of the same prompt, so a single daily run per prompt measures that noise as much as your visibility. Treat each answer as a sample and report rates over pooled samples.

For a rate p measured over n answers, the 95% margin of error is about 1.96 × √(p(1−p)/n):

In practice:

• Pool before you read: weekly rates per topic, not daily rates per prompt.
• More distinct prompts reduce phrasing bias; more runs per prompt reduce run-to-run noise. Three runs a day across 15 prompts in a topic gives 315 answers per topic per week.
• Keep markets separate. Pooling the US and Germany mixes two populations.

The sampling vs census post and AI answer volatility cover this in more depth.

A pipeline on AnswerLine

One Python file with three commands: submit queues today’s runs, collect fetches finished tasks and extracts brand signals, and report prints weekly rates. It uses requests and SQLite against the HTTP API, and polls for results to stay self-contained; at larger volume, use a webhook instead.

Inputs

prompts.csv:

id,topic,prompt
p001,category,best payroll software for small business
p002,category,easiest payroll software for restaurants
p003,comparison,Gusto vs ADP for a 20-person company

The script

import csv, datetime, os, re, sqlite3, sys
from urllib.parse import urlparse

import requests

API = "https://api.answerline.dev"
COUNTRIES = ["US"]
RUNS_PER_DAY = 3
BRANDS = {
    "acme": {"patterns": [r"\bacme payroll\b"], "domain": "acmepayroll.com"},
    "gusto": {"patterns": [r"\bgusto\b"], "domain": "gusto.com"},
    "adp": {"patterns": [r"\badp\b"], "domain": "adp.com"},
}

http = requests.Session()
http.headers["Authorization"] = f"Bearer {os.environ['API_KEY']}"
db = sqlite3.connect("chatgpt_visibility.db")
db.executescript("""
create table if not exists tasks (
  task_id text primary key, prompt_id text, topic text, country text, day text, run int, status text);
create table if not exists hits (
  task_id text, brand text, mentioned int, cited int, best_position real, in_products int,
  primary key (task_id, brand));
create table if not exists fanout (task_id text, query text);
""")


def submit():
    day = datetime.date.today().isoformat()
    with open("prompts.csv", newline="", encoding="utf-8") as f:
        prompts = list(csv.DictReader(f))
    meta, tasks = [], []
    for p in prompts:
        for country in COUNTRIES:
            for run in range(RUNS_PER_DAY):
                meta.append((p["id"], p["topic"], country, day, run))
                tasks.append({
                    "taskType": "CHATGPT",
                    "payload": {
                        "prompt": p["prompt"],
                        "country": country,
                        "include": {"searchQueries": True, "shopping": True},
                    },
                    "idempotencyKey": f"{p['id']}-{country}-{day}-r{run}",
                })
    for start in range(0, len(tasks), 500):
        res = http.post(f"{API}/v1/async/task/batch", json=tasks[start:start + 500], timeout=60)
        res.raise_for_status()
        for item in res.json()["results"]:
            m = meta[start + item["index"]]
            if item["success"]:
                db.execute("insert or ignore into tasks values (?,?,?,?,?,?,?)",
                           (item["task"]["id"], *m, "QUEUED"))
            else:
                print("not queued:", m, item["error"]["code"])
    db.commit()


def on_domain(url, domain):
    host = (urlparse(url).hostname or "").lower()
    return host == domain or host.endswith("." + domain)


def product_text(result):
    parts = []
    for card in result.get("shoppingCards", []):
        for prod in card.get("products", []):
            parts += [prod.get("title", ""), prod.get("merchant", "")]
    for item in result.get("inlineProducts", []):
        prod = item.get("product", {})
        parts.append(prod.get("title", ""))
        parts += [o.get("merchant_name", "") for o in prod.get("offers", [])]
    return " ".join(p for p in parts if p).lower()


def extract(task_id, result):
    text = result.get("text", "").lower()
    products = product_text(result)
    for brand, cfg in BRANDS.items():
        mentioned = any(re.search(p, text) for p in cfg["patterns"])
        positions = [s["position"] for s in result.get("sources", [])
                     if on_domain(s.get("url", ""), cfg["domain"])]
        in_products = any(re.search(p, products) for p in cfg["patterns"])
        db.execute("insert or replace into hits values (?,?,?,?,?,?)",
                   (task_id, brand, mentioned, bool(positions),
                    min(positions) if positions else None, in_products))
    db.executemany("insert into fanout values (?,?)",
                   [(task_id, q) for q in result.get("searchQueries", [])])


def collect():
    pending = db.execute("select task_id from tasks where status in ('QUEUED','PROCESSING')").fetchall()
    for (task_id,) in pending:
        res = http.get(f"{API}/v1/async/task/{task_id}", timeout=30)
        res.raise_for_status()
        body = res.json()
        status = body["task"]["status"]
        if status == "COMPLETED":
            extract(task_id, body["response"]["result"])
        db.execute("update tasks set status = ? where task_id = ?", (status, task_id))
        db.commit()


def report():
    rows = db.execute("""
      select strftime('%Y-%W', t.day) as week, t.topic, h.brand,
             count(*) as answers,
             round(100.0 * avg(h.mentioned), 1) as mention_pct,
             round(100.0 * avg(h.cited), 1) as citation_pct,
             round(avg(h.best_position), 2) as avg_position,
             round(100.0 * avg(h.in_products), 1) as product_pct
      from hits h join tasks t using (task_id)
      where t.status = 'COMPLETED'
      group by week, t.topic, h.brand
      order by week, t.topic, mention_pct desc""").fetchall()
    for row in rows:
        print(*row, sep="\t")


if __name__ == "__main__":
    {"submit": submit, "collect": collect, "report": report}[sys.argv[1]]()

Run python tracker.py submit once a day, python tracker.py collect every few minutes until nothing is pending, and python tracker.py report for the table. Test it on a few prompts before scheduling the full set.

Design choices

• Idempotency keys from prompt, market, day and run. Running submit twice on the same day creates nothing new: duplicates come back as RESOURCE_ALREADY_EXISTS, and the first submission’s tasks are already stored. See idempotency keys explained.
• Batches of 500, the maximum per request. Each item succeeds or fails on its own, so the code checks every item.
• include.searchQueries and include.shopping together. Both belong to the same add-on group, so enabling both costs 2 extra credits, not 4.
• shoppingCards read with .get. The field is omitted when shopping is not requested and can be absent from answers without products.
• Failed tasks are recorded but excluded from rates, so only COMPLETED answers form the denominator. Failed tasks are not charged.
• One row per brand per answer. A brand’s share of voice for a topic is its mention count divided by the sum across BRANDS for that topic and week.

Extending it

1. More markets: add codes to COUNTRIES, or US states with "state": "TX" in the payload and the state in the key.
2. Webhooks: add "webhook": {"url": ...} to each task and move extract into the handler, reading response.result. Verify signatures first.
3. Other engines: set taskType to PERPLEXITY, GEMINI, COPILOT or AIMODE and remove the ChatGPT-only include flags, which other engines reject. The text and sources[] fields used here exist on those responses too; shopping fields differ by engine.
4. Fan-out analysis: group fanout.query by frequency per topic to find the searches ChatGPT runs most; see query fan-out.
5. Mention detection: regex on brand names misses misspellings and catches homonyms. Keep an alias list per brand and review a sample of matches each month.
6. Alerts: compare this week’s rates with the previous four weeks and alert when a change exceeds the margin of error in the sampling table.

Reading the weekly report

report prints one row per week, topic and brand. Read it in this order:

1. Answer count. A week with far fewer completed answers than planned has wider margins; check the sampling table before comparing it with other weeks.
2. Mention rate against citation rate. High mentions with low citations means ChatGPT knows the brand but cites other sites. Low mentions with some citations means your pages are retrieved but the answer recommends others. The fan-out table shows which searches produced the sources cited instead of yours.
3. Average position, only when citation rate is stable. Position over a handful of cited answers swings widely.
4. Product presence in commercial topics. Where ChatGPT shows shopping cards, a brand can lead the text and still be missing from the carousel.
5. The answers behind any change. The script stores only extracted signals; add a table for text and sources so a moved number can be explained from the answers.

Cost of this pipeline

100 prompts, three runs a day, one market, 30 days: 9,000 ChatGPT answers a month.

API and scraper lines cover collection, and your pipeline covers the rest. Platform lines include reports but fix run frequency and prompt allowance. AI monitoring cost planning shows how cadence and runs per prompt change the total.

Off-the-shelf ChatGPT trackers

These platforms include ChatGPT (entry details checked 2026-09-17):

• Peec AI: Starter €85 a month, 50 prompts, three models, daily; visibility, position, sentiment and share of voice; API on Enterprise.
• Otterly.ai: Lite $29 a month, 15 prompts, daily, ChatGPT, AI Overviews, Perplexity and Copilot; API from Standard.
• Semrush AI Visibility Toolkit: $99 a month per domain, 25 prompts tracked daily.
• Scrunch: Starter $300 a month month-to-month, 350 prompts.
• AthenaHQ: free Essential tier with 300 credits; Starter $295 a month.
• Ahrefs Brand Radar: custom prompts from $50 a month.
• Profound: free trial on ChatGPT (10 prompts, run once); Enterprise custom.

Before choosing, ask how many runs per prompt the platform takes and how it defines visibility; the sampling table applies to platforms too. The AI visibility tools comparison covers each in more detail.

Pitfalls

• Measuring the API instead of the app. A model API with web search is a different product; see provider APIs vs consumer answers.
• Mixing answers with and without web search in one series. An answer produced without searching has no citations. AnswerLine forces web search by default; if you set disableWebSearch, keep those runs in a separate series.
• Reading single prompts daily. Pool runs, then read.
• Changing the prompt set mid-series. Version it and add cohorts.
• Ignoring shopping blocks in product categories. See ChatGPT shopping cards.

The method behind mention measurement is in measuring brand mentions in ChatGPT, and the ChatGPT response fields are on the ChatGPT engine page.

Keep reading