2026-09-17, ChatGPT · Brand Monitoring · Tutorials

ChatGPT visibility tracker: metrics, prompts and sampling

A ChatGPT visibility tracker runs a fixed set of buyer prompts through ChatGPT on a schedule, several times per prompt and market, and reduces each answer to a few facts: was the brand mentioned, was its site cited and where, did its products appear. Those rates, over time and against competitors, are the visibility metric. You can buy this as a platform (Peec AI, Otterly.ai, Profound, Scrunch, AthenaHQ, Semrush, Ahrefs Brand Radar) or build it on an answer API. The method comes first here, since it determines whether either approach gives numbers you can trust; then a Python pipeline on AnswerLine; then the off-the-shelf options.

Third-party facts are from the vendors’ own pages, checked 2026-09-17, with links. For the same measurement across engines rather than ChatGPT alone, see brand mentions per engine.

Options

Metrics

Define these before collecting anything. Each maps to one part of the ChatGPT answer.

To keep them comparable over time:

1. Only completed answers go in the denominator. A failed collection is not an answer without you.
2. Mentions and citations stay separate. A brand can be recommended without a link and linked without a recommendation. See mentions vs citations.
3. The competitor set is fixed and versioned, because share of voice changes meaning whenever the set changes. The share of voice framework covers the arithmetic.

Building the prompt set

The prompt set decides what “visibility” means for you.

1. Start from buyer questions rather than keywords. Sales calls, support tickets, site search and People Also Ask are better sources than a keyword list; People Also Ask for prompt research shows one method.
2. Cover the funnel: category questions (“best payroll software for small business”), comparisons (“Gusto vs ADP”), problems (“how to run payroll for contractors in two states”), and brand questions (“is Acme Payroll good for restaurants”).
3. Tag every prompt with topic, funnel stage and whether it names a brand. Unbranded prompts measure discovery and branded prompts measure reputation, so report them separately.
4. Write prompts the way people type: short, conversational, sometimes with context (“for a 20-person agency”). Do not name your brand in unbranded prompts.
5. Size by topic. Ten to twenty prompts per topic gives a rate that no single phrasing dominates.
6. Keep a stable core. Changing prompts breaks trend lines, so add new prompts as a separate cohort and fold them in after a few weeks.

Prompt set design goes deeper on balance and size.

Sampling: how many runs you need

ChatGPT’s answers vary between runs of the same prompt, so a single daily run per prompt measures that noise as much as your visibility. Treat each answer as a sample and report rates over pooled samples.

For a rate p measured over n answers, the 95% margin of error is about 1.96 × √(p(1−p)/n):

In practice:

• Pool before you read: weekly rates per topic, not daily rates per prompt.
• More distinct prompts reduce phrasing bias; more runs per prompt reduce run-to-run noise. Three runs a day across 15 prompts in a topic gives 315 answers per topic per week.
• Keep markets separate. Pooling the US and Germany mixes two populations.

The sampling vs census post and AI answer volatility cover this in more depth.

A pipeline on AnswerLine

One Python file with three commands: submit queues today’s runs, collect fetches finished tasks and extracts brand signals, and report prints weekly rates. It uses requests and SQLite against the HTTP API, and polls for results to stay self-contained; at larger volume, use a webhook instead.

Inputs

prompts.csv:

id,topic,prompt
p001,category,best payroll software for small business
p002,category,easiest payroll software for restaurants
p003,comparison,Gusto vs ADP for a 20-person company

The script

import csv, datetime, os, re, sqlite3, sys
from urllib.parse import urlparse

import requests

API = "https://api.answerline.dev"
COUNTRIES = ["US"]
RUNS_PER_DAY = 3
BRANDS = {
    "acme": {"patterns": [r"\bacme payroll\b"], "domain": "acmepayroll.com"},
    "gusto": {"patterns": [r"\bgusto\b"], "domain": "gusto.com"},
    "adp": {"patterns": [r"\badp\b"], "domain": "adp.com"},
}

http = requests.Session()
http.headers["Authorization"] = f"Bearer {os.environ['API_KEY']}"
db = sqlite3.connect("chatgpt_visibility.db")
db.executescript("""
create table if not exists tasks (
  task_id text primary key, prompt_id text, topic text, country text, day text, run int, status text);
create table if not exists hits (
  task_id text, brand text, mentioned int, cited int, best_position real, in_products int,
  primary key (task_id, brand));
create table if not exists cited (task_id text, domain text);
""")


def submit():
    day = datetime.date.today().isoformat()
    with open("prompts.csv", newline="", encoding="utf-8") as f:
        prompts = list(csv.DictReader(f))
    meta, tasks = [], []
    for p in prompts:
        for country in COUNTRIES:
            for run in range(RUNS_PER_DAY):
                meta.append((p["id"], p["topic"], country, day, run))
                tasks.append({
                    "taskType": "CHATGPT",
                    "payload": {
                        "prompt": p["prompt"],
                        "country": country,
                        "include": {"shopping": True},
                    },
                    "idempotencyKey": f"{p['id']}-{country}-{day}-r{run}",
                })
    for start in range(0, len(tasks), 500):
        res = http.post(f"{API}/v1/async/task/batch", json=tasks[start:start + 500], timeout=60)
        res.raise_for_status()
        for item in res.json()["results"]:
            m = meta[start + item["index"]]
            if item["success"]:
                db.execute("insert or ignore into tasks values (?,?,?,?,?,?,?)",
                           (item["task"]["id"], *m, "QUEUED"))
            else:
                print("not queued:", m, item["error"]["code"])
    db.commit()


def on_domain(url, domain):
    host = (urlparse(url).hostname or "").lower()
    return host == domain or host.endswith("." + domain)


def product_text(result):
    names = [e.get("name", "") for e in result.get("entities", []) if e.get("type") == "product"]
    return " ".join(n for n in names if n).lower()


def extract(task_id, result):
    text = result.get("text", "").lower()
    products = product_text(result)
    for brand, cfg in BRANDS.items():
        mentioned = any(re.search(p, text) for p in cfg["patterns"])
        positions = [s["position"] for s in result.get("sources", [])
                     if on_domain(s.get("url", ""), cfg["domain"])]
        in_products = any(re.search(p, products) for p in cfg["patterns"])
        db.execute("insert or replace into hits values (?,?,?,?,?,?)",
                   (task_id, brand, mentioned, bool(positions),
                    min(positions) if positions else None, in_products))
    db.executemany("insert into cited values (?,?)",
                   [(task_id, (urlparse(s.get("url", "")).hostname or "").lower())
                    for s in result.get("sources", [])])


def collect():
    pending = db.execute("select task_id from tasks where status in ('QUEUED','PROCESSING')").fetchall()
    for (task_id,) in pending:
        res = http.get(f"{API}/v1/async/task/{task_id}", timeout=30)
        res.raise_for_status()
        body = res.json()
        status = body["task"]["status"]
        if status == "COMPLETED":
            extract(task_id, body["response"]["result"])
        db.execute("update tasks set status = ? where task_id = ?", (status, task_id))
        db.commit()


def report():
    rows = db.execute("""
      select strftime('%Y-%W', t.day) as week, t.topic, h.brand,
             count(*) as answers,
             round(100.0 * avg(h.mentioned), 1) as mention_pct,
             round(100.0 * avg(h.cited), 1) as citation_pct,
             round(avg(h.best_position), 2) as avg_position,
             round(100.0 * avg(h.in_products), 1) as product_pct
      from hits h join tasks t using (task_id)
      where t.status = 'COMPLETED'
      group by week, t.topic, h.brand
      order by week, t.topic, mention_pct desc""").fetchall()
    for row in rows:
        print(*row, sep="\t")


if __name__ == "__main__":
    {"submit": submit, "collect": collect, "report": report}[sys.argv[1]]()

Run python tracker.py submit once a day, python tracker.py collect every few minutes until nothing is pending, and python tracker.py report for the table. Test it on a few prompts before scheduling the full set.

Design choices

• Idempotency keys from prompt, market, day and run. Running submit twice on the same day creates nothing new: duplicates come back as RESOURCE_ALREADY_EXISTS, and the first submission’s tasks are already stored. See idempotency keys explained.
• Batches of 500, the maximum per request. Each item succeeds or fails on its own, so the code checks every item.
• include.shopping only. It is free and adds shoppingCards, empty on the ChatGPT answers served today (no merchant cards), and inlineProducts, the products from ChatGPT’s shopping search with price, link and merchant offers. Product presence reads entities, which is free. include.searchQueries is free, but it comes back empty on the ChatGPT answers served today, so the script does not ask for it.
• entities read with .get. It is empty when an answer names no brand or product.
• Failed tasks are recorded but excluded from rates, so only COMPLETED answers form the denominator. Failed tasks are not charged.
• One row per brand per answer. A brand’s share of voice for a topic is its mention count divided by the sum across BRANDS for that topic and week.

Extending it

1. More markets: ChatGPT answers from 213 countries. Add the ones you sell in to COUNTRIES; the country is already part of the key.
2. Webhooks: add "webhook": {"url": ...} to each task and move extract into the handler, reading response.result. Verify signatures first.
3. Other engines: set taskType to GEMINI and remove the ChatGPT-only include flags, which other engines reject. The text and sources[] fields used here exist on Gemini responses too. Perplexity, Copilot, Grok and AI Mode work the same way (PERPLEXITY, COPILOT, GROK, AIMODE); AI Mode’s shopping fields differ from ChatGPT’s.
4. Cited-domain analysis: group cited.domain by frequency per topic to find the sites ChatGPT cites most; see how AI engines choose citations.
5. Mention detection: regex on brand names misses misspellings and catches homonyms. Keep an alias list per brand and review a sample of matches each month.
6. Alerts: compare this week’s rates with the previous four weeks and alert when a change exceeds the margin of error in the sampling table.

Reading the weekly report

report prints one row per week, topic and brand. Read it in this order:

1. Answer count. A week with far fewer completed answers than planned has wider margins; check the sampling table before comparing it with other weeks.
2. Mention rate against citation rate. High mentions with low citations means ChatGPT knows the brand but cites other sites. Low mentions with some citations means your pages are retrieved but the answer recommends others. The cited table shows which domains were cited instead of yours.
3. Average position, only when citation rate is stable. Position over a handful of cited answers swings widely.
4. Product presence in commercial topics. A brand can lead the text and still have none of its products named.
5. The answers behind any change. The script stores only extracted signals and cited domains; add a table for text and full source URLs so a moved number can be explained from the answers.

Cost of this pipeline

100 prompts, three runs a day, one market, 30 days: 9,000 ChatGPT answers a month.

API and scraper lines cover collection, and your pipeline covers the rest. Platform lines include reports but fix run frequency and prompt allowance. AI monitoring cost planning shows how cadence and runs per prompt change the total.

Off-the-shelf ChatGPT trackers

These platforms include ChatGPT (entry details checked 2026-09-17):

• Peec AI: Starter €85 a month, 50 prompts, three models, daily; visibility, position, sentiment and share of voice; API on Enterprise.
• Otterly.ai: Lite $29 a month, 15 prompts, daily, ChatGPT, AI Overviews, Perplexity and Copilot; API from Standard.
• Semrush AI Visibility Toolkit: $99 a month per domain, 25 prompts tracked daily.
• Scrunch: Starter $300 a month month-to-month, 350 prompts.
• AthenaHQ: free Essential tier with 300 credits; Starter $295 a month.
• Ahrefs Brand Radar: custom prompts from $50 a month.
• Profound: free trial on ChatGPT (10 prompts, run once); Enterprise custom.

Before choosing, ask how many runs per prompt the platform takes and how it defines visibility; the sampling table applies to platforms too. The AI visibility tools comparison covers each in more detail.

Pitfalls

• Measuring the API instead of the app. A model API with web search is a different product; see provider APIs vs consumer answers.
• Counting answers without search. ChatGPT decides on its own whether to search, and an answer with no sources showed no search. Compute citation metrics over answers with sources and report the share without.
• Reading single prompts daily. Pool runs, then read.
• Changing the prompt set mid-series. Version it and add cohorts.
• Ignoring product names in product categories. Today they come in entities; see ChatGPT shopping cards.

The method behind mention measurement is in measuring brand mentions in ChatGPT, and the ChatGPT response fields are on the ChatGPT engine page.

Keep reading