2026-09-18, GEO · Fundamentals · Analytics

AI answer volatility: why one check is an anecdote

AI answers vary because every answer is a fresh sample: the engine generates text probabilistically over search results that change from day to day. One run shows what an answer can say, not how often it says it. The measurement that holds up is a rate over repeated runs, reported with its uncertainty, with the number of runs sized from the precision you need.

Definitions

• Cell: one prompt, one market, one engine. It is the unit you repeat.
• Run: one collected answer for a cell.
• Mention rate: the share of runs in a cell whose answer names the brand, p = mentions / n.
• Standard error (SE): the typical distance between an observed rate and the true rate you would see with infinitely many runs.
• Margin of error: the half-width of a confidence interval; for a 95% interval under the normal approximation it is 1.96 × SE.

Why answers vary

Four mechanisms act at once. None of them has a published rate you can plan around, so you measure the outcome instead of predicting it.

Sampling during generation

Language models pick each next token from a probability distribution. Google’s Gemini API documentation describes temperature as controlling “the degree of randomness in token selection” and notes that only a temperature of 0 always selects the highest-probability response (Gemini prompting strategies, checked 2026-09-17). Consumer assistants do not publish the sampling settings they use, so treat answer wording, list order and the set of named brands as random variables.

Retrieval drift

When an assistant searches the web before answering, the answer is conditioned on whatever those searches return at that moment. New pages get indexed, rankings move, and the pool of candidate sources shifts. You see this layer through the cited sources: sources[] lists the pages an answer used, in order, and that list moves as retrieval moves. The searches themselves are not visible on ChatGPT today: its searchQueries[] field comes back empty on the answers served. The query fan-out post covers how to reason about them anyway.

Market and session context

The same prompt in a different country or language can produce a different answer. Every request names its market with country. ChatGPT and Gemini answer from 213 countries; Google News takes the 71 countries with a Google News edition. Gemini also takes a US state. Keep those inputs identical across the runs of a cell; a rate computed over mixed markets blends two distributions into one misleading number. See state-level AI answers for market design.

Model updates

Providers change the models behind their products over time, and a change can shift answer style, length and source selection for many prompts at once. ChatGPT responses include a model field; store it with every row so you can check whether a step change lines up with a model change.

Quantifying variance: the binomial standard error

If each run of a cell either mentions the brand or does not, and runs are independent, the count of mentions follows a binomial distribution. The standard error of the observed rate is:

SE = sqrt( p × (1 − p) / n )
95% margin ≈ 1.96 × SE

Worked values, computed from the formula (not measured data):

Two things follow. Uncertainty is largest when the true rate is near 50%, which is where a brand is contested. And the margin shrinks with the square root of n: halving it takes four times the runs.

Sizing n for your own margin

Invert the formula. For a target margin E at 95% confidence:

n = p × (1 − p) × (1.96 / E)²

Use p = 0.5 when you have no prior estimate, since it gives the largest n:

• E = 0.15: 0.25 × (1.96 / 0.15)² = 0.25 × 170.7 = 42.7, so 43 runs
• E = 0.10: 0.25 × (1.96 / 0.10)² = 0.25 × 384.2 = 96.0, so 97 runs
• E = 0.05: 0.25 × (1.96 / 0.05)² = 0.25 × 1,536.6 = 384.2, so 385 runs

If a pilot shows p near 0.2, recompute: E = 0.10 gives 0.16 × 384.2 = 61.5, so 62 runs.

Detecting a change between two periods

A drop from last month to this month is only a finding if it exceeds the noise in both measurements. The standard error of a difference between two independent rates is:

SE_diff = sqrt( SE_1² + SE_2² )

With p = 0.5 and n = 100 in each period: SE_diff = sqrt(0.0025 + 0.0025) = 0.0707, so the 95% margin is 1.96 × 0.0707 = 0.139. A change smaller than about 14 points is not distinguishable from sampling noise at that sample size. If you need to detect a 10-point move, size each period for a smaller individual margin.

Rates near 0% or 100%

The normal approximation breaks down when p is close to 0 or 1 or n is small. Use the Wilson interval instead:

center = (p + z²/(2n)) / (1 + z²/n)
half   = z × sqrt( p(1−p)/n + z²/(4n²) ) / (1 + z²/n)

For 0 mentions in 20 runs with z = 1.96: center = 0.0960 / 1.1921 = 0.081, half = 1.96 × 0.049 / 1.1921 = 0.081, so the interval is 0% to 16.1%. “Never mentioned” after 20 runs means “mentioned in fewer than about one in six answers”.

When runs are not independent

The formulas assume independent runs. Runs collected in the same minute share the same search results and index state, so they are correlated and the true uncertainty is larger than the formula suggests. Spread a cell’s runs across hours and days. If you must report from a burst, treat the interval as a lower bound on uncertainty.

A method for measuring volatility on your prompts

1. Pick the cells. Start with your highest-value prompts on one market and the engines your audience uses. The prompt set design guide covers selection.
2. Choose a target margin and compute n per cell from the sizing formula. Budget the cost before you run (see below).
3. Schedule the runs over time. Split n across days, for example a fixed number per day, instead of one burst.
4. Submit as async batches with meaningful idempotencyKey values that include the run index, so a retried submission cannot double-collect.
5. Extract one row per run: cell, run time, model where present, mentioned (0/1), cited domains, brands named in the text.
6. Compute p and its interval per cell, and per period if you track change.
7. Classify cells by the interval, not by the point estimate (see the table at the end).

Credit arithmetic for a pilot

Prices at the Starter tier: ChatGPT $0.0006, Gemini $0.0006 per request. Example sizing, 20 prompts in one market at 43 runs each (E = 0.15):

ChatGPT: 20 × 43 × $0.0006 = $0.516
Gemini:  20 × 43 × $0.0006 = $0.516
total                 = $1.03

Setting include.searchQueries on ChatGPT is free, so it does not change these numbers. Cost planning covers monthly budgets and tiers.

Collecting the runs

A batch of ChatGPT runs for one prompt and market, one of several you would submit across the day:

curl -X POST https://api.answerline.dev/v1/async/task/batch \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {"taskType": "CHATGPT", "idempotencyKey": "vol:2026-09-17:chatgpt:US:p01:r01",
     "payload": {"prompt": "best project management tools for agencies", "country": "US"},
     "webhook": {"url": "https://example.com/hooks/answers"}},
    {"taskType": "CHATGPT", "idempotencyKey": "vol:2026-09-17:chatgpt:US:p01:r02",
     "payload": {"prompt": "best project management tools for agencies", "country": "US"},
     "webhook": {"url": "https://example.com/hooks/answers"}}
  ]'

Each item in the response has its own index and either a task or an error, so check every item. Results arrive at the webhook as the full task JSON; for a COMPLETED task the engine result is at response.result. The webhook signature post covers verification.

Computing the rate and Wilson interval from stored rows, in Python:

import math
from collections import defaultdict

def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return (max(0.0, center - half), min(1.0, center + half))

def mentioned(result, brand):
    text = result.get("text") or ""
    return brand.lower() in text.lower()

# rows: (cell_key, result) pairs taken from COMPLETED tasks' response["result"]
def rates(rows, brand):
    counts = defaultdict(lambda: [0, 0])
    for cell, result in rows:
        c = counts[cell]
        c[0] += 1 if mentioned(result, brand) else 0
        c[1] += 1
    return {cell: (k / n, *wilson(k, n)) for cell, (k, n) in counts.items()}

The check reads the answer text, which works the same on every engine; on ChatGPT, entities[] also lists the brands and products an answer names. Substring matching is a starting point; handle brand aliases and false positives before you publish numbers (measuring brand mentions goes further).

Reporting variance

Show the interval with the point. “Mentioned in 58% of runs (95% interval 48–68%)” is a finding stakeholders can act on; “58%” invites a comparison with last week’s 61% as if the difference meant something. Conventions:

• Plot rates with interval bars per cell and per period.
• Only call a change when the intervals of the two periods do not overlap, or when the difference exceeds its own margin.
• Report n next to every rate.
• Annotate periods where the model value changed.

When variance itself is the signal

The interval tells you what kind of prompt you are looking at:

The thresholds are illustrative; set your own. Contested prompts are where better source coverage can shift the distribution; see how engines choose citations and sampling vs. census.

Pitfalls

• Calling a single run a drop. A missing mention at p = 0.7 happens three times in ten.
• Mixing markets or engines into one rate. Compute per cell, then aggregate with the counts visible.
• Bursty collection. Correlated runs understate uncertainty.
• Reading optional arrays without a default. They are omitted when empty.
• Rewording the prompt mid-series. A reworded prompt is a new cell.

Set up collection with the async task guide, or start with the ChatGPT engine page.

Keep reading