2026-09-18, Analytics · Fundamentals

Sampling vs. census in AI-answer monitoring

The same prompt asked twice can return different wording, citations and recommendations, so one AI answer is one draw from a distribution. A monitoring program has to decide which prompt, engine and market combinations to collect, and how many runs of each it needs before trusting a number. Below: the statistics for both decisions, the cases where you should collect everything, and code for either approach.

Definitions

Sampling is collecting a subset of the combinations or runs you care about and estimating rates, with stated uncertainty, from that subset.

A census is collecting every combination you care about (every prompt × engine × market) in every period, so that each individual answer is on record.

The two can be combined. You can take a census of your prompt set and still hold only a sample of each prompt’s possible answers, because no number of runs exhausts an engine’s output. What a census removes is uncertainty about which prompts were covered; what repeated runs reduce is uncertainty about what a prompt tends to return.

Why n = 1 misleads

A single run per prompt per period tells you what one user might have seen at one moment. Reported as “was our brand recommended: yes/no”, it produces false alarms in both directions: a lucky run hides a problem and an unlucky run invents one. The interval around a single observation covers almost the whole range.

The 95% Wilson score interval shows how wide the uncertainty stays at small run counts:

“Cited in 17 of 20 runs” is a useful statement; “cited: yes” is not.

import math

def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """95% Wilson score interval for k successes in n observations."""
    if n == 0:
        return 0.0, 1.0
    p = k / n
    d = 1 + z * z / n
    center = (p + z * z / (2 * n)) / d
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return max(0.0, center - half), min(1.0, center + half)

Measure your own variance before choosing n

How much answers move depends on the engine, the category and the prompt, so measure it instead of assuming it. A reasonable starting hypothesis is that wording changes on nearly every run, the top cited sources repeat more than the tail, and the first recommendation is steadier than positions two to five. The baseline will confirm or reject it.

Collect a variance baseline

1. Pick 20 to 50 prompts that represent your set.
2. Run each 10 times on each engine you monitor, spread over a week and different times of day. A run every few hours captures more of the variation users see than ten runs in the same minute.
3. For each prompt, compute the appearance rate of every cited URL and of your brand: runs where it appears divided by runs.

from collections import Counter

def appearance_rates(runs: list[dict], brand: str) -> dict:
    """runs: result objects for ONE prompt on ONE engine and market."""
    n = len(runs)
    urls = Counter()
    brand_hits = 0
    for r in runs:
        urls.update({s["url"] for s in r.get("sources", []) if s.get("url")})
        brand_hits += brand.lower() in (r.get("text") or "").lower()
    return {
        "runs": n,
        "brand_rate": brand_hits / n,
        "brand_interval": wilson(brand_hits, n),
        "stable_sources": [u for u, c in urls.items() if c / n >= 0.8],
        "rotating_sources": [u for u, c in urls.items() if c / n < 0.3],
    }

result.sources[] and result.text exist on every AI engine, so the same function works for any of them. Run it on result from synchronous calls or response.result from completed tasks.

Read the baseline

• Rates near 0% or 100% are cheap to track: a few runs per period detect a collapse.
• Rates in the middle (30% to 70%) are expensive: the answer is unstable for that prompt, and you need many observations to see a change.
• Many rotating sources mean citation metrics for that prompt will be noisy; report them pooled across prompts, not per prompt.

Re-run the baseline after a visible model change on an engine, since variance can shift with it. AI answer volatility covers the causes.

How many observations detect a real change

Monitoring is mostly about change: did our citation rate drop this month? For a two-sided test at 95% confidence with 80% power, the observations needed in each period are approximately:

Large drops are cheap to detect and small ones are not, and the same 10-point change costs about twice as many observations around 50% as around 90%.

An “observation” here is one answer, but runs of the same prompt are correlated, so 100 answers from 10 prompts carry less information than 100 answers from 100 prompts. For portfolio metrics such as mention rate across a prompt set, spend the budget on more prompts before more runs per prompt. Add runs when you must report on individual prompts, or when the baseline shows those prompts are unstable.

A sampling design

1. Keep the prompt set fixed and cover all of it every period: a census of prompts, so the trend is never confounded with which prompts were drawn. Prompt set design covers building it.
2. Set runs per prompt from the baseline: one or two for stable prompts, three to five for unstable ones you report individually.
3. Spread runs across the period, for example one on Monday and one on Thursday, rather than back to back.
4. Report rates with counts and intervals, pooled at the level where you have enough observations.
5. Alert on tested change, not on single flips. Monitoring cadence covers matching frequency to how fast you need to react.

Encode the run slot in the idempotency key so repeated runs are distinct tasks and resubmissions are not:

import { Client } from "@answerline/sdk";

const client = new Client({ apiKey: process.env.ANSWERLINE_API_KEY! });
const prompts = ["best payroll software for startups", "gusto vs rippling"];
const slot = `${new Date().toISOString().slice(0, 10)}-am`; // one slot per scheduled run

const tasks = prompts.flatMap((prompt, i) =>
  (["CHATGPT", "GEMINI"] as const).map((taskType) => ({
    taskType,
    payload: { prompt, country: "US" },
    idempotencyKey: `sov:${taskType}:US:${i}:${slot}`,
    webhook: { url: "https://example.com/hooks/answerline" },
  })),
);

const batch = await client.createBatch(tasks);
console.log(batch.summary);

When to take a census

Sampling estimates rates. Some jobs need the individual answers:

Brand crisis monitoring covers the crisis case in detail.

Run the census as tasks, not synchronous calls

A census is large and scheduled, which is what async tasks are for. Batches take up to 500 tasks, and webhooks deliver each result when it finishes.

Worked example at Starter-tier prices from the billing docs: 1,000 prompts on ChatGPT ($0.0006) and Gemini ($0.0006) in the US is 1,000 × $0.0012 = $1.20 per daily run. Over 30 days the task census is $36.00; running it three times a week instead of daily is $3.60 a week. See the tiers on pricing, and see cost planning.

At census scale:

• A task that has not started within 4 hours of creation fails uncharged; size submissions so they complete well inside that.
• GET /v1/async/status shows queued and processing counts, so the scheduler can hold the next batch while the previous one is still running.
• FAILED tasks are not charged and have no answer. In a census, resubmit them; a missing answer is a gap in the record, not a zero.

The reporting rule

Whatever design you choose, publish the n. “ChatGPT recommends us” means nothing without “in 17 of 20 runs across 10 prompts, 1 to 30 September, US”. A rate without its sample size and interval invites the wrong conclusion, and a census without its failure count overstates its completeness. The report template shows how to present both.

Pitfalls

• Adding runs in one burst. Ten runs in a minute understate the variance users see across a week.
• Per-prompt alerts on unstable prompts. A prompt with a 50% brand rate flips often by chance; alert on the pooled rate.
• Changing n between periods without saying so. Intervals change width, and apparent volatility changes with them.
• Pooling engines to reach a sample size. A bigger blended sample measures a mix no user sees.
• Counting failures as negatives. Exclude or resubmit, and report the count.

Set up the collection with the async guide, or see the AI visibility tracking use case for the full workflow.

Keep reading