2026-09-16, GEO · Fundamentals

How often do AI answers change? Setting a monitoring cadence

The right monitoring cadence is the slowest one that still tells you about a change before it costs you. Checking too often spends money on run-to-run noise; checking too rarely means you learn weeks late that a competitor replaced you in the answer. Choose cadence per group of prompts from three inputs: the detection delay you can accept, the noise in the metric, and the cost of each run.

Definitions

• Cadence: how often a cell (one prompt, one market, one engine) is collected.
• Detection delay: the time between a real change in answers and the moment your data confirms it.
• Confirmation rule: how many consecutive or aggregated runs must show the change before you act.
• Run cost: the price per collected answer, from the price table.

What changes an answer

These act continuously and on no published schedule.

• Retrieval drift. Assistants that search the web answer from whatever those searches return now. As pages are published, updated and re-ranked, the source pool changes and the answer follows.
• Sampling. Generation is probabilistic, so identical inputs can yield different wording, order and brand lists even when nothing else changed. The volatility post shows how to quantify this with standard errors.
• Model changes. Providers update the models behind their products. Store the model field ChatGPT returns so you can line up step changes with model changes.
• Market. The same prompt in another country, state or city can produce a different answer. Every request carries country. ChatGPT, Gemini, AI Mode and Google Search answer from 213 countries; Copilot, Perplexity and Grok take any country, and Google News the 71 countries with a Google News edition. Gemini also takes a US state.
• Prompt wording. A reworded prompt is a different measurement. Fix the text of each tracked prompt; if phrasing matters, track variants as separate cells.

Detection delay and confirmation

A run detects a change only when it is collected, and one run does not confirm it:

worst-case detection delay ≈ cadence interval × runs needed to confirm

Suppose a brand is normally mentioned in 70% of runs for a prompt. If runs are independent, the chance of a miss on a healthy day is 0.3, two misses in a row 0.3² = 0.09, three in a row 0.3³ = 0.027. A “three misses in a row” rule fires on noise about 2.7% of the time per sequence, and at daily cadence it confirms a real disappearance on day three. At weekly cadence the same rule takes three weeks.

For a brand mentioned in 90% of runs, a single miss has probability 0.1 and two in a row 0.01, so a two-run rule is enough. The more stable the baseline, the fewer runs you need to confirm, and the lower the cadence can go for the same delay.

Daily runs on your top prompts mean a disappearance shows up in the next day’s data; the confirmation rule decides how many more days pass before you act on it.

A method for choosing cadence

1. Group prompts by consequence: brand and high-intent commercial prompts, category prompts, long-tail prompts. Prompt set design covers the grouping.
2. Write down the acceptable detection delay for each group: a day, a week, a month.
3. Measure the baseline rate for a sample of cells over two to four weeks at a daily cadence. Stable, high rates need fewer confirming runs; contested rates near 50% need more.
4. Set cadence = acceptable delay ÷ runs to confirm, rounded to a schedule you can run (daily, twice weekly, weekly).
5. Cost it with the formula below and check it against your budget.
6. Revisit monthly. Promote cells that turned contested to a faster cadence; demote cells that stayed locked.

Different metrics, different cadences

Metrics collected from the same prompts have different noise:

• Mention and recommendation rates are binary per run and noisy near 50%. They need many runs to estimate, which argues for steady collection spread across the period rather than bursts.
• Citation share aggregates many source entries per answer, but sources within one answer move together, so treat the answer as the sampling unit and do not assume more precision than the run count supports.
• Fan-out queries would show what the assistant searched, but ChatGPT’s searchQueries[] comes back empty on the answers served today, so there is no query list to sample from it. Cited sources are the retrieval signal you can collect.
• Classic rankings for your target queries can run on their own schedule on Google Search.

Decouple the schedules: a daily alert set, a weekly reporting set, and a weekly or monthly research set can share prompts without sharing cadence.

Cadence by use case

These are starting points; step 3 of the method replaces them with your own numbers. For crisis situations, see brand crisis monitoring.

The cost arithmetic

monthly cost = prompts × markets × engines' price per run × runs per month

Prices per request at the Starter tier: ChatGPT $0.0006, Gemini $0.0006, Google AI Mode $0.00206, Google Search $0.00103, Google News $0.0002. Add-ons change the per-run price and cost the same on every tier (ChatGPT’s include flags and Google News pages beyond the first are free): Google Search adds $0.00071 per page beyond the first and $0.00071 for the AI Overview.

You pay from a prepaid USD balance, topped up from $1.00; a new account starts with 100 free requests. Prices drop on larger tiers, reached by 30-day spend (Growth from $95.00, Max from $1,900).

Worked example: everything daily

200 prompts, US, 2 engines (ChatGPT, Gemini), daily for a 30-day month, async:

cost per prompt-day   = $0.0006 + $0.0006        = $0.0012
runs per day           = 200 × 2          = 400
runs per month         = 400 × 30         = 12,000
cost per month         = 200 × 30 × $0.0012   = $7.20

That is $7.20 a month at Starter prices. Scheduled work belongs in async tasks, which wait in the queue and can report results by webhook.

Turning on include.searchQueries for ChatGPT on every run is free, so the total stays at $7.20.

Worked example: tiered cadence

Same 200 prompts, but only the top 40 run daily; the other 160 run weekly (counted as 4 runs in a 30-day month):

top 40, daily:          40 × 30 × $0.0012 = $1.44
long tail 160, weekly: 160 × 4 × $0.0012  =  $0.768
total                                = $2.21

That is $2.21 a month. Moving 80% of prompts to weekly cut $4.99 while every prompt kept its engines.

Worked example: trimming engines instead

All 200 prompts stay daily, but the long tail runs on ChatGPT only:

top 40, daily, both engines:      40 × 30 × $0.0012 = $1.44
long tail 160, daily, ChatGPT:   160 × 30 × $0.0006 = $2.88
total                                           = $4.32

That is $4.32 a month with every prompt still collected daily. Engines and markets are multipliers on every prompt, so removing one that does not change decisions saves as much as a large cut in cadence. Trim those first, then set cadence by consequence. Cost planning has more combinations, and /pricing has every tier’s price.

News cadence

Ten news queries collected hourly on Google News, one page each: 10 × 24 × 30 × $0.0002 = $1.44 a month. Extra pages are free, so pages: 2 costs the same.

Scheduling the runs

Run the schedule from your own job runner (cron, a workflow tool, a queue worker) and submit each slot as one batch of up to 500 tasks. Give each task an idempotencyKey derived from what it measures, so a retried submission never collects the same slot twice:

import datetime, os, requests

API = "https://api.answerline.dev"
HEADERS = {"Authorization": f"Bearer {os.environ['API_KEY']}", "Content-Type": "application/json"}
ENGINES = {"CHATGPT": "chatgpt", "GEMINI": "gemini"}

def daily_batch(prompts, markets, webhook_url):
    day = datetime.date.today().isoformat()
    tasks = [
        {
            "taskType": task_type,
            "idempotencyKey": f"cadence:{day}:{slug}:{country}:{pid}",
            "payload": {"prompt": text, "country": country},
            "webhook": {"url": webhook_url},
        }
        for pid, text in prompts
        for country in markets
        for task_type, slug in ENGINES.items()
    ]
    for start in range(0, len(tasks), 500):
        r = requests.post(f"{API}/v1/async/task/batch", json=tasks[start:start + 500], headers=HEADERS, timeout=30)
        r.raise_for_status()
        for item in r.json()["results"]:
            if not item["success"] and item["error"]["code"] != "RESOURCE_ALREADY_EXISTS":
                print("task", start + item["index"], item["error"]["code"], item["error"]["message"])

Queue rules that shape the schedule:

• Tasks wait in your queue until they are picked up; no count limits the queue (it holds about 100,000 tasks of a typical prompt), and there is no concurrency cap. A task that has not started within 4 hours fails uncharged, so submit each run close to when you need it.
• priority (1–10, higher first) orders only your own queue: give daily brand prompts a higher priority than the weekly long tail submitted at the same time.
• RESOURCE_ALREADY_EXISTS on a batch item means that slot was already submitted, which is what you want on a retry.
• Failed tasks are not charged; a failed run costs a retry, not money.

Results arrive at your webhook as the finished task JSON; the engine result of a COMPLETED task is at response.result. See webhook reliability design for the receiver.

Don’t alert on noise

• Alert on sustained patterns: N consecutive misses sized from the baseline rate, or a weekly rate whose change exceeds its margin of error.
• Separate alert cells from report cells. Alerts need fast cadence on few prompts; reports need many prompts at slower cadence.
• Compare like with like: same prompt text, market and engine across periods.
• Record the model value where the engine returns one, and check it before explaining a step change as a content effect.
• Log the dollar charge per run from credits.creditsCharged so the bill and the schedule stay reconciled.

The metric definitions behind these alerts are in AI share of voice. To start collecting, follow the quickstart.

Keep reading