2026-09-17, Rank Tracking · Analytics

AI rank tracking: what "rank" means inside an AI answer, and how to measure it

Rank tracking inside AI answers starts with a definition problem. A search results page has one ordered list, so “rank 4” is unambiguous. An AI answer has several ordered things at once: brands named in the prose, sources in a list, inline citation chips, product cards, map listings. And from one run to the next, you may not appear at all.

Below: six rank definitions, how to aggregate when you are absent, the statistics and sample sizes behind a trustworthy number, and the collection pipeline in brief. The full operational build is in the batch pipeline tutorial.

Six definitions of rank

Pick the definition that matches the business question, and name it in every report. Mixing them silently is the most common source of contradictory dashboards.

Notes that matter when you implement them:

• Mention rank needs a brand set. “Position 2” only means something relative to the brands you track. Fix the competitor list per prompt cluster and version it, or ranks shift when you add a competitor.
• Mention rank on non-ChatGPT engines is text order. entities[] exists only on ChatGPT. Elsewhere, find the first match of each brand’s aliases in text with word boundaries and sort by offset.
• Source lists and inline citations differ. A source can be listed without appearing inline. Treat citationPills[] as the more visible placement if that distinction matters to you; group entries by citationPillId to rebuild each chip.
• Organic rank is an input to AI rank. It belongs in the same report because assistants search before answering: ChatGPT returns its searches as searchQueries[] with include.searchQueries, Copilot and Grok return searchQueries[], and Perplexity returns search_model_queries. Your organic rank on those queries explains part of your source rank. See query fan-out.

The absence problem

A tracked page usually has some position in a results page. In an AI answer, a brand is either present or absent on each run. How you treat those runs decides what your metric says.

A worked example (illustrative numbers)

Twenty runs of one prompt, mention rank among tracked brands:

• Your brand appears in 12 runs, at ranks 1, 1, 2, 1, 3, 2, 1, 4, 2, 1, 1, 2.
• Competitor appears in 19 runs, always at rank 3.

“Mean rank when present” says you win comfortably. Top-3 rate says the competitor is in the visible top three almost every time and you are there about half the time. Mean reciprocal rank, which weights position 1 heavily, still favours you. Which metric fits depends on the question; the error is reporting conditional rank alone.

Aggregations that handle absence

• Presence rate plus conditional rank. Always reported together. Never show conditional rank without its presence rate beside it.
• Top-k rate. Share of all runs in which you appear at position k or better. Easy to explain; choose k to match what users see (for a list of five recommendations, k = 3 is often meaningful).
• Mean reciprocal rank (MRR). Average of 1/rank over all runs, counting absence as 0. Bounded between 0 and 1, rewards top positions, and needs no arbitrary penalty.
• Penalized mean rank. Assign absent runs a fixed rank (list length + 1). Avoid it unless the list length is fixed: the penalty value drives the result.

Pick one headline metric per report and show presence rate beside it. For a broader metric set (mention share, citation share, recommendation share), see the AI share of voice framework.

Why repeated runs are required

The same prompt, engine and market can give a different answer on each run. Sampling during generation, changes in the pages the engine retrieves, and model updates all move the output; AI answer volatility covers the causes. So a rank for one prompt is a distribution, and you estimate it by sampling.

Intervals on presence and top-k rates

Presence and top-k rates are proportions. For small samples or rates near 0% or 100%, use the Wilson interval rather than the normal approximation:

from math import sqrt

def wilson(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    if n == 0:
        return (0.0, 1.0)
    p = successes / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return (centre - half, centre + half)

print(wilson(12, 20))  # (0.387, 0.781)
print(wilson(19, 20))  # (0.764, 0.991)

From the example: your 60% presence over 20 runs is compatible with anything from about 39% to 78%. The competitor’s 95% is 76% to 99%. Twenty runs separate these two, but they would not separate 60% from 70%.

Intervals on MRR and mean rank

MRR and conditional rank are not proportions. Bootstrap them: resample the runs with replacement a few thousand times, recompute the metric each time, and take the 2.5th and 97.5th percentiles. When you aggregate across prompts, resample prompts (with all their runs) rather than individual runs, for the reason in the next section.

import random

def bootstrap_mrr(runs_by_prompt: dict[str, list[int | None]], iters: int = 2000) -> tuple[float, float]:
    prompts = list(runs_by_prompt)
    stats = []
    for _ in range(iters):
        sample = [random.choice(prompts) for _ in prompts]
        rr = [0.0 if r is None else 1.0 / r for p in sample for r in runs_by_prompt[p]]
        stats.append(sum(rr) / len(rr))
    stats.sort()
    return (stats[int(0.025 * iters)], stats[int(0.975 * iters)])

Detecting a change between two periods

To detect a presence rate moving from p1 to p2 with a two-sided test at 5% significance and 80% power, the observations needed per period are approximately:

n ≈ (1.96 + 0.84)² × (p1(1 − p1) + p2(1 − p2)) / (p2 − p1)²

For 30% to 40%: 7.84 × (0.21 + 0.24) / 0.01 ≈ 353 observations per period. For 30% to 50%, the same formula gives about 90. Small changes need a lot of data, which is why rank reports should be aggregated over prompt clusters and weeks rather than read per prompt per day.

Runs of one prompt are not independent

Twenty runs of one prompt tell you a lot about that prompt and little about the next one. Statistically, observations cluster by prompt, and the effective sample size shrinks by the design effect:

effective n = total observations / (1 + (runs per prompt − 1) × ρ)

where ρ is the intra-prompt correlation of the outcome. With 50 prompts × 8 runs = 400 observations and an illustrative ρ of 0.5, the effective n is 400 / 4.5 ≈ 89. Estimate ρ from your own baseline. When it is high, which is common for stable prompts, adding prompts buys far more precision than adding runs. Sampling vs census covers designs that balance the two.

Comparing ranks across engines

Position numbers are not portable between engines. Engines return different numbers of sources for similar prompts and display them differently, so “source rank 3” is a stronger placement in a short list than in a long one.

Rules that keep comparisons honest:

1. Compare trends within an engine, not levels across engines.
2. Prefer rate metrics across engines. Presence rate and top-3 rate survive list-length differences better than mean position.
3. Record list length per run. Store the count of sources[] with each observation so you can normalize later (for example, rank divided by list length) if needed.
4. Keep engine as a dimension in every table. A blended AI rank hides that engines cite different sources for the same prompt.

Choosing the prompt and run design

Prompt set design covers sourcing and clustering prompts; monitoring cadence covers how often to run them.

The collection pipeline, briefly

1. Plan tasks. Expand prompt × engine × market × run slot into tasks with an idempotencyKey such as p042-chatgpt-US-2026-09-17-r3, so a resubmission can never double-count.
2. Submit in batches. POST /v1/async/task/batch accepts 1 to 500 tasks; each item is admitted or refused on its own.
3. Receive by webhook. Each finished task is posted to your webhook.url with task, credits and response. Deduplicate by task.id.
4. Store raw JSON. Keep the full result; you will add rank definitions later and want to recompute history.
5. Extract observations. One row per task per brand per rank type: (prompt_id, engine, market, run_at, brand, rank_type, rank or null, list_length).
6. Aggregate with intervals. Wilson for rates, bootstrap by prompt for MRR and conditional rank.

A submission of two tasks for one prompt on two engines:

curl -X POST https://api.answerline.dev/v1/async/task/batch \
  -H "Authorization: Bearer $ANSWERLINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {
      "taskType": "CHATGPT",
      "payload": { "prompt": "best accounting software for a small construction company", "country": "US", "include": { "searchQueries": true } },
      "idempotencyKey": "p042-chatgpt-US-2026-09-17-r1",
      "webhook": { "url": "https://hooks.example.com/answers" }
    },
    {
      "taskType": "COPILOT",
      "payload": { "prompt": "best accounting software for a small construction company", "country": "US" },
      "idempotencyKey": "p042-copilot-US-2026-09-17-r1",
      "webhook": { "url": "https://hooks.example.com/answers" }
    }
  ]'

Extracting mention rank from a stored result:

import re

def mention_ranks(result: dict, engine: str, brands: dict[str, list[str]]) -> dict[str, int | None]:
    """brands maps a brand key to its aliases. Returns rank among tracked brands, or None if absent."""
    if engine == "CHATGPT" and result.get("entities"):
        names = [e["name"] for e in result["entities"]]
        first = {}
        for i, name in enumerate(names):
            for key, aliases in brands.items():
                if key not in first and any(a.lower() == name.lower() for a in aliases):
                    first[key] = i
    else:
        text = result.get("text", "")
        first = {}
        for key, aliases in brands.items():
            hits = [m.start() for a in aliases for m in re.finditer(rf"\b{re.escape(a)}\b", text, re.IGNORECASE)]
            if hits:
                first[key] = min(hits)
    order = sorted(first, key=first.get)
    return {key: (order.index(key) + 1 if key in first else None) for key in brands}

Note that ChatGPT’s entities[] can include entities that are not in your brand set; the function ranks only tracked brands, by design.

What it costs

Credits per async task: ChatGPT 5 (7 with include.searchQueries, include.shopping, include.ads or include.rawResponse, charged once), Copilot 5, Grok 5, Gemini 4, Perplexity 4, AI Mode 4, Google Search 3 for the first page. Synchronous calls add 2.

A design of 60 prompts × 4 runs per week on ChatGPT with search queries and on Copilot is 60 × 4 × (7 + 5) = 2,880 credits per week. Adding a weekly organic check on 150 deduplicated fan-out queries adds 150 × 3 = 450. See cost planning and pricing to size a plan.

Pitfalls

• Conditional rank without presence. The most common way AI rank reports mislead.
• Undefined rank type. “Rank” on a dashboard must say whether it is mention, source, inline citation, product, local or organic rank.
• Per-prompt, per-day readings. The intervals are too wide; aggregate by cluster and week.
• Counting failed tasks as absent. A FAILED task is a missing observation, not a zero.
• Changing the brand set silently. Mention ranks shift when you add a competitor; version the set.
• Cross-engine position comparisons. Compare rates across engines and positions within one.

To start collecting observations, see the rank tracking use case or the quickstart.

Keep reading