2026-09-17, GEO · Fundamentals

llms.txt: what it is, who uses it, and how to test whether it matters

llms.txt is a proposed convention for a Markdown file at /llms.txt that gives language models a concise summary of a website and a curated list of links to its most useful content. It was proposed by Jeremy Howard on llmstxt.org (published September 3, 2024; checked 2026-09-17). It is not a web standard, and as of the check date, no major AI search vendor we checked documents reading it from your site; Google explicitly says Google Search doesn’t use it.

So documentation can’t tell you whether it helps AI citations. You can test it yourself; the method is below, after the format and the evidence that exists.

The format, per the proposal

The proposal on llmstxt.org specifies a file at the root path /llms.txt (optionally in a subpath) containing, in order:

1. An optional byte-order mark.
2. An H1 with the project or site name. This is the only required section.
3. A blockquote with a short summary containing the key information needed to understand the rest of the file.
4. Zero or more Markdown sections of any type except headings, with more detail.
5. Zero or more H2 sections containing “file lists”: Markdown lists where each item is a [name](url) link, optionally followed by : and notes.

An H2 named Optional has a special meaning: its URLs can be skipped when a shorter context is needed.

# Example Analytics

> Example Analytics is a web analytics service for small online stores. This file lists the pages that explain pricing, setup and data retention.

Answers about plan limits should use the pricing page, not blog posts.

## Docs

- [Quickstart](https://example.com/docs/quickstart.md): Install the tracking snippet and verify events
- [Data retention](https://example.com/docs/retention.md): How long raw events and reports are kept

## Optional

- [Changelog](https://example.com/changelog.md)

The proposal makes a second, related suggestion: pages useful to language models should offer a clean Markdown version at the same URL with .md appended (index.html.md for URLs without a file name).

It also positions the file relative to existing standards. robots.txt is “generally used to let automated tools know what access to a site is considered acceptable”, while llms.txt information “will often be used on demand when a user explicitly requests information about a topic”. The author expects it to be “mainly useful for inference”, not training, and notes it isn’t a substitute for sitemap.xml.

What vendors have said

All checked 2026-09-17.

Google: Search doesn’t use it

Google’s guide to optimizing for generative AI features on Google Search (last updated 2026-07-10, per the page) lists llms.txt under things you can ignore for Google Search: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.” It adds that creating and maintaining llms.txt files “for other services or systems that use these files” is fine, and that doing so “will neither harm nor help your site’s visibility or rankings in Google Search”.

That covers AI Overviews and AI Mode. It doesn’t cover every Google product: Chrome’s Lighthouse has an experimental “Agentic Browsing” category with an llms.txt audit, which calls the file “an emerging convention” and says that without it, “agents may spend more time crawling the site”. The audit flags server errors when fetching the file and marks a missing file (404) as not applicable, “as providing the file is optional at the moment”. That is a readiness check for browsing agents, not a statement about search ranking or citations.

OpenAI, Perplexity, Anthropic, Microsoft: no documentation found

We read the pages where these vendors tell site owners how their crawlers and answer products work:

• OpenAI: Overview of OpenAI Crawlers and the Publishers and Developers FAQ
• Perplexity: Perplexity Crawlers
• Anthropic: Does Anthropic crawl data from the web?
• Microsoft: Bing crawlers and AI Performance in Bing Webmaster Tools

None of them states that the vendor’s search or answer product reads a website’s llms.txt. What they document as the controls for AI search are robots.txt tokens (OAI-SearchBot, PerplexityBot, Claude-SearchBot), published IP ranges and, for Bing, robots.txt and meta directives. Absence of documentation is not proof of non-use, but it means any claim that these engines use llms.txt needs a source it currently doesn’t have.

Adoption: who publishes one

The proposal site links two community directories of published files (llmstxt.site and directory.llmstxt.cloud). We didn’t use them for counts. Instead we checked a handful of well-known documentation sites directly on 2026-09-17; each returned HTTP 200 with a text or Markdown body at:

• https://docs.perplexity.ai/llms.txt
• https://developers.openai.com/llms.txt
• https://platform.claude.com/llms.txt
• https://ai.google.dev/gemini-api/docs/llms.txt
• https://docs.x.ai/llms.txt
• https://docs.stripe.com/llms.txt
• https://developers.cloudflare.com/llms.txt

AI companies do publish llms.txt for their developer documentation, where coding agents and developer tools can use it. Publishing a file is different from reading yours, though: none of these companies’ crawler pages says their answer engine consumes it.

The clearest current use case is documentation for tools that a user points at a site: IDE assistants, agents with a fetch tool, and chat products where someone pastes a URL. That matches the proposal’s own framing of on-demand use at inference time.

Whether to publish one

Writing a small llms.txt takes little time. Keeping it accurate is the ongoing cost: a stale file that points agents at outdated pricing or deprecated docs is worse than none.

How to test whether it matters

Two measurements answer different questions, and you need both.

Measurement 1: is the file fetched at all?

Filter your access logs for requests to /llms.txt (and to any .md URLs it links). For each request, record the user agent and check the IP against the vendor’s published ranges (OpenAI, Perplexity, Google, Bing’s Verify Bingbot tool). An unverified user agent string proves nothing, because anyone can send one.

A fetch shows access, not use. No fetches by any verified AI crawler over several weeks is a strong sign the file can’t be affecting that vendor’s answers.

Measurement 2: does listing a page change its citation rate?

Because llms.txt is one file per site, a site-level before/after is confounded by everything else that changes. A within-site design is stronger.

1. Pick comparable pages. For example, 40 documentation or guide pages of similar type and traffic, each the natural answer to at least one prompt.
2. Write one prompt per page that the page should be cited for, in the wording a buyer would use. Prompt set design covers wording.
3. Randomize. Assign half the pages to listed and half to unlisted. Don’t pick by hand; you’ll pick the pages you expect to win.
4. Record a two-week baseline. Run every prompt on each engine several times and record, per answer, whether the target page’s URL is in sources[].
5. Publish llms.txt listing only the listed pages, with .md versions if you plan to keep them.
6. Measure for two more weeks with the same prompts and schedule.
7. Compare the change in citation rate for listed pages with the change for unlisted pages, per engine: (listed after − listed before) − (unlisted after − unlisted before).

A clear positive difference on one engine, with verified fetches of /llms.txt from that vendor in the logs, is evidence. A difference with no fetches is almost certainly something else.

Sample sizes

The unit is an answer to a prompt about a page. At 95% confidence, a rate near 20% measured from 100 answers has a margin of about ±7.8 points; from 400 answers, about ±3.9. To detect a lift from 20% to 30% at 80% power you need roughly 290 answers per group per period. With 20 pages per group, that is about 15 runs per prompt per period on each engine, so plan for the effect you can afford to detect, and treat repeated runs of one prompt as correlated when you analyze.

Collecting the answers

Submit each run as an async batch:

curl -X POST https://api.answerline.dev/v1/async/task/batch \
  -H "Authorization: Bearer $ANSWERLINE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {
      "taskType": "CHATGPT",
      "payload": { "prompt": "how long does Example Analytics keep raw event data", "country": "US" },
      "idempotencyKey": "llmstxt-p07-chatgpt-2026-09-17-r1",
      "webhook": { "url": "https://example.com/hooks/answers" }
    },
    {
      "taskType": "GEMINI",
      "payload": { "prompt": "how long does Example Analytics keep raw event data", "country": "US" },
      "idempotencyKey": "llmstxt-p07-gemini-2026-09-17-r1",
      "webhook": { "url": "https://example.com/hooks/answers" }
    }
  ]'

In the webhook handler, tag each result with page id, group, engine, run and period, then compute the rates:

from collections import defaultdict
from urllib.parse import urlsplit


def cited(result: dict, target_url: str) -> bool:
    target = urlsplit(target_url)
    for source in result.get("sources", []):
        url = urlsplit(source.get("url", ""))
        if url.hostname == target.hostname and url.path.rstrip("/") == target.path.rstrip("/"):
            return True
    return False


def did(rows: list[dict]) -> dict:
    """rows: {engine, group: listed|unlisted, period: before|after, result, target_url}"""
    counts = defaultdict(lambda: [0, 0])
    for row in rows:
        key = (row["engine"], row["group"], row["period"])
        counts[key][0] += cited(row["result"], row["target_url"])
        counts[key][1] += 1
    rate = lambda k: counts[k][0] / counts[k][1] if counts[k][1] else 0.0
    return {
        engine: (rate((engine, "listed", "after")) - rate((engine, "listed", "before")))
        - (rate((engine, "unlisted", "after")) - rate((engine, "unlisted", "before")))
        for engine in {k[0] for k in counts}
    }

Cost

Async tasks cost 5 credits on ChatGPT, Copilot and Grok, and 4 on Gemini. 40 prompts × 15 runs × 2 periods on ChatGPT and Gemini is 1,200 answers per engine: 6,000 + 4,800 = 10,800 credits. Synchronous calls add 2 credits each. Start with a pilot of 10 prompts and 3 runs to check the pipeline. See credits and pricing.

Caveats

• A null result in September says nothing about December. Re-run the test after vendor announcements, and re-check the vendor pages linked above.
• Publishing llms.txt tends to come with page edits and new .md versions. Change only the file during the test, or you measure the edits.
• Citations move between runs of the same prompt, which is why the design needs a control group and repeated runs (answer volatility).
• Even if a file helps a crawler discover pages, citations still depend on the engine’s searches and selection (how AI engines choose citations).
• A result on one engine doesn’t transfer to another.

For the documented controls that do affect AI search, see AI crawlers and the AI search engines. To collect answers for a test like this, start with the quickstart.

Keep reading