llms.txt: The Complete Guide for AI Search in 2026

The llms.txt file is a plain-text manifest you place at your domain root that tells large language models which pages on your site they should read first. It mirrors the spirit of robots.txt but inverts the goal. Robots.txt tells crawlers what to avoid. The llms.txt file tells reasoning models what to prioritize, in what order, and which canonical URLs to cite when they generate answers.

The proposal landed in September 2024 from Jeremy Howard at Answer.AI, and adoption moved faster than any other web standard I’ve watched in 18 years. By Q1 2026 the index at llmstxt.directory tracked 11,400 plus public llms.txt files. Anthropic, Cloudflare, Vercel, Stripe, Supabase, GitHub, Cursor, Mistral, ElevenLabs, and Perplexity all ship one. None of these companies waste engineering time on speculative protocols, which tells you what they think this file is going to be worth.

I deployed an llms.txt file across 14 client sites between November 2025 and April 2026. The pattern is consistent. Sites with a clean, curated llms.txt got cited 30 to 70 percent more often in ChatGPT Search, Perplexity, and Claude responses inside 60 days, with no other on-page changes. The file is not a ranking factor in the classic sense. It is a routing document, and routing is exactly what AI search runs on.

What llms.txt Is, In Plain English

An llms.txt file is a Markdown file served at the root of your domain. Two URLs are conventional: /llms.txt for a curated index and /llms-full.txt for the entire documentation in one file. The format has six required sections in order: an H1 with the project or site name, an optional blockquote with a short summary, free-form paragraphs that give context, then H2 sections that group canonical URLs as a Markdown link list, with optional “Optional” sub-sections that the model may skip when its context budget is tight.

The point of the format is brutal simplicity. A reasoning model can ingest the entire file in a single retrieval pass, decide which 3 to 8 URLs to follow, and burn far less context on navigation than it would by crawling a sitemap. That matters because every token spent on a nav menu is a token not spent on your actual content.

The proposal is not a W3C standard. There is no enforcement, no certifying body, no Google Search Console widget. It is a community convention that Anthropic, Vercel, and Cloudflare adopted because their own internal Claude Code, v0, and AI Gateway products needed a way to feed external docs to language models without scraping the open web. Once those three names shipped one, every developer-tools company followed inside 90 days.

Why llms.txt Matters In 2026

llms.txt matters in 2026 because the share of search-style queries that get answered without a click is past 35 percent on informational intent. Semrush’s March 2026 study across 10 million keywords reported organic CTR drops of 34 to 62 percent on queries that triggered AI Overviews. ChatGPT Search crossed 250 million weekly active users in February 2026. Perplexity is doing more than 780 million queries per month. The click is no longer the primary distribution channel; the citation is.

If LLMs are going to summarize your content with or without your permission, your only leverage is to make the summary accurate, attributed, and easy to extract. llms.txt does three things that move that needle:

• Routes the model to your canonical URL. Instead of an LLM picking between 18 partially overlapping pages on your site, you tell it which one is the source of truth.
• Strips navigation from the retrieval payload. The model sees a clean Markdown link list, not your header, footer, cookie banner, and related-posts widget.
• Signals freshness and authority. A maintained llms.txt with recent canonical URLs is a stronger trust signal to Claude or ChatGPT than a stale sitemap of 4,000 thin pages.

None of this replaces the deeper work of generative engine optimization or paragraph-level extractability. llms.txt is the routing layer. The content layer still has to earn the citation. But routing without quality is invisible, and quality without routing is wasted.

llms.txt vs Robots.txt vs Sitemap.xml

The three files solve three different jobs and do not replace each other. Robots.txt is a crawl-permission boundary. Sitemap.xml is an exhaustive URL discovery list. llms.txt is a curated reading order for reasoning models. You ship all three.

The simplest mental model: robots.txt is the front-door sign, sitemap.xml is the building directory, llms.txt is the receptionist who walks the visitor to the right office.

If you also want LLMs to cite your structured data, ship JSON-LD schema in tandem with llms.txt. I cover the pairing in my structured data for AI search guide.

The llms.txt Specification In Detail

The llms.txt format that Jeremy Howard published is intentionally minimal. The order matters because some retrieval systems truncate the file at a token budget. Put the most important sections first.

1. H1 title (required). The project, product, or site name. Exactly one H1 in the file.
2. Blockquote summary (recommended). One to three sentences. The model uses this as the doc card when it cites you. Make it the most precise sentence you write all year.
3. Free-form context paragraphs (optional). Anything the model needs to know before reading: stack assumptions, audience, version, license terms.
4. H2 sections with link lists. Each H2 is a topic. Each list item is a Markdown link with a colon-separated description: [Title](url): One-line description.
5. Optional H2 (literal). A section literally named ## Optional tells the model these URLs can be skipped if context is tight.

The companion file /llms-full.txt follows the same structure but inlines the full Markdown body of every linked page. Cloudflare, Supabase, and Stripe all serve both. Smaller sites usually only need llms.txt; the full version is for documentation hubs that want to be ingested whole.

Two things the spec does NOT do, despite what you’ll read on Reddit. It does not block AI training; that is robots.txt territory plus your terms of service. And it does not guarantee citation; it just makes citation cheaper and more accurate when the model decides to cite you.

Real llms.txt Files From Anthropic, Cloudflare, And Vercel

Reading other companies’ llms.txt files is the fastest way to get your own right. I pulled these three on May 9, 2026 with a plain curl. Each takes a different approach and each one is worth studying.

Anthropic, docs.anthropic.com/llms.txt

Anthropic’s file is the largest of the three and is structured around language editions of the Claude Developer Platform Console docs. The H1 is “Anthropic Developer Documentation”. The first H2 lists 11 language editions of the docs (English, German, Spanish, French, Italian, Japanese, Korean, Portuguese, Russian, Simplified Chinese, Traditional Chinese), with English (1,413 pages) marked as the canonical edition that has its content inlined elsewhere. Subsequent sections cover API references, model cards, and the prompt library. The file leans toward thoroughness over brevity, which fits a company whose primary audience is engineers wiring up the API.

Cloudflare, developers.cloudflare.com/llms.txt

Cloudflare’s file is a directory of directories. The H1 is “Cloudflare Developer Documentation”. A blockquote at the top points readers to llms-full.txt for the entire archive in a single file (intended for offline indexing and bulk vectorization), and notes that every product ships its own product-scoped llms.txt. The body is a flat Markdown link list grouped under H2s like “Application performance”, “Application security”, “Developer platform”, “Network services”, and “Reference architectures”. Each link follows the canonical pattern: [Argo Smart Routing](https://developers.cloudflare.com/argo-smart-routing/llms.txt): Route web traffic across the fastest, most reliable network paths. This nested-llms.txt approach is the cleanest pattern I’ve seen for any vendor with more than 50 products.

Vercel, vercel.com/docs/llms.txt

Vercel’s file is the most marketing-aware of the three. The H1 is “Vercel Documentation” and the first line points to llms-full.txt for the whole archive. Sections cover “Getting Started”, “Fundamental Concepts”, framework guides, deployment, and the AI SDK. Each link description is a single declarative sentence, written in product-marketing voice (“Vercel is the AI Cloud, a unified platform for building, deploying, and scaling AI-powered applications and agentic workloads”). That sentence is what shows up in many ChatGPT responses about Vercel; the file is doing double duty as documentation index and brand summary.

Three patterns to copy: Anthropic’s exhaustive language coverage, Cloudflare’s nested per-product files, Vercel’s marketing-grade descriptions. Pick the one that fits the shape of your site.

Other production llms.txt files worth studying in the same session: Stripe (docs.stripe.com/llms.txt), Supabase (supabase.com/llms.txt), GitHub (docs.github.com/llms.txt), Cursor (docs.cursor.com/en/llms.txt), Mistral (docs.mistral.ai/llms.txt), and ElevenLabs (elevenlabs.io/docs/llms.txt). All six were live and returning 200 on May 9, 2026. Each one is worth a read; patterns repeat across all of them.

How To Write A Good llms.txt

A good llms.txt is short, opinionated, and accurate. The mistake I see most often is treating it like a sitemap and dumping every URL. That defeats the point. The model already has a sitemap. What it needs is your editorial decision about which URLs are canonical.

1. Cap the file at 200 URLs on a marketing or content site, 2,000 on a documentation hub. More than that and the model truncates anyway.
2. Group URLs under 4 to 8 H2 sections by topic, not by site navigation. The model uses the H2 names to disambiguate intent, so name them after the questions a user would ask.
3. Write each link description as a complete sentence that is true on its own. The model will paraphrase your description in many citation chips. If the description is a noun phrase, the citation reads as fragment.
4. Put the canonical pillar URLs first in each section. Truncation cuts from the bottom.
5. Use the literal H2 “Optional” for things you’d be happy for the model to skip when context is tight: changelog, release notes, archived posts.
6. Refresh the file monthly. Add new pillar URLs, drop pages you’ve redirected or unpublished. Stale llms.txt is worse than none, because it sends the model to 404s.

One non-obvious detail: keep absolute URLs throughout. Relative URLs technically work but every retrieval system I’ve tested hits edge cases when the file is fetched from a different host than the canonical one (CDN, reverse proxy, or cached copy in a vector database).

llms.txt Generator Tools And Validators

You can write the file by hand in 30 minutes for a 50-page site. For larger sites or to keep it fresh, a generator is worth setting up. The tools I’ve used in production:

• llmstxt.firecrawl.dev. Free hosted generator from the Firecrawl team. Crawls a domain and emits both llms.txt and llms-full.txt in Markdown. Best for one-time bootstrapping.
• Mintlify. If your docs are on Mintlify, llms.txt and llms-full.txt are auto-generated and served on every deploy. Zero configuration.
• llms-txt CLI by Answer.AI, on GitHub at AnswerDotAI/llms-txt. The reference implementation. Parses an llms.txt and validates structure.
• llms.txt Generator plugin for WordPress. Several exist; the one with the most active maintenance as of May 2026 is “Website LLMs.txt” by Search Atlas. Generates from your post types and updates on publish.
• llmstxt.directory. Public index of public llms.txt files. Useful for studying patterns and checking who in your industry has already deployed one.

Auto-generators are fine for first deployment. They are bad for maintenance because they treat every URL as equal weight. Hand-curate the top-level llms.txt for any site that earns more than 10,000 monthly organic sessions. Use the generator for llms-full.txt, where exhaustiveness is the goal.

WordPress Implementation, Step By Step

Three deployment paths work on WordPress. Pick by how much control you want.

Path 1: Static file via SFTP

Write the Markdown by hand, name it llms.txt, drop it in the WordPress root next to wp-config.php. Add a RewriteRule to .htaccess if your server tries to send it through PHP. This is the cleanest setup for sites with stable URL structures and an editor who’s willing to update the file once a month.

Path 2: Plugin-generated

Install Website LLMs.txt by Search Atlas, or LLMs.txt for WordPress by Rank Math (shipped in version 3.0.95, March 2026). Both crawl your post types, build a curated index, and serve /llms.txt via a rewrite rule. Rank Math’s version reads your existing focus keywords and primary categories to group sections automatically. This path is best for content sites with 50 to 500 posts and a publishing cadence higher than once a week.

Path 3: Custom theme function

Add a template_redirect hook in your theme’s functions.php that listens for the /llms.txt path, runs a WP_Query across your relevant post types, and prints a Markdown file with appropriate Content-Type: text/markdown; charset=utf-8 and Cache-Control headers. This is the path for developer-led teams that already script their LLM SEO stack and want full control over URL selection logic.

Whichever path you take, verify with curl -I https://yoursite.com/llms.txt. You want a 200 response, content-type text/plain or text/markdown, and a Last-Modified header that reflects your actual publishing cadence. If you see 301, 302, or text/html, the file isn’t being served correctly and reasoning models will skip it.

Validating And Tracking llms.txt Performance

Validation runs in two layers. First, the file itself: structure, links, freshness. Second, the impact: are you getting cited more after deployment.

For structural validation, paste your file into the Answer.AI llms-txt CLI or the validator at llmstxt.directory. Both flag missing H1, malformed links, and links that 404. Run the check before every deploy. A broken llms.txt is silently ignored by most retrieval systems, which makes it impossible to debug from the outside.

For impact tracking, you need an AI citation monitoring tool. Three I’ve used: Profound, Otterly.ai, and Peec.ai. Each runs a panel of representative queries through ChatGPT, Perplexity, Gemini, and Claude on a daily cadence and tracks whether your domain shows up as a cited source. Compare the citation rate 30 days before and after deploying llms.txt. The pattern across my client work: a 30 to 70 percent lift in citation share inside 60 days, larger on niche topics, smaller on commodity ones. None of the lift is guaranteed; it depends entirely on whether the underlying content was already strong enough to be worth citing.

For a deeper dive on tracking citations across engines, see my AI search optimization playbook and the breakdown of Perplexity SEO mechanics.

Common llms.txt Mistakes And How To Avoid Them

Six failure modes account for almost every llms.txt that doesn’t work. None of them are subtle; all of them are common.

• Treating it like a sitemap. 4,000 URLs in alphabetical order. Models truncate at 1 to 5 percent of that and you have no editorial control over which 1 percent survives.
• Marketing fluff in the blockquote. “We are passionate about innovation” is a wasted token. Use the blockquote to state what your site actually is, in domain-specific terms.
• Relative URLs. Break in 30 percent of retrieval contexts. Always absolute.
• No description on links. Bare URLs are 4 to 6 times less likely to be cited than URLs with one-line descriptions, in my A/B tests across two client domains.
• Stale file. 404 links inside llms.txt make the model distrust the entire file. Re-validate weekly on a content site, monthly on a docs site.
• Duplicating in robots.txt. robots.txt is a permission boundary, not a content list. Putting llms.txt content in robots.txt does nothing.

The single highest-leverage fix on a sloppy llms.txt is the link descriptions. Spend an hour rewriting them as complete sentences and you’ll outperform a generator-built file at 10x the size.

What Comes After llms.txt

llms.txt is the entry point. Once the model is reading the right URLs, the next layer is paragraph-level extractability: answer-first paragraphs, named entities, specific numbers, schema markup. None of that is replaced by llms.txt; it is amplified by it. A bad page with a perfect llms.txt still gets passed over. A great page with no llms.txt gets cited less than it should because the model spends its retrieval budget navigating instead of reading.

Beyond llms.txt, watch for two adjacent proposals being floated through 2026: ai.txt from the SpawningAI team (which is more about training-data opt-outs than retrieval) and IETF discussions around a more formal “AI Preferences” header. Neither has the adoption llms.txt has earned. Build for what’s shipping.

Ship llms.txt this week. Hand-write it for your top 50 URLs. Validate it. Track citation share for 60 days. Then expand. The cost is one afternoon. The upside is showing up in the answer instead of being summarized off it.

Frequently Asked Questions