The web after search: why llms.txt is the new robots.txt

Lorem ipsum dolor sit amet, consectetur adipiscing elit. For twenty years, the default contract between a website and the rest of the web was a robots.txt file and a sitemap. You told crawlers where to go. They indexed. People found you through a ranked list of ten blue links.

That contract is quietly being rewritten.

What changed

Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. When a user asks ChatGPT, Claude, Gemini, or Perplexity a question, the model doesn't return ten blue links. It returns an answer — possibly with one or two citations tucked underneath. That answer was synthesized from content the model either trained on, retrieved live, or both.

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris. The sites that show up in those answers aren't necessarily the sites that ranked #1 in search. They're the sites that were legible to the model — the ones that offered a clean, self-describing version of their content for a non-human reader.

The llms.txt spec in one paragraph

Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore. llms.txt is a proposed standard: a single markdown file at the root of your domain that tells an LLM who you are, what your site is for, and which pages matter. Think of it as the README for your entire web presence, written for a model instead of a developer.

A minimal manifest looks like this:

# Acme Corp

> Acme builds payroll software for remote teams.

## Docs
- [Getting started](https://acme.com/docs/quickstart)
- [API reference](https://acme.com/docs/api)

## Company
- [About](https://acme.com/about)
- [Pricing](https://acme.com/pricing)

Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. That's it. No JavaScript, no schema, no crawling. Just markdown.

Why you probably don't have one

Sed ut perspiciatis unde omnis iste natus error sit voluptatem. Three reasons:

1. It's new. The spec landed in 2024 and most frameworks don't ship it by default.
2. It looks optional. Your SEO team isn't asking for it yet.
3. It's boring. Writing a good manifest is tedious.

Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit. The third one matters most. The hand-written manifest is a chore; the auto-generated manifest is a joke. The right answer is a tool that reads your site, writes the manifest, and keeps it updated as you ship.

What good looks like

Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur. A good manifest is:

• Accurate. Every link resolves. Every section reflects real content.
• Current. When you publish a new docs page, the manifest updates within a day.
• Scoped. It covers the pages you want models to read, not every URL in your sitemap.
• Under 50KB. LLMs don't reward verbosity.

At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores. Get the basics right and you'll be in the answers before your competitors notice the protocol exists.