llms.txt Explained: The New robots.txt for AI Assistants

llms.txt is a plain-text file placed at the root of your website (at yourdomain.com/llms.txt) that gives large language models a curated, markdown-friendly map of your most important content. Think of it as a concierge note for AI: instead of letting an assistant guess which of your hundreds of pages matter, you hand it a short, clean list of your best, answer-ready content with a one-line description of what each page covers. It complements — it does not replace — your existing robots.txt and sitemap.xml.

The idea is a direct response to how AI assistants now read the web. Tools like ChatGPT, Claude, Gemini, and Perplexity work best with clean, structured, markdown-style text and operate within limited context windows. A sprawling HTML site full of navigation, scripts, and boilerplate is noisy and hard to distill. llms.txt cuts through that noise by pointing models straight at the pages you most want them to understand and cite.

llms.txt vs robots.txt vs sitemap.xml

These three files all live at your domain root and all speak to automated visitors, but they answer completely different questions. Confusing them is the most common mistake, so it helps to keep their jobs separate:

• robots.txt — the access rules. It tells crawlers (including AI bots like GPTBot, PerplexityBot, and Google-Extended) where they are and aren't allowed to go. It is about permission, not guidance. A blocked bot reads nothing at all.
• sitemap.xml — the full inventory. It lists every URL you want indexed, in machine-readable XML, so search engines can discover all of your pages. It is comprehensive and undifferentiated — your contact page sits next to your flagship guide with equal weight.
• llms.txt — the curated guidance. It highlights the handful of pages that matter most, in human-readable markdown, with short descriptions that explain what each one is for. It is opinionated by design: you are telling an AI assistant where your best answers live.

Here is the one-line way to remember it: robots.txt controls access, sitemap.xml lists everything, and llms.txt recommends what's important. A complete site benefits from all three. llms.txt does not override robots.txt — if a bot is blocked from a page in robots.txt, listing that page in llms.txt will not grant access.

What Goes in an llms.txt File

The format is intentionally simple and readable. At minimum, an llms.txt file contains an H1 with your site or brand name, an optional blockquote summarizing what you do, and then one or more sections of markdown links to your key pages — each link followed by a short description. Optional ## sections let you group those links (for example, “Core Services,” “Guides,” or “Company”), and an “Optional” section can flag lower-priority links a model may skip if context is tight.

A realistic example for a small business looks like this:

# Acme Plumbing

> Licensed plumbing and drain services for the greater Chicago area,
> offering emergency repairs, water heater installation, and sewer
> inspections with upfront pricing and same-day appointments.

## Core Services

- [Emergency Plumbing Repair](https://acmeplumbing.com/emergency): 24/7 repair for leaks, burst pipes, and clogs.
- [Water Heater Installation](https://acmeplumbing.com/water-heaters): Tank and tankless installation and replacement.
- [Drain & Sewer Cleaning](https://acmeplumbing.com/drains): Camera inspections and high-pressure jetting.

## Guides

- [What to Do in a Plumbing Emergency](https://acmeplumbing.com/blog/emergency-guide): Step-by-step first response before help arrives.
- [Tankless vs Tank Water Heaters](https://acmeplumbing.com/blog/water-heater-guide): How to choose, with cost and lifespan tradeoffs.

## Company

- [Service Area & Pricing](https://acmeplumbing.com/areas): Neighborhoods served and how pricing works.
- [Contact & Booking](https://acmeplumbing.com/contact): Phone, hours, and online scheduling.

## Optional

- [Customer Reviews](https://acmeplumbing.com/reviews): Verified ratings and testimonials.

Notice what makes this useful: every link has a plain-English description, the most commercially important pages come first, and the whole file is short enough to read in under a minute. You are not dumping your sitemap here — you are curating. A good llms.txt for most sites is a few dozen links at most, not a few hundred.

Why It Matters for AEO and GEO

Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO) are about being the source an AI assistant chooses when it answers a question. To pick you, the assistant first has to find and understand your most relevant, answer-ready content — quickly, and without getting lost in page chrome. That is exactly the problem llms.txt is designed to solve.

By handing models a clean shortlist of your strongest pages with context attached, you make it easier for an assistant to match a user's question to the right page on your site and to summarize it accurately. It is the difference between leaving an assistant to scrape and infer from messy HTML versus giving it a tidy table of contents written specifically for machines. As AI-driven discovery grows, that small head start in clarity can be the margin between being cited and being skipped.

How to Create and Publish an llms.txt File

Creating one is genuinely a short task. The steps are:

1. List your most important pages. Start with the pages that answer real customer questions or drive your business — core services or products, your best guides, pricing, and contact. Resist the urge to include everything.
2. Write the file in markdown. Begin with # Your Brand Name, add a one- or two-sentence blockquote summary, then group your links under ## headings with a short description after each link.
3. Save it as a plain-text file named llms.txt.
4. Upload it to your site root so it is reachable at https://yourdomain.com/llms.txt — the same place your robots.txt lives. Most content management systems and hosts let you add a root-level file through the file manager, an SEO plugin, or FTP.
5. Verify it loads by visiting the URL in a browser. It should render as readable text, not download or 404.

If you would rather not hand-write the markdown, our free llms.txt generator will draft a structured file from your site in a couple of clicks. While you are at it, run our AI Bot Access checker to confirm your robots.txt actually allows the AI crawlers to reach the pages you just curated — guidance is wasted if access is blocked. And for the full picture of how visible you are in AI search, a complete AI website audit shows where you stand across crawlability, citations, and answer readiness, with fixes ranked by impact.

An Honest Caveat: It's an Emerging Standard

It is worth being clear-eyed here. llms.txt is a voluntary, emerging proposal, not an official requirement, and support from AI vendors is still evolving. No assistant is obligated to read or honor it, and you should not expect publishing one to instantly change how you appear in AI answers. There is no guaranteed ranking boost.

What it is, is a low-effort, low-risk way to get ahead of where AI discovery is heading. The file takes minutes to create, costs nothing, and cannot hurt you — at worst it is ignored. Treat it the way smart teams treated structured data and sitemaps early on: a small, forward-looking investment that complements the fundamentals (allowing AI crawlers, writing clear answer-ready content, earning citations) rather than replacing them. Get robots.txt right first; add llms.txt as the curated layer on top.

Frequently Asked Questions

What is llms.txt?

llms.txt is a plain-text, markdown-formatted file placed at the root of your website (yourdomain.com/llms.txt) that gives AI assistants a curated map of your most important pages, each with a short description. It helps models quickly find and understand your best, answer-ready content instead of guessing from messy HTML.

What is the difference between llms.txt and robots.txt?

robots.txt controls access — it tells crawlers which pages they are allowed to read. llms.txt provides guidance — it points AI models to your most important content and explains what each page is for. They work together: robots.txt grants permission, llms.txt offers a curated tour. llms.txt cannot override a block set in robots.txt.

Is llms.txt the same as a sitemap?

No. A sitemap.xml lists every URL on your site in machine-readable XML so search engines can discover all of it. llms.txt is a short, human-readable markdown file that highlights only your most important pages with descriptions. A sitemap is comprehensive; llms.txt is curated and opinionated.

How do I create an llms.txt file?

List your most important pages, write the file in markdown with an H1 brand name, a blockquote summary, and grouped sections of links with short descriptions, then save it as llms.txt and upload it to your site root so it loads at yourdomain.com/llms.txt. You can also use a free llms.txt generator to draft it automatically.

Do AI assistants like ChatGPT and Perplexity actually use llms.txt?

llms.txt is a voluntary, emerging standard, and adoption by AI vendors is still evolving. No assistant is required to read or honor it. Treat it as a low-effort complement to the fundamentals — allowing AI crawlers and writing clear content — rather than a guarantee of better AI visibility.

Where do I put the llms.txt file?

Place it at the root of your domain so it is reachable at yourdomain.com/llms.txt — the same location as your robots.txt. Most content management systems and hosting control panels let you add a root-level file through the file manager, an SEO plugin, or FTP. Confirm it loads as readable text in a browser.