AEO and AI search

What is llms.txt, and do you need one?

IN SHORT

llms.txt is a proposed plain-text Markdown file at the root of a website that gives language models a short, curated guide to the site’s most useful pages. Google says Search ignores it, and no major assistant has confirmed using it as a citation or ranking signal. It is cheap to publish, but do not expect it to move results.

By Hristijan Najcheski, Head of SEO and AEO at Webtec: technical SEO and AI SEOReviewed October 11, 20269 min read

It lives at /llms.txt and is written in Markdown. Jeremy Howard proposed it in September 2024. It is not an official standard.
Google’s guide says Google Search ignores llms.txt, so it neither helps nor harms visibility there.
Ahrefs’ May 2026 study of 137,210 domains found that 97 percent of valid llms.txt files received no requests that month.
It does not replace crawlable pages, schema or good content, and it controls nothing, unlike robots.txt.

TRY IT

See what goes in the file

example.com/llms.txt

Line 1Title

The name of the business, as one clear title. It tells a machine who the file is about.

What a machine reads

Honest note: no major AI assistant relies on llms.txt alone, and it will not rank you by itself. It is cheap, tidy and a sign of a well-organised site, so we include it. The work that moves answers is positioning, content and proof.

What llms.txt contains

The proposal, written by Jeremy Howard and published in September 2024, describes a Markdown file that gives an AI agent a short, curated overview of a site with links to the detail. The file must start with a single H1 with the site or project name, which is the only required part.

After that comes an optional blockquote summary, optional body text, and any number of sections headed by H2 titles that contain lists of links, each with a short note. By convention a section called "Optional" holds secondary links an agent can skip. The proposal also suggests offering clean Markdown versions of pages at the same address with .md added.

The point is to give a model a clean map without making it parse your navigation, scripts and ads. The file is meant to be small enough to fit in a model’s context window.

What the evidence says about whether it works

Google has been explicit. In June 2025, John Mueller of Google wrote that no AI system currently uses llms.txt, and that consumer chatbots fetch pages for training and grounding but none of them fetch the llms.txt file. Google’s generative AI optimization guide (last updated July 2026) says you do not need new machine-readable or AI-specific files to appear in Search because Search does not use them. It adds that creating llms.txt for other services is fine, and that Google ignores it.

Usage data points the same way. Ahrefs, an SEO software vendor, analyzed 137,210 domains in its web analytics for May 2026 and published the study in June 2026. About 28 percent had a valid llms.txt file, but 97 percent of those files received no requests that month. SEO audit tools were the largest category of requesters, and AI retrieval bots made up about 1.1 percent of total requests. Ahrefs notes that its customers skew technical, so the adoption figure is an upper bound, and that a fetch does not prove a bot used the file.

The honest summary: no major assistant has confirmed using llms.txt for citations, and the available data shows little use. Some tools, such as coding agents and documentation readers, do fetch it, so it can have a role for developer documentation. Check current provider statements, because this can change.

How to create one, step by step

Choose 8 to 25 pages: core services or products, pricing or how pricing works, key guides, about, contact and policies.
Write one plain sentence for each saying what a reader will learn there, not marketing copy.
Group them under short H2 headings, and put secondary pages under "Optional".
Publish it at /llms.txt as plain text with a Markdown content type, and check it loads without a redirect.
Generate it from your content system or sitemap, so it never drifts out of date.

What it cannot do

It will not make an assistant cite you. Pages still have to be crawlable and genuinely the best answer. It does not block or allow anything either: access rules live in robots.txt, per AI crawler, as explained in AI crawlers and robots.txt.

Ahrefs also flags a risk. It mentions prompt injection, the risk that text a model fetches is treated as instructions, and a poorly maintained file that points to outdated or wrong pages can mislead an agent. Keep it short and accurate.

Our recommendation

Treat llms.txt as optional documentation for machines. It is one small part of the wider difference between SEO and AEO work. If you can generate it automatically in a few minutes, publish it, because it forces a useful discipline: deciding which pages matter. If it would need manual upkeep, skip it and spend the time on the pages themselves.

Do not sell it as a ranking tool, and do not rank it above crawl access, schema markup or answer-first content. Sites with developer documentation or an API have the strongest case, because coding agents are the visible users today.

What may change

The file is a proposal that its author wants to standardize, and the proposal was still being updated in 2026. A major assistant could announce support at any time.

The cost of waiting is low and the cost of publishing is low, so neither choice is risky. Revisit when a provider documents support, and read the provider’s page, not a headline. The larger gains sit in how assistants choose sources.

Examples by industry

Illustrative scenarios, not client results. Pick the one closest to your business.

B2B SOFTWARE AND STARTUPS
A SaaS company with docs, pricing and a changelog. The site has hundreds of pages and a deep menu.
1. List the pages that matter: product, pricing, security, key docs and integrations.
2. Write one plain sentence about each.
3. Generate the file from the sitemap so it never goes stale.
4. Put the changelog and old release notes under Optional.
Takeaway: The file forces the useful discipline of deciding which pages matter most.
ACCOUNTING AND FINANCIAL FIRMS
A firm with 40 pages. Services, a few guides, team and contact pages.
1. Pick the eight pages a prospect or assistant most needs.
2. Describe each in one sentence.
3. Publish at /llms.txt and review it twice a year.
Takeaway: It is cheap, so publish it, and do not expect it to move results by itself.
E-COMMERCE BRANDS
A store with 2,000 products. Listing every product would make the file useless.
1. List the main collections, the buying guides and the policies instead.
2. Leave individual products to the sitemap and structured data.
3. Keep it short.
Takeaway: Curate the file. Its value is in what you leave out.

THE WEBTEC ANSWER METHOD · STEP 5 OF 6

Plumb: how we do it

llms.txt is part of the technical plumbing that makes your site easy for machines to read.

Publish and maintain llms.txt where it is cheap to do so
Set robots.txt rules deliberately
Add structured data to key pages

See all six steps

Common mistakes

Selling llms.txt as a ranking shortcut, when Google says it ignores the file and no major assistant has confirmed using it.
Listing every page instead of the important ones, which defeats the purpose of a curated guide.
Letting it go out of date, so it points models to pages that have moved or been removed.
Blocking AI crawlers in robots.txt while publishing the file, which sends contradictory signals.
Expecting it to control access. It blocks and allows nothing.
Spending hours on it before fixing broken indexing, thin pages or crawl blocks.

Checklist

llms.txt is live at the root and returns plain text without a redirect.
It starts with an H1 and a one-sentence summary.
It lists only the pages that matter, with a clear description for each.
It is generated from your content, or reviewed at least twice a year.
robots.txt is consistent with your intent for each AI crawler.
Your core pages are crawlable and indexed first.

Questions

Do ChatGPT, Gemini or Perplexity use llms.txt?

To our knowledge, none of the major providers has confirmed that they use it as a signal for answers or citations. Google says its Search ignores the file, and John Mueller of Google said in June 2025 that no AI system currently uses it. Server-log studies show very few requests from AI retrieval bots. Check current provider documentation, because this can change. If you want evidence for your own site, check your server logs for requests to /llms.txt and see which user agents made them.

Is it harmful to publish one?

No. Google says publishing it will neither help nor harm your visibility in Search. The risks are indirect: overselling what it does, letting it go stale so that an agent reads wrong information, or spending time that should go to your core pages. Keep it short, accurate and generated from your content. If you publish one, do not include anything you would not want quoted back to a customer. It is also worth checking that the file is not blocked by a redirect or password, so what you publish is what a reader sees.

Is llms.txt the same as robots.txt?

No. Robots.txt tells crawlers which parts of your site they may fetch, and the major crawlers document that they honor it. Llms.txt is a reading guide. It does not allow or block anything, and no standard says a crawler must follow it. If you want to control AI access, use robots.txt and your firewall, as covered in our crawler guide. Both are plain files in your site root, which is why they are confused, but only one carries rules that crawlers document that they follow.

How long does it take and what does it cost?

A basic file takes under an hour for a small site once you have decided which pages matter, and less if it is generated from your sitemap or CMS. Ongoing cost is a review twice a year, or nothing if it is automated. We do not quote prices in guides. For most clients it is a small line item inside broader technical work. Ahrefs’ study suggests that for most sites the return is small, so keep the effort proportional to that.

Is llms.txt right for my business?

It is most useful for sites with documentation, APIs or many product pages, where coding agents and tools may read it. For a local service business with 30 pages, it is optional, and your time is better spent on clear service pages, reviews and consistent business details. Publish it if it costs you little, and do not expect results from it. If a developer or agency maintains your site, ask them to generate it from the sitemap rather than writing it by hand.

Should I also create Markdown copies of my pages?

Only if you have a documentation-heavy site and a technical reason. The proposal suggests offering clean .md versions of pages, but Google says you do not need to rewrite content or create special formats for its AI features. Good HTML with clear headings and text content is what search crawlers use. Do not duplicate pages without a clear use. If you do publish Markdown versions, make sure they match the HTML exactly, since a mismatch creates contradictory versions of your facts.

Sources

Webtec reviews this page regularly. Search and platform features change, so check provider documentation for the latest. How we research and verify our guides.

Hristijan Najcheski

Head of SEO and AEO at Webtec: technical SEO and AI SEO. Leads technical SEO and AI SEO at Webtec: crawling, indexing, site speed, structured data and how AI systems read a site. He also verifies the numbers in our case studies.

Related guides

Pick up where this one leaves off.

All AEO and AI search guides

PreviousQuery fan-out NextSchema markup for AI

Want this done for your business?

Our AEO and GEO service We run this method for clients, with a dedicated growth manager.

Book a consultation

Explore our services

Start with the one closest to what you just read.

All services