Large language models (LLMs) often struggle to understand websites effectively. Unless they rely on grounding (connecting to real-time data sources), they can’t tell which pages matter most on your site.
Some say that the llms.txt solves this problem. A few create a .md version of every page. Others call it pointless. Let’s take a look at what it actually does and whether it’s worth your time.
What is the llms.txt?
llms.txt is a simple, machine-readable text file that gives LLMs (such as ChatGPT and Claude) context about your website. Think of it like a helpful guide that points AI agents toward your most important content.The file uses markdown formatting and lives in the root of the domain.
The protocol actually includes two files:
llms.txt: Links to your most important pages.
llms-full.txt: Contains detailed content in a single file, often with examples and concrete steps to perform certain tasks.
Both use standard markdown formatting to organize the information.
An abstract example of an llms.txt file.
Where did llms.txt come from?
In September 2024, Jeremy Howard published the proposal for the llms.txt protocol. Howard is the founder of FastAI and AnswerAI, two research labs in the AI and deep learning space, and a lecturer at the Universities of Queensland and Stanford.
As of July 2025, more than 600 websites have adopted the standard. Among them Perplexity, Anthropic, Cursor, Stripe, Solana, ElevenLabs, Hugging Face, Raycast, Yoast, DataForSEO, Zapier, and Cloudflare.
How does it compare to existing standards?
llms.txt vs robots.txt
These serve completely different purposes. robots.txt is a file webmasters use to tell crawlers which areas of a website they’re allowed to crawl. This is relevant for all kinds of crawlers. SEOs have been using the robots.txt to handle how search engines crawl their websites for more than 20 year. If you want to control how LLM crawlers access your website, that still happens in the robots.txt as well.
The llms.txt is just a helper file with no directives. You can use it neither to allow nor forbid crawling of your website.
Ilms.txt vs sitemap.xml
Both help make crawlers aware of certain content, but they work at different scales.
Traditional crawlers aim to crawl a website in its entirety and recrawl pages after an update. That’s where the XML sitemap shines.
The llms.txt usually contains only a small subset of pages.
Let’s take an online shop as an example. In your XML sitemap, you’d want to include every single product and category page - potentially millions of URLs. In your llms.txt, you’d probably only want to link to the return policy, and explain that product and category pages exist. And in the llms-full.txt you would explain how to use the search function and the checkout process. And maybe how to integrate all of that via an API.
What does the llms.txt look like in practice?
llms.txt example
Here’s how Perplexity structures their llms.txt:
Notice how Perplexity makes heavy use of headlines (including a hierarchy), lists, and labeled links.
llms-full.txt example
Perplexity’s llms-full.txt goes much deeper. The file has more than 7,000 lines and contains over 30,000 words. Here’s a short extract:
The file contains detailed instructions and examples on how to work with the Perplexity API.
SEOs & the llms.txt
In recent months, some members of the SEO community have started to create llms.txt files that either contain links to all their SEO landing pages - or outright include all website content.
There are already 10+ tools that automatically create your llms.txt for you.
The markdown copy-paste approach (and why it doesn’t work)
A few SEOs have even started creating markdown copies of every blog article as .md files, then linking all these .md files from their llms.txt. This approach creates unnecessary duplicate content without clear benefits.
This trend has led to questions in the Google Search Console Help Community, where people ask whether they need separate markdown versions of all their content. Here is a recent example:
My website may serve two versions of the same content via different URLs:
https://mywebsite.com/about-my-website
https://mywebsite.com/about-my-website.mdThe llms.txt file will reference the .md (markdown) versions
John Müller from Google addresses this directly:
AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag - this is what a site-owner claims their site is about ... (Is the site really like that? well, you can check it. At that point, why not just check the site directly?)
llms.txt & Answer Engines
Not a single LLM-driven search and answer engine has announced support for the llms.txt protocol. With limited evidence of actual usage and no official backing from major AI companies, widespread adoption seems unlikely.
When you see Generative Engine Optimization (GEO) checklists claiming everyone needs an llms.txt, or consultants recommending markdown copies of every blog post, there’s no credible evidence this helps with GEO. It might actually hurt by creating duplicate content, diluting your ranking signals across multiple URLs, and wasting crawl budget.
However, if you have an API or a website/app that you want AI agents and vibe coders (AI-assisted developers and builders) to use, investing in an llms.txt, with a focus on your API documentation, seems like a good bet.
Getting started with llms.txt
If the llms.txt fits your specific use case, here’s how to approach it:
Create the basic file
Create a plain text file named llms.txt.
Add a list of your most relevant pages.
Host it at the root of your domain: example.com/llms.txt
Follow these best practices:
Keep it short and curate strictly - quality over quantity.
Use plain URLs, or even better, markdown with one URL per line.
Add brief comments or page titles to give context on links.
Focus on pages that help AI agents complete specific tasks on your website.
Testing your llms.txt
We recommend a multi-tiered approach to verify your implementation actually works:
Test 1: Verify retrieval
Ask ChatGPT to retrieve your llms.txt file. You might want to repeat this with multiple LLMs if your IT team has blocked AI bots in the past.
Test 2: Chat
Upload or copy and paste your file content into a new chat session. If possible, disable web search / grounding. Then instruct the LLM to answer questions based only on your file.
Then chat away. Ask the LLMs questions and see if the answers are correct.
Especially with the llms-full.txt file, the LLM should be able to answer even complex questions.
Test 3: Test agent performance (optional)
If you want AI agents to interact with your website, give an OpenAI (or other) agent a task on your website that typically takes too long or fails regularly. Then give it the llms.txt as context and measure whether performance improves.
Test 4: Test vibe coding tools (optional)
Provide a vibe coding tool (like Cursor, or Claude in coding mode) of your choice with your llms.txt and see if you can build a simple prototype that interacts correctly with your API.
Our verdict on llms.txt
The llms.txt is a helpful tool to give AI agents and coding tools a pointer on how to interact with your website. And the llms-full.txt can give AI coding tools all the context they need in a single place.
If AI gents, AI coding tools, and vibe coders are your target audience, you should seriously consider adding an llms.txt.
If you are an SEO looking to get more traffic from ChatGPT, Perplexity & Co, llms.txt and .md files are a distraction without any upside. And the .md files are potentially hurting your traditional SEO performance, which can negatively impact your AI visibility via web search / grounding.
