THE SHORT ANSWER
Robots.txt is an access-control file that reputable crawlers obey: it says which paths may be fetched. llms.txt is an advisory index in markdown that says which pages are worth reading and what each one contains. One is enforcement, the other is curation. They do not overlap, they are not alternatives, and a site that wants AI visibility benefits from having both plus a real XML sitemap.
The two files are constantly confused because they share a location and a naming convention. They do entirely different jobs.
A useful way to hold the difference: robots.txt is the lock on the door, llms.txt is the directory in the lobby.
The numbers, at a glance
robots.txt: access control, established since 1994, honoured by reputable crawlers
llms.txt: content curation, proposed 2024, partial adoption, advisory only
sitemap.xml: discovery, universal support, machine-readable list of every indexable URL
Correct answer: run all three; they answer three different questions
What each file actually says
robots.txt says "you may or may not fetch this path". It is a permission statement, addressed to named user-agents, and it has real consequences: a disallowed path generally will not be crawled by any operator that follows the convention.
llms.txt says "here is what is worth reading and why". It is a recommendation, addressed to nobody in particular, with no enforcement mechanism and no obligation on the reader.
sitemap.xml says "these URLs exist". It is a discovery aid with universal support and no opinion about value.
Why you cannot substitute one for the other
Publishing llms.txt does not grant access to anything robots.txt blocks. If a page is disallowed and also listed in llms.txt, the disallow wins with any well-behaved agent, and you have published a public index pointing at content you have simultaneously locked. That combination is common and always accidental.
Equally, allowing everything in robots.txt does not tell a model which of your four hundred pages carries the answer. A permissive robots.txt with no curation means a crawler treats your cookie policy and your pricing page as equally plausible sources, and it will sometimes choose wrong.
The order of operations that actually matters
Fix robots.txt first. Nothing downstream works if access is blocked, and inherited blanket disallows are the single most common cause of invisible sites.
Then the XML sitemap. Complete, current, submitted, and free of URLs that return anything other than a 200.
Then page structure. Self-contained sections, answers in the first fifty words, dated figures.
Then llms.txt. A curated index is only useful once there is something worth curating.
Teams routinely do this in reverse, publishing an llms.txt on a site whose robots.txt still blocks the whole domain from a staging configuration. The index is then a beautifully written map of a locked building.
How to audit both in five minutes
Fetch each file directly in a browser and read it, rather than trusting the platform's settings panel. For robots.txt, confirm there is no blanket disallow and that each AI agent you care about is either explicitly allowed or not mentioned. For llms.txt, confirm it returns plain text rather than a rendered HTML page, and that every link in it resolves to a live URL.
Then check one thing that catches almost everyone: whether your CDN's bot protection is rejecting AI agents at the network layer. That override makes both files irrelevant, and it produces exactly the same symptom as a misconfigured robots.txt.
The five-minute audit
Fetch robots.txt in a browser and read every line.
Confirm no inherited blanket disallow from staging or a migration.
Fetch sitemap.xml and confirm the URL count matches what you expect.
Fetch llms.txt and confirm it serves as plain text, not a rendered page.
Open the CDN bot-protection panel and check the AI-crawler category.
Want leads like this in your pipeline?
Flock runs the campaigns, screens the enquiries and hands you only the ones that match your service area, job size and capacity. You pay per lead, not per month.
Book a 15-minute fit check | See lead package pricing
Related answers
Frequently asked questions
Do I need llms.txt if I already have a good sitemap?
A sitemap lists everything; llms.txt says what matters and describes it. The descriptions are the added value, and they cost an hour.
Can llms.txt block AI crawlers?
No. It has no enforcement mechanism at all. Blocking belongs in robots.txt, and genuinely sensitive material belongs behind authentication.
Which do the major assistants actually read?
Robots.txt, universally, by every reputable operator. llms.txt support is partial and evolving, which is why it is a cheap bet rather than a priority.
What is the most common mistake with these files?
Disallowing a path in robots.txt while listing it in llms.txt or the sitemap. Keep the three files consistent, and check consistency after every migration.
NEED A CLEARER PLAN?
Let’s turn your next move into momentum.
Talk to us →