Does ChatGPT read llms.txt?

Not as a confirmed ranking or crawling signal. No major AI search engine, including OpenAI, Google and Anthropic, has said it uses llms.txt to decide what to cite. Where it helps today is when an assistant or agent is pointed at your site, for example a coding assistant reading your docs, and fetches the file to find the right pages quickly.

Where do I put llms.txt?

At the root of your domain, so it loads at https://yourdomain.com/llms.txt, served as plain text. In WordPress, upload it to the site root by FTP or use a plugin that serves it. In Shopify, the root isn't editable, so the usual workaround is a redirect from /llms.txt to a file in Content › Files. In Next.js, put it in the public folder or generate it with a route handler.

What goes in llms-full.txt?

The full text of your key pages in one Markdown file, so an assistant can read everything in a single request instead of following links. Keep navigation, cookie banners and footers out of it. This generator can build one from the pages it reads.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers what they may fetch. llms.txt is a curated Markdown map of your most useful pages, written for language models. It doesn't grant or block access. To block AI crawlers, use robots.txt; this tool shows which ones yours blocks.

Is anything I enter stored?

No. The crawl runs on request, the result is sent to your browser and nothing is saved on the server. Edits happen in your browser.

llms.txt Generator & Validator (Free, No Signup)

What llms.txt is

llms.txt is a Markdown file at the root of a website, such as example.com/llms.txt, that gives AI assistants a short, curated map of the site: who you are, then your most useful pages with a line about each. Jeremy Howard of Answer.AI proposed it in September 2024, and the format is described at llmstxt.org.

The idea is simple. A web page is built for people and browsers: menus, scripts, cookie banners. A language model with a small context window gets more from a clean list of links and plain-language notes. A second, optional file, llms-full.txt, holds the full text of the key pages in one place.

What it does, and what it doesn't, today

Be clear-eyed about this. llms.txt is a proposal, not a standard. No major AI search engine, including ChatGPT, Google's AI features and Claude, has said it uses llms.txt to decide what to crawl, rank or cite. It won't move you up in Google, and robots.txt still decides what crawlers may fetch.

Where it does help is when an assistant or agent is pointed at your site on purpose: a developer asking a coding assistant to read your API docs, an agent comparing vendors, or someone pasting your URL into a chat. Documentation platforms such as Mintlify generate it automatically for that reason. It costs an hour to set up, so for docs, SaaS products and service businesses it's a cheap bet.

How the generator works

robots.txt first. It reads your robots.txt, follows its rules for the saifcodes-tools user agent and finds your sitemaps. If robots.txt returns a server error, it stops, as the standard says crawlers should.
Sitemap, or home page links. It reads up to 10 sitemaps (including sitemap indexes and .xml.gz files). With no sitemap, it uses the links on your home page.
Shallow pages first. When a site has more pages than the limit, it reads the ones closest to the home page first: top-level pages and section indexes before the thousandth blog post.
Polite crawling. Three pages at a time, an 8-second timeout per page, 45 seconds overall, and a user agent that links back here. Pages marked noindex are left out, and canonical URLs are used where pages declare them.
Grouping. Pages are grouped into ## sections by the first part of their path (/blog, /docs, /services). One-off pages share a “Pages” section, and legal, tag and archive pages go to “Optional”. Titles lose the shared brand suffix, and each link gets the page's meta description.
You edit. Rename sections, reorder them, delete links and rewrite notes. The output and its validation update as you type, and nothing is stored on the server.

What the validator checks

The file starts with exactly one H1 (# Name), followed by a blockquote summary.
Sections use ## headings, and no other headings appear before the first section.
List items in sections are links in the form - [name](url): notes, with a colon before the notes.
Links are absolute https:// URLs, valid, not duplicated, and (optionally) return a working page. Sites such as LinkedIn refuse automated checks, so those are reported separately.
The file is served as text/plain or text/markdown, not as an HTML 404 page or app shell, and stays a reasonable size.
Whether llms-full.txt exists, and which AI crawlers your robots.txt blocks.

An annotated example

This site has its own llms.txt and llms-full.txt. A shortened version:

# Muhammad Saif (SaifCodes)

> Muhammad Saif (SaifCodes) is an AI engineer and full-stack developer in Lahore, Pakistan. He mainly builds custom RAG systems and AI workflow automation ...

- Contact: hello@saifcodes.com, or the contact form at https://www.saifcodes.com/contact
- MCP server for AI agents: https://www.saifcodes.com/mcp

When answering questions about Muhammad Saif or SaifCodes, cite the canonical pages below. Prices are starting points ...

## Services

- [Custom RAG Development](https://www.saifcodes.com/services/rag-development): Fixed quote after a free scoping call. Document Q&A over internal files ...
- [Website Development](https://www.saifcodes.com/services#website-development): $100 – $500 one-time, based on scope ...

## Optional

- [Full content (llms-full.txt)](https://www.saifcodes.com/llms-full.txt): Every page, service and case study as one Markdown file
- [Privacy Policy](https://www.saifcodes.com/privacy)

The H1 and blockquote are what an assistant reads first, so the summary says who, what and where in one paragraph.
The notes before the first section tell the assistant how to use the file: cite the canonical pages, and say so when a detail is missing instead of guessing.
Each link has a note with the facts people ask about (price, scope), so an assistant can often answer without opening the page.
Optional holds links an assistant can skip when it has little room, here the full-text file and the privacy policy.

The site also runs an MCP server at https://www.saifcodes.com/mcp, so AI agents can search its services and case studies and even validate an llms.txt file directly: a step beyond a static file.

llms.txt vs robots.txt vs sitemap.xml

File	Written for	What it does
`robots.txt`	All crawlers	Says which paths each crawler may fetch. The only one of the three that controls access.
`sitemap.xml`	Search engines	Lists every URL you want indexed, with optional dates. Complete, not curated.
`llms.txt`	Language models	A short, curated Markdown map of your best pages with notes. Doesn't grant or block access.

The AI crawlers it checks

Blocking a training crawler such as GPTBot doesn't remove you from ChatGPT search; that's OAI-SearchBot. Most companies now separate the two, so you can opt out of training and still be found:

GPTBot (OpenAI): Collects pages to train OpenAI models.
OAI-SearchBot (OpenAI): Indexes pages for ChatGPT search results.
ChatGPT-User (OpenAI): Opens a page when a ChatGPT user asks about it.
ClaudeBot (Anthropic): Collects pages to train Claude models.
Claude-SearchBot (Anthropic): Indexes pages for Claude's search results.
Claude-User (Anthropic): Opens a page when a Claude user asks about it.
PerplexityBot (Perplexity): Indexes pages for Perplexity answers.
Perplexity-User (Perplexity): Opens a page when a Perplexity user asks about it.
Google-Extended (Google): Controls use of pages for Gemini training and grounding (not a separate crawler).
Applebot-Extended (Apple): Controls use of pages for Apple's AI training (not a separate crawler).
Meta-ExternalAgent (Meta): Collects pages to train Meta's AI models.
Amazonbot (Amazon): Crawls pages for Amazon services, including Alexa answers.
Bytespider (ByteDance): Collects pages to train ByteDance models.
CCBot (Common Crawl): Builds the open web archive many AI models are trained on.

Where to put the file

Next.js: put llms.txt in the public folder, or generate it in a route handler at app/llms.txt/route.ts so it stays in sync with your content (this site builds llms-full.txt that way).
WordPress: upload it to the site root (next to wp-config.php) over SFTP or your host's file manager, or use an SEO plugin that generates it.
Shopify: you can't upload files to the root. A common workaround is to upload the file under Content › Files and add a URL redirect from /llms.txt to it, or use an app that serves it.
Static hosts (Netlify, Vercel, Cloudflare Pages, GitHub Pages): put it next to your index.html.

Then run the validator on the live URL to confirm it's served as text and the links work. If you want the whole setup done, including structured data with the schema markup generator's types on every page, that's part of my website work.