The Complete Guide to llms.txt: How to Prepare Your Site for AI Agents

In short: llms.txt is a format proposed at llmstxt.org, not a standard. There is no evidence that ChatGPT, Gemini, Claude or Perplexity read this file when they answer, so do not expect more mentions from it. You can add it in 20 minutes as a tidy site summary for the tools that do use it.

When an AI agent visits a website, it needs to understand quickly what the site does. An llms.txt file is a plain-text document at the root of the site that holds such a summary. In the authors' idea it is something like robots.txt, but descriptive rather than restrictive.

What Is llms.txt?

llms.txt was proposed by Jeremy Howard (fast.ai) in 2024 as a lightweight format for helping AI systems understand websites. The concept is simple: a Markdown file at yourdomain.com/llms.txt that gives AI agents a structured overview of your site. The specification is at llmstxt.org. It is not a standard of any standards body.

Unlike robots.txt (which tells crawlers what not to do), llms.txt is descriptive: it says what your site is, what you do and where to find your most important content.

The llms.txt Format

The file uses Markdown with a specific structure. Here's the minimal valid format:

# Your Company Name

> One-line description of what you do and who you serve.

## Product
- [Feature 1](https://yourdomain.com/feature-1): What it does
- [Feature 2](https://yourdomain.com/feature-2): What it does

## Documentation
- [Getting Started](https://yourdomain.com/docs/start): Installation and setup
- [API Reference](https://yourdomain.com/docs/api): Full API documentation

## About
- [Pricing](https://yourdomain.com/pricing): Plans and pricing
- [About Us](https://yourdomain.com/about): Company background

An Example: Causabi's llms.txt

Six more real files — Stripe, Vercel, Cloudflare, Svelte and others — are broken down in llms.txt examples.

# Causabi

> Causabi checks how AI crawlers read a website and creates the files that
> make it readable for AI: robots.txt, Schema.org markup, FAQ, llms.txt.

## Product
- [AI score](https://causabi.com): Free check of how ready a site is for AI search (0-100)
- [Dashboard](https://causabi.com/dashboard): Fix files
- [Pricing](https://causabi.com/pricing): Mention monitoring is part of the paid plans

## Check areas
- robots_txt: AI bot access
- schema_org: Organization/LocalBusiness/SoftwareApplication structured data
- faq_schema: FAQPage JSON-LD
- content_depth: Amount of text on the page
- brand_signals: Name, phone, address and profile links (sameAs)
- freshness: Publication and update dates

## Integrations
- [PyPI package](https://pypi.org/project/causabi-geo/): CLI tool for local analysis
- [Open source](https://causabi.com/open-source): CLI and MCP server

## Contact
- Email: hello@causabi.com
- GitHub: https://github.com/SHADRINMMM/causabi-geo

llms-full.txt: The Extended Version

llms-full.txt is not part of the llmstxt.org proposal: it is a convention documentation sites use — the full text of the docs in one file rather than just links. It is mainly for developer tools and API docs where agents need the complete content without many requests. Real examples and sizes: llms.txt examples.

For most websites, llms.txt with links is enough. llms-full.txt makes sense if you have:

• An API or SDK with documentation you want AI agents to be able to use
• Content that changes frequently (agents can re-fetch the full file)
• Docs that are hard to navigate via regular links

What to Include in Your llms.txt

Required sections

• H1 header: Your brand/product name
• Blockquote description: 1-3 sentences describing what you do and for whom. This is the most important field of the format: by the spec it comes right after the title.

Strongly recommended sections

• Key features/product pages — link to pages that explain what you do in depth
• Documentation or guides — your best explanatory content
• Contact/About — helps show who is behind the site

What NOT to include

• Don't list every page on your site — link to the most important ones
• Don't include transactional pages (cart, checkout, login)
• Don't add marketing fluff: factual, concrete descriptions are more useful to people and agents alike
• Don't include internal URLs, staging links, or anything not publicly accessible

How to Create It

Option 1 — Manual: Create a llms.txt file in your public directory and write it yourself using the format above.

Option 2 — Automated with Causabi CLI:

pip install causabi-geo

# Create the fix files, llms.txt included
geo-optimizer fix https://yourdomain.com --output ./out

# Output: llms.txt and the other files are in ./out

The fix command analyzes the page and writes the fix files: a robots.txt patch, schema.json, faq-schema.json and llms.txt. There is no separate command for llms.txt alone.

Technical Deployment

Place the file at /llms.txt (root of your domain). Serve it with:

• Content-Type: text/plain
• Charset: UTF-8
• No authentication — must be publicly accessible without login
• Cache-Control: public, max-age=86400 (1 day is fine)

For Next.js, add the file to /public/llms.txt. For static sites, add it to your root. For WordPress, add it via a plugin that serves static files or via functions.php.

Does It Affect AI Answers?

The honest answer: there is no data. Major assistants have not said they read llms.txt, and we have seen no evidence that the file changes their answers. Treat it as hygiene, not a lever. What speaks for it:

• It costs nothing — it is a small text file
• Writing it makes you state in two or three sentences what the site does; that summary can be reused elsewhere
• Some developer tools and site audits look at it

Do not expect more mentions in ChatGPT, Gemini, Claude or Perplexity from it. If you have an evening for one thing only, first check that AI bots are not blocked in robots.txt.

Alternatives (without Causabi)

You don't need our tool to create llms.txt. Here are the alternatives:

• llmstxt.org — the spec site, with a simple generator tool
• Write it manually — takes about 20 minutes, just follow the format above. For small sites this is often the best option.
• Causabi WP Plugin — if you're on WordPress, our open source plugin serves an llms.txt file on the paid Pro plan and refreshes it every 7 days along with the markup (per the plugin's WordPress.org page; it does not mention updating on publish)
• Next.js app route — for Next.js sites, add app/llms.txt/route.ts to build it dynamically from your content

Related GEO guides