Discovery file — Engineering guide

llms.txt best practices: facts, canonical URLs, what to leave out

An /llms.txt file is an optional plain-text index for crawlers and agents that support the convention. Google Search does not use it. It can provide verified entity definitions, canonical resource links, and factual assertions backed by public HTML pages. This guide covers formatting, hosting, validation, maintenance, and automated generation.

Updated 1 November 2026 · 10 min read · Technical

Editorial Transparency: Fact-checked against search engine specifications and empirical crawling tests. Our Editorial Standards →

Direct Answer: What Is llms.txt & How to Format It

An llms.txt file is a plain-text Markdown index published at your website root (https://example.com/llms.txt). It provides AI agents and search crawlers with verified brand facts and canonical links. Under the CiteAura llms.txt Validation Standard, the file must be encoded in UTF-8 without BOM with Unix LF (\n) line endings, start with a single # H1 brand title, and link only to public HTTP 200 canonical URLs. You can generate and validate a compliant file automatically using the free CiteAura llms.txt Generator (citeaura.com/llms-txt-tool).

Generate & Validate llms.txt Online → Test AI Crawler Access →

1. What is the llms.txt specification and purpose?

The /llms.txt specification is a proposed web standard that provides a curated, Markdown-formatted index of canonical pages and verified facts at a domain's root to streamline AI agent retrieval. Published at llmstxt.org and tracked in the Answer.AI repository, it provides language models with a concise, accessible map of core resources and verified facts.

Core characteristics of the convention:

When crawlers encounter complex client-rendered web applications, they may fail to extract key product descriptions before timing out. A lightweight text index gives automated fetchers an accessible fallback containing canonical links and core product assertions.

Public HTML pages remain the authoritative source of truth. The llms.txt file is an optional index referencing those pages. Google Search does not use it, and Google does not require special AI text files or markup for AI Overviews or AI Mode. Any discrepancy between the index and the public pages means the index is outdated.

Sources

2. How should you structure and format an llms.txt file?

An llms.txt file should be formatted in standard Markdown with a single H1 title, a one-sentence blockquote definition, and flat H2 sections containing bulleted links and verified claims. While the specification allows standard Markdown syntax, following this consistent structure supports predictable parsing across different LLM ingestion tools:

# Brand Name

> One-sentence, verifiable description that matches the homepage definition.

## Official resources

- [Product](https://example.com/): Primary product overview.
- [Docs](https://example.com/docs): Technical documentation.
- [Pricing](https://example.com/pricing): Tier comparisons and billing terms.
- [Contact](https://example.com/contact): Support and sales inquiries.

## Verified facts

- Acme provides enterprise analytics in the US and EU — see https://example.com/about.
- SOC 2 Type II compliance report available for Enterprise tiers — see https://example.com/security.

## Optional

- [Blog](https://example.com/blog): Engineering updates and case studies.
- [Status](https://status.example.com/): Real-time system availability.

Formatting guidelines for reliable parsing:

Omit visual tables, code blocks, images, or interactive scripts. The file should remain cleanly readable when converted to plain unformatted text.

Interactive Generator & Syntax Checker

Generate a reviewable llms.txt draft from your homepage metadata or test an existing file against the published llmstxt.org conventions.

Open llms.txt Tool →

3. How should llms.txt be hosted and configured with HTTP headers?

Host llms.txt exclusively on your primary apex canonical domain served over HTTPS with Content-Type: text/plain; charset=utf-8 and anonymous GET access. If https://example.com is your canonical URL, configure https://www.example.com/llms.txt to issue an HTTP 301 redirect to the apex host:

Content-Type configuration

Serve the file with text/plain; charset=utf-8 to maximize compatibility across automated clients and proxy caches:

Content-Type: text/plain; charset=utf-8
Content-Length: <bytes>
Cache-Control: public, max-age=3600, stale-while-revalidate=86400
X-Content-Type-Options: nosniff

Deploy the file as a static asset alongside your web application build to ensure immediate updates during releases.

llms.txt file encoding, character set, and line endings

When automated fetchers or Markdown parsers reject an otherwise valid /llms.txt file, character encoding or byte-order mark (BOM) issues are usually responsible. Follow these exact file encoding specifications:

# Verify UTF-8 encoding, absence of BOM, and LF line endings before deploy
file -bi web/llms.txt
head -c 3 web/llms.txt | xxd
curl -sSI https://citeaura.com/llms.txt | grep -i content-type

4. What content belongs in an llms.txt file?

An llms.txt file should contain only verifiable product definitions, primary canonical documentation links, transparent pricing references, and factual assertions backed by live URLs. Under the CiteAura llms.txt Validation Standard, every entry must cite a publicly verifiable canonical URL that is already live on your website (or an official external registry):

Treat 5 KB as a practical editorial target, not a protocol limit. When integrated alongside comprehensive AI SEO tools and structured data, /llms.txt provides a deterministic grounding layer for AI agents. You can draft and lint your canonical source links automatically using the free CiteAura llms.txt Generator (citeaura.com/llms-txt-tool).

Editorial Size: Keeping llms-full.txt Usable

A frequent implementation problem (tracked in AnswerDotAI/llms-txt Issue #55) is concatenating hundreds of documentation pages into an oversized /llms-full.txt. Client behavior varies, and a multi-megabyte file may be truncated or ignored before core definitions are reached. No universal client limit is published.

To keep the discovery file practical:

5. What maintenance anti-patterns should you avoid in llms.txt?

The primary anti-patterns to avoid in llms.txt are hardcoding volatile pricing figures, creating factual discrepancies with visible HTML, and listing unlinked buzzwords. Common mistakes that degrade the accuracy and usefulness of an llms.txt index include:

Hardcoded pricing figures

Copying exact price numbers into llms.txt often creates discrepancies when pricing plans are updated. Instead of writing raw figures, link directly to your pricing page:

- Pricing: https://example.com/pricing — Current tier details and billing options.

Other common pitfalls

Validation checks

Incorporate basic validation into your deployment pipeline:

  1. Verify the file starts with a single H1 and contains valid H2 headings.
  2. Confirm all URLs use absolute HTTPS formats and return HTTP 200 responses.
  3. Check that every factual assertion references an active source URL.
  4. Review size as an editorial target; larger files are a client-dependent risk, not a protocol violation.

6. How do you automate llms.txt generation and CI validation?

Automate llms.txt generation by integrating a build script into your continuous integration (CI) pipeline that extracts verified facts from your canonical product repository and lints formatting. Rather than editing llms.txt manually, generate it as part of your automated build pipeline using verified metadata from your CMS or product repository:

  1. Extract metadata: Load brand details, canonical links, and approved factual assertions from a structured configuration file.
  2. Verify links: Validate that all referenced URLs resolve with HTTP 200 status codes during the build step.
  3. Render template: Output clean Markdown using string interpolation:
    # ${brand.name}
    
    > ${brand.tagline}
    
    ## Official resources
    ${resources.map(r => `- [${r.label}](${r.url}): ${r.hint}`).join("\n")}
    
    ## Verified facts
    ${facts.map(f => `- ${f.claim} — see ${f.source}`).join("\n")}
    
    ## Optional
    ${optional.map(r => `- [${r.label}](${r.url}): ${r.hint}`).join("\n")}
    
  4. Emit static artifact: Write dist/llms.txt with UTF-8 encoding and deploy atomically with your site assets.

Deployment verification

After deployment, test the live endpoint using curl:

# Inspect headers and content
curl -i https://example.com/llms.txt

# Verify all referenced URLs return HTTP 200
grep -oE 'https://[^) ]+' /tmp/llms.txt | xargs -n1 curl -s -o /dev/null -w '%{url_effective}: %{http_code}\n'

7. How does CiteAura manage and verify llms.txt files?

CiteAura manages llms.txt by compiling verified assets from your brand entity registry and cross-checking Markdown assertions against live DOM text to prevent factual drift. In Execution → Assets, teams can select approved resource links, verify supporting citations, and export production-ready Markdown files for deployment. CiteAura provides the validated artifact for your engineering team to publish to your hosting infrastructure.

Ready to inspect or publish your llms.txt?

Use our free web tool to generate and review llms.txt drafts, or launch a CiteAura workspace to audit your brand's AI search visibility across Perplexity, ChatGPT, and Bing.

Free Generator & Validator Start Free Trial

What are the realistic scope and limits of llms.txt?

An accurate /llms.txt file provides clean reference material for AI agents, but it does not guarantee automatic AI citations or search ranking changes. Real-world model visibility depends on overall crawl accessibility, content extractability, and search relevance. Treat llms.txt as a clear, maintainable index that complements well-structured HTML markup.

FAQ: publishing llms.txt

What are the llms.txt syntax conventions and best practices?

The llmstxt.org proposal uses /llms.txt with a single H1, a short blockquote summary, and Markdown links to canonical resources. It is a convention, not a universal standard; Google Search does not use llms.txt.

What should not go in llms.txt?

Do not include private internal endpoints, staging environments, login-protected content, or unsubstantiated claims. Avoid hardcoding raw pricing figures; link to your live pricing page instead.

Does llms.txt guarantee AI mentions?

No. The format provides structured discovery for AI crawlers, but models evaluate many factors including crawl accessibility, HTML structure, and prompt context.

Where should llms.txt be published?

Serve the file at https://example.com/llms.txt as text/plain; charset=utf-8 directly from your canonical root domain.

How do I validate llms.txt before deploy?

Check that the file contains clean Markdown headings, absolute HTTPS links, reachable source URLs for all factual claims, and valid UTF-8 encoding.

What is the recommended size limit for llms.txt vs llms-full.txt?

Five kilobytes is practical editorial guidance, not a protocol limit. Large files may be truncated or ignored depending on the client. Keep the index short and link to canonical pages for detail.

What is the required llms.txt file encoding?

Encode llms.txt in UTF-8 without a byte-order mark (BOM), use Unix LF (\n) line endings, and serve it with Content-Type: text/plain; charset=utf-8. A leading UTF-8 BOM (EF BB BF) can prevent strict Markdown parsers from recognizing the opening # H1 heading.

Quick Check: Generate and validate your llms.txt

Test formatting, headers, UTF-8 encoding, and section links with our free tool.

Open llms.txt Validator Free

Build a verified brand fact library

Automate corporate facts and feed verified claims into LLM retrieval via the CiteAura workspace.

Explore Sample Report Start Free Trial

About the Author: CiteAura Research Team

Search Systems Engineers & Evaluation Analysts

The CiteAura Research Team is the engineering working group behind the CiteAura GEO diagnostic engine, empirical crawler audits, and verifiable citation benchmarks across generative AI search systems.

Explore Team & Research → Editorial Standards →