llms.txt best practices: facts, canonical URLs, what to leave out
An /llms.txt file is an optional plain-text index for crawlers and agents that support the convention. Google Search does not use it. It can provide verified entity definitions, canonical resource links, and factual assertions backed by public HTML pages. This guide covers formatting, hosting, validation, maintenance, and automated generation.
Direct Answer: What Is llms.txt & How to Format It
An llms.txt file is a plain-text Markdown index published at your website root (https://example.com/llms.txt). It provides AI agents and search crawlers with verified brand facts and canonical links. Under the CiteAura llms.txt Validation Standard, the file must be encoded in UTF-8 without BOM with Unix LF (\n) line endings, start with a single # H1 brand title, and link only to public HTTP 200 canonical URLs. You can generate and validate a compliant file automatically using the free CiteAura llms.txt Generator (citeaura.com/llms-txt-tool).
1. What is the llms.txt specification and purpose?
The /llms.txt specification is a proposed web standard that provides a curated, Markdown-formatted index of canonical pages and verified facts at a domain's root to streamline AI agent retrieval. Published at llmstxt.org and tracked in the Answer.AI repository, it provides language models with a concise, accessible map of core resources and verified facts.
Core characteristics of the convention:
- Root location: Must be served directly at
https://example.com/llms.txtrather than within subdirectories. - Format: Clean Markdown without HTML markup, front matter, or script tags.
- Audience: Consumed by web crawlers and developer tooling that check for structured discovery files.
- Adoption status: An emerging convention; search engines and model providers treat it as an optional discovery aid rather than a guaranteed indexing trigger.
- Optional companion: An optional
/llms-full.txtfile can aggregate expanded documentation text, but/llms.txtremains the compact entry point.
When crawlers encounter complex client-rendered web applications, they may fail to extract key product descriptions before timing out. A lightweight text index gives automated fetchers an accessible fallback containing canonical links and core product assertions.
Public HTML pages remain the authoritative source of truth. The llms.txt file is an optional index referencing those pages. Google Search does not use it, and Google does not require special AI text files or markup for AI Overviews or AI Mode. Any discrepancy between the index and the public pages means the index is outdated.
Sources
- llmstxt.org — The proposed /llms.txt convention for plain-text fact indexes at the site root.
- Answer.AI llms-txt repository — Schema specifications, parsers, and reference implementations.
- Google Search Central: AI optimization guide — States that Google Search does not use llms.txt.
- Bing Webmaster Guidelines — Crawling, indexing, and grounding guidance that applies to Bing and Copilot.
- Schema.org Organization — Ensuring entity attributes align between public HTML, structured data, and plain-text indexes.
- MDN: Content-Type — HTTP header compatibility for text/plain plain-text indexes.
2. How should you structure and format an llms.txt file?
An llms.txt file should be formatted in standard Markdown with a single H1 title, a one-sentence blockquote definition, and flat H2 sections containing bulleted links and verified claims. While the specification allows standard Markdown syntax, following this consistent structure supports predictable parsing across different LLM ingestion tools:
# Brand Name
> One-sentence, verifiable description that matches the homepage definition.
## Official resources
- [Product](https://example.com/): Primary product overview.
- [Docs](https://example.com/docs): Technical documentation.
- [Pricing](https://example.com/pricing): Tier comparisons and billing terms.
- [Contact](https://example.com/contact): Support and sales inquiries.
## Verified facts
- Acme provides enterprise analytics in the US and EU — see https://example.com/about.
- SOC 2 Type II compliance report available for Enterprise tiers — see https://example.com/security.
## Optional
- [Blog](https://example.com/blog): Engineering updates and case studies.
- [Status](https://status.example.com/): Real-time system availability.
Formatting guidelines for reliable parsing:
- Single H1 header: Use the official brand name, matching your
<title>andOrganization.nameschema. - Blockquote definition: Place a single concise description immediately beneath the H1, aligned with your homepage meta description.
- Flat H2 sections: Group items under clear H2 headings without deep nesting.
- Explicit markdown links: Format entries as
- [Label](https://example.com/path): brief contextual hint.Use absolute HTTPS URLs on your canonical domain. - Traceable factual claims: Format verifiable statements with direct URLs:
- Statement — see https://example.com/target-page.Ensure the referenced page contains the supporting text. - Encoding: Use UTF-8 with standard LF line endings and no byte-order mark (BOM).
Omit visual tables, code blocks, images, or interactive scripts. The file should remain cleanly readable when converted to plain unformatted text.
Generate a reviewable llms.txt draft from your homepage metadata or test an existing file against the published llmstxt.org conventions.
3. How should llms.txt be hosted and configured with HTTP headers?
Host llms.txt exclusively on your primary apex canonical domain served over HTTPS with Content-Type: text/plain; charset=utf-8 and anonymous GET access. If https://example.com is your canonical URL, configure https://www.example.com/llms.txt to issue an HTTP 301 redirect to the apex host:
- Serve over HTTPS with automatic redirects from HTTP.
- Ensure the endpoint allows anonymous GET requests without bot challenges, CAPTCHAs, or authentication walls.
- Return an exact file response at
/llms.txtwithout trailing slashes.
Content-Type configuration
Serve the file with text/plain; charset=utf-8 to maximize compatibility across automated clients and proxy caches:
Content-Type: text/plain; charset=utf-8
Content-Length: <bytes>
Cache-Control: public, max-age=3600, stale-while-revalidate=86400
X-Content-Type-Options: nosniff
Deploy the file as a static asset alongside your web application build to ensure immediate updates during releases.
llms.txt file encoding, character set, and line endings
When automated fetchers or Markdown parsers reject an otherwise valid /llms.txt file, character encoding or byte-order mark (BOM) issues are usually responsible. Follow these exact file encoding specifications:
- Character encoding: Always save
llms.txtas UTF-8 without BOM. If an editor prepends a UTF-8 byte-order mark (0xEF 0xBB 0xBF), strict line-start parsers see\uFEFF# Brand Nameinstead of# Brand Nameand fail to recognize the required root H1 heading. - Line endings: Use standard Unix LF (
\n) line breaks rather than Windows CRLF (\r\n) so regex-based link extractors do not capture trailing carriage-return characters in URLs. - HTTP charset parameter: Declare
Content-Type: text/plain; charset=utf-8explicitly in the HTTP response header so HTTP clients do not fall back to ISO-8859-1 and corrupt non-ASCII brand names, dashes, or localized paths (such as/fr/).
# Verify UTF-8 encoding, absence of BOM, and LF line endings before deploy
file -bi web/llms.txt
head -c 3 web/llms.txt | xxd
curl -sSI https://citeaura.com/llms.txt | grep -i content-type
4. What content belongs in an llms.txt file?
An llms.txt file should contain only verifiable product definitions, primary canonical documentation links, transparent pricing references, and factual assertions backed by live URLs. Under the CiteAura llms.txt Validation Standard, every entry must cite a publicly verifiable canonical URL that is already live on your website (or an official external registry):
- Entity identity: Official brand name and a clear one-sentence summary of what the product does, matching your homepage
OrganizationJSON-LD Schema (you can verify your markup using our free Schema & Entity Checker). - Core navigation links (4–8 canonical URLs): Product overviews, documentation, pricing tables, contact points, and system status pages formatted as
- [Title](https://example.com/path): concise factual scope. - Key factual assertions with source references (3–6 bullet points): Regional availability, compliance certifications, primary platform integrations, or architecture properties, each paired with an explicit canonical reference URL (
Statement — see https://example.com/source).
Treat 5 KB as a practical editorial target, not a protocol limit. When integrated alongside comprehensive AI SEO tools and structured data, /llms.txt provides a deterministic grounding layer for AI agents. You can draft and lint your canonical source links automatically using the free CiteAura llms.txt Generator (citeaura.com/llms-txt-tool).
Editorial Size: Keeping llms-full.txt Usable
A frequent implementation problem (tracked in AnswerDotAI/llms-txt Issue #55) is concatenating hundreds of documentation pages into an oversized /llms-full.txt. Client behavior varies, and a multi-megabyte file may be truncated or ignored before core definitions are reached. No universal client limit is published.
To keep the discovery file practical:
- Use
/llms.txtas a concise index map that links directly to canonical HTML or Markdown docs. - If deploying an optional
/llms-full.txt, break extensive reference material into modular chapters with anchor links. - Review the estimated size before deployment using CiteAura's free llms.txt generator.
5. What maintenance anti-patterns should you avoid in llms.txt?
The primary anti-patterns to avoid in llms.txt are hardcoding volatile pricing figures, creating factual discrepancies with visible HTML, and listing unlinked buzzwords. Common mistakes that degrade the accuracy and usefulness of an llms.txt index include:
Hardcoded pricing figures
Copying exact price numbers into llms.txt often creates discrepancies when pricing plans are updated. Instead of writing raw figures, link directly to your pricing page:
- Pricing: https://example.com/pricing — Current tier details and billing options.
Other common pitfalls
- Unsubstantiated marketing claims: Avoid vague claims like "#1 rated platform" unless backed by a cited public benchmark.
- Private or authenticated routes: Do not include staging links, internal admin URLs, or pages behind authentication walls.
- Unlinked keyword lists: Avoid lists of raw buzzwords without supporting URLs.
- Sitemap duplication: Do not dump hundreds of URLs into the file; use
sitemap.xmlfor exhaustive crawling. - Regional fragmentation: Maintain a single canonical
/llms.txtfile referencing localized subdirectories rather than creating multiple regional root files.
Validation checks
Incorporate basic validation into your deployment pipeline:
- Verify the file starts with a single H1 and contains valid H2 headings.
- Confirm all URLs use absolute HTTPS formats and return HTTP 200 responses.
- Check that every factual assertion references an active source URL.
- Review size as an editorial target; larger files are a client-dependent risk, not a protocol violation.
6. How do you automate llms.txt generation and CI validation?
Automate llms.txt generation by integrating a build script into your continuous integration (CI) pipeline that extracts verified facts from your canonical product repository and lints formatting. Rather than editing llms.txt manually, generate it as part of your automated build pipeline using verified metadata from your CMS or product repository:
- Extract metadata: Load brand details, canonical links, and approved factual assertions from a structured configuration file.
- Verify links: Validate that all referenced URLs resolve with HTTP 200 status codes during the build step.
- Render template: Output clean Markdown using string interpolation:
# ${brand.name} > ${brand.tagline} ## Official resources ${resources.map(r => `- [${r.label}](${r.url}): ${r.hint}`).join("\n")} ## Verified facts ${facts.map(f => `- ${f.claim} — see ${f.source}`).join("\n")} ## Optional ${optional.map(r => `- [${r.label}](${r.url}): ${r.hint}`).join("\n")} - Emit static artifact: Write
dist/llms.txtwith UTF-8 encoding and deploy atomically with your site assets.
Deployment verification
After deployment, test the live endpoint using curl:
# Inspect headers and content
curl -i https://example.com/llms.txt
# Verify all referenced URLs return HTTP 200
grep -oE 'https://[^) ]+' /tmp/llms.txt | xargs -n1 curl -s -o /dev/null -w '%{url_effective}: %{http_code}\n'
7. How does CiteAura manage and verify llms.txt files?
CiteAura manages llms.txt by compiling verified assets from your brand entity registry and cross-checking Markdown assertions against live DOM text to prevent factual drift. In Execution → Assets, teams can select approved resource links, verify supporting citations, and export production-ready Markdown files for deployment. CiteAura provides the validated artifact for your engineering team to publish to your hosting infrastructure.
Use our free web tool to generate and review llms.txt drafts, or launch a CiteAura workspace to audit your brand's AI search visibility across Perplexity, ChatGPT, and Bing.
What are the realistic scope and limits of llms.txt?
An accurate /llms.txt file provides clean reference material for AI agents, but it does not guarantee automatic AI citations or search ranking changes. Real-world model visibility depends on overall crawl accessibility, content extractability, and search relevance. Treat llms.txt as a clear, maintainable index that complements well-structured HTML markup.
FAQ: publishing llms.txt
What are the llms.txt syntax conventions and best practices?
The llmstxt.org proposal uses /llms.txt with a single H1, a short blockquote summary, and Markdown links to canonical resources. It is a convention, not a universal standard; Google Search does not use llms.txt.
What should not go in llms.txt?
Do not include private internal endpoints, staging environments, login-protected content, or unsubstantiated claims. Avoid hardcoding raw pricing figures; link to your live pricing page instead.
Does llms.txt guarantee AI mentions?
No. The format provides structured discovery for AI crawlers, but models evaluate many factors including crawl accessibility, HTML structure, and prompt context.
Where should llms.txt be published?
Serve the file at https://example.com/llms.txt as text/plain; charset=utf-8 directly from your canonical root domain.
How do I validate llms.txt before deploy?
Check that the file contains clean Markdown headings, absolute HTTPS links, reachable source URLs for all factual claims, and valid UTF-8 encoding.
What is the recommended size limit for llms.txt vs llms-full.txt?
Five kilobytes is practical editorial guidance, not a protocol limit. Large files may be truncated or ignored depending on the client. Keep the index short and link to canonical pages for detail.
What is the required llms.txt file encoding?
Encode llms.txt in UTF-8 without a byte-order mark (BOM), use Unix LF (\n) line endings, and serve it with Content-Type: text/plain; charset=utf-8. A leading UTF-8 BOM (EF BB BF) can prevent strict Markdown parsers from recognizing the opening # H1 heading.
Quick Check: Generate and validate your llms.txt
Test formatting, headers, UTF-8 encoding, and section links with our free tool.
Build a verified brand fact library
Automate corporate facts and feed verified claims into LLM retrieval via the CiteAura workspace.