llms.txt

What is llms.txt?

The llms.txt file is a standardized text file in Markdown format designed to improve how Large Language Models (LLMs) access, understand, and process information from websites. Hosted at the root path of a website (e.g., /llms.txt), this file acts as a curated index that provides structured and summarized content specifically optimized for machine consumption during inference. Its primary goal is to bypass the complexities of traditional HTML content—such as navigation menus, advertisements, and JavaScript—by presenting clear, human- and machine-readable data.

Unlike other web standards like robots.txt or sitemap.xml, llms.txt is tailored explicitly for reasoning engines, such as ChatGPT, Claude, or Google Gemini, rather than search engines. It helps AI systems retrieve only the most relevant and valuable information within the constraints of their context windows, which are often too small to handle the entirety of a website’s content.

Origins of llms.txt

The concept was proposed by Jeremy Howard, co-founder of Answer.AI, in September 2024. It emerged as a solution to the inefficiencies faced by LLMs when interacting with complex websites. Traditional methods of processing HTML pages often lead to wasted computational resources and misinterpretation of content. By creating a standard like llms.txt, website owners can ensure that their content is parsed accurately and effectively by AI systems.

How is llms.txt Used?

The llms.txt file serves several practical purposes, primarily in the realm of artificial intelligence and LLM-driven interactions. Its structured format enables efficient retrieval and processing of website content by LLMs, overcoming limitations in context window size and processing efficiency.

Structure of an llms.txt File

The llms.txt file follows a specific Markdown-based schema to ensure compatibility with both humans and machines. The structure includes:

1. H1 Header: The title of the website or project.
2. Blockquote Summary: A concise description or summary of the website’s purpose and key features.
3. Detailed Sections: Freeform sections (e.g., paragraphs or lists) for additional context or critical details.
4. H2-Delimited Resource Lists: Categorized links to important resources, such as documentation, APIs, or external references. Each link may include a brief description of its content.
5. Optional Section (## Optional): Reserved for secondary resources that can be omitted to save space in the LLM’s context window.

Example:

# Example Website  
> A platform for sharing knowledge and resources about artificial intelligence.  

## Documentation  
- [Quick Start Guide](https://example.com/docs/quickstart.md): A beginner-friendly guide to getting started.  
- [API Reference](https://example.com/docs/api.md): Detailed API documentation.  

## Policies  
- [Terms of Service](https://example.com/terms.md): Legal guidelines for using the platform.  
- [Privacy Policy](https://example.com/privacy.md): Information on data handling and user privacy.  

## Optional  
- [Company History](https://example.com/history.md): A timeline of major milestones and achievements.

Key Features

• AI-Readable Navigation: Provides a simplified view of the website’s structure, making it easier for LLMs to identify relevant content.
• Markdown Format: Ensures human readability while allowing for programmatic parsing using tools like parsers or regex.
• Context Optimization: Helps LLMs prioritize high-value content by excluding unnecessary elements like ads or JavaScript.

Use Cases

1. Technical Documentation: Developers can link API references, quickstart guides, and other technical resources to facilitate coding assistants like GitHub Copilot or Codeium.
2. E-Commerce: Online retailers can use llms.txt to direct AI systems to product taxonomies, return policies, and sizing guides.
3. Education: Universities can highlight course syllabi, schedules, and enrollment policies for AI-driven student assistants.
4. Corporate FAQs: Businesses can streamline customer support by linking FAQs, troubleshooting guides, and policy documents.

Examples of llms.txt in Action

1. FastHTML

FastHTML, a Python library for building server-rendered web applications, uses llms.txt to simplify access to its documentation. Its file includes links to quickstart guides, HTMX references, and example applications, ensuring developers can quickly retrieve specific resources.

Example Snippet:

# FastHTML  
> A Python library for creating server-rendered hypermedia applications.  

## Docs  
- [Quick Start](https://fastht.ml/docs/quickstart.md): Overview of key features.  
- [HTMX Reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): Full HTMX attributes and methods.  

2. Nike (Hypothetical Example)

An e-commerce giant like Nike could use an llms.txt file to provide AI systems with information about their product lines, sustainability initiatives, and customer support policies.

Example Snippet:

# Nike  
> Global leader in athletic footwear and apparel, emphasizing sustainability and innovation.  

## Product Lines  
- [Running Shoes](https://nike.com/products/running.md): Details on React foam and Vaporweave technologies.  
- [Sustainability Initiatives](https://nike.com/sustainability.md): Goals for 2025 and eco-friendly materials.  

## Customer Support  
- [Return Policy](https://nike.com/returns.md): 60-day return window and exceptions.  
- [Size Guides](https://nike.com/sizing.md): Charts for footwear and apparel sizing.

llms.txt vs. Robots.txt vs. Sitemap.xml

Comparison

While all three standards are designed to assist automated systems, their purposes and target audiences differ significantly.

• llms.txt:

  • Audience: Large Language Models (e.g., ChatGPT, Claude, Google Gemini).
  • Purpose: Provides curated, context-optimized content for inference.
  • Format: Markdown.
  • Use Case: AI-driven interactions and reasoning engines.

• robots.txt:

  • Audience: Search engine crawlers.
  • Purpose: Controls crawling and indexing behavior.
  • Format: Plain text.
  • Use Case: SEO and access management.

• sitemap.xml:

  • Audience: Search engines.
  • Purpose: Lists all indexable pages on a site.
  • Format: XML.
  • Use Case: SEO and content discovery.

Key Advantages of llms.txt

1. AI-Specific Optimization: Unlike robots.txt and sitemap.xml, llms.txt is designed for reasoning engines, not traditional search engines.
2. Noise Reduction: Focuses only on high-value, machine-readable content, omitting unnecessary elements like ads or navigation menus.
3. Integration with Markdown: Aligns with the LLM-friendly format for easier parsing and processing.

Integration and Tools

Creating an llms.txt File

• Manual Creation: Use a text editor to write the file in Markdown format.
• Automated Tools:
  • Mintlify: Automatically generates llms.txt and llms-full.txt for hosted documentation.
  • Firecrawl Generator: Scrapes your website and creates llms.txt.

Hosting and Validation

• Place the file in the root directory of your website (e.g., https://example.com/llms.txt).
• Validate the file using tools like llms_txt2ctx to ensure compliance with the standard.

Integration with AI Systems

• Direct Upload: Some AI tools allow users to upload llms.txt or llms-full.txt files directly (e.g., Claude or ChatGPT).
• Frameworks: Use tools like LangChain or LlamaIndex to integrate the file into retrieval-augmented generation workflows.

Challenges and Considerations

1. Adoption by Major LLM Providers: While llms.txt has gained traction among developers and smaller platforms, it is not yet officially supported by major LLM providers like OpenAI or Google.
2. Maintenance: The file must be updated regularly to reflect changes in content or structure.
3. Context Window Limitations: For extensive documentation, the llms-full.txt file may exceed the context window size of some LLMs.

Despite these challenges, llms.txt represents a forward-thinking approach to optimizing content for AI-driven systems. By adopting this standard, organizations can ensure their content is accessible, accurate, and prioritized in an AI-first world.

Research: Large Language Models (LLMs)

Large Language Models (LLMs) have become a dominant technology for natural language processing, powering applications such as chatbots, content moderation, and search engines. In “Lost in Translation: Large Language Models in Non-English Content Analysis” by Nicholas and Bhatia (2023), the authors provide a clear technical explanation of how LLMs work, highlighting the data availability gap between English and other languages and discussing the efforts to bridge this gap through multilingual models. The paper details the challenges of content analysis using LLMs, especially for multilingual contexts, and offers recommendations for researchers, companies, and policymakers regarding the deployment and development of LLMs. The authors emphasize that while progress has been made, significant limitations remain for non-English languages. Read the paper

The paper “Cedille: A large autoregressive French language model” by Müller and Laurent (2022) introduces Cedille, a large-scale French-specific language model. Cedille is open source and demonstrates superior performance on French zero-shot benchmarks compared to existing models, even rivaling GPT-3 for several tasks. The study also evaluates the safety of Cedille, showing improvements in toxicity through careful dataset filtering. This work highlights the importance and impact of developing LLMs optimized for specific languages. The paper underscores the need for language-specific resources in the LLM landscape. Read the paper

In “How Good are Commercial Large Language Models on African Languages?” by Ojo and Ogueji (2023), the authors assess the performance of commercial LLMs on African languages for both translation and text classification tasks. Their findings indicate that these models generally underperform on African languages, with better results in classification than translation. The analysis covers eight African languages from various language families and regions. The authors call for greater representation of African languages in commercial LLMs, given their rising adoption. This study highlights the current gaps and the need for more inclusive language model development. Read the paper

“Goldfish: Monolingual Language Models for 350 Languages” by Chang et al. (2024) investigates the performance of monolingual versus multilingual models for low-resource languages. The research demonstrates that large multilingual models often underperform compared to simple bigram models for many languages, as measured by FLORES perplexity. Goldfish introduces monolingual models trained for 350 languages, significantly improving performance for low-resource languages. The authors advocate for more targeted model development for lesser-represented languages. This work contributes valuable insight into the limitations of current multilingual LLMs and the potential of monolingual alternatives. Read the paper