Sitemap to llms.txt AI Converter

Transform any sitemap.xml into a well-structured llms.txt format using AI. This workflow fetches URLs from a sitemap, retrieves and processes their content, and leverages an AI agent to generate an optimized llms.txt file suitable for AI training or knowledge ingestion.

Thumbnail for Video
How the AI Flow works - Sitemap to llms.txt AI Converter

Flows

How the AI Flow works

Display Welcome Message.
Shows a welcome or instructional message when the chat is opened to guide the user.
User Inputs Sitemap URL.
User submits the sitemap.xml URL through the chat interface.
Retrieve URLs and Content.
Fetches all URLs from the provided sitemap and retrieves their web content.
AI Agent Converts Content.
An AI Agent analyzes the retrieved content, determines importance, and structures it into llms.txt format.
Output Generated llms.txt.
Displays the generated llms.txt file to the user for download or further use.

Prompts used in this flow

AI Agent

AI agent that formats sitemap.xml into llms.txt using a detailed prompt and example, given webpage content. Custom prompt provided in 'goal' field.

You are an AI assistant that formats sitemap.xml into llms.txt .
Below is the full text content of a webpage, including the page title and body.
Your task:
- Identify the title (either from the first line or a heading).
- convert the content of the xml sitemap into an llms.txt SIMILAR TO THE EXAMPLE GIVEN BELOW.
-YOU CAN DECIDE WHICH PAGES ARE MORE IMPORTANT AND PUT THEM AT THE START AND OTHER THINGS LIKE CATEGORIES AND TAGS CAN COME AND THE VERY END.
this is an example from langfuse:

**example of llms.txt**

**YOUR OUTPUT SHOULD LOOK AND HAVE A VERY SIMILAR FORMAT TO SOMETHING LIKE THE EXAMPLE ABOVE.**

Components used in this flow

Below is a complete list of all components used in this flow to achieve its functionality. Components are the building blocks of every AI Flow. They allow you to create complex interactions and automate tasks by connecting various functionalities. Each component serves a specific purpose, such as handling user input, processing data, or integrating with external services.

Chat Opened Trigger

The Chat Opened Trigger component detects when a chat session starts, enabling workflows to respond instantly as soon as a user opens the chat. It initiates flows with the initial chat message, making it essential for building responsive, interactive chatbots.

See Chat Opened Trigger

Message Widget

The Message Widget component displays custom messages within your workflow. Ideal for welcoming users, providing instructions, or showing any important information, it supports Markdown formatting and can be set to appear only once per session.

See Message Widget

Chat Output

Discover the Chat Output component in FlowHunt—finalize chatbot responses with flexible, multi-part outputs. Essential for seamless flow completion and creating advanced, interactive AI chatbots.

See Chat Output

ChatInput

The Chat Input component in FlowHunt initiates user interactions by capturing messages from the Playground. It serves as the starting point for flows, enabling the workflow to process both text and file-based inputs.

See ChatInput

URL Retriever

Unlock web content in your workflows with the URL Retriever component. Effortlessly extract and process the text and metadata from any list of URLs—including web articles, documents, and more. Supports advanced options like OCR for images, selective metadata extraction, and customizable caching, making it ideal for building knowledge-rich AI flows and automations.

See URL Retriever

AI Agent

The AI Agent component in FlowHunt empowers your workflows with autonomous decision-making and tool-using capabilities. It leverages large language models and connects to various tools to solve tasks, follow goals, and provide intelligent responses. Ideal for building advanced automations and interactive AI solutions.

See AI Agent

GoogleSearch Component

FlowHunt's GoogleSearch component enhances chatbot accuracy using Retrieval-Augmented Generation (RAG) to access up-to-date knowledge from Google. Control results with options like language, country, and query prefixes for precise and relevant outputs.

See GoogleSearch Component

Flow description

Purpose and benefits

This workflow automates the process of converting a website’s sitemap.xml into a structured and AI-friendly llms.txt format. The flow leverages AI agents and retrieval tools to streamline preparing your site’s content for use in large language models (LLMs) and other AI applications. Below is a detailed breakdown of its steps and components.

1. User Experience & Entry Point

  • Welcome Message on Chat Open
    When a user opens the chat, a message widget displays a friendly prompt:

    🗂️ Drop your sitemap.xml URL below!
    I’ll convert it into a clean llms.txt format, perfect for feeding into AI models 🤖📄

    This sets clear expectations and guides the user to submit the correct input.

2. User Input Handling

  • Chat Input Node
    The user pastes the URL of their sitemap.xml file into the chat interface.
  • Triggering the Workflow
    The input URL is automatically passed to the next step for processing.

3. Retrieving and Processing Sitemap URLs

  • URL Retriever (Primary)
    The workflow uses a URL Content Retriever node to:

    • Fetch the content from the provided sitemap.xml URL.
    • Apply extraction strategies (e.g., extracting from headings, paragraphs).
    • Cache results for efficiency.
    • Limit content using token and header strategies (e.g., start from H1, skip last header).
    • Optionally process metadata (such as product data).

  • Advanced Settings

    • Handles up to 300,000 tokens per document.
    • Can include content from various HTML elements (H1–H6, paragraphs).
    • Supports strict whitelisting, caching (2 weeks), and loading strategies.
    • Can integrate OCR for image-based content (disabled by default).

4. Augmenting with External Knowledge

  • Google Search Tool
    The AI agent is equipped with a Google Search tool, allowing it to:

    • Search for additional context or related documentation if needed.
    • Limit results (default: 3), specify country/language.
    • Use cached results for efficiency.

  • Secondary URL Retriever
    An additional retriever node can be configured to fetch content from URLs found via Google Search, further enriching the AI agent’s knowledge base if required.

5. AI Agent for Content Conversion

  • AI Agent Node
    The core of the workflow is an AI agent configured with the following:
    • Goal:
      Transform sitemap.xml content into a well-structured llms.txt according to a provided example.
    • Backstory & Role:
      The agent is positioned as a specialist in generating llms.txt files.
    • Tools:
      Uses both Google Search and URL Retriever tools for access to site and web content.
    • Operation:
      The agent:
      • Reads all fetched page content.
      • Identifies the title and structure.
      • Decides which pages are most important, listing them first.
      • Places less critical pages (e.g., categories, tags) at the end.
      • Follows a strict format to ensure the output matches the llms.txt requirements for LLM ingestion.

6. Output and User Feedback

  • Chat Output Node
    The AI agent’s formatted output (the llms.txt content) is displayed to the user in chat, ready to be used for AI training or ingestion.

Workflow Structure (Process Overview)

StepComponentPurpose
1Chat Opened TriggerShows welcome/instruction message
2Message WidgetGuides user to input sitemap.xml URL
3Chat InputReceives user-submitted sitemap.xml URL
4URL RetrieverFetches and parses URLs/content from sitemap
5Google Search Tool(Optional) Finds additional context for pages
6URL Retriever (Google)(Optional) Fetches content from Google-found URLs
7AI AgentConverts all page data to formatted llms.txt
8Chat OutputPresents formatted llms.txt to user

Why Is This Workflow Useful?

  • Scalability:
    Automates a time-consuming manual process, allowing you to convert any site’s sitemap into a usable format for LLMs without technical expertise.

  • Quality and Consistency:
    Ensures that the output matches a strict format, improving the quality of your AI training data.

  • Extensibility:
    Can be customized to include additional knowledge sources or apply more advanced extraction logic.

  • Efficiency:
    Integrates caching and token limits to handle even large websites quickly and reliably.

  • AI-Driven Decisions:
    The agent can prioritize important pages and structure the output intelligently—something that would be tedious or error-prone to do by hand.

Example Use Cases

  • Preparing your entire website as a single, well-structured document for AI ingestion.
  • Automating content curation for knowledge bases or documentation systems.
  • Scaling up dataset creation for fine-tuning LLMs with domain-specific site content.

This workflow makes it easy, fast, and reliable to convert website sitemaps into AI-optimized text files, saving you hours of manual work and ensuring your AI models get high-quality, structured input.

Learn more

Sitemap to LLM.txt AI Converter
Sitemap to LLM.txt AI Converter

Sitemap to LLM.txt AI Converter

Transform your website's sitemap.xml into LLM-friendly documentation format automatically. This AI-powered converter extracts, processes, and structures your we...

2 min read
AI Documentation +4
llms.txt
llms.txt

llms.txt

The llms.txt file is a standardized Markdown file designed to optimize how Large Language Models (LLMs) access and process website content. Hosted at a website'...

7 min read
AI LLMs +3