URL Retriever

The URL Retriever is a versatile flow component designed to fetch and process web content from specified URLs, returning the information as structured documents. It serves as a bridge between external online content and your AI workflow, enabling you to integrate, analyze, or process web-based information efficiently.

What Does It Do?

This component retrieves the content of one or multiple URLs provided as input. It can extract the main text, metadata, and even process content from images using Optical Character Recognition (OCR). The retrieved data is then made available in various structured formats suitable for downstream AI tasks such as summarization, question answering, or knowledge extraction.

Input Options

You can supply URLs to the component in two ways:

• Text URLs:

  • Input Type: Message
  • Description: A list of plain URL links for the component to fetch content from.

• URL Records:

  • Input Type: UrlRecord
  • Description: A list of structured URL records, which may include additional metadata.

Advanced Input Parameters

Outputs

The URL Retriever provides its outputs in several formats, allowing flexible integration with various AI processes:

Why Use the URL Retriever?

• Integrate External Knowledge: Seamlessly bring web-based information into your AI applications, such as chatbots, search engines, or knowledge bases.
• Customizable Extraction: Fine-tune what content and metadata you want, control the amount of data, and use OCR for images.
• Performance & Efficiency: Use caching to avoid redundant downloads, and limit token output for performance.
• Flexible Output Formats: Choose the output format that best fits your next workflow step—structured document, message, or tool.

Example Use Cases

• Building knowledge-grounded conversational agents that answer questions using up-to-date web content.
• Aggregating product data from e-commerce sites for comparison or analytics.
• Monitoring and analyzing blog or news articles based on specific topics or keywords.
• Extracting information from web pages containing mixed media (text and images).

Summary Table

The URL Retriever is a powerful and flexible bridge between web content and your AI workflows, offering granular control over content extraction and integration.