Retrievers
URL Retriever
Get the content of URLs as Documents.
Category: Retrievers · Plans: Trial, Starter, Pro, Premium, Enterprise · Type: URLContentRetriever
Inputs#
| Field | Type | Required | Default | What it does |
|---|---|---|---|---|
| URLs | List of text | No | {input} | List of URLs to retrieve content from. Each can be a direct URL or variable reference. |
| From H1 if exists | Yes/No | No | On | Start from the H1 tag of the document if H1 exists. |
| Skip Last Header | Yes/No | No | Off | Skip the last header and its content in the document. |
| Include Metadata | Text | No | — | Include metadata in the output if available. Options: Author, Product, BreadcrumbList, VideoObject, BlogPosting, FAQPage, WebSite, opengraph |
| Export Content | Text | No | — | Content types to include in the output. Options: H1, H2, H3, H4, H5, H6, Paragraph |
| Load from pointer | Yes/No | No | On | Load from the pointer best matching the input query in the document. |
| Max Tokens | Number | No | 3000 | The maximum number of tokens in the output text. |
| Strategy | Text | No | Include equal size from each documents | The strategy to use for transforming the knowledge document to text. Options: Concat documents, fill from first up to tokens limit, Include equal size from each documents |
| Cache TTL | Text | No | 2 weeks | Cache the content of the URLs for defined amount of time. Options: No cache, 5 minutes, 30 minutes, 1 hour, 12 hours, 1 day, 1 week, 2 weeks, 1 month, 3 months, 6 months, 1 year |
| Hide Resources | Yes/No | No | Off | Hide resources that were retrieved by the retriever. |
| Apply OCR | Yes/No | No | Off | Apply OCR to the document. Useful for documents containing images. |
| Page Offset | Number | No | 0 | Skip the first N pages (PDF/DOCX/PPTX) before extracting content. Use with max_tokens to paginate through large documents. Has no effect on HTML or plain-text URLs. |
| Raw Output | Yes/No | No | Off | Return raw documents instead of text. (Advanced) |
| Raw HTML | Yes/No | No | Off | Return the raw HTML of each URL instead of parsed text. Overrides Raw Output and other extraction options. |
| Allowed Domains | Text | No | (empty) | Newline-separated list of allowed domains. Use wildcards (*) for subdomains. |
| Allowed URL Paths | Text | No | (empty) | Newline-separated list of allowed URL paths. Use wildcards (*) for patterns. |
| Strict Whitelist Mode | Yes/No | No | On | When enabled, only URLs matching BOTH domain AND path restrictions are allowed. |
Outputs#
| Output | Type |
|---|---|
| Documents | Any |
| Document Count | Any |
Use as an agent tool#
Tool name: url_retriever
Fetch and scrape webpage content from one or more URLs. Sends HTTP requests to the given URLs, extracts the main page text (or raw HTML), and returns the result as documents. Use this for web scraping, crawling sites, reading article content, or retrieving text from any webpage. Pass urls as an array of URL strings.