Retrievers
URL Discovery Crawler
Recursively discover all reachable URLs from a seed URL up to a given depth. Returns a deduplicated flat list of discovered URLs.
Category: Retrievers · Plans: Trial, Starter, Pro, Premium, Enterprise · Type: URLDiscoveryCrawler
Inputs#
| Field | Type | Required | Default | What it does |
|---|---|---|---|---|
| Seed URL | Text | No | {input} | The starting URL to crawl. All reachable links will be discovered from here. |
| Crawl Depth | Number | No | 1 | How many levels deep to follow links. Depth 1 fetches only the seed page and returns its links. Depth 2 also fetches those links, and so on. |
| Same Domain Only | Yes/No | No | Off | When enabled, only URLs on the same domain as the seed URL are followed and included in the output. |
| Parallelism | Number | No | 2 | Maximum number of URLs to fetch concurrently. Higher values speed up crawling but increase load on the target server. |
| Max URLs | Number | No | 1000 | Maximum number of URLs to collect before halting discovery. Prevents unbounded memory usage on large or infinite-URL sites. |
| From H1 if exists | Yes/No | No | Off | Start link extraction from the H1 tag of the document if H1 exists. Anchors before the first H1 are skipped; falls back to the full page when no H1 is present. |
| Skip Last Header | Yes/No | No | Off | Skip the last header (H1-H6) and any anchors that follow it. Falls back to the full page when no header is present. |
Outputs#
| Output | Type |
|---|---|
| Urls | Any |
| Url Count | Any |
Use as an agent tool#
Tool name: urldiscoverycrawler
Recursively discover all reachable URLs from a seed URL up to a given depth. Returns a deduplicated flat list of discovered URLs.