What is the Schedules feature?

Schedules allow you to power your Chatbots and Flows with your knowledge sources by periodically crawling websites or single URLs.

How frequent should the crawling be?

It depends on your use case and needs. However, crawling bigger (100+) domains too often can get expensive quickly. We recommend only using daily schedules for single URLs that you know change frequently. You can also run a schedule manually whenever you need to.

How do I check if the crawling was successful?

The status and number of crawled URLs in the list inform you of any possible errors. To see the exact URLs crawled, click Show Found URLs. You can go even deeper by clicking Show URL Tree in the detail of a URL. This will reveal the page structure and all the content.

How do I make the Flow use information from my Schedules?

Use the Document Retriever component to allow the Flow to look up information in crawled URLs.

Schedules

Automate data retrieval for your chatbots and flows by scheduling periodic crawls of websites and YouTube channels with FlowHunt’s Schedules feature.

AI Schedules Chatbots Knowledge Sources

Try it Now Schedule a Demo

How to add a Schedule

Navigate to the Schedules screen and click Add Schedule:

A pop-up will appear:

Pick the type of crawl:
- Domain crawl – Goes through your entire domain, learning from every page.
- Single URL crawl – Good for pages that need daily crawling or for adding external sources in a controlled way.
- Sitemap crawl – Learns information about your pages, linked media, and files to understand your site better and crawl it more efficiently.
- YouTube channel – Index the closed captions from your YouTube videos.
FlowHunt uses video captions to index the content of YouTube videos. However, YouTube’s auto-generated captions aren’t always accurate. If you want to ensure your Chatbot has the right information, we suggest you take the time to create the captions yourself, or at least check the auto-generated ones.
YouTube’s auto-generated captions aren’t always accurate. To ensure your Chatbot has the right information, check the auto-generated captions or take the time to create the captions yourself.
Pick the frequency – how often is the schedule repeated:
- Daily
- Weekly
- Monthly
- Yearly
Remember that crawling costs credits, and larger schedules can get quite pricey. When choosing the frequency, consider how often you update the data and how important it is for the flow to have the latest information. You probably won’t need your whole domain crawled daily, but you might benefit from daily crawls of frequently updated URLs.
Input the URL. Based on the type of crawl, you will want to use these URL structures:
- Domain crawl: https://www.example.com
- Single URL crawl: https://www.example.com/blog/article1
- Sitemap crawl: https://www.example.com/page-sitemap.xml
Click Add New Schedule.

Managing Schedules

Your schedule has been created and will show as pending. To see the progress, refresh the page:

The crawling shouldn’t take more than a couple of minutes. In this example, we’re doing a single URL crawl, which takes less than five seconds. If your schedule runs into an error, just hover over the red ‘Error’ status tag to learn more.

The Schedule can always be edited or deleted. Once finished, you don’t need to wait for the next scheduled time. If you need to, you can re-run the schedule manually by clicking the repeat icon next to the status tag:

You can easily check the crawled URLs, ensuring all the necessary information is gathered. To do so, click Show found URLs. This will show you a list of all the URLs crawled within the schedule. To see more about a specific URL, click Details:

To go even deeper, click Show URL Tree in the detail of a URL. This will allow you to see the page structure by headlines. Clicking one will reveal the content:

Frequently asked questions

: Schedules allow you to power your Chatbots and Flows with your knowledge sources by periodically crawling websites or single URLs.
: It depends on your use case and needs. However, crawling bigger (100+) domains too often can get expensive quickly. We recommend only using daily schedules for single URLs that you know change frequently. You can also run a schedule manually whenever you need to.
: The status and number of crawled URLs in the list inform you of any possible errors. To see the exact URLs crawled, click Show Found URLs. You can go even deeper by clicking Show URL Tree in the detail of a URL. This will reveal the page structure and all the content.
: Use the Document Retriever component to allow the Flow to look up information in crawled URLs.

Start Automating Knowledge Updates

Discover how FlowHunt's Schedules feature can keep your AI chatbots and flows smart, relevant, and always up-to-date.

Try it Now Schedule a Demo

Learn more

Scheduling Automated Website Crawls

Learn how to set up automated schedules for crawling websites, sitemaps, domains, and YouTube channels to keep your AI Agent knowledge base up-to-date.

Sep 11, 2025 6 min read

Schedules Crawling +3

Reduce AI Hallucinations By Adding Custom Knowledgebases

Reduce AI hallucinations and ensure accurate chatbot responses by using FlowHunt's Schedule feature. Discover the benefits, practical use cases, and step-by-ste...

May 30, 2025 8 min read

AI Chatbot +4