Website crawling

A schedule tells FlowHunt which pages to crawl and how often to crawl them again. Crawled pages are indexed, so a retriever can search them. Open Knowledge sources → Schedules to manage them.

Create a schedule#

1. Add the schedule

   Click Add new Schedule. A dialog opens with a note about credits.

2. Choose the type

   Under Type, choose what to crawl (see the table below). You cannot change the type once the schedule exists.

3. Choose the frequency

   Under Frequency, choose Daily, Weekly, Monthly, Quarterly or Yearly. A new schedule defaults to Daily.

4. Enter the address

   Enter the Domain or URL. For a sitemap crawl the field is Sitemap URL. For a URL list, paste one URL per line or load them from a file.

5. Save

   Click Add new Schedule. The first crawl starts straight away, and later crawls follow the frequency.

Schedule types#

The knowledge base syncs need the matching integration and a Pro plan or higher. Trial workspaces can only create Single URL, Sitemap and URL List schedules; other types are greyed out or refused.

Control what gets indexed#

Open the advanced settings of a schedule to narrow the crawl.

• Index just matching URLs: one string per line. A URL is crawled only if it contains one of them.
• Skip matching URLs: one string per line. A URL that contains one of them is not crawled.
• Follow links (Process also URLs found in page): also crawl links found on each page. You must fill in Index just matching URLs when this is on, otherwise saving fails. Every URL checked counts against your credits, including ones that do not match.
• With Browser (extra credits): use a real browser. Slower and more expensive. Use it only for sites that load their content entirely with JavaScript.
• Take screenshot (extra credits): store a screenshot, useful when pages have no og:image.
• With Proxy Rotation (extra credits): change IP address with each request, for sites whose firewall blocks the crawler.
• Custom Headers: one NAME=value per line, sent with every request. You can allow these headers in your firewall.

Read the schedule list#

Each row shows the domain, frequency, last update, a status and counts of URLs Scheduled and Failed URLs. Open a schedule to see its URLs. From there you can open a URL’s details or remove it with Delete URL. In the confirmation, tick Skip this URL in future schedule runs to keep it from being crawled again.

Row buttons let you Run Schedule, Cancel running schedule and Show Error logs. Select several rows and use Run Selected to run them together.

Credits#

Credits are consumed when content is crawled. If a scheduled crawl finds no changes, half of the credits are consumed.

Next steps#

Add exact answers for the questions you care about in Questions and answers, or read Integration disconnected if a knowledge base sync stops.