Crawle
Start monitoring

Documentation

Crawl settings

Control rendering, crawl policy, URL budgets, screenshots, and crawl behavior per site.

Crawl settings

Crawl settings

Control how Crawle discovers and re-crawls northwind.com

Crawling active

Crawl settings

Control crawl cadence and how much of the site to discover.

Crawling enabled

Pause or resume scheduled crawls for this site.

Paused
Crawl delay

Minimum time between requests (1–60s).

1s
Faster (1s)Gentler (60s)
URL budget

Maximum URLs to crawl for this site.

50,000
Crawled vs budget0 / 50,000

Fetch & render

How pages are downloaded and rendered during a crawl.

JavaScript rendering

How to handle JavaScript-rendered content.

Page screenshots

Capture visual thumbnails of each page via a headless browser.

Ignore cache headers

Always perform full downloads (no 304 Not Modified shortcuts).

Custom user agent

Override the default crawler user agent (empty for default).

Mozilla/5.0 (compatible; CrawleBot/1.0)

Crawl region

Where crawl requests originate from — affects geo-targeted pages.

Site-level settings control crawl budget, rendering behavior, screenshots, cache handling, user agent, and crawl region.

Rendering

Rendering is configured per site so user choices are respected instead of silently changed.

  • Auto mode lets Crawle choose rendering when useful.
  • JavaScript rendering is useful for dynamic sites and SPAs.
  • Screenshots and lab performance checks can run on a lower cadence than crawl checks.

Budgets and suppression

URL budgets and suppression rules help keep crawls focused on useful URLs.

  • Suppression should be used for known traps and low-value parameter URLs.
  • Crawle can surface parameter trap suggestions for review.
  • Suppressed URLs stay out of crawl pressure and can be purged when appropriate.

Related features

Full-site cloud crawling without local machines

Explore

Link monitoring for broken links and redirect chains

Explore

SEO alerts that surface real problems, not noise

Explore

Continuous monitoring

Put these docs to work on a real client site.

Set up continuous monitoring across your client portfolio.

Start monitoring