Crawle docs

Parameter traps and suppressed URLs

Review parameter explosion warnings and suppress low-value URL patterns safely.

Parameter trap manager

Crawl exclusion rules suppress noisy parameter URLs, show matched counts, and keep purge controls explicit.

Parameter traps

Parameter traps happen when a site exposes many low-value URL variants through filters, sorting, tracking, or repeated query parameters.

Crawle should flag parameter variants, not ordinary folder paths.

Review examples before suppressing a rule.

Suppression rules should target the parameter pattern that creates crawl noise.

Suppressed URLs

Suppressed URLs stay visible as a state but are removed from crawl pressure.

Use suppression for URLs that should not be crawled again.

Purge suppressed URLs only when you intentionally want to delete them from the workspace.

Keep important URL patterns unsuppressed even if they are large.

Private beta note

Crawle is currently invite-only. Some features depend on workspace permissions, connected accounts, API quotas, or integration setup by the organization.

Related features

See where this fits

View all docs

Full-site cloud crawling without local machines

Run full client-site crawls in the cloud, monitor discovered URL coverage, render JavaScript when needed, and export crawl datasets.

Link monitoring for broken links and redirect chains

Find broken internal links, links to redirects, insecure links, redirect chains, source pages, and final targets from one link explorer.

Monitoring SEO performance, continuously and in real time

Track SEO performance continuously across client sites with live crawl coverage, change detection, alerts, reports, and exports.