Crawle docs

Add and verify a site

Verify ownership with Google Search Console, DNS, or meta verification before monitoring starts.

Add and verify site

The site wizard verifies ownership, applies client and tag context, and allocates URL budget before the first crawl starts.

Verification options

Crawle uses the strongest available verification path first and falls back when needed.

Google Search Console is preferred when the user has access to the property.

DNS TXT verification works when GSC access is unavailable.

Meta verification can be used when DNS access is slower to coordinate.

After verification

Verified sites become eligible for crawling, alerts, reports, exports, and integrations.

Initial crawls prioritize discovered URLs that have not been crawled yet.

Crawl settings remain site-specific.

Client groups and tags can be assigned from the sites list.

Private beta note

Crawle is currently invite-only. Some features depend on workspace permissions, connected accounts, API quotas, or integration setup by the organization.

Related features

See where this fits

View all docs

Full-site cloud crawling without local machines

Run full client-site crawls in the cloud, monitor discovered URL coverage, render JavaScript when needed, and export crawl datasets.

Monitoring SEO performance, continuously and in real time

Track SEO performance continuously across client sites with live crawl coverage, change detection, alerts, reports, and exports.

Google Search Console and field CWV enrichment

Combine Crawle's crawl data with Google Search Console, URL Inspection, and CrUX field data when the workspace has access.