Crawle
Log inStart monitoring
Crawle24/7 SEO crawling

Catch SEO issues before Google does.

Google, Bing and AI bots never stop crawling your clients' sites. Neither does Crawle. It crawls 24/7, compares every page with its last crawl and alerts your team the moment something breaks.

Start monitoringChoose a workspace size

What Crawle does

Alerts land where your team already works.

Critical incidents reach Crawle's Slack and Microsoft Teams apps, email or your webhooks the moment a crawl finds them. In Slack, acknowledge or resolve from the message.

Email · Daily digests

Illustrative example

Every change. Two years of history.

Crawle logs what changed on each page: titles, canonicals, status codes, redirects and indexability. Every paid workspace keeps two years of history.

On-demand recrawl

Illustrative example

Full exports. Anytime.

Download URLs, issues, broken links, changes and technology risks as CSV, JSON or ZIP. Large exports run in the background while you keep working.

7, 30 and 90-day ranges · Custom range

Illustrative example

Can AI bots get in?

One score for AI crawlers, one for AI agents: can they reach your pages and get through your key flows? Crawle lists what to fix first and alerts you when a score drops.

Bot access · Indexability · Structured data · Agent access · Score history

Illustrative example

Find the issue. Confirm the fix.

Issues are grouped by type, with every affected URL. Ship a fix and hit Validate Fix: Crawle recrawls the pages and marks each one Fix Verified or Issue Persists.

Health score · Validation progress · robots.txt · Sitemaps

Illustrative example

Let your AI agents work in Crawle.

Connect Claude, ChatGPT and Gemini over OAuth MCP to read and act on live crawl data, with scoped read and write permissions limited by the active workspace role. API keys remain read-only.

Connected apps

Illustrative example

Ship the migration. Crawle watches the redirects.

Crawle alerts you when pages start redirecting or redirect to a broken page, and shows each redirect chain and its linking pages. Move detection is on request via support.

Pages started redirecting · Redirect to broken page · Redirect loop

Illustrative example

Know the stack. See when it changes.

Crawle detects frameworks, CMS and CDN passively from normal crawl evidence, alerts you when the stack changes and flags risk signals with the evidence behind them.

Detected components · Recent stack changes · Risk findings

Illustrative example

Run every client from one account.

Organize clients with workspaces, client groups, tags and four team roles. Invite a client as a site-only viewer: they see their site, not the rest of the workspace.

Weekly reports · Segments

Illustrative example

Always on

What 24/7 crawling means

A weekly audit looks once a week. Crawle never stops looking: every page is recrawled on its own schedule, key pages most often, and every crawl is compared with the last.

Illustrative example

  1. Tuesday morning

    A release goes live

    A template change quietly adds noindex to every product page.

  2. Next crawl

    Crawle recrawls the product pages

    Each page is compared with its last crawl. The noindex is new.

  3. Same crawl

    A critical incident opens

    Your team gets it in Slack, Teams, or email, not in a digest.

  4. Tuesday afternoon

    Fixed the same day

    The template is reverted. Crawle sees the fix and closes the incident.

With a weekly audit, the same noindex waits for next Monday's run. Google may get there first.

Workspace overview

Dashboards built for the work, not vanity charts

A live view across client sites: URL capacity, crawl progress, active incidents, link issues, and the technical SEO signals that actually need triage.

Workspace overview

Overview

Workspace health across 5 sites

Last crawl 14m ago
Critical issues
12
−21%
Pages crawled
17,452
+6.4%
Broken links
106
−8%
Index coverage
85.4 %
+3.1%

Sites

5 sites in this workspace

  • crawle-demo.io86B60417,428
  • docs.crawle-demo.io94A9312
  • shop.crawle-demo.io72C1484,980
  • blog.crawle-demo.io58D831,264
  • staging.crawle-demo.iounverified———

Crawl activity

URLs crawled and errors over the last 30 days

  • Crawled
  • Errors

Site health

Weighted technical score

Open issues by type

  • Links To Redirect
  • Missing Meta Description
  • Broken Links
  • Canonical Changed
  • Title Removed
A workspace overview with sample data: client sites, URL capacity, crawl progress, and active technical SEO signals.

Cloud crawler

Full-site crawling, no local machine

Crawle runs in the cloud 24/7, so nothing waits on a laptop: shared crawl data, rendered-DOM support, exports, and workspace capacity sized around the client.

Open crawler feature

URL coverage summary

Coverage

How much of the workspace Crawle has discovered and crawled

Crawling · 2m ago
Discovered URLs
0+1,204 today
Crawled
0
20% crawl coverage
In queue
0
awaiting crawl
Crawl coverage
0%
of 15.8k discovered

Index coverage

Crawled · indexed · in sitemap

  • Crawled
  • Indexed
  • In sitemap
Coverage cards show how much of the workspace Crawle has discovered, crawled, queued, and flagged.

Real-time alerts

Catch the regression while the team still remembers the release

Critical incidents fire in real time. Warnings, informational changes, and resolved incidents can roll into digests for Slack, Teams, email, and webhooks.

See alert routing

Slack notification routing

Notification channels

Route alerts to the right Slack channels

Workspace connected

Slack

Active4 channels

Connected to Acme Workspace

Notification channel

alerts
  • growth
  • migrations-qa
  • seo-alerts
    Default
  • site-health

Currently sending to #seo-alerts

Send test
Alerts routed (30d)0
Channels wired0
Alert volume (12d)
Slack routing lets teams choose channels for critical incidents, digests, migration QA, and test notifications.

AI agents

Bring live crawl context into Claude, ChatGPT and Gemini

Crawle exposes workspace-aware crawl data over OAuth MCP with scoped read and write permissions for approved agents. Access is limited by the granted scopes and active workspace role. API keys remain read-only.

Explore AI workflows
Claude MCP issue triage
Why is /pricing missing from Google?
Calling Crawle MCPMCPhttps://api.crawle.io/mcp
issues:readlinks:readsites:readread-only · scoped

Critical
Noindex on indexable page
X-Robots-Tag: noindex · app.crawle.io
High impact
Affected URLs7
Internal inlinks42
Trend (7d)+12%
7-day
Top affected pages
  • /pricing0.98
  • /pricing/teams0.81
  • /pricing/enterprise0.74
Recommended next actions
  • Remove the X-Robots-Tag: noindex header from /pricing
  • Re-crawl the 7 affected URLs to confirm the fix
  • Request indexing in Search Console once headers are clean
Claude triaging a noindex issue through Crawle's MCP connector, recreated with sample data.

Everything after the crawl starts

The workflows agencies actually need

Crawle is not just another crawl snapshot. It is shared monitoring, history, routing, exports, and agent context that stays alive after the audit.

  • Client workspaces

    One operating context per client: capacity, team access, integrations, reports, and history kept separate.

  • 24/7 crawling

    Every URL runs on its own recrawl schedule, prioritizing key pages. Each crawl checks for changes in status, metadata, links, robots, and canonicals.

  • 2-year change history

    Timelines prove when a change happened and what it affected. Crawle retains 2 years of change history; change exports cover the most recent 30 days.

  • Migration monitoring

    Detect mass redirects, verify target domains, export mappings, and clean up old URLs when the migration is stable.

  • Link monitoring

    Broken links, redirect targets, source pages, and link exports stay grouped for action.

  • AI agent workflows

    OAuth MCP gives Claude, ChatGPT, Gemini, and internal agents live Crawle context with scoped read and write permissions, limited by the active workspace role. API keys remain read-only.

  • White label

    Brand reports and client-facing surfaces when the agency needs Crawle to sit behind its own service model.

  • Exports and automation

    Download current or historical datasets, send weekly summaries, or trigger customer-owned workflows with webhooks.

How it works

From client workspace to always-on monitoring

01

Create a client workspace

Keep each portfolio in its own context with capacity, team access, integrations, alerts, and reports.

02

Add and verify domains

Verify with Google Search Console when available, or use DNS/meta verification when needed.

03

Crawl 24/7

Crawle checks every crawl against the last: status, metadata, links, robots, canonicals, rendering, indexability, and custom extraction values.

04

Route the right alerts

Critical incidents can interrupt immediately; lower-priority changes roll into digests for email, Slack, Teams, and webhooks.

Agency operating model

One workspace per client when separation matters

Use dedicated workspaces for clients that need their own capacity, permissions, alerts, integrations, exports, and reports. Group smaller sites only when that is the cleaner account structure.

Explore workspaces

Workspace clients and tags

Clients & tags

Organize agency domains and apply crawl defaults in bulk

Bulk crawl defaults

Save crawl defaults for a client, preview matching sites, then apply.

Target type
Client
Client
Northwind Retail
JavaScript rendering
No change
Screenshots
Enabled
Cache headers
Respect
Crawl delay0.5s
0.5s (fast)30s (gentle)
Preview sitesSave & applyApplied to 14 sites

14 matching sites for Northwind Retail

  • northwind.com
  • shop.northwind.com
  • help.northwind.com
  • northwind.co.uk
  • northwind.de

Clients

Reuse clients across workspaces; defaults apply locally per assigned domain.

ClientSitesWorkspacesCrawl defaults

Northwind Retail

E-commerce storefront + regional microsites

142 shared
JS: OnScreenshots1.5s delay

Harborlight Media

Publishing network, high page volume

91
JS: OffNo cache4s delay

Vantage SaaS

Product, docs, and marketing domains

61
JS: On2s delay

Meridian Health

Regulated content, gentle crawl

41None set

Tags

Reuse tags across workspaces for priority, market, or migration wave.

TagSitesWorkspaces
Priority113 shared
Migration wave72 shared
JS-heavy51
Client and tag defaults let teams organize domains and apply crawl settings across matching sites.

Technology context

Give AI agents the site's implementation context

Technology stack evidence, security-header posture, stack changes, and risk signals give teams and AI workflows the context needed before suggesting technical SEO fixes.

Open technology context

Stack changes and risk findings

Technology & risk signals

Evidence-backed posture signals and stack deltas for crawle-demo.io

Jun 9
High-risk open0critical + high
Open findings0of 4 total
Detected components0across the stack
Recent stack changes0since last baseline

Risk findings

High-severity findings require deterministic public evidence; informational rows are context for developers and agents.

4open
Actions
Critical
No Content-Security-Policy header

Responses ship without a Content-Security-Policy. A baseline policy reduces the blast radius of injected or third-party script.

CRAWLE-CSP-001
Site posture
OpenJun 9
High
jQuery exposes known prototype-pollution advisory

The detected jQuery version matches a published advisory. Upgrading clears the exposed surface on rendered pages.

GHSA-gxr4-xjj5-5px2CVE-2020-11023
jQuery
v3.4.1
OpenJun 9
Warning
Missing Strict-Transport-Security header

HSTS is not set, so browsers will not pin HTTPS for return visits. Add a long max-age once HTTPS is fully enforced.

Site posture
OpenJun 9
Warning
Server version disclosed in response headers

The Server header advertises an exact nginx build. Suppressing the version removes a low-effort fingerprinting signal.

nginx
v1.21.6
OpenJun 9

Recent stack changes

Compared against the previous completed scan. The first scan becomes the baseline.

Version changed
Next.js14.1.0 → 14.2.3Framework · npm
meta_generatorJun 8, 3:20 PM
Added
Plausible AnalyticsAnalytics & tags
script_srcJun 7, 10:02 AM
Removed
LodashJavaScript library · npm
script_srcJun 5, 6:47 PM
1

Baseline changes The first scan becomes the baseline; later scans can show added, removed, or changed technologies.

2

Risk language Findings are framed as posture signals with evidence, not as proof from an intrusive security test.

Stack changes and risk findings separate baseline changes from evidence-backed hardening signals that developers and AI agents can act on.

Free account · paid workspaces

Choose capacity around the client, not your backend

Most agencies use one workspace per larger client. The workspace size defines URL capacity; prices exclude VAT.

Workspace capacity planner

Are you planning Crawle for one company or many clients?

Crawle accounts are free. Costs start when you add active workspaces. Agencies usually use one workspace per client so access, alerts, reports, and historical exports stay cleanly separated.

Crawl rate and capacity

Suggested setup

10 x Small workspace

Estimated from 10,000 URLs and 2 domains per client.

€390

/mo, excluding VAT

Workspaces

10

URLs per workspace

10,000

Capacity

10,000 URLs / workspace

White label

Recommended add-on

This assumes clients are roughly similar. If one client is much larger, size that workspace separately and keep smaller clients on smaller workspaces.

Suggested fit

Small workspace

€39/mo

€328/year with annual billing

10,000URLs
3domains
1 team member
  • Alerts for critical, warning, and informational changes
  • Integrations and read-only API access
  • CSV, JSON, and ZIP exports
  • Weekly reports
  • JavaScript rendering when required
  • Rendered-page screenshots
Start monitoring
Most popular

Medium workspace

€79/mo

€664/year with annual billing

50,000URLs
10domains
5 team members
  • Alerts for critical, warning, and informational changes
  • Integrations and read-only API access
  • CSV, JSON, and ZIP exports
  • Weekly reports
  • JavaScript rendering when required
  • Rendered-page screenshots
Start monitoring

Large workspace

€149/mo

€1,252/year with annual billing

150,000URLs
25domains
10 team members
  • Alerts for critical, warning, and informational changes
  • Integrations and read-only API access
  • CSV, JSON, and ZIP exports
  • Weekly reports
  • JavaScript rendering when required
  • Rendered-page screenshots
Start monitoring

X-Large workspace

€299/mo

€2,512/year with annual billing

500,000URLs
100domains
25 team members
  • Alerts for critical, warning, and informational changes
  • Integrations and read-only API access
  • CSV, JSON, and ZIP exports
  • Weekly reports
  • JavaScript rendering when required
  • Rendered-page screenshots
Start monitoring

XX-Large workspace

€549/mo

€4,612/year with annual billing

1,000,000URLs
200domains
50 team members
  • Alerts for critical, warning, and informational changes
  • Integrations and read-only API access
  • CSV, JSON, and ZIP exports
  • Weekly reports
  • JavaScript rendering when required
  • Rendered-page screenshots
Start monitoring

Agency white label

Add-onRecommended

Workspace costs separate

  • Custom domain
  • Own logo and branded emails
  • Applies across purchased workspaces
€99/mo
Add white label
Every paid workspace retains 2 years of change history: URL state, detected changes, issue lifecycle, and GSC signals where connected. Change exports cover the most recent 30 days.

Continuous monitoring

Put Crawle into your next client QA loop.

Continuous technical SEO monitoring with per-client workspaces, field-level change alerts, exports, reports, and OAuth MCP access.

Start monitoring