Crawle
Start monitoring

AI readiness

Know if AI agents can reach, read, and use your site

Collect every agent-relevant crawl signal on one page, score what is measurable from 0 to 100, and get an alert when a deploy or robots.txt change moves it.

Start monitoringRead setup docs

AI readiness

shop.example.com

AI readiness

Ecommerce — detected, high confidence
Last crawl 14m ago
0/ 100

5 of 6 applicable signals measured · computed from the latest crawl and poll data

An agent can reach and read this site; some controls are missing the accessible names agents act on.

Top cause: 214 of 1,986 interactive controls lack an accessible name (−10 pts)

  • Reach

    Can an agent get the bytes — robots.txt access for AI crawlers.

    82/ 100

    Good

    −4 / 22 pts

    Worst finding: OAI-SearchBot blocked in robots.txt — as of last poll.

  • Understand

    Can an agent parse the content — text before JavaScript runs, structured data, headings.

    83/ 100

    Good

    −5 / 30 pts

    Worst finding: 78% of rendered text is present before JavaScript runs.

  • Operate

    Can an agent act — accessible names on interactive controls, layout stability.

    64/ 100

    Mixed

    −10 / 28 pts

    Agent accessibility

    Heuristic name/role/state check on the page as crawled

    −10 / 28 pts

    214 of 1,986 interactive controls lack an accessible name — 11% · 2 of 38 buy/login/search controls unnamed

    Layout stabilityNot measured

    Not measured — no CLS data yet. Drops out of the denominator; never counts as zero.

    no effect on score

  • Transact

    Can an agent buy — product schema completeness plus checkout-scoped controls (ecommerce only).

    92/ 100

    Good

    −1 / 12 pts

    Worst finding: product schema complete on 96% of product-pattern URLs.

Forward-looking — not scoredllms.txtPresentWebMCP toolsNot checkedUCP manifestAbsentZero score weight — presence earns recognition, absence never deducts.
InfoAI readiness dropped on shop.example.com

robots.txt started blocking OAI-SearchBot — detected on the robots.txt poll path.

Cross-references the robots.txt change alert — one root cause, one warning.

The AI readiness page scores what agents can measurably do — reach, read, operate, transact — with deduction math per signal and forward-looking standards kept visibly unscored.

The problem

ChatGPT, Perplexity, Claude, and Google's AI features do not behave like Googlebot: most AI crawlers render no JavaScript, each obeys its own robots.txt rules, and agents operate pages through the accessibility tree. A site can rank fine and still be unreachable or unusable for agents — and nobody notices until traffic shifts.

The outcome

Crawle collects every agent-relevant signal on one page and scores what is measurable: 0-100, renormalized over the signals that apply to the site — an ecommerce site is graded on checkout operability, a blog never is — with an alert when a deploy or robots.txt change moves the number.

  • Per-AI-bot robots.txt access matrix: 17 crawlers grouped by class — search-index, assistant fetch, training, opt-out — with change alerts on the robots.txt poll path.
  • Raw-vs-rendered content share: how much of each page's text exists before JavaScript runs — no major AI crawler except Google's renders JavaScript.
  • Heuristic agent-accessibility census on rendered pages: interactive controls checked for accessible names, roles, and state, including buy, login, and search buttons.
  • Structured-data and heading coverage: JSON-LD presence and type coverage plus single-H1 outline checks.
  • Site-type-aware Transact pillar: product schema completeness and checkout-path controls, scored only for ecommerce sites — the score renormalizes for everyone else.
  • Forward-looking badges for llms.txt, WebMCP, and UCP/AP2: shown with adoption facts, never scored.
  • Not AI-citation or brand-visibility tracking: Crawle measures the crawl side — whether agents can reach, read, and operate the site. Badges carry zero score weight, and blocking training bots is treated as a policy choice, not a failure.

Workflow

From crawl signal to team action

01

Crawl and collect

Every signal comes from data Crawle already gathers: robots.txt polls, raw and rendered HTML, the accessibility census, structured data, and lab CLS on rendered pages.

02

Score what applies

Crawle classifies the site type, drops signals that do not apply or are not yet measured, and renormalizes the rest to 0-100 — the page always shows how many applicable signals were measured.

03

Alert on the drop

Daily snapshots form the baseline. When a deploy or robots.txt change drops the score past the threshold, one incident routes through Slack, Teams, or email — robots-attributed drops cross-reference the robots.txt change alert instead of notifying twice.

Where teams use it

Release QA for agent-facing regressions.
Robots.txt policy review across AI crawlers.
Ecommerce checkout operability checks.
Client AI-readiness reporting.

Related documentation

AI readinessAlerts and digestsAPI and MCP setup

Related features

Keep exploring Crawle

View all features

AI agent workflows

The SEO MCP server for AI agents

Connect Claude, ChatGPT, Codex, Gemini, and internal agents to a live SEO MCP server backed by Crawle's continuous crawl data.

Explore

Alerts and digests

SEO alerts that surface real problems, not noise

Detect critical regressions quickly while collecting lower-priority changes into digest windows your team can actually read.

Explore

Continuous monitoring

Monitoring SEO performance, continuously and in real time

Track SEO performance continuously across client sites with live crawl coverage, change detection, alerts, reports, and exports.

Explore

Continuous monitoring

Put Crawle into your next client QA loop.

Continuous technical SEO monitoring with per-client workspaces, real-time alerts, and white-label reports.

Start monitoring