Common Crawl in AI readiness

What Common Crawl is

Common Crawl is an open web archive that many AI models are trained on. Pages in a Common Crawl collection are available to anyone training on it and can shape what AI models learn about your site and brand. A page missing from the collection shown here is not available through it, but models can also learn from other sources.

• Being in the archive does not prove that any model used your pages, and it is not a ranking signal.
• The Common Crawl view is shown for context. It is not included in your score.
• Allowing or blocking CCBot in robots.txt is your decision, so it never changes your score.

How Crawle reads it

Crawle reads Common Crawl's public index for your site's host in the newest completed collection, and compares the pages it finds there with your site's own page list.

• Only captures that returned status 200 as HTML are counted.
• From your page list, Crawle compares the pages whose latest status is 200 or not known yet, since other pages are not expected in the archive.
• URLs are matched after a light normalization: lower-case scheme and host, no default port, an empty path read as /, and no fragment. Nothing else is merged, so the same path with a different query string is a different URL.
• A read that is interrupted or capped is stored as incomplete, and it never turns one of your pages into missing.
• Pages found per collection shows the most recent collections Crawle has read. A collection is a dated Common Crawl index, not a score or ranking.
• Outside inventory counts Common Crawl URLs for your site that are not in Crawle's page list.

What a result means

Each result says what Crawle found, what that means, and what to do. None of them changes your score.

• Present

  What Crawle found

  Crawle read the whole Common Crawl index for this archive and found pages from your site in it. Those pages can be part of the data AI models learn from; that says nothing about how they are used.

  What to do

  Nothing is required.

• Partially read

  What Crawle found

  Crawle has read only part of the archive index so far. The pages found are in it; pages not seen yet are not counted as missing, so the share is a lower bound.

  What to do

  Nothing. Crawle keeps reading and updates the result.

• Not found

  What Crawle found

  Crawle read the whole index for this archive and found none of your pages. That is common for new or small sites and says nothing about your search visibility.

  What to do

  Nothing is required.

• Not measured

  What Crawle found

  Crawle has not read the index for this site yet, Common Crawl limited the requests or was unavailable, or Crawle has not collected your page list yet.

  What to do

  Nothing. Crawle tries again in an upcoming cycle, or compares once it has discovered your pages.

• In the URL list, In Common Crawl, Missing and Not seen yet filter your pages by what the comparison found.

Example: what the collection shows

For shop.example.com the latest Common Crawl collection shows most pages as Present and a few as Not found. Common Crawl is not part of the score, so neither result changes it.