Allow Crawle's test server

Why the test needs it

Each day Crawle's AI access test fetches one page of your site, the key page you chose or else the start page, once with a normal browser user agent and once with each AI bot's user agent. A stopped bot request only counts as a user-agent block when the browser request gets through. When both are stopped, your CDN or firewall is blocking Crawle's test server itself, and the result is inconclusive.

• The test never claims that the real bot is blocked: its requests only carry the bot's user agent.
• robots.txt is checked for each user agent before every request.
• Allow Crawle's test server only for IP reputation and bot challenges, not for your own user-agent rules. A broad allow-all for the IP stops the test from seeing user-agent blocks; such results are marked as tested from an allowlisted IP.

What to allow

In the site's AI readiness, open Allow Crawle's test server. It shows the IPv4 and IPv6 addresses the test uses, ready to copy, and the steps for your provider with those addresses filled in. The steps below are the same, with the addresses written as the test addresses.

• If Crawle marks the addresses as shared, other Crawle customers' tests and crawls use the same addresses. Allow only the addresses and the optional test header Crawle shows.
• Optional: you can also match the X-Crawle-Test request header, which carries this site's test token, together with the IP address. The header is a label, not authentication: anyone who learns the token can send it, so do not publish it and do not trust the header alone.
• Keep your user-agent rules active. An allow-all for this IP hides user-agent based blocks, and results from it are marked as tested from an allowlisted IP.
• When the change is in place, choose Run the test again and read the result. The retest stays within the site's daily test budget.
• Providers change their controls and plans. Each step names the provider page it comes from and the day Crawle read it.

Cloudflare

Cloudflare's controls differ by plan, so Crawle shows separate steps for Free with Bot Fight Mode, Pro and up with Super Bot Fight Mode, and Enterprise with Bot Management.

• Free / Bot Fight Mode: Cloudflare documents that Bot Fight Mode cannot be bypassed or skipped with WAF custom rules or Page Rules, so a WAF custom rule for the test addresses does not help. You can turn Bot Fight Mode off (Security settings, Bot traffic) or leave it on and accept that the test stays inconclusive on pages it challenges.
• Free / Bot Fight Mode, not recommended: an IP Access rule for the test addresses stops Bot Fight Mode from triggering, but Cloudflare documents that an allowed IP also bypasses your custom rules, so your own user-agent rules would no longer apply to it.
• Pro+ / Super Bot Fight Mode: create a WAF custom rule that matches the IP source address of the test addresses and your site's hostname, with the action Skip. Tick 'All Super Bot Fight Mode rules'. If Browser Integrity Check challenges the test, also tick it under the products to skip. Super Bot Fight Mode runs after custom rules, which is why a custom rule can skip it.
• Super Bot Fight Mode is included in Pro, Business and Enterprise. Enterprise without the Bot Management add-on has the Business version, so the Pro and Business step applies to it.
• Do not tick 'All remaining custom rules' or 'User Agent Blocking' for this rule: they would also skip your own user-agent rules.
• Enterprise / Bot Management: Cloudflare shows that Enterprise Bot Management custom rules can combine an IP source, hostname and cf.bot_management.score. In the specific bot-score rule that blocks or challenges the test, exclude requests that match the test addresses and your site's hostname. Do not add a broad rule that skips all remaining custom rules; keep separate user-agent blocking rules active.
• If a built-in Bot traffic setting, rather than a custom rule, blocks the request and cannot be scoped to exclude the test addresses, use Cloudflare's Bot Management guidance or support. Do not rely on a WAF Skip rule to bypass Bot Management.
• Sources: Cloudflare's developer documentation on Bot Fight Mode, IP Access rules, custom rule Skip options, Super Bot Fight Mode, custom rules for customers and partners, and feature interoperability, read 2026-10-13 and 2026-10-14.

Akamai

Akamai products differ and its documentation describes separate controls. Use the one that matches what stops the request. If you are not sure which control fires, ask your Akamai administrator or Akamai support before you change anything.

• IP, geo and ASN blocks: put the test addresses in a Client List used only for the Crawle address and add that list to Block exceptions in the security policy that protects your site. Block exceptions override IP, geo and ASN blocking only, not other protections such as DoS protection and the Web Application Firewall.
• Web Application Firewall: Akamai documents a bypass network list whose networks are allowed to bypass the Web Application Firewall. Its Terraform page says that applies to Web Application Protector configurations, so check that it applies to your product. It is a broad exemption: results are then marked as tested from an allowlisted IP.
• Bot Manager: Akamai documents custom bot categories for traffic you know about, for example internal tools or vendor bots, which you can define as friendly in order to allow them. The pages we read do not list which conditions a custom bot can use, so check Akamai's Bot Manager documentation or support for the conditions your product offers, and define the Crawle test server as narrowly as they allow.
• Akamai also recommends starting Bot Manager actions in monitor mode.
• Sources: Akamai TechDocs on setting protections, bypass network lists, detection methods and handling adversarial bots, read 2026-10-13 and 2026-10-14.

Fastly

This covers the Fastly Next-Gen WAF. For another Fastly security product use that product's own documentation.

• In the Fastly control panel go to Security, then Next-Gen WAF, then Rules. Select your workspace, choose Add workspace rule and leave the type as Request. Match all of: IP address equals the test addresses, and Domain equals your site's domain.
• Set the action to Allow. Fastly documents that Allow lets the request through regardless of any blocking rules that matched, and that an allow rule takes precedence over a block rule. That is broad: for this address and domain it skips your own blocking rules, including user-agent rules, so results are marked as tested from an allowlisted IP.
• If one specific signal is what tags and stops the test requests, a signal exclusion rule is narrower: Fastly documents that it stops requests from being tagged with the signal you choose.
• The action types you can use may depend on your platform or packaged offering, so check that Allow is available to you.
• Sources: Fastly's Next-Gen WAF documentation on defining rule conditions and about rules, read 2026-10-13.

Vercel

Vercel Firewall custom rules can let the test through ahead of the managed rulesets, as long as the rule order keeps your own user-agent rules in front.

• In your Vercel project open Firewall, add a custom rule and set the condition IP address equals the test addresses. If the project serves several domains, add a host name condition too. Custom rules are available on all plans.
• Set the action to Bypass: Vercel documents it as bypassing any remaining custom rules, and custom rules run before managed rulesets such as Bot Protection.
• Order matters: Vercel documents that a Deny action ends evaluation and that a Bypass rule must run before the rule you want it to skip. Our reading: place the Bypass rule below your own user-agent rules, so a Deny there still ends evaluation first and the test can see it.
• If you block AI bots with Vercel's AI Bots managed ruleset, a Bypass rule skips that ruleset for this address too, and the test cannot see that block.
• If Vercel's system mitigations, such as DDoS Mitigation, are what stop the request, a Bypass rule does not help: Vercel documents that WAF bypass rules do not bypass system-level mitigations. On Pro and Enterprise plans add a System Bypass rule with the IP address and the domain instead (limited to 25 rules per project on Pro and 100 on Enterprise).
• Vercel's Attack Mode page describes no exemption for a single IP address, and says non-recognized automated services may not be able to pass its challenges. While Attack Mode is on, the test can stay inconclusive.
• Sources: Vercel's documentation on WAF custom rules, rule configuration, managed rulesets, system bypass rules and Attack Mode, read 2026-10-13.

AWS CloudFront WAF

For AWS CloudFront and WAF, exclude the test addresses from the rules that stop automated requests instead of allowing them ahead of everything else.

• AWS WAF protects a CloudFront distribution through a protection pack (web ACL) and evaluates rules from the lowest priority number up. Do not add a top-priority Allow rule for the Crawle address; exclude it from the rules that stop automated requests instead.
• AWS documents Allow as a terminating action: it stops evaluation, so every later rule, including your own user-agent rules, would be skipped.
• Create an IP set with the scope Global (CloudFront) and add the test addresses as CIDR ranges: a /32 for one IPv4 address and a /128 for one IPv6 address.
• Edit the managed rule group that stops automated requests, for example the bot control group, and give it a scope-down statement that puts the whole exception inside one NOT statement. If the protection pack covers only this site, the exception is the IP set match: NOT (the request matches the IP set). Only requests that match the scope-down statement are evaluated by the rule group, so the Crawle address is not, and your other rules still apply. Do the same for a rate-based rule if one stops the test.
• If one protection pack covers several sites, the exception is the IP set match AND a Host header match for your site, and all of it goes inside the NOT: NOT (the request matches the IP set AND its Host header is your site's host). Do not write NOT (IP set) AND Host: the rule group would then stop checking all traffic to your other sites, including the Crawle address there. AWS's visual rule editor cannot nest an AND statement, so write this scope-down statement in JSON.
• If the rule group or rate-based rule already has a scope-down statement, keep it: combine it with the NOT statement in an AND statement, so the rule still covers what it covered before, minus the Crawle exception.
• Sources: the AWS WAF developer guide on web ACL processing order, rule actions, IP sets and scope-down statements, read 2026-10-13, and on scope-down statements and the NOT and AND rule statements, read 2026-10-29.

Sucuri

Sucuri's website firewall allows an address from the dashboard. Allow only the address Crawle shows.

• In the Sucuri dashboard select your website, open the Access Control tab, enter the test addresses under Allow IP Addresses and select Allow.
• Sucuri accepts CIDR ranges between /16 and /32. Allow the single address Crawle shows, not a wider range: Sucuri says allowing many addresses is not recommended unless it is absolutely necessary.
• Sucuri's page does not say which firewall protections an allowed address skips. Assume it may skip the whole firewall for that address, including user-agent blocks, so results are marked as tested from an allowlisted IP.
• Source: Sucuri's website firewall documentation on allowing an IP address, read 2026-10-13.

Imperva

Imperva calls these settings security exceptions and IP whitelists. The guidance we checked is a 2016 post from when the product was called Incapsula, so confirm the current screen names in Imperva's documentation or with Imperva support.

• Prefer a narrow exception to a whitelist. Imperva describes exceptions that can be limited by URL, IP address, country, client application and, for bot access control, user agent, and recommends combining at least two conditions to keep the exception specific.
• For a block that comes from bot access control, create an exception for the test addresses limited to your site's URLs instead of whitelisting the address everywhere.
• Imperva describes an IP whitelist entry as making traffic from that address bypass the WAF and security settings altogether. Use it only if narrower exceptions do not work; results are then marked as tested from an allowlisted IP.
• Source: Imperva's post on best practices for setting security exceptions in the Incapsula WAF, read 2026-10-13.

Server

Your own server, with nginx, Apache or fail2ban: exempt the Crawle address only from the limits and bans that stop automated requests, and leave your user-agent rules alone. If a firewall or hosting provider in front of the server blocks by IP address, allow the address there as well.

• fail2ban: if the test addresses were banned, unban them, and add them to ignoreip for the jails that read your web server logs. The fail2ban manual says ignoreip is a list of IPs not to ban and can include CIDR masks and DNS names.
• nginx: to exempt the test addresses from rate limits, flag them with a geo block, map the flag to an empty key and use that key in limit_req_zone and limit_conn_zone. For example: geo $crawle_test { default 0; <Crawle IP address> 1; } map $crawle_test $limit_key { 0 $binary_remote_addr; 1 ""; }. The nginx documentation says requests with an empty key are not accounted.
• Apache 2.4: Require ip in mod_authz_host allows an address or a CIDR range, for example inside a RequireAny block. It does not exempt a client from a rate limiter or a ban module.
• For the third-party mod_evasive module, add the test addresses with DOSWhitelist, which its README describes as keeping trusted clients from being denied. For other limiters use their own exemption setting.
• Sources: the fail2ban jail.conf manual, the nginx limit_req module, Apache 2.4 mod_authz_host and the mod_evasive README, read 2026-10-13.

Example: a site behind Cloudflare

shop.example.com sits behind Cloudflare, and the access test is challenged by IP reputation. You follow the Cloudflare section to allow Crawle's test addresses, keep your user-agent rules active, and run the access test again.