API access
The public REST API is available at https://api.crawle.io/v1/api. Send your scoped API key in the Authorization header as Bearer <API_KEY>.
- Create an API key in workspace settings and grant only the scopes your integration needs.
- Keep keys in server-side secret storage, never in a browser or source repository.
- For OAuth MCP connections, follow the API and MCP setup guide in Docs.
Request limits
These are the default limits for a rolling 60-second window. Plan limits can be configured differently; the response headers report the limit applied to your request.
| Plan or traffic | Requests per 60 seconds |
|---|---|
| Starter | 200 |
| Pro | 300 |
| Business | 1,000 |
| Enterprise | 5,000 |
| Scale | 10,000 |
| IP fallback, including OAuth MCP | 30 |
- An active API key on the public API or MCP uses its workspace plan limit. Requests using the same key share a limit; different keys have separate counters.
- Requests authenticated with a Crawle sign-in token share a limit for the resolved workspace, including its team members.
- OAuth MCP tokens do not select a workspace plan limit. They fall back to the IP limit, shared by requests from the same IP address.
- Other requests that cannot be resolved to an active API key or signed-in workspace also use the IP limit.
- Health checks, API schema and documentation endpoints, internal routes, and OPTIONS requests are excluded from this global limit.
- Additional endpoint limits can apply. POST /v1/export/request is limited to 3 requests per 5 minutes, shared by its workspace, in addition to the global limit. This endpoint limit does not apply to POST /v1/api/exports or the MCP create_export tool.
- If rate-limit storage is unavailable, requests may proceed without enforcement.
Response headers and retries
Globally rate-limited responses include X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset. Reset is a Unix timestamp in seconds.
- HTTP 429 can mean a request limit or an export concurrency limit was reached. If Retry-After is present, wait for the number of seconds it gives before retrying.
- Avoid tight retry loops: rejected requests are also counted in the global rolling window.
- Endpoint-specific limits may return Retry-After without the full set of global rate-limit headers. If Retry-After is absent, read the error message. When the concurrent export limit is reached, let pending or processing jobs finish before creating another export.