What is the adoption and standardization status of Cloudflare Pay per crawl?

Closed beta; documentation updated 2026-07-28 — phase candidate, governed by Cloudflare (single vendor). Adoption: unmeasured. Version: closed beta (2026-07-28). Last verified 2026-10-16.

How does Cloudflare Pay per crawl compare to other pay protocols?

Cloudflare Pay per crawl is one of 14 standards at step 7, Pay (Settle the bill). It pairs with Signature Agent Card (Web Bot Auth registry), Web Bot Auth HTTP Message Signatures, x402, Machine Payments Protocol. It builds on HTTP Message Signatures, Web Bot Auth HTTP Message Signatures. Head-to-head comparisons on this site: x402, L402, Machine Payments Protocol. Every standard at this step is listed at https://agenticweb.wiki/category/pay/.

Cloudflare Pay per crawl

7 · Pay

Cloudflare Pay per crawl
Category	Pay
Governed by	Cloudflare (single vendor)
Status	Closed beta; documentation updated 2026-07-28
Phase	candidate
Implementation	developer trial
Adoption	unmeasured
Version	closed beta (2026-07-28)
Specification	https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/
Found at	HTTP 402 response carrying a crawler-price header
Last checked	2026-10-16
Freshness	current · checked 2026-10-16
Solves	Accept payments from agents, Control AI crawling and training use, Prove an agent's identity

Cloudflare Pay per crawl is an agentic-web standard (HTTP header) at step 7, Pay, governed by Cloudflare (single vendor); its own status is "Closed beta; documentation updated 2026-07-28" (candidate). It answers: How can a site charge an AI crawler per request, and how does the crawler agree to pay? Adoption is unmeasured, last verified 2026-10-16.

Pay Pay per crawl is Cloudflare's way for a site to charge AI crawlers per request. The site answers with HTTP 402 and a crawler-price header; the crawler agrees by sending crawler-exact-price or crawler-max-price, and a charged response carries crawler-charged ^[1]. Crawlers must sign requests with Web Bot Auth, so a priced crawl is also an identified one ^[1].

Overview

Cloudflare documents it as a closed beta, and the documentation was last updated on 2026-07-28 ^[1]. It is a single-vendor mechanism, not an open standard, and sits in the HTTP 402 family with x402 and MPP.

The record could equally sit under permit; it is placed at pay because the exchange is a payment.

Discovery and probe

HTTP 402 response carrying a crawler-price header

GET {origin}/{priced-path}
parse:     HTTP 402 with a crawler-price response header; a crawler pays by sending crawler-exact-price or crawler-max-price, and the charged response carries crawler-charged. The crawler must sign requests with Web Bot Auth.
on absent: A 200 or a 402 without a crawler-price header means this path is not priced by pay per crawl; the probe cannot see an unsigned-agent decision.

Not probed live: no-fixed-path. candidate record; probe rule pending

History

2026-10-15 — curated update 2026-10-15
2026-10-16 — use-case tags (auto)

Instances

No instances recorded yet.

Comparisons

References

Cloudflare's documentation says pay per crawl uses HTTP 402 with crawler-price, crawler-exact-price / crawler-max-price and crawler-charged headers, requires Web Bot Auth signing, and is in closed beta (documentation updated 2026-07-28). — https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/ (2026-07-28) VERIFIED
Cloudflare's introductory post describes pay per crawl. — https://blog.cloudflare.com/introducing-pay-per-crawl/ REPORTED

JSON · Markdown