agenticweb.wiki

Web Bot Auth

Last checked · 30 primary sources

This page is about Web Bot Auth, the IETF webbotauth working group's protocol that lets bots and AI agents sign their HTTP requests with HTTP Message Signatures (RFC 9421) and publish their public keys in a key directory [1][11]. It is also called "HTTP Message Signatures for automated traffic", the title of its drafts [4][11]. For the underlying signature standard, see HTTP Message Signatures; for the separate registry and Signature Agent Card draft (an individual draft [29]), see Signature Agent Card (Web Bot Auth registry).

Overview

Web Bot Auth lets an automated client prove which operator sent a request by signing it, instead of relying on User-Agent strings or IP address lists, which the working group charter says have significant limitations [1]. The client signs each request under RFC 9421 HTTP Message Signatures and adds a Signature-Agent header giving the URL of its public keys; the site, or the CDN in front of it, fetches that key directory and checks the signature [11]. The protocol identifies the operator only; whether to allow, challenge or charge a verified bot is left to the site's own policy [23]. It began as two Cloudflare-authored individual drafts in April 2025 and became an IETF working group draft in September 2026 [4][5][30].

Who uses it

Each key directory below was fetched on 2026-10-30 and returned HTTP 200 with no redirect, Content-Type application/http-message-signatures-directory+json, a JSON "keys" array of Ed25519 keys, and Signature and Signature-Input headers on the response. A published directory shows the operator is set up to sign; it does not show that every request it sends is signed. Cloudflare's demo directory and agenticweb.wiki are not listed. DuckDuckGo (DuckAssistBot), Meta and You.com directories, all in the Cloudflare Radar registry [22], also passed these checks on 2026-10-30 but are not listed, because no operator page describing them was found.

Who reads it

Not listed: DataDome, whose changelog page blocked our fetches, and Fastly, for which no statement that it verifies Web Bot Auth signatures was found.

Recent coverage

See also

References

  1. IETF, Web Bot Auth (webbotauth working group charter, approved 2025-10-23): "Current solutions (such as IP allowlisting, User-Agent strings, and shared API keys) have significant limitations regarding security, scalability, and manageability." "The Web Bot Authentication (webbotauth) Working Group will standardize methods for cryptographically authenticating automated clients and providing additional information about their operators to Web sites." — https://datatracker.ietf.org/doc/charter-ietf-webbotauth/
  2. IETF Datatracker, Web Bot Auth (webbotauth) working group page: "Area Web and Internet Transport (wit)" "Chairs David Schinazi, Rifaat Shekh-Yusef Area Director Mike Bishop" "2026-04-30 Apr 2026 Standards track specification(s) describing authentication technique(s) sent to the IESG" "2026-08-31 Aug 2026 Best Current Practice operational specification sent to the IESG" — https://datatracker.ietf.org/wg/webbotauth/about/
  3. IETF Datatracker, Web Bot Auth (webbotauth) group history page: "2025-10-23 Morgan Condie Charter approved, group active" — https://datatracker.ietf.org/wg/webbotauth/history/
  4. T. Meunier (Cloudflare), HTTP Message Signatures for automated traffic Architecture, draft-meunier-web-bot-auth-architecture-00, 15 April 2025: "Web Bot Auth T. Meunier Internet-Draft Cloudflare Intended status: Informational 15 April 2025" "This document describes an architecture for identifying automated traffic using [HTTP-MESSAGE-SIGNATURES]. The goal is to allow automated HTTP clients to cryptographically sign outbound requests, allowing HTTP servers to verify their identity with confidence." — https://www.ietf.org/archive/id/draft-meunier-web-bot-auth-architecture-00.txt
  5. T. Meunier (Cloudflare), HTTP Message Signatures Directory, draft-meunier-http-message-signatures-directory-00, 15 April 2025: "HTTP T. Meunier Internet-Draft Cloudflare Intended status: Standards Track 15 April 2025" "This document describes a method for clients using [HTTP-MESSAGE-SIGNATURES] to advertise their signing keys." — https://www.ietf.org/archive/id/draft-meunier-http-message-signatures-directory-00.txt
  6. IETF Datatracker, draft-meunier-web-bot-auth-architecture (status page): "Authors Thibault Meunier, Sandor Major Last updated 2026-03-02 Replaced by draft-meunier-webbotauth-httpsig-protocol" — https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/
  7. IETF Datatracker, draft-meunier-http-message-signatures-directory (status page): "Authors Thibault Meunier, Sandor Major Last updated 2026-03-02 Replaced by draft-meunier-webbotauth-httpsig-directory" — https://datatracker.ietf.org/doc/draft-meunier-http-message-signatures-directory/
  8. IETF Datatracker, draft-meunier-webbotauth-httpsig-directory (status page): "Replaces draft-meunier-http-message-signatures-directory Replaced by draft-meunier-webbotauth-httpsig-protocol" — https://datatracker.ietf.org/doc/draft-meunier-webbotauth-httpsig-directory/
  9. IETF Datatracker, draft-meunier-webbotauth-httpsig-protocol (status page, latest revision 2026-08-18): "Replaces draft-meunier-web-bot-auth-architecture, draft-meunier-webbotauth-httpsig-directory Replaced by draft-ietf-webbotauth-httpsig-protocol" — https://datatracker.ietf.org/doc/draft-meunier-webbotauth-httpsig-protocol/
  10. IETF Datatracker, draft-ietf-webbotauth-httpsig-protocol (status page): "Active Internet-Draft (webbotauth WG) Authors Thibault Meunier, Sandor Major Last updated 2026-10-29" "WG state WG Document Document shepherd (None) IESG IESG state I-D Exists" — https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/
  11. T. Meunier (Cloudflare) and S. Major (Google), HTTP Message Signatures for automated traffic, draft-ietf-webbotauth-httpsig-protocol-01, 29 October 2026: "Web Bot Auth T. Meunier Internet-Draft Cloudflare Intended status: Standards Track S. Major Expires: 2 May 2027 Google 29 October 2026" "This document describes a protocol for identifying automated traffic using [HTTP-MESSAGE-SIGNATURES]. The goal is to allow automated HTTP clients to cryptographically sign outbound requests, allowing HTTP servers to verify their identity with confidence." "It defines the Signature-Agent header field for in-band key discovery, a key directory format based on JWKS, and a well-known URI at which that directory is served." "tag="web-bot-auth" Signature-Agent: sig="https://signer.example.com"" "A signed request carries three headers 1. Signature defined in Section 5.2 2. Signature-Input defined in Section 5.2 3. Signature-Agent defined in Section 5.2.1" "A directory served at the well-known URI registered in Section 8.1 MUST be served with media type application/http-message-signatures-directory+json." ""HTTP Message Signatures", RFC 9421, DOI 10.17487/RFC9421, February 2024" "Make every Signature-Agent value a direct JWK Set URI. Remove the directory, jwks_uri, and cimd types." "Recommend that an Origin expecting a signature request one with Accept-Signature. Describe signing only on request." "State that identifiers allow correlation across origins." "Signers MUST send the dictionary form." "An Origin that expects a signature SHOULD request one from clients that did not provide it, using the Accept-Signature field" — https://www.ietf.org/archive/id/draft-ietf-webbotauth-httpsig-protocol-01.txt
  12. IANA, Well-Known URIs registry (CSV export). Checked 2026-10-30: it has no http-message-signatures-directory entry — https://www.iana.org/assignments/well-known-uris/well-known-uris-1.csv
  13. Thibault Meunier and Mari Galicer, Cloudflare Blog, Forget IPs: using cryptography to verify bot and agent traffic, 2025-05-15: "For those building bots, we propose signing the authority of the target URI, i.e. www.example.com, and a way to retrieve the bot public key in the form of signature-agent, if present" — https://blog.cloudflare.com/web-bot-auth/
  14. Cloudflare Blog, Message Signatures are now part of our Verified Bots Program, simplifying bot authentication, 2025-07-01: "Starting today, Cloudflare is ramping up verification of cryptographic signatures provided by automated crawlers and bots." "For site owners, no additional action is required – Cloudflare will automatically validate signatures on our edge" — https://blog.cloudflare.com/verified-bots-with-cryptography/
  15. Cloudflare, Web Bot Auth (bot solutions docs), last updated 2026-10-08: "Host a key directory at /.well-known/http-message-signatures-directory (note that this is a requirement)." "Cloudflare implements the Signature-Agent format from draft-meunier-http-message-signatures-directory-03, where the header value is a structured string such as "https://signature-agent.test"." "The message uses the dictionary form from later drafts, such as sig2="https://signature-agent.test"." — https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/
  16. Vercel changelog, Vercel's bot verification now supports Web Bot Auth, 2025-08-12: "We collaborated with industry partners to advance the IETF proposal for Web Bot Auth and Vercel's bot verification system supports the new protocol." "For example, ChatGPT operator signs its requests using Web Bot Auth, so is now allowed." — https://vercel.com/changelog/vercels-bot-verification-now-supports-web-bot-auth
  17. Rupesh Chokshi, Akamai Blog, Redefine Trust with Web Bot Authentication, 2025-11-10 (the live page returns 403 to our fetcher; quoted from the Wayback Machine copy of 2026-06-09): "Web Bot Auth is already supported by Akamai's bot & abuse solutions, inclusive of bot visibility and mitigation in Akamai App & API Protector." "Edge verification — The Akamai edge retrieves the bot's signature directory, validates the signature using RFC 9421 rules" — https://web.archive.org/web/20260609113733/https://www.akamai.com/blog/security/redefine-trust-web-bot-authentication
  18. AWS What's New, AWS WAF announces Web Bot Auth support, 2025-11-21: "Today, we're excited to announce the addition of Web Bot Auth (WBA) support in AWS WAF, providing a secure and standardized way to authenticate legitimate AI agents and automated tools accessing web applications." "AWS WAF now automatically allows verified AI agent traffic. Verified WBA bots will now be automatically allowed by default." — https://aws.amazon.com/about-aws/whats-new/2025/11/aws-waf-web-bot-auth-support/
  19. Ben Diamant and Tomer Elias, HUMAN Blog, Introducing HUMAN Verified AI Agent: An Open-Source Foundation for Trustworthy Agent Identity, 2025-07-21: "These signatures are verified by a gateway service using a public key registry, allowing web services to cryptographically validate that a specific agent sent a given request." — https://www.humansecurity.com/learn/blog/human-verified-ai-agent-open-source/
  20. Google for Developers, Authenticate requests with Web Bot Auth (experimental), last updated 2026-10-13: "Google is testing usage of the Web Bot Auth IETF internet draft, which is a new cryptographic protocol that helps websites to validate that bots are authentic." "A subset of requests made by the Google-Agent are signed with Web Bot Auth; in these cases, they are authenticated as https://agent.bot.goog." — https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth
  21. Harsehaj Dhami, Browserbase Blog, What is Web Bot Auth?, 2026-09-30: "It's why we're part of Cloudflare's Signed Agents program, so participating websites can explicitly allow Browserbase sessions." — https://www.browserbase.com/blog/what-is-web-bot-auth
  22. Cloudflare Radar, signature-agent-registry.txt (plain-text list of key directory URLs, undated): "https://chatgpt.com/.well-known/http-message-signatures-directory" "https://www.browserbase.com/.well-known/http-message-signatures-directory" "https://cloudsurf.shopify.io/.well-known/http-message-signatures-directory" "https://crawler.exa.ai/.well-known/http-message-signatures-directory" "https://assistbot.duckduckgo.com/.well-known/http-message-signatures-directory" "https://www.meta.com/.well-known/http-message-signatures-directory" "https://you.com/.well-known/http-message-signatures-directory" — https://assets.radar.cloudflare.com/bots/signature-agent-registry.txt
  23. Maria Paktiti, WorkOS Blog, Web Bot Auth: How to identify AI agents and set policy for them, 2026-10-19: "Web Bot Auth is an IETF draft protocol that lets automated clients, including AI agents, sign their HTTP requests with HTTP Message Signatures (RFC 9421) and point to a public key directory, so a website can verify which operator sent a request." — https://workos.com/blog/web-bot-auth-identify-ai-agents
  24. Luis Rijo, PPC Land, Explaining Web Bot Auth, 2026-09-12: "Web Bot Auth is a protocol that lets an automated client attach a cryptographic signature to the HTTP requests it sends, so the website receiving them can establish which service sent them rather than taking the sender's word for it." — https://ppc.land/web-bot-auth/
  25. Lantad Blog, Web bot authentication: 2 of 9 AI crawler vendors publish a key directory, 2026-09-15: "Web bot authentication replaces a crawler's user agent claim with a signature a server can check, and the only half of it a site can inspect from outside is the operator's published key directory." — https://lantad.co/blog/web-bot-authentication-two-of-nine-publish-a-directory
  26. hCaptcha Blog, Web Bot Auth: How hCaptcha Supports it, 2026-10-02: "hCaptcha Enterprise now supports Web Bot Auth (WBA), giving partner agent platforms a cryptographic way to identify their traffic." — https://www.hcaptcha.com/blog/web-bot-auth
  27. Exa, Exa Search Crawler (crawler documentation), last updated 2026-07-30: "Every ExaSearchBot request is cryptographically signed using HTTP Message Signatures (RFC 9421), following the Web Bot Auth scheme." — https://crawler.exa.ai/
  28. IANA, Media Types registry, application subtype list (CSV export). Checked 2026-10-30: it has no http-message-signatures-directory+json entry — https://www.iana.org/assignments/media-types/application.csv
  29. IETF Datatracker, draft-meunier-webbotauth-registry (status page; revision -04, an individual draft): "Document Type Active Internet-Draft (individual) Authors Maxime Guerreiro, Ulas Kirazci, Thibault Meunier Last updated 2026-10-27" — https://datatracker.ietf.org/doc/draft-meunier-webbotauth-registry/
  30. IETF Datatracker, draft-ietf-webbotauth-httpsig-protocol history page: "2026-09-01 00 Thibault Meunier New version available: draft-ietf-webbotauth-httpsig-protocol-00.txt" "2026-09-01 00 Rifaat Shekh-Yusef This document now replaces draft-meunier-webbotauth-httpsig-protocol instead of None" — https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/history/

JSON · Markdown · OKF v0.2