User-Agent & IP Allowlists vs Web Bot Auth (RFC 9421)
HTTP User-Agent strings & reverse-DNS / IP allowlists is the classic web primitive built for human browsers and pre-integrated services (Identify the client software name and verify known search crawlers via static IP CIDR lists or reverse DNS), whereas Web Bot Auth HTTP Message Signatures (WebBotAuth) and its step 2 (Identify) peers provide the machine-native agentic surface (RFC 9421 Signature-Agent) so autonomous agents can answer: "Which automated client made this HTTP request, and can the origin verify it cryptographically?" Across the agenticweb.wiki conformance panel, Web Bot Auth HTTP Message Signatures is multi-implementation (9 of 574 probed origins (last checked 2026-10-23)).
How User-Agent & IP Allowlists compares to Web Bot Auth (RFC 9421) at step 2 (Identify): why the classic primitive breaks for software agents, and how sites deploy both together.
| Dimension | HTTP User-Agent strings & reverse-DNS / IP allowlists Classic web | Web Bot Auth HTTP Message Signatures WebBotAuth Agentic web |
|---|---|---|
| Primary consumer | Human driving a web browser or pre-integrated developer code | Autonomous software agent, coding harness, or AI search/RAG crawler |
| Lifecycle step | Classic web baseline for step 2 (Identify) | 2 Identify: Who is this agent? Who is this site? |
| Core question answered | Identify the client software name and verify known search crawlers via static IP CIDR lists or reverse DNS. | Which automated client made this HTTP request, and can the origin verify it cryptographically? |
| Wire format & entrypoint | User-Agent header + source IP range JSON | HTTP header at RFC 9421 Signature-Agent |
| Failure mode for agents | Any scraper can spoof a User-Agent header in one line of code, and user-triggered agents running on cloud workers, residential proxies, or laptops do not come from a fixed publisher IP range. | Solved natively: Web Bot Auth is an IETF specification that lets an automated HTTP client (a crawler, agent, or bot) cryptographically sign its outbound requests using RFC 9421 HTTP Message Signatures, so the receiving server can verify the client's identity and intent without relying on spoofable signals like the User-Agent string or source IP. |
| Governance & status | Established IETF / W3C / industry baseline | IETF webbotauth Working Group (chartered; document adopted as a WG item 2026-09-01) — Active Internet-Draft, WG Document, Intended Status: Standards Track; expires 5 March 2027 |
| Live adoption (/check/ census) | Universal across the classic web | multi-implementation — 9 of 574 probed origins (last checked 2026-10-23) |
| Primary specification | https://www.rfc-editor.org/rfc/rfc9110#name-user-agent | https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/ |
Why User-Agent & IP Allowlists breaks when the caller is an agent
Any scraper can spoof a User-Agent header in one line of code, and user-triggered agents running on cloud workers, residential proxies, or laptops do not come from a fixed publisher IP range.
When to use which (and how they work together)
- Keep HTTP User-Agent strings & reverse-DNS / IP allowlists: You are logging basic browser analytics or verifying legacy search crawlers that still operate from static datacentre ASN blocks.
- Add Web Bot Auth HTTP Message Signatures (WebBotAuth): You need cryptographic proof of which agent made an HTTP request regardless of IP address, using RFC 9421 HTTP Message Signatures (Signature, Signature-Input, Signature-Agent) verified against /.well-known/http-message-signatures-directory.
All agentic web standards replacing or extending User-Agent & IP Allowlists
- Web Bot Auth HTTP Message Signatures (WebBotAuth) — multi-implementation · 9/574 probed origins: Which automated client made this HTTP request, and can the origin verify it cryptographically?
- HTTP Message Signatures — multi-implementation: How can HTTP messages carry end-to-end component integrity and proof of possession, independent of transport-layer security?
- Signature Agent Card (Web Bot Auth registry) — prototype · 1/574 probed origins: What identity, purpose, rate expectations and cryptographic keys belong to a signed bot?
- SPIFFE (Secure Production Identity Framework for Everyone) — multi-implementation: Which workload instance — potentially including an AI agent process — is making this request, verifiable independent of network location?
- Workload Identity in Multi System Environments (WIMSE) Architecture — prototype: How do workloads — including, per a related draft, AI agents — authenticate to each other and to services across platforms and trust domains?
- ERC-8004: Trustless Agents — multi-implementation: Who is an agent, and what onchain identity, reputation and validation record does it carry?