Update (2026-10-07): Rechecked the IETF boundary, Cloudflare workflow, and Google rollout.
A Web Bot Auth signature does not prove who operates an AI agent, whether the agent is benign, or whether its submitted content is accurate. In draft-ietf-webbotauth-httpsig-protocol-00, resolving a Signature-Agent URL only proves that the server at that URL published the key that signed the request. Verifying the signature validates key possession, not editorial authority.
Web Bot Auth key facts
draft-ietf-webbotauth-httpsig-protocol-00is dated 2026-09-01 and lists Thibault Meunier of Cloudflare and Sandor Major of Google as authors. It definesSignature-Agentfor in-band key discovery.- Section 4.4 limits the proof to key hosting: resolving the URL over TLS establishes only that the named host served the key set at fetch time.
- Sections 4.1 and 4.6 leave operator identity, human authentication, authorization, delegation, and trust exchange outside the protocol.
- As of 2026-10-07, Google's guide calls Web Bot Auth experimental, says not all Google user agents use it, and recommends keeping IP, reverse DNS, and user-agent checks as fallbacks.
- Cloudflare's Verified Bots page, last updated on 2026-07-01, separates honest self-identification from non-abusive behavior and requires an application before approval.
- Appendix C.1 defines
verified,invalid, andunverified; it says origins can apply local policy to each outcome.
What does a Signature-Agent URL actually prove?
A valid Web Bot Auth signature proves that the sender holds the private key corresponding to a public key published at the given Signature-Agent URL. It does not prove that the operator is honest, known, authorized, or benign.
Under Section 4.1 of draft-ietf-webbotauth-httpsig-protocol-00, an unresolved Signature-Agent header is a claim rather than an identity until the origin fetches the directory. Even after fetching /.well-known/http-message-signatures-directory, Section 4.4 defines the boundary of that proof:
"Resolving a Signature-Agent URL over TLS establishes that the host named in the URL served this key set at fetch time. Whoever controls that URL says this key signs for it. That is what makes the URL usable as an identifier, and all it gives you. It does not say that the operator of that URL is honest, or that it is the same party everyone knows about."The protocol leaves reputation and content decisions to origin policy. If an origin accepts automated corrections, a valid signature should be an intake signal, not a publish decision. The companion explainer How AI agents sign HTTP requests with Web Bot Auth covers the wire mechanics; this post covers the trust boundary.
Why does Cloudflare still require an application for verified bots?
Cloudflare does not grant automatic Verified bot status merely because requests carry cryptographic signatures. On its Verified Bots page last updated on 2026-07-01, Cloudflare requires bots to satisfy two bars:
- Honest self-identification: declaring identity deterministically via Web Bot Auth signatures, a published IP list, or reverse DNS.
- Non-abusive behavior: obeying
robots.txt, observing crawl rates, and avoiding scraping abuses.
A signature satisfies only honest self-identification. Because a signature does not show whether an agent respects crawl directives or request rates, operators must submit an application in the Cloudflare dashboard. Only after Cloudflare approves the submission does the agent appear in BotBase and Cloudflare Radar.
On 2026-07-01, Cloudflare also separated operating models into Direct and Intermediary. Intermediary services create transitive trust because the operator and end user are different parties. Cloudflare is experimenting with the Forwarded header from RFC 7239 to carry end-user information. These edge protections build on the controls examined in What Cloudflare turns on for AI bots on a new domain and Cloudflare Content Signals in robots.txt: what search, ai-input, and ai-train mean.
Cloudflare expects three headers on signed agent requests: Signature-Agent, Signature-Input (specifying tag web-bot-auth), and Signature:
Signature-Agent: "https://signature-agent.test"
Signature-Input: sig2=("@authority" "signature-agent");created=1735689600;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";alg="ed25519";expires=1735693200;nonce="e8N7S2MFd/qrd6T2R3tdfAuuANngKI7LFtKYI/vowzk4lAZYadIX6wW25MwG7DCT9RUKAJ0qVkU0mEeLElW1qg==";tag="web-bot-auth"
Signature: sig2=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA1nZDsi6nJKuHxUi/5Syp3rLWBA==:Cloudflare's example uses the quoted-string form. The 2026-09-01 IETF draft defines Signature-Agent as a structured dictionary, so check the syntax your verifier implements.
How should an agent correction queue handle incoming signatures?
When an origin exposes an automated correction queue, it should separate signature verification from content publishing. A verified result says that the signature and key material validate. It does not say that the correction is true.
Section 7.2 of draft-ietf-webbotauth-httpsig-protocol-00 recommends keys that represent a role, company, or automation identity rather than a specific human:
"A key tied to a specific human individual exposes personally identifiable information and makes the key usable for user tracking or profiling. A key that represents a role, company, or automation identity (e.g., 'news-aggregator-bot', 'example-crawler-v1') avoids this."
Because the protocol does not authenticate a human editor, incoming submissions must not bypass editorial review. Following Appendix C.1 of the draft, a queue can handle three states:
Verifier outcome | Cryptographic check | Ingestion behavior | Editorial routing |
|---|---|---|---|
| Signature and key material validate | Keep the payload; record the | Place it in a private queue for human review; never auto-publish |
| Directory discovery failed or the key is unknown | Hold the payload or return a retry signal | Quarantine the submission and require evidence checks |
| Signature, covered components, key, or freshness checks fail | Reject or isolate the payload and log the event | Do not process it as an accepted correction |
Recording the Signature-Agent URL and verification result provides an audit trail while keeping a human on the accept button.
How do Web Bot Auth signatures differ from paid crawling?
Cryptographic identity and monetization answer different questions. Web Bot Auth provides identity discovery; Cloudflare's payment features handle access and settlement.
In Cloudflare's Pay Per Crawl beta, an AI crawler either presents payment intent for a successful HTTP 200 response or receives HTTP 402 Payment Required with pricing. Site owners configure a minimum price of $0.001 USD per crawl. Separately, Cloudflare announced its Monetization Gateway closed beta on 2026-09-30, using Payment-Method: x402 and settlement with USDC on Base.
Property | Web Bot Auth (IETF draft) | Cloudflare Pay Per Crawl | Cloudflare Monetization Gateway |
|---|---|---|---|
Status | IETF draft (published 2026-09-01) | Beta (in AI Crawl Control) | Closed beta (announced 2026-09-30) |
Mechanism | Structured HTTP Message Signatures | Payment intent with |
|
Minimum pricing | No price | $0.001 USD per crawl | Variable per request |
What it proves | Host published the signing key | Payment intent for a priced crawl | Payment authorization and settlement for a resource |
Does not prove | Editorial accuracy or human authority | Crawler honesty | Factual truth |
Neither paying $0.001 USD per crawl nor signing with an Ed25519 key makes crawler output suitable for automatic publication.
Sources
- HTTP Message Signatures for automated traffic (draft-ietf-webbotauth-httpsig-protocol-00) (read 2026-10-07)
- Cloudflare Verified bots documentation (read 2026-10-07)
- Cloudflare Web Bot Auth (read 2026-10-07)
- Google Web Bot Auth verification guide (read 2026-10-07)
- Cloudflare What is Pay Per Crawl documentation (read 2026-10-07)
- Cloudflare Set a pay per crawl price documentation (read 2026-10-07)
- Cloudflare Monetization Gateway beta announcement (read 2026-10-07)
Last verified: 2026-10-07.