| Internet-Draft | Crawler Traffic Verification Levels | November 2026 |
| Arsentev | Expires 19 May 2027 | [Page] |
The User-Agent header is written by the client. It is a claim, not an identity. Reports of crawler and AI crawler traffic that count requests by User-Agent token alone present claims as activity. This document defines four counts a crawler traffic report should give per crawler token: declared, route-consistent, verified, and unverifiable. It states which verification check was possible at the vantage point where the log was taken. It also defines a small JSON report object.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 19 May 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document.¶
Site operators, CDNs, analytics vendors and researchers publish counts of "AI crawler traffic". They produce these counts by matching the User-Agent string (Section 10.1.5 of [RFC9110]) against lists of crawler tokens. The User-Agent value is chosen by the client.¶
Crawler operators increasingly publish IP address ranges for their crawlers, and some offer reverse DNS checks. Work on cryptographic bot authentication with HTTP Message Signatures ([RFC9421]) is under way in the IETF (webbotauth).¶
Whether a check can be applied depends on where the log is recorded. An origin server behind a CDN usually sees the CDN's address, not the client's address. A count that mixes checked and uncheckable requests cannot be compared across sites.¶
The goal of this document is to make the verification level of every reported number explicit.¶
This document does not define how to verify a crawler. It does not define access policy, which is the subject of robots.txt ([RFC9309]) and of AI preference work. It does not say whether unverified traffic should be blocked.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
For each crawler token and reporting period, the report gives the following counts.¶
The report MUST give the declared count together with at least one of the other three counts. A report MUST NOT present the declared count alone as crawler activity.¶
Requests that were checked and failed are the declared count minus the verified count minus the unverifiable count. The report SHOULD state that number.¶
When the operator of a token publishes no ranges and no other check exists, all requests under that token are unverifiable. The report says so instead of reporting zero verified.¶
The report names the vantage point, the period, and the token list with its version or date. For each token, the report states which checks were available: address range, reverse DNS, message signature, or none.¶
A count from a CDN edge and a count from an origin are different measurements. They are not added together.¶
If address ranges were fetched, the report gives the date they were fetched, because ranges change.¶
The report is carried as a JSON object ([RFC8259]). The object carries one entry per crawler token with the four counts, the checks performed, and the vantage point. The example in the appendix uses illustrative numbers, not measurements.¶
The report is a JSON object [RFC8259] with the following members.¶
A report MUST satisfy, for every entry, verified + unverifiable + failed = declared when all three are present. Members not defined here MAY be added; consumers MUST ignore members they do not understand.¶
This section reports one case. The data come from one origin server behind a CDN, with full logs for 15 days. The server logged 345,808 requests in total. Of these, 46,155 carried one of 22 published crawler tokens.¶
Of those declared requests, 35.7% reached the origin directly, bypassing the CDN. None of the 11,276 direct requests under tokens whose operators publish address ranges came from those ranges. In other words, these requests were not confirmed by the published ranges.¶
The direct traffic never requested robots.txt or the sitemap. It used about 3.5 times more distinct User-Agent strings per request.¶
A token that Google states it never sends as a User-Agent arrived through the CDN 293 times.¶
On this origin, no request could be verified by address, because CDN traffic is logged with the CDN's address. A declared-only report from this server would have shown 46,155 crawler requests. The four counts show that none of them was verified.¶
This is one server, and the result is not a statement about the web in general. The analysis is published as a preprint on Qeios, DOI 10.32388/SRD4TB [QEIOS].¶
Verification results can themselves be wrong. Published ranges change. Shared cloud addresses may host other clients. Reverse DNS needs forward confirmation. A report that names its checks lets readers judge these limits.¶
Publishing per-token counts may help an attacker learn which tokens pass a site's filters. Operators MAY delay or aggregate reports.¶
The report carries counts only. It MUST NOT carry client IP addresses, full User-Agent strings of individual requests, request paths with query strings, or visitor identifiers.¶
Small counts for rare tokens can still point to individuals or organisations. Operators SHOULD suppress or round small cells.¶
This document has no IANA actions.¶
The numbers in this example are illustrative and are not measurements.¶
{
"vantage_point": "origin",
"period": {"start": "2026-11-01T00:00:00Z",
"end": "2026-11-15T00:00:00Z"},
"token_list": {"source": "example-token-list",
"as_of": "2026-11-01"},
"tokens": [
{"token": "ExampleBot",
"declared": 1200, "route_consistent": 800,
"verified": 0, "unverifiable": 800, "failed": 400,
"checks": ["address-range"], "ranges_fetched": "2026-11-01"},
{"token": "OtherBot",
"declared": 300, "route_consistent": 300,
"verified": null, "unverifiable": 300, "failed": 0,
"checks": ["none"]}
]
}
¶
In the first entry, 800 requests arrived through the CDN and are logged with the CDN's address, so they cannot be checked at the origin; 400 arrived directly and none of them came from the published ranges. In the second entry the operator publishes no ranges, so every request is unverifiable.¶