Internet-Draft Attested Collection November 2026
McKenna Expires 20 May 2027 [Page]
Workgroup:
Network Working Group
Internet-Draft:
draft-mckenna-webbotauth-collection-attestation-00
Published:
Intended Status:
Informational
Expires:
Author:
S. McKenna
Independent

Attested Conduct for Anonymous Web Data Collection: Use Cases, Requirements, and a Tiered Model

Abstract

Anonymous Bot Authentication lets a web site learn that an automated client holds a credential from an Anchor whose policies it accepts, without learning which client it is. This document describes web data collection use cases that need that property, and what they require of the evidence an Anchor evaluates, of what a site can verify about conduct it cannot attribute, and of what a buyer of collected data needs to see. It proposes a tiered model. Collectors register signed statements of their own conduct in a SCITT Transparency Service; the site, its security intermediary, or a network intermediary may corroborate them; and an Anchor acting as a certifying body may appraise them. Coarse, unlinkable attributes about the data's eventual recipient let a site restrict use by category of recipient without learning who the recipient is.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 20 May 2027.

▲

Table of Contents

1. Introduction

A large share of automated web traffic is data collection: crawlers, monitors, and research tools that retrieve public pages for indexing, archiving, price and availability tracking, financial research, academic study, and consumer protection. Sites have good reasons to manage this traffic, and the IETF is developing two ways to do so: identified bots that sign their requests [I-D.ietf-webbotauth-httpsig-protocol], and Anonymous Bot Authentication (ABA) [I-D.rescorla-anonymous-webbotauth], in which a bot presents an anonymous credential issued under the policies of an Anchor.

Many legitimate collectors cannot use the identified path. A financial firm that reveals which retailers it monitors reveals its research. A consumer protection monitor that is identified can be served different content or blocked. ABA addresses this, and its authors note the tension directly: abuse controls that depend on knowing the identity of the abusive party do not work with anonymous authentication.

ABA specifies how a credential is issued and presented. It leaves open three questions this document takes up: what evidence an Anchor relies on when it decides that a collector complies with its policies, how parties who do not trust the collector can check that evidence, and what a site or a data buyer can learn about conduct it cannot attribute.

This document defines no new protocol. It shows how existing pieces (the SCITT architecture [RFC9943], the RATS roles [RFC9334], and ABA with MoLE [I-D.jms-mole-architecture]) fit this use case, and where gaps remain.

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

2. Terminology

Collector:

An automated client that retrieves web content in order to produce data for its operator or for a recipient.

Recipient:

The party that ultimately receives the collected data, which may be the collector's operator or a downstream buyer.

Site:

The web site whose content is collected, together with any security intermediary (for example a bot management or WAF provider) acting for it.

Anchor:

As in [I-D.rescorla-anonymous-webbotauth]: a party that evaluates whether a collector complies with its policies and issues endorsements. In RATS terms [RFC9334], an Anchor appraising evidence about a collector acts as a Verifier.

Conduct Statement:

A signed statement by a collector describing its own behavior toward a site over a stated interval.

Transparency Service:

As defined in the SCITT architecture [RFC9943]: an append-only, verifiable log that registers signed statements and returns receipts.

3. Use Cases

3.1. Financial Research

Investment firms buy data on prices, inventory, job postings, and product listings. Their compliance teams increasingly ask for evidence of how the data was collected: whether robots exclusion [RFC9309] and other stated preferences were honored, whether request rates were reasonable, and whether the data was collected under the terms the vendor claims. The firms also cannot let sites learn which companies they watch.

3.2. Price Comparison and Consumer Protection

[I-D.rescorla-anonymous-webbotauth] lists shopping sites blocking price comparison bots, and employment or housing sites blocking bots that monitor for discrimination, among the harms of identity-only schemes. These collectors have to show good conduct because they cannot show who they are.

3.3. Archiving and Research

Archives and academic projects collect at scale for public-interest purposes. They are usually willing to be identified, and a verifiable record of conduct helps them when a site questions their behavior.

4. Requirements

R1 (Unlinkability):

A site MUST NOT be able to link a collector's requests to a specific collector or recipient beyond what the credential scheme in use allows.

R2 (Verifiable conduct):

The behavior that matters to sites (request rate, backoff when a site slows or signals overload, adherence to robots exclusion and other published preferences) SHOULD be recorded in a form that a party other than the collector can check.

R3 (Non-equivocation):

A collector MUST NOT be able to present one record of its conduct to a site or Anchor and a different record to a recipient.

R4 (Progressive assurance):

The model SHOULD work without any certifying body, and SHOULD let stronger assurance be added later without changing what collectors record.

R5 (Minimal attributes):

Attributes disclosed to sites MUST be coarse. As participants noted when Web Bot Auth discussed ABA, fine-grained attributes become fingerprints that defeat anonymity.

R6 (Recipient restrictions):

A site SHOULD be able to express that certain categories of recipient may not receive data collected from it, and a collector SHOULD be able to prove membership in a permitted category without revealing the recipient.

R7 (Accountability without identity):

Misconduct SHOULD have consequences, chiefly loss of endorsement at the Anchor, without requiring that the site learn who misbehaved.

R8 (Collector-governed rates):

A well-behaved collector SHOULD set and enforce its own request rate for each site, and record the basis for it. Sites keep full control of their access policy. The aim is to give them a reason to rely on attested self-governance instead of limits per client, which can be used to discriminate among collectors and are not tied to the site's actual load. A site MAY publish capacity hints that apply equally to all automated clients. As with TCP congestion control, collectors need not coordinate: if each backs off when a site slows, their combined load falls as the site's load rises.

5. A Tiered Model

The model has three tiers. Each adds a party and a stronger claim. A collector at any tier records the same Conduct Statements.

5.1. Tier 0: Self-Attested, Logged Conduct

The collector produces Conduct Statements automatically from its own operation. A statement covers one site over one interval and contains:

  • the interval;

  • a keyed reference to the site, so that the log does not disclose which sites the collector visits;

  • request counts and the maximum and mean request rate;

  • the rate ceiling and per-interval budget the collector set for itself, and the inputs it used to set them (see Section 5.1.1);

  • evidence of backoff: response-time and error-rate observations, how each slowdown was classified, and the collector's corresponding rate changes;

  • a digest of the robots exclusion file and of any other preference signals in force during the interval, and whether they were followed;

  • a digest of the extraction configuration and software version used.

The collector signs each statement and registers it with a Transparency Service [RFC9943]. The receipt proves that the statement existed at that time, and the collector cannot later present a different one (R3).

Keyed site references are not enough on their own. A long-lived signing key, or software and configuration digests repeated in every statement, would let anyone reading the log link all of one collector's statements and see its whole footprint. Collectors SHOULD therefore sign with keys that are not reused across sites or epochs, for example keys bound to the credential or derived per site and epoch, and SHOULD salt or coarsen digests in the log, disclosing exact values only to a verifier that needs them.

Tier 0 proves what the collector said and when, not that it is true. It is cheap to adopt, it creates a consistent record that later tiers can check, and it lets a recipient check a vendor's claims against a log the vendor cannot rewrite.

5.1.1. Collector-Governed Rates

Under R8 the collector, not the site, sets its rate:

  1. It estimates the site's normal traffic and capacity from available signals: third-party traffic estimates, observed response times, error rates and page weight, time-of-day patterns, and any capacity hint the site publishes for all automated clients.

  2. It sets its rate ceiling to a small fraction of that estimate, and keeps its own request budget per site and interval.

  3. It monitors response times continuously against the site's baseline.

  4. It separates site slowdowns from slowdowns in its own network path, such as a proxy vendor, by comparing measurements across several independent egress paths, and where available consults the site's status page or a third-party uptime monitor. A slowdown seen on most paths is treated as the site's; one confined to a single path or vendor is treated as the path's.

  5. It separates overload signals from path-specific refusals. An HTTP 429 or 503, a Retry-After header, or rising errors seen across several paths indicate that the site is under load. A refusal confined to one path, such as a 429 on the first request from an address the site or its bot-management service has blocked, while other paths and ordinary browsers reach the site normally, reflects the reputation of that path rather than the site's load.

  6. It reduces its rate on a site slowdown or overload signal, honors any Retry-After, and recovers gradually once the site returns to baseline. On a path-specific slowdown or refusal it may move requests to another path. Rerouting changes the path, never the rate: the collector's total rate to the site stays within its ceiling.

The Conduct Statement records each of these decisions and the measurements behind them, including each reroute and the evidence that the refusal was confined to one path. A verifier can then judge whether the collector backed off and whether its ceiling and classifications had a reasonable basis. Estimation methods are left to implementations; recording their inputs and outputs is what makes them checkable.

5.2. Tier 1: Corroborated Conduct

A site, or a security intermediary acting for it, observes requests that present a given credential. ABA lets the site apply budgets and rate limits per credential; under R8 these are a backstop, not the main control. In this tier the site also signs a statement of what it observed for that credential over an interval (request volume and rate, and any limits it applied), and either returns it to the collector or registers it in a Transparency Service.

Matching the site's observations against the collector's Conduct Statements removes the main weakness of Tier 0: the collector is no longer the only source of evidence about its own behavior. How a site statement is bound to a collector's statements without creating a linkable identifier is an open question (see Section 8).

5.2.1. Intermediary Witnesses

Corroboration need not come from the site. A network intermediary that carries the collector's traffic, such as a proxy network, VPN or egress gateway, scraping-infrastructure provider, or cloud network service, sees that traffic directly, and the collector cannot alter what it sees. Such an intermediary can sign a witness statement for a keyed site reference and interval.

What it can attest to depends on what it can see. A proxy that only tunnels encrypted connections sees the destination host, timing, and volume: it can attest to connection and request rates and how they changed, but not to response statuses or headers. A service that terminates TLS for the collector, such as an unblocking API or a hosted browser, can also attest to the statuses, response times, and Retry-After values it relayed, and to how the collector's rate changed afterward. When a collector uses several paths or intermediaries, each records which of its paths received refusals or slowdowns, so a verifier can confirm that a reroute followed a refusal confined to one path and did not raise the collector's total rate to the site.

Witness statements use the collector's keyed site references, so they reveal nothing about which sites were visited beyond what the intermediary already knows, and they SHOULD use signing keys that do not link the collector across sites. They can be registered in a Transparency Service, matched against the collector's Conduct Statements, combined across independent intermediaries so that none is trusted alone, and shown to Anchors, auditors, or data buyers. They add evidence from a party other than the collector without any change at the site, but they supplement site corroboration rather than replace it. Each site and each Anchor decides whether to accept witness statements, and from which intermediaries.

5.3. Tier 2: Certified Conduct

An Anchor, acting as a certifying body, appraises the collector's Conduct Statements and any corroborating statements against its policies, in the sense of a RATS Verifier [RFC9334]. It may also verify the identity of the collector's operator and of the recipient, and keeps both to itself. On a favorable appraisal it issues an endorsement under [I-D.rescorla-anonymous-webbotauth].

Because the Anchor sees identity and the site sees only the credential, identity checks can be strict without weakening R1. An Anchor that endorses a collector whose logged conduct contradicts its policies can be held to account, because the evidence it relied on is in a log it does not control.

6. Recipient Categories

Some sites object less to collection itself than to who receives the data: for example, direct competitors, or automated pricing systems in the same product category. Competition law in several jurisdictions is also concerned with the exchange of commercially sensitive information among competitors, including through intermediaries and algorithms. This document takes no position on any law.

Recipient categories are not a general tool for sites to choose who may read public data; used that way, they would bring back the discrimination R8 is meant to avoid. They SHOULD be limited to restrictions with a legal basis, such as competitors and pricing systems in the same product category, and SHOULD NOT be used to exclude journalists, researchers, archives, or consumer protection monitors.

A Tier 2 Anchor can verify a recipient's category and include it as a coarse attribute of the endorsement, for example "financial research" or "not a seller in category X". A site can then state, alongside its other preferences, which categories may not receive its data, and the collector proves membership in a permitted category without revealing the recipient. Attributes MUST be few and broad (R5).

7. Relationship to Other Work

8. Open Questions

  1. How can a site's corroborating statement be matched to a collector's Conduct Statements without creating a linkable identifier? Candidate approaches include per-interval pseudonyms derived from the credential, and corroboration only in aggregate per Anchor.

  2. Should Transparency Services for Conduct Statements be public, permissioned, or operated by Anchors?

  3. Which conduct fields are useful enough to standardize, and which should stay implementation-specific?

  4. What minimum set of recipient categories is useful without becoming a fingerprint?

  5. What should happen to a collector's endorsement when its logged conduct and a site's corroborating statements disagree?

9. Security Considerations

Tier 0 statements are self-serving: a collector can log favorable statements that are false. The tiers let reliance grow only as independent evidence is added.

Logs can leak metadata. Site references MUST be keyed so that the log does not reveal which sites a collector visits, and registration timing can itself correlate collectors with sites. Batching and delayed registration reduce this.

An intermediary that signs witness statements is usually paid by the collector, and some intermediaries also sell help in getting past site blocks. A witness may therefore have an interest in attesting that a refusal was confined to one path, or in understating rates. A witness statement is only as credible as its signer, so sites choose which witnesses to accept, and the site's own corroboration, where available, carries more weight.

An Anchor and a collector can collude. Logged evidence makes collusion detectable after the fact but does not prevent it. Sites choose which Anchors to accept.

Attributes and timing patterns can fingerprint collectors despite anonymous credentials (R5).

10. Privacy Considerations

The model is intended to protect the identity of collectors and recipients from sites, and the identity of sites from public logs. An Anchor at Tier 2 holds identity information and becomes a sensitive party: a record that a given recipient buys data collected by a given operator is exactly what the model hides from sites, and held in one place it is a target for breach, subpoena, or sale. Anchors SHOULD retain only what an appraisal needs and delete it after a stated period, SHOULD verify a recipient's category without retaining the recipient's identity where the category can be established that way, and SHOULD publish their retention and disclosure policies.

11. IANA Considerations

This document has no IANA actions.

12. References

12.1. Normative References

[RFC2119]
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <https://www.rfc-editor.org/info/rfc2119>.
[RFC8174]
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, , <https://www.rfc-editor.org/info/rfc8174>.
[RFC9309]
Koster, M., Illyes, G., Zeller, H., and L. Sassman, "Robots Exclusion Protocol", RFC 9309, DOI 10.17487/RFC9309, , <https://www.rfc-editor.org/info/rfc9309>.

12.2. Informative References

[I-D.farzdusa-webbot-datacollection]
Farzaneh, Levy, J., and S. McKenna, "Best Practices for Responsible Web Data Collection", Work in Progress, Internet-Draft, draft-farzdusa-webbot-datacollection-01, , <https://datatracker.ietf.org/doc/html/draft-farzdusa-webbot-datacollection-01>.
[I-D.gilda-audit-agent-audit-record]
Gilda, S., "An Audit Record Format for AI Agent Authorization Decisions", Work in Progress, Internet-Draft, draft-gilda-audit-agent-audit-record-03, , <https://datatracker.ietf.org/doc/html/draft-gilda-audit-agent-audit-record-03>.
[I-D.ietf-webbotauth-httpsig-protocol]
Meunier, T. and S. Major, "HTTP Message Signatures for automated traffic", Work in Progress, Internet-Draft, draft-ietf-webbotauth-httpsig-protocol-01, , <https://datatracker.ietf.org/doc/html/draft-ietf-webbotauth-httpsig-protocol-01>.
[I-D.illyes-webbotauth-cbcp]
Illyes, G., Kühlewind, M., and K. Aj, "Crawler best practices", Work in Progress, Internet-Draft, draft-illyes-webbotauth-cbcp-01, , <https://datatracker.ietf.org/doc/html/draft-illyes-webbotauth-cbcp-01>.
[I-D.jms-mole-architecture]
Schlesinger, S., Jackson, D., and T. Meunier, "Moderation of unLinkable Endorsements (MOLE) Architecture", Work in Progress, Internet-Draft, draft-jms-mole-architecture-01, , <https://datatracker.ietf.org/doc/html/draft-jms-mole-architecture-01>.
[I-D.rescorla-anonymous-webbotauth]
Rescorla, E. and R. Barnes, "Anonymous Bot Authentication: Authorization and Rate Limiting for Web Agents", Work in Progress, Internet-Draft, draft-rescorla-anonymous-webbotauth-01, , <https://datatracker.ietf.org/doc/html/draft-rescorla-anonymous-webbotauth-01>.
[RFC9334]
Birkholz, H., Thaler, D., Richardson, M., Smith, N., and W. Pan, "Remote ATtestation procedureS (RATS) Architecture", RFC 9334, DOI 10.17487/RFC9334, , <https://www.rfc-editor.org/info/rfc9334>.
[RFC9943]
Birkholz, H., Delignat-Lavaud, A., Fournet, C., Deshpande, Y., and S. Lasker, "An Architecture for Trustworthy and Transparent Digital Supply Chains", RFC 9943, DOI 10.17487/RFC9943, , <https://www.rfc-editor.org/info/rfc9943>.

Author's Address

Sarah McKenna
Independent