<?xml version="1.0" encoding="UTF-8"?>
<rfc xmlns:xi="http://www.w3.org/2001/XInclude" ipr="trust200902" docName="draft-mckenna-webbotauth-collection-attestation-00" category="info" submissionType="IETF" consensus="true" tocInclude="true" sortRefs="true" symRefs="true" version="3">
<front>
<title abbrev="Attested Collection">Attested Conduct for Anonymous Web Data Collection: Use Cases, Requirements, and a Tiered Model</title>
<seriesInfo name="Internet-Draft" value="draft-mckenna-webbotauth-collection-attestation-00"/>
<author fullname="Sarah McKenna"><organization>Independent</organization><address><email>mckenna.sarah+ietf@gmail.com</email></address></author>
<date/>
<area>Security</area>
<keyword>web bot auth</keyword><keyword>anonymous credentials</keyword><keyword>data collection</keyword><keyword>transparency</keyword><keyword>attestation</keyword>
<abstract><t>Anonymous Bot Authentication lets a web site learn that an automated client holds a credential from an Anchor whose policies it accepts, without learning which client it is. This document describes web data collection use cases that need that property, and what they require of the evidence an Anchor evaluates, of what a site can verify about conduct it cannot attribute, and of what a buyer of collected data needs to see. It proposes a tiered model. Collectors register signed statements of their own conduct in a SCITT Transparency Service; the site, its security intermediary, or a network intermediary may corroborate them; and an Anchor acting as a certifying body may appraise them. Coarse, unlinkable attributes about the data's eventual recipient let a site restrict use by category of recipient without learning who the recipient is.</t></abstract>
</front>
<middle>
<section anchor="introduction"><name>Introduction</name>
<t>A large share of automated web traffic is data collection: crawlers, monitors, and research tools that retrieve public pages for indexing, archiving, price and availability tracking, financial research, academic study, and consumer protection. Sites have good reasons to manage this traffic, and the IETF is developing two ways to do so: identified bots that sign their requests <xref target="I-D.ietf-webbotauth-httpsig-protocol"/>, and Anonymous Bot Authentication (ABA) <xref target="I-D.rescorla-anonymous-webbotauth"/>, in which a bot presents an anonymous credential issued under the policies of an Anchor.</t>
<t>Many legitimate collectors cannot use the identified path. A financial firm that reveals which retailers it monitors reveals its research. A consumer protection monitor that is identified can be served different content or blocked. ABA addresses this, and its authors note the tension directly: abuse controls that depend on knowing the identity of the abusive party do not work with anonymous authentication.</t>
<t>ABA specifies how a credential is issued and presented. It leaves open three questions this document takes up: what evidence an Anchor relies on when it decides that a collector complies with its policies, how parties who do not trust the collector can check that evidence, and what a site or a data buyer can learn about conduct it cannot attribute.</t>
<t>This document defines no new protocol. It shows how existing pieces (the SCITT architecture <xref target="RFC9943"/>, the RATS roles <xref target="RFC9334"/>, and ABA with MoLE <xref target="I-D.jms-mole-architecture"/>) fit this use case, and where gaps remain.</t>
<section anchor="requirements-language"><name>Requirements Language</name>
<t>The key words "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>", "<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL NOT</bcp14>", "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>", "<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>", "<bcp14>MAY</bcp14>", and "<bcp14>OPTIONAL</bcp14>" in this document are to be interpreted as described in BCP 14 <xref target="RFC2119"/> <xref target="RFC8174"/> when, and only when, they appear in all capitals, as shown here.</t>
</section>
</section>
<section anchor="terminology"><name>Terminology</name>
<dl newline="true"><dt>Collector:</dt><dd><t>An automated client that retrieves web content in order to produce data for its operator or for a recipient.</t></dd><dt>Recipient:</dt><dd><t>The party that ultimately receives the collected data, which may be the collector's operator or a downstream buyer.</t></dd><dt>Site:</dt><dd><t>The web site whose content is collected, together with any security intermediary (for example a bot management or WAF provider) acting for it.</t></dd><dt>Anchor:</dt><dd><t>As in <xref target="I-D.rescorla-anonymous-webbotauth"/>: a party that evaluates whether a collector complies with its policies and issues endorsements. In RATS terms <xref target="RFC9334"/>, an Anchor appraising evidence about a collector acts as a Verifier.</t></dd><dt>Conduct Statement:</dt><dd><t>A signed statement by a collector describing its own behavior toward a site over a stated interval.</t></dd><dt>Transparency Service:</dt><dd><t>As defined in the SCITT architecture <xref target="RFC9943"/>: an append-only, verifiable log that registers signed statements and returns receipts.</t></dd></dl>
</section>
<section anchor="use-cases"><name>Use Cases</name>

<section anchor="financial-research"><name>Financial Research</name>
<t>Investment firms buy data on prices, inventory, job postings, and product listings. Their compliance teams increasingly ask for evidence of how the data was collected: whether robots exclusion <xref target="RFC9309"/> and other stated preferences were honored, whether request rates were reasonable, and whether the data was collected under the terms the vendor claims. The firms also cannot let sites learn which companies they watch.</t>
</section>
<section anchor="price-comparison-and-consumer-protection"><name>Price Comparison and Consumer Protection</name>
<t><xref target="I-D.rescorla-anonymous-webbotauth"/> lists shopping sites blocking price comparison bots, and employment or housing sites blocking bots that monitor for discrimination, among the harms of identity-only schemes. These collectors have to show good conduct because they cannot show who they are.</t>
</section>
<section anchor="archiving-and-research"><name>Archiving and Research</name>
<t>Archives and academic projects collect at scale for public-interest purposes. They are usually willing to be identified, and a verifiable record of conduct helps them when a site questions their behavior.</t>
</section>
</section>
<section anchor="requirements"><name>Requirements</name>
<dl newline="true"><dt>R1 (Unlinkability):</dt><dd><t>A site <bcp14>MUST NOT</bcp14> be able to link a collector's requests to a specific collector or recipient beyond what the credential scheme in use allows.</t></dd><dt>R2 (Verifiable conduct):</dt><dd><t>The behavior that matters to sites (request rate, backoff when a site slows or signals overload, adherence to robots exclusion and other published preferences) <bcp14>SHOULD</bcp14> be recorded in a form that a party other than the collector can check.</t></dd><dt>R3 (Non-equivocation):</dt><dd><t>A collector <bcp14>MUST NOT</bcp14> be able to present one record of its conduct to a site or Anchor and a different record to a recipient.</t></dd><dt>R4 (Progressive assurance):</dt><dd><t>The model <bcp14>SHOULD</bcp14> work without any certifying body, and <bcp14>SHOULD</bcp14> let stronger assurance be added later without changing what collectors record.</t></dd><dt>R5 (Minimal attributes):</dt><dd><t>Attributes disclosed to sites <bcp14>MUST</bcp14> be coarse. As participants noted when Web Bot Auth discussed ABA, fine-grained attributes become fingerprints that defeat anonymity.</t></dd><dt>R6 (Recipient restrictions):</dt><dd><t>A site <bcp14>SHOULD</bcp14> be able to express that certain categories of recipient may not receive data collected from it, and a collector <bcp14>SHOULD</bcp14> be able to prove membership in a permitted category without revealing the recipient.</t></dd><dt>R7 (Accountability without identity):</dt><dd><t>Misconduct <bcp14>SHOULD</bcp14> have consequences, chiefly loss of endorsement at the Anchor, without requiring that the site learn who misbehaved.</t></dd><dt>R8 (Collector-governed rates):</dt><dd><t>A well-behaved collector <bcp14>SHOULD</bcp14> set and enforce its own request rate for each site, and record the basis for it. Sites keep full control of their access policy. The aim is to give them a reason to rely on attested self-governance instead of limits per client, which can be used to discriminate among collectors and are not tied to the site's actual load. A site <bcp14>MAY</bcp14> publish capacity hints that apply equally to all automated clients. As with TCP congestion control, collectors need not coordinate: if each backs off when a site slows, their combined load falls as the site's load rises.</t></dd></dl>
</section>
<section anchor="a-tiered-model"><name>A Tiered Model</name>
<t>The model has three tiers. Each adds a party and a stronger claim. A collector at any tier records the same Conduct Statements.</t>
<section anchor="tier-0-self-attested-logged-conduct"><name>Tier 0: Self-Attested, Logged Conduct</name>
<t>The collector produces Conduct Statements automatically from its own operation. A statement covers one site over one interval and contains:</t>
<ul><li><t>the interval;</t></li><li><t>a keyed reference to the site, so that the log does not disclose which sites the collector visits;</t></li><li><t>request counts and the maximum and mean request rate;</t></li><li><t>the rate ceiling and per-interval budget the collector set for itself, and the inputs it used to set them (see <xref target="collector-governed-rates"/>);</t></li><li><t>evidence of backoff: response-time and error-rate observations, how each slowdown was classified, and the collector's corresponding rate changes;</t></li><li><t>a digest of the robots exclusion file and of any other preference signals in force during the interval, and whether they were followed;</t></li><li><t>a digest of the extraction configuration and software version used.</t></li></ul>
<t>The collector signs each statement and registers it with a Transparency Service <xref target="RFC9943"/>. The receipt proves that the statement existed at that time, and the collector cannot later present a different one (R3).</t>
<t>Keyed site references are not enough on their own. A long-lived signing key, or software and configuration digests repeated in every statement, would let anyone reading the log link all of one collector's statements and see its whole footprint. Collectors <bcp14>SHOULD</bcp14> therefore sign with keys that are not reused across sites or epochs, for example keys bound to the credential or derived per site and epoch, and <bcp14>SHOULD</bcp14> salt or coarsen digests in the log, disclosing exact values only to a verifier that needs them.</t>
<t>Tier 0 proves what the collector said and when, not that it is true. It is cheap to adopt, it creates a consistent record that later tiers can check, and it lets a recipient check a vendor's claims against a log the vendor cannot rewrite.</t>
<section anchor="collector-governed-rates"><name>Collector-Governed Rates</name>
<t>Under R8 the collector, not the site, sets its rate:</t>
<ol><li><t>It estimates the site's normal traffic and capacity from available signals: third-party traffic estimates, observed response times, error rates and page weight, time-of-day patterns, and any capacity hint the site publishes for all automated clients.</t></li><li><t>It sets its rate ceiling to a small fraction of that estimate, and keeps its own request budget per site and interval.</t></li><li><t>It monitors response times continuously against the site's baseline.</t></li><li><t>It separates site slowdowns from slowdowns in its own network path, such as a proxy vendor, by comparing measurements across several independent egress paths, and where available consults the site's status page or a third-party uptime monitor. A slowdown seen on most paths is treated as the site's; one confined to a single path or vendor is treated as the path's.</t></li><li><t>It separates overload signals from path-specific refusals. An HTTP 429 or 503, a Retry-After header, or rising errors seen across several paths indicate that the site is under load. A refusal confined to one path, such as a 429 on the first request from an address the site or its bot-management service has blocked, while other paths and ordinary browsers reach the site normally, reflects the reputation of that path rather than the site's load.</t></li><li><t>It reduces its rate on a site slowdown or overload signal, honors any Retry-After, and recovers gradually once the site returns to baseline. On a path-specific slowdown or refusal it may move requests to another path. Rerouting changes the path, never the rate: the collector's total rate to the site stays within its ceiling.</t></li></ol>
<t>The Conduct Statement records each of these decisions and the measurements behind them, including each reroute and the evidence that the refusal was confined to one path. A verifier can then judge whether the collector backed off and whether its ceiling and classifications had a reasonable basis. Estimation methods are left to implementations; recording their inputs and outputs is what makes them checkable.</t>
</section>
</section>
<section anchor="tier-1-corroborated-conduct"><name>Tier 1: Corroborated Conduct</name>
<t>A site, or a security intermediary acting for it, observes requests that present a given credential. ABA lets the site apply budgets and rate limits per credential; under R8 these are a backstop, not the main control. In this tier the site also signs a statement of what it observed for that credential over an interval (request volume and rate, and any limits it applied), and either returns it to the collector or registers it in a Transparency Service.</t>
<t>Matching the site's observations against the collector's Conduct Statements removes the main weakness of Tier 0: the collector is no longer the only source of evidence about its own behavior. How a site statement is bound to a collector's statements without creating a linkable identifier is an open question (see <xref target="open-questions"/>).</t>
<section anchor="intermediary-witnesses"><name>Intermediary Witnesses</name>
<t>Corroboration need not come from the site. A network intermediary that carries the collector's traffic, such as a proxy network, VPN or egress gateway, scraping-infrastructure provider, or cloud network service, sees that traffic directly, and the collector cannot alter what it sees. Such an intermediary can sign a witness statement for a keyed site reference and interval.</t>
<t>What it can attest to depends on what it can see. A proxy that only tunnels encrypted connections sees the destination host, timing, and volume: it can attest to connection and request rates and how they changed, but not to response statuses or headers. A service that terminates TLS for the collector, such as an unblocking API or a hosted browser, can also attest to the statuses, response times, and Retry-After values it relayed, and to how the collector's rate changed afterward. When a collector uses several paths or intermediaries, each records which of its paths received refusals or slowdowns, so a verifier can confirm that a reroute followed a refusal confined to one path and did not raise the collector's total rate to the site.</t>
<t>Witness statements use the collector's keyed site references, so they reveal nothing about which sites were visited beyond what the intermediary already knows, and they <bcp14>SHOULD</bcp14> use signing keys that do not link the collector across sites. They can be registered in a Transparency Service, matched against the collector's Conduct Statements, combined across independent intermediaries so that none is trusted alone, and shown to Anchors, auditors, or data buyers. They add evidence from a party other than the collector without any change at the site, but they supplement site corroboration rather than replace it. Each site and each Anchor decides whether to accept witness statements, and from which intermediaries.</t>
</section>
</section>
<section anchor="tier-2-certified-conduct"><name>Tier 2: Certified Conduct</name>
<t>An Anchor, acting as a certifying body, appraises the collector's Conduct Statements and any corroborating statements against its policies, in the sense of a RATS Verifier <xref target="RFC9334"/>. It may also verify the identity of the collector's operator and of the recipient, and keeps both to itself. On a favorable appraisal it issues an endorsement under <xref target="I-D.rescorla-anonymous-webbotauth"/>.</t>
<t>Because the Anchor sees identity and the site sees only the credential, identity checks can be strict without weakening R1. An Anchor that endorses a collector whose logged conduct contradicts its policies can be held to account, because the evidence it relied on is in a log it does not control.</t>
</section>
</section>
<section anchor="recipient-categories"><name>Recipient Categories</name>
<t>Some sites object less to collection itself than to who receives the data: for example, direct competitors, or automated pricing systems in the same product category. Competition law in several jurisdictions is also concerned with the exchange of commercially sensitive information among competitors, including through intermediaries and algorithms. This document takes no position on any law.</t>
<t>Recipient categories are not a general tool for sites to choose who may read public data; used that way, they would bring back the discrimination R8 is meant to avoid. They <bcp14>SHOULD</bcp14> be limited to restrictions with a legal basis, such as competitors and pricing systems in the same product category, and <bcp14>SHOULD NOT</bcp14> be used to exclude journalists, researchers, archives, or consumer protection monitors.</t>
<t>A Tier 2 Anchor can verify a recipient's category and include it as a coarse attribute of the endorsement, for example "financial research" or "not a seller in category X". A site can then state, alongside its other preferences, which categories may not receive its data, and the collector proves membership in a permitted category without revealing the recipient. Attributes <bcp14>MUST</bcp14> be few and broad (R5).</t>
</section>
<section anchor="relationship-to-other-work"><name>Relationship to Other Work</name>
<ul><li><t><xref target="I-D.rescorla-anonymous-webbotauth"/> and <xref target="I-D.jms-mole-architecture"/> define how endorsements and credentials are issued and presented. This document describes evidence an Anchor can rely on, and does not change those mechanisms.</t></li><li><t><xref target="I-D.ietf-webbotauth-httpsig-protocol"/> covers identified collectors. Conduct Statements apply to them as well.</t></li><li><t><xref target="RFC9943"/> provides the transparency log. Conduct Statements are SCITT Signed Statements.</t></li><li><t><xref target="RFC9334"/> provides the Verifier and Relying Party roles used in Tier 2.</t></li><li><t><xref target="I-D.gilda-audit-agent-audit-record"/> defines per-call audit records for agent actions. Conduct Statements are aggregate and collection-oriented, and need an unlinkable mode that per-call records with identified principals do not provide.</t></li><li><t><xref target="I-D.illyes-webbotauth-cbcp"/> and <xref target="I-D.farzdusa-webbot-datacollection"/> describe the conduct that Conduct Statements would record.</t></li></ul>
</section>
<section anchor="open-questions"><name>Open Questions</name>
<ol><li><t>How can a site's corroborating statement be matched to a collector's Conduct Statements without creating a linkable identifier? Candidate approaches include per-interval pseudonyms derived from the credential, and corroboration only in aggregate per Anchor.</t></li><li><t>Should Transparency Services for Conduct Statements be public, permissioned, or operated by Anchors?</t></li><li><t>Which conduct fields are useful enough to standardize, and which should stay implementation-specific?</t></li><li><t>What minimum set of recipient categories is useful without becoming a fingerprint?</t></li><li><t>What should happen to a collector's endorsement when its logged conduct and a site's corroborating statements disagree?</t></li></ol>
</section>
<section anchor="security-considerations"><name>Security Considerations</name>
<t>Tier 0 statements are self-serving: a collector can log favorable statements that are false. The tiers let reliance grow only as independent evidence is added.</t>
<t>Logs can leak metadata. Site references <bcp14>MUST</bcp14> be keyed so that the log does not reveal which sites a collector visits, and registration timing can itself correlate collectors with sites. Batching and delayed registration reduce this.</t>
<t>An intermediary that signs witness statements is usually paid by the collector, and some intermediaries also sell help in getting past site blocks. A witness may therefore have an interest in attesting that a refusal was confined to one path, or in understating rates. A witness statement is only as credible as its signer, so sites choose which witnesses to accept, and the site's own corroboration, where available, carries more weight.</t>
<t>An Anchor and a collector can collude. Logged evidence makes collusion detectable after the fact but does not prevent it. Sites choose which Anchors to accept.</t>
<t>Attributes and timing patterns can fingerprint collectors despite anonymous credentials (R5).</t>
</section>
<section anchor="privacy-considerations"><name>Privacy Considerations</name>
<t>The model is intended to protect the identity of collectors and recipients from sites, and the identity of sites from public logs. An Anchor at Tier 2 holds identity information and becomes a sensitive party: a record that a given recipient buys data collected by a given operator is exactly what the model hides from sites, and held in one place it is a target for breach, subpoena, or sale. Anchors <bcp14>SHOULD</bcp14> retain only what an appraisal needs and delete it after a stated period, <bcp14>SHOULD</bcp14> verify a recipient's category without retaining the recipient's identity where the category can be established that way, and <bcp14>SHOULD</bcp14> publish their retention and disclosure policies.</t>
</section>
<section anchor="iana-considerations"><name>IANA Considerations</name>
<t>This document has no IANA actions.</t>
</section>
</middle>
<back>
<references><name>References</name>
<references><name>Normative References</name><xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9309.xml"/></references>
<references><name>Informative References</name><xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9334.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9943.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.rescorla-anonymous-webbotauth.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.jms-mole-architecture.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.ietf-webbotauth-httpsig-protocol.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.illyes-webbotauth-cbcp.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.farzdusa-webbot-datacollection.xml"/><xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.gilda-audit-agent-audit-record.xml"/></references>
</references>

</back>
</rfc>
