AI training preference versus retrieval access

Definition & limits

Training permission, retrieval access and action authorization are three independent relationships. A publisher can permit a crawler to fetch a page while withholding permission to use it to train a model. A retrieval system may show the page as evidence without certifying its truth. And a user may ask an assistant to read the page without authorizing the assistant to follow instructions embedded in it.

The October 30 IETF Internet-Draft proposes a robots.txt group keyed by User-Agent: __usage_ai-training with normal Allow and Disallow path rules. It introduces a purpose-specific declared preference, not a new HTTP authentication or crawler-ban mechanism. The draft states that an ordinary wildcard user-agent group does not stand in for the special token. Its absence is not a specific expression of consent or refusal. The text remains an Internet-Draft rather than an adopted RFC, and provider implementations or private licenses may differ.

That publisher-to-provider declaration should also be distinguished from a developer's Gemini API log-sharing setting, which governs a different data flow, and the data-void search manipulation case, where untrusted retrieved material affects what an agent presents as fact. A robots.txt preference neither establishes the veracity of a fetched page nor grants the page authority to make the agent spend money, disclose private data or change instructions. Responsible reports specify which relationship and vendor implementation are actually being tested.

Examples

• The IETF __usage_ai-training Internet Draft expresses path-level training preferences without changing ordinary crawler rules.
• Search-grounded models may encounter poisoned source pages regardless of whether the publisher consents to model training.

Explicit mappings

Research using this topic (1)