OpenAI, Google, Microsoft and Anthropic Back One Shared robots.txt Signal to Opt Out of AI Training

If you run a fan site, a webcomic page or an art portfolio, opting out of AI training has meant keeping a growing list of crawler names in your robots.txt and hoping you did not miss one. That chore may be about to shrink.

One signal instead of a dozen rules

OpenAI, Google, Microsoft and Anthropic say they will respect a shared signal in robots.txt that tells AI companies a site does not want its content used for training generative models. Instead of writing separate rules for each company's crawler, a site owner states the preference once and every participating provider is supposed to honor it.

The idea comes out of the IETF's AI Preferences working group, the standards body discussion where this kind of web plumbing gets argued over. According to The Verge, the proposal was put forward by a standards lead at Anthropic.

When it actually starts working

The rollout is uneven, so do not delete your existing rules yet:

• Anthropic has already adopted it.
• OpenAI says it will implement it "in the coming months" and keep its current crawler controls in place, per its own announcement.
• Google and Microsoft plan to support it next year. Microsoft also promises separate switches for training and for grounding, and says sites will stay in Bing search either way.
• Meta voiced support but has not said it will honor the signal.

Why fans and artists should care

A lot of anime fandom lives on small, personal sites: fan art galleries, translation notes, episode trackers. Those owners rarely have time to audit crawler lists. A single, shared opt-out is a real improvement for them, even if it is still voluntary and only binds the companies that sign on.

The catch is that it only covers future training by participating companies, and it relies on them keeping their word. I would treat it as one more layer, not a lock.