AI Crawler Controls in 2026
We checked robots.txt and llms.txt across the top 100 companies in the 2026 Fortune 500 to measure how their public controls treat AI search/discovery, training or broader AI use, and user-directed retrieval.
Only 8 of the 100 published a named rule for at least one measured search or training control. A real llms.txt text file appeared on 12. Partial-path restrictions were much more common than root-level restrictions, largely because general crawler policy can apply even when an AI agent is never named.