AI Crawler Controls in 2026: What the Top 100 Fortune 500 Companies Publish

A lot of advice about AI crawler access begins with a block of robots.txt syntax.

Typical implementation advice is explicit: name GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, then add Google-Extended if that use case matters to you.

The implementation looks explicit because the example is explicit. Most of the companies we checked were not publishing policy that way.

Only 5 had a named group for at least one of the three search/discovery tokens. Eight had a named group for at least one of the training or broader-AI-use controls. Across the combined six-token headline family, the total was still 8 companies because those sets overlap.

The effective rule often came from User-agent: * instead.

That difference matters in a real audit. A site can have no named GPTBot group and still restrict GPTBot through a wildcard group. Calling that “no AI crawler policy” would miss the rule that actually applies. Calling it an explicit GPTBot decision would overstate what the file says.

I separate two questions:

What rule applies to the agent?

Did the site name that agent specifically?

They are not interchangeable.

Our AI Crawlers, llms.txt & Machine Discovery guide handles the implementation side. This study has a different job: measure what a defined sample of large U.S. companies actually published.

The strongest declared restriction in our classification is a root-path restriction.

We found one company with a root restriction affecting at least one measured search/discovery token and three with a root restriction affecting at least one training/broader-AI-use token. The counts are too small for sweeping claims, so we report them descriptively.

llms.txt gets a lot of attention relative to how often it appeared in this sample.

We found a real text file at 12 of the 100 governed corporate origins. Eleven matched the proposal structure used by the study. One was text, but did not meet that structural definition.

Fifty-nine were absent. The remaining states stayed separate: 11 denied, 14 temporarily unavailable and 4 HTML or site fallbacks.

An HTTP 200 at /llms.txt was not enough to count as adoption.

Walmart returned a bot-interstitial HTML page at that path, so we classified it as HTML_OR_SITE_FALLBACK. CVS Health returned a text file whose first non-empty line was an H1 and passed the structural test. Cisco published something that looked close to the proposal, including an H1, an H2 section and Markdown links, but the first non-empty line was a literal URL before the H1. Under the rule locked before collection, Cisco was PRESENT_NONCONFORMING_TEXT.

I like that Cisco example because it makes the classification rule visible. We did not decide after seeing the files which ones felt close enough to count.

The llms.txt proposal was modified to v2 in August 2026. We froze the proposal snapshot for this study before measuring the 100 companies, so the definition could not move halfway through collection.

And no, 12/100 is not evidence that llms.txt improves AI visibility. We did not test that question here.

The study reinforced one practical rule for me: do not start an AI crawler audit with a list of bot names and stop when the names are absent.

Start with the decision the site owner is trying to make. Search discovery, model training and user-directed retrieval do not automatically have the same answer.

Then fetch the actual robots.txt response. Inspect the body, resolve the matching group for each relevant token, and keep named-versus-wildcard provenance alongside the effective result. Root and partial-path restrictions belong in separate fields. An inaccessible or malformed response should remain inaccessible or malformed until there is evidence to classify it differently.

Inspect /llms.txt the same way. A 200 is not the classification. The body is.

Only after that would I move into the separate question of whether the brand is appearing, cited or recommended in AI answers. Our How to Measure AI Search Visibility guide covers that measurement layer.

Crawler controls and observed visibility can affect the same strategy without being the same evidence. Our AI search citation study is a separate example of why citations, mentions and recommendations need their own observation layer too.

A public robots.txt rule tells us what the site declared under the parsing rules used in the study. It does not prove that a crawler complied with that rule.

The study does not establish network access or blocking from robots.txt alone. It does not show that llms.txt improves citations, rankings, traffic or recommendations. It does not show that a missing named AI-agent group is a deliberate decision to allow AI use.

The top 100 companies in the 2026 Fortune 500 are a reproducible sample of very large U.S. companies. They are not a proxy for all websites, all industries or smaller businesses.

User-directed retrieval also needs care. The vendor documentation does not describe every user-directed agent as behaving like an automatic crawler. We retained those tokens in the dataset but kept them outside the main search-versus-training headline.

I would rather leave the conclusion narrower than the evidence than turn a useful AI crawler study into a claim it cannot support.