Published study

Current study

RSR 002Status:PUBLISHED

AI Crawler Controls in 2026

We checked robots.txt and llms.txt across the top 100 companies in the 2026 Fortune 500 to measure how their public controls treat AI search/discovery, training or broader AI use, and user-directed retrieval.

Only 8 of the 100 published a named rule for at least one measured search or training control. A real llms.txt text file appeared on 12. Partial-path restrictions were much more common than root-level restrictions, largely because general crawler policy can apply even when an AI agent is never named.

READ THE AI CRAWLER CONTROLS STUDY

  1. METHODOLOGY LOCKED
  2. IN PROGRESS
  3. PUBLISHED— current stage
Where the RSR 002 crawler-controls study sits in its own protocol: the methodology is locked, collection is complete, and the study and its dataset are published.

01

The State of AI Search Visibility 2026

The separate State benchmark remains in methodology/in-progress state. Its locked v1.2 method uses 32 prompts across eight topic strata and 176 final-study observations. Final collection is still deferred, so that page publishes the method, measurement rules and limitations rather than findings.

SEE THE LOCKED STATE STUDY METHODOLOGY

02

What We Research

The work starts with the sources and page types that keep showing up around commercial questions. From there, we compare citation overlap across AI systems and look at cases where a brand is mentioned without its own site being cited.

We separate buyer-intent prompts from informational ones, then repeat a smaller commercial subset to see how stable the recommendations and source patterns are. We can study those patterns even though we do not have access to the ranking systems behind them.

  1. sources and page types
  2. AI systems
  3. citation overlap
  4. rerun the same set
What We Research

03

Research Is Not PROOF With More Rows

PROOF follows specific DankRank builds, baselines and experiments. Research looks across a larger sample and asks a broader question.

The DankRank launch baseline can tell us what happened to one new domain. I won't turn it into an industry benchmark simply because the chart looks interesting. Keeping those evidence types separate matters.

PROOF

builds, baselines and experiments

Research

a larger sample

Research Is Not PROOF With More Rows

04

The Methodology Stays Visible

Every study should tell you enough to judge the finding:

  • sample size
  • prompt design
  • engines observed
  • dates
  • repeat frequency
  • what counted as a citation, mention or recommendation
  • competitor or brand selection
  • known exclusions
  • limitations
  • material methodology changes

If a later version changes the method, I’ll document the change.

the methodology

the finding

The Methodology Stays Visible

05

Market Share Requires Different Evidence

A visibility study can tell us about the prompt set we observed. Questions about total consumer use, influence on commercial decisions or the revenue created by an AI citation need different evidence.

I keep the conclusion inside the boundaries of the data.

the prompt set we observed

requires different evidence

  • total consumer use
  • commercial decisions
  • revenue
Market Share Requires Different Evidence

06

Research Should Be Useful Outside DankRank

The best outcome isn't a report that says "hire DankRank for your AI SEO." It's a dataset or finding another search practitioner can challenge, repeat or use to ask a better question.

Commercial “money” pages can link to the research when it supports a claim. The research itself shouldn't need a sales pitch to be worth reading.

For practical measurement, use How to Measure AI Search Visibility.

a dataset or finding

  • challenge
  • repeat
  • ask a better question
Research Should Be Useful Outside DankRank

Related resources

Practical guides for measuring AI search visibility, understanding AI search systems and managing crawler discovery.

Explore all resources