DankRank AI Crawler Controls Study 2026 - Public Research Dataset

Study page: /research/ai-crawler-controls-2026/
Sample: ranks 1-100 of the domestic U.S. 2026 Fortune 500
Final collection date: 2026-09-22
Default published denominator: N=100

What is included
- final_study_rows.csv: the governed 100-row final study dataset (72 fields)
- data_dictionary.csv: definitions for all 72 fields
- final_data_schema_v1.2_LOCKED.csv: locked final schema
- methodology_v1.2_LOCKED.md: locked methodology used for collection and calculation
- outputs/headline_metrics.csv: eight governed headline calculations
- outputs/cross_use_case_counts.csv: stored cross-use-case label distribution
- outputs/llms_classification_counts.csv: /llms.txt classification distribution
- outputs/token_policy_counts.csv: per-token policy distributions
- outputs/family_policy_counts.csv: search, training/broader-AI-use and user-directed family distributions
- outputs/endpoint_state_counts.csv: robots.txt and /llms.txt endpoint-state distributions
- outputs/company_calculation_trace.csv: company-level calculation trace for ranks 1-100
- outputs/calculation_summary.json: machine-readable calculation definitions and guardrails
- MANIFEST.sha256: SHA-256 for every published file in this package

Important interpretation boundaries
A public robots.txt rule is a declared control under the study's parsing rules. It does not prove crawler compliance or network access. The study does not establish that llms.txt improves citations, rankings, traffic or recommendations. Absence of a named AI-agent rule is not evidence of a deliberate allow decision. The Fortune 500 top-100 sample is not representative of all websites or industries. User-directed agents remain separate from the main search-versus-training headline comparison.

Reproducibility
All published percentages in the article use N=100. Unavailable, denied, malformed and ambiguous cases remain in the denominator unless a file explicitly states otherwise. Small-N findings with numerator below 5 are descriptive only.

The raw endpoint bodies and internal execution logs are not included in this public package. They were retained for governed QA but are not necessary to reproduce the published calculations from final_study_rows.csv.
