AI Crawlers, llms.txt & Machine Discovery

A blanket rule for "OpenAI" or "Anthropic" can hide meaningful differences between their crawlers. Before changing robots.txt, check the current provider documentation and understand which product or use case the user agent controls.

The names and purposes can change over time, so this is one of the pages we expect to maintain.

OpenAI documents OAI-SearchBot as the crawler associated with making public web content discoverable in ChatGPT search experiences. A publisher that wants its pages available to be surfaced and cited in ChatGPT search would work against that goal by blocking it.

Model-training access is a separate decision.

GPTBot has been used by OpenAI for model-related web crawling, according to OpenAI's crawler documentation. A GPTBot rule and an OAI-SearchBot rule control different access, so a site can make a deliberate search-visibility choice without treating every OpenAI user agent as one thing.

Anthropic documents separate robots for different purposes. ClaudeBot is associated with web content collection that can contribute to model development, while user-directed retrieval uses separate Claude user agents.

That lets publishers make more specific access choices than one "allow Claude" decision. Check Anthropic's current bot documentation before shipping rules because the exact user-agent set and product behaviour can change.

Perplexity documents PerplexityBot as its search crawler used to surface and link websites in Perplexity search results, and Perplexity-User for user-requested page access.

The company notes that the user-requested fetcher can behave differently from the crawler with respect to robots.txt. That is another reason to classify automated requests by purpose.

Google-Extended is a robots.txt product token rather than a separate HTTP crawler user agent. Google documents Google-Extended as a control over whether crawled content may be used for certain Gemini model-training and grounding uses, while leaving inclusion and ranking in Google Search unaffected.

That gives a site a separate choice about those Gemini uses without giving up normal Google Search visibility.

robots.txt tells compliant crawlers which URLs they may request. Google's documentation distinguishes crawler access from indexing controls, and a blocked URL can still be known through links or other signals in some contexts.

If the business wants a page out of search entirely, use the provider's documented indexing or removal controls rather than expecting a crawl block to do both jobs.

A site can allow a crawler in robots.txt and still block it at Cloudflare, Akamai, a CDN, a firewall or an application-level challenge. That creates a common debugging problem: the policy says allow, but the request receives a 403 or never gets past bot mitigation.

When crawler access matters, check the full request path. Server logs and provider-published verification methods are more useful than staring at the robots.txt file alone.

llms.txt is a proposed convention for publishing an LLM-friendly summary and a curated set of links, normally at /llms.txt. The proposal offers language models a cleaner map of important site information than parsing a complex website from scratch.

It remains a proposal rather than a universal web standard.

There is no basis for treating an llms.txt file as a guaranteed AI-search ranking or citation factor, and major AI-search products don't universally require it for eligibility.

Documentation-heavy sites or projects with machine-consumption use cases may still find it convenient. It can also expose a clean, curated set of canonical resources. The mistake is promoting implementation into proof of visibility.

The commonly discussed proposal uses llms.txt with the plural "llms." Search demand and informal references sometimes use llm.txt, which is one reason both terms appear.

If you implement the proposal, follow the current specification rather than the misspelled query variant.

I wouldn't start there by default. First make sure the important pages are crawlable, indexable where appropriate, internally linked and full of information worth retrieving.

A machine-friendly directory file can't repair weak content architecture. For some sites, testing llms.txt is reasonable. For others, it sits far below fixing a blocked crawler, an accidental noindex, a broken canonical or a site that barely explains what the company does.

/markdown-for-bots.md is a plain-text reference to DankRank, its service families and the canonical public pages behind them. It gives systems that can read Markdown another clean way to understand what the company is and where the primary resources live.

It doesn't replace robots.txt or llms.txt, and its presence doesn't mean an AI system will crawl, use, cite or rank the site.

READ MARKDOWN FOR BOTS →

When we're checking how AI systems can discover the site, we work through the actual controls:

Which AI-related user agents are requesting the site?Which ones matter to the business?What does the current robots.txt allow or block?Does the CDN/WAF permit the same access?Are important pages returning normal HTML to those requests?Are canonical and indexing directives consistent?Can the pages be reached through internal links?Is the information useful enough to retrieve once the crawler gets there?Is an llms.txt experiment worth running, or is it a distraction from a larger discovery problem?

I use that sequence to keep the technical work tied to an actual visibility question.

A crawler reaching the page doesn't guarantee retrieval. Retrieval doesn't guarantee citation. Citation doesn't guarantee recommendation.

Crawler access is the foundation. The broader client work lives under AI Search, while Organic Search owns the crawl, index and site-architecture foundation.

For the larger glossary, use the AI Search Field Guide.