The Web Is Splitting AI Access Into Training, Search, and User Fetches

The web is no longer asking one question about an AI crawler. It is asking three: is this system training a model, answering a search query, or fetching a page for a named user? Kavel’s October 2026 study examined robots.txt rules at 945 well-known websites and found that the distinctions are already appearing in production. Of the 746 sites whose files could be read, 193 blocked at least one of 14 AI crawlers. Sixty-nine blocked AI training crawlers while leaving every AI search crawler in the study open. Those figures do not measure what companies secretly do. They measure what sites request in a public control file, which is precisely why they are useful.

flowchart LR
  A[Website policy] --> B[robots.txt]
  B --> C[Training crawler]
  B --> D[Search crawler]
  B --> E[User fetcher]
  C --> F[Model data pipeline]
  D --> G[Cited answer and referral]
  E --> H[User-requested page]

Robots.txt has become a negotiation layer

The study reports a completion date of October 22, 2026 and presents the results as an October check. That is different from the date on which an individual website changed its policy, and it is different from a legal determination about permission. Kavel says the sample is its own list, not a statistical sample of the whole web. It also says 199 sites were excluded from the robots.txt figures because they returned bot protection, server errors, or could not be reached. A responsible reading therefore treats 25.9 percent as a property of the measured set, not a census of the internet.

What the October sample actually measured

The crawler breakdown is revealing. Kavel lists CCBot and Bytespider as training-oriented crawlers, ClaudeBot and GPTBot as training crawlers, Applebot-Extended as an AI-use control, and PerplexityBot, OAI-SearchBot, and Claude-SearchBot as search crawlers. It also separates user fetchers such as ChatGPT-User and Claude-User. The categories are not merely branding. They map to different publisher interests. A newspaper may object to training on its archive but want a search system to send readers to a current article. A commerce site may allow a user-requested page fetch while blocking bulk extraction. A single ‘allow AI’ switch cannot express that policy.

Training is not search is not a user request

RFC 9309 describes the Robots Exclusion Protocol and clarifies the scope of robots.txt. The file communicates a crawler policy; it does not create a cryptographic barrier. Kavel explicitly warns that robots.txt is a request, not a lock, and that its Cloudflare-based test requests cannot establish how a real crawler behaves from its production network. That limitation should be front and center. A robots file is closer to a public sign than a firewall rule. It is still important because it communicates intent, gives responsible operators a machine-readable instruction, and creates an observable record of how access expectations are changing.

Why publishers are making different choices

The most striking result is the difference between news and software. Kavel reports that 79.7 percent of its international news sample blocked at least one AI crawler, compared with 9.2 percent for SaaS and developer tools. Reference sites were also highly restrictive. The pattern makes economic sense. Publishers bear the cost of original reporting and may see search summaries replace visits. Software companies may prefer to be discoverable in an AI answer, or they may want their documentation available to developers. Neither choice is automatically correct. The policy depends on whether the owner values training exposure, referral traffic, direct product discovery, or protection against unmetered extraction.

A robots file is a signal, not a lock

The phrase ‘no to training, yes to AI search’ describes a new bargain but not a stable one. Search agents can send traffic, yet an answer engine may satisfy a query without a click. Training can create a durable representation of a publisher’s work, yet the publisher may gain no attribution or control over how the representation is used. User fetches create a narrower relationship: a person asks an assistant to retrieve a page now. Website owners need a way to distinguish those use cases operationally. Crawler identity is one signal; rate, request path, authentication, and referral behavior are others.

The crawler names are part of the policy

Website teams should stop treating robots.txt as a forgotten SEO file. It is now part of an AI access policy that should be versioned, reviewed, and tested. Record the date a crawler rule changed. Explain why GPTBot is allowed or blocked. Check whether the policy applies to all paths or only a content directory. Review whether a new vendor uses a separate user agent. Google’s crawling documentation and the public documentation for OpenAI, Anthropic, Perplexity, and Common Crawl show why names matter: different services describe different uses, and a site needs to map those descriptions to its own rights and goals.

Search visibility now has a technical price

A mature access policy also needs enforcement evidence. Cloudflare’s bot-learning material is useful here because it frames automated traffic as a behavior problem, not only a user-agent string. A crawler can claim an identity while arriving from an unexpected network, ignoring rate limits, or requesting pages in a pattern that no human search session would produce. Conversely, a legitimate assistant may be blocked by a generic anti-bot challenge. Owners should log user agent, source network, requested path, response status, rate, and referral when available. The point is not to build a perfect detector. It is to know whether the public policy matches observed traffic.

What website owners should log

The study’s monthly recheck is a valuable design choice because web policies drift. A company changes providers, launches a licensing deal, or adds a search feature. A robots file can be edited in minutes while a model-training contract lasts years. Monthly comparison can show whether a site added a crawler, narrowed a rule, or accidentally blocked an agent it intended to support. The same process can catch stale documentation. A vendor may rename a crawler or add a search-specific user agent, leaving an old rule ineffective. Policy maintenance is not glamorous, but it is now part of content operations.

The limits of the study

There is a fairness question in selective access. Large publishers can negotiate licenses and operate sophisticated bot controls. Small sites may rely on a simple robots file and lack the staff to monitor requests. If search systems prioritize content that is easy to fetch, a cautious small publisher could lose visibility even when its work is valuable. If AI companies ignore publisher signals, smaller creators may have no practical route to object. Standards work should therefore make access preferences easy to express and attribution easy to verify. The goal is not to freeze the web. It is to make the terms of machine access legible to both sides.

The next web contract will be more explicit

For AI builders, the distinction between a user fetch and training data should affect product architecture. A user-requested page can be fetched with a narrow purpose, limited retention, and a citation. A training pipeline may store, transform, and redistribute patterns over a much longer period. A company that treats both as ‘web access’ will create predictable conflict. The public crawler documentation is a starting point, not a complete consent system. Builders should honor robots rules, publish clear agent identities, give sites a way to correct misclassification, and avoid suggesting that a search result is a licensed endorsement.

Field notes for operators and builders

Robots.txt is easy to edit and easy to misunderstand. A rule for GPTBot does not automatically govern every OpenAI crawler, and a rule for one path may not express the owner’s policy for another. Website teams should maintain a crawler inventory rather than assume a familiar brand name covers every use.

The difference between a search fetch and a training crawl is partly temporal. A search request may need one page for one answer. Training collection may revisit the page, store it, transform it, and mix it with other material. Policy should account for retention and reuse, not only the first HTTP request.

Publishers often have different rights across their sites. A public article, a licensed photograph, a reader comment, and a paid report may share a domain but not a permission model. Path-specific directives and explicit APIs can express that difference better than a single global allow or disallow.

User agents are useful identifiers but weak authentication. A site that needs stronger certainty can use tokens, signed requests, or a licensed feed. Those mechanisms cost more to operate, which is why a public robots signal remains valuable for low-friction communication.

AI search creates a referral paradox. A cited answer can send a high-intent visitor, but a complete summary can reduce the reason to click. Publishers should measure qualified visits, subscription conversion, and citation placement instead of assuming that every answer is beneficial or harmful.

Bot protection can create accidental exclusion. A legitimate search agent may be challenged while a scraper rotates identities. Rate limits, behavior analysis, and documented escalation paths are more precise than a blanket block when the site wants machine discovery.

Kavel’s monthly check is a useful operational model for content teams. Store a snapshot, compare directives, test representative requests, and notify the owner when the effective policy changes. The same process can detect a deployment that accidentally replaced robots.txt with a landing page.

AI companies should publish crawler purpose and retention behavior in language a publisher can map to a decision. A vague statement that a bot helps AI is not enough. Owners need to know whether the request supports indexing, training, user retrieval, evaluation, or product telemetry.

Attribution is not permission, and permission is not compensation. A search engine can cite a page without licensing its archive. A training pipeline can respect a robots rule while still needing a commercial agreement. These concepts should not be collapsed into a single badge that gives users a false sense of settlement.

Web standards work should prioritize stable vocabulary. Training, search, user fetch, and agent action are different purposes. If each vendor invents a new user agent with no public mapping, website owners will either block everything or allow everything. Both outcomes create worse incentives.

Small publishers need templates and monitoring that do not require a legal department. A practical policy can state which public paths are available to search, which are excluded from training requests, and where licensing questions go. Clarity is more valuable than a policy so complex that nobody updates it.

Government and health sites deserve special care because machine access can expose sensitive context even when pages are public. A crawler policy cannot replace privacy controls, but it can reduce unnecessary collection of documents that were meant for human access at a particular moment.

AI search vendors should honor cache freshness. A page may be corrected, retracted, or updated after the crawler’s first visit. User answers that rely on a stale copy can damage the publisher and the person asking the question. Freshness metadata and visible retrieval dates are practical safeguards.

Website owners should test their own policy from outside the corporate network. Internal monitoring can miss a CDN rule, bot challenge, or regional difference. Kavel’s warning that a Cloudflare request is not proof of real crawler behavior applies in both directions: a successful test does not guarantee broad access.

Legal terms and robots rules should not contradict one another without an explanation. A site that disallows training in robots.txt but grants a broad license in terms may create confusion for partners and researchers. Coordinate the public files with the actual business policy.

For AI builders, respect should be measurable. Log when a request is blocked, record the reason, and avoid retrying indefinitely with alternate identities. A crawler that treats a refusal as an obstacle to route around is not behaving like a responsible search service.

The web’s new access layer will likely combine public signals with paid agreements. That can fund original work, but it may also concentrate visibility among publishers able to negotiate. Standards should preserve a useful free web while making large-scale commercial extraction accountable.

The useful result of the Kavel study is not a universal percentage. It is the evidence that site owners are already differentiating AI purposes. The industry should treat that differentiation as an invitation to build clearer tools, not as an inconvenience to be ignored.

Applied tests and deployment details

One practical mistake is to block a crawler by brand while forgetting a purpose-specific user agent. A publisher can believe it denied training while allowing a separate collection path. Vendor documentation should be checked against observed traffic, and the robots file should be reviewed whenever a product introduces a new bot.

Publishers also need a response to cached content. If an article is corrected or withdrawn, a search assistant should not continue presenting the old version without a date. A takedown and correction process for machine indexes will matter as much as the initial robots rule.

Licensing models may make selective access more attractive. A publisher could provide a structured feed for a search service, with attribution and freshness guarantees, while blocking uncontrolled archive collection. That arrangement is more work than a robots file but can align incentives better when the content has clear commercial value.

Content management systems should make the policy visible to editors. A reporter publishing a sensitive investigation may need a different default from a marketing team publishing a product page. If the control exists only in infrastructure, the editorial owner may not know what machines can do with the work.

AI companies should treat blocked requests as an opportunity to explain. A clear response can identify the crawler, purpose, and documentation page rather than forcing a site owner to guess. Good error handling reduces the temptation to retry with a different identity.

Robots rules do not solve access to private data. Authentication, authorization, and data minimization remain essential. A site must never publish a secret and rely on a crawler instruction to protect it. The public policy is for public access preferences, not a substitute for security.

Researchers using Common Crawl or similar archives should preserve the collection date and source policy. A dataset can contain pages collected before a rule changed. Downstream users need enough metadata to understand that historical presence is not proof of continuing permission.

The web’s machine-readable future will be healthier if refusal is treated as a legitimate outcome. Search quality cannot be measured only by the number of pages an agent can retrieve. It also depends on whether the sources it uses have a clear relationship with the people who created them.

A final operational check

The immediate opportunity is a better machine-readable declaration that combines purpose, freshness, attribution, and refusal. Robots.txt is a foundation, but the next layer should let a publisher state what an agent may do with a page after retrieval. That would turn a vague allow-or-block signal into a more useful web contract.

Additional verification scenarios

A publisher may want search access for a page but not automated action. An agent that can buy, book, submit, or post has a different risk from one that summarizes public text. Future access declarations should therefore distinguish retrieval from side effects. The current crawler debate is a first step toward a broader agent-permission model.

Freshness is also a quality issue for training. A model may learn an old price, policy, or medical instruction and repeat it after the page changes. Publishers cannot solve model staleness through robots.txt alone, but AI companies can publish update mechanisms and dates that help users understand the age of a source.

Citation links should remain available when a search agent uses a page. A citation that points to a blocked, paywalled, or changed article may not give the reader a practical way to verify the claim. Retrieval systems should preserve source title, date, and relevant version context rather than offering a bare URL.

Kavel’s sample includes finance, government, education, retail, reference, and technology sites. Different sectors face different harms. A news archive worries about substitution, a government site worries about stale public information, and a commerce site worries about automated extraction of inventory. Sector-specific policies are more useful than a single industry average.

AI crawler measurement should become a shared research practice. Independent checks can compare declared identity, network origin, request rate, and cache behavior. Publishing methodology allows websites and vendors to dispute a result with evidence rather than with competing assertions about intentions.

The policy file itself should be included in deployment review. A new CDN, redesign, or static-site build can overwrite it accidentally. Treating robots.txt as code—with ownership, tests, and change history—reduces that risk and makes the organization’s AI position durable.

Closing control notes

A site can also publish a contact route for crawler disputes. When a vendor believes a page should be accessible, the answer should be a documented policy conversation rather than repeated requests from new identities. That preserves the publisher’s ability to change its mind and gives the vendor a path to explain purpose, attribution, retention, freshness, and citation behavior. It also creates an accountable record for editors and engineers.

This record supports later review when policy, traffic, or vendor behavior changes.

Sources and evidence

The article distinguishes vendor descriptions, research observations, and editorial analysis. Primary and methodological references used near the claims above: