How to Audit Pages for AI Retrieval

The audit applies to any public page a team wants AI search engines and assistants to fetch, quote or cite, on any CMS, with or without a CDN. Passing every gate makes a page retrievable, not cited. Whether AI answers use it depends on its content and its competition, which this audit does not measure.

What An AI Retrieval Audit Checks

An AI retrieval audit checks whether a page passes four gates in order, crawl permission for each AI crawler in robots.txt, the CDN or firewall rules in front of the site, content that is present in the raw HTML, and Google index eligibility for AI Overviews. The audit in this playbook adds a fifth step, proof in your logs, because the four gates describe what should happen and only logs show what did.

The split between retrieval and training is the base of the whole audit. OpenAI and Anthropic each run separate crawlers for search, for training and for fetches a person triggers. Perplexity separates its search crawler from its user fetcher. A robots.txt rule only affects the token it names, so blocking a training crawler does not block search, and allowing a search crawler does not opt a site in to training.

The gates also sit at different layers. A CDN can add its own robots.txt rules above the site's file, so the file in the repository is not always the file crawlers read. Google adds no extra requirements for its AI features, so the Google gate is ordinary index and snippet eligibility.

This audit is the access layer of AI search visibility. It covers whether crawlers can reach and read a page, not how to write for AI answers or how to measure them.

For how retrieval differs from classic indexing and which technical base SEO, AEO and GEO share, see the comparison of AEO, GEO and SEO in practice.

Which AI Crawlers Should You Allow?

OpenAI runs separate crawlers for separate jobs, so OAI-SearchBot decides whether pages can appear in ChatGPT search answers while GPTBot covers only model training and ChatGPT-User fetches pages when a person asks. For retrieval, allow the search and user-fetch tokens of each operator, and treat training tokens as a separate licensing decision.

The tokens that matter for a retrieval audit, by operator:

OpenAI says a site that opts out of OAI-SearchBot will not appear in ChatGPT search answers, though it can still show as a navigational link. Disallowing GPTBot opts content out of training and nothing else. ChatGPT-User is not used for automatic crawling or to decide search inclusion.

Anthropic follows the same split. Blocking Claude-User or Claude-SearchBot may reduce a site's visibility in Claude's user-directed search, while blocking ClaudeBot excludes future content from training. Anthropic's bots honor robots.txt and support the non-standard Crawl-delay directive.

Perplexity draws the line differently. PerplexityBot surfaces and links pages in Perplexity search results and is not used to crawl content for AI foundation models, and Perplexity recommends allowing it. For fetches a person starts, Perplexity-User generally ignores robots.txt rules.

Google-Extended works another way again. It is a robots.txt product token with no HTTP user agent of its own, so it never appears in logs. It controls whether content Google crawls may train future Gemini models and ground answers in Gemini Apps and the Vertex AI API. It does not affect inclusion in Google Search and is not a ranking signal.

Decision rule: Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot on every page you want cited, leave ChatGPT-User, Claude-User and Perplexity-User open, and decide GPTBot, ClaudeBot and Google-Extended on licensing grounds rather than visibility.

Robots.txt is a signal, not access control. OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says the same of Perplexity-User. A page that must stay out of AI tools needs a block at the CDN or firewall, which the next steps audit.

How To Audit Pages For AI Retrieval

To audit pages for AI retrieval, run five checks on a sample of priority URLs, robots.txt rules for each AI crawler token, CDN and firewall bot settings, raw HTML content, Google index eligibility and crawler hits in your logs, and fix failures in that order.

Gather these inputs before the first check:

• A sample of priority URLs, taken from the top pages in Google Search Console plus your key money pages
• The robots.txt file exactly as the live site serves it
• Login access to the CDN or firewall dashboard
• Server logs or CDN request data for the same URLs

Then run the checks and record a pass or fail for each:

• Robots.txt per token passes when every search and user-fetch token you want is allowed on the sample URLs.
• CDN and firewall rules pass when no bot policy, managed rule or WAF rule blocks those tokens.
• Raw HTML passes when main copy, headings, links and metadata appear without JavaScript.
• Google eligibility passes when URL Inspection shows each page indexed, with snippet controls sparing key passages.
• Log proof passes when each allowed crawler receives 200 responses on the sample URLs.

The order matters because a later check means nothing when an earlier gate fails. Clean raw HTML does not help a page whose CDN returns a 403 to OAI-SearchBot. A perfect index status does not help a page whose robots.txt disallows the AI search tokens.

Each step below follows the same pattern, what to check, the tool, the pass condition, the access needed and the limitation, so the results drop straight into one audit sheet.

How To Run A Crawler Access Audit

A crawler access audit has to cover the CDN as well as robots.txt, because Cloudflare blocks Training and Agent crawlers by default on ad-displaying pages of new domains from September 15, 2026 even when the site's own robots.txt allows them. Run the two checks on the same URL sample.

Read Robots.txt As The Edge Serves It

Start with the robots.txt the edge returns, not the file in the repository or CMS. Cloudflare's managed robots.txt, available on all plans under Security, Bots, adds directives that disallow known AI training and AI agent bots. Those lines sit above the site's existing rules in one response, or stand alone when the site has no file.

Then test each AI token against that file in Screaming Frog. Screaming Frog has presets for Googlebot, Bingbot and browsers but not for AI bots, so each AI token goes in as a custom user agent. The tool sets the HTTP request user agent and the robots user agent separately. Its custom robots.txt feature downloads, edits and tests the file with the selected agent and never changes the live file.

Google Search Console's robots.txt report adds Google's side. It lists the files Google found for the top 20 hosts of a Domain property, with fetch status and parsing issues. Errors stop a rule being used, while warnings do not.

Check CDN And Firewall Bot Rules

In Cloudflare AI Crawl Control, the Crawlers table shows each AI crawler's operator, category, request count, robots.txt violations and current action. Any crawler can be allowed, allowed with robots.txt still enforced, or blocked with a 403 or 402 response, and WAF rules can extend a block to more paths or user agents.

Cloudflare now groups bots by function, Search, Agent and Training, with a separate policy for each on every plan. Multi-purpose crawlers such as Googlebot, Applebot and Bingbot fall under both the Search and Training policies, so blocking Training blocks them even when Search is allowed. Owners can opt out of the new-domain default in Security settings.

A quick, reproducible check from any terminal compares the status codes the edge returns for different tokens:

# 1. Read robots.txt as the edge serves it
curl -s {{site-root}}/robots.txt

# 2. Status code for one AI token on one page
curl -s -o /dev/null -w "%{http_code}\n" -A "{{user-agent-token}}" {{page-url}}

# 3. Compare several tokens against a browser baseline
for ua in "Mozilla/5.0" "OAI-SearchBot" "GPTBot" "ChatGPT-User" "Claude-SearchBot" "ClaudeBot" "Claude-User" "PerplexityBot" "Perplexity-User"; do
  printf "%s  " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" {{page-url}}
done

A 403 or 402 for one token and a 200 for the browser points to a bot rule. The result shows the CDN's answer to that string, not proof of what the real crawler receives, because verified-bot rules can treat a genuine crawler differently.

What to check: Each search, training and user-fetch token against the served robots.txt, then each crawler's action at the CDN.

Tool: Screaming Frog custom user agents and custom robots.txt, then Cloudflare AI Crawl Control or your CDN's bot settings.

Pass condition: Every token you chose to allow reaches the sample URLs with no disallow line and no block action.

Access needed: Cloudflare dashboard access for AI Crawl Control, while Screaming Frog needs no access to the site.

Limitation: Cloudflare evaluates multi-purpose crawlers under both Search and Training, so a Training block can stop Googlebot without any robots.txt edit.

How To Check HTML Content Access Without JavaScript

Content that only appears after JavaScript runs is invisible to most AI crawlers, because none of the major AI crawlers render JavaScript in Vercel's analysis of its network, with Google's Gemini and AppleBot as the exceptions. Vercel's one-month sample, published in December 2024, covered OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot and others.

ChatGPT and Claude crawlers did fetch JavaScript files, at 11.50% and 23.84% of their fetches, but did not execute them. Content in the initial HTML, including JSON data, can still be read. The rule for the audit follows from that. Main content, links and metadata must be in the raw HTML response.

To check it, crawl the URL sample twice in Screaming Frog. Text Only mode extracts from the raw HTML and ignores client-side JavaScript. JavaScript mode runs the page in headless Chrome and extracts from the rendered HTML. Compare the copy, headings and internal links each crawl finds, and any gap is content a non-rendering AI crawler misses.

Sitebulb adds two views. Its Response vs Render report compares response and rendered HTML for six elements, meta robots, canonical, title, meta description, internal links and external links. Single Page Analysis shows the raw HTML and the rendered DOM of one URL side by side, highlighting what JavaScript adds, modifies or removes.

What has to be in the raw HTML:

A render difference is not automatically a problem. Sitebulb itself notes that differences in rendered HTML are not inherently bad, so judge each gap by whether it removes copy, links or directives. Its meta robots and canonical checks also do not account for HTTP headers, so check header-level directives separately.

In Sitebulb's own JavaScript SEO Report survey in 2025, 81% of respondents said they use tools to compare response and rendered HTML.

What to check: Main copy, headings, internal links and metadata in the raw HTML response.

Tool: Screaming Frog Text Only and JavaScript crawls, with Sitebulb Response vs Render and Single Page Analysis for detail.

Pass condition: The Text Only crawl finds the same main copy, headings and internal links as the JavaScript crawl.

Access needed: No site login, though a server that blocks Sitebulb's user agent needs a pre-allowlisted or authorised one.

Limitation: Response vs Render runs only with Sitebulb's Chrome Crawler, and its meta robots and canonical checks ignore HTTP headers.

What Makes A Page Eligible For AI Overviews And AI Mode?

Google's AI Overviews and AI Mode add no technical requirements of their own, since a page only needs to be indexed and eligible to be shown in Google Search with a snippet to appear as a supporting link. The audit step is therefore a standard index check, plus a check that snippet controls leave your key passages usable.

Crawl access for these features runs through ordinary Googlebot rules in robots.txt. Run each sample URL through URL Inspection in Search Console:

1. Confirm the URL is on Google, the page fetch succeeded, robots.txt allows crawling and indexing is allowed.
2. Compare the user-declared canonical with the Google-selected canonical, since a mismatch means Google indexes another URL.
3. Run Test live URL, then open View tested page to see the raw HTML, HTTP headers and, in a successful live test, a screenshot.
4. Check that nosnippet, data-nosnippet, max-snippet and noindex do not strip the passages you want AI features to use.

URL Inspection also points back to earlier gates. A result that says robots.txt blocks crawling sends you back to the robots.txt check, and a failed page fetch sends you back to the CDN rules. View tested page shows JavaScript console output and loaded resources too, which helps explain a raw versus rendered gap found in the HTML check.

Search Console also limits what the step can prove afterwards. Traffic from AI features sits inside the Performance report's Web search type and is not reported separately, so this step confirms eligibility rather than AI traffic.

What to check: Index status and snippet eligibility for each sample URL.

Tool: Search Console URL Inspection, with Test live URL and View tested page.

Pass condition: The page is indexed and eligible for a snippet, with no snippet control removing its key passages.

Access needed: A Search Console property you own that contains the URLs.

Limitation: Inspections, live tests and indexing requests have per-property daily limits, and a passing live test does not guarantee indexing.

How To Confirm AI Crawlers Reach Your Pages

A robots.txt that allows AI crawlers proves nothing until CDN or server logs show those crawlers receiving 200 responses on the pages you care about, and Cloudflare's AI Crawl Control lists request counts and robots.txt violations for each crawler. Pull those figures for the sample URLs over a recent period.

Watch the error share as well as the hits. Vercel's analysis found that 34.82% of ChatGPT fetches and 34.16% of Claude fetches hit 404 pages, against 8.22% for Googlebot. A high 404 share on your own logs points to broken links or stale URLs that AI crawlers keep requesting.

Three sources show crawler hits:

• Cloudflare AI Crawl Control: Request counts, violations and actions per crawler. On the Free plan it identifies crawlers by user agent string only, and the unsuccessful-request count can include failures from any rule, not only AI Crawl Control blocks.
• Server or CDN logs: Status codes per user agent and URL, filtered to the AI tokens you allowed.
• Scrunch Agent Traffic: AI bot visits through a direct CDN or hosting integration with Akamai, AWS CloudFront, Cloudflare, Fastly, Vercel or WordPress, with Shopify, Webflow and Wix sites connecting through a Cloudflare proxy.

What to check: Requests and status codes per AI crawler on the sample URLs.

Tool: Cloudflare AI Crawl Control, server or CDN logs, and Scrunch Agent Traffic once a CDN or host is connected.

Pass condition: Each allowed AI crawler shows 200 responses on the sample URLs, with few 404s.

Access needed: A CDN or hosting integration for Scrunch, and log or dashboard access for the rest.

Limitation: Any user agent can be imitated, so only crawlers checked against published IP ranges count as verified.

Which Tools Do You Need For An AI Retrieval Audit?

Five tools cover the whole audit between them, with Screaming Frog and Sitebulb for raw versus rendered HTML, Google Search Console for Google index eligibility, Cloudflare for edge bot rules and logs, and Scrunch for AI agent traffic and page-level AI audits. Each covers different steps, so none replaces the others.

What drives the cost of each tool:

Screaming Frog: Each user needs their own licence, and licences must be renewed. Crawl size on a licence depends on the memory and storage you allocate, and AI tokens go in as custom user agents.

Sitebulb: The Response vs Render report runs only with the Chrome Crawler, so switch that crawler on for the audit's HTML step.

Google Search Console: The cost is access rather than a licence. It works only on properties you own and reflects Google's crawlers, not third-party AI user agents.

Cloudflare: On the Free plan, AI Crawl Control detects self-identifying crawlers by user agent only. Paid plans add Bot Management detection IDs and custom block responses. Pay per crawl, the Charge action, is in closed beta and not generally available.

Scrunch: Page audits are capped per plan, with 5 on Starter, 10 on Growth and expanded audits on Enterprise, and agent traffic monitoring comes with every plan. Its Deep AI Audit scores access controls, content delivery, content quality and content alignment with pass and fail checklists, but the thresholds are Scrunch's own, such as an unsourced under-5-seconds page-load guideline. Scrunch is now a Sitecore company, with the product name unchanged.

Teams that need measurement once access is fixed can compare AI citation tracking tools next.

What Access And Permissions Does The Audit Need?

Search Console's URL Inspection works only on properties you own, and Scrunch shows agent traffic only after a CDN or hosting integration is connected, so request every login and integration before the audit starts. Ask whoever owns DNS, the CDN and analytics.

Two Search Console details trip teams up. URL Inspection needs a property that contains the URLs, while the robots.txt report exists only for Domain properties, so a URL-prefix property alone covers one check but not the other.

Scrunch has three separate prerequisites. Agent Traffic needs the CDN or host connection, AI Referrals for human visits needs Google Analytics 4, and the Deep AI Audit needs a brand and domain set up in AI Context plus no CDN or security rule blocking Scrunch's user agent.

Cloudflare settings sit with whoever owns the zone. The managed robots.txt toggle and the new-domain default opt-out both live in the dashboard's Security settings.

Decision rule: Request access before the audit starts, from whoever owns DNS and the CDN, and log which checks can run without it.

How To Fit The Audit Into An AI Visibility Workflow

Run the retrieval audit before any AI visibility measurement, because a page that AI crawlers cannot fetch cannot be cited however strong its content is, and rerun it after every CDN, robots.txt or rendering change.

A workable AI visibility workflow runs in three stages:

1. Access, with this audit on the priority URL sample.
2. Content, written and structured for the pages that pass.
3. Measurement, once crawlers can reach what you want measured.

Rerun the audit when any of these change:

• CDN or security settings, including Cloudflare's function-based bot policies and the default block for new domains from September 15, 2026
• Robots.txt, allowing about 24 hours for OpenAI's search systems and up to 24 hours for Perplexity to reflect the edit
• The front-end framework or rendering setup, since a move to client-side rendering can empty the raw HTML
• New domains or subdomains added behind the CDN

Keep the URL sample and the audit sheet from each run, so a rerun compares against the last pass instead of starting over. A token that passed last time and fails now points straight at the change that caused it.

Measurement comes next. To report results cleanly, measure AI visibility without double-counting and separate AI brand mentions from citations in the same report.

Some platforms go a step further after the audit. Scrunch's Agent Experience Platform serves an AI-optimised version of a site to AI retrieval agents at the CDN layer, with native integrations for Akamai, Cloudflare and Vercel, while the human-facing site stays unchanged. That is an option once pages pass the audit, not part of it.

Common AI Retrieval Audit Mistakes

The first AI retrieval audit mistake to rule out is blocking GPTBot to stop training and assuming that also controls ChatGPT search, because OpenAI handles search through OAI-SearchBot as a separate token. The other four mistakes share the same root, auditing the rule you wrote instead of what crawlers meet.

1. Blocking GPTBot to control ChatGPT search. Set OAI-SearchBot and GPTBot as two separate rules, one for search and one for training.
2. Treating robots.txt as access control. OpenAI says its rules may not apply to ChatGPT-User, and Perplexity-User generally ignores them, so use a CDN or firewall rule for anything that must stay out.
3. Auditing the repository robots.txt. Cloudflare's managed robots.txt places its own AI disallow lines above your file, so always read the file the edge serves. The robots.txt violations column in AI Crawl Control then shows which crawlers ignore those lines.
4. Blocking Training on Cloudflare. Multi-purpose crawlers such as Googlebot fall under the Training policy too, so a Training block can cut them off even when Search is allowed. Review the Training policy before the Search policy, and opt out of the new-domain default in Security settings when it does not suit the site.
5. Relying on llms.txt. Search Engine Land notes that no major AI engine has confirmed it reads llms.txt or llms-full.txt, and it describes ai.txt as an academic proposal no major AI company honors. Fix access, raw HTML and indexing first, since those are the gates the operators document.

Frequently Asked Questions

Sources