AI Visibility Audit: Why Most Teams Start in the Wrong Place
Start your AI visibility audit with robots.txt: 60% of reputable sites block an AI crawler. The three layers to check, in order, and what each one costs.
Most teams open an AI visibility audit by typing twenty prompts into ChatGPT and writing down who got named. That tells you the score without telling you why, and it skips the one layer that can make the score zero on its own.
A study of robots.txt files published in October 2025 found that 60% of reputable sites now disallow at least one AI crawler, up from 23% in September 2023. The better your domain's reputation, the likelier it is that somebody on your team already locked the door.
Below is the order that finds the cause instead of the symptom: access first, then what your site hands a machine, then whether the page itself can be quoted.
Table of contents
What is an AI visibility audit?
An AI visibility audit is a check of whether answer engines can reach your site, parse what it offers them and quote a passage cleanly, run in that order so a failure at one layer explains the layers above it. It diagnoses the cause, while prompt tracking reports the score.
The distinction matters because the two get sold as the same thing. A dashboard that says you appear in 4% of answers is a measurement, and a measurement with no mechanism behind it leaves you guessing at which of a dozen possible reasons applies.
Running the layers in order also saves money. The access layer costs nothing and takes minutes, and finding a blocked crawler there means you were about to pay a subscription to watch a number sit at zero for reasons no tracker would have surfaced.
Layer 1: can the crawlers reach you at all?
Open your robots.txt first. If it disallows the bots that answer engines use, those engines cannot read a single page, and everything downstream is unmeasurable.
60%
of reputable sites disallow at least one AI crawler, against 9.1% of misinformation sites. Over 50% block GPTBot specifically. Source: Steinacker-Olsztyn, Gosain and Dao, "Is Misinformation More Open?", 2025
That finding has an uncomfortable shape. The sites an answer engine would most want to quote are the ones that locked it out, usually through a plugin default or a blanket rule added during the 2023 scraping panic and never revisited, as the jump from 23% to 60% in twenty months suggests. You can read the full study on arXiv.
Which user agent costs you which engine
| User agent | Engine | What blocking it costs |
|---|---|---|
| GPTBot, OAI-SearchBot | ChatGPT | Training and citation, together |
| PerplexityBot | Perplexity | Citation, which is its whole model |
| ClaudeBot, anthropic-ai | Claude | Retrieval when browsing is on |
| Google-Extended | Gemini, grounding | Gemini only, not Search ranking |
| CCBot | Common Crawl | Training datasets, no live citation |
Source: published crawler documentation from OpenAI, Anthropic, Perplexity and Google, September 2026
That last row is the useful one. Blocking CCBot keeps your pages out of training corpora while leaving the search-and-cite bots free, which is the sensible middle position for a brand that wants answers to mention it without feeding the next model.
Do not read the file by eye. A robots.txt has user-agent groups, Allow rules that override Disallow, and wildcards, so a specific group for PerplexityBot cancels a blanket block above it. Telling a client they are blocked when they are not is a false statement about their own site, and it is an easy mistake to make from a quick scan.
Layer 2: what your site hands a machine
Once the crawlers can get in, the second layer asks what they find waiting. Two plain-text files do most of the work here, and both are optional in the sense that nothing breaks without them.
An llms.txt at your root states what your product does, who it is for and where the important pages live. It is a summary written for a parser rather than a visitor, which spares the machine from inferring your positioning out of a marketing homepage.
A pricing.md matters more than it sounds. Agents that compare products on a buyer's behalf discard whatever they cannot read, so a price locked behind "contact sales" or rendered by JavaScript removes you from the shortlist before a human ever sees it.
Google says plainly in its AI features optimization guide that neither file is required for AI Overviews or AI Mode, and that is worth repeating so you size the effort correctly. They help with the non-Google engines and they cost an afternoon, so treat them as cheap upside and not as the reason your visibility is low.
Layer 3: can the page itself be quoted?
The third layer is where most of the recurring work lives, because a page can be fully crawlable and still never get quoted. An engine lifts passages, not pages, so the unit that has to survive removal from its context is the paragraph.
What makes a passage quotable
The answer comes first
Each heading's question is answered in its first two sentences, complete, without depending on anything above it.
Figures carry their source
Source and year sit in the same sentence as the number. A statistic in a footnote is a statistic nobody can lift.
Sentences stand alone
"It is 42%" needs context to mean anything. "The average SaaS NPS is 42%" survives being quoted on its own.
Comparisons live in tables
Comparative prose extracts badly. A table of the same content gives the engine rows it can read without interpreting.
Source: MARK content criteria, 2026
Add the structured data while you are in there. A FAQ block with its schema markup turns four paragraphs into four question and answer pairs a machine can address individually, which is the difference between content that reads well and content that gets extracted.
How to run an AI visibility audit with MARK
The idea called "Can AI read you?" runs the first two layers as one command. You give it any URL on your site and it reports which engines can reach you today, which cannot, and what your domain currently offers a machine.
What the audit does, step by step
-
1
Fetches your robots.txt and parses it with the standard library parser, so user-agent groups, Allow overrides and wildcards resolve the way a crawler resolves them.
-
2
Reports allowed and blocked engines by name, and lists CCBot separately because blocking training while allowing citation is a position, not an error.
-
3
Checks llms.txt and pricing.md, and treats a 200 that returns your theme's HTML 404 page as missing, which is how half of these checks quietly give a false pass.
-
4
Hands back a verdict in one line, then offers the third layer: rewriting a specific article so its passages can be extracted.
Source: MARK site audit workflow, 2026
Two design choices are worth calling out because they change what you do with the result. A blocked bot is presented as your decision to make rather than a defect to fix, since blocking prevents training and citation at once and only you know which trade your brand wants.
The audit also consumes no article credits. It reads and reports without creating anything, which matters on a trial where every published piece counts against a small allowance.
What an audit will not tell you
An audit explains why you might not be cited. It cannot tell you whether you are cited, because that requires querying the engines with your prompts on a schedule and comparing against competitors over weeks.
Those are two different products and the honest setup uses both. The AI visibility trackers start around $29 a month and give you the score, while the audit gives you the mechanism behind it.
The order still favours the audit. Paying for a tracker before checking access means buying a dashboard to confirm a zero whose cause was sitting in a text file the whole time, which is the single most common way this budget gets wasted.
Where to start this afternoon
Open yourdomain.com/robots.txt in a browser tab right now and search it for GPTBot. That one lookup resolves the layer most likely to be silently costing you everything, and it takes less time than reading this paragraph.
If the crawlers are allowed, move to the page that should be winning your most valuable query and read its first two sentences under each heading. Most pages fail there, and the fix is rewriting rather than buying anything.
MARK runs the access audit, the file check and the rewrite in one place, and generative engine optimization is the discipline it automates. The trial lasts 14 days and the audit itself costs nothing to run.
Frequently asked questions
What is an AI visibility audit?
An AI visibility audit checks whether answer engines can reach your site, parse the files it offers them and quote a passage from your pages, examined in that order. It diagnoses why you are or are not appearing in AI answers, which is different from prompt tracking, which measures how often you appear.
How do I check if AI crawlers are blocked on my site?
Open yourdomain.com/robots.txt and look for GPTBot, PerplexityBot, ClaudeBot and Google-Extended. Reading the file by eye is unreliable because user-agent groups and Allow rules override each other, so a specific group can cancel a blanket block above it; a parser that implements the standard gives the answer a crawler would actually reach.
Should I block AI crawlers in robots.txt?
That is a business decision rather than a technical one, because blocking stops both training and citation at the same time. A common middle position is blocking CCBot, which feeds training datasets, while allowing the search-and-cite bots such as GPTBot and PerplexityBot so answers can still name you.
Do I need an llms.txt file?
Google states that llms.txt is not required for AI Overviews or AI Mode, so it will not fix low visibility on its own. It helps the non-Google engines understand what your product is and where your key pages are, and it costs an afternoon, which makes it worthwhile as cheap upside rather than a priority.
How often should I run an AI visibility audit?
Re-run the access layer whenever someone changes hosting, installs an SEO plugin or edits robots.txt, since those are the moments a crawler rule appears without anyone deciding on it. The page-level layer is continuous work rather than a scheduled check, because every new article either is extractable or is not.