AI readiness
You have decided what AI crawlers may do
ai.crawler-directives
Why this matters
Companies training AI models and answering questions with your content read your site using crawlers with names like GPTBot and ClaudeBot. Your robots.txt says nothing about them, so by default they are all allowed. That may be exactly what you want — being quoted in an AI answer is visibility — but it is worth being a decision rather than an accident.
Who fixes it
You can, usually
Roughly how long
Minutes
Care needed
Low risk to change
How to fix it
To keep allowing them, do nothing. To opt out, add a section per crawler to robots.txt — `User-agent: GPTBot` followed by `Disallow: /`. Opting out means your content is less likely to be cited in AI answers, which cuts both ways.
How we score it
Failing this check takes up to 0 points off your ai readiness score. It is a fact about your site rather than a measurement, so it reads the same on every scan until you change something.
Does your site pass this one?
This check runs on every scan, along with the other 106. Free, no account, and you see the evidence for each result.
Check my site