Search
robots.txt is present and valid
seo.robots-txt
Why this matters
robots.txt is the first file a search engine asks for. Mistakes in it are quiet — a typo'd rule is simply ignored, so you think a page is blocked when it is not, or the reverse.
Who fixes it
You can, usually
Roughly how long
Minutes
Care needed
Test before and after
How to fix it
Add a robots.txt at the root of the domain containing at minimum a Sitemap: line pointing at your sitemap. Correct any misspelled directives — 'Disalow' does nothing at all.
On your platform
WordPress
WordPress serves a virtual robots.txt when no real file exists, and Yoast and Rank Math both let you edit it from their Tools screen. A real file in the site root beats the virtual one — so if editing in the plugin changes nothing, there is a file on disk to find and edit instead.
Shopify
Editable since 2021, through Online Store → Themes → Edit code and a `robots.txt.liquid` template. Shopify's default is sensible and already points at the sitemap, so only add the template if you have a specific reason: an empty or half-written one is worse than the default.
Drupal
Drupal ships a `robots.txt` in the docroot, so this is a file to edit rather than a module to install. The RobotsTxt module is the alternative where your deployment keeps overwriting the file, which is common.
Joomla
Joomla ships it as `robots.txt.dist`, and on some installs it has to be copied to `robots.txt` before it does anything — which is the usual reason it is missing. Add the Sitemap: line yourself; Joomla's default does not include one.
How we score it
Failing this check takes up to 12 points off your search score. It is a fact about your site rather than a measurement, so it reads the same on every scan until you change something.
Does your site pass this one?
This check runs on every scan, along with the other 106. Free, no account, and you see the evidence for each result.
Check my siteOther search checks
- Canonical tags point somewhere validA canonical tag tells Google which address is the real one for a page. When it points somewhere wrong — another domain, a dead page, or the wrong protocol — Google follows it anyway, and the page you wanted indexed can be dropped in favour of one that does not exist.
- Content has been updated recentlySome content is meant to sit still, and age alone is not a fault. But prices, opening hours and staff pages that have not been touched in over a year are usually wrong by now — and a visitor who acts on the wrong opening hours does not come back to check whether the rest was right.
- Descriptions are a sensible lengthLong descriptions get truncated mid-sentence; very short ones waste the space you are given.
- Each page has exactly one title tagA page can only have one title. When there are several, the browser and Google each pick one — and they do not always pick the same one, so what you see in the tab may not be what appears in search.
- Each page has one main headingSeveral top-level headings makes the structure of the page ambiguous. Worth checking rather than urgent — modern HTML permits it, and Google handles it, but it usually signals that headings are being chosen for their size.
- Every page has a main headingThe h1 is the headline of the page. Without one, both visitors skimming and search engines parsing the page have nothing telling them what it is about.