Will AI chatbots recommend your website?

How an assistant ends up mentioning a site at all

There are two distinct routes, and conflating them is the source of most bad advice.

The first is training data: text absorbed when the model was built, months or years ago, with no attribution and no way to add to it after the fact. Nothing you do this week affects a model already trained, and no technique gets you into one retrospectively.

The second is retrieval, and it is the one that matters. When an assistant looks something up while answering — browsing, or searching an index it maintains — it fetches pages in the moment and can cite them. This is where a site can be included or excluded by things you control, and where effort is worth spending.

Retrieval usually rides on a search index

Several assistants find candidate pages using a conventional search engine before reading them. That has a blunt consequence: if you do not appear in ordinary search results for a question, you are frequently not a candidate for the AI answer to it either. Conventional search visibility is a prerequisite rather than an alternative.

Being fetchable is not the same as being indexed

A page can be indexed by Google and still be unavailable to an assistant, because a separate crawler was blocked. The two are governed by different directives, and a site can accidentally be visible in search and invisible to every assistant.

The crawlers, and what blocking each one actually does

AI companies publish the names their crawlers use, and robots.txt controls them the same way it controls any other. The important thing is that different crawlers from the same company do different jobs, so blocking the wrong one has consequences you did not intend.

This is a decision rather than a best practice. Publishers with content worth licensing block deliberately; a business that wants to be recommended usually wants the opposite. What matters is that it is chosen rather than inherited from a template.

Google-Extended is not Googlebot

Google-Extended controls whether your content may be used to improve Google's generative models. It does not affect Googlebot, your ranking, or whether you appear in ordinary search results. Blocking it is a licensing decision with no search penalty attached, which is unusual and worth knowing.

GPTBot, ClaudeBot, PerplexityBot and their siblings

OpenAI, Anthropic and Perplexity each publish crawler names and each respects robots.txt. Several run more than one, separating the crawler that gathers training data from the one that fetches a page live to answer a question in front of a user. Blocking the second is what removes you from answers; blocking the first is a licensing choice.

A blanket block is almost always an accident

The most common real-world configuration is a site that blocked every AI crawler by pasting a list from an article in 2023, and whose owner now wants to be recommended by assistants. If that describes you, robots.txt is the first thing to read and the change takes a minute.

What genuinely helps, and why

Everything below follows from one idea: a model quoting you has to be able to extract a self-contained, attributable statement. Content written so that a single paragraph makes sense on its own is easier to quote than content whose meaning depends on the three paragraphs above it.

None of this is exotic, and most of it is indistinguishable from writing clearly for people. That is not a coincidence, and it is the reason this work is safe to do even as the mechanisms change.

Answer the question in the first sentence, then elaborate

A page that opens with two paragraphs of preamble before the answer gives a model nothing clean to lift. Stating the answer plainly and immediately, then giving the detail and the caveats underneath, is the single most effective structural change — and it is also better for the human who is scanning.

Make facts self-contained

A sentence like our opening hours are listed above cannot be quoted usefully. A sentence naming the business, the thing and the value survives being lifted out of the page and still means something. Assume every paragraph will be read alone, because for this purpose it will be.

Structured data, because it removes the guesswork

Marking up who you are, what you sell, where you are and when you are open states plainly what would otherwise have to be inferred from prose. It is a long-established part of ordinary search and it is straightforwardly useful to anything trying to extract facts reliably.

Say when the page was written, and by whom

Anything summarising sources has to weigh them, and an undated page from an unnamed author is harder to trust than a dated one from an identified business. This costs nothing and is among the few things you can do that is unambiguously in your control.

Keep the content in the HTML

Text that only appears after JavaScript has run is reliably read by Google and unreliably read by everything else. If your key content is rendered on the client, a fetcher that does not execute scripts sees an empty page. Server-rendering the substance removes a whole class of invisibility.

What is unproven, and what is simply false

This is the part most articles on the subject leave out, and it is the part worth reading twice.

llms.txt is a proposal, not a standard

A file at the root of your site summarising its content for language models, suggested in 2024. It is a sensible idea and it costs almost nothing to publish. What it does not have is confirmed adoption by the major assistants, and there is no public evidence that publishing one causes you to be cited. Treat it as a cheap bet rather than a technique, and be sceptical of anyone selling it as one.

Nobody can promise you a mention

Any service guaranteeing that an assistant will recommend your business is selling something it cannot deliver. There is no submission process, no ranking factor to tune and no relationship to buy. Where visibility can be influenced, it is influenced indirectly, through the things above.

Writing for models rather than people does not work

Pages stuffed with question-shaped headings and repetitive phrasing read badly to humans and are not known to help with anything. The same overcorrection happened with keyword stuffing twenty years ago and ended the same way. Clear writing has the useful property of being the right answer under both regimes.

You cannot measure this well yet

There is no equivalent of Search Console for AI assistants. Referral traffic from them is partially visible in analytics, but an answer that satisfies somebody without a click is invisible by construction — and that is the majority case. Anyone reporting a precise AI visibility score is estimating and should say so.

A realistic order to do this in

Start by finding out whether you are blocked, because that is binary and takes two minutes: read your own robots.txt and look for the crawler names above. Everything else is a matter of degree; this one is a switch.

Then make sure the content exists in the HTML and that your most important pages state plainly who you are, what you do and where. Then add or repair structured data on the pages that describe the business itself. Then, if you like, publish an llms.txt, understanding that it is speculative.

And keep doing ordinary search work, because the retrieval route mostly runs through conventional indexes. A site nobody can find in Google is rarely cited by an assistant either, and the two efforts overlap far more than the current volume of writing about AI search would suggest.

The checks behind this guide

• You have decided what AI crawlers may doCompanies training AI models and answering questions with your content read your site using crawlers with names like GPTBot and ClaudeBot. Your robots.txt says nothing about them, so by default they are all allowed. That may be exactly what you want — being quoted in an AI answer is visibility — but it is worth being a decision rather than an accident.
• You guide AI tools to your best contentAn emerging convention: a plain text file listing what your site is and which pages matter, written for AI assistants rather than search engines. Nobody is penalised for not having one, and support is still patchy — but it is cheap, and it is how you get described accurately rather than guessed at.
• Search engines are told who runs the siteNothing on a site says, in a form a machine can read, which organisation it belongs to unless somebody puts it there. That block is what connects a website to a business: it is what a knowledge panel is built from, what ties a company to its social accounts, and increasingly what an AI assistant reads when deciding whether a site is a business it can name. It is one block, written once, and it goes on every page.
• Structured data parses correctlyThe markup that produces star ratings, prices and FAQ dropdowns in search results is invalid on these pages, so it is silently ignored. You get none of the benefit and no warning that anything is wrong.
• robots.txt is present and validrobots.txt is the first file a search engine asks for. Mistakes in it are quiet — a typo'd rule is simply ignored, so you think a page is blocked when it is not, or the reverse.
• Pages have enough content to rankA page with very little text gives a search engine almost nothing to work with, so it rarely ranks for anything. This is worth checking rather than assuming: a contact page or a gallery is legitimately short, and padding it out would make it worse for the person reading it.
• robots.txt is not blocking pages you want foundA page disallowed in robots.txt is a page search engines will not read. That is often deliberate — admin screens, carts, internal search results — but it is worth confirming none of them matter. The serious case is a page that appears in your sitemap and is blocked at the same time: the sitemap says index this, robots.txt says do not look, the block wins, and the contradiction usually means one of the two files is out of date.