Showing up in AI answers: what helps, and what you can control

People now put to AI assistants the questions they used to type into a search box. Whether your company appears in those answers depends mostly on things you already know from search, and on a few settings most sites have never looked at. The settings are the quick part, so we start there.

Training and search are different crawlers

Every major AI company now separates two things: collecting pages that may be used to train future models, and fetching pages to answer questions today. They use different crawlers, with different names, and robots.txt treats each one separately.

What each provider says about blocking them:

• OpenAI: disallowing GPTBot indicates your content should not be used in training. Disallowing OAI-SearchBot prevents your site from appearing in ChatGPT search answers.
• Anthropic: blocking ClaudeBot excludes your future content from training. Blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude's search results and answers.
• Google: Google-Extended controls whether your content is used to train Gemini models and for grounding in some of Google's other systems. It does not affect your inclusion or ranking in Google Search. AI Overviews are part of Search and follow the Search controls described below.

That means a company can say no to training and yes to being found. Whether to do that is a business decision, not a technical one, and the answer can differ between a news publisher and a software consultancy.

# AI crawlers
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Fetches a person asks for

The third column behaves differently. When someone pastes your link into a chat, or the assistant opens your page to answer them, the provider treats that as the user's request. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic says all of its bots, Claude-User included, follow robots.txt.

So robots.txt is a strong signal for crawlers and a weaker one for what people do with your page. Anything that must not be read by machines should not be public.

What Google says about AI Overviews

Google's guidance is short. There are no additional requirements to appear in AI Overviews or AI Mode, and no special optimisation is needed. You do not need new machine-readable files, AI text files or markup. To limit what appears, you use the same controls as for ordinary snippets: nosnippet, data-nosnippet, max-snippet or noindex. Those also limit how your page appears in regular results, so use them with care.

What about llms.txt?

llms.txt is a proposal, first published by Jeremy Howard in September 2024, for a plain-text file that points AI agents to the most useful pages on a site. It is open for community input. It is not a standard, and Google says it does not need one. We publish one on this site because it costs little to keep. We would not expect it to change how often you are cited.

What we would do first

1. Decide on your crawlers. Read your robots.txt today. Many sites block or allow AI crawlers by accident, through a template or a security plugin.
2. Make sure your pages render without JavaScript. If the content only appears after scripts run, some crawlers will not see it. Server-rendered pages avoid the question.
3. Answer the questions your customers ask, one page each. Put the answer in the first paragraph, then the detail. That helps search engines, AI assistants and the people reading.
4. Keep your company facts consistent. The same name, address and description of what you do on your site, your profiles and the directories you appear in.
5. Watch where visits come from. Check your analytics for referrals from AI assistants, so you can tell whether any of this is moving.