Make your site readable to ChatGPT, Claude, Perplexity and Google's AI answers

Know which crawler does what

Each AI company runs separate crawlers for search and for training, and each honours robots.txt by its own documentation. Blocking a training crawler keeps your pages out of future models. Blocking a search crawler keeps you out of the answers.

• Search and answers: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude), PerplexityBot (Perplexity), Googlebot (Google Search, AI Overviews and AI Mode), bingbot (Bing and Copilot), Applebot (Siri and Spotlight).
• Training: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), meta-externalagent (Meta). Google-Extended and Applebot-Extended are opt-out tokens, not crawlers: they control model training and change nothing in search.
• Fetches a person asks for: ChatGPT-User, Claude-User and Perplexity-User fetch a page when someone asks about it. OpenAI and Perplexity say robots.txt may not apply to these.

Write robots.txt for search, not against it

A robots.txt that blocks every bot (User-agent: * with Disallow: /) blocks search too. If you want to stay out of training but in the answers, name the training crawlers and leave the rest open. Point to your sitemap from the same file.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Check the firewall as well. A bot challenge on your host or CDN can turn crawlers away before robots.txt is ever read. Google's AI features guide names the CDN alongside robots.txt for this reason.

Put your text in the HTML

Googlebot renders JavaScript before it indexes a page, and Google still recommends server-side rendering or pre-rendering because not every crawler does. OpenAI, Anthropic and Perplexity publish nothing on whether their crawlers run JavaScript. Vercel's study of its own traffic in December 2024 found GPTBot, ClaudeBot and PerplexityBot downloading JavaScript files without running them.

Test it yourself: open your page, view the source (not the inspector), and search for a sentence from your headline. If it is not there, a crawler that skips JavaScript sees an empty page. Sites built with a client-side app builder are the usual case. Turn on server rendering or pre-rendering in your framework, or publish the landing pages as static HTML.

curl -s https://example.com | grep -c "your headline text"

Answer one question per page

Bing's guidelines, which also cover Copilot, ask for the key facts near the top of a page, one topic per page and the same names for the same things throughout. Google says the same SEO basics decide whether a page can appear in AI Overviews and AI Mode: it has to be indexed and allowed to show a snippet, and needs no special markup.

• A title of its own that names the thing and what it does.
• A meta description that answers the page's question in one or two sentences.
• An opening paragraph that gives the answer before the detail.
• Structured data (JSON-LD) that matches the page's visible content: Organization on the home page, SoftwareApplication or Product for what you sell, FAQPage only beside questions the page prints.

llms.txt: cheap, but not a fix

llms.txt is a proposal from September 2024: a Markdown file at your site's root that lists your important pages for language models. None of the major AI companies says its crawlers read it, and Google says AI files are not needed to appear in its AI features. Add one if it takes you ten minutes; do the steps above first.

# Example

> Example is invoicing for freelancers in the EU.

## Pages

- [Pricing](https://example.com/pricing): plans and what each includes
- [Docs](https://example.com/docs): setting up your first invoice

Tags that keep you out by accident

noindex removes a page from Google and from Bing's Copilot answers. nosnippet and max-snippet:0 stop Google from using the page's text in AI Overviews and AI Mode. Bing's NOCACHE and NOARCHIVE limit what Copilot can quote. Check that none of these sits in a shared layout where it reaches pages you want found.