TerlihatAI

How-to Guide

How to write a robots.txt that allows AI crawlers

To allow AI crawlers, add explicit allow rules for the major AI user-agents — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, and Bingbot — in your robots.txt, while keeping private and admin paths blocked. The complete policy is below; paste it into your robots.txt and adjust the blocked paths to your site.

  1. 01

    Paste the AI crawler allow policy

    Add allow rules for each major AI crawler. This makes your public pages readable by the engines that build AI answers.

    # GEO crawler access policy
# Keep private/admin paths blocked separately.

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Bingbot
Allow: /
  2. 02

    Keep private and admin paths blocked

    Allowing AI crawlers does not mean exposing everything. Keep cart, account, admin, and API paths disallowed so they stay out of AI answers and the index.

    User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /account
Disallow: /api/
  3. 03

    Verify the file is live and correct

    Fetch https://yoursite.com/robots.txt and confirm the allow rules are present and the syntax is valid. A typo here silently blocks the crawlers you meant to allow.

  4. 04

    Rescan to confirm

    After the file is live, run the audit again. The goal is not a one-time score — it is verifying the crawler access, schema, and content actually changed. TerlihatAI generates these files for your site and re-checks them on rescan.

FAQ

Which AI crawlers should I allow?

The main ones are GPTBot, OAI-SearchBot, and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google AI), and Bingbot (Microsoft/Copilot).

Does allowing AI crawlers hurt my SEO?

No. AI crawler allow rules are separate from search indexing. You can allow AI crawlers for citation while keeping normal SEO controls in place, and keep private paths blocked for both.

Skip the manual work

Run a free audit — TerlihatAI generates these files for your site and verifies them on rescan.

Run a free audit