Is Your Site Blocked from ChatGPT? How to Check (and Fix) AI Access

To check whether ChatGPT can read your site, open your robots.txt file (at yourdomain.com/robots.txt) and look for rules that block the AI crawlers GPTBot, OAI-SearchBot, and ChatGPT-User. If any of those bots are disallowed, your pages cannot be crawled, indexed for AI search, or cited inside ChatGPT's answers. The fix is to add explicit Allow rules for the assistants you want to appear in — and the whole check takes a couple of minutes.

Search is no longer just blue links. A growing share of buyers now ask ChatGPT, Google's Gemini, Perplexity, and Claude for recommendations instead of scrolling a results page. Those assistants can only mention, summarize, or link to a business if their crawlers were allowed to read the site in the first place. A single overly broad line in robots.txt can quietly make your brand invisible to an entire category of AI-driven discovery — and most owners never realize it.

Which AI Crawlers Actually Matter

Each major AI assistant uses one or more named bots (user agents) to fetch web pages. Blocking a bot blocks the assistant that depends on it. Here are the ones worth knowing:

• GPTBot — OpenAI's primary crawler. It gathers content used to build and improve the models behind ChatGPT. Block it and your pages are excluded from that training and indexing pipeline.
• OAI-SearchBot — OpenAI's search crawler. It powers ChatGPT's web-search and source-citation feature. If you want to be cited as a source when ChatGPT browses the live web, this bot needs access.
• ChatGPT-User — the agent used when a ChatGPT user (or a plugin/action) fetches a specific URL on demand. Blocking it can stop ChatGPT from opening your page when a user explicitly asks about it.
• Google-Extended — Google's control for whether your content can be used by Gemini and its AI features. It is separate from the normal Googlebot, so allowing Google search does not automatically allow Gemini.
• PerplexityBot — Perplexity's crawler for its answer engine. It is heavily citation-driven, so allowing it is one of the more direct ways to earn AI citations.
• ClaudeBot and Claude-Web — the crawlers associated with Claude. Allow them if you want your site readable by that assistant.
• CCBot — the Common Crawl bot. Its public dataset feeds many AI systems indirectly, so it is a quiet but meaningful access point.

The important takeaway: there is no single “AI bot.” Each assistant has its own user agent, and your robots.txt has to address them individually (or with a catch-all) if you want broad visibility.

How to Check Your robots.txt

Your robots.txt is a plain-text file that lives at the root of your domain. Visit https://yourdomain.com/robots.txt in a browser and read it. A User-agent line names a bot, and the Disallow and Allow lines that follow tell that bot where it may and may not go. The two patterns that silently block AI assistants are an explicit block on a named AI bot, or a blanket User-agent: * / Disallow: / that catches everything.

Watch for blocks that look like this — each one removes a path to AI visibility:

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

If you want AI assistants to read and cite your site, your robots.txt should explicitly welcome them instead. A clean, AI-friendly configuration looks like this:

# Allow major AI assistants to read the site
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: CCBot
Allow: /

# Keep normal search crawlers welcome too
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Two things to verify after editing. First, make sure no later, broader rule re-blocks a bot you just allowed — robots.txt is read top to bottom and the most specific matching User-agent block wins. Second, confirm there is no leftover Disallow: / under a wildcard that the AI bots fall back to.

Why a Blocked Bot Means Zero Citations

AI assistants cannot cite what they have never read. If GPTBot is disallowed, ChatGPT has no record of your pages to draw on. If OAI-SearchBot is blocked, ChatGPT's live web search will skip your site and cite a competitor or a roundup article instead. The same logic applies to every assistant: no crawl access means no understanding of your products, services, or expertise, which means no mention and no link when a prospective customer asks a relevant question.

This is a harsher cutoff than traditional SEO. In classic search you can rank on page two and still get some traffic. In an AI answer there is no page two — the assistant names a handful of sources, and if you were never crawlable, you are simply not in the conversation. Worse, the block is invisible from the front end: your website looks perfectly normal to human visitors while being completely dark to the assistants your buyers are asking.

How to Fix It

The fix is usually small. Add explicit Allow: / rules for the AI user agents listed above, remove or narrow any Disallow: / that catches them, and re-upload robots.txt to your site root. Most content management systems let you edit robots.txt directly or through an SEO plugin; some hosts require editing the file over FTP or in the file manager. After you publish the change, re-fetch yourdomain.com/robots.txt in a private browser window to confirm the new rules are live and not cached.

Don't Forget llms.txt

An emerging companion to robots.txt is llms.txt — a proposed plain-text file, also placed at your domain root, that gives AI assistants a curated, structured map of your most important pages and context. Where robots.txt controls access, llms.txt is about guidance: it helps an assistant find your best content and understand what your business does. Adoption is still early and it is not a substitute for fixing robots.txt, but publishing one is a low-cost way to get ahead of where AI discovery is heading.

How Custom Web Audits Helps

Checking these rules by hand is doable, but it is easy to miss a buried Disallow, a conflicting wildcard, or one assistant you forgot to allow. Custom Web Audits runs an AI-bot crawlability check as part of every audit, flagging exactly which assistants can and cannot read your site and pointing to the precise robots.txt lines responsible. If you just want a fast answer, try our free AI Bot Access checker to see at a glance whether GPTBot, PerplexityBot, and the rest are allowed, or our robots.txt tester to validate your rules against any URL. For the full picture — AI visibility, citations, technical SEO, and the fixes ranked by impact — run a complete AI website audit and find out where you stand in AI search before your competitors do.

Frequently Asked Questions

How do I know if my site is blocked from ChatGPT?

Open your robots.txt file at yourdomain.com/robots.txt and look for Disallow rules under User-agent: GPTBot, OAI-SearchBot, or ChatGPT-User, as well as any blanket User-agent: * with Disallow: /. If those bots are disallowed, ChatGPT cannot read or cite your pages. A free AI bot access checker can confirm this in seconds.

What is GPTBot and should I allow it?

GPTBot is OpenAI's primary web crawler, used to gather content for the models behind ChatGPT. If you want your business to be understood and potentially cited by ChatGPT, you should allow GPTBot in your robots.txt. Blocking it removes your pages from that pipeline.

Does blocking AI crawlers affect my Google rankings?

Blocking AI-specific crawlers like GPTBot, Google-Extended, or PerplexityBot does not directly change your traditional Google search rankings, because those bots are separate from Googlebot. However, it does remove you from AI-powered answers and citations, which are an increasingly important source of discovery and traffic.

How do I allow GPTBot in robots.txt?

Add a block to your robots.txt that reads User-agent: GPTBot followed by Allow: /, then re-upload the file to your domain root. Make sure no broader rule later in the file re-blocks it, and confirm the change is live by reloading yourdomain.com/robots.txt.

What is llms.txt and do I need it?

llms.txt is an emerging plain-text file placed at your domain root that gives AI assistants a curated map of your key pages and context. It complements robots.txt: robots.txt controls access, while llms.txt offers guidance. It is optional and adoption is still early, but it is a low-effort way to prepare for AI-driven discovery.

Will allowing AI crawlers slow down my website?

No. AI crawlers fetch pages the same way traditional search engine bots do, and reputable ones respect crawl-rate signals. Allowing them adds negligible load for a normal business website while opening the door to AI visibility and citations.