AI-Agent Readiness Checker

How it works

What we check

• AI crawler access (robots.txt) — Checks whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot are allowed or blocked from crawling your site. A blocked bot means that AI assistant can never read or cite your content, no matter how good it is.
• llms.txt presence — llms.txt is an emerging standard — a plain-Markdown file at your domain root that gives AI assistants a curated map of what your site is and which pages matter most, instead of making them guess from a full crawl.
• sitemap.xml presence — Confirms a sitemap exists and is referenced from robots.txt, so both search engines and AI crawlers can discover every page on your site without relying purely on internal links.
• Structured data (JSON-LD) — Counts your JSON-LD blocks and the distinct schema.org types they declare (Organization, Product, FAQPage, Article, and more). Structured data gives AI assistants machine-readable facts instead of forcing them to parse prose.
• Semantic HTML landmarks — Checks for real <header>, <nav>, <main>, and <footer> elements (or their ARIA-role equivalents). These landmarks let an AI agent instantly separate your primary content from navigation and boilerplate.
• Heading hierarchy — Confirms a single, clear H1 and a heading structure with no skipped levels, so the topic and sub-topic structure AI extracts matches the structure you actually intended.
• Content extractability — A text-in-raw-HTML word count, a text-to-HTML ratio, and a JavaScript-rendering risk heuristic. Most AI crawlers fetch raw HTML and do not execute JavaScript — content that only appears after a client-side render is invisible to them.
• Machine-parsing basics — Title tag, meta description, canonical URL, and an HTML lang attribute — the first things any parser, human or machine, reads to understand what a page is and where it lives.

Common issues we catch

• AI crawlers blocked without realizing it — A boilerplate robots.txt copied from another project, or a security plugin's 'block bad bots' preset, can silently disallow GPTBot, ClaudeBot, or PerplexityBot. Your site can rank perfectly in Google and still be invisible to every AI assistant.
• No llms.txt at all — Most sites have never heard of llms.txt. Without it, AI assistants have to infer your site's purpose and key pages from a full crawl — slower and less reliable than being handed a clean, curated map.
• Content trapped behind client-side rendering — Single-page apps (React, Vue, Next.js client components) that render their real content only after JavaScript executes leave AI crawlers — which typically fetch raw HTML — looking at an empty shell.
• Structured data missing or type-thin — A site might have one generic JSON-LD block and stop there. Without Organization, Product, FAQPage, or Article markup, AI assistants have no structured facts to extract and fall back on unreliable prose-parsing.
• Multiple H1s or missing landmarks — Page builders frequently wrap the logo, hero banner, and page title all in H1 tags, or skip semantic elements entirely in favor of generic <div>s. AI agents lose the clear signal of 'this is the main content' vs. navigation chrome.
• No sitemap, or one not referenced from robots.txt — A sitemap.xml that exists but is never declared with a 'Sitemap:' line in robots.txt is effectively undiscoverable to any crawler that doesn't already know the URL.
• Thin or bloated pages — Pages with very little visible text relative to their markup and script weight — common on template-heavy sites — give AI assistants too little substance to reliably summarize or cite.

Where this matters

• ChatGPT (OpenAI) — GPTBot, OAI-SearchBot, and ChatGPT-User are the crawlers behind ChatGPT's training, search, and live-browsing features respectively. This checker verifies your robots.txt policy for all three.
• Claude (Anthropic) — ClaudeBot is Anthropic's crawler. If it's blocked, Claude cannot read or cite your site when answering user questions that touch on your business or content.
• Gemini & Google AI Overviews (Google) — Google-Extended controls whether your content can be used for Gemini and AI Overviews specifically — separate from the standard Googlebot that powers classic Search.
• Perplexity — PerplexityBot is Perplexity's dedicated crawler for its answer-engine results. Perplexity leans heavily on live citations, making crawl access especially important.
• Common Crawl-derived models — CCBot feeds the open Common Crawl dataset, which numerous AI models (beyond the big four) train on or reference. Blocking it removes you from that entire downstream ecosystem.
• llms.txt-aware tools — llms.txt is a community-driven, still-emerging convention (llmstxt.org). Support varies by tool today, but publishing one costs nothing and future-proofs you as more AI products adopt it.