CanAgentUse

CanAgentUse products

Agentic Browsing AuditCheck dialogs, menus, drawers, popovers, form errors, checkout steps, and other UI states that a standard Lighthouse scan may not capture.View product MCP ServerInstall CanAgentUse in your coding agent so scan findings become repo-ready fixes.View product WooCommerce PluginPublish UCP discovery and agent-ready commerce signals from WooCommerce without rebuilding checkout.View product Shopify PluginAdd UCP discovery, storefront metadata checks, and controlled cart-to-checkout flows for shopping agents.View product AgentReady CIScan commits and PRs, publish reports, and choose which agent-readiness failures should block merges.View product

CanAgentUse tools

UCP Suite

Validate Universal Commerce Protocol discovery, then test product search, carts, checkout links, and merchant handoff flows.

UCP ValidatorUCP Playground

MCP Playground

Connect to remote MCP servers, inspect tools and resources, test prompts, auth, headers, notifications, and JSON-RPC responses.

MCP Playground

A2A Playground

Inspect Agent Cards, validate advertised endpoints, and prepare safe requests for agent-to-agent workflows.

A2A Playground

Agent Website Viewer

Enter a public URL and see the roles, names, landmarks, controls, and blockers that shape how AI agents understand the page.

Agent Website Viewer

AI crawler compatibility tool

AI Crawler Compatibility Checker

Check whether AI crawlers like GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and other AI user agents can discover, crawl, and understand your website.

Run the complete agent readiness scan

Live crawler access diagnostic

Bot access map

The scan follows the public discovery path an AI retrieval system would depend on: robots.txt, bot-specific policy, sitemap hints, canonical links, and AI-readable guidance.

GPTBot
ClaudeBot
PerplexityBot
Google-Extended
Crawler rules
AI bots
Discovery
Sitemap
Guidance
llms.txt

What the checker tests

The scan runs a focused slice of the CanAgentUse assessment suit for crawler discovery and AI-readable access.

  • robots.txt availability
  • AI bot rule interpretation
  • sitemap discovery
  • canonical and link hints

Robots.txt AI crawler rules

The report checks whether important AI user agents are allowed, blocked, or caught by broad wildcard rules.

  • GPTBot and OAI-SearchBot
  • ClaudeBot
  • PerplexityBot
  • Google-Extended

WAF and bot-blocking issues

Crawler access can fail even when robots.txt looks open. The scanner records fetch evidence and highlights blocking symptoms when checks detect them.

Sitemap and canonical checks

AI crawlers need stable discovery paths. The diagnostic validates sitemap candidates and link-header signals that help agents choose canonical resources.

llms.txt and AI-readable guidance

A present and useful llms.txt file can give AI systems a concise map of your important pages, policies, and source material.

How to fix issues

Every failing or warning check includes evidence and remediation guidance from the same versioned checks used in full reports.

FAQ

What people ask before scanning

What is AI crawler compatibility?

AI crawler compatibility means your site can be fetched, discovered, and interpreted by AI search and answer systems without accidental robots.txt, sitemap, header, or AI guidance blockers.

Does this check GPTBot specifically?

Yes. The focused scan checks AI bot access rules including OpenAI crawlers such as GPTBot and OAI-SearchBot, alongside other major AI crawler families.

Is this a full AI agent readiness scan?

No. This diagnostic focuses on crawler access and discovery. The full AI Agent Readiness Scanner adds structured data, APIs, OAuth, MCP, skill discovery, commerce, performance, and accessibility checks.