Search engines
Googlebot and Bingbot. Block them and search engines stop reading the site, so its pages lose their descriptions and fade from search, and from Copilot answers that rely on Bing.
robots.txt // free
Free robots.txt checker
See who your robots.txt lets in: Google and Bing, AI search crawlers such as ChatGPT Search and Perplexity, and the assistants that visit when a person asks about you.
Clunky Watch
Sites break quietly: a renewal fails, a setting changes, a robots.txt rule ships by mistake. Watch checks your homepage on a daily schedule and emails you when something breaks. £19 a year.
Who is asking
Three kinds of crawler, three different choices
AI search
OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch pages so AI search can cite them.
AI training
GPTBot, ClaudeBot, Google-Extended and CCBot. Blocking them is a legitimate choice and does not affect search.
FAQ
Questions people ask
More free checks:
What does the robots.txt checker test?
Whether robots.txt answers as a plain text file, whether its rules let search engines, AI search crawlers and AI assistants read your homepage, which AI training crawlers it blocks, and whether it lists a sitemap.
Should I block GPTBot or ClaudeBot?
That is your choice. GPTBot, ClaudeBot, Google-Extended and CCBot mainly collect data for AI training, so blocking them does not remove you from search. Blocking OAI-SearchBot or PerplexityBot stops those services indexing your pages for their AI search answers, so you are much less likely to be cited there.
What happens if I have no robots.txt?
Nothing bad. When robots.txt returns 404, crawlers treat the whole site as open to crawl. A robots.txt that returns a server error is worse: Google pauses crawling for a while, then falls back to the last copy it saved, or crawls freely if it has none.
Does robots.txt keep a page out of Google?
Not reliably. It stops crawling, but a blocked address can still be listed if other sites link to it. Use noindex to keep a page out of search results.