Crawl Gate
Edit robots.txt in the dashboard with a line checker and preview, and block AI crawlers like GPTBot, ClaudeBot and CCBot in one click.
Coming to WordPress.org soon
WordPress makes a robots.txt for you but gives you no screen to change it. Crawl Gate adds one under Settings, with a checker that points at the exact line that is wrong.
It also lists the known AI crawlers by company and by what they do: training, AI search, or fetching a page for a user. Block the training bots and still show up in AI search.
Nothing changes on activation. WordPress keeps serving its own file until you tick Serve these rules and save.
What the free version does
- Edit the rules served at /robots.txt
- Line checker with line numbers for typos, unknown directives, bad paths and sitemap URLs
- One-click blocks for 41 known AI crawlers, with the date the list was last checked
- Sitemap line added for you, plus extra sitemap URLs
- Preview of the exact file crawlers get
- URL tester: pick a crawler and a path, see if it is allowed and which line decides
- Warning when a real robots.txt file on the server would override your rules
Pro
The free version is complete on its own. Pro adds:
- noai headers: X-Robots-Tag noai, noimageai on every page, plus the meta tag.
- Folder exceptions: block a crawler but let it read chosen folders.
- Crawler log: which bots fetched robots.txt, how often and when.
Privacy
No outside requests of its own. The free version stores nothing about visitors.
Questions
Will blocking AI crawlers remove me from Google?
No. Google-Extended only covers Gemini training. Googlebot is not on the list. Blocking OAI-SearchBot or PerplexityBot can take you out of those AI search tools, so they are marked as search bots.
My changes do not show at /robots.txt
A real robots.txt file in the site's root folder wins, and Crawl Gate warns you about it. A page cache can also keep the old copy for a while.
Where does the crawler list come from?
From each company's crawler docs and the community list at github.com/ai-robots-txt. Plugin updates bring new names.
Does robots.txt stop every bot?
No, it is a public request. The big AI companies say their crawlers follow it.