Sparrow Plugins

Crawl Gate

Edit robots.txt in the dashboard with a line checker and preview, and block AI crawlers like GPTBot, ClaudeBot and CCBot in one click.

Coming to WordPress.org soon

WordPress makes a robots.txt for you but gives you no screen to change it. Crawl Gate adds one under Settings, with a checker that points at the exact line that is wrong.

It also lists the known AI crawlers by company and by what they do: training, AI search, or fetching a page for a user. Block the training bots and still show up in AI search.

Nothing changes on activation. WordPress keeps serving its own file until you tick Serve these rules and save.

What the free version does

Pro

The free version is complete on its own. Pro adds:

Privacy

No outside requests of its own. The free version stores nothing about visitors.

Questions

Will blocking AI crawlers remove me from Google?

No. Google-Extended only covers Gemini training. Googlebot is not on the list. Blocking OAI-SearchBot or PerplexityBot can take you out of those AI search tools, so they are marked as search bots.

My changes do not show at /robots.txt

A real robots.txt file in the site's root folder wins, and Crawl Gate warns you about it. A page cache can also keep the old copy for a while.

Where does the crawler list come from?

From each company's crawler docs and the community list at github.com/ai-robots-txt. Plugin updates bring new names.

Does robots.txt stop every bot?

No, it is a public request. The big AI companies say their crawlers follow it.