Block unwanted bots with clearer rules
Running a site means dealing with automation you did not invite: scrapers, aggressive SEO tools, AI crawlers, and agents that ignore polite requests. Bot Rules is a practical web tool from Trip Consulting. You search a published bot catalogue, select the agents you care about, and generate pasteable rules — robots.txt, a Cloudflare WAF custom-rule expression, or an Apache .htaccess snippet.
The list in the browser is not a live scrape. Catalogue records are imported, reviewed, and published as a revision. What you see in the generator is that published set, so the tool stays consistent while the catalogue is maintained behind the scenes.
Open the generator · Help & format comparison
How it works
- Browse — Open the generator and search by name. Filter by category, country of origin, or a preset (for example top most-blocked agents). Sort by name or reported block rate, and choose how many results per page.
- Select — Tick individual bots into your selected list, or use Add this page to take everything visible under the current filters.
- Generate — Choose an output format (robots.txt, Cloudflare WAF, or .htaccess), press Generate, then Copy the result into your configuration. On smaller screens, use Selected — Review & generate to open the build panel.
There is a short help guide on the site if you want a side-by-side comparison of the three formats before you deploy anything.
Things to remember
robots.txt
A useful first signal: you declare which crawlers should stay away. Compliance is voluntary. Many bots ignore it. Treat robots.txt as a starting point, not a complete defence. More on robots.txt
Cloudflare WAF rules
WAF rules can enforce policy at the edge. Cloudflare limits each custom rule to 4096 characters. Large selections often need splitting into several rules. Review carefully so you do not block traffic you still want. Cloudflare custom rules
Apache .htaccess
Stronger than robots.txt: the server can reject requests by User-Agent before your application runs. Needs Apache 2.4+ with the right modules and .htaccess overrides enabled. Merge into an existing file; do not blindly replace a CMS .htaccess. Keep a backup and test first — a mistake can take a site offline. Details are on the Bot Rules help page.
“% blocked” is not a risk score
Where shown, the percentage comes from Known Agents’ reporting on how widely a bot appears in robots.txt block lists. It is a prevalence signal for restrictions — not proof that a bot is malicious. Unknown means no published measurement was available.
Privacy, terms, and cookies for the tool are published on the site (Privacy, Terms, Cookies).
* A Web Application Firewall (WAF) sits between visitors and your application. It inspects HTTP traffic and can block or challenge requests that match your rules — useful against abuse patterns such as scraping floods and other unwanted automation — while allowing legitimate traffic through.
Thanks
Grateful thanks to Gavin King and the Known Agents work (evolved from Dark Visitors) for helping keep operator-facing bot information available and useful. Bot Rules also uses Cloudflare Radar directory data where identities have been linked for verification context — that is about identity confidence, not a behavioural verdict on its own.
If you need something heavier on the server itself — Nginx bad-bot blocking, spam referrers, scanners, Fail2Ban jails, and similar — see this broader GitHub project: Nginx Ultimate Bad Bot Blocker.
Disclaimer
Bot Rules is provided as-is, without warranty. Always review generated rules against your own traffic and policies before applying them. Blocking the wrong agent can break search, previews, monitoring, or features you still rely on. Use at your own judgement. See the terms of use on the site.