The Nginx Bad Bot Blocker and Referrer Spam Blocker for Nginx Web Servers is now available for free from Github. This centralized Nginx script will strengthen your Nginx server defenses against web spammers, content scrapers, image thieves, pornography web sites, bad referrer spam, spam referrers, spy web sites, wordpress theme detectors, sites with viruses, ransomware and malware and also fake web site hits. It also has a rate limiting built in for aggressive bots and additionally an anti DDOS filter.
Since inception the blocker has advanced as each day goes by and it is updated almost daily which assures you always have the latest protection. It now protects against well over 4000 bad domains, bad user agents and bad IP addresses and is fast becoming the most popular Nginx Bot Blocker out there.
The blocker now includes a script to facilitate those seeking a fool proof installation process and it also now includes an auto update script which can be run as a daily cron ensuring your Nginx server is always up to date with the latest build release. I introduced Travis CI testing too which also ensure that all new builds and releases are tested and pass with TravisCI. Try it, you won’t be dissapointed and you’ll wonder how you ever did without it !!!
For Nginx Web Server – https://www.nginx.com/
Update: 2016/10/22 – All known domains, referers and IP ranges being used in the new Mirai BotNet Attack have been implemented into the Nginx Bad Bot Blocker as of this morning and will be kept up to date.
Update: 2016/12/04 – Added the creation of a Google Disavow File (google-disavow.txt) and a Google Exclude file (google-exclude.txt). The Google Disavow file can be used to block all spam domains using Google’s Webmaster Tools so they are not considered as incoming links to your site. The Google Exclude File can be used to create filters or segments in Google Analytics to exclude all the spam domains from showing up in your Analytics.
Why? …. because all files located in /conf.d/ are automatically loaded by Nginx in the main nginx.conf file.
- Bad Referers
- Spam Referers
- Spam bots
- Vulnerability scanners
- E-mail harvesters
- Content scrapers
- Aggressive bots that scrape content
- Bots or Servers linked to viruses or malware
- Government surveillance bots
Bots attempt to make themselves look like other software or web sites by disguising their user agent. Their user agent names may look harmless, perfectly legitimate even.
For example, “^Java” but according to Project Honeypot, it’s actually one of the most dangerous BUT a lot of legitimate bots out there have “Java” in their user agent string so the approach taken by many to block “Java” is not only ignorant but also blocking out very legitimate crawlers including some of Google’s and Bing’s and makes it very clear to me that those people writing bot blocking scripts seldom ever test them.
Unfortunately most bot blocker scripts out there are simply copy and pasted from other people’s scripts and made to look like their own work. This one was inspired by the one created by https://github.com/mariusv and I contributed to that project but went off into a totally new layout, cleaned it up big time and started from scratch. It is now a completely independent project. It’s clean, it works and has been thoroughly tested.
Welcome to the UltimateNginx Bad Bot Blocker and Referer Blocker for Nginx Web Server with and Anti DDOS system
This bot blocker list is designed to be a global Nginx include file and uses the Nginx map $http_user_agent, map $http_referer and geo $validate_client directives.
This way the .conf file is loaded once into memory by Nginx and is available to all web sites that you operate. You simply need to use an Include statement in an Nginx vhost conf file.
My methods uses no complex regex other than the Name of the Bot. Nginx case matching will do the rest. You can use Regex if you like but it’s NOT needed and I proved it by testing with the Chrome extension User-Agent Switcher for Chrome. (handy util and a must for everyone to test these kinds of blocking scripts)
- The user agent “Aboundex” is found without using “~*Aboundex” … which means a case insensitive match and is much simpler for anyone to maintain than other lists using complicated and messy Regex patterns.
- If we have a rule, like “~*Image\ Stripper” and a bot decides to change its User-Agent string to “NOT Image Stripper I Promise” he is picked up regardless and blocked immediately.
I only capitalise bot names in my list for ease of reading and maintenance, remember its not case-sensitive so will catch any combination like “Bot” “bOt” and “bOT”.
For those of you who SUCK with Regex this is your saviour !!!
The beauty of this is that it is one central file used by all your web sites. This means there is only place to make amendments ie. adding new bots that you discover in your log files. Any changes are applied immediately to all sites after a simple “sudo service nginx reload”. But of course always do a sudo nginx -t to test any config changes before you reload.
The file is tiny in size. At the time of this writing and the first public commit of the Nginx bad bot blocker file size including all the commenting “which nginx ignores” is a mere 58 kb in size. It is so lightweight that Nginx does not even know it’s there. It already contains hundreds of entries.
Unlike many other bad bot blockers out there for Nginx and Apache where people simply copy and paste lists from others, this list has been built from the ground up and tested thoroughly and I mean thoroughly. It comes from actual server logs that are monitored daily and there are at least 3-10 new additions to this file almost daily.
It has also been throughly tested for false positives using months of constant and regular testing and monitoring of log files.
All web sites listed in the bad referers are checked one by one before they are even added. Simply copying anything that look suspicious in your log file and adding it to a blocker like this without actually seeing what it is first …. well it’s foolish to say the least.
Nginx has a lovely error called 444 which just literally drops the connection. All these rules issue a 444 response so if a rule matches, the requesting IP simply get’s no response and it would appear that your server does not exist to them or appears to be offline.
For bot’s or spiders that you still want to allow but want to limit their visitation rate, you can use the built in rate limiting functions I have included. The file is extensively commented throughout so you should figure it out otherwise simply message me if you are having problems.
FEATURES of the Nginx Bad Bot Blocker:
- Extensive Lists of Bad and Known Bad Bots and Scrapers (updated almost daily)
- Alphabetically ordered for easier maintenance
- Commented sections of certain important bots to be sure of before blocking
- Includes the IP range of Cyveillance who are known to ignore robots.txt rules and snoop around all over the Internet.
- Your own IP Ranges that you want to avoid blocking can be easily added.
- Ability to add other IP ranges and IP blocks that you want to block out.
Usage: recommended to be saved as /etc/nginx/conf.d/globalblacklist.conf
The configuration instructions for the Nginx Bad Bot Blocker are below !!!!
Please understand why you are using the Nginx Bad Bot Blocker before you even use this. Please do not simply copy and paste without understanding what this is doing. Do not become a copy and paste Linux “Guru”, learn things properly before you use them and always test everything you do one step at a time.
Make sure to add all your own IP addresses the white list section near the bottom of the globalblacklist.conf file for the Nginx bad bot blocker !!!!
MAKE SURE to monitor your web site logs after implementing this. I suggest you first load this into one site and monitor it for any possible false positives before putting this into production on all your web sites.
Also monitor your logs daily for new bad referers and user-agent strings that you want to block. Your best source of adding to this list is your own server logs, not mine.
Feel free to contribute bad referers from your own logs to this project by sending a PR. You can however rely on this list to keep out 99% of the baddies out there.
sudo mkdir /etc/nginx/bots.d
- copy the blockbots.conf file into that folder
- copy the ddos.conf file into the same folder
sudo nano /etc/nginx/nginx.conf
limit_req_zone $ratelimited zone=flood:50m rate=90r/s;
limit_conn_zone $ratelimited zone=addr:50m;
PLEASE NOTE: The above rate limiting rules are for the DDOS filter, it may seem like high values to you but for wordpress sites with plugins and lots of images, it’s not. This will not limit any real visitor to your WordPress sites but it will immediately rate limit any aggressive bot. Remember that other bots and user agents are rate limited using a different rate limiting rule at the bottom of the globalblacklist.conf file.
Open a site config file for Nginx (just one for now) and add the following lines
Make sure to edit the globalblacklist.conf file near the bottom there is a section to whitelist your own IP addresses. Please add all your own IP addresses there before putting this into operation.
sudo nginx -t (make sure it returns no errors and if none then) sudo service nginx reload
- This Nginx Bad Bot Blocker is free to use and modify as you wish.
- No warranties are express or implied.
- You use this entirely at your own Risk.
- Fork your own copy from this repo and feel free to change it to your needs or contribute to it.