Web SEO and Design Guide: Effectively Block Known Spam Bots from Your Website and Analytics
Before we get started, we need to explain what is an internet bot?
What Is A Bot?
For all of your newbies to the web world, bots are a slang for the term for ‘robot.’ Bots can be a good. For example, they are used by search engines to retrieve pertinent data from your website. However, bots can also be harmful and can be used to create spam to your email as well as fill out web forms.
We Hate Spam!
It’s no surprise, everyone, I mean, everyone hates spam. Not only is spam annoying, but it also can lead to phishing websites or sites that host malware. So, here’s the real question: how do you get rid of these little critters and avoid them? They mess up our analytics! Bots are always going to be one of those nagging problems. You may get rid of some, but then new ones will find you later on. It’s a battle we have to constantly fight. There may never be a time that you can get rid of all of them. Nonetheless, it’s smart to do the best you can and find ways to eliminate them. We have simplified the process so you can get rid of these bots. Learn how to block spam bots from google analytics below:
Step: 1 Identify the bot you want to block
You first must find the origins of these little critters. To do this, you need to locate and find the IP address of the bot. You can find this in your raw web log (this allows you to access your website). The raw web log allows you to see who is accessing your file in a quick way. You will most likely find this through your web host, FTP server.
If you think you have found bots using your raw log, you need to find their IP address. You can use the below external lookup services:
- Inspect an IP at Project Honey Port which knows about many email address harvesting web robots.
- Cybernet Quest IP Address Lookup
- Network Tools Whois Lookup
Here is another way to find individual bots from Google Analytics, go to Reporting at the top of the Header page. Half way down the left column, click on the term Acquisition. Once there, then click on all traffic.
This feature will show you exactly where your traffic is coming from. We are looking for the source of the ‘referrals.’ Click on Source/Medium. This is an exhaustive list of all the sites that are visiting your page.
Once you see an unrecognized, spam address, copy and paste the name onto any search engine to find the IP address. For example, free-share-buttons.com is a well-known referral spam address found in Google. It is probably one of the worst ones and it’s not alone. It’s often accompanied by similar ones such as social-buttons.com. But you may have a hard time finding the IP address. In this case, you can go straight to create a filter on Google Analytics.
Go back to the Admin Tab, go to view then filters and select Filter. Enter Free Share Button.com as the name for the filter. Select Filter Type Custom. In filter field, find and select Campaign Source. In the filter Pattern text box, enter name of spam such as Free Share Buttons.com then press save. Once you are done, you will see the below screen. You have now created a filter to exclude these buttons to your website using Google Analytics. See below.
Another option is to go through your Google Analytics account. Go to your Admin settings then under view panel, you’ll find view settings then click there. Toward the bottom of the options, look for Bot Filtering, check that box.
This feature will filter all the known bots and spiders from your Analytics collecting. The list is compiled from IAB/ABC International Spiders & Bots Lists, which is an independent, industry website well-recognized for finding and maintaining Spiders and Bots List. Check it out at this site: http://www.iab.net/1418/spiders.
Step: 2 Update your .htaccess File
Once you have found the source of your bot, now it’s time for you download the .htaccessfile. Go to your website’s server and navigate to the root level. (likelypublic_html/ ).
After you have found .htaccess, download it your computer then open it with a text editor, such as Notepad in Microsoft Windows, TextEdit in Mac OS X or gedit in Linux. Note: Do not use Microsoft Word for this task.
Notepad in Microsoft Windows, You should see a screen like below.
Step 3: Block That Bot!
To block the IP address, say, 184.108.40.206, go to the bottom of the .htaccess file and add the following:
Order, DenyDeny from 220.127.116.11
Note: You should of course change the numbers “18.104.22.168” to point to the correct IP address you want to block.
Here you are telling the website to deny this IP address (and its bot) from visiting the site. When the bot originating from that IP address tries to visit your website, your server will automatically know to deny access to that bot. If you have more than one address you want to deny access too, you will follow the same code as above. For example:
Order, DenyDeny from 22.214.171.124Deny from 126.96.36.199
Save Your New Updated .htaccess File
Once you’ve finished creating the code to deny the uninvited bots to your .htaccess file then you will want to save the new “.htaccess” file back where you originally found it.
TIP: Use quotations around the file (“.htaccess”) as above or it will be read with the .txt tag or file name instead of a new .htaccess file.
Summary: Block Spam Bots From Google Analytics
So, there you have it! You’ve blocked the bot! Now go have a drink. Man, this web stuff is tough If you need help with your website contact us today for help.
What other methods do you use to “block the bots?” I look forward to reading your comments below!