Protect your Magento store from malicious bots and automatic SPAM

Protect your Magento store from malicious bots and automatic SPAM

What would you say if you find out that you have traffic twice bigger than the one you see in your analytics data? And the most intriguing fact is that for each human visitor on your site, you have 1 bot visitor. Quite shocking fact, as we always knew that bots crawl our websites. But the fact is that, for the last 3 years, bots have been dominating website traffic. The good news is that 2015 was a better year for people and therefore for every store owner. People are now dominating bots again with 51% part in website visitors according to Incapsula.

Sometimes your store is crawled by good bots such as Google spider, which will crawl your site in order to show it later on search results pages. However, bad bots are much more popular nowadays. According to 2015 bots traffic report provided by Incapsula, 30% of your visitors were malicious bots for the same year. Quite shocking - yes that is true, every third visitor to your site is here with bad intentions. Before starting to panic, let’s find out more facts about bots.

Our friends: Google bots, BIng bots and other

1 of every 5 visitors to your sites are still Good bots. Probably the most well known is Google bot aka “spider”. Googlebot was created in order to quickly crawl billions of web pages and collect required information in order to improve Google search results.

Quite many people still think that Google Bot is the only one spider from Google. But that isn’t true. There are around 10 different bots released by the search giant that can crawl your site. For example, if are participating in Google Adwords, then you can also see Google Adsbot at your store. Complete list of all Google bots can be found at Google official page.

But let’s focus on the most popular and common Google bot. Google Web search bot is a good bot so it will obey all instructions placed in a robots.txt file. But still, Robots.txt instructions are directives only, so you simply ask good bots to behave well. A robots.txt file that you can indicate those parts of your site you don’t want to be accessed by search engine crawlers. If you are interested in gathering more detailed information, the best place to find it is the official Google webmaster documentation.

The important fact is that Magento is missing a default robots.txt file so you have to create one. The good place to start is using Astrio sample robots.txt.

Googlebot will also download your store sitemap if it was configured at the Google search console. Its main activity will be of course crawling your site pages using internal and external links and the provided sitemap.

If you want to check Googlebot activity details, you can always do that with the Google search console, formerly known as Google Webmaster tools. By using this Google service, you can always check if Googlebot has any errors accessing your site, how many pages and data were downloaded from your site, find out what pages have errors, test changes to your robots.txt file and much more.

Google is dominating search market but it isn’t only one player here. So each search engine has its own bot that will do the exact same things as Google spider does. So in your server access logs, you will also find records of Bing, Baidu, Yandex and even Facebook.

These are big players. Nowadays, many of webmaster are also using a few SEO and security cloud services that are also using spider technology so your site can be also crawled by Semrush, Ahref or McAfee bot.

Our enemies: SPAM bots, content scrapers, DDoS bots and much more….

Hacking bots

One of the most popular and most dangerous usages of bandit bots is the hacking of different sites. Usually, these bots are trying to steal credit cards numbers, personal information or inject malware into your site. Sometimes they are also used to bring your store down or remove important content from your site. The main risk of being hacked of course is that your customers can lose confidence in your store security and will afraid to place order on your store.

So what is the problem with Magento? Magento is the most popular e-commerce platform with more than 240 000 clients and that creates both benefits and disadvantages. For the last 2 years, we have seen releases of multiple security patches for Magento 1.x. The most serious problems such as SUPEE-5344, SUPEE-6285, SUPEE-6482 allowed attackers to upload their content to the server or even take full control over Magento stores. The same story happened again with release of Magento 2.x.

Using these security holes lead to massive site hacking and injection of malware. For example one of the most large-scale attack caused injection of malware to thousands sites. Serving thousands of clients worldwide, we have faced traces of hacking attempts on almost 1 of 10 sites our developers had access to. And the real situation is even more dangerous. According to byte.nl investigation, 80% of all Magento stores are still open to many attacks.

So how can we protect from this kind of bots? Fighting hacking bots is a complex task. The first and most important step is keeping your site software and modules up to date and constantly looking for news about new security holes in your software. Complete Magento security manual is available at Magento security center. If you are interested in more detailed information about keeping your Magento secure, you can check our 10 pieces of advice to keep your Magento store admin secure.

Also, last but not least, an efficient step is using software firewalls such as CloudFlare, Akamai or Incapsula.com. Such kind of services are usually provided as services and the don’t require you to install any software on your server. For example, CloudFlare displays a set of special software firewall rules aimed to stop all the most common attacks to your Magento store.

Spam bots

Bad bots became part of the black hat SEO a long time ago. They are used to submit spam comments, reviews and contact forms filled with spammy links and spam messages. They will try to turn your site into a link farm and show you visitors malware or phishing links. The main risk here, except ruining your online business reputation, is, of course, getting the manual or automatic penalty from Google. According to Google webmaster manual, having such links on your store can cause in getting your site blacklisted in a search engine and it will simply lead to business annihilation. Recovery from such penalty can last months and that means simply death for the most part of online retailers.

Spam bots will also be responsible for creating fake accounts at your store using hacked emails from popular email providers. That will make your email marketing almost impossible as your email bounce rate will be increased greatly and that will make email campaigns analysis almost impossible.

Recent trends in using spam bots got even further. If you have some dishonest competitor in your niche, then you should also be prepared for getting a large number of fake orders. You can find out more details in a real story from Ticketmaster attacked by spam bots making more than 200 000 fake tickets orders per day for the most popular events

Such kind of bots are more easy to fight compared to automatic hacking attacks. First of all, your should start using simple tools such as CAPTCHA. The good news is that Magento does have a basic CAPTCHA support. If you are interested in more user-friendly ReCAPTCHA from Google, then stay tuned as we are going to publish a ReCAPTCHA module next week.

The second step is, of course, reviewing all user-generated content at your site. We strongly recommend moderating all type of content your users submit to you. It could be customers reviews and products questions and even customers testimonials at your site. Every piece of customer-generated content should be moderated by your editor. In order to automate that process, you can also use Akismet. Akismet Magento module integration is part of our Products questions module. Akismet was created by the same company that brought us Wordpress and helps to remove almost 99% of all spam content submitted to your store.

An additional step is using ours Magento Bot protection extension. Usually spam comments are still posted from the same server so it’s quite easy to blacklist that one IP and stop spam at your store.

Content scrapers

This bad bots will try to benefit from all your hard works. Content scrapers will try to retrieve all useful content from your sites such as article content, products price and description and personal information of your site visitors. Scrapers bots will crawl both your sites pages and RSS feeds and can create duplicate pages at the exact same time you publish a new article or add a new product.

You might not even notice for months that your site has been attacked by scrapers. However, ignoring that problem can bring you serious issues. Google and other search engines are not good in the definition of content authorship and having copies of same content on dozens of scrapers sites can lead to multiplying duplicate content even further or low content quality penalties. And that will lead to disappearance of your site from search engines results.

Fighting such kind of activity requires different steps. First of all, you can consider using Copyscape service. This search will help you to check your site content page by page and find out if there are any duplicates on other websites. For large store owners, we recommend using Copyscape premium API that allows checking thousands of pages per day.

But what should we do with copies of our site? The first step is, of course, to try contacting the webmasters by using the contact information provided on the site or the information provided in domain contact records. The truth is that chances to get the reply from such site owners are minimal so you should most likely rapidly go to your next step, which will be submitting DMCA removal request to Google. After a few weeks, a page with stolen content on scrappy site or even an entire script site can disappear from search results.

That will certainly not stop other content scrapers and your content will be regularly stolen and used on other sites. So the final protection for your store should be to deny access to scrappy bots using web server configuration at your server. If you are using Apache then a good place to start is using rewrite rules instructions placed in your .htaccess file.

Price monitoring bots are another kind of content scrapers. This bots will not publish your content on scraper sites or private blog networks but they will try everything possible in order to bring your sales down. This happens because digital economy has changed forever and now your competitors can have all information about your products prices and promotions without any additional cost.

This all happens because now scraping and monitoring rivals prices doesn’t require any serious technical skills and can be done by the sole store owners. By using tools such as Scrapy or EasySync, your rivals can always set their prices just 1-2% below yours and therefore win the price battle.

The most serious price battles usually occur on Black Friday and other sales days. During these days, prices on popular products can be changed dozens of times per day. An interesting example of such price fluctuation can be found at. Rivals prices on a same product are just one of the factors for these changes, but no one can be sure that competitors are not trying to get more market share by using prices information scraped from your page.

Click fraud

Click fraud bots can cause more or less harm to your store unless you have Adwords block in your store. This types of bandit bots usually greatly reduce the efficiency of your PPC advertisement by clicking on all your ads. This bots can be sometimes supported by publishers but also by your rivals. That will simply lead you to spend all your advertising budget without getting real visitors. And of course, that will reduce your advertisement quality score and simply decrease the efficiency of your PPC campaigns. According to 2015 reports, almost every second dollar invested in online advertising is stolen by click bots. Unfortunately, you don’t have any control over such kind of bots and you can only rely on the algorithms of Google or Facebook that are constantly trying to fight click bots.

DDos Attacks

This is another destructive way of using bad bots against your business. In that case, malicious bots will not try to hack your site but simply will send you an enormous amount of request from hacked servers and desktop computers. The biggest attacks in 2015 sent more than 300 Gigabits per second to target servers. According to 2016 Q1 Akamai research, we can notice a 125% increase in number of DDos attacks. Such attacks usually last for 1-2 days as each hour of such type of attack can cost thousands of dollars. But it can be enough to destroy your online reputation and send your customers to your rivals. If you are looking on more information on that problem, then a good place to start is this article

Defending from DDoS is something that was very difficult and required a set of technical skills 10 years ago. Hardware solutions created for that purpose usually cost ten thousand dollars and were simply not available for small and medium-sized business. But with the increase of DDoS attacks, many security companies also started offering affordable cloud-based services. So before your store got down because of a DDoS attack, you should consider starting using anti-DDoS services such as CloudFlare, Incapsula or Akamai.

Tell about tool that can help you to protect

So let’s summarize all the given advices in order to protect your store from bads bots activity.

Block bad bots via webserver configuration

It&rsquo's the easiest way to protect your store from bad bots without putting any additional budget on it. All you need is analyzing your store access logs using text editor or logs analyzer tools and then block bots IP in your web server configuration file. In case you are an Apache user, then you can simply add it to your .htaccess file.

You can also find popular instructions for htaccess files that should block most popular bots

You should be really careful modifying that file and blocking access to your site. In case you have applied wrong instructions, you can block access to your customers or good bots and that can lead you to disappear from search results pages. So after applying changes to that file, you should try accessing your site as Google bot in Google search console.

Finding bad bots can be automated by using different custom solutions such as Black hole bot trap.

CAPTCHA

The easiest method to stop SPAM bots from submitting fake reviews and creating fake user accounts in your store. You can simply enable CAPTCHA at admin > system > configuration > customer configuration settings. The main disadvantages of using the old traditional CAPTCHA is that your site visitors will simply hate it. According to report from Casey Henry published several years ago, disabling CAPTCHA increased his conversion on 3%.

The best advice, for now, is to replace default Magento CAPTCHA with the new Google ReCaptcha service. It can be done by using third-party modules. You should also notice that Magento doesn’t support the default CAPTCHA for the contact form and you should install third party modules.

Protect your store using software Firewall

This is one of the easiest steps to implement. All you need is to sign up for cloud computing services from one of the most popular providers:

These security services are now evolving into the most comprehensive and effective security software. They will not just protect your from the most common attacks but will also help you to fight DDoS attacks, stop content scrapers and even make you faster with built-in CDN services.

Block bad bots using Magento extension

In order to make the first easy step to protect your store from malicious bot, we created Bot protection extension. Our module provides a secure protection from unwelcome visitors like spam bots, crawlers, spiders and other kinds of robots which try to steal your website content. After the visitor was identified as a bot by analyzing user agent string, he will be forwarded either to blacklist immediately or to suspicious list response page with captcha field shown. In order to avoid banning Google and Bing bots, Bot Protection module comes with an included whitelist IP option. Our module is based on the default Magento Log Visitors functionality.

Monitor your site content copies

Start using Copyscape.com in order to get notified when your content got stolen. The sooner you will find out about stolen content and you will take action against stolen content, the less damage will be done to your store.

Don’t feed malicious bots and get benefits for your store

As soon as you will start using this tools in order to stop malicious bots, you will not only make your store more secure and improve your online reputation, but your store will also get faster. As soon as you will block bandit bots, your server resources will get free for real visitors. Also, a secure site with unique content will perform much better in search results.

Have you experienced problems caused by bad bots? If you have effective new solutions or a real story, please share with us. It will be very valuable for all of us.

Popular Magento Extensions

One thought on “Protect your Magento store from malicious bots and automatic SPAM”

Leave a Reply