WAF vs BOT Manager: Do you need both?

We have all heard of Web Application Firewall. What do you picture when you think about WAF? A red coloured wall which protects malicious traffic from getting in. You’re not wrong. Let me tell you what happens inside that red coloured wall. 

In this blog, we will mostly be on the defensive side. We already know how web application attacks are performed, now we will see how firewalls defend it and do we need WAF as well as BOT Manager.

Fun fact - The term firewall originally referred to a wall intended to confine a fire within a line of adjacent buildings.

What is Web Application Firewall?

Well, what do you understand about web application attacks? Attacks like SQL Injection, XSS, Command Injection, etc are some of web application attacks. To protect a website from these kinds of attacks we have a Web Application Firewall in place (between the server and the internet).

Simple, isn’t it? 

How does it work?

A web application firewall is placed in front of web applications that performs a deep inspection of all HTTP traffic inspecting both GET and POST requests, detecting and blocking anything malicious. WAF detects malicious activities based on the kinds of rules that are present in the firewall. Multiple rules for detecting SQLI, XSS, XML Injection, File execution, LFI, etc can be created and are predefined in WAF. This inspection helps the WAF to identify and block threats, preventing them from reaching the server.

Let’s say you are hitting a URL - “https://abc.com/<script>alert(1)</script>”. Do you think this is malicious and it should be blocked? Yes of course this is malicious, when this request is sent to abc.com it first reaches the firewall where there are multiple rules in place for detecting if the request is malicious or not. Let’s say rule XSS is triggered for this, and we have set an action to Deny requests for XSS, the request will be denied.

But.. does it capture all the malicious requests and are we secure??

The most frequent malicious attacks are usually automated (meaning where an attacker is using automated scripts to make continuous requests on the server). These attacks are difficult to detect as they are often designed to mimic human traffic and go undetected.

A lot of companies and security professionals still rely on their WAF for protection against unwanted bot traffic. A WAF will only be able to block familiar threats, such as known malicious user agents and IP addresses, WAFs are not designed to detect real-time automated threats. They will also struggle to recognize the behavior of many of today’s sophisticated bots.

WAFs are generally unable to answer this simple question: “Is the site visitor a human or a bot?”

Here comes BOT Manager..

What is a BOT?

A Bot (short for robot) is a software program that runs repetitive/automated tasks over the internet. Since it is automated, it is operated in a much higher rate as compared to human Internet activity.

There are good bots and bad bots. Good bots such as, Googlebot - is an application used by Google to crawl the Internet and index it for search. Other bots are malicious such as bots used to automatically scan websites for software vulnerabilities and execute attack patterns.

How can the BOT Manager solve this issue?

Bot Manager are designed to identify human vs bot traffic which a WAF cannot differentiate. Bot managers should be able to block malicious bots and allow good bots through, instead of simply blocking all non-human traffic. If all bots are blocked and Google bots aren't able to index a page, for instance, then that page can't show up in Google search results, resulting in greatly reduced organic traffic to the website.

How?

Well, as you read previously, WAF has some predefined rule, that, when triggered, an action to Deny or Alert is set. Likewise, BOT Manager also has some predefined rules and custom rules can also be created. Well the question arises, on what basis is it defined that the visitor is human or bot??

There are different categories for which checks are made for e.g. It checks the protocol version, whether the request originates from a cloud provider, or whether the client wants to keep a connection open. It can also identify bots that are impersonating web browsers (eg. Using a Firefox user-agent but without other headers sent by Firefox). Declared Bots that declare the operator’s website in the user-agent, or self-identify as a specific category, such as aggregator, spider or scraper. It can identify scraper bots based on aggressiveness, persistence, scope(no. of sites impacted) & request characteristics. Some common HTTP libraries that make the request to the site for e.g - Curl, Perl, Python, etc and development frameworks like Node.js, Ruby, Java, etc. 

Bots that also spoof other bots like GoogleBot(good bot) which usually is bypassed but a good BOT Manager can detect all spoofed bots by checking for incorrect/spoofed user-agents & if the traffic source is not from a true service provider. Since we know user-agents can easily be tampered with, but the BOT Manager solution is not just based on one test case, it checks for multiple instances of the issue based on the type of requests, request count, heavy traffic, session validations, etc. These test results are computed and used to qualify the client as a bot.

A good bot manager accomplishes the following goals. It can:

• Identify bots vs. human visitors

• Identify origin IP addresses and block based on IP reputation

• Analyze bot behavior

• Good bots can be added to allow lists

• Challenge potential bots via a CAPTCHA test, JavaScript injection, or other methods

• Rate limit rules for any potential bot over-using a service

• Deny access for "bad" bots

• Serve alternative content/page to bots

What kinds of bot attacks does WAF and bot management mitigate?

Why WAFs aren’t effective against malicious bots?

WAFs are specially designed to protect web applications from attacks that are trying to exploit common software vulnerabilities, such as cross-site scripting (XSS), SQL injection and session hijacking. The WAF first analyzes all incoming traffic, for both GET and POST-based HTTP requests, and applies a set of predefined rules to filter out suspicious traffic with familiar attack signatures.

However, many bots do not target vulnerabilities, and they don’t carry attack signatures. Instead, they are aiming to mimic the behavior of real, human users.

For example, they click on ads, scrape unprotected content, and use stolen credentials to try and log in to protected areas of your site. None of these behaviors will be detected by a WAF. Thanks to botnets, IoT deployments and IPv6, bot operators with malicious intents can easily rotate through hundreds, thousands, or even millions of different IPs to work around such filters. IP-based rules are simply no match for them.

Products?

Now that we know that BOT Manager solutions are important, which product you should go look for BOT Manager protection. There are multiple products in the market that provide BOT Manager protection. You can check Akamai BOT Manager Solution, Cloudflare BOT Management and Radware BOT Manager.

Can there be cases of attacks not detected in any of the WAF/BOT/DOS Rules?

Well, Yes, In this kind of case you first create a visibility rule to understand the type of traffic received. Here mostly your experience on handling such kind of cases comes in picture. You analyse the traffic based on the IP Address, ASNs, countries, TLS hases, User agents, etc. Try to get the pattern on how the requests are being received, if it is from a single IP or multiple ASNs, any common pattern noticed in all the requests. Based on the analysis, take further action.

References

https://datadome.co/bot-detection/what-is-bot-management-how-a-bot-manager-can-help-you-control-bot-traffic/

https://www.imperva.com/learn/application-security/what-are-bots/

https://blog.fabianpiau.com/en/tag/security/

https://www.linkedin.com/pulse/blocking-bots-why-wafs-fall-short-how-fix-jordan-phillippi