Bot/DDoS Protection
Bot/DDoS Protection is the DDoS and bot protection module of TR7 that works at the application layer (L7). It evaluates the traffic coming to a vService from three separate angles before it reaches the Backend Services: how much is coming, where it is coming from and who is sending it. Each of the three questions is answered by a separate protection; the decisions are combined in a single chain and at the end of this chain the request either passes, is directed to a verification, or is stopped.
The module does not consist of a single threshold. The volume limits, the reputation lists and the bot score work independently of each other; each one stays in its own mode, carries its own rule list and recognizes its own exemptions. The enforcement layer, on the other hand, is common: whichever protection makes the decision, Captcha, Quarantine and Waiting Room use the same common infrastructure.
The protection is not a single module that is switched on and off across the device: it is enabled separately for each vService. At the moment of enabling, the configuration of all three protections is created, but they do not all start with the same mode: the volume and behavior protections open with the monitor mode, and the reputation protection directly with the enforcement mode.
The inspection is not an independent calculation made per request. The volume measurement accumulates in counter tables that age with a time window (some tables are kept per source, some with a single key for the whole vService); the reputation decision is a single table read; the bot score, on the other hand, is recalculated for every request. The three results are combined on the same request, are reduced to a single decision and are written to the event log with the same result vocabulary. The enforcement is applied within the traffic engine itself and does not create an additional network hop; only the requests of the challenge page itself go to a separate component bound to a local socket.
It works on the vService
The Bot/DDoS protection works on the vService that meets the traffic: after the request is met by the vService and TLS is terminated, it passes through this inspection before the target is selected. For this reason, the protection is enabled on the vServices of the HTTP and L7 TCP types; since Network (Layer 4) traffic passes through a separate data plane, it cannot enter the application layer inspection.
Its Place in the TR7 Product Structure
Bot/DDoS Protection corresponds to two parts in the product structure of TR7: bot management is within the scope of the TR7 WAAP product, and the L7 DDoS protection is the L7 DDoS Protection add-on. The standard L7 DDoS protection scope comes with the product license; the add-on extends the protection to more vServices. The licensing details are available on tr7.com.
The protection works within the traffic engine of the vService. There is no separate component to be installed, no separate service to be started and no separate device to be routed to; when the box in the vService form is checked, the configuration is added to the same engine.
The protection does not carry a separate quota for how many vServices it can be enabled on. The limits are within the module itself and are per configuration: such as at most six advanced notification triggers per vService and a Quarantine list of 5,000 records.
Changing the mode, the limit or the rule list is applied without interruption: the open connections are not dropped and no separate publishing or restart step is run.
What Is Bot/DDoS Protection?
The unwanted traffic directed at a web service does not come in a single form. Sometimes it is a volume problem: a single source or thousands of sources send more requests than the service can bear. Sometimes it is a source problem: the address that sends the request has been marked earlier with an attack, scanning or abuse record. And sometimes it is a behavior problem: the number of requests is normal, the source is clean, but the one sending the request is not a browser but a piece of software that imitates a browser.
These three problems do not have a common solution. A limit that solves the volume problem cannot catch a content collecting software (scraper) that sends requests one by one. A reputation list stays silent against a new botnet that is not on the list. And the bot score alone is not sufficient in a distributed flood in which each of thousands of different addresses sends only a few requests.
Three Separate Questions, Three Separate Protections
It measures the volume of the traffic. It counts the number of requests, connections and errors that a single address or all the sources together produce within a certain time window; in addition, by monitoring the number of different addresses that produce traffic at the same time, it catches distributed request floods. It looks at the number, not at the identity.
It looks at the history of the source. If the address falls into a category in the continuously updated reputation data, the rule list applies the enforcement bound to that category. The decision is based on a single table read: it does not require a time window like volume or a calculation like a score, and for this reason it is both the cheapest layer and the layer that is valid from the first request of the traffic on.
It looks at the behavior of the client. It collects many signals, from the TLS fingerprint to the header order and from the request pattern to browser consistency, in a bot score. Even a single request is scored: independently of the volume and the list, it answers the question "is this a browser?".
Behavioral Analysis
Two of the three protections look at a single moment of the request: the volume counter says how many requests came in that window, and the reputation list says which category the address falls into. Both of them measure not what the client does but what it is.
Behavioral Analysis is one of the factor groups of the bot score and closes exactly this gap: it looks not at a single request of the client but at its pattern over time. Even a client whose every request looks flawless, whose headers are complete and whose fingerprint belongs to a real browser gets points from this group when it shows the following behaviors:
- sending requests to the same path at a constant rate throughout a certain window,
- trying addresses that cannot be found one after another,
- reaching for paths that look for a configuration file and a backup file.
The other factor groups say what the client is, and this group says what it does. The signals that catch well-hidden content collecting software and scanners mostly come from here; since the signals of the group weigh heavily on the maliciousness axis, one of them alone can push a client to the top band.
The scoring has two axes
The bot score is not a single "is it a bot or not" measure. Two questions are measured separately: is this a machine? and is this hostile? A monitoring system or a verified search engine robot stays high on the first axis and low on the second; a client that carries no harmful sign but looks for a configuration file shows the opposite. For the score class to be Bad Bot is possible only by passing through the maliciousness axis.
The full list of the factors and the weight of each one is on the Factors page.
The Difference from WAF
The Bot/DDoS protection and WAF work at the same layer, see the same request and are frequently enabled together on the same vService. However, they answer different questions:
| Bot/DDoS Protection | WAF | |
|---|---|---|
| Question | How much, from where and from whom is the request coming? | Is the content of the request harmful? |
| What is looked at | Counters, source reputation, client behavior | The content of the path, the query, the header and the body |
| What it typically catches | Request flood, scraper, credential stuffing, botnet | SQL injection, XSS, file path attack |
| Decision time | Before the content is opened | After the content is parsed |
| vService type | HTTP and L7 TCP | Only HTTP |
The two layers do not take the place of each other
A request with clean content that is repeated thousands of times per second is not a problem for the WAF; for Bot/DDoS it is an indicator of a flood. With a harmful payload that is sent only once, the situation is reversed. When the two modules are enabled together on the same vService, both of them evaluate the same request and they do not take the place of each other: one questions the content of the request, the other the source and the frequency of the request.
Bot/DDoS in TR7
In TR7 these three protections are gathered under a single module and are managed from a single management screen. The module has three design decisions:
- The protections are independent. Every protection carries its own mode. While one applies enforcement, another can only monitor and the third can be completely off. There is no common mode switch that turns all three at once; the whole module, on the other hand, is switched on and off with a single box in the vService form.
- The enforcement layer is common. All three protections use the same Captcha, the same Quarantine and the same Notifications infrastructure. Which protection put a source into Quarantine is written in the record, but the list is single.
- The decisions are combined in a single event log. Whichever protection makes the decision, the result is written to the same event log with the same result vocabulary. Three separate places are not looked at in order to understand why a request stopped.
Why Is It Needed?
- Availability. A request flood can put the service out of service without using any vulnerability. The limits keep the load passing to the backend in a predictable band.
- Resource cost. Content collecting software (scraper) and automatic clients consume bandwidth, database connections and cache space even if they do no harm at all.
- Account security. Credential stuffing and trial-and-error attacks consist of requests that look ordinary when they are looked at one by one; however, the source-based error counter and the bot score catch them together.
- Content and price protection. Automatic collection carries content and price data that is open to competition out in bulk.
- Demand burst. Not every pile-up is an attack. The legitimate density that occurs after a campaign or an announcement is managed by being put in a queue instead of being blocked.
How Does It Work?
A request has a single decision
The same request can fit the criterion of more than one protection; however, the enforcement is applied only once. Which protection will make the decision is resolved with a fixed priority order: L7 DDoS → IP Reputation → Bot Protection. For this reason, every request has a single decision in the event log and the protection that made the decision is also written in the record.
The order is independent of whether the protections are on or off: if the protection above is off, the decision passes to the one below it.
The monitor mode is a rehearsal mode
A protection in the Monitor mode makes the whole calculation, produces all the decisions and writes all of them to the event log; however, it does not stop any request. In the records these decisions appear with the Would… stamp. When these stamped lines are read before the mode is taken to enforcement, which traffic will stop when enforcement is started is seen in advance.
Three Protections
L7 DDoS Protection
It limits the traffic over the volume. It carries eight separate controls; every control keeps its own metric, its own limit and its own enforcement and is switched on and off independently. The controls that measure a rate also carry their own time window.
Seven of the controls ask the same question at different scales: how much? Some of them monitor a single address (the number of requests, connections and errors), and some of them total all the traffic (the total requests and connections). The eighth one, on the other hand, stands on a separate axis and asks from how many places?: since it measures the number of different addresses that produce traffic at the same time, it catches a distributed flood in which each of tens of thousands of sources sends only a few requests. In this picture no source-based limit is triggered and the total limits look the same as legitimate campaign traffic.
The country and network type controls, on the other hand, can also work without looking at the number at all: when no limit is written on a rule row, the row turns directly into a membership rule.
The four controls that measure a rate work by default in a 10-second window: 1000 requests and 100 connections for a single address, 10000 requests and 5000 connections for the whole vService. When the request limits are exceeded, Captcha is applied, and when the connection limits are exceeded, Silent Drop. The control that measures the number of different addresses producing traffic at the same time carries not a rate but a count: its default threshold is 10000.
In the source IP and total request/connection controls, a second tier above the soft limit can be defined. This hard ceiling is empty by default and its enforcement cannot be selected: in the source IP controls Quarantine is applied directly, and in the total controls Silent Drop. In the error counter and in the active source and geographic controls, the hard ceiling field is not present.
IP Reputation Protection
It looks at the reputation of the source. The reputation data that is kept across the device and refreshed regularly separates the addresses into categories such as attack type, abuse form and network quality. The protection carries a rule list: which categories will be bound to which enforcement is defined row by row and the first matching row wins.
The category list consists of 25 entries: 23 threat categories plus the operator's own blacklist and the external feed sources they define. The operation of all three is the same, their difference is where the data comes from.
This is the exception among the three protections: when the protection is enabled, it starts directly with the Active mode, it does not pass through the monitor mode. The default rule list applies enforcement from the first day on: eight attack categories together with the operator's own list are taken to Quarantine, and the remaining categories except vpnIp are blocked. The practical result of this is that a part of the traffic can start to be blocked at the moment the protection is switched on.
Bot Protection
It looks at the behavior of the client and produces a bot score between 0-100 for every request. The score comes not from a single signal but from the evaluation of the factors gathered in eight groups together: the fingerprints at the connection and TLS level, the presence and the order of the headers, the quality of the address, the request pattern and the consistency between the information with which the client introduces itself and its behavior. Three of the factors are a trust signal that lowers the score: the score of a client that has logged in, solved the Captcha or shown real interaction is pulled down.
Which factors will enter the score can be selected; when the selection is left empty, all the factors are considered active and the factors added later also enter the scope by themselves. On an HTTP vService 46 factors are evaluated; on L7 TCP the 17 that do not require application layer information work.
The score is separated into four classes: 0-39 Clean, 40-59 Suspicious, 60-84 Bot, 85-100 Bad Bot. The rule list determines from which threshold on which enforcement will be applied; the default list applies blocking from 90 points on and Captcha from 75 points on.
There is a separate trust switch for the verified search engine robots. The verification is based not on the name with which the robot introduces itself but on the address range: a client that introduces itself as a search engine but comes from outside the verified ranges is not kept exempt.
Operating Modes
Each of the three protections works in one of the same three modes and the modes are selected independently of each other.
No measurement is made and no record is produced. This value is always a conscious decision of the operator and the learning assistant does not break this decision with its recommendations.
All the limits are measured, all the scores are calculated and all the decisions are written to the event log; however, no request is stopped. It is the rehearsal mode before passing to enforcement.
The rule list and the limits are in force: the requests are blocked, directed to Captcha or taken to Quarantine.
The status strip of the Bot/DDoS Management of the vService shows a reading derived from these three modes: if at least one of them applies enforcement, Active, if none of them applies but at least one monitors, Monitoring, and if all three are off, Not Active. This text on the strip is not a setting; the modes are changed only from the bands of the protections themselves.
The Common Enforcement Layer
All three protections apply their decisions with the infrastructure offered by the same four cards. The cards do not belong to the protections: even while not a single one of them is being used, their settings continue to be kept and when a protection starts to direct to that card, it works with the values that have been written.
| Card | Its job |
|---|---|
| Captcha | Instead of blocking the suspicious source, it directs it to a verification. A source that passes the verification is not questioned again for a defined period; a source that continuously fails, on the other hand, can be taken to Quarantine. |
| Quarantine | It is the single list kept per vService and carries at most 5,000 records. A source that is taken in meets the same enforcement on every request until its period expires (30 minutes by default); the box membership takes precedence over the decisions of the protections and the source does not need to catch a rule again. |
| Waiting Room | It is not an attack enforcement: in a legitimate demand burst it limits the number of visitors served at the same time with a ceiling (1,000 by default) and puts the excess in a queue according to the order of arrival. It is present only on HTTP vServices and, unlike the three protections, comes with the Off mode by default. |
| Notifications | It does not touch the traffic at all: it determines which events will give rise to a notification and with which of the four channels (e-mail, SMS, SNMP, interface) these notifications will be delivered. By default only the interface channel is open. |
Learning (Smart Advisor)
The correct value of a volume limit changes from service to service: the same number is loose for one service and narrow enough to cut the service for another. The module does not wait for the operator to guess this value. The learning engine reads the past traffic of the vService, calculates the suitable range for every setting and presents its result as a recommendation.
What Does the Engine Read, What Does It Produce?
The engine monitors the traffic on four axes: source address, country, network type and bot behavior. From these measurements it produces three kinds of output:
| Output | Meaning | Can it be applied? |
|---|---|---|
| Suggestion | A numeric limit or a rule row recommendation. It is the result that the engine both calculates and can apply. | Yes, with a single operation |
| Advice | The value that the engine calculates but whose decision is left to the operator. The Waiting Room capacity and the sudden rise threshold are in this class: the decision to put a visitor in a queue is not left to the engine. | No, it is read only |
| Attention | A finding that is wanted to be looked at. It is not a recommendation and does not enter the pending recommendation counter. | No |
Three Operating Modes
| Mode | Behavior |
|---|---|
| Off | The periodic evaluation is not made at all. When the screen is opened and when a record is edited, an instant evaluation is still run. |
| Smart Advisor | The default mode. The engine learns, writes its reason and recommends; it applies nothing by itself. The evaluation runs at a fixed frequency, every hour. |
| Automatic | The engine applies what it has learned itself in the defined period. When the Tune Limits Only brake is switched on, it touches only the numeric limits and does not touch the rule structure. |
The learning window comes from the Learning Lookback setting and its default is one month. The value entered is rounded to the ready windows in which the engine works: approximately one week, two weeks, one month and two months. In addition, the countries and the network type groups that are considered trusted are listed and kept outside the learning; these lists do not change the learned numbers, they shape the rule structure produced.
The Limits of the Application
The recommendations are selected from the review window on the Learning strip and are sent as a single package. The application is in the form of all or none: if a single selection in the package is invalid, none of them is applied.
The authority of the engine is consciously limited. The server rejects three situations: the enabling of a protection that has been made Off manually, writing over a full rule list and changing the rows that the engine does not own. That is, a rule written by the operator is not silently overwritten by the learning.
The numbers can look full even while the learning is off
What stops in the Off mode is only the periodic run. Since an instant evaluation is made over the settings in force every time the screen is opened, the counters on the card can still look full. There is a second situation that stops the periodic run: if Mute Suggestion Notifications is checked in the Smart Advisor mode, the run is not made at all.
Monitoring and Visibility
The threat, result and status data in the selected time range is gathered in the panels; the configuration of the vService is also summarized as a flow diagram.
Counters refreshed once a second, the result distribution, the rule tables and the live boxes of the cards. It is used in order to see the effect of a setting right after changing it.
Which protection, which rule and which threshold every request was caught by is read at the record level. There is a separate global screen for scanning across the device.
The threat status band, the learning recommendations and the scope counters defined by the operator; they are delivered through the e-mail, SMS, SNMP and interface channels.
Management and Operation
- The protection is switched on per vService. The enabling is done from the vService edit form or in bulk from the Bot/DDoS Protected vService List screen.
- The settings are portable. The whole Bot/DDoS configuration of a vService is exported to a file and loaded to another vService. The import is a full replacement.
- The changes are uninterrupted. A configuration change takes effect without dropping the open connections.
- There is no separate permission area. Displaying the screens requires monitoring permission in the Load Balancer area, and all the edits require change permission in the Load Balancer area.
- The counters are specific to the device. In a cluster installation the configuration is spread to the peer device, but the counter tables are not synchronized: after a failover, Quarantine, the Captcha counters and the Waiting Room start empty on the peer device.
Use Cases
An intense request flood coming from a small number of sources is cut with the source-based limits before it reaches further to the backend. When a hard ceiling is defined above the limit, the insisting source is taken directly to Quarantine and is not evaluated again on every request.
In a botnet in which each of tens of thousands of addresses sends only a few requests, no source-based limit is triggered. The control that measures the number of different addresses producing traffic at the same time catches this picture, because it monitors not the volume but the breadth of the traffic.
A scraper that never catches the limits by keeping its volume low is caught with the inconsistencies in its browser imitation. Since the verified search engine robots are kept exempt from the same rules, the indexing of the site is not broken.
Login attempts look ordinary when they are looked at one by one. The source-based error counter counts these attempts together; the reputation data, on the other hand, catches the same source on the first request if it has also been marked at other institutions.
The pile-up that comes at a ticket sale or a campaign opening is not an attack. The Waiting Room limits the number of visitors served at the same time with a ceiling and puts the excess in a queue: nobody is rejected and the backend sees a constant load.
A service that serves only certain countries limits the traffic coming from the other countries directly with a membership rule. Anonymizing networks and data center ranges are also bound to separate rules in the same way.
When the defined scope counters exceed the threshold, a notification is produced; from the event log the source of the attack, the protection that made the decision and the rule triggered are examined and the source is taken to the exemption or the blocking list from the same window.
The configuration that has matured on one vService is exported to a file and loaded to similar services; the same tuning process is not lived through from the beginning for every service.
Terminology
| Term | Description |
|---|---|
| Off · Monitor · Active | The operating mode of a protection. The three protections carry their own mode separately, there is no common main switch; the distribution of the modes across the vService is monitored in the Protection Modes panel on the dashboard. |
| L7 DDoS Protection | The protection that measures volume. It carries eight controls; the ones that measure a rate keep their own time window and all of them keep their own limit. |
| IP Reputation Protection | The protection that looks at the category of the source in the reputation data. It is the only protection among the three that opens directly with the enforcement mode. |
| Bot Protection | The protection that looks at the behavior of the client and produces a bot score. |
| Action Rules | The ordered list of a protection that consists of condition and enforcement rows. The first matching row wins. In the country and network type controls of L7 DDoS the same structure is called Rules. |
| Hard Ceiling | The second tier above the soft limit. It is empty by default and its enforcement cannot be selected: Quarantine in the source IP controls, Silent Drop in the total controls. The field is present only in these four controls. |
| Bot Score | The number between 0-100 that summarizes the probability of the client being automation. It is measured on two axes: is this a machine? and is this hostile? |
| Clean · Suspicious · Bot · Bad Bot | The four classes into which the bot score falls. They correspond to the 0-39, 40-59, 60-84 and 85-100 ranges respectively; the Bad Bot class is reached only through the maliciousness axis. |
| Behavioral Analysis | One of the factor groups of the bot score. It looks not at a single request of the client but at its pattern over time. |
| Verified Bot | A search engine or AI crawler verified from its address range. Its exemption from the bot rules and the IP reputation rules is open by default; the exemption from the volume limits, on the other hand, is provided with a separate switch on the L7 DDoS side and is off by default. |
| Captcha | The verification step offered to a suspicious source instead of blocking. In the action list it is called Show captcha. |
| Quarantine | The list, single per vService, in which the sources taken into quarantine are kept. The panel title of its live table is Quarantine Penalty Box. |
| Waiting Room | The capacity tool that limits the number of visitors served at the same time and puts the excess in a queue. It is not an attack enforcement. |
| Silent Drop | The closing of the connection without a response being given to the request. It is the equivalent of blocking on L7 TCP vServices. |
| Smart Advisor | The engine that learns from the traffic and produces setting recommendations but does not apply them by itself. It is at the same time the name of one of the learning modes. |
| Would… | The decision that a protection in the monitor mode writes to the event log and that is not applied to the traffic: such as Would Block, Would Show CAPTCHA and Would Quarantine. |
| Advanced Notifications | The scope and window counters defined by the operator. It produces a notification when the threshold is exceeded and does not touch the traffic. |
