Overview of Big Data Defense

Big data DDoS defense is the application of big data technology to the process of detecting and mitigating Distributed Denial of Service (DDoS) attacks. By utilizing cloud-scale compute and storage resources, big data DDoS makes it possible to collect and examine network flow data in highly granular fashion, at great scale and with superior baselining intelligence.  The combination of scale, granularity and baselining smarts contributes to greater accuracy in stopping DDoS attacks than is possible with legacy DDoS defense appliances. 

Big data DDoS defense has evolved because the limiting, scale-up architecture of appliance-based DDoS detection is no longer sufficient to track the massive amount of flow records and individual IP addresses that operationally characterize today’s digital business networks.

The primary goal of a DDoS attack is to either limit access to an application or network service, thereby denying legitimate users access to the services. There are many types of DDoS attack schemes that are used today and they are steadily becoming more sophisticated.  However, their common goal is to overwhelm targeted network resources with traffic or requests for service from many different sources – potentially hundreds of thousands or more. This effectively makes it impossible to stop the attack simply by identifying and blocking a single IP address.  The sheer distribution of attacking sources also makes it very difficult to distinguish legitimate user traffic from attack traffic when spread across so many points of origin.

DDoS defense deploys manual or automated detection, plus one or more remediation techniques including traffic flow filtering, rate-limiting, and blocking/dropping, so attacks can’t consume resources needed to service legitimate traffic.  In order to detect an attack, one has to gather a sufficient amount of data by monitoring network-wide traffic flows, then rapidly and accurately analyze that data.

The Omnipresent DDoS Threat

There is no doubt, as evidenced in the alarming rise of DDoS attacks, that identifying and stopping DDoS attacks is an absolute necessity for businesses that rely on Internet traffic to in order for them to avoid disruption of applications and services, revenue loss, and brand damage.

The majority of DDoS attacks are volumetric in nature, with the intent to congest all of the target network’s bandwidth related to a service or network segment. Roughly 90% of all DDoS attacks are volumetric, with application-layer attacks making up the remaining 10%. According to Akamai’s Q3 2016 State of the Internet Report, the majority of volumetric attacks are IP flood attacks involving a high volume of spoofed packets such as TCP SYN, DNS, UDP, or UDP fragments.

DDoS attacks are rapidly increasing in frequency and size. While mega attacks that last for many hours and reach 200 Gbps or more make the news, the vast majority of attacks last under an hour and are less than 1 Gbps in volume. Smaller attacks often happen without being noticed, though they may be harbingers of larger attacks to come. Mid-sized attacks are more readily felt, but distinguishing between a friendly surge in normal traffic and an attack is key to timely response. Large attacks are fairly obvious, and in these cases diagnosing the traffic is important to understand network entry points and sources. In all cases, a clear detection assessment is important to understand the best most expedient way to mitigate the attack.

In-line versus Out-of-band DDoS Defense

There are two primary means of implementing DDoS defense: an in-line defense appliance or out-of-band detection via traffic flow record analysis that integrates with any mitigation technique, whether on-premise or in the cloud. When performing in-line DDoS defense, all traffic is sent through one or more DDoS defense appliances that support deep packet inspection. The appliances examine the incoming traffic, and if particular traffic flows or packets are determined to be attacks, they are discarded, while legitimate traffic is allowed to pass through.

The basic in-line DDoS defense capabilities of network devices such as load balancers, firewalls or intrusion prevention systems may have once provided acceptable detection and mitigation when DDoS attacks were less sophisticated but high-volume attacks can overwhelm these devices, and sophisticated multi-vector DDoS attacks can often evade them because they are network location specific.

Out-of-band DDoS detection utilizes flow data from NetFlow, J-Flow, sFlow, and IPFIX-enabled routers and switches to understand unusual traffic spikes that occur across the network and detect attacks.  Attack mitigation via in-line devices or routing methods that drop selected traffic is then triggered manually or automatically.

DDoS Mitigation Options

The remediation portion of DDoS mitigation can be accomplished via multiple mechanisms, including dedicated in-line appliances, routing techniques, cloud-based services, or some hybrid of these mechanisms.

In-Line Appliance Mitigation

When performing in-line DDoS mitigation, all traffic is sent through one or more DDoS protection appliances that support deep packet inspection. The appliances examine the incoming traffic, and if particular traffic flows or packets are determined to be attacks, they are discarded, while legitimate traffic is allowed to pass through.

Routing Techniques:  Remote Triggered Black Hole (RTBH)

Black holing is a form of DDoS mitigation achieved by dropping traffic via changing routing parameters. By utilizing BGP to redistribute attack traffic to the null interface on edge routers, multiple types of black holing are possible.  The most common form of black holing is destination-based Remote Triggered Black Hole (RTBH). When a service or website is under attack,  a network operator configures a /32 host “black hole” route. The route is then redistributed via BGP — along with a ‘no-export’ community and a next-hop address — to the routers where the attack traffic is entering the network. These routers then route the traffic to a destination that doesn’t exist (the black hole), for example a null interface. Black holing can be very powerful and effective but with the caveat that legitimate traffic will also get dropped.

Traditionally, the detection of attacks for RTBH was performed by basic alerts, and the triggering of the remote black hole was performed by a manual routing parameter configuration.  However, it is possible to use SaaS-based DDoS detection to automatically trigger BGP-based black holes.

Cloud-Based DDOS Defense Services

Cloud-based DDoS defense services detect and mitigate attacks without requiring the network under attack to deploy on-premises resources. This is accomplished by redirecting all traffic through the DDoS protection cloud provider’s network, where the traffic is most commonly run through in-line DDoS mitigation devices in a scrubbing center.  Attack traffic is detected and removed, and legitimate traffic is routed backed to the customer’s network.    Such services are offered on an “always-on” or “on-demand” basis. While cloud-based traffic scrubbing services are very convenient, they can also be quite expensive.

Hybrid Defense

Hybrid DDoS defense is performed by a combination of on-premises mitigation devices and cloud-based mitigation services. This approach is utilized to enable the fastest response (from the on-premises appliance) with the cost-effectiveness of bursting to the cloud on-demand when mitigation requirements rise above the capacity of the on-premises device.

Evolving DDoS Detection to Big Data

A common thread to historical defense solutions is the use of single-server appliances for DDoS detection.  The first generation of out-of-band DDoS defense solutions accomplished attack detection based on single server software design, mostly running on standalone rackmounted server appliances.  While far better than nothing, single servers simply don’t have the compute, memory and storage resources to track high volumes of traffic data on a network-wide basis.  This is particularly true when attempting to perform dynamic baselining to identify attacks, which requires scanning massive amount of flow data to understand what is normal, then looking back days or weeks in order to assess whether current conditions constitute an anomaly.  Regardless of whether it is deployed on-premises or in the cloud, the single server approach is insufficient to accurately detect today’s attacks in a consistently reliable fashion.  The result is that a significant percentage of attacks sneak through.

The key to solving the DDoS defense accuracy issue is big data. By using a scale-out system with far more compute and memory resources, a big data approach to DDoS defense can continuously scan network-wide data on a multi-dimensional basis without constraints.

Cloud-scale big data systems make it possible to implement a far more intelligent approach to the problem, since they are able to:

  • Track and baseline millions of IP addresses across network-wide traffic, rather than being restricted to device level traffic baselining
  • Monitor for anomalous traffic using multiple data dimensions such as the source geography of the traffic, destination IPs, and common attack ports. This allows for greater flexibility and precision in setting detection policies.
  • Apply learning algorithms to automate the upkeep of detection policies to include all relevant destination IPs.

Of course, constructing an open-source big data system for DDoS detection isn’t trivial.  However, cloud-based/SaaS network visibility and DDoS detection systems are making it far easy to get the advantages of big data for DDoS defense without making major capital and R&D investments.

These advances are making a new hybrid model possible, where DDoS detection is performed by a best-of-breed, cloud service that automates the triggering of RTBH, on-premises and cloud-based mitigation appliances.  Big data detection systems also provide the added benefit of deep, forensic analytics, plus the ability to incorporate network performance, planning and other capabilities.

On Kentik & More Reading

Kentik Detect offers the industry’s only big data network visibility and DDoS defense solution built from the ground up on big data and delivered as a cost-effective SaaS.  Kentik Detect offers the industry’s most accurate DDoS detection, and can automatically trigger mitigation via RTBH, Radware DefensePro or A10 Thunder TPS mitigation.

For more information on how big data delivers 30% greater DDoS detection speed and accuracy, check out the blog post on Big Data for DDoS Protection, read the PenTeleData case study, or download The Case for Big Data-Powered DDoS Protection white paper. Know you want to get big data-powered DDoS protection today?  Start a free trial.