Overview of Big Data DDoS Detection

Big data DDoS detection is the application of big data technology to the process of detecting and mitigating Distributed Denial of Service (DDoS) attacks. By utilizing cloud-scale compute and storage resources, big data DDoS makes it possible to collect and examine network flow data in highly granular fashion, at great scale and with superior baselining intelligence.  The combination of scale, granularity and baselining smarts ensures greater accuracy in stopping DDoS attacks than is possible with legacy DDoS detection appliances. 

Big data DDoS detection has evolved because the limiting, scale-up architecture of appliance-based DDoS detection is no longer sufficient to track the massive amount of flow records and individual IP addresses that operationally characterize today’s digital business networks.

The primary goal of a DDoS attack is to either limit access to an application or network service, thereby denying legitimate users access to the services. There are many types of DDoS attack schemes that are used today and they are steadily becoming more sophisticated.  However, their common goal is to overwhelm targeted network resources with traffic or requests for service from many different sources – potentially hundreds of thousands or more. This effectively makes it impossible to stop the attack simply by identifying and blocking a single IP address.  The sheer distribution of attacking sources also makes it very difficult to distinguish legitimate user traffic from attack traffic when spread across so many points of origin.

The first step in avoiding or stopping a DDoS attack is knowing that an attack is taking place. To detect an attack, one has to gather a sufficient network traffic information, then perform analysis to figure out if the traffic is friend or foe.  This process can be performed manually or in an automated fashion. DDoS detection is the key to quickly stopping or mitigating attacks and in order for this to happen, two success criteria need to be met: 1) speed of detection and 2) accuracy of detection. So detection methods are a key consideration in formulating a strong DDoS defense.

The DDoS Threat

There is no doubt, as evidenced in the alarming rise of DDoS attacks, that DDoS detection is an absolute necessity for businesses that rely on Internet traffic in order for them to avoid disruption of applications and services, revenue loss, and brand damage. Neustar, Inc. and others regularly publish reports on DDoS attack and protection trends. Neustar’s October 2016 report highlights that DDoS attack volume has remained consistently high and that these attacks cause real damage to organizations.  Some highlights from that report:

  • DDoS attacks are unrelenting and show no sign of abating: The overwhelming majority of surveyed organizations (73 percent) suffered a DDoS attack. Eighty-five percent of attacked organizations were attacked more than once and 44 percent were attacked more than five times.
  • DDoS attacks are only the tip of the spear in complex assaults: The majority of organizations that suffered a DDoS attack (53 percent) also experienced some form of additional compromise. Forty-six percent of breached organizations discovered a virus, malware was activated at 37 percent of breached organizations, and ransomware was encountered at 15 percent of breached organizations.
  • DDoS attacks are time-consuming and expensive: It can take hours to detect and mitigate a DDoS attack at significant cost to the organization. Seventy-one percent of organizations took an hour or more to detect a DDoS attack and 72 percent took an additional hour or more to respond to the attack. Forty-nine percent of surveyed organizations lose $100,000 or more per hour of downtime during these attacks.

The most common type of DDoS attack is the volumetric attack, with the intent to congest the target network’s bandwidth related to a service or network segment. Roughly 90% of all DDoS attacks are volumetric, with application-layer attacks making up the remaining 10%. According to Akamai’s Q3 2016 State of the Internet Report, the majority of volumetric attacks are IP flood attacks involving a high volume of spoofed packets such as TCP SYN, DNS, UDP, or UDP fragments. A growing percentage of attacks are reflection and amplification attacks using small, spoofed SNMP, DNS, or NTP requests to many distributed servers to bombard a target with the much more bandwidth-heavy responses to those requests.

DDoS attacks are rapidly increasing in frequency and size. While mega attacks that last for many hours and reach 200 Gbps or more make the news, the vast majority of attacks last under an hour and are less than 1 Gbps in volume. Smaller attacks often happen without being noticed, though they may be harbingers of larger attacks to come. Mid-sized attacks are more readily felt, but distinguishing between a friendly surge in normal traffic and an attack is key to timely response. Large attacks are fairly obvious, and in these cases diagnosing the traffic is important to understand network entry points and sources. In all cases, a clear assessment is important to understand the best way to mitigate the attack.

In-line versus Out-of-band DDoS Detection

There are two primary means of implementing DDoS detection: an in-line detection appliance or out-of-band detection via traffic flow record analysis that integrates with any mitigation technique, whether on-premise or in the cloud. When performing in-line DDoS detection, all traffic is sent through one or more DDoS detection appliances that support deep packet inspection. The appliances examine the incoming traffic, and if particular traffic flows or packets are determined to be attacks, they are discarded, while legitimate traffic is allowed to pass through.

The basic in-line DDoS detection capabilities of network devices such as load balancers, firewalls or intrusion prevention systems may have once provided acceptable detection and mitigation when DDoS attacks were less sophisticated but high-volume attacks can overwhelm these devices, and sophisticated multi-vector DDoS attacks can often evade them because they are network location specific.

Out-of-band DDoS protection utilizes flow data from NetFlow, J-Flow, sFlow, and IPFIX-enabled routers and switches to understand unusual traffic spikes that occur across the network and detect attacks.  Attack mitigation via in-line devices or routing methods that drop selected traffic is then triggered manually or automatically.

How Cloud and Big Data Help Detection Accuracy

The first generation of out-of-band DDoS detection solutions were based on single server software design, mostly running on standalone rackmounted server appliances.  While far better than nothing, single servers simply don’t have the compute, memory and storage resources to track high volumes of traffic data on a network-wide basis.  This is particularly true when attempting to perform dynamic baselining, which requires scanning massive amount of flow data to understand what is normal, then looking back days or weeks in order to assess whether current conditions constitute an anomaly.  Regardless of whether it is deployed on-premises or in the cloud, single server DDoS detection is insufficient to accurately detect today’s attacks in a consistently reliable fashion.

The key to solving DDoS detection accuracy issue is big data. By using a scale-out system with far more compute and memory resources, a big data approach to DDoS detection can continuously scan network-wide data on a multi-dimensional basis without constraints.

Cloud-scale big data systems make it possible to implement a far more intelligent approach to the problem, since they are able to:

  • Track and baseline millions of IP addresses across network-wide traffic, rather than being restricted to device level traffic baselining.
  • Monitor for anomalous traffic using multiple data dimensions such as the source geography of the traffic, destination IPs, and common attack ports. This allows for greater flexibility and precision in setting detection policies.
  • Apply learning algorithms to automate the upkeep of detection policies to include all relevant destination IPs.

Of course, constructing an open-source big data system for DDoS detection isn’t trivial.  However, cloud-based/SaaS network visibility and DDoS detection systems are making it far easy to get the advantages of big data for DDoS detection without making major capital and R&D investments.

These advances are making a new hybrid model possible, where DDoS detection is performed by a best-of-breed, cloud service that automates the triggering of RTBH, on-premises and cloud-based mitigation appliances.  Big data detection systems also provide the added benefit of deep, forensic analytics, plus the ability to incorporate network performance, planning and other capabilities.

On Kentik & More Reading

Kentik Detect offers the industry’s only big data network visibility and DDoS protection solution built from the ground up on big data and delivered as a cost-effective SaaS.  Kentik Detect offers the industry’s most accurate big data DDoS detection, and can automatically trigger mitigation via RTBH, Radware DefensePro or A10 Thunder TPS mitigation.

For more information on how big data delivers 30% greater DDoS detection speed and accuracy, check out the blog post on Big Data for DDoS Protection, read the PenTeleData case study, or download The Case for Big Data-Powered DDoS Protection white paper. Know you want to get big data-powered DDoS protection today?  Start a free trial.