With Kentik customers now starting to take advantage of True Origin™, the time is right to learn more about the benefits of this new feature-set. In the sections that follow we’ll explain what True Origin is, what operational needs it addresses, how it works, and the steps involved in putting it to use.
By the end of this post you should have a pretty good idea of how True Origin helps network operations and engineering, particularly for Internet Service Providers. The following capabilities provide a taste of how useful True Origin can be:
To understand how True Origin helps us with the above scenarios, let’s start by putting ourselves in the place of an (eyeball) ISP with a large number of subscribers that consume content from the Internet. The content comes into the ISP network from various sources via various paths with varying cost and performance. For the ISP’s executive and engineering teams, these variations increase the complexity of meeting two main operational goals:
To succeed at these goals, an ISP needs to define a robust, performant architecture and also a set of best practices. To do that, you need real-time, high-quality information about two things: The traffic over your network and the cost to you of carrying that traffic.
Until now, there hasn’t been a single tool that provides the required visibility—in the same place, at the same time—into both traffic and cost-of-carrying. That’s why True Origin is a major step forward for every ISP carrying a significant volume of Internet-sourced content.
Kentik’s foundational mission is to bring our users network traffic visibility that’s fast, easy to consume, and highly actionable. By correlating an ever-wider set of traffic data into a single, instantly-queryable dataset (the Kentik Data Engine, aka KDE), we’re able to generate technical and business insights with direct, powerful relevance to your network operations. True Origin falls right in line with this mission, building on a set of key Kentik Detect features that are geared towards eyeball ISPs:
The above features are each powerful by themselves. But working together they uniquely address the questions faced daily by our ISP partners. The main goal is to provide actionable, human-readable answers about the relationship between specific content and the methods by which that content is delivered to subscribers. Here’s a look at some ISP questions that True Origin can help answer:
Flow data alone (source and destination ASNs and IPs) lacks the information needed to answer the questions above. True Origin addresses that limitation by enhancing KDE flow records with additional data that enables Kentik Detect to provide richer context. The following factors are now either included in or derived from these enhanced records:
We’ve already implemented OTT tagging and CDN tagging as dimensions that you can filter and group-by in Kentik Detect, so to get a feel for how True Origin works we’ll start with an example that uses both features.
Consider a simple (conceptually) transaction in which a client requests OTT content from a server, which we’ll assume in this case is managed by a CDN:
As DNS responses and flow data are ingested, KDE correlates flows and DNS queries in which:
Based on the transaction described above, True Origin may seem relatively simple in theory. In practice, though, successful execution requires resolving the following challenges:
CDN Tagging by itself isn’t sufficient to obtain a clear picture:
CDNs come in many different shapes and forms, each involving its own mix of infrastructure and connectivity, which brings up the following considerations that can affect performance and cost:
With True Origin, Kentik has solved the challenges above, enabling ISPs—as well as the operators of enterprise and campus LANs—to get a clear picture of how your network utilization and costs are impacted by Internet-sourced content.
To do so, you’ll use the following source and destination dimensions, whose values are stored in each flow record at ingest:
The key factor in OTT tagging is the hostname that, as described earlier, is looked up in a DNS query. That’s the source from which we derive the OTT Service value; the other OTT dimension values are associated with the service via our curated list of providers and service types, which continually evolves as we discover and analyze new sources of traffic.
Now that we understand a bit about how True Origin works, let’s start thinking about how to put it to use in Kentik Detect. Once again, we’ll be approaching this from the point of view of an ISP with eyeball subscribers. Because our subscribers pull far more content from the Internet than they push to it, our ingress traffic has a much greater effect on volume and costs than our egress traffic. So what we’ll mainly focus on is traffic that enters our network towards our subscribers.
With most flow visibility tools it wouldn’t be easy to segment our traffic this way, but Kentik Detect’s Interface Classification feature is designed to handle exactly this kind of challenge. So to get the results we need when using True Origin, we’ll start with a quick detour into Interface Classification.
Interface classification works in multiple stages. First we apply rules against which interfaces are evaluated and classified, then the flow records from traffic across those interfaces is enriched with information about the interfaces, and then queries on those flow records can group-by or filter based on any or all of the following Interface Classification dimensions:
Among the types of connectivity covered by Interface Classification, Embedded Cache is particularly relevant to traffic delivered by some of the major CDNs. As an ISP, I may have interfaces that are connected to a cache appliance server that’s been provided by a CDN or content provider—such as a Facebook appliance (FNA), Google Global Cache (GGC), Netflix Open Connect (OCA), or Akamai cache.
If so, I want to be sure that those interfaces are classified as such. To do that, I’ll set an Interface Classification rule as shown below:
Using regex, the above rule will evaluate all SNMP-polled interface descriptions. Matching interfaces will be labeled with these three dimensions:
Once Interface Classification is applied we can take advantage of it to better understand how traffic from content providers (via CDNs) is affecting our network utilization and costs. In Data Explorer, for example, we can set filters on Interface Classification dimensions so our visualizations and tables are focused on the traffic that’s of greatest interest. To see only traffic coming from the outside, we’d set a filter to SOURCE Network Boundary EQUALS External. To see only traffic coming out of the network the filter would instead be DESTINATION Network Boundary EQUALS External.
Now that we see how Interface Classification helps tailor our queries we can combine it with True Origin dimensions to really narrow down to a specific type of content. We’ll focus on video traffic because these days it represents the lion’s share of traffic volume.
We’ll start in the Filtering pane of the Data Explorer sidebar, where we’ll create a new ad hoc filter made up of two filter groups (see Filter Groups Interface). One group will contain two ORed conditions that cover traffic from two possible sources, either outside the network (via externally facing interfaces) or from caches embedded inside the network. A second filter group, ANDed with the first, will use the OTT Service Type dimension with a value of Video. The resulting filter will look like this:
Next, still in the sidebar, we’ll choose our group-by dimensions in the Query pane as shown below (see Query Basic Options), so that the query will show us top-X traffic by a combination of OTT dimensions, connectivity type, and CDN.
Finally, from the View Type menu at the top right of the chart display area, we choose Sankey diagram (see Chart View Types). When we run the query with these settings, the resulting Sankey diagram (example below) shows the video traffic coming from external sources or an embedded cache, ranked (top-X) according to a key built from our five group-by dimensions:
Let’s first look at relative traffic volume to confirm that what we see represents reality. Based on the volume of video-only traffic, our top OTT service providers (3rd column) are Netflix, Google’s YouTube, and Hulu. Nothing surprising there, but it’s good that our query results are consistent with the usual suspects and our general knowledge about their traffic.
Now let’s dig into a few more interesting things. First, Netflix’s OpenConnect CDN is known to have a nearly perfect offload score. Based on these results that appears to be warranted, because we can see that the Netflix-supplied traffic is coming exclusively from the embedded caches in our network. If we ever see Netflix “shedding” traffic into connectivity types like Transit or Peering, we’ll know that this is an anomaly that needs to be investigated.
Second, it’s worth noting (though it may be assumed) that the vast majority of our video traffic is being supplied via CDNs, either commercial (Akamai, Level3, etc.) or in-house (Netflix, Google CDN, Amazon CloudFront, etc.). This underscores the value to ISPs of being able to see beyond CDNs to identify the true providers of their bandwidth-heavy traffic.
Next, by hovering over any listed OTT service (2nd column) we can easily identify both the connectivity type (4th column) and the CDN (5th column). This is important because, as previously noted, it’s extremely important for ISPs to deliver OTT services with the best possible performance at the best possible cost. Content whose connectivity type is transit makes those goals more challenging for two reasons:
Despite the potential drawbacks above, the delivery policies of some video content providers leverage the full spectrum of available means. In our Sankey, for example, we can hover over Hulu to see the following delivery methods in use:
Information like the above enables you to plan intelligently for the demands that CDN-delivered content puts on your infrastructure. And it also alerts you to opportunities to either lower costs or boost performance (e.g., deliver video at a higher bitrate) by trying to find alternatives to transit — more embedded caches, perhaps? — for traffic from content providers. In future posts about True Origin we’ll dive deeper into specific use cases, such as troubleshooting traffic that should be going to caches but is instead arriving via transit. In the meantime, if you have questions or need help with using True Origin, don’t hesitate to contact Kentik support.