Network Interconnection: Part 1 of a new series Building and Operating Networks
Networks come in all shapes and sizes, and they all have different purposes. Whether you’re a network service provider or an enterprise running your own hybrid, global network, each network scenario has a unique set of constraints and optimal operating conditions.
At Kentik, we pride ourselves as a pioneer and leader in network observability. Our platform caters to both standard and custom use cases depending on the unique needs of our customers.
This blog examines several different network interconnection models, essential considerations for each, and how “what matters” can be optimized with network observability.
First, let’s lay out a few simple principles that govern the way networks are interconnected.
As trivial as it seems, no network originates and terminates traffic within itself as a closed environment. Hence the importance of the name “internet,” meaning between and among networks.
Of course, there are corner cases, such as residential ISPs that offer IPTV services living within their own boundaries. But for this blog, let’s set aside any corner cases.
For a given network, traffic sourced from within the network is considered free. However, content sourced from outside needs to be carried over to the eyeball network’s (defined below) subscribers in the local loop at a cost.
Handovers are configured on the edge of a network, at locations where multiple networks meet and are usually described as interconnects. These interconnects have a cost associated with them (explicit or hidden). These costs are directly related to the amount of traffic flowing through them.
There is a hierarchy of interconnection methods based on technical and financial attributes.
Let’s summarize the different types of interconnects:
Given these different types, defining a sensible interconnection policy is not a trivial task. Network operators need to track multiple metrics, but we’ll come back to this later.
Let’s start with a simple categorization of the types of networks out there. This basic classification will help us to understand the underlying goals of each.
This term refers to residential or enterprise ISPs. A simpler description is that these networks have users directly attached to them who consume content traffic (hence “eyeball”).
Here are some of main properties of eyeball networks:
A good example of an eyeball network is unbundling of the copper local loop in some countries where the incumbent is legally obligated to offer other ISPs the possibility to rent the last mile copper pair to deliver DSL-based internet access. In these situations, this legal aspect drove internet plans prices down and made them very competitive markets.
Content networks are the reverse of the eyeball networks: these networks originate the majority of traffic, destined to be consumed by end-users.
Content networks’ main attributes are:
These networks are the providers of all the other networks: in essence, they transport packets from content networks to eyeball networks across the globe and are “the backbone of the internet.”
Here are their main characteristics:
There’s an interesting parallel to be drawn between building and operating a network and engineering a distributed datastore. Some may already be familiar with the CAP Theorem, formulated by Eric Brewer. In this theorem, C stands for Consistency, A for Availability, and P for Partition Tolerance — it lays out this simple rule:
“When building a distributed datastore, you can only select two of these three as design constraints.”
On the consumption side, when selecting a datastore for a given task, one must evaluate which of these two is more important for the underlying storage task and choose the datastore that fits accordingly.
The dynamic is very similar for each network. As listed in the quick shorthand summary in the previous section, each network comes with its own challenges, but building and operating a network always requires an understanding and observation of three well-known dimensions:
While less drastic than the CAP theorem (you can effectively get a variable amount of the three without having to exclude one completely), the art of the game for any network is to be able to make the right decisions, according to their constraints in each of these categories.
Since we’ve determined that Interconnection is one of the defining elements of any network, it only makes sense to leverage these to evaluate the constraints of any interconnection.
Generally, a transit interconnect is more expensive than peering on a cost per Mbps basis. More so, peering at an IX mutualizes the cost of interconnection over multiple peerable networks. Both require common locations and cross-connects that need to be factored into the eventual cost per Mbps.
Cost is an important factor for eyeball networks, as they usually operate in a competitive market.
To give the reader an idea: a pretty common plan in the US can be found for $40 for 100Mbps, which means $0.4/Mbps, without even considering the cost of labor and maintenance. This traffic cost needs to be put in perspective with the interconnection cost at the edge, more specifically, the cost of transit.
The diagram below illustrates the different modes of interconnection together with the associated money-flows that can take place between a variety of networks.
Performance relies among others on the number of hops to reach a specific destination. A blunt estimate is that performance decreases as the number of hops between a source and a destination increases.
With that in mind, a direct peering connection allows for content to go directly from the content provider to the subscriber. Additionally, the distance between the content and the subscriber is a crucial factor: because of the limits of the speed of light, the further the source from the destination, and the higher the latency.
These two aforementioned aspects are the key reasons behind the rise of content delivery networks in the early 2000s, whose value add is to both reduce the number of hops between content and consumer as well as the geographical distance across the broadest footprint possible. For a content provider with a user base across the world, performance to users becomes problematic because of the need to optimize for a large set of local eyeball networks.
As a rule of thumb, the more available paths to choose from between a source and a destination, the more reliable the eventual service rendered between the two. For example, if the primary path fails, a backup path is immediately available.
With the diversity of paths comes additional costs, as multiplying logical and physical routes implies diversifying providers, i.e. multiplying the number of interconnecting interfaces towards these next or previous hops.
With a larger number of interfaces on the edge comes a sizable management cost overhead because of the additional number of moving pieces.
Network reliability is one of the areas most prone to the law of diminishing returns. Every additional path added inbound or outbound becomes increasingly costly and decreasingly beneficial. Some experts go as far as saying that reliability never being total; choosing carefully when not to be reliable is a key strategic consideration for a network operator.
The diagram below illustrates diverse paths between two given networks, and the associated failover modes (configuration dependent).
We’ve established earlier that the top dimensions against which any network should benchmark itself are cost, performance and reliability. Given this, the underlying success on any network-reliant enterprise lies in its ability to do the following for any of these attributes:
In short, these are all the crucial activities that network observability can deliver.
Rightfully, the reader may ask, “It’s not like anything here is novel. Why does network observability brand itself as a novel concept?”
One of the answers to this question relates to the lack of a cohesive approach to the tasks of managing networking costs, performance and reliability, throughout the industry.
While network and infrastructure-heavy companies naturally structure themselves around functions (sales, network engineering, reliability engineering, software engineering, operations, support), they often suffer from the perennial organizational silos when the service they deliver demands horizontal attention across goals (cost, performance, reliability) and tasks (measure, track, tune, fix).
Some of the usual symptoms are:
Network observability’s mandate is to bring down the boundaries between teams, tools and methodologies as it pertains to the network. Its ambition is to make it trivial to answer virtually any question about the network and easily understand the answers. Lastly, network observability streamlines the activity of building and running networks.
In the next blog post, we’ll discuss how Kentik instruments and supports tracking and planning of your connectivity costs.