Kentik - Network Flow Analytics
More Posts

How to Maximize the Value of Streaming Telemetry for Network Monitoring and Analytics

Streaming telemetry is no longer an unfamiliar term in the network monitoring realm. In fact, interest in streaming telemetry is increasing over recent years, while SNMP is falling, according to Google Trends:

streaming telemetry versus snmp search popularity

Streaming telemetry aims to modernize the collection of network and device metrics to keep up with the scale of next-generation networks, and provide new ways to access the huge variety of metrics that network devices can now generate.

Steaming telemetry uses a push-based mechanism with which data can be transmitted automatically and continuously from various remote sources (such as routers, switches, firewalls, etc.) to some centralized platform for storage and analysis. Selecting a proper telemetry architecture can potentially remove many of the issues—such as security, scaling, polling gaps, resource utilization of the polled device, and so on—around sending and receiving streaming telemetry data.

In this blog, we will review the current state of streaming telemetry and its ecosystem, discuss our take on the value that streaming telemetry brings to the network analytics table, and outline Kentik’s approach to powering-up network teams by leveraging streaming telemetry.

Streaming Telemetry: Where are We Now?

Core Technology

Streaming telemetry has the potential to accelerate network troubleshooting, automation, and traffic optimization. Core components of the technology to support those goals include:

  • Near real-time network data, achieved with push-based data collection
  • A programmatic way of configuring and managing network devices, achieved by a data model, which describes the specific metrics and metadata to include (for example, IETF YANG, OpenConfig, and other vendor proprietary models
  • A highly-scalable architecture and framework with more data point granularity and superior performance

The Vendor Ecosystem

Major networking vendors now support streaming telemetry on many of their hardware platforms, including:

  • Cisco - OS: IOS XE, XR and Nexus OS; Platform: ASR9K, CRS, NCS 6K
  • Juniper - OS: Junos OS; Platform: MX, QFX, EX, vMX
  • Arista - EOS
  • Nokia - SR OS
  • Ciena
  • Infinera
  • … and many more.

We are also seeing many new open source projects related to streaming telemetry, such as:

Streaming Telemetry: State of Adoption

With increasing interest from major technology companies and growing support from network hardware vendors, we’re seeing that early deployments of streaming telemetry have picked up speed in recent years, especially in organizations with large-scale infrastructures.

However, since the technology is still not standardized, there are many choices and variables, leading to different flavors of streaming telemetry that could make deployment more complex and slow down adoption. For example:

  • Transport options: Many choices like TCP, UDP, and gRPC
  • Session Initiation options: Dial-out (i.e., device sends data to collector) versus Dial-in (i.e., collector connects into the device)
  • Encoding options: Choices of JSON, XML, and Google Protocol Buffers (GPB)

Clearly, there’s still a long way to go in terms of standardizing streaming telemetry interfaces, which will ultimately likely boil down to either (1) picking a winner or (2) coming up with some best practice solutions and reference guides on when to use each option.

A long-term commitment to consistency and effort (from both networking vendors and the open-source community) will be required in order to move the technology forward over the coming years. As such, we expect to see that SNMP and streaming telemetry will coexist for a very long time.

A new specification, gRPC Network Management Interface (gNMI), is currently one of the main efforts to standardize streaming telemetry and other areas of network management. gNMI is a gRPC-based protocol for state management on network elements. Current participants in the project include big tech brands such as Google, Facebook, Microsoft, Apple, Netflix, AT&T, T-mobile, Comcast and others.

From a streaming telemetry perspective, the goal of gNMI is to normalize and control telemetry streams across multiple vendors, with consistent data elements and interfaces for data collection.

Combining Streaming Telemetry with Other Data Sources

From the network operations perspective, streaming telemetry can improve efficiency in many use cases, including:

  • Detecting problems by setting up network monitors and alerts based on pre-configured thresholds or network performance baselines
  • Troubleshooting connectivity and performance issues
  • Planning for network capacity according to usage and budgets
  • And much more… especially when we are able to use AI or machine-learning techniques to make automated decisions based on telemetry data.

However, streaming telemetry shouldn’t be the only data source that drives these capabilities. As an example:

Let’s say that you, as a network operator, want to be notified when utilization is high for critical backbone links. The next step would be to determine the characteristics of the traffic that’s driving up utilization. Which applications, clients, and servers are prominent on the highly-utilized links and can thus be used to make vaious optimization decisions (e.g., changing traffic patterns)?

An appropriate approach could be:

  1. Use streaming telemetry metrics as a set of indicators of thresholds, and then
  2. Use NetFlow to figure out what type of traffic is causing it.

As another example, streaming telemetry can also report real-time information on packet drops across links. This information can then be used via a network automation workflow to provision new paths and optimize traffic across the network.

The idea is to correlate all relevant data, with multidimensional data enrichment ― regardless of whether the data is sourced from streaming telemetry, network flows, or events and logs ― to see the bigger picture and learn the story behind the superficial symptoms.

Kentik’s Approach to Streaming Telemetry

At Kentik, we’ve been evaluating the market readiness to support this exciting technology, and current customers have asked for streaming telemetry support to help take advantage of the wide variety of data that can be sourced from streaming telemetry sources.

That’s why Kentik now officially supports an MVP release of streaming telemetry.

We are bringing all of our innovation for flow data to streaming telemetry. Unlike traditional approaches, Kentik’s AIOps platform for network monitoring and analytics allows users to easily combine flow data with streaming telemetry. Kentik’s backend architecture is designed to receive a high volume of streaming data, contextualized with Interface Classification, flow enhancements, flow tagging and more.

Basically, Kentik gives real meaning to the data, which is one of the major differentiators compared to other tools in the market today. Legacy tools may be able to collect the data, but do not provide deep insights into it.

As shown below, Kentik ingests telemetry data at scale, just like every other type of data we collect. Then via enrichment and machine learning, Kentik surfaces potential problems in real time so that network teams can quickly and accurately respond to incidents, proactively recognize and prevent issues from impacting service and business, and focus on network optimization rather than firefighting.

Kentik support for streaming telemetry

Additionally, we have a robust roadmap of capabilities that will dramatically expand the usefulness of an already-powerful technology over time.

Current Support and Roadmap

Kentik’s product team has always employed a user-centric approach to feature development. We usually implement features in multiple phases, in order to gather feedback from customers as we iterate during the development process.

The thought process behind this is, first, to design a scalable mechanism for ingesting and storing data, along with UI components so we can lay down a solid architectural foundation to leverage streaming telemetry. Second, we bring basic support to customers to gather feedback, and evolve iteratively. Third, we bring all of it together, normalize the requirements, and build the workflow to collect metrics and understand the data.

Kentik’s Phase 1 support for streaming telemetry includes:

  • Direct collection of telemetry data
  • Interface classification support
  • Support for Juniper “gNMI” JTI with UI support
  • Interface metrics (partial support)

Please contact our Customer Success Team if you want to get a preview of this early version of streaming telemetry support.

With access to the streaming telemetry features you can get statistics and visualizations of network ingress and egress traffic, via which interfaces, with connectivity types and so on:

kentik-support-streaming-telemetry-table-1 Kentik visualization of streaming telemetry data

In subsequent phases, we will add support for more vendors (e.g., Cisco Dial-Out for ASR), full interface metrics, more sample interval options, full alerting on metrics and state changes, and much more. The goals are to eliminate gaps in visibility and understand the complete health of the network, as well as relate this information to applications and traffic throughout the entire infrastructure.

Conclusion

Digital businesses drive the fastest revenue growth in history, and networks underpin all of it. New network monitoring and management capabilities are in urgent demand, and streaming telemetry is filling the visibility gap by providing real-time and HD-like visibility. Consuming telemetry data at scale while correlating it with all the other aspects of network context can be challenging. Kentik is well on the way to solving this difficult network monitoring problem.

Keep an eye on our product updates page, where we post regular recaps of what’s new. And to be the first to know about our latest developments, subscribe to the Kentik blog. You can also request a personalized demo to see Kentik’s powerful network analytics—and our latest streaming telemetry features—for yourself.

We use cookies to deliver our services.
By using our website, you agree to the use of cookies as described in our Privacy Policy.