Ultimate Exit Release #1
Fasten your seat-belts, this one is a big deal. It’s the first release within a bigger plan for end-to-end visibility of your traffic, which is a holy grail objective of flow data reconciliation. What do we mean by “end-to-end visibility”? It means an easy way to figure out what volumes of traffic are flowing in and out of your network, from any source to any destination network.
A great example of this is assessing potential peer or transit prospects. How many times have you had to toggle between multiple spreadsheets that contain only approximations of traffic to or from various ASNs, getting bogged down in hacked, convoluted excel formulas, all in order to guess the ROI of what should be a simple decision?
What about trying to figure out how much traffic from a peer is being routed locally versus over more costly long-haul links? You need to able to figure out precisely at the site and device level — and at the interface level in the future — the traffic flowing between network entry and exit points.
It turns out that the sophistication of flow consolidation and reconciliation needed to achieve this task is beyond home-grown tools, data infrastructure, and software engineering capacities of many network engineering teams. And for good reason. It’s a hard problem.
Voila! New Exit Dimensions in Kentik Detect
Introducing two newly added destination dimensions (fanfare, please):
- Ultimate Exit Site
- Ultimate Exit Device
How do you use these? Let’s say you are a transit provider. You move packets from content providers to eyeball ISPs, and carry them over a costly global backbone. You want to look at the traffic you’re exchanging with one of the major content providers like Google, and see where it comes in, and where it comes out of your network.
Let’s further assume that you run a well organized network, so you indicate within your Interface description nomenclatures any interconnections with Google. This means you can easily include these interconnects with a simple filter. For example:
BTW, if you know that you’re going to be looking at these often, you can also make yourself a nice Saved Filter (see below) and just apply it any time you need it.
Then you can use that saved filter in any Data Explorer query you’re working on.
So here’s what you want to look at, in sequence:
- The site where the traffic enters the network.
- The site where the traffic leaves the network.
- The next-hop Network.
- Which eyeball network it is terminating at, i.e. Destination AS.
Using Kentik Detect’s handy new dimensions you can now answer this question with the following query:
For a useful visualization, select the Sankey display type:
Looking at the generated Sankey diagram (above), you can now instantly see what traffic is flowing between the entry Site and the Ultimate Exit site, and which eyeball networks are reached. What you would typically do at this point is look at where transport is the most expensive or least performant between your Entry Site and Ultimate Exit site and optimize for either of them.
In the above Sankey chart, you can see that you’re shipping a lot of traffic from Frankfurt to Marseilles. So a few questions come to mind that can be explored further in Kentik Detect:
- Should you track Google’s ability to PNI in Marseilles and save yourself some Frankfurt→Marseilles transport costs?
- Do you want to review your prices for transport for London→Marseilles based on how much of that capacity is consumed by your Google PNI?
- What portion of the private links between Frankfurt and Marseilles is going to those Google PNIs, and therefore what’s the real ROI you’re getting from these links?
You can’t even start this ROI exploration when you’re stuck in spreadsheet hell. Stay tuned, because there’s a lot more coming over the next few months in this arena.
Enjoy!
Custom Dimensions Update
Our Custom Dimension infrastructure has been upgraded, allowing us to upgrade our default provisioning rules:
- Max Custom Dimensions per customer account: increased from 5 to 10.
- Max characters per dimension: increased from 12 to 128.
- Max populators: upgraded from 5000 per dimension to 10,000 overall across all dimensions (no additional per-dimension limit).
User Based Filtering [PREVIEW]
Every now and then we will preview an upcoming feature. We also believe that occasionally there is value in releasing an early/crude version of a feature-set in order to get early feedback from our users, which we can then use to quickly iterate until we arrive at the feature that users really want. In the case of User-Based Filtering (see Knowledge Base article), we are previewing here a feature that we have decided to introduce as an early release.
Kentik Detect currently supports two different user levels: Member and Admin. User-Based Filtering allows an Admin user to apply a user filter that restricts the data available to a Member user. The underlying idea is for Admins to be able to grant (very) granular rights on what specific Members are allowed to see and/or query.
Admin users can set up a user filter on the Users page (Admin » Users).
A user-based filter is composed like a filter in the Filters pane of the Data Explorer sidebar. Once a user filter is associated with a given user, these filters are systematically appended (ANDed) with any query run by that user, including:
- Data Explorer queries via Kentik Portal UI
- SQL queries from the SQL Query explorer or via PGSQL connections
- API queries
One use case example is allowing only certain users to query flows from backbone routers, as shown in the following screenshot:
Another example, shown below, allows certain users to query only flows for CUSTOMER interfaces on ‘Ashburn DC3’ and ‘Ashburn DC4’:
As explained above, we have released the minimum amount of functionality for this feature, and hope to leverage the feedback of interested users to iterate it.
Some open questions we have for this feature include:
- Should filtered users be made aware in the UI that they are being filtered? In the current version of this feature, the user wouldn’t know.
- If filtered users are made aware, should we indicate a permanently locked filter setting in the Data Explorer?
- Should we let users know they are being administratively filtered, but not indicate what the filter constraints are?
- Should the display of filtering information be administratively configurable at the user level?
- How do we mention or indicate user filtering in the API and SQL? For example, when a user submits a SQL query, should we return a modified version of the submitted requests with the appended filtering in its SQL form?
Please let us know your feedback on support@kentik.com. Is this a useful feature that you would like to rely on? What should the next iteration look like?
Sampling Rate
This is one for the nerdier users out there. As you may know, our ingest platform includes smart ways of re-sampling flows exported by your devices to match your contracted FPS. We’ve been improving this functionality quite a lot recently. Our goal is to resample accurately and keep the resampling-bound distortion as close to zero as possible.
In order to keep our engineering work accurate, we actually had to add Sampling Rate to our available dimensions, metrics, and filters, as shown in the images below:
Available Dimensions:
Available Metrics:
Available Filters:
This could come in handy on your end when debugging potential flow sampling misconfigurations.
Extra Data Explorer niceties
As we see our customer’s usage of the Data Explorer evolve we often throw in additional convenience features that we think will streamline the overall user experience. This time around, we’ve added a couple of convenience tweaks, both of which are geared towards making queries return faster by allowing users to optionally skip certain processes.
- Disable Total: You can now disable computation of Total over a metric if you already know you aren’t interested in looking at the total value for your breakdown. This saves processing time on our mid-layer, resulting in faster return of query results.
- Disable Hostname lookups: You can also now disable Hostname lookups directly from the Data Explorer query panel, which reduces query response time because IPs won’t need to be reverse DNSed before returning the results of an IP/CIDR breakdown.
With reverse DNS enabled:
With reverse DNS disabled:
Alerting Update
Syslog Alert Notification Channel
We’ve just added the capability for you to ship alert notifications to good ole Syslog infrastructure. This has been a recurring ask since we’ve released v3 of our Anomaly Detection and Alerting platform. Your voice has been heard! Syslog alerting works in the same way than the JSON Webhook feature does, which is by offering a new type of notification channel, aptly named “Syslog.”
When configuring a threshold in an Alert Policy (Alerting → Alert Policies → edit a policy), you will notice that in addition to the existing Email and JSON webhook options a new entry has been added to the Create Notification Channel button. You can tune all of the config knobs when you create the channel, including Port, UDP/TCP transport, Syslog Severity, and Syslog Facility.
Alerting: new dimensions and filters
We’ve just added new support in our Alert Policies for:
- IPV6 (for Dimensions as well as Filters)
- inet_family (for Dimensions as well as Filters – this is to select IPv4 vs IPv6)
