It’s been a busy time around here, so we’re just catching up with product update announcements for the last couple of months. Several of the enhancements we’ll look at this time around are foundational, meaning that they’re not only useful now, but they also lay groundwork that will enable us to further extend the capabilities of Kentik Detect in the future. We’ll start, though, with a housekeeping announcement that may affect those of you who’ve been accessing Kentik via a PostgreSQL client…
Deprecated: KDE Access Via Native SQL
Kentik Detect currently supports several ways for customers to use SQL to access their network traffic data stored by Kentik Detect in the Kentik Data Engine (KDE):
- RESTful Query API:
- Query Editor, an editor for ad-hoc queries in the Kentik Detect portal:
- Native PostgreSQL client (db.kentik.com:10376)
As Kentik Detect has expanded it’s become increasingly important to instrument interactions with KDE so that we can measure the impact of ongoing improvements on query performance. Our Query API provides this increased level of instrumentation, while access via native PostgreSQL (used by very few users) does not. So we’ve decided to deprecate direct PostgreSQL access.
As of Tuesday, June 26th, 2018, we’re phasing out support for KDE access via PostgreSQL clients. Current users will have continued access until January 1, 2019 (your IPs are whitelisted for this interface). If you find that your queries are now blocked, please contact Kentik Customer Success.
Without the constraints of PostgreSQL clients we’ll be better able to expose and support the full capabilities of Kentik Detect via our RESTful APIs. We apologize for any inconvenience caused by this change, and we encourage you to contact us for help in working through any issues that may arise.
PREVIEW HyperScale Custom Dimensions & API
Custom Dimensions provide a mechanism by which Kentik Detect customers can effectively add custom columns — queryable dimensions — to the main tables of the Kentik Data Engine (see Main Table Schema). Our developers have worked up a major upgrade to our Custom Dimensions engine, which we’re calling Hyperscale Custom Dimensions (HSCD). This new approach increases the capacity and agility of Custom Dimensions by eliminating many constraints of the legacy system, particularly with regard to the cardinality of values.
|Legacy, UI driven Custom Dimensions||Hyperscale Custom Dimensions (HSCD)|
|Maximum of 10,000 populators
over 12 Custom Dimensions
|~100,000s of populators for shared Kentik SaaS setup
~Millions of populators for single-tenant
Kentik On Prem setup
|~20 min target ingest-to-flow latency||<5 min target ingest-to-flow latency|
HSCD Availability and Use
While HSCD is already fully functional, our engineering team is currently completing final refinements in preparation for General Availability release. In the meantime, we encourage you to get familiar with it to see what it can do. So we’re launching HSCD in preview mode and putting it into the hands of customers who request it from Customer Success (email@example.com).
Note that to define HSCD dimensions you’ll use our new HSCD API. Instead of exposing direct REST endpoints like we do for our query and admin APIs, we’re making HSCD available via an Open Source client library, written in Python, that we’ve published on GitHub. We will also soon add the API itself to our API Tester and publish API documentation.
New Custom Dimension UI
Once we activate the new feature for you, you’ll be able to use the Python client to push Custom Dimension mappings, and you’ll see some changes to the controls in your Custom Dimensions dialogs (see screenshot below). The new UI allows:
- Paging when there is a large number of populators.
- Searching for Custom Dimension values across a large number of populators.
Stay tuned to this space as more details will follow soon about the powerful features that are enabled by the increased scale made possible by HSCD.
Selecting devices is one of the foundational functions required to return meaningful results in Kentik Detect, not only from ad-hoc queries in Data Explorer but also elsewhere, including the Library’s Saved Views and Dashboards. Our device selector has allowed you to choose groups of devices by type (router or host) or by site (physical location; see About Sites). But until now you haven’t been able to flexibly define your own device groups or to change the devices covered by Dashboards and Saved Views (e.g. add a newly deployed router) without editing the device assignments for each specific view.
Introducing Device Labels
With the recent introduction of Device Labels, those limitations are now gone. A device label is simply a non-exclusive property, any number of values for which can be assigned to (or removed from) any device at any time. Labels allow you to define groups of devices based on any criteria you choose: capacity, the customers they serve, the device manufacturer, etc.
When a query is defined using a device label then the devices whose flow data will be included in the query are determined at run time based on which devices have that label. This means that if you refer to devices via labels when building components such as dashboards, saved views, reports, and saved filters then you can change the devices that a given component covers without going back and revising the component itself. This makes the assignment of devices to queries much less tedious, and it makes the views that use those queries (Dashboards, Saved Views) much easier to maintain as your network evolves.
Using Device Labels
Device Labels are implemented in the portal with a new page (Admin » Device Labels) where you assign labels to devices as shown below.
Depending on where you are in the portal you’ll also see a new Device Selector that’s been redesigned to take advantage of device labels. As shown below, you can still select individual devices from the sidebar, or you can select by label, so that all devices that have the label at query run-time will be covered by the query.
As shown below, device labels are also now available for filtering in the Ad_Hoc filter Groups dialog in both Data Explorer and Alerting:
It’s important to note one area where you won’t find device labels, which is in the Group-By Dimensions selector. Because multiple overlapping labels can be assigned to a given device, it’s not workable for us to break down traffic by labels. Even so, we’re sure that you’ll find device labels to be a very flexible and efficient enhancement to Kentik Detect.
Subtenancy provides a mechanism by which a Kentik customer (e.g. an ISP) can enable each of its own external or internal customers to see a curated set of visualizations and metrics of their own traffic. This has been a recurring ask from existing and prospective Kentik users, and we’re excited to announce the availability of this feature. Subtenancy needs to be activated by Kentik (contact Customer Success for details), after which you’ll see a new page (Admin » Subtenancy) that we’ve created for you to manage subtenancy in the Kentik Detect portal.
As shown above, you begin configuration by defining the properties of your subtenant portal (subdomain, logo, etc.; see Subtenancy Page in our Knowledge Base), which can be thought of as a customer-branded version of the portal Library (see below).
Next you’ll use the Add Subtenant dialog (shown below) to create subtenants that can access the subtenant portal. If you’re an ISP, each subtenant might correspond to one of your business accounts. Alternatively, a large Enterprise might create subtenants that each correspond to an internal team that would benefit from access to specific reports generated by Kentik. For each subtenant you can determine the reports (e.g. Dashboards and Saved Views) that will be available, the traffic that’s covered in those reports, and the subtenant users (identified by email address) that will have access.
The subtenant portal feature will help Kentik customers boost revenue by selling subtenancy as a value-added service. And subtenancy will also increase the “stickiness” of ISP service offerings, which can provide a significant edge in a competitive provider market.
Performance Metrics in Sankey Diagrams
Sankey flow diagrams have long been an important feature of Kentik Detect, used to represent the flow of network data from hop to hop. We’ve recently refined the calculations underlying our Sankeys related to performance metrics, such as percent retransmits or latencies, to give a more accurate picture of these metrics. Our Sankeys also now support bracketing, which enables the coloring of nodes — and the links between them — based on your bracket specifications, which is very useful for performance diagrams.
BGP Status Indicators
We’ve reworked the BGP status indicators in the Device List (in Admin » Devices) to better explain the shown status. Previously, if you had v4 BGP enabled, it would show v6 BGP as “not established” even if you had not set it up. Now a tooltip (displayed on mouseover) will now tell you about non-established and not-configured BGP states. As shown below, you’ll see a separate indicator of states for v4 and v6 BGP.
- Session is not configured:
- Session is configured but not established:
- Session is established (therefore also configured):
CSV of Unclassified Interfaces
Interface Classification is a quick and easy process that reveals the role of each interface through which your traffic enters and leaves the network. The higher the percent of interfaces you’re able to classify, the better you’ll be able to optimize your network for cost and performance. We now make available a downloadable CSV file that lists all of the interfaces that can’t currently be classified, so that you can take steps to facilitate a better interface classification percentage (e.g. revise SNMP-retrieved description strings).
To export the CSV file, click on the Unclassified Interfaces button below the ring diagram in the Classified Devices pane (right sidebar) of the Interface Classification page (Admin » Interface Classification).
In the resulting Unclassified Interfaces dialog you’ll see the list of unclassified interfaces as well as a new Export CSV button at the upper right. When you click the button you’ll see alerts indicating the progress of the export, after which you’ll be able to access the Recent Reports dialog, from which you can download the Unclassified Interfaces report.
CSV of Alert History
Another report that you can now download as a CSV file is a log of all Alerts within a chosen time range and with specific attributes. To enable this oft-requested feature we’ve now added an Export to CSV button on the History page (Alerting » History). The download process is the same as for the Unclassified Interfaces report described above.
Once downloaded and opened in a spreadsheet, the CSV file will appear as shown in the screenshot below: