Kubecon 2023: Code, Culture, Community, and Kubernetes

I didn’t find out I was headed to Kubecon 2023 until the Thursday before the event. Which was, for all you procrastinators out there, still enough time to buy my ticket, find a hotel, and make reservations at Milt’s BBQ for the Perplexed.

While last-minute travel is always a bit of a nail-biter for me, I’m glad I decided to take the risk. The event was both massive and massively impactful in ways I couldn’t have imagined and in ways that no other event has been (at least for me). In this post, I wanted to share some of what I saw, thought, and learned.

Yes, “the network” is still a thing

I won’t pretend this is the first time I’ve heard the question, but it still catches me up short when I get it.

People see the big “network observability” banner at the Kentik booth and come up to ask some variation of a question I’ve heard since attending my first DevOpsDays back in 2014:

“Are networks even really something we need to think about anymore?”

I understand why they’re asking. Between containers, orchestration, cloud platforms, microservices, and the application itself, there are so many layers of abstraction that the packets, bits, wiring, and routing have all faded into the background. But it only takes a moment’s introspection to realize that, of course, they are all still there and still matter — both to IT practitioners who manage that infrastructure and the applications whose performance (let alone availability) rely on it.

Ironically, people also came to the booth to give examples of when the network really was the problem, but they had no way to detect it because that type of data didn’t show up in their tools.

My colleague Justin Ryburn echoed this observation in his Kubecon review:

Last year, I had to describe what eBPF was and how it might provide value when it comes to understanding the network traffic in a k8s pod. This year, most of the attendees asked me about eBPF.

So yes, the network matters.
It never didn’t matter.
Some folks just forgot.

The Kubernetes connection

One of the points my colleague Mike Krygeris was quick to make was that every Kubernetes cluster looks like this:

He follows it up with this explanation:

One original creator of the Calico CNI said, “Every Kubernetes node has a router built right into it. IP Tables.” Kubernetes still uses IP, and BGP is a battle-tested routing protocol with way less overhead than other tunneling techniques. The challenge is that the resources that are connected to that router are very ephemeral, so we need to add context to that in order to get useful observability from it. That context comes from metadata gleaned from the Kubernetes API. When you combine flow logs with all of the associated metadata, you get useful information like what services are being used and which are generating costs.

eBPF allows us to add context and information about performance. We are able to add insight like connection latency and TCP retransmission information, which identifies when packet loss is causing application slowdowns.

It’s important to recognize that this sort of insight is much harder to obtain further up the stack.

Mike stressed that folks who need higher performance often replace the kernel’s routing functions with an eBPF XDP data plane. Still, the trade-off is that these replacement data planes no longer work with standard kernel observability tools, and you have to turn to other solutions. An example would be using Hubble for Cilium’s XDP data plane.

All of this should provide you with all the context for why we announced Kentik Kube on the first day of Kubecon. If, as I’ve already explained, the network is still a thing that matters; and, as Mike has explained, every Kubernetes cluster has a router baked into it:

Then, it stands to reason that Kentik ought to be right in the middle of that Venn diagram. And that’s what Kentik Kube does: It collects network data from your Kubernetes instances — whether they are on-premises, in the cloud, in multiple clouds, or spanning any or all of those locations — and adds it into the not-insignificant network data already being collected. The result is that it provides you with deep and rich insights into your application architecture, performance, and uptime.

Code, Community, Culture

The keynotes across all three days had wonderful speakers and amazing themes — from lessons learned from outages to Destiny O’Connor and Catherine Paganini explaining essential truths about accessibility (more on that in a minute). One of the statements that stood out for me came during the day two keynotes when one of the speakers celebrated the way Kubecon is centered around “Code, Community, and Culture.”

If you read my retro blog from All Things Open, you already know the importance I am placing on building and nurturing a strong community at Kentik and how there will be much more to discuss in the coming weeks and months. What I loved about this framing was recognizing how it all has to fit together. Building a strong community by itself, simply for the sake of having a community, isn’t enough. Nor is it enough to establish and be an exemplar of a particular culture (of curiosity, being data-driven, celebrating achievement, or any other laudable trait) among users or of “code” which, in the case of Kubecon and the CNCF, translates to the overall product.

The lesson I learned is that all three of those things need to be imagined from the very beginning as being woven together into a complementary and harmonious whole — each supporting and supplementing the other. It’s a powerful vision and one which I wholeheartedly agree with and embrace. I also plan to use this as a model here at Kentik. More on that coming soon.

It also has to be said that part of the Kubecon 2023 culture was food-focused. Barely a single presentation failed to include images of deep dish pizza, hot dogs, and other well-known Chicagoan culinary delights.

Honestly, I left almost every workshop, talk, and keynote hungrier than when I’d sat down.

And communication

There aren’t many people (especially folks reading this blog, for whom English is a familiar if not native, mode of communication) who would choose to repeatedly attend conferences where nobody spoke their language, where every session — along with all hallway conversations, vendor interactions, and after-hours events — was foreign. And to be clear, I’m not talking about “foreign” as in “this is a technology I haven’t worked with,” but completely foreign. Every noun, verb, conjunction, and adjective is unintelligible.

Sure, some of us are lucky enough to travel to other countries, where we might experience a bit of a struggle getting through the airport, into our hotel, ordering food, or reading street signs. But once those relatively minor hurdles are cleared and we get to the conference, we usually find our way.

However, for the tech workers who are among the 430 million people worldwide with significant hearing impairment (to say nothing of the millions more with audio processing challenges), this is precisely their experience.

I learned American Sign Language (ASL) in college and was privileged to meet incredible folks in the Deaf community and count them among my friends to this day. So, this is both a passion (accessibility) and a pet peeve (the lack thereof) of mine.

In the ten years I’ve worked as a technical evangelist — attending conferences, user groups, meetups, and conventions running the gamut in scale from a modest handful to tens of thousands — I can count on one hand (with fingers to spare) the number of times I met up with someone for whom sign language was their primary mode of communication.

This is not because Deaf and Hard of Hearing people don’t work in tech — quite the contrary. The reason Deaf/HoH people don’t come to conferences is that the effort and challenge (not to mention the cost) far outweigh the value.

I’m proud to say this appears set to change. In June 2023, the CNCF established a Deaf and Hard-of-Hearing working group tasked with identifying conference accessibility shortcomings and creating a list of recommendations and guidelines to overcome them. The group — sponsored by Catherine Paganini and co-chaired by Rob Koch and Destiny O’Connor — will work immediately, drawing on the input from a diverse group of members. A draft document was published in September 2023.

The CNCF showed exactly how serious it was about this by expending both the effort and the investment to contact and contract interpreters, provide scholarships, and reach out to the Deaf/HoH members so that everyone understood this would be an accessible event. Ten people may not sound like a lot for a conference that boasts ten thousand attendees. Still, given the short notice and lack of promotion, and in light of my earlier comment that conferences are simply not something Deaf/HoH folks expect to attend, it’s positively massive. Add to that a few hearing ASL speakers like me, who joyfully and enthusiastically glommed onto the main group, and you had a respectable crowd.

A significant portion of the people in that picture had never attended a conference before, simply because of the hurdles I described. Others have attended, but at significant cost to themselves or their companies, and the experience was always constrained by the limits of what they — individually — could advocate for.

This is the part of the blog where someone usually expresses hope for the future, something like “I can only hope future events will be even more accessible.” And I do hope that. But I have more than hope. As a founding member of the Accessibility working group, I know there’s a plan to continue this effort and a commitment by the CNCF to ensure Kubecon 2023 is the rule rather than an exception.

Summary

Kubecon 2023 was more than just another conference to check off my list. First, it marked my first chance to work in the Kentik booth and learn from incredible colleagues.

Second, it was a chance to dive deep into the code, community, and culture of Kubernetes.

Third, it was a moment for a few members of an underrepresented group to meet in person and experience an event that was previously not an option.

As a postscript, I’ll mention that I was able to take advantage of Chicago’s famous kosher food scene. Kentik CEO Avi Friedman and I made it to Milt’s BBQ for the Perplexed and enjoyed a mind-boggling seven-pound rack of ribs. I fear those calories will be with me, at least as long as the rush I got from attending the conference.