Career and networking evolution with BGPMon's Founder Andree Toonk

Welcome to Network AF, a journey of super nerd proportions into the world of networking cloud and the internet.

Avi Friedman, Networker, Coder, husband, and CEO of network observability company can On this first episode, we'll talk with my friend Andre Thunff of Open DNS, PGP, Cisco, and multiple networking projects, about career and networking evolution. Some of the takeaways you'll hear are about overcoming intimidation, staying connected to mentors, Driving your own growth. And on the networking side, we'll talk about the inter networking world, Sre and networking in the relationship.

And some of the trends in disaggregation and Cloud Scale Networking.

Welcome everyone to network a f. I'd like to introduce my friend and fellow networker Andre Tungk. Andre, if you could give us a brief background about yourself and, what you've been up to and what you're up to now.

Alright. Yeah. Thanks for having me, Avi. So my name's Andre. I am based out of Vancouver, Canada, like everything networking, and I guess what we now call DevOps So I've been sort of doing that for the last twenty years.

I think what's unique is that I've always had one lag in what I call DevOps now and SIS admin in the past maybe. And networking, in the other. So, really always try to take the learnings from mostly sort of the the dev ops world into the network world. It's been a lot of fun.

Started my career at the Amsterdam internet exchange, worked for a few ISPs, moved to Canada, Vancouver, fifteen years ago.

In two thousand twelve, joined, a company that that really changed a lot for me, called Open DNS. As one of the early, what was back then called ops engineers. Mhmm. That was a lot of fun.

And, you know, when it joined, opening us had a few pops or data centers. I recently left last fall, and and I think they're up to forty. And so that's been a very exciting and an interesting journey and learned a lot there. And then also, I think one other thing a lot of folks might know me from is I founded a company called BGPMon, and I know a lot of Fjokes.

Yeah. I know Kantec, obviously, and Avi threw that into Cisco. And I know at Cisco, we were big fans of, of Kantec.

And so, yeah, everything around, BGP monitoring, hijacks, outages, and, and, and that kinda cool stuff. And I don't know, Avi or super interested in that. And and you even had a predecessor to that. I think it was called, watch my net or something. Yeah. It was watch my net.

Andy Freed. Actually, I met Gotti have run at a convention for people that run science fiction conventions.

And, he did the, you know, taunting me to discover whether I was a worthy nerd, which I don't really enjoy. But, anyway, I did that And then he's like, you can't build a BGP monitoring system. And I'm like, well, I don't really do the front end, but I could certainly go through. I remember the front end was like a Pearl CGI PHP.

Yeah. With Oh, yeah. Andy. Andy, now it's spamhouse actually did. But I think on the second week, I'm like, and I tried to compile my old BGP implementation that I wrote my interview at AboveNET that used to run the RBL.

And I was like, oh, I guess BGP's changed a little. So I found some BGP pro thing and made it, made it work, and, you know, dump the things and did the grabbing and, yeah, I just didn't really try to put, well, I should have, but, you know, b g b man did did well and maybe not. Yeah. It did well.

Got a huge community. It's Yeah. I got a used community and, eventually got acquired and it is now still, I think, being run by Cisco. Although I think they're shutting it down, but, you might actually know better by now, but, yeah, rewritten and put into something.

And now they're a respected competitor because can't take just launched. Can't seem to get something similar. Yeah. Yeah.

I know, obviously, you guys have, Doug Madore, who's, like, an expert in this field too. So, you know, anyways, that was a really fun journey, and, and then I left Cisco, so a lot of that was around that last fall. And right now, I'm a bit of a sabbatical and try and figure out what I'm gonna do next. Okay.

Well, we can talk about that because, I am only one of many people. I'm sure that has tried to recruit you, and you'd be like, no, it's okay. But have been following some of the fun stuff that you've been doing, you know, which again we can talk about. So so what what got you, you know, you said systems I know a lot of people are like, oh, networking is hardware.

It's like, really? I mean, it's hardware that runs software. Systems are software software, you know, that can run networking. I mean, I have always seen that it can be a continuum.

But, like, what got you into the networking especially inter networking, but networking over all side of things. Yeah.

Well, I I guess I was lucky. When I was in university, we had a, a teacher, that somehow, I guess, got a deal with Cisco and he had these Cisco labs. And cc and A was just becoming a big thing. And I guess Cisco had sponsored a lab in my university.

And, so they had all the gear, and then they had also just released all these kind of flash based tutorials, like, excel material that you could basically follow. And then, and then you learn about networking. To indulge that, I had just moved out of my parents' place to, like, the university there, and I didn't have too many friends there yet. And, and the ones that I had, they they still live at home.

So I I kind of got used to that lab, and it was open basically till midnight. And I was like, this is all I got addicted to it. That's a lot of fun. And then I started doing them, and then, you know, I got really interested in it and got really good at it.

So that's how it started. That's how I originally got into it.

A lot to be honest. I guess I worked at a help desk before as well. But this is really how we got into the networking. Then I got a, a gig at AMSick the Amsterdam index exchange, which is like, you know, one of the larger internet exchanges in the world.

That's where I even learned more. That's for me. This is where the door opens be beyond the default route. So I didn't know, like, I knew BGP.

It's like, you set a default route to your ISP, but it never occurred to me, like, what happens after the default route? And then I ended up there and was like, wow, that's the whole world behind here, which is super fascinating. And I really never really left that world. Like, that's you know, it's so fascinating.

That's how I really got into it. Cisco labs help desks, and then the door to the, you know, core of the internet between quotes, and, yeah. I remember the first time I visited Amzacs, the university park side. And I'm not used to smelling smoke in a data center, but, like, people are still that was long enough.

Oh, yeah. That's right. Yeah. Smoking in the lobby outside the data center. And I'm like, usually, it's bad if you let the smoke out of the computer.

The magic smoke, you know, makes it run. And then I remember they had, like, that was, I think, the first time I saw, like, forklift for, like, sticking stuff in. And, of course, the the glimmer glass, which if it was in the United States, you would think was for the NSA, but was actually for an interesting approach to to layer layer zero, H. A.

That's, it's interesting. You bring that up. So, so, my first real job was at Sara which is in the in the science park where I love the Amsix stuff is.

And, yeah, some of my coworkers are still smoking when I worked there. And so I worked on the network called surf just sort of the internet too of the US.

But, my first one of my first automation experiences, like network automations, was all around t l one. And I worked a lot with someone at Amzyx and Arjan, if you're listening, and they automating the glimmer glass with t l one. And we had a lot of optical gear, like, Nortel gear, So telco gear has a standardized language. It's called tier one for folks that don't know that.

This is a really obscure thing. I didn't know that. I didn't know that. Okay. Yeah.

So it's all the telco gear seems to have, at least in the past, different vendors, Cisco, Nortelceana, all spoke the same kind of language.

And so we built some Pearl libraries to automate together with this with this Also, it's a girl. M6. Yeah. To to do the, the gluomiglass.

Yep. Actually, for those that don't know, it is an interesting take. Don't know if they still do it anymore, but the glimmerglass was sort of like it was a active. The idea was you'd have a standby port it would be watching.

So it'd be doing the Mac learning. And then when they wanted to maintain a switch, they'd was it all a switch they'd flip over to another was like a Yeah. So I think they just had basically two switches.

So what the the glimmer glass is basically a big switch with mirrors in it. Right? And it's an optical switch. But it's a very dumb switch, right?

So it's either goes to switch a. So the cable comes in and says, oh, I want it to go to switch a. And then switch a is unhealthy, well, then they changed the mirror slightly, and then they would go to switch b. And so the advantage as a customer was that you only needed one connection and then, sort of on the on the provider side, they they would just say, oh, you know, switch a is unhealthy.

We'll switch you to switch b. Yeah. And then you wanna automate out. But the cool thing is that it was all optical with mirrors, essentially.

Yeah. Up the automated mirrors pretty cool. Yep. Automated patch panel, whatever you wanna call it.

Yeah. So were there did you have great mentors? Were there, you know, how did you, you know, get get was it was it, you know, you did the lab yourself. Did you have people pointing you at the things to learn and and, you know, was was there that kind of culture there at the time?

I think, sort of the, you know, if I think back about the lap and really getting into networking and Cisco and, you know, the, you know, a lot of people give, crap on, like, the certification programs. But to be honest, I think if it weren't for those, I would not not have gotten into it. I think it gave me a solid kind of background in terms of, like, what is BGP, what's OSF, what's, you know, whatever, and and actually some hands on them, and and getting some of the basic in. And so for me, that was a great way to get in. I was surrounded by, sort of that teacher and a few other folks that that shared that passion. Don't think if there was really a mentor there at that point, it was pure passion, but certainly the opportunity was created for me to to do how spend however much I wanted on, like, all cool gear.

I if I think about not a mentors, I've had a few over my career, but I think, early on, when I was still in the Netherlands, there was someone called named Ronald, and, Esara. And he really got me into the automation type thing. It was a really smart guy. And, in university, we had to do some programming, but I never enjoyed it.

I never really got into it. And, but but I guess the problem was you were just doing puzzles. And I was like, I I don't get it. It's kinda boring.

And, I wanna go back to the lab. You know?

And then all of a sudden I was running this this was at Servinet. We were running just, you know, fair that's a very advanced, network, and, very large And, because all these new gear and, and, and, for especially the Nortel stuff, the network management systems didn't exist yet for Nortel. So again, in the in the telco world, you basically buy the network management system from the vendor, but because the it was also new, it didn't exist or it was or it was very expensive.

And so, he and I kind of started going, like, okay. How would we do this ourselves? And it was a whole, you know, back down Pearl was a big thing. And, yeah, and he took me under his wing, and then there was another guy Marco, who helped me with a lot of PHP stuff.

And all of a sudden, my passion for programming, kind of came back because all of a sudden, I had a itch to scratch. I had an actual problem. And, you know, it would solve me sometime in the middle of the night or whatever when we took you know, I would get I was, you know, an end an, a new engineer sort of, so they gave me all the crappy jobs. Like, go collect all the serial numbers from all the devices.

Well, that's a three week job. Right? Or I can spend two days programming this thing, and I run it in an hour, and it's done. Laziness breeds elegance, you know, in many people.

Right? It's the why do I wanna do this again and again and again? And and it's an interesting theme. I guess we'll talk about some of the things that you've done are now living between networking and systems.

But, in the they're what we're in some ways more advances in networking automation, at least the life cycle, even before that really became, you know, crawled out of the high performance computing world and became cloud. And, you know, SRE, and then I guess it sort of stopped for a while.

But, no, that's that's definitely interesting. So So, obviously, just maybe one step back with the mentors, right, just so there's been a few throughout my career and those are the ones that really helped me going. And then, I guess my message would be to folks out. Yeah. Do find out or find mentors, people that wanna help you. And, you know, to be honest, it always sounds a little bit scary, but a lot of times it's just like, find someone you respect or who is, like, you know, a few years ahead of you. And just, you know, this can be a lot of things, but a lot of times it's just, like, unofficial, right, just, have a coffee with that person once a week or once a month.

And, and, and, and that relationship changes over time and with whom, but There's so much you can learn from others if you're just kind of open to it. And even for me, I even now, I've made to it a lot of folks and sometimes they're just chats. And sometimes, you're like, wow, they're full of gold. So I think it's very important.

And I and I also tell people, like, a lot of folks within a company have one on ones. So, like, you know, if I work for Cantic, I would have a one on one with with with you, for example, right, every week. But more and more, I'd find, like, it's very important to so if you're doing that, keep doing that, but also do it with people outside of your company. See, that's a very well understood thing.

As you go further on your career, there's only so much you can learn, and the company is a little bit of a bubble in itself. So work on, like, hey, someone you worked with in the past and that's a one on one too. Don't feel guilty that you spend an hour talking to someone at another company, because you're learning from that too. Yeah.

No. Absolutely. Sometimes I get daunted when I looked at my LinkedIn I try to accept the only people that I know, but you never you never really know, what, you know, can come in the future, you know, from staying connected to people.

You know, definitely for sure. The other thing I would encourage is, you know, there's lots of different terms for the bright shiny eyed, kid. But if you demonstrate that you are interested in learning and picking things up.

You know, there's often, especially in university, but even in companies, ways to get involved in, you know, projects and learn and you know, in a in a healthy company, in a healthy environment, people will invest in you, you know, if it's clear that you're that you're, you know, learning and growing, and hopefully we'll pay it back too, you know, at some point. With COVID, we need to figure out some of these things.

You know, how that's all gonna work and how we grow the community but, you know, those are all things that we as a community are thinking about. So you mentioned sort of, I guess, now we have to say SRA or DevOps or dev net ops, but, you know, the the link between these. And, you know, you certainly have come from the world of, well, the labs, you know, the the the actual go twenty five hundreds and seventy five hundreds and such. And also, you know, on the internet routing where it's already virtual and open DNS was I mean, services on top of, right, a network, you know, with any cast and load balancing and things like that. But I see you've been playing a lot with data plane networking, EBF.

Actually, the pre history of Kentech was network sensors doing you know, packet stuff. And then I just got really interested. Customers said, what do we do with all the data?

But, you know, what what what was your interest in, you know, playing with high speed networking through, systems and all the, you know, the Linux, you know, evolution around that that, you know, stuff on.

I think there are a few things that drove me to sort of dig into it deeper.

You know, some of it was driven by the things that were happening in my work environment back then where, you know, at Cisco umbrella, and a lot of other folks are doing this is, with the, what's called Sasi, where basically what's happening is your you're you're offering the the typical network services as a service. Right? It's a typical network components, like a load balancer or firewall or netting or IP stack. And you see this in all the old cloud providers, right, Amazon has IP set gateway, so whatever not gateways. But, and so traditionally, we would ship lots of you know, appliances.

But so that that doesn't really work anymore. And, and so the question then becomes, how do you build these things in a virtual environment in a cloud native way. And so that that was, something I worked on a lot and was super fascinated about, because, again, it brought together sort of those two worlds that I was, was kind of interested in. And so that worked really well for me.

And so there's two challenges to be solved. One of them is sort of the, the implementation on the control plane and management plane, like, because, you know, nice thing about an appliances, it has a six, and and, you know, basically, they scale vertically. Right? They can do a hundred gig in one device, for example.

Well, there's no way you can get that out of a VM. Right? So how so then you're faced with, you have to basically disaggregate that. And so now you need to have I don't know.

Let's call it ten VMs to do the equivalent of one big box.

And now you're basically into the distributed computing problem. And, like, how do you synchronize state and, like, all that kind of stuff. So that's a whole interesting problem. So that was part of the problem space, which is super interesting.

And, you know, some people call it microservices, but basically it's a distributed computing problem. And then the other part is, speed. These big boxes have, you know, ASIC, and they go like crazy. Like they're really fast, right?

If you do this in virtual environments, you actually have a problem because well, there's a few problems. The first one you often hear about, was Linux networking is slow. Right? And, you know, you know, that's kinda true, I guess.

Right? And, so part of my journey was to define what is slow, like, what is slow. And so the conclusion was in Linux, and I'm I'm gonna, you know, skip a whole bunch of details, but sort of you can do a million packets, per second per core. That's sort of the rough through the kernel.

So just to be clear, through the kernel, forwarding using the IP stack. Yes. Yeah. Yeah.

Thanks for clarifying. That's the baseline. Right? The base space. Yeah. And and there's a whole bunch of details there, but that's a rough if you wanna take a number.

If you just use kernel networking, you you say the kernel does the forwarding. It's a million packets per second per core. So that's, you know, whether that's slow or not. It really depends on your situation, but certainly if you're trying to do a hundred gig through a firewall service in a data center and you want to do that through that.

Well, that's okay. Now now you really have to do, first of all, again, I'm talking about hundred gigs, but really what the the the number is we should be talking about is the number of packets second. And the reason why we say packets per second, because roughly speaking, again, I'm skipping a bunch of details is every packet per second is an interrupt or a soft interrupt. Essentially.

Right? And this is where the CPU speed then kicks in. So, anyway, so that's, and actually it will go a little bit deeper since we're here with Avi and we can go So that's just the kernel, right?

But a lot of and let's say you're a fancy shop and like we're doing Kubernetes. Okay. So what does a typical Kubernetes will it look like? Oh, actually, you run Kubernetes, basically docker containers on a VM host on hardware.

Right? And so, typically, in the world of Linux, you have these, what we call, Vee virtual network interfaces connecting them. So if you now look at the path and the cost that it takes to take a packet off the physical wire into, say, the hypervisor. So that's the first Nick, you got into the network interface.

So that's an interrupt. So then you go into the, sort of the the VM host and then you go into the docker. And so there's typically three physical, fifth to the VM host, fifth to the container. You might have.

Yeah. Yeah. A magic puddling policy enforcement. Oh, yeah. There's that too. So, but, yeah, just as an example, so there's a few of these, like, virtual interfaces, but they all have a cost.

They all have that, you know, one million packets per second per core type thing. So now if you have a simple scenario, as I just explained, and, you know, there's lots of variants, then all of a sudden your budget's cut in three, right?

Very crude. So anyways, that was one something that, when I was in that world, it's like, well, this sucks because how are we ever gonna build a hundred gig IP set gateway, for example, at, at a reasonable cost.

And so that then So the long story short, and then I started exploring what are the other alternatives. And so there's two main alternatives that that have a bit of traction One of them, people have probably heard of is, DPDK.

And so DPDK is basically a very fast, driver, that, that bypasses the kernel.

But it's only well, with spin locking. Right? So Yeah. So basically, the CPU runs all the time whether you have packets or not.

So it's a little bit of cheating, but, you know, if you work in network heavy environments, you're fully care about giving up a CPU just for the networking. So, basically, takes away and this is part of the challenge. Linux is a multi user multifunction system. And so it's optimized to be as generic as possible.

Right. Whereas for a lot of these workloads, actually, you don't want, you know, you don't want time sharing. You basically say, hey, this CPU, all you're doing is doing network packets. So then you could leverage the l one caching and and all that kind of stuff.

So, basically, that's what DPDK does. But then DBK only does sending and receiving. It doesn't do forwarding, for example. So now you need to build a network function that understands.

Well, this backup with this destination IP or Mac should go there. Right. That's what VPP does. So that we can dig deeper there, but DBK and VPP is one option.

And then you have, in the BPF world, you have XDP. So DBK and VPP completely bypass the kernel. Which is very interesting. So, those the, the Nick, the network interface literally disappears from if you do conflict or IP link, you don't see it anymore.

You gave it to a userland program, and that's taken care of it. Can you can you use SROV and make some of the neck interrupts disappear, or it takes over the whole neck only? Yeah. You can, yeah, you can do s r I o v, and then, it makes it a little bit easier because then you don't have to have dedicated mix.

But just in the slide. Right. Yeah. Yeah. Yeah. Certainly good for testing. Like, if you don't have a machine with multiple because then you need an out of band for management.

The other one is XTP, which actually is a little bit of a hybrid between the two, and it allows you to execute network code. So XTP is basically, part of BpF. So a lot of folks have heard of BpF. And this specific BpF codes related to networking.

It's called XTP, express data path, and it basically executes some of the network codes a lot earlier on in the Linux kernel stack. And so as a result, it's a lot cheaper. It's a lot So before it's above the driver, but before the IP stack, Yeah. It's, it's after the driver, and it's, like, almost immediately after the driver code.

And what they basically what they say in Linux, you have this concept of SKBs.

Which is a structure for sockets, and as soon as you got there, you got very slow. And so they said we wanna do a lot of their work before that. And so that's what you can do. And so one in one of my blogs, I I said, okay.

Let's say you have two network interfaces, and I want to route between them. I still wanna use BERT or whatever as a BGP daemon and a BERT and built on the forwarding in XDP. Which based it's really nice because she got the use still the Linux forwarding table, like the kernel routing table. So you can say IP route, this IP address list behind that Nick, which again, if you use DBGain VVP, you can't.

Like, everything has to be done on the user list. Zero kernel from Yeah. But you get some kernel Okay. Gotcha.

And that this was the big lesson for me. Right? Like, as we went into this journey, I was like, okay. What does this really mean?

And it became very obvious. Yeah. If you take away from the kernel, and you have to do everything in user land. You have to reimplement everything.

So there's no TCP dump. There's no IP tables. You can't type in IP route this. In fact, there's no TCP stack, right?

So all of that has to be reinstated. It's like the monoclonal. It's like people trying to make a monoclonal to run a micro service. But for networking.

Yeah. You know, we're yeah. So you could do it really fast, but you better have to go all in and the team to to actually do that. This is not something you can just kinda doing a second here.

So what was the answer? So what was the answer? One million packets a second through the core through the Oh, yeah. Yeah.

Yeah. Yeah. What what was the answer with, with XTP. I don't have the I think that XTP was something like ten million packets per second and with, the VPP think it was around fourteen million packets per second per quarter.

So, you know, significantly more expensive or, you know, significantly cheaper in terms of budget.

So just switching to XTP, which is sort of a nice middle ground where you get some kernel functionality, but a lot of, speed improvements, Yeah. You got a ten x improvement or so. If if you guys are interested, look at, that's where all the details are. I don't recall the exact, but it was pretty impressive.

It's a very steep learning curve, though. And so you you have to figure out if that's what you need. But certainly some of the new CNIs in the world of Kubernetes are leveraging a lot of that stuff, and a lot of the work VPP and XVP have, C and I implementations. So, yeah, that's a very interesting topic.

You've come a long way from not one not liking coding. But, again, you have to solve the right find the right problem to solve it. Yes. Yes.

Yeah.

No. That's pretty cool. I I remember when I first started seeing people try to do this, and I was looking at Snap Switch, and I'm like, That's pretty fast for lua. Like, what's going on there? That's pretty interesting.

Yeah. Snap is another one. There's a few there. But I think, yeah, XTP is gaining a lot of momentum and I think VPP is is super interested in as well. But, by the way, if you guys are interested in VPP, the project It also goes by Fido, and the project name is f d dot I o. It's a, it's actually a Cisco technology that they open source Apparently, some of the Cisco products, she use it, like, internally.

But, it's pretty powerful.

But I guess that that's it. If you need raw power, go look at it. But you really need it because it comes at a pretty high cost in terms of, like, investing and understanding what you're doing.

Like I'm gonna replace a party with NGINX or something. Right? And it will take ten minutes to figure out. No.

This is gonna be measured in days or weeks. That's a a separate topic maybe for another time that the whole idea of white box and what's the idea of running Linux and it's nice, but you know, be careful what you wish for. You know, when you don't have vendor support for everything. And I think, you know, again, a separate topic, but I think has been some interesting middle ground.

But I wanna ask you, so I saw all that work, which I consider to be low level. You know, you have to know how the systems work. You have to understand what's in the kernel, what layer. You know, I like that stuff.

But then I've been looking at my socket. I'm like, looks like Sassy, but and, you know, I was following Tail Scale on some of that stuff. I'm like, so that's, like, you know, sort of up a level I was like, well, that seems pretty simple, but it seems like there's some interesting things that you can do with it. So I'm sort of curious what what motivates you.

Yeah. You know, myself. You're kinda curious. Like, what's this guy up to? He's all over the place.

Starn is doing this, and doing that.

Yeah.

So maybe, yeah, so first, that was all around the cloud native networking type stuff, the data plane networking and the control plane. And then, Another project I was working on, yeah, is my socket.

So maybe, a little bit like what that is. So so sort of the ten thousand foot level thing is, you know, what is my socket? It's kind of an alternative to remote access VPN type stuff. So I think the, the what is the challenge with the typical VPN is, like, when I think, like, when I was at Cisco, any big orb, Like, I want to onboard Avi as a contractor, like, and he needs access to wiki dot corp dot com.

So I have to remember to create an IT ticket two weeks ahead of time. And then, you know, we trade a corporate account. They would send you the AnyConnect line, then you could VPN in. And then the challenge to that is, well, Avi is now VPN ing in.

Now he basically becomes a member of the network.

Again, I'll use Cisco as an example. That's a very big network. So every smart guy, he's gonna poke around. Right?

Yeah. See what he can do. And, you know, it turns out we also gave him corporate credentials because they use SSO for Vicky. And then so now he finds a GitHub server, and chances are you can actually log in.

Right? So that's a typical problem. And so I was like, okay, how do we solve this? Now, there's a lot of people that have thought about this.

And and essentially what my socket was is like, how would you build a solution to that? Something like what Google calls beyondcorp, other people have called. Private access, zero trust type stuff. So that's what my focus is.

So the idea is that, you know, you have this Wiki server, it actually kind of dials out to, in this case, the my socket Edge secure tunnel for just, say, point four four three, that particular service.

And then Avi says, okay. I wanna go there. Week e dot corp dot com, and then we act sort of as this bouncer in the middle and say, okay. We have a checklist.

It's Avi allowed. We also check what Avi is doing exactly. And then, you know, the more, and then we stitch those two connections together. And the nice thing is that the the Wiki example it can be in a private VPC behind NetS because it's an outbound connection, so no VPN changes are needed.

And, and, you know, Avi can be anywhere. And it can also be clientless. You don't need a VPN client or something like that. And so one of the things I'm doing, if it's sort of from a technical perspective, really drives me, it's like, Okay.

HTTP was relatively easy because there's a lot of proxies out there. So how do you build more application aware proxies? And and with application aware proxy, I mean, like, it can speak the protocol like HTTP. And so I just is is quick tough to I haven't looked at quick the ones that I've been looking at are more of the typical remote access use cases like, SSH.

So, you know, have a bastion host. Normally, you need a VPN, right, or a router. Like, how I don't wanna use the VPN. Okay.

So how do you do it, so then I built in my own sort of assess proxy, your own implementation of that. And with things like goal, it's actually relatively easy. So I learned a lot of go. And because you speak to protocol, we can we can do session recording we can kill sessions or that kind of stuff building an SQL proxy right now where we can log all the queries or can even modify what's getting back.

If Avi doesn't select on users, I'm garbling the email addresses or something. So that's the kind of stuff I'm working on with with my socket, where you're basically sitting in the middle as this application aware and provide, policy, authorization, authentication, and enforcement, and recording, and stuff like that. Yeah. I would encourage people that are looking at trying to technology to look at what Andre's done.

I like the, you know, I try to think the same way. What's the minimum covering set that can that I can build to demonstrate the concept? And then build on top of it towards and then all of a sudden it's like, oh, look, it's a Sassy zero trust and everything platform that you could do, you know, whatever. Then that'll be the interesting question to ask next time is what are you gonna do with it?

You know, just project or, you know, company side because, There's lots of different takes, on on the direction things are going. I'm actually here. I'm actually in Vegas. This is my virtual background, but of my actual home, But, I'm in Vegas at Black Hat, you know, as we're trying to it's much smaller than before, but, you know, it's just interesting to try to poke through the marketing and find figure out what people are actually doing.

And that was part of my journey as well. Like, a lot of these things I start with, like, you know, I don't really understand what this I've keep hearing about it, and then I get slightly frustrated because I don't understand it. Right? So, and that's how it well, let's just build this DPDK thing or STP thing.

And and in this case, like, well, zero trust are very popular. And I was like, I don't really understand anymore what this is. And I just go ahead and go, like, well, if I would build it, what would it do? And then it's kind of a forcing function to go and figure this out.

So I'm sure now that you're at Black Hat or Defcon in Vegas. You've heard zero trust, like, a million times, and everybody means something different.

Yeah, it's been a good it's just a great exercise to figure out what you think it is, because you're forced to, if you're implemented, you better understand what you're doing. Yeah. Actually, in two thousand three, for the first issue of ACMQ, Eric Allman, who I was like, oh, mister Send Mel. Reached out and was like, Would you do an article for this?

And I'm like, what do you wanna write about? I was like, I don't know what to write about. I said, what what does port eighty make you think of? You're you're a Akamai.

You know? I thought. I'm like, it makes me think oh, I I think I can say this, of the illicit tunnels that I have through port eighty to my home stuff because they're firewalls. I didn't like the firewalls.

And I'm like, so I wrote it, you know, said securing the edge. So if I had said zero trust, it would have been really cool. But Yeah. You were way ahead of your time then.

But, actually, I was just I just talking about IP. So I didn't have the user concept, you know, in there. But, you know, obviously, nowadays, that makes sense. Right?

Who's the user or the role or the service or, you know, that tie it all together, right, user, expected role, like, and then, like, the application, and you can tie in contact Like, hey, Avi is all of a sudden in Vegas at Black Ave. Maybe we want to. But, yeah, and his laptop's doing all kinds of weird things. So maybe we wanna drive back a little bit of his permissions and stuff.

You mentioned something which I think is pretty cool because, you know, it's been part of the journey of networking for me is I describe networking sometimes as lots of little simple things that interact in complex ways convolved with vendor bugs.

And, you know, the average, sysadmin might find a kernel bug in their lifetime nowadays.

Didn't used to be the case, but nowadays, but, you know, the average network practitioner, you know, it might be monthly or quarterly or yearly depending on you know, how active you are. And, sometimes, you know, the other thing with networking that I found is that you know, it can be hard until you put your hands on. You think there's complexity that there isn't. Like, I remember when I was learning BGP, I'm like, there must be performance based stuff. Something must make make things change when the performance is bad. And, like, no, there isn't.

And so, you know, what you just said is really powerful, which is, like, sometimes you just need to put your hands on. Well, that's because it's all marketing or because, you know, it can be hard in any kind of technology.

You know, to see the description and put it in your head until you really put your hands on.

And, you know, I love that you're making, you know, all the blogs available and all that, but, you know, encode available for people to play with too. That's pretty cool. Yeah. No better way to demystify some of this stuff than, than just getting started.

But you know, to be frank, it is scary. It's intimidating, and it takes a lot of time. So you do have to have, that time. Not everybody has that.

And especially not not every employer allows you to do some of that stuff. But that's the best way. Just make keep your hands dirty and, you know, keep, you know, stay in shape, essentially.

So, you know, one of the things I like to ask people is sort of, what's what's hot and what's hype So it sounds like you're you're, big on you're both big and hypey. I'm sassy, but you're big on on the what you can do with Linux and disaggregating, but I'll I'll ask you, you know, the question. Like, what do you think in networking in general is, like, you know, really, really cool? And what do you think is being talked about maybe more than than it should be? Oh, that's a deep question. Yeah. There's a there's a lot of hype around, well, you know, zero trust and stuff like that.

There's a segment of things called, SDP software defined perimeter, which is actually quite interesting. So I don't know if this is exactly a question, but certainly what I've seen over time and, you know, a lot of actually folks like you will like, So traditionally, networks are dumb. Like, this is a quick dumb fossil pipes. And then there's been a strength over the last twenty years. You know, somewhat driven by people like us that wanna make more money out of the network where our employers want is like to put more smarts and services in the network. In the network gear into firewalls and, like, all that kind of stuff.

In fact, you know, can take is a company like, we need we we wanna extract value from the network value in terms of visibility and stuff like that. And unfortunately, as an industry, we have been trying to put more and more stuff like that.

Into the devices, into the networking. And so never gear has gone from just doing layer two layer three all the way up to layer seven. And this has been very, I've seen this around me, I guess stressful or, difficult on network engineers. And you know, that's why I heavily biased think in terms of engineers, internet engineers, network engineers are some of the best around.

Because they are forced to understand the full stack. If if, you know, I remember outages in various companies, that if there was an outage, it typically the network engineering team that, first of all, they had so many scars. They were battle hardened. So they were cool under pressure.

But they also understood in a lot of chaos. They were very good in, like, trying to figure out where is the problem? Is it in this stack, this stack where, because they understand all but there's also a lot of cognitive overload. So this has been great or has worked well for people like me and you that have been every year, we learn something new and sort of driving, or, or, sort of, serving that wave.

It's a lot more challenging for people that come out of university, and want to become a network engineer. Right. But first off, there is I don't think there is an education or a program. There's lots of computer science stuff you know, so that's that's a problem.

And I don't know if that really ever existed, but the world into sort of network engineering was a lot easier. So there's this problem for new engineers and even existing ones that are called, you know, just cognitive overload. You need to do so much nowadays. And, depending on where you work, the network side are seen as a valuable asset.

You know, certainly the world there I came from is, like, we were delivering network rattle and services. Right? Was seen as a strategic advantage if we and and same for Amazon. The more we invest in it, you know, the better and stuff.

Whereas if you work in a more enterprise environment, you're typically just the call center. Right? And so that's that's very challenging. But you still have to deal with this cognitive overload.

So why I'm saying this like, I feel like we've reached a top, like, a peak. And right now what you're seeing more and more is that networks are made dumber again. It's just a good thing. In my opinion, And we talked a little bit about, service mesh, and a lot of that stuff is now built into applications, essentially.

And I think service mesh as hypey as it may be. And, you know, like, that's the whole question. What does service mesh mean to you and me?

But a lot of times I think that's interesting that you see. So basically what surface mesh provides, it's an alternative to your traditional load balancer or traditional firewall or whatever, right, where traditionally we would, put a load balancer in the network, and we would funnel all the traffic through there and from there on, it would, you know, go to the things again.

Service mesh is different. It's also in a much more meshed way, if you will. But it's interesting that those, responsibilities are now being basically put in a different part of the stack and also in different teams.

Similarly with security where you have firewalls. But if you look at the cloud, we don't have firewalls anymore. Now we have security groups and whatever, under managed by the teams itself. And I think Those are all good things be because no network engineer I know ever, like, doing ACLs and stuff like that. Although, they will always the voice of reasons, like, Are you sure you wanna open this up to the whole world? Right.

So I think that's, a bit of a trend that that I'm seeing where, and I think that's good for network engineers. It's also good for innovation in general because, other teams, can can kind of reimagine that. Now whether that's going to be as scalable and as good. Like, look at, Kubernetes networking.

It's, it's like, well, it's getting better slowly, but it's been a big Corbernetes networking is flexible. It's flexible, but, if you came from the world where I came from, where you want to build network functions, like, you want to build a firewall? Okay. So first I have to go through these three levels of network interfaces.

Oh, and by the way, there's like three levels of padding as well. It's like This is great for proof of concept lab stuff, but there's no way I'm gonna get significant amounts of traffic through this. Right? So there is, anyway, that's part of just the evolution, eventually, they'll solve that.

But I think, certainly a trend that I'm seeing. And I'm not quite sure if it's good or bad, but it's certainly good for network engineers that have is tremendous cognitive overload, I think. I think it's, you know, when I was a an ISP in the nineties, every year, it's like a a paradigm shift. And it was awesome, you know, for people that hate being bored, which I hate being bored from a technology perspective, Uh-huh.

But I do see a lot of customers, a lot a lot of people sort of confused between the service mesh and then istio is gonna be hot, but it's a coordinator what are you running underneath? Then network mesh and what some of the network meshes are doing? Because ultimately there's, like, there's policy, there's load balancing, there's telemetry, there's all these things. And and I do I do sort of like the separating into dump pipes versus the services on top.

But I think there still is a lot of question about, is it gonna be, how much is sort of the load balancing service meshy stuff? And how much of that network intelligence and balancing and things like that are gonna happen down there. But it's interesting to say. I I I agree there's It is an area of of hotness and hype, you know, altogether.

And, you know, we see a lot of innovation, and I look forward to to seeing and figuring it out. We keep it pretty simple with our infrastructure back end, you know, because we want the training to be simple and, you know, stuff like that.

Enough of that. Yeah. It's super fascinating. The network service mesh stuff, like, you know, basically, all these overlay tunnels going everywhere, it certainly allows for more innovation I find, I I know we got a lot of questions from that sort of from other teams to the network teams that I led. And, I mean, we we we wanted to do it, but There just wasn't enough time. And then, what you see in in several companies is where there's now two teams. So now there's the network team, the big pipe, don't pipe stuff.

And then there's an overlay team, for example, somehow embedded, or, you know, however you want to implement that. And that's where you get to do a lot of smarts. Okay? I only can see the networks that Avi has invited me to. And, you know, we provide the encryption on that level for sort of the the the underlay novel.

And we provide authentication on top of that and failovers. And and now it's all software. And so you're no longer limited in what you could do on the big Cisco juniper Nokia, whatever our box is. Right? And and now what you wanna do is only limited by your imagination.

And so that's Good. Makes me very excited. It's good. It's exciting, but it's scary because how do you document? Like, all these again, it's many simple things, but they can evolve in complex ways.

You know, something else you said know, you're talking about the lab days when we started talking.

And it used to be said that, you know, until you've destroyed a system, you weren't a real just admit. So you've taken down the internet. You're not a real, you know, internet worker.

Is that easier or harder? I mean, do we have labs that simulate that? Was there more freedom, less freedom. You know?

Yeah. That's a great question. Yeah. I, I have a few of those scars myself. I'm very proud of them.

I I wear them with pride, but, if I think back of them, they were very stressful.

And I can, you know, having worked in a lot of operational environments, and seeing, you know, training new engineers sometimes I feel a little bit bad because, it's a lot harder. Well, it's still very easy to make those mistakes, unfortunately. Right? So although we try to put in more, controls, but the cost of making those mistakes are so much higher nowadays. Right?

Where when you created some kind of routing loop, or I dropped their NACL somewhere and, you know, a pop went off, it sucked for a few minutes, but then it wasn't the end of the world. But now, you know, depending on your environment, there's, like, millions of dollars and, just really bad press or whatever. And so I think those mistakes will still get made, but the cost is a lot higher, and, it will make a lot more impact on the on the engineer. So I think that's a very So you see this trend now with hookups, you know, when there's and I think that's great, that's been a change over the last maybe two or three years, because, you know, I'll just do happen and humans do make mistakes.

So, hopefully, we'll learn from it. But it's certainly, I think, harder. Well, it's not harder to make mistakes, or maybe it is, but the cost is so much high higher. So people are less willing to to experiment.

They're a lot more careful.

Yeah. No. It's it's, definitely interesting. There are people are trying to do labs, like, to simulate the internet and peering and all that, especially because It's not just about technology.

It's about politics, which is sort of layering, layer, depending on how you look at it, you know, especially with internet, you know, engineering, but you know, how do you, you know, help there, and, you know, give give people. Because sometimes until you really get it in your head, like, the dangers of redistributing routing, which hopefully no one's doing much, but, you know, routing protocols. But, like, until you see it and and it can be hard to really, you know, internalize. And the thing about automation is it can save a lot of time or it can do the wrong thing really fast and really well because, computers are are you dumb and will do what you tell them to do.

Yeah. I don't need to remove this thing. I've done. I've done it everywhere now. Like, you know, and there's a global outlet.

And even every now and then we hear, like, I mean, recently, there was, you know, something with Fossely and then Akamai, I think, two, three weeks ago, and it's just hard down, right? And, you know, whether that's not related or DNS, it's the same type of challenge.

I I fully agree. And the other challenge is as we are trying to scale these environments, we're actually putting layers on top of that. So there's a lot of network engineers between quote, and I don't mean this in a bad way that actually never log in to routers. All they do is make changes through a GUI or a pull request an agenda template, and then magic happens, which is great for scaling and automation and and makes the network safer, assuming you do testing on them.

But the challenge then because, okay, now it broke. Eventually, something is gonna break, whether it's a bug or whatever. Right? And now, all of a sudden, you do have to log in.

And then, you know, one of my my worry is is that over time, we lose that knowledge. Like, you know, that some someone that can log in and understands the TCP three way handshake to BGP, BGP handshakes. And, you know, why why is this route flapping or stuff like that? And That's a very interesting challenge as we scale up, like, how do you keep that sort of more and more niche knowledge, how how there are enough people that that keep doing that.

Yeah. I, as an observability vendor, I like to think at some point we'd have all the telemetry and whatever to do that. But as you said, sometimes You need to get the TCP dump because there's something in the middle, which you're never gonna get telemetry from, which is behaving poorly, and you need to be able to point the finger and go figure it out.

And, yeah, I mean, it's, it's, sometimes it has been handy sometimes my fingers actually know more than my brain what to do with the device.

As as you mentioned, you you do need to get baked a little bit under pressure sometimes and sometimes it's helpful to have your own parallel processing of, like, there's the incident, here's what I'm looking at, and here's what I'm there. So, yeah, it's something I'm thinking about, and I think the community's thinking about is especially with COVID, you know, because because networking, especially internet working, has always been a little bit of a a tribal knowledge and an apprenticeship and you know, in the nineties, yes, you didn't graduate to, above journeyman until you actually broke something and, like, you know, Yeah.

Congratulations. So Congratulations. Here's your medal. Now you have, you know, root access. Exactly. But I it's hard to really understand how bad the internet was in the nineties and and get it all worked, but, now that we live connected over it.

So, those are things, you know, we're all gonna think about. Log about it if you have any great ideas and something I'm gonna keep asking people about. On that note, any advice you give younger Andre.

You know, for the career. Younger Andre. I don't know. I think, I was very lucky to just stumble into the right things and be surrounded by, you know, this lab and the right people early on that you just kind of I guess directed me in the right way and was lucky to take the right risks and just have a good environment and good people around me.

If I think about new network or people that are interested in this, or maybe you're just getting started or earlier in your career, Yeah. I think find those those people that you think you can learn from and that are willing to take, spend some time with you. Find an environment in a company where, things like experimentation are encouraged versus, like, you know, just don't touch it, that, you know, is this fear, and you'll never learn much.

And those companies do exist, just talk to a lot of folks.

Read things like nanog and stuff like that, you know, even that has changed a lot. But, you know, just even by learning and writing the presentations, you'll you'll learn so much.

And to be honest, the one thing that was very important to me was, that I early on learned how to code essentially. And I still think that some of the best people I've worked with both understood basic. And I'm not, like, you don't have to be DIAD programming, basic scripting. And nowadays, you know, especially if you're networking, Python is kind of the go to language. There's tons of libraries out there for network folks. I think that's very important.

It will set you apart from some of the others. It will make your job a lot more fun too. I think there's just so much more. All of a sudden, you're no longer you know, handcuffed by what the vendor allows you to do. You can build your own systems and change things. So I think that one is definitely one of my big tips. Spend some and it's if you if you're not if you don't know how to do it or if you're never done, it can be very intimidating.

But this this has been the same for me with, like, d p d k or x d p or or a service mesh. Like, I didn't know any of that. And so what's hard is to get started. And it's almost like, you know, I don't know, this weekend or I'm gonna take two days off or maybe If you're if you're lucky, you can talk to your, your manager or your boss and say, Hey, I wanna spend two, three days just to get going because those first two, three days are the hardest.

And just watch some videos do a course on, like, you know, there's lots of course websites out there. And then once you get going, you know, once you you've written your first few scripts, then all of a sudden it starts to unlock, and then you can start to, solve your own little problems. And then the future is, limitless, essentially. Yeah.

So I guess I'll follow that up with a few thoughts. Yeah. You know, first, I wanna encourage everyone listening to do what Andre's done. It's something I I have done too.

It's like, as you learn, document and and teach, If you have a question that that that was confusing for you, guarantee you. It was confuse it's confusing to someone No. You will not look stupid. You will look smart by breaking down the things that were confusing and then how you got unconfused.

Right? You know, it's like the kernel is this big thing, and then we try to understand what it is. And here in a different way, because different people learn differently you know, and need that. And so we'll just encourage people that that that also is really helpful to the community. And also acknowledge what you've said a couple times, Andre, you know, we've both been pretty fortunate and and lucky that the things we we found our way in, we had access, We had privilege to, you know, take time, and do what we thought was interesting.

And, you know, if people are looking to get in, people are looking for mentors, people are looking for pointers, feel free to ping me. You know, I'm Avi at avi dot net, avi at kentech dot com. People like Andre, you know, if you demonstrate that you're reading and thinking and and interested and can be passionate, We're happy to help connect you to people, you know, in your area of interest and try to grow the community because it's something we're all actively thinking about. As things get more and more abstract and and, even as the world is opening up, hopefully, safely post COVID, there's gonna be different patterns and we need to figure out how to how to teach and grow these communities. So, thank you for being on and sharing, Andre. How can people find you and reach you?

Yeah. I'm on Twitter at a toonk, a t o o n k. That's the best way to reach me or check out my website at about all my adventures around exploring some of these things and learning sort of in the public.

Cool. Well, thank you very much, and I look forward to, maybe in another year, seeing, what progress you've made on my socket, and whether you have a third, project, maybe product company, coming coming down the place. So Awesome. Well, thanks for having me, man.