Building and operating a network has certainly changed over the last few years. It’s no longer a matter of just knowing how spanning tree works, or all about OSPF, or how to configure a VPC on your data center switches. Looking over the landscape of the industry today, we can see network engineers very much involved in public cloud, in programming, and developing an understanding of the nuance of how applications actually work over the network.
In this episode, Tony Efantis, CCIE and principal network security engineer working joins us to discuss what we expect from network engineers today, and what it means to build a successful networking team in 2023.
Tony Efantis is a Principal Network Security Engineer at SealingTechnologies. For 10 years he's enjoyed the wild ride straddling Network & Security Engineering with packet level deep-dives and enterprise visibility.Tony's Blog
Phillip Gervasi: Building and operating a network has definitely changed over the last few years. It's no longer a matter of just knowing how spanning tree works or all about OSPF or how to configure a VPC on your data center network switches. Those things are all still important, but expectations have changed. Just looking over the landscape of the industry today, we can see that network engineers very much need to be involved in public cloud and programming and developing an understanding of the nuance of how applications actually work over the network. And this makes sense to me at least because when it comes down to it, the network is the actual delivery mechanism underlying what we all care about, accessing our applications and getting to our data. So the network touches everything. The need to know networking fundamentals is still there, but so is this other host of technical knowledge just to figure out why something isn't working right in today's very complex environment. So with me today is Tony Efantis, CCIE and principal network security engineer working very much hands on and in the trenches on a variety of networking projects. And we're going to be talking about this in detail. What do we expect from network engineers today and what does it mean to build a successful networking team? My name is Philip Gervasi and this is Telemetry Now. Tony, welcome to the podcast. It's good to see you and well, I can see you. Most of our audience cannot because this is an audio only podcast, but it is good to see you again. Last time I saw you, I think was at Cisco Live a few years ago. No, last year. When was that?
Tony Efantis: Yeah, yeah. No, that was, I'm looking at my calendar now. That was 2022, right? That was in Vegas.
Phillip Gervasi: Okay. It was last year.
Tony Efantis: Yeah, that's right. You were there. You gave a great, what was the name of that section of the World of Solutions area? You gave a talk over there, a little private thing in front of the, it looks like a little projector and stuff. A little side room.
Phillip Gervasi: That's right. That's right. I don't remember the name of the area, but I gave a talk on network observability. We got into some data science stuff. I love that stuff. It's really interesting for me to talk about right now. But now before we get into anything, I mean, I'm really excited to talk about this because it's so top of mind to talk about careers and transitions, especially because I made a big transition from the traditional network engineer to what I do now, not even two years ago. So it's top of mind for me. But I'd like to hear a little bit about your background from a technical perspective, what you've been doing as an engineer, what kind of engineer you are, what you're doing right now.
Tony Efantis: Yeah. Oh, first of all, I want to tell you, Telemetry Now, every time I hear that, I can't help but think Jerry Stiller on Serenity Now, on the, gosh, what is the name of that episode? I think that's called The Strike. Is it called The Strike?
Phillip Gervasi: Is that the name of the episode?
Tony Efantis: No, I'm sorry. Gosh, The Strike is the Festivus episode. Anyhow, every time I hear telemetry now I hear it in Jerry Stiller's voice. So rest in peace Jerry Stiller.
Phillip Gervasi: That's exactly what we were intending. So I'm really glad to hear that.
Tony Efantis: Yeah, yeah. So anyhow, for me, I'm actually coming up on 10 years in IT, 10 years in IT. I made a career change when I was 28, and I went and got my CCNA, and that's when I left the construction field behind and started basically a help desk job. And I just loved networking. So even though I was doing help desk and kind of mundane windows management tasks, I kept studying. I pursued my CCMP on my own and did that. Finally, I got the opportunity to actually be a network engineer for a company as a contractor, and that was a lot of fun, and I just kept pushing and doing the grind. I think most people that I've met, if they know me, it's from my CCIE studies and how I broadcast that on YouTube. So I worked my way up all the way to the CCIE achieving that in a fairly short amount of time. But it was fun. I didn't feel rushed at all, and I enjoyed it. It was stressful at times, but it was cool. But at that time in my career, because I got my CCIE, I become this principal in my company, and my company is a small business, I don't have hundreds of people underneath me or anything. I was the lead network engineer out of a couple dozen people who do a little bit of networking and who have expertise in other domains. And at that time, for about the next year or two, I was, I don't want to say promoted, but I kind of say this jokingly when I'm having a sidebar conversation with people, but I got promoted off the keyboard. There was actually a period of time when it was a year and a half before I saw the command line of a router. Because I was now the principal in this role, I had to become a mentor and a guide, and I had to handle proposals. I had to be part of proposal writing teams and all this stuff to help grow the company and show off the best networking face the company had and network security to our clients and things. And it was like a year and a half before I touched command line interface, and then COVID hit, and I had a fun project then, but I wasn't doing a lot of hands- on command line jockey stuff. That's what I call them now, command line jockeys, inaudible getting in there, show IP interface brief. I wasn't doing a lot of that. I was actually doing a migration of physical firewalls to virtual firewalls. So there was a lot of VMware work. We weren't doing NSX or anything. It was basically a one for one, we called it a P to V basically a physical to virtual migration, moving the rules one to one right over. And so that is not your traditional networking. I mean, I went through the entire Cisco NetAcad all the way to CCIE, and I don't ever remember getting in VMware and configuring Palo Alto firewalls. It wasn't part of their portfolio, but the expectation was there that like, Hey, this is our project. You need to be able to handle that. And was no problem for me. I had some basic VMware skills. I think as you progressed through your career, you may focus on something, but just the more work you do, you always get a little bit more experience that's out of your wheelhouse. And for me, a lot of labbing for networking is done on a VM, on VMware or something. And so I had that experience and it wasn't complicated, but I needed to know that. But now we're talking, I'm getting up on my 10- year mark. I recently started a contract. I'm supporting an agency of the DOD again, and we've built an awesome networking team, and I am surrounded by command line jockeys, and I love it. I love sitting here. We got an excellent team of about 15 people, varying expertise, varying backgrounds, and you look on every single one of the monitors and there is more than one command line open. And it's been a long time since I've seen that, and that was the environment I thought I was going to be in, that I was doing the certification grind to be part of. And finally I'm in it, albeit I have a little bit of a different role, but I'm in it in the middle of it, and it's very cool. It's very cool now.
Phillip Gervasi: Do you think that the help desk job that you started off with many years ago now set you up to have a broader understanding of how these technologies all work together as opposed to having a very narrow focus on networking only with the blinders on, so to speak, the tunnel vision, so that way you don't really see how other technologies and networking interact with each other and how they're interrelated? Does that make sense?
Tony Efantis: Yeah. I'm actually, I'm going to tell you straight up, no.
Phillip Gervasi: That's not what I expected to hear.
Tony Efantis: I'm going to tell you no, I'm going to tell you what the help desk did teach me. I actually think no matter what you do, you're going to get those basic fundamental technological skills in your career, which is how to plug in the printer, how to ping the printer, how to ping your gateway, the Internet's not working. Do you have DNS? Can you resolve DNS? These fundamental components of a working network, whether it's a home or a business or an enterprise, there's some fundamentals. Can you ping the gateway? Can you resolve DNS? Can you externally resolve DNS? Is there a trace route? These are simple. I think anyone is going to pick that up whether they're in help desk or anything else. But what working the help desk did prepare me for was the end user experience. I was actually taking calls for end users, end users that would say... We used to joke and say, we used to take calls all the time, and the call was a single sentence, or the ticket that would come in was a single sentence, " Internet's broke." That could be thousands or I don't know, innumerable amount of things, but they just boil it down to Internet's broke. But that taught me what the end user experience was, right? Because from an IT professional, even though I was new in my career at the time working help desk, you learn those fundamentals of can you ping your gateway? And they're like, I don't know what that is. So you walk them through it on the command line, or if you have the ability to remote into their machine, sometimes you can say, " Hey, do you mind taking your hands off the keyboard? Is it okay if I remote into your machine? I'm going to do a couple of tests." You get to the point where you're pinging Google and you're like... That's always the test, right? You're pinging quad eight, you're pinging Google's DNS, and you're like, Internet's not broke. But it doesn't matter because that's not the perception of what the problem is to the end user. The perception is when they opened their web browser and they went to whatever domain, it didn't work. And so that's really what the help desk trained me for. So now since then I've been in lots of organizations and lots of teams where I am miles and many levels removed from an end user, but I still have to be aware of what those impacts are, how that end user is impacted, and how the issues that they see are reported back to us. In fact, today, I'll give you a little behind the scenes on my work, we suffered a minor outage. It wasn't a big thing, it was a minor blip. It went down for a minute, but a lot of messages were sent via text, via email, all these sort of different ways to communicate when your network is down and the network wasn't down. It was actually just a functional component of our network that enables remote workers. And because the workforce today has to be enabled to work remote, if you're inable to do that, I don't know if it's unable or inable, but if you're unable to work remotely, then it's a complete outage. It's considered the network is down and it's what's going on. And that wasn't the case this time, but that's the perception. And so getting the rest of the networking team I have, they're all there saying, " I can ping out to Google and I can trace out across our network. And all these things are up and monitoring shows all these services up," but the user's perception is the network's down because they can't get online. And so understanding that and how we can build resiliency into those components is sort of how I'm looking at things now.
Phillip Gervasi: And that makes sense. I think it's important for us in the networking industry to remember that what we do in essence is construct and maintain the delivery mechanism for the services, usually in the form of applications, that actual human beings are consuming on their phone, on their tablet, on their computer. And so nobody cares about the network. The network is the delivery mechanism, and it's only important until it doesn't work. And at the same time, it's profoundly important because it is the delivery mechanism. So here we are. Now, Tony, you mentioned that you just recently built a team of engineers where you work. Can you explain a little bit about what you were looking for when you were interviewing candidates?
Tony Efantis: Yeah, a little bit. And first of all, I don't want to take a hundred percent of the credit or the majority of the credit for building the team. I was part of establishing this networking team, but I think it ranges a lot. So you want a well- rounded team is what you want. And I'm so fortunate now today that's exactly what we have. And what I mean by that is you want a couple of seniors, a couple of, I call them journeymen and then a couple of juniors.
Phillip Gervasi: Okay, what's a journeyman? What do you mean by that?
Tony Efantis: A journeyman would be, I mean, if you wanted to put a certification equivalent, I would compare it to a CCMP. Someone that can... I did this CCMP it was called, was it called Route Switch. It was the three part test, whatever it was. So someone who's well- rounded, who knows a pretty good amount of what's going on in the network and how the network supports the other business functions, I think that's really what they try to get across at that level. And so I consider that sort of a journeyman, you're a professional, you're getting paid to do your job. You are expected to know the fundamentals at that level. There should be no question. You're not expected to know everything. No one's touting you as the expert. So it's okay to keep studying. It's okay to ask questions, it's okay to learn. In fact, it's okay to do that at all phases and levels. But it's expected. And the junior people, they're expected to be exposed to the fundamentals. They should be learning those fundamentals. Some of them might already have them. Maybe they have a good grasp of fundamental. It's always a weird thing getting into the start of someone's career because they may have really good book knowledge and the ability to pass a certification exam, but if they don't have any experience, even a year or two at a help desk somewhere, they might not cut it as a junior engineer for some teams. But that's generally sort of how we break it down. And it's not just in networking. We also needed the network team to have a wide range of domain experience across networking. We got to have a UC person. We actually have someone on the team. I'm going to give him a call- out right now as UC Rob. We have many Robs. And on day one, when he showed up, when I met him, he realized there were multiple Robs and he was like, " I'm UC Rob."
Phillip Gervasi: There you go.
Tony Efantis: And so he named himself and now he wishes he didn't. But there's almost everything we do has a component of security. And I love, let me give you a little peek inside my mind, I love network security. I have always had one foot in networking and one foot in security. Now, security is a huge domain. It covers hosts, it covers containers, it covers all kinds of stuff. And the kind of security that I like is network security. I really enjoy proxies, firewalls, deep packet inspection. I like all the cool things that are coming out or that have been out now with SaaSy products and things. I really love being able to leverage the network as part of the security architecture across the entire thing. So anyhow, so many network engineers that I've seen in my professional career might be great at routing and switching or VoIP or something else, but they don't sort of understand how stateful firewalls work or how our particular brand of firewall treats packets differently than another particular brand.
Phillip Gervasi: Which is especially important if you're in UC.
Tony Efantis: So while we're a networking shop, we expect most people to have some fundamental understanding of the firewalls across the network. And maybe that's just a team specific thing. And I think it very much is, I've talked with other departments and other places and other team members who have worked elsewhere where sometimes there's a firewall team and that's what they do, they manage either the rules or the devices themselves or whatever it is. And for us, maybe we hope to spin off this firewall rule responsibility onto someone else, but at the moment, there's no better team that understands the flow of the network better than the network team. So we sort of take that responsibility on our team because we want to make sure when the firewall rules are entered in the device, that they're accounting for directional flow, they're accounting for what kind of traffic, dynamic ports. There's a lot that really goes into something as simple as a firewall rule. And I know that the network team that I help support has a good grasp of that. And when we bring people into the team, when we hire new people, we're looking for that blend. You got to be a good command line jockey as well as have not just a routing and switching background or if so, we're going to expose you to that and I want you to be perceptive and enjoy that as well. Enjoy having multiple domains under your responsibility. So yeah, I mean just a well blended team, a blended team of experience and a blended team of skills and knowledge domains sort of going vertically and horizontally.
Phillip Gervasi: So can you explain why you organized your team in this way? Is it to have a more efficient workflow based on skill level or specialty areas?
Tony Efantis: I'm not sure. I'm not sure, but it certainly has its advantages. There are definitely some tasks that as a team or as an organization, you want to give to your senior people because maybe they've seen it before. Maybe you just have that trust in them and you know that the task will get carried through from A to Z, or the expectation is that someone wouldn't have that domain of knowledge at a junior level, so it misses them. But by having the team spread out again, sort of vertically in inexperience and horizontally across knowledge domains allows us to move those tasks around the team. We got to do a bunch of firmware upgrades and stuff like that. Hey, almost anyone can do that. We can show you how, I give you a five- minute demonstration, got to upload this thing, schedule it for an outage and hit the reboot button. And while other times we're doing circuit engineering and working with the ISPs and different things like that, so we do pass those tasks around and sort of assign them that way and it just works out great.
Phillip Gervasi: You have your subject matter experts that are going to do some very advanced troubleshooting that a CCNA would likely have never seen before or doesn't understand how a BGP adjacency even is formed in the first place. So you're going to assign that by default to a more senior person, whether they have a CCIE or not by the way. I know that the certification may not be there officially, but likewise, you probably wouldn't say, all right, we have to rename all of these interfaces and then you have the most senior CCIE on staff, multiple CCIEs. Let's do it that way. And this is your job for the next three weeks is to manually maybe even make a script and make it a little bit easier, but nevertheless, super mundane. I can't imagine that you did assign that to that person, or maybe you're all that humble and willing to get the job done.
Tony Efantis: If you want to ask me, my absolute most humble opinion is like, I'll take the interface renaming job because I want to script it out.
Phillip Gervasi: There you go. Make it a project.
Tony Efantis: Because you know what? I don't care if it takes me all day to write the script to cover every interface in the entire enterprise. I'll do it because I find that challenging. For me, it's just a personal growth thing. I've really been embracing scripting and automation lately. I've really done a lot more than I would say, dip my toes in. I'd say I'm in the swimming pool and I'm treading in the deep end. I'm not an expert by no means. I'm learning every day, but there has been nothing I can't accomplish yet with some python CHOPs. There's no task I haven't been able to sort out with some rudimentary python CHOPs. I think I'm doing okay. I might be at that peak where I come down to reality here soon, but I feel pretty good. And so what I'm trying to do at my role, at least within the team and the organization, I always wanted to be surrounded by command line jockeys, people just like configs on every screen and everyone working together and collabing and whiteboards everywhere with scribbles and all that kind of stuff. That was what I thought I was going to be handed when I became a CCIE. And instead I got taken off of a keyboard. And it's so exciting to be back in that environment because now at least where I am in my career, maybe it's just my personal age or the experience that I've gathered in my career, but I'm starting to mature in my job. And so some of the things that I'm trying to help the networking team with isn't all the time solving the most complex problem, they don't bring all their most complex problems to me. They'll bring some and some I will take. Some I'll say, " I'm working that." And it's just for fun, not for fun because we have a job to do.
Phillip Gervasi: No, it's totally for fun, Tony. Give me a break. I do think you and I are alike in one thing. I've seen this thread being kind of woven through a lot of what you said, that you enjoy the puzzle. And I think a lot of us engineers in this field, network engineers, previous network engineer now, really enjoyed the puzzle of figuring something out and then the satisfaction of having figured it out, right?
Tony Efantis: 100%. It's getting to walk away with the trophy that you gave yourself because you chose the task.
Phillip Gervasi: Totally.
Tony Efantis: But sort of where I am now is a little bit more mature in my professional career and in my personal life. And so I have a great team of network engineers around me of great varying levels of experience and across a wide domain of knowledge. And so what I'm trying to do is bring innovation. Almost everyone that we have on the team is an excellent command line jockey, whether they like that term or not. A lot of them will take on a task like, hey, I got to rename these interfaces and they'll just copy and paste everything in a notepad, do a find and replace and blah, blah, blah, and paste it back in. But I'm trying to look at things a little bit differently and say, I know you can do that. That's great if we wanted to get the job done, go ahead, but let's figure out how we can script this." And let's not just developed a single shot script that we put in a folder and go, this is the rename interface script for this device that can't really be reused. We can check the code, but then we have to change it for another device. Let's not just write the scripts, but lets sort of develop things in the most modular way and the most reusable way possible so that once we have the boilerplate done, everything else just goes on top. Whatever your job is goes on top. And there's lots of frameworks out there that sort of do that already, but just getting people's minds set up for that. Gosh, there was a thing today, I think I told you about a little network outage, and one of the things was a device that needs to be rebooted. And we were kind of joking in our network chat channel that Tony is probably thinking about how to script this reboot process. I'm like, I have a Cron Job ready for you. I'm ready to go. You know what I mean? But that's how I get to give back is trying to broaden everybody's way of thinking about these as different problems. Something recently is we have a new team and a new contract and in an existing organization, so we got a lot of people that are coming in blind. I say coming in blind, but I mean haven't seen this network before. And the organization doesn't have a ready tool set like a networking tool set that's ready for us. So we have to figure out how to make the things that are existing on our provisioned laptops work for us. And let me start with the most simplest thing is SSH access to a network device.
Phillip Gervasi: That is pretty basic.
Tony Efantis: This is the fundamental way to access a networking device. I don't even think books today even mentioned Telnet. I think it's some devices, it's not even an option. SSH is the bottom line default for some devices. My point is this is even challenging because we don't have things like secure CRT. That needs to be bought as part of the organization, and that was never purchased. So Windows 10 comes with open SSH if you include that module in your baseline image. And so we're having to do everything through the Windows command line via SSH. And so one of the things I started doing was creating these SSH config files, that allow me to automatically jump through jump servers using key based authentication. So I'm trying to remove as much friction as possible so that the team of network engineers, the good people I have, can do their jobs more easily. And that's sort of how I look at this problem now. I'm not just solving some network engineering problems. I am, that's fun, but I'm also solving some process problems, some team- based problems, some organization problems. And I love coming up with those solutions because that stuff they don't teach you in CCNA, CCNP, CCIE. That is really just from being around, just from being around shops like this and sort of understanding how these systems interwork and what can we do to make it easier for ourselves. I'm enjoying the hell out of it, man. I'm enjoying the hell out of it.
Phillip Gervasi: Yeah, you're enjoying it. Yeah. Yeah. Well, you're enjoying figuring out how to solve the problem. And the very beginning, you mentioned how you changed careers in your late twenties. I did as well, maybe for different reasons, but I changed careers looking among other things for a larger paycheck, but looking for also a career that I can sink my teeth into. And that was a never ending challenge, a never ending project. There may be other ones out there that would've fit me, but falling into technology and then very shortly thereafter, networking perfectly fit the bill because it truly is a never ending project. But I think what you're talking about there is the maturity of having worked in environments, having dealt with these devices, but also just being a more mature human being and understanding how people work together and what can I do to remove the friction between my coworker and the devices that they support that makes their job easier. And I assume they'll be more successful then, and the network will be more stable. You can even take it to that extent, right?
Tony Efantis: Yeah. I mean, that example I laid out is not the end goal, is not the perfect picture of how we want to do things, but it is reducing enough friction that we don't have to get frustrated while we wait for the, what do you call it? The purchase of a proper multiple SSH manager.
Phillip Gervasi: And that was the thing I remember, oh, actually this was in a job interview where they asked me some question about managing some devices and what would you do here? And I gave them this spiel about how I believe that we're engineers, and so we're trying to find a solution to the problem like I've been saying for the past few minutes, and how you don't necessarily need to just rip and replace all your gear. How do the protocols work? How do these devices interact with each other? How can I make this work for the organization in the constraints, the business constraints that I have? Maybe the business constraint is we can't replace this gear or I can't shift traffic from here to here for whatever reason, security possibly. And so therein lies the puzzle, but therein also lies the engineering component that I really enjoyed just figuring out how do I make this work with what I have? Sounds like that that's kind of an important thing for any engineer applying to your team to have if they're showing up to a job interview. I mean, they don't necessarily have to have the terminal degrees from MIT, some PhD in computer science from MIT or multiple CCIEs, but they need to have that drive to want to solve the problem. Maybe the curiosity, I don't know. And that's probably not right.
Tony Efantis: Yeah, yeah, definitely. I mean, just what you described, I'm just smiling over here as you were describing it because that is the best part of some of the network challenges that I have. It is not what is some nerd knob I can turn on some protocol. Hey, come over here, check this thing out. Look what I can make OSPF do. It's not that because in a wide open network, any solution can be used for anything, but it's these constrained environments, these environments where things either can't change or won't change or these are the sort of roadblocks and still making the work, excuse me, the network work. Work with resiliency, work with high availability, being able to provide a redundancy plan for the network in all of these constraints, that's the engineering challenge. It's not a greenfield solution. So one of the things that I'm getting ready to face in the next couple of days is I have to talk about the IPv6 deployment plan for this agency, and I am so excited to be doing this. To be honest, if someone asked me to do it two years ago, I would've been extremely scared. I'm like, I am so ready for it because I realize it's not actually going to be that difficult. I've gotten over the monster that is the IPv6 addressing, right?
Phillip Gervasi: Okay. It is kind of a monster.
Tony Efantis: Just the fact of seeing letters and numbers together just was like, I'm used to seeing things like in numeric characters only, but I'm so excited about this because even though the underlying plumbing of the network is not changing at all, it is almost like a greenfield deployment because the IPv6 and IPv4 are like ships in the night. They don't have to interact with each other. They can occupy the same cables, and so we can actually build a new network if we want to. And it's just so exciting to have that flexibility because I can't move the IPv4 space that's there now occupying the network and these buildings and these lands and everything else, but we can do whatever we want with IPv6. And it's just super cool to, again, there are business constraints. There's a lot of constraints as well as there's an existing network that we have to make it work. But I get to lay down an IPv6 addressing schema across the country. It's not greenfield, but I'm calling it greenfield because there is no existing IPv6. There isn't one. So yeah, we get to do it for the first time, and I'm so excited about it.
Phillip Gervasi: That's awesome. It's been a while since I've been, I use the term field engineer sometimes, but just an engineer out working at the command line and building networks and fixing networks. It's been a while. It hasn't been, it's been more than the two years that I've been working in network observability, because prior to that, I was a solutions architect for a few years, and my experience in the field quote unquote air quotes was in a lab learning Viptela when they were purchased by Cisco or SD access, when that came out and that kind of stuff. So still very cool, but it's been a while since I was in a cut over that went south, and I'm literally sweating from my forehead and ignoring the customer's phone calls because they're just, every 10 minutes, it's like, when will this be up? I'm like, well, if I knew that, then that implies I know the answer, and it would be up. And then eventually you figure out what's going on and the pings go through and you see all the exclamation marks. And I remember even calling my wife and saying, " It's working." I also remember calling my wife and saying, " I am quitting." Both of those things were true on the same night. And during cut overs, I'm, I miss those days a lot. On one hand, I don't miss doing cut overs at two in the morning. I work banker's hours now, but I miss that a lot because I miss that puzzle and figuring stuff out. Do you think that if somebody came to you looking for a job, you have a job opening, you got a rack open and you're interviewing people, and you could kind of sniff out that if you had an IPv6 project coming down the road and it's not on their resume, they're not super familiar, and you could tell that they weren't ambitious enough to go out and learn it because there was an upcoming project, they probably wouldn't be a good candidate. You could say no, it's okay.
Tony Efantis: Yeah. I'm going to tell you, there isn't one thing that makes any candidate a good candidate or a bad candidate.
Phillip Gervasi: Okay, that's fair. Yeah. Maybe I look at things a little bit too black and white.
Tony Efantis: Yeah, I'm sorry I have to be that way, but I'm not sure. They could be dynamite in so many other ways. But what I have a really good team, and I'm pretty sure most of them are not a hundred percent comfortable with IPv6. In fact, I'm pretty sure some of them might be a little bit scared. I'm not going to name names, but I on the other hand, and maybe it's false confidence, sometimes it's hard to tell the difference, but I'm excited for the challenges that we will uncover because... I'm not implying that I have the answer to everything and I'm going to know I'm going to do this right, and I'm going to be a sniper and get everything in one shot. No, but what I am confident in is we are going to uncover some challenges and we're going to work through them and we are going to solve them. And I think that's what makes us as a team. Phil, I wanted to tell you when you were just talking about that cut over. Yeah. I just told you that once I got my CCIE, I didn't touch a command line for about a year and a half. I was doing proposals and business development kind of stuff and doing road shows and things. And two weeks ago, I did my first command line cut over, it was so exciting. Dude, it was so exciting.
Phillip Gervasi: How did it go?
Tony Efantis: It was a disaster.
Phillip Gervasi: It was a disaster. Oh, no.
Tony Efantis: Now listen, I was doing this cut over and the team lead now for the team, the network team that I'm a part of, he is actually someone that I brought in as an intern seven years ago. He was an intern to me, to my team, and now he's the networking team lead. Very experienced, knows this environment and just couldn't be a better person for the job, but because he knew I was doing the cut over, he was like, all right, I'm going to stay tonight. And so it was just going to be me. I was going to do it by myself. And it was a router swap. We were going ASR 1006 to an ASR 1006- X. So a little small router, not a lot of interfaces, but still. I'm going to start it at 6: 00 PM Eastern. And he's like, " Yeah, I'll hang out tonight. I'll give you a hand, whatever you need." And I'll like, " Hey, that would be awesome." He and I just had a blast because when he was an intern for me, when I brought him in as a junior engineer, we did a hundred cut overs. I was the lead at the time and we did a hundred cut overs. And there's just some camaraderie that happens when you stay up all night with somebody.
Phillip Gervasi: Oh, totally.
Tony Efantis: And you two are just working a problem or two people or however many is on your team, but where you and your team, you become so in sync with what the other people are doing that you're like, you're just talking over cubicles because the rest of the office is empty because it's nighttime. So you're just talking over cubicles and you guys know exactly what the other person needs as far as information. I know exactly what you need. I'm going to show you this. It's on this VLAN. It's this subnet here, I'll send it right over, blah, blah, blah. And it was just like we were just falling into an old rhythm and we got those pulse checks from the customer. Any update, guys? I need 30 more minutes please. And then finally we're into it for a few hours, and I was coming up on the boundary of my change window and we got the pulse check, " How much more time, guys? It's getting pretty late over here." And I'm like, " I'm not answering him. I'm not answering." And he's like, " I'll answer, I'll answer. We need some more time guys." Anyhow, it was successful. We had a blast. I mean, when it was done, it was just falling into old times. And I think it might be just unique to network engineering. I mean, I'm sure other industries have it, but when I talk about it with another network engineer, there's some camaraderie there, even amongst strangers that you just know what the other people have gone through, what they're dealing with. And really, I love it. I mean, I'm a little bit older now. Staying up late is sometimes painful, and I'm glad I don't have to do it all the time. But I'm glad that when I do, I'm not doing it because I was assigned this. I was doing this as a proof of concept for this kind of architecture change that we were doing so that I can map it out and hand it down to that networking team and those other engineers in a sort of here's the playbook. You know what I mean? This is what you need to do. Go ahead and run this. And it was so much fun. So yeah, man, I don't know. Gosh, I'm glad I don't have to do a lot of them, but I'm glad I have a long history of cut overs. They're so much fun.
Phillip Gervasi: Oh, absolutely. I think that for me, the most meaningful times in my career as a network engineer out in the field were those cut overs when sometimes things went south and I had to figure it out. And then of course the sense of accomplishment that you did that, the technical lessons that you learned when you figure out a technical issue, you take that with you into the rest of your career for sure. But also when things go south and you can't figure it out and the slice of humble pie that you have to eat to roll things back, you really don't want to and your customer doesn't want to, believe me. And you have to explain to your customer what happened. But I think I was in that position as a VAR engineer working solo almost the entire time in my career because I had an understanding about their technologies, whether that's VMware or navigating a Linux file system or troubleshooting wireless or how active directory structure works. We're pulling that into the wireless system. So if you're troubleshooting a core cut over, not going well, you might need to know all those other technologies as well. And so it gave me a lot of advantage and I think propelled me in my career, even though I was focused on a particular network cut over, having that other knowledge and those skills, I think really gave me an edge. Very much. So I have to imagine then for an entire network team, you want to have that variety of skill set because of the reality of what it means to do networking tasks today and then touching all these other technologies in order to be successful at least. And so for me, I was always alone because I was a VAR engineer, but if I had an entire team, I'd have my VMware specialist and my cloud specialist and my security specialist, all networking but with all those specialty skills as well, because of the reality of what networking does, it touches everything.
Tony Efantis: Yeah, touching everything. I mean, like I mentioned about the team I'm working with now, we have a couple of cloud projects, and it's more than just standing up a tunnel to a cloud environment. It's way more than that. It's orchestrating whole network address management deployments in cloud environments. And they don't even really exist. You know what I mean? They don't exist yet. They're deployable or BGP, I was going to say BGP routes, BGP peers that aren't routers. They're just like software you turn on and now you get BGP routes and wait a second, what routes are these? And there's a whole different sort of cloud networking component. And I'm not an expert in that. I'm very much just have one ear turn to my coworkers who are in deep to that. So I just hear some of those struggles. But that's all within our team, right? The cloud networking team, security's within our team so much we have to touch. And I don't know if that's the right answer. What I will tell you is we knew a lot of these challenges going in and that's why we built the team the way we did with a wide domain of different skill sets and different experience levels. But I know there's a common saying, which is a lot of times the network has to do what the applications can't. And one of those things like you just mentioned, the developers, they write an application and then they're done. And that's right because a lot of the development teams, well I'm just painting with a broad brush, maybe I shouldn't, but I have heard some developers firsthand that when they write an application, when they want to make a network call, they're just calling the operating system and just saying, " Hey, open a socket," or Hey, I need to make a network call. Just let me know when it's ready." From a software developer standpoint. And they're not maybe taken into account the firewall rules that need to happen and the things like that. So again, they're developing a tool for something, I don't know, for whatever accounting or something, but they're leaving the security part to the network, so they're putting that onto us. And I've heard so many discussions recently in different webinars and different companies and different things about doing DevSecOps and trying to bring the security back into the actual development of the application and about how challenging that can be, especially for things that are already written, can't like unwrite the code to put right security in it. So we're still having to enforce it at the network layer. And I'm not saying that security at the network layer, network security needs to go away, but it should be changed. And the developers need to do maybe more than just telling the operating system, " Hey, open a socket for me, let me know when it's ready and I'll give you some data," and just letting it pipe it over the network and leaving network security to something else. But that's the way it is today. And so the team of people I have, they're all really well- rounded each with their own sort of expertise and it's been really awesome.
Phillip Gervasi: Do you care about seeing certifications or college degrees or anything like that on resumes when you're talking to folks?
Tony Efantis: I'd say in my industry, if I call networking my industry, I don't see a ton of value in college degrees. But let me back that up. I'm not going to say the networking industry as a whole. What I will say is network operations, network administration, applied network security in college degrees because they don't teach the kind of challenges that you're going to see on the battlefield, so to speak. Experience is king, right? And everyone knows you got to have a personal brand. If you got to a blog, if you got to a GitHub, if you got whatever and you got stuff worth sharing, share it. That's your personal brand, put that on your resume. I want to see the blog about the BGP changes you made on your last cut over or whatever. That adds a ton of value that lets me know more about your experience. So experience is king. Certifications are, I would say second in line of importance. And this is just my opinion, but because I've been through the cert program, I know what it can do for you when you apply yourself, you can learn a lot. I have learned a lot. I've probably forgotten more than I've learned, but there is a ton of value in actually getting the certifications. As someone who's going to do the interviewing or the hiring, there are a lot of people out there who try to game the certification system either through cheating or dumps or whatever the case may be. And it's your job to suss that out. So don't put all of your weight into someone's certifications they present on their resume or that they talk about. It's your job to suss out whether or not they have the experience to back it up. And I think that's the bottom line. Experience is king. Certifications help, but ultimately experience needs to back up what you have.
Phillip Gervasi: So then if one is trying to build a successful networking team, if you are trying to build a successful networking team, are you looking for candidates that have that broader knowledge outside of networking as well, that know something about public cloud, that knows something about Kubernetes, that knows something about something deeper than the layperson of network security?
Tony Efantis: Yeah, absolutely. Although it's harder to find those people than it is your focused network engineer.
Phillip Gervasi: Is it?
Tony Efantis: Yeah, someone that knows networking. It's much harder to find someone who knows Kubernetes and Cisco and Juniper. Normally they sort of play in two different playgrounds. But you would get the best of both words. I mean if you find those people, absolutely. One of the things that I'm pushing for now is I want all of the engineers on our team to learn some automation and to start to understand what automation can do for you personally, us as a team and the organization as a whole, which is how us automating the network can move the organization and the other teams that support the organization forward. And that's challenging right now as well. I know there's a huge support through network engineers getting into Ansible and Python and everything. And actually when I started my CCIE at that time, I saw a split in the industry. About half the industry went to automation and half stayed on the certification route.
Phillip Gervasi: I remember that. Sure.
Tony Efantis: I went on the grind and so I abandoned automation and I didn't pick up automation seriously until the last two years. And now I'm like, I don't even want to touch the command line anymore. I just want to do everything through some sort of API. So if you have those skills, you are a threat. You are an amazing threat if you are a decent network engineer with experience to back it up and you can code or script even just a little bit, but enough to make it useful in your field. That's amazing. That's amazing. I think I'm just getting to that point where I can start to make useful products for other people in my team to use, pushing out simple scripts, checking different things, go do the config backups for all the devices in the network. Just those simple things are starting to become valuable or are valuable and we're just starting to make use of them. Phil, I don't mean to derail you, but I just had a thought about something that I mentioned earlier, which was maturity.
Phillip Gervasi: Sure.
Tony Efantis: And I mentioned I had a little bit of professional maturity going on and how I always envisioned myself as this network engineer in this awesome team with all these screens around and all these command line prompts and the terminals blinking and whiteboards with markers everywhere and everything. And that's all very exciting and it still is. I talked about how I enjoyed my cut over a couple of weeks ago, but I'm also looking at the network. Well, I guess I should start out by saying at that time when I was coming up, and even sometimes now, you want to show off the network, it's this pet that you built. I know there's an argument of pet versus cattle in the Kubernetes world. But this pet that you built and you want to show people how all your access lists are all capital letters with underscores, no hyphens, and how you adhere to all these interface description standards and how all these things and your protocols are written just this way. And it's an advantage for your topology because it makes this easier. And you want to be able to show people that and show off the network and come over here and look at this and this is cool and this is why it's cool and blah, blah, blah. And I'm mature now and I realized no one gives a shit.
Phillip Gervasi: Nobody cares.
Tony Efantis: No one. And here's what I discovered through my sort of maturity, my personal maturity, professional maturity is we should actually be building networks that are invisible. The more invisible the network, the better. And what I mean is, I think you started our talk by saying this, you're a user, you're using an application or you're accessing an application through the network. You actually don't give a crap about the network. And you know what? It shouldn't matter. It should be the cleanest, fastest pipes possible to get the job done. And it shouldn't matter. It shouldn't matter to the user and it shouldn't matter to the application. Yet a lot of times I see us sort of compensating for different network designs or different applications and sort of doing so with the network and trying to achieve this beautiful topology, this topology that's like a perfect tree of having some sort of root and branching out and branching out and you get to the edges of the network. And I used to think that I wanted to build these perfect networks that the configs were pristine. I think I always hear Greg talk about the artisanal command line, your prompt that you made by my artisanal fingers. And I used to think that was a thing and that when you were a mature good network engineer, you'll get to that point. And I realized that's so stupid. That's totally not the case. The realism is simple is king in network design, simple is king and the network doesn't matter. My point is I have a team of 15 people now to operate a network. The goal would be in a number of years to get that down to three, four people.
Phillip Gervasi: Three, really?
Tony Efantis: You know what I mean? If we can simplify the network, the architecture, simplify the touchpoints and the interfaces, and I don't mean the interfaces like you do interface gigabit zero slash zero. I mean the interfaces between devices and applications in the network. The parts where two unlike devices touch, we simplify that. That needs less maintenance, right? Less administration, less operational changes. Simple is king and the network shouldn't be seen. That's where I'm trying to get to now. So I have a very complex network that me and my team members work on and it's a lot of fun and we have a blast. But ultimately I'm trying to use the knowledge I've gained through other networks and some things I've learned from other people and also through certifications and things like that, and using everything to my advantage, not to build complexity into the network, but to remove it.
Phillip Gervasi: Yeah.
Tony Efantis: And that's like when I had that realization, I realized like, oh crap, I'm a different engineer now because there was a time when I was trying to put complexity into the network to show off what the network can do and that doesn't matter. So yeah, it was a big growing point. And I'm happy to say that making things simple is easier than making them complex. So it's the right decision.
Phillip Gervasi: At the application's expense, right? It's true or maybe I'll make it broader, making the network more complex at the expense of service delivery. So that's the point is to deliver that application or whatever service it is. When I was a VAR engineer, sometimes I'd go into a customer's network and you can tell that the person running the network, they're studying for the CCIE because they are implementing or all these tunneling protocols and you're like, why are you doing this stuff? To be fair, I was also studying for the CCIE. So I'm like, all right, we're going to get our hands dirty here. But I mean we know better now. We know that that adding complexity actually causes problems. Not only is it a waste of time and adding complexity that you have to sift through later on, it actually increases the likelihood of a failure in the whole point of the network, which is delivering the service or the application to a human being. And the reality is, it's not even the application that we care about. When I go to my bank's app on my phone, I don't actually care about the app either. Not only do I not care about the network, I don't also care about the app. I care about managing my money, the actual task that the application is enabling me to do. And so in building a successful network team, I have to imagine that that's a very important component, that understanding that, hey, what we're doing here is all about delivering an application to a person. And so finding ways to simplify the network, not adding complexity, at the cost of the cool factor, of course, should be paramount in any engineer's mind as they're constructing a network. And also managing the day- to- day, finding solutions to problem or how can we continue over the course of time to simplify the system that we manage and thereby increase its reliability.
Tony Efantis: Another way to phrase it, just sort of what you were saying and what I was just talking about previously was we shouldn't build networks just to build networks. We build networks to connect people to data. I mean, that's as simple as I can put it where I think sometimes and maybe my younger years, and I think I'm guilty of probably practicing my CCIE on a few production networks, I'm guilty okay, sorry. But I was building a network to build a network. I was doing it to do it, and now I realize that network should be transparent. It's about delivering data to people or data to other data, system to system, there doesn't have to be a human in it. But yeah, that's what it's really about.
Phillip Gervasi: Yeah, absolutely. But to be fair, sometimes there is inherent complexity in achieving a particular goal. So when we're trying to make managing the network easier, like our WAN for example, and then so we start looking at technologies like some vendor of SD-WAN and under the hood there's complexity, there's overlays. Maybe they're using VXLAN as a control plane or something else. Who knows? I do believe that these are tools, right? But again, it's using the correct tool to solve a specific problem and not just throwing my entire tool back at the problem and saying, " I got this new saw. What can I cut? Or I got this new hammer." And looking for nails everywhere. But sometimes you do need a saw and sometimes you need a hammer. And sometimes you do need to implement a complex technology to solve a problem. So it's not that we don't like complexity. I think it's that we don't like complexity for complexity's sake.
Tony Efantis: When it's unnecessary.
Phillip Gervasi: Our networks are complex by virtue of the fact that they're distributed. And we have public cloud and containers, networks, we own networks. We don't own networks that we don't manage, that we do. And then of course, how do we secure all of that? There's a lot of stuff going on. Inherently complex, whether we want it or not.
Tony Efantis: I forget what the saying is, or maybe it's a quote and I'm not sure who it's from, but it's where when there's nothing left to remove, it's not when there's nothing left to add, it's when there's nothing left to remove. And what that means is you have, whatever you're building is as simple as it can be. It has only the components that are necessary.
Phillip Gervasi: And Tony, I was Googling that while you were speaking. The direct quote is, perfection is achieved not when there is nothing more to add, but when there is nothing left to take away. And yes, I agree with that. Now Tony, I do have to say before we close out that I was following you when you were looking at the results from your CCIE exam. And I remember even now as I'm talking to you, I remember very clearly the look on your face, the hand gestures and the facial expression that you had when you opened up the results when you passed. I think you were streaming it live, I think, right? I assume.
Tony Efantis: Yep.
Phillip Gervasi: I remember that. And you were just like, you can't see this. This is audio only. I wish the audio could see it, but you were shocked in awe. Were your kids there? Were your kids around?
Tony Efantis: Yeah. Yeah.
Phillip Gervasi: I remember that.
Tony Efantis: My kids, I always brought them in for the reveal. I don't know why I chose to do that. But throughout the CCIE studies, it's kind of a common knowledge or I don't know, a commonly talked about thing that your entire family pays the price.
Phillip Gervasi: Oh yeah, sure.
Tony Efantis: For you to study, for you to achieve your CCIE. And so in doing so, I wanted my entire family to be part of the moment of success, of the joy of success. And so I brought them in even on the first attempt when I was pretty confident I was a zero, but I ended up walking away with a, I think one W there, what do they call it? Pass. So you need four passes to pass. And I had one on that first one, but yeah, the last one. Yep. My kids were there.
Phillip Gervasi: Yeah, I remember that. Very cool.
Tony Efantis: It was great.
Phillip Gervasi: Yeah. So Tony, I think this is a good place for us to end here. And I just want to say thank you for joining me today. It really has been a pleasure to talk to you, engineer to engineer, and swap war stories about cut overs and the lessons learned from over the years about being a network engineer. And also to get your thoughts about what it means to be a network engineer moving into 2023 and what it is to build a successful networking team. That's really been the underlying theme of today's podcast, though we wandered all over the place and I love that. So again, Tony, thank you. Much appreciated.
Tony Efantis: Yeah, thanks for having me.
Phillip Gervasi: So Tony, if folks want to reach out to you with a question or comment, how can they find you online?
Tony Efantis: You can find me on Twitter, I'm inaudible and that's probably the best way. Look for me there.
Phillip Gervasi: Great. And you can find me on Twitter as well, network_phil. Still very active there. And you can also search my name in LinkedIn, which I also am pretty active in these days. Now, if you have an idea for an episode or if you'd like to be a guest on the show, please reach out to us at telemetrynow@ kentik. com. And you can also follow Telemetry Now on both Twitter and on LinkedIn. So until next time, thanks very much. Bye-bye.
Do you dread forgetting to use the “add” command on a trunk port? Do you grit your teeth when the coffee maker isn't working, and everyone says, “It’s the network’s fault?” Do you like to blame DNS for everything because you know deep down, in the bottom of your heart, it probably is DNS?
Well, you're in the right place! Telemetry Now is the podcast for you!
Tune in and let the packets wash over you as host Phil Gervasi and his expert guests talk networking, network engineering and related careers, emerging technologies, and more.