Service Provider Cord Cutting with Greg Villain

Phil Gervasi : My name is Phil Gervasi and you're listening to Telemetry Now. And with me today is someone who I'd say has the, I'm going to say, second best taste in music of pretty much all time. Greg Villain is the senior director of technical product management at Kentik. And today we're going to be talking about cord cutting and the impact that it's had on service provider networks. So let's get started. Greg is an expert in service provider networking. He has worked with web scale companies, companies that you'd recognize. He's the president of France- IX, the list goes on and on. And if you look at his LinkedIn profile like I did earlier today, it's going to make you want to go study and just become a better engineer. Greg, it really is great to have you on the program today. Good to talk to you.

Greg Villain: Thanks for having me here, Phil. And also you mentioned previously that I was your second best music choice person. Now I want to know who the first one is. I got to unseat him.

Phil Gervasi : You're curious about that, huh? So you and I are similar in age, we grew up with a lot of the same bands, I know from having hung out with you, and so we have that same foundation, so that's why you're right there alongside me as second best. But what I saw was that you tended toward the hardcore punk scene from the grunge days, and then I went in the direction of progressive rock, progressive metal, but we have that same foundation, so that's why I still count you as a close second, but it is second best.

Greg Villain: We'll battle this offline, don't worry.

Phil Gervasi : Yeah, battle of the bands. So before we get eyeball deep into this conversation, I do want to ask you what do you mean by cord cutting? I think I know what it means, but I want to hear it from you, and how long has that really been happening in the industry?

Greg Villain: Well, as far back as I can remember, in the early, I'd say, 2000s we started seeing internet service providers going beyond the duty of just offering internet access. We started seeing what you'd call back then triple play offers, which would be a bundle of internet access, voice over IP and also IPTV, and so back then it was just using internet as a conduit to relay IPTV on a set- top box, but it was really what gave what we now call OTT, over- the- top, content provider the idea that independent content from linear programming could emerge and be brought to end users this way. So that's cord cutting. It starts with voiceover IP really and it is what it is now with user- generated video content, streaming services that have been multiplying over the recent years and so on, and gaming also if you'd like. So it's a broad topic, but it's basically users progressively going away from, I'd call them, monopolistic broadcast methods such as radio or plain old Hertz TV, we had Hertz TV back home, not a lot of cable TV back then, I hear cable TV was a much bigger thing in the United States, and so it's moving away from that to online content basically.

Phil Gervasi : Which makes sense. That's been going on for the past 10, 15 years, like you said. But you're not casting that as a negative thing, right? Because honestly from a consumer perspective, like me personally, not from a business perspective or anything like that, it makes sense. I'm consuming the content that I want, I'm getting it over high speed internet, I feel like it's a good thing for the consumer. Do you agree?

Greg Villain: For the consumer, no doubt. The non- linearity, if this is even a word of it, it is the product people were waiting for, instead of needing to be in front of your TV, and at that specific hour to watch that specific program and not having choice based on your mood, you had a limited number of channels. Like back home when I was a kid we had six channels and we had one or two additional cable providers that were very expensive, but the consumption methods demanded to change, the users wanted something else, and something else appeared. Now, people will make a case that it is starting to become just a multiple amount of streaming providers that are basically the early broadcasters that we had back then, this may or may not be true, but even the methods of producing content have changed accordingly, and the formats have changed, so it's much more of a global change than just transport over the internet.

Phil Gervasi : Absolutely. And my overall spend, my bill, or total of bills for my streaming services pretty much equates to what would've been paying 15 years ago for cable. So I really feel like it's just a better version of that for the most part.

Greg Villain: This is where it's slightly different in the US than in some other countries in Europe that I'll take as an example because cable TV was not a huge thing. As I said before, I come from France and so you'd have Canal +, and not to name brands, but a few ones that you'd pay for, but most people would use free public TV or private TV, but using the public broadcast channels, and so you wouldn't have any spent per se. And so that exploding number of steering providers is perceived differently in Europe than it is in the United States. In the United States, in the end you're eventually paying the same amount that you used to pay on a regular cable plan, but you're just paying it over multiple services.

Phil Gervasi : Pretty much. And I didn't grow up with cable because my parents just didn't want to pay for it, but all my friends did. And I remember that if the president was doing a speech that night, then forget it, my four channels were all consumed with the speech and we were all bummed out because I wasn't able to watch Star Trek reruns, which is what I like to do at night. But what if you add up all of the bandwidth for each of those broadcasts and high def with high def audio, high def video, it's not a tremendous amount of bandwidth, but if you add that up in a particular region, a geographic region, that's a tremendous amount of bandwidth. So really where I want to go with this conversation now is what's been happening to the service providers, what's been its effect on service providers that are trying to deliver that same TV, movie, video content, but using a different medium?

Greg Villain: Here's how it used to work, Hertz and air broadcast had this bandwidth that was shared amongst the limited number of channels, and everybody was fine with that, then came IPTV, but IPTV you'd basically get the broadcast of the linear into your ISP network, and the ISP would encode it and then deliver it using multicast. Nobody remembers that technology. I could claim to be obsolete to the point that I know how multicast works, but then multicast it through their networks, so it'd be somewhat efficient because it builds a tree and it tries to not distribute the same thing and times. We're both on Comcast, say, if my neighbor watches the same show as I am at the same time, it's being sent twice, and so HTTP as a transport for TV or any video content is ubiquitous, that's for sure. Now the smallest devices, probably, your razor uses it, your fridge uses it, so it's there everywhere to consume, but we've got that additional complexity where I'll be watching something on Netflix, my wife will be watching something else on Hulu, and my son will be at the same time streaming something else on YouTube on his iPad, and so three people in the house will consume bandwidth from ISP, which wasn't the case before. And so I remember back in a day, because in... What was it? Around 2000 is when in France local loop unbundling showed up, and basically what it meant was that the incumbent, France Télécom by then, would be legally obligated to rent the copper pair to other telcos so that they could deliver services using DSL over that copper pair. And so when we started there, bandwidth was not the same orders of magnitude than there are now. It's the early days of DSL, so it's like anything was sub- two meg, and basically the capacity planning we made to build these networks was based on statistical consumption. I clearly remember early on in the days of early DSL in, I'd say, 2004 or something, the assumptions we made was that a user would consume in average 150K per second. And so to give you an idea, now you stream HD at above five megs per second, multiply that by the number of people in the house. And bear in mind, infrastructure investments, they can't happen overnight, because this is significant and you need to deploy stuff. There's actual gear that you need to put in the local loop, and depending on what your geographic footprint is, it can be an insane amount of investment. And so on the one end you've got the rise of OTT and everyone pushing their own content through a parallel channel to the others. And the fact that... Here's the thing, and I always boil it down to that, what type of internet access do you have currently at home?

Phil Gervasi : I get fiber to the house.

Greg Villain: So you get fiber to the house. So basically in most countries in the world now you get one gig for 50 bucks, right?

Phil Gervasi : Yeah.

Greg Villain: So one gig for 50 bucks is five cents a megabit per second, right?

Phil Gervasi : Mm- hmm.

Greg Villain: And so the price that your internet service provider pays to access the internet is probably above that, and so it is a business of very, very thin margins, and back in a day you had to keep a customer for at least five years to make it worth it.

Phil Gervasi : Oh, really? Okay.

Greg Villain: And so they're stuck between the increasing consumption and the competition. In the US you've got some kind of geographical local monopolies like in San Francisco I will only have a choice between two or three providers, but in some countries they've made it so easy to switch from one to another, local loop unbundling being one of the vectors of it, that people can switch whenever they want. So you have to make sure that they stay. And that's the other problem, right?

Phil Gervasi : Yeah. I have three main providers in my area, plus two or three smaller regional, which piggyback off the main ones anyway, and I get constant email and phone calls asking me to switch and giving me the introductory rates and stuff. So it sounds like one of the impacts that this change in how we consume media affects service providers is that they had to change their business model. How can we now deliver all this media? This is what customers want, this is what people want, how can we deliver that and still turn a profit considering that, like you said, the margins are razor thin? The only thing question I have here though is the reality is, I'm trying to do the math in my head, it really isn't that much bandwidth to my house. If I have five people all streaming at the same time, which doesn't really happen. If we're all streaming a movie, as a dad I'm going to be upset, we all need to be sitting in the family room together, but even then, what is that? 25 mags? That's not a huge amount of bandwidth.

Greg Villain: That's for what you're looking at, but now a lot of providers will pre- cash software. You probably have your iPhone set to auto update, which would be the safe thing to do nowadays, and so it'll get an update, and then your Mac's locally, if you have Mac's, will do that too, and your Windows setups will pre- fetch stuff, and your Xbox will start downloading game updates offline. And so it's funny because working on that OTT detection product that I work on, I can take, I have to look at my internet traffic closer than I used to, and the amount of stuff that's constantly pulling from the internet is insane, and I don't always remember it. There's a sunken part of the iceberg here.

Phil Gervasi : So is providing residential consumer grade internet almost like a lost leader for service providers today?

Greg Villain: I would think it is. I'm not a macro economy expert, but it seems to be a business of very thin margins, because I mentioned the price of bandwidth, but there's the price to operate such a network, and that price I would think increases with the footprint that you'd have and the number of metros that you serve internet access in, so it makes these matters worse. But to give you a scale, an internet service provider such as Comcast has a network pretty much as big as any transit tier one provider in the United States, to the extent that they've leveraged their eyeballs to become a seller of bandwidth, because they have built such a huge network, they need to profit from. It makes a lot of sense.

Phil Gervasi : And so how does that compare to business customers and the kind of legacy connections that are still very, very expensive, getting a high bandwidth MPLS circuit to my office building, for example, things like that?

Greg Villain: And this is where things start to getting interesting. So I dabbled a little bit in the enterprise business early on. In the first company that I worked at, I did a lot of pre- sales engineering on DSL specific and NPLS VPNs, and you could see back then that building NPLS VPN, you would price the internet access part of it a whole lot higher than residential, to the extent that people considered, well, I'm just going to buy routers and do it myself instead of renting a service from a service provider. And so this is how large companies ended up having... They were basically two paths. One would contract everyone for everything, and that's a very French and European way, to go through a company to contract a huge NPL and VPS. But some smaller companies that were a little bit crafty would buy their own gear and make their own tunnels themselves, because internet access, prices kept decreasing back then, and you could use that commodity as a building block for a VPN.

Phil Gervasi : And that's something that I'm seeing on the residential side, but as a network engineer having worked with large enterprises, bandwidth has just become incredibly cheap, but it's not just cheap, the quality of the connections are better than they have been. The fact that I don't really need a private NPLS circuit like I used to for the quality sake, because the internet... I make Zoom calls all the time and it's just HTTPS, and it's over the public internet from my house or from a business, and it's just fine, I can see everybody. We had the technology and the error correction built in to accommodate any kind of latency, well, minimal latency, that sort of thing, so I feel like the changes that are occurring in service providers aren't necessarily as technical. So they're not going to be using multicast as much, but more in how do I strategize around this? How do we charge differently? How do we change our backbone infrastructure to accommodate? So the advent of CDNs, I think you started to allude to that, how do you build an over- the- top, OTT, network? And then what is it called when you have those locations like an Amazon or a Netflix would have in the local, regional COs they would have their stack of content servers?

Greg Villain: I don't know precisely how this works in the US, but COs would be highly regulated because belonging to the incumbent, and usually in the COs you'll have all the telcos that want to have access to the local loop that used to be operated by the incumbent or by the local incumbent. So quick flashback in history, CDNs were born with Yahoo. Akamai was born with the Yahoo, to give you an example, the very first CDN, and they built it for Yahoo, and they were like, " We can probably make a business out of it." So basically back in the day it was all squids, squid proxies all over the place, and this is when the advent of global content happened. Usually you'd have local content, and it didn't really matter from where it was distributed because eventually it was meant to be consumed locally, but with, I figure this would be around 2005 or some, the rise of YouTube was the game changer here. You had to deliver content from all over the place, your audience was from the United States to Seoul, Korea, and so you couldn't just rely on connectivity from your local data center to distribute it. Online video is tolerant, as long as you have enough bandwidth, it can have long latencies, it doesn't really matter, but still you'd have to somehow deliver it locally. And so this is when they started to build pops in near ISPs network, so usually in data centers whereas there's a lot of other ISPs to connect with, and they would directly connect with them. So they'd have their servers in that data center and they would interconnect directly with the local ISPs or through an internet exchange peering fabric. And then the next step was we can reach further towards the last mile and help these ISPs, because, think of it, if I'm a content provider and I'm blasting content throughout the place, I need to help the eyeballs distribute my content, because if I don't, their users are going to get bad quality and they're just not going to use my service, and so I need to meet them further in the middle of the road. And so the idea came up, and Akamai pretty much started that, of embedding caches within the ISPs network. So they'd put caches in your ISPs network, and Akamai, for instance, is a commercial CDM, so they have multiple content customers that are located on their cache on the edge, and it would've distributed edge between their own facilities and the facilities of the ISPs that would agree to have these caches. And then large content providers thought, well, you know what? I can probably be Akamai for myself. And this is what we did over at Netflix back in the day, this is what I did at Dailymotion initially, and this is what I did at Netflix back in the day when we started helping out these eyeball ISPs, broadband providers, save bandwidth by literally offering servers to put in their own networks. And while we still paid for the servers, it was still very much worth it for us to do that, and for them because the quality would be better and they would save a lot of cost of upstream.

Phil Gervasi : Oh, sure. And from a service provider's perspective, whether it's regional or very large, national or international, as far as consumption of your own resources, you're talking about a unicast or a small subset of multicast to a region to those pops, and then from there those can multicast or unicast whatever out to the individual homes and businesses, things like that. So it does change the nature of how we're going to route traffic and how bandwidth is consumed on the service providers network itself. So where do you think we're going next as far as service providers, their relationship with their customers and the way the landscape looks?

Greg Villain: And every time I've made...

Phil Gervasi : Good question, isn't it?

Greg Villain: Oh yeah, it definitely is because I don't think that I have a valid answer for that. First off, broadband internet service provider, streamlining the way they produce a lot of these internet accesses, there's always been some amount of vendor lock- in, much less in a broadband industry you could switch pieces of your backbone to other providers from one contract to another and so on and so forth. But I think I've seen it happen with providers of large scale that are slowly moving towards automating their production mechanisms and changing, switching towards programmable routing that uses white box gear out there, and I know as a gearhead it might be a topic of interest to you, I've seen the rise of companies where back in the day if you'd come up with an engineering plan that would not include either Cisco or Juniper, there were very few ones back in the day, people would frown at you. But now it's I think I recently saw that AT& T is redoing their entire backbone using one of these recent software defined network services. What's their name? DriveNets, I think.

Phil Gervasi : DriveNets, okay.

Greg Villain: And I've heard them hiring like there's no tomorrow. So I think there's a massive evolution of the service provider industry towards programmable networks. And I think that one is a very interesting one because it's going to fundamentally change the shape of network engineering in the years to come.

Phil Gervasi : In my experience in the enterprise space, it is much more heavily vendor lock in because of whether it's politics, really good salespeople taking out the CIO to a state dinner, whatever it happens to be, you have those shops that are only Cisco, only Juniper, only a risk in the data center, whatever, or anything but Cisco, whatever it happens to be. I haven't seen that as much in the service provider realm. I have in small regionals. There's one in my area that is 99% Cisco. But generally speaking I don't see it as much. I see more of an openness to not necessarily white box, just non- major vendors, but sometimes white box. And I know especially when you get into really large web scale or global enterprise that's almost like service provider like, you want some of that disaggregation between your software and your hardware, so you can have a programmable interface. Not because it's cool and hipster networking, the cool thing to do, but because you have no choice, you're not making any money unless you find ways to reduce your opex. And that's one way to do it. And I have to imagine we're going to only see more and more and more of that in the service provider space when margins are so thin, and demand is even greater. For example, when I was working for VARS, I was selling SD- WAN all the time. That was the hot ticket. And people were not always getting rid of their NPLS, because either it was cheap and so they kept it or they were locked into a contract. But that was the idea that we're just going to use the public internet because it's cheap and it's good. And so I think that there is a difference between enterprise and service provider networking in that sense where the enterprise, I don't think it grabbed onto programmability and network automation as much as we thought it was five or seven years ago. The whole industry was talking about it. And I don't think they did quite as much as we thought it would. Obviously there's exceptions, large organizations that have huge... I get it. But your local company with a thousand employees, I don't think it really happened, but at the service provider level for sure. Do you think that we're just going to see a greater acceptance of white box, white box switching and routing, open networking, that sort of thing as margins get more and more thin, yet the demand grows more and more?

Greg Villain: Definitely. And I even see that it's a scale thing, I believe. Past a certain scale it makes sense for you to invest into that. And SPs have learned it from content providers that have become... Because of their scale of their user base, think Facebook with their wedge routers, Facebook runs on an entirely wide box, a gear. I remember already in 2010 or something, Google was already relying on they would buy Junipers just for the gear and they would run their own stuff with NPLS down to the server. And so interestingly enough, the service providers are absorbing these trends from the very web scale companies out there and in turn it gets accepted, it gets in people's brains as CIOs start to get acceptant of it. Think about how Amazon made its way into the larger companies out there, it really was because the CEO kept hearing about, oh, they do this thing very agile with that cloud provider and we've got our servers and it takes forever. The act of repeating it over and over makes its way to the enterprise, and so there's... How can you say that? Staggered delay in acceptance, but it won't be long until enterprises adopt it, the same way they adopted using Word processing and spreadsheet in the cloud. Back when I started working, it was unthinkable, first off from a technical standpoint, and then the next thing you know, nobody wants to hear about software licenses anymore, it's all about SaaS. So it's inevitable, I think. And to some extent I've been involved in teaching in university back home and that's the one thing I wanted to teach network engineers is that don't get focused around what a certain vendor can do because in the end it's the best documented industry ever, so we're very lucky about that. The protocols are explicitly documented everywhere. It'll come a time where these functions will be software, and that was probably the one prediction that I was right about. And so you have to, A, become a software engineer, because being a zealot for this provider or this provider of gear is not going to help you all your career and you're going to need to run networks, and running network is not logging on a router every day. And so be a software engineer, as much as you're a network engineer, the protocols are clearly explained. If you care to read about them, you'll understand what they do and you'll use them, but this is, I think, what's going to change in the industry is the type of profiles that are going to be required to run networks.

Phil Gervasi : There's a lot to say about the future of network engineering, which is a different podcast for sure. Very interesting topic too. What does it take to be a network engineer going into 2023 into 2024? That's really neat. Maybe we'll have you on again and talk about that, and focus on that. But right now we are at about time, so I would like to say thank you, Greg, for coming on and sharing your experience, your expertise. You have a lot of experience. I really value that.

Greg Villain: Thank you.

Phil Gervasi : And I got to say, it really is conversations like these that I love and I miss because I'm not in the field every day installing networks like I was years ago in the data center, so I do miss it. But anyway, it's been great, Greg. And if someone has a comment or question for you specifically, you directly, how can they find you online?

Greg Villain: I don't know if people do that, but they can just reach out to me at Greg at Kentik. They can also find me on LinkedIn, Greg Villain. I'm on Twitter, but I spend most of my time ranting on Twitter, so it might not be the best to follow me there, but that's about it. And again, if you dwell conferences such as ... or RIPE meetings, or even the general meeting for the French- IX Internet Exchange, you may see me there, so just grab me and have a chat. I'd love to.

Phil Gervasi : And you can find me on Twitter, @ network_phil. Search my name in LinkedIn, my blog network phil. com. You can also follow Telemetry Now on LinkedIn and Twitter. And if you are interested in being a guest on Telemetry Now, or if you have an idea for an episode, we'd love to hear from you. You can always email us at telemetrynow @kentik. com. So until next time, bye- bye.