Service Provider Cord Cutting with Greg Villain

My name is Phil Giovanni, and you're listening to telemetry now. And, with me today is someone who I'd say has the, I'm gonna say second best taste in music, pretty much all time. Greg Villan is the senior director of technical product management at Kentech. And today, we're gonna talking about cord cutting and the impact that it's had on service provider networks. So let's get started.

Okay.

So Greg is an Burton service provider networking.

He has worked on, with web scale companies, companies that you'd recognize. He's the president of France I X, The list goes on and on. And if you look at his LinkedIn profile, like I did earlier today, it's gonna make you wanna go study and just become a better engineer. So, Greg, It really is great to have you on the program today. Good to talk to you.

Thanks for having me here, Phil. And, also, you mentioned previously that I was the your your second best music choice person. Like, I now wanna know who the first one is. I gotta I gotta unseat him.

Yeah. Yeah. You're curious about that. So you and I are similar in age. We grew up with a lot of the same bands I know from having hung out with you. And, so we have that same foundation.

So that's why you're right there alongside me as second best, but what what I saw was that you sort of tended toward the the hardcore punk scene from the grunge days. Right? And then I sort of went in the direction of like progressive rock progressive metal, but we did have that same foundation So that's why I still count you a second, a close second, but it is second mast.

We'll battle this offline. Don't worry.

Yep. Battle the bands. So, before we get eyeballed even to this conversation, I do want to ask you, what do you mean by cord cutting? I mean, I think I know what it means, but I wanna hear it from you. And how long has that really been happening in the industry?

Well, so as far back as I can remember, in the early, I'd say, two thousand we started seeing, like, internet service providers going beyond the duty of just offering internet access. Right? We started seeing what you'd call back then triple play offers, which would be a bundle of internet access, voice over IP, and and also IPTV. And so back then, it was just using internet as a conduit to relay IPTV on a set top box.

But it was really what gave, what we now call OTT over the top content provider the idea that independent content from linear programming, could emerge and, and, and be brought to end users this way. So that's cord cutting, it it starts with voice over IP, really, and it it is what it is now with user generated video content, streaming services that have been multiplying over the recent years and so on. And gaming also, if you'd like. So, it's a it's a broad topic, but it's basically users progressively going away from, I'd call monopolistic, broadcast methods such as radio or or or plain old Hertz TV.

We had Hertz TV back home. Not a lot of cable TV back then, here cable TV was a much bigger thing in the United States. And so it's moving away from that to online content, basically.

Yeah. Yeah. Which makes sense. I mean, that's what's been going on for the past ten, fifteen years, like you said.

But you're not casting that as a negative thing. Right? Because, I mean, honestly, from a from a consumer perspective, like me personally, not from a business perspective or anything like that. I mean, it makes sense.

I mean, I I'm consuming the content that I want. I'm getting it over high speed internet. I I mean, I feel like it's a a good thing for the consumer.

I mean, the Oh, for for the consumer.

No doubt. I mean, the non linearity, if this is even a word of it, what people I mean, it's the product people we're waiting for. Right? Instead of needing to be in front of your TV, and at that specific hour towards that specific program and not having trolleys based on your mood.

You had a limited number of channels. Like back home when I was a kid, we had six channels. And we had like one or two additional cable providers that were very expensive, but, the consumption methods demanded to change. The users wanted something else and something else appeared.

Now, people will make a case that it is starting to become just a multiple amount of streaming providers that are basically the early broadcasters that we had back then This may or may not be true, but the even the methods of producing content have changed accordingly, and the formats have changed. So it's it's a it's much more of a global change than just like transport over the internet.

Yeah. Absolutely. And my overall, spend my bill or, you know, total of bills for, my streaming services, pretty much equates to what I would have been paying fifteen years ago for cable.

So I really feel like it's just a better version of that for the This is where it's slightly different in the US than in some other countries in Europe that I'll take as an example because Cable TV was not a huge thing.

They were like, as I said before, I come from France. And so, you'd have canal plus and and not to name brands, but a few ones that you'd pay for, but most people would use free public TV or private TV, but using the public broadcast channels.

And so you wouldn't have any spent per se. Right? And so that the that kind of like exploding number of steering providers is a diff is perceived differently in your advantage in the United States. In the United States, in the end, you're Eventually paying the same amount that you used to pay on a on a regular cable plan, but you're just paying it over multiple services.

Yeah. Pretty much. And, I mean, I I didn't grow up with cable because my parents just didn't wanna pay for it. But all my friends did, and I, you know, I remember that if the president was doing a speech that night, then forget it.

You know, my four channels were all consumed with the speech, and we were all bummed out because I wasn't able to watch Star Trek reruns, which is what I to do at night. But but what if you add up all of the the bandwidth for each of those broadcasts in high def with high def audio, high def video. I mean, it's not a tremendous amount of bandwidth, but if you add that up in a particular region, a geographic region. That's a tremendous amount of bandwidth.

So really where I wanna go with this conversation now is what's been happening to the service providers? What's been its effect on service providers? That are trying to deliver that same TV movie video content, but in a different using a different medium.

Here's how it used to work. Hertz and Air broadcast had this bandwidth that was shared amongst the limited number of channels and everybody was fine with that, then came IPTV.

But IPTV, you'd basically get the broadcast of the linear into your ISP network and the ISP would encode it and then deliver it using multicast.

Yeah. Yeah. Nobody remembers that technology.

I I I could claim to be obsolete to the point that I know how multicast works. Yeah. But it then multicast it through their network So it'd be somewhat efficient because, I mean, it builds a tree and it tries to not distribute the same thing and times. Right now, if my neighbor watches, we're both on Comcast say, if my name of watch is the same show as I am at the same time, it's being sent twice.

Right? And so It's a ubiquitous HTTP as a as a transport for TV or or any video content is ubiquitous. That's the show. It it now the smallest devices you're probably your razor uses it.

Your fridge uses it. So it's it's there everywhere to consume, but we've got that additional complexity where I'll be watching something on on Netflix. My wife will be watching something else on Hulu and my son will be at the same time streaming something else on YouTube on his iPad. And so three people in the house will consume bandwidth from or ISP, which wasn't the case before.

And so I remember back in the day because, in what was it? Around two thousand is when in France, local loop unbundling showed up. And basically what it meant was that the incumbent, French telecom by then, would be, legally obligated to rent the copper pair to other telcos so that they could deliver services using DSL over that copper pair. Right?

And so when we started there, Benwidth was not the same orders of magnitude than than they are now. Right? It's, the early days of DSL, So it's like anything was sub two meg. And and basically the capacity planning we make to build these networks was based on statistical consumption. I clearly remember early on in the days of, like, early DSL in, like, I'd say, two thousand and four or something.

The assumptions we made that was that a user would consume in average one hundred and fifty k per second.

And so to give you an idea, now you stream HD at above five megs per second, multiply that by the number of people in the house. And so, and and bear in mind, infrastructure investments, they can't happen overnight because this is significant. And you need to deploy stuff. There's actual gear that you need to put in the local loop and and depending on what your geographic footprint is.

It's it can be an insane amount of investment. And so On the one end, you've got the rise of OTT and everyone pushing their own content through a parallel channel to the others. Right? And and the fact that Here's the thing, and I always boil it down to that.

What type of internet access do you have currently at home?

I get fiber to the house.

So you get fiber to the house. So basically, in in most countries in the world now, you get one gig for like fifty bucks. Right? Yeah.

So one gig for fifty bucks is five cents a megabit per second. Right? Mhmm. And so the price that your internet service provider pays to access the internet It's probably above that.

And so it is a business of very, very thin margins. And back in a day, you had to keep a customer for at least five years to make it worth it.

Oh, really? Okay.

And so they're stuck between the increasing consumption and the the competition.

In the US, you've got some kind of geographical local monopolies, like Yeah. In San Francisco, I will only have a choice between, like, two or three providers, but in some countries, they've made it so easy to switch from one to another.

Local loop unbundling, being one of the, vectors of it, that people can switch whenever they want. So you have to make sure that they stay. And that's the other problem. Right?

Yeah. I mean, I have three main providers in my area plus, two or three smaller regional, which piggyback off the main ones anyway.

And I get constant, email and phone calls, you know, having, you know, asking me to switch and giving me the introductory rates and stuff. So it sounds like one of the impact that this change and how we consume media affects service providers is that they had to change their business model. How can we can we now deliver all this media? This is what, you know, customers want. This is what people want. How can we deliver that and still turn a profit considering that like you said, the margins are razor thin.

Not to I mean, the the only thing the only question I have here, though, is The reality is I'm I'm trying to do the math in my head. It really isn't that much bandwidth to my house. I mean, if I have five people all streaming at the same time, which doesn't really happen. I mean, if we're all streaming a movie, as a dad, I'm gonna be upset.

Like, we all need to be sitting in the family room together. But but even then, what is that? Twenty five megs? That's I mean, that's not a huge amount.

Well, that that's for what you're looking at, but, like, now a lot of providers will pre cash software. Right? Mhmm. You probably have your iPhone set to auto update, which would be the safe thing to do nowadays.

That's cool. And so it'll get an update. And then your max locally, if you have max will do that too, and your Windows setups, little prefetch stuff. So and and your Xbox, we'll start downloading game updates offline.

And so It's funny because working on the on that OTT detection product that I that I that I work on at Kantec, like, I have to look at my Internet traffic like closer than I than I did used to. And the amount of stuff that's constantly pulling from the internet is insane, and and I don't always remember it. Right? So it there's there's a sunken part of the iceberg here.

So is providing residential consumer grade internet almost like a lost leader for service providers today?

I would think it is. I don't I'm I'm not a a macro economy expert. Right? But, it seems to be a business of very thin margins.

Because, I mean, I mentioned the price of bandwidth, but there's the price to operate such a network. And that price, I would think increases with the footprint that you'd have and the number of metros that you serve internet access in. So it makes it makes these matters worse. But to to give you a scale, like, an internet service provider such as Comcast has a network pretty much as big as any transit tier one provider in the United States. To the extent that they've leveraged their eyeballs to become a seller of bandwidth because they have built such a huge network. They need to profit from it. It makes a lot of sense.

And so how does that compare to business customers and the kind of legacy connections that are still very, very, expensive, you know, getting a high bandwidth MPLS circuit to my office building, for example, things like that.

And this is where things, start getting interesting. So I dabble a little bit in the enterprise business early on in the first company that I worked at, I did a lot of, presales engineering on on DSL specific and NPLs VPNs.

Yeah.

And and you could see b back then that building an NPLs VPN, you would price Ener the internet access part of it a whole lot higher than residential, to the extent that people considered well, I'm just gonna buy routers and do it myself instead of renting a service from a service provider. Right? And so this is how large companies ended up having They were like basically two paths. One would contract everyone for everything and that's a very French and European way to do to go through a company to contract a huge MPL and VPS But some smaller companies that were a little bit crafty would would buy their own gear and make their own tunnels themselves because internet access prices kept decreasing back then. Yeah. And so, you could use that commodity as a building block for for a VPN.

Yeah. And that's something that I mean, I'm seeing, yeah, on the residential side, but, you know, as a network engineer having worked with large enterprises, bandwidth has just become incredibly cheap. But it's not just cheap. The quality of the connections are better than they have been.

You know, I mean, the the fact that I don't I don't really need a private MPLS circuit like I used to, you know, for the quality sake because the internet I mean, I I make Zoom calls all the time, and it's just HTTPS and it's over the public internet from my house or from a business, and it's just fine. Like, I can see everybody and we had a technology and the error correction built in to to accommodate any kind of latency. Well, you know, minimal latency, that sort of thing. So I feel like The changes that are occurring in service providers aren't necessarily as technical.

Okay. So they're not gonna be using multicast as much.

But more in like how do I strategize around this? How do we charge differently?

How do we change our backbone infrastructure to accommodate? So, you know, the advent of CIDR, I think you started to allude that. Right? How you how do you build an over the top OTT network? And, and then what what is it called when you have those locations? Like, you know, like an Amazon or a Netflix would have in the local regional CEOs, they would have, you know, their stack of content servers?

I I I don't know precisely how this works in the US, but CEOs would be highly regulated because belonging to the incumbent. And usually, This in the CEOs, you'll have all the telcos that want to have access to the local loop that used to be operated by the incumbent or by the local incumbent. So quick flashback in history. CDNs were born with Yahoo.

Akamai was born with Yahoo to give you an example. The ver the verifies CIDR they built it for Yahoo and they were like, oh, we can probably make a business out of it. So basically, back in the day, it was all squids, squid proxies all over the place.

And and this is when the advent of global content happened. Usually, you'd have local content, and it didn't really matter from where it was distributed because eventually it was meant to be consumed locally. Right? But with the rise of I figured this would be around two thousand and five or some the rise of YouTube was the game changer here. You had to deliver content from all over the place. Right?

Yeah.

Your your audience was from the United States to Seoul Korea. And so you couldn't just rely on connectivity from your local data center to distribute it. I mean online video is tolerant. It's a it's a fairly it it it can have as long as you have enough bandwidth, it can have long, latencies. It doesn't really matter.

But still you'd have to somehow somehow deliver it locally. And so this is when they started to build pops in ISP, near ISP's network. So usually in, data centers where there's a lot of other, ISPs to connect with. Right?

And they would directly connect with them so they'd have their servers in that data center and they would interconnect directly with the local ISPs or through an Internet exchange, peering fabric. And then the next step was we can reach further towards the last mile and help these ISVs because Think of it. If I'm a content provider and I'm blasting content throughout the place, right? I need to help the eyeballs distributed my content because if I don't, their users are gonna get bad quality and they're just not gonna use my service.

And so I need to meet them further, in the middle of the road. And so the idea came up. And how come I pretty much started that of embedding caches within the ISP's network. Yeah.

So they put caches in your ISP's network and Akema, for instance, is a is a commercial CIDR. So they have multiple content customers, that are located on their cash, on the edge. And it would have distributed edge between their own facilities and the facilities of the ISPs that would agree to have these caches. And then like, large content providers thought, well, you know what?

I can probably be Acamai for myself, and this is what we did over at Netflix back in the day. This is what I did at daily motion initially, and this is what I did at at Netflix back in the day when we started helping out these IS, Ebola ISPs Broadband providers save bandwidth by offering them literally offering servers to put in their, in their own networks. And while we still paid for the servers, It was still very much worth it for us to do that. And for them, because the quality would be better, and it would save a lot of cost of upstream.

Oh, sure. And and from a service provider's perspective, whether it's regional or or very large national or international, I mean, you're talking about as far as consumption of your own resources, you're talking about a unicast or a small subset of multicast to a region to those pops And then from there, those can multicast or unicast whatever out to the, individual homes and businesses, things like that. So it does change the nature of how we're gonna route traffic and how bandwidth is consumed on the, the service service providers network itself. So where do you think we're going next? I mean, as far as service providers, their relationship with their customers and the way the landscape looks.

So that's and every time I've made, I've made like Good question, isn't it?

Oh, yeah. No. It it definitely is because, I don't think that I have a valid answer for that. First off, streamlining the way, internet service providers, broadband, internet service provider, streamlining the way to produce a lot of these, internet accesses.

There was there's always been some amount of vendor lock in, much less in a in a broadband industry. You would you could switch pieces of your backbone to other providers from one contract to another and so on and so forth. But I think I've seen it happen, with providers of large scale, that are slowly moving towards automating their production mechanisms and automating, changing switching towards programmable, programmable routing that uses kind of like white box gear out there. And I know as a gear head, it might be a topic of interest to you.

Mhmm. Yep.

I've seen the rise of of companies where back in the day, If you'd come up with a biz with a with a an engineering, plan that would not include either Cisco or Juniper or There were very few ones back in the day.

People would kind of frown at you, but now it's like, I think I recently saw that AT and T is redoing their entire backbone using, one of these recent, software defined work services. What's their name? Drivenights, I think.

Drivingnights? Okay.

And I've I've heard like Oh, yeah.

Them hiring, like there's no tomorrow. So I think There's a massive evolution of the service provider industry towards programmable networks. I think that one is a very interesting one because It's gonna fundamentally change the shape of network engineering in the yes to com.

Yeah. I mean, in my experience in enter in the enterprise space, it is much more heavily vendor lock in because, whether it's politics, really good sales people taking out the CIDR to, you know, a state center. Whatever it happens to be, you have those shops that are only Cisco, only Juniper, only nearest in the data center, whatever, or anything but Cisco, whatever it happens to be. I haven't seen that as much in the service provider realm.

I have I have in small regionals, like, there's one in in my area that is, like, ninety nine percent Cisco. But generally speaking, I don't see it as much. I see more of an openness to not necessarily white box, just non major vendors. But sometimes white box.

And I know, especially when you get into really large web scale or global enterprise that's almost like service provider like.

You you want some of that, disaggregation between your software and your hardware. So you can have a programmable interface, not because it's cool and hipster networking, you know, the cool thing to do. But because you have no choice, you're not making any money unless you find ways to reduce your your opex. And that's one way to do it.

And I have to imagine we're gonna only see more and more and more of that in the service provider space when margins are so, so, so thin. And, and the and then and the demand is even greater. Like, for example, you know, I I when I was, working for Vars, I was selling, like, a the yuan all the time. That was the hot ticket, and people were not always getting rid of their MPLS because either it was cheap and so they kept it or they were locked into a contract, but that was the idea that we're just gonna use the public internet because it's cheap and it's good.

And so I think that there is a difference between enterprise and service provider networking in that sense where the enterprise, I don't think it grabbed on to programmability and network automation as much as we thought it was five or seven years ago. The whole industry was talking about it, and I don't I don't think they did quite as much as as we thought it would. Obviously, there's exceptions, large organizations that have huge info. I get it.

But your local, you know, your local company with a thousand employees it's just, I don't think it really happened, but at the service provider level for sure, do you think that we're just gonna see a greater acceptance of white box white box switching and routing, open networking, that sort of thing, as margins get more and more thin, yet the demand grows more and more.

Definitely. Like, and I and I even see that it's a scale thing, I believe, past a certain scale it makes sense for you to invest into that. And and and SPs have learned it from content providers that I've become because of their scale of their user base. Right?

Like, think Facebook with their wedge routers, like Facebook runs on an entirely, wide box. A gear. So does I remember already, like, in two thousand ten or something, Google was already relying on. They would buy juniper just for the gear, and they would run their own stuff with MPLS down to the server.

And so interestingly enough, the service providers are absorbing, these trends from the very, like, web scale companies out there And and in turn, kind of like it gets accepted. It gets in people's brains as CIOs start to get accepted of it. Think about how, Amazon made its way into the larger, companies out there. It it really was because CEO kept hearing about, oh, like, they do this thing very agile with, like, that cloud provider and we've got our servers and and it takes forever.

Like the act of repeating it over and over, makes its way to the to the enterprise. And so there's I could say that staggered delay in in, acceptance, but it won't be long till, enterprises adopted the same way they adopted using word processing and and spreadsheets in the cloud, like, back when I started working, like, it was unthinkable, first off, from a technical standpoint.

And then and then the next thing you know, like, Nobody wants to hear about software licenses anymore. Like, it's all about SaaS. So it's it's inevitable, I think.

And, and to some extent, I've been involved in in teaching, in university back home, and that's the one thing I wanted to teach network engineers is that don't get focused around what a certain vendor can do because in the end, It's the best document in industry ever. So you we're very lucky about that. Like, the protocols are like explicitly documented everywhere.

It'll come a time where these functions will be software, and you can, and you can, that was probably the one prediction that I was right about. And so you have to, a, become a software engineer because being, a Zillot for this provider or this provider of of gear is gonna help you all your career, and you're gonna need to run networks, and running network is not logging on a router every day. And so be a software engineer as much as you're a network engineer, the protocols are clearly explained. If you care to read about them, you'll under and what what they do and you'll use them. But this is this is I think what's gonna change in the industry is the ability, the the the type of profiles that are gonna be required to run networks. Yeah.

Yeah. There's, there there's a lot to say about the future of network engineering, which is a a different a different podcast for sure. Very interesting topic too. What does it take to be a network engineer going into twenty twenty three into twenty twenty four. You know, that's that's that's really neat. Maybe we'll have you on again and talk about that and, and focus on that. But right now, I, you know, we are we are at about time.

So I would like to say thank you, Greg, for coming on and sharing your experience.

Your expertise. You have a lot of experience. I really value that.

Thank you.

And I gotta say. It really is conversations like these that I that I love and, and I I kinda miss because I'm not in the field every day installing networks, you know, like, like I was years ago in the data center. So I do miss it. But anyway, it's been great, Greg, and, and if someone has a comment or question for you specifically, you directly. How can they find you online?

I don't know if it if if people do that, but it can just reach out to me, at Greg at Kentek. They can also find me on LinkedIn Greg Villan.

I'm on Twitter, but, I I spend most of my time ranting on Twitter, so it might not be the best to follow me there. But that's about it. And and again, like, if you dwell conferences such as Nanog or or Rype, meetings or even the general meeting, for the French Sykes internet exchange, you may see me there. So just grab me and and have a chat. I'd love it.

And you can find me on Twitter at network underscore fill, search my name in LinkedIn, my blog network fill dot com.

You can also follow telemetry now on LinkedIn Twitter. And, if you are interested in being a guest on telemetry now, or if you have an idea for an episode, we'd love to hear from you. You can always email us at telemetry now ending dot com. So until next time. Bye bye.