Kentik - Network Observability
More episodes
Network AF  |  Season 1 - Episode 12  |  March 1, 2022

Dave Schaeffer on leadership and operations at Cogent Communications

Play now

 

Dave Schaeffer, founder and CEO of Cogent Communications, discusses his path in networking and the innovations that have propelled their ISP to success. In this episode Avi and Dave touch on:

  • The differences between developing a product and a business
  • Focusing on the product and knowing when to pivot to new ideas
  • Raising investor money and the dynamics it creates
  • The decreasing costs of internet service and Cogent's role in creating affordable access
  • How COVID-19 has impacted culture and operations at Cogent
  • Hiring practices when looking for early career vs senior employees
  • Dave's advice for his younger self

Transcript

Hi. Welcome to network a f. Today, I'm here with my friend Dave Schaffer who's built of the leading carriers that connects us all in the digital world cogent. You may notice my mask is on because he is graciously hosting me here in beautiful downtown DC. Today, we're gonna talk about passing to technology, science and technology, culture, a lot of business topics around capital and growth and focus and execution.

And learn a little bit about how the company was made and where it's going, which as you'll see, is a lot of, like, where it's been.

Hello, and welcome to network a f. I'm here with my friend Dave Schaper, leader of Cogent in Beautiful Downtown DC. I'm also here for the World Science Fiction but thought we'd do this episode in person. And hi, Dave.

You could introduce yourself briefly. Yeah, hi. Thanks for joining me, Avi. Dave Shaffer, Coaches, Founder, CEO.

So I always ask people to start how did you get into networking? I know I'll go back a little bit maybe, Dave, and how did you get interested in in in technology?

Whoa. You know, it should be a short answer, but in this case, it's probably gonna be a little bit of a long answer. So I got involved, well, first of all, in technology, going to school as I was physics major, but, in business, I had The first business I ran was actually a taxicab company and related to the taxicab business was the need for mobile radios for Disback.

And at that time, two major changes were occurring.

One motorola which dominated that market went from being a manufactured direct distributed model to establishing dealers, which I became one of those, and then secondly, the frequency bands were also being allocated on a proprietary basis as opposed to a shared basis.

So, you could actually own them but you did not have to pay for them if you met certain learning requirements.

And as a result, I started developing a number of SMR systems or specialized mobile radio systems that eventually got rolled up to what became Nextel.

I also from that expanded into a second technology venture, which was paging. I was a paging carrier up and down the East Coast, including covering your market, Philadelphia, where you were based.

And this was in the early nineties.

Had the good fortune to sell that business while there was still a business to sell.

My third, technology business was based on a regulatory complexity created when the government decided to reallocate spectrum for personal communication services or PCS creating additional cellular carriers. So initially, There were only two licenses granted an A and B license.

The government decided to ultimately grant four additional licenses, but in doing so, when they granted those licenses, the winners and an auction of those licenses had the obligation to relocate the exist employees and Cs who were not communications companies for individuals who was using the frequency of the spectrum. That's correct. And I created a business that manage that process for those companies, create an excess capacity and aggregate and build a nationwide network, through that regulatory catalyst and, actually that company raised a significant amount of capital.

It actually had public debt, but, attempted to go public and then markets were not in our favor, eventually, was able to sell that business and realizing that the network that had both the lowest cost and the most ubiquity would eventually win I became convinced in the mid 90s that the internet would be the only network that mattered and by building a internet network, most companies overlaid either a telephone network a cable TV network or a mobile phone network, and it was possible by the late 90s to build a much more efficient network, capitalize in what two major, changes the marketplace.

First, the ability by dark fiber rather than build all of your network truck, which had traditionally been the way in which carriers built their network, taking, you know, three to four years, costing you know, three to five billion dollars and carrying huge amounts of excess inventory.

By the late nineties, you can go out and purchase dark fiber and with advances in wave division multiplexing, that fiber would have virtually unlimited capacity.

The second key technological change was the advances in enterprise land routing.

So traditionally carrier networks were built on Sonet STH, which had a number of expensive overhead protocols that were not relevant to the internet, and by taking land equipment and then deploying it on a global scale, you could drive down the cost of network construction.

So those two events gave us the ability to build a purpose built facilities based network.

And I think cogent, even to this day remains unique in that. So a long winded answer your flushing holiday, but, you know, how I got here was probably not the most conventional round.

Well, no. It's fascinating to me because when I talk sometimes to to, I guess, I'm old enough that sometimes I say kids, but people early in the career about different paths. And, you know, when we got started, it was all nerd, was all technology, and and before the great specialty. But today, It's like how often will you find a Linux kernel book versus how often will you find a, you know, routing book and the analogy I often use is physics.

Like, the people that I had physics with used to say, you know, starting from the basic laws of physics, you know, how fast would the moon go around, whatever. And and, you know, like, in in networking, there's enough bugs and complexity that sometimes you just need to get back to first principles and say what? Should be. And I think that's fascinating.

Some people like it, you know, more than others, but, it's interesting to see the physics background. Do you remember when you developed the ability to sort of look at something and say, this should exist, but doesn't like, as a as a as a business opportunity, like, you know, because it seems like, you know, the story you told all three of those, like, well, there's this thing and taxi and, but there's this optimization here or, you know, this is gonna be this coming problem with people on the spectrum that to move in, or as you said regulatory, you didn't say arbitrage, but, you know, complexity or opportunity.

When when did you, you know, do you think that's something that people are born with or is Is it is it, you know, family? Or Well, probably a little bit of both. I mean, my dad was a cab driver, and I'm very never had a real job, started seven companies, then no one job I've ever had, CEO, and in my mind, that's overhead all the people that really need to work. So I'm very conscious of, trying to add value when I can.

And you know, I think many entrepreneurs confused developing a product with developing a business, which needs to be self sustaining and able to grow.

And then I think the second kind of opportunity comes from looking at the market and rarely does a company invent something that no one has ever seen before. I know She, Steve Jobs said, I imagine because no one else has ever seen this before. Right. But, you know, whether it be you know, his, you know, I guess borrowing the concept of the mouse Right. And Jira's part to the graphic interface. There's a new book about that I wanna read. Yeah, on the first Newton.

You know, I think the reality is everybody builds off of existing technology. And to me, the opportunity comes from taking any business and being able to do it faster, cheaper, and better. If you could do those three things, typically, you know, you'll succeed.

And in running a business, there are really two fundamental questions that you need to ask yourself, one, do people need what I'm selling?

And then two, will They buy for me versus someone else.

Once you've answered those two questions, you've got the basis for a business.

It's actually the third one. That's maybe the hardest one for most companies, and, that is Can I create a profit, an economic profit? That doesn't mean GAAP, EPS profits it means, cannot Well, you said that grow and make money. And that's not just one or the other.

Right? That's correct. That's they will do both. You know, it's the old saying I can sell a lot of dollars if I sell them for fifty cents or years.

You know, so You know, it all starts with fundamental demand, your ability to meet that demand and differentiate yourself from others. And then the final question after you check those two boxes is, is this self sustaining? And it's, it's why So many venture backed companies never reach kind of escape velocity.

They typically get bought Mhmm. By other companies because you're really a product, or they're not self sustaining.

So when I look at the idea to build an internet provider, it seemed like the market was huge, the differentiation was substantial enough to give people a reason to buy from us, and then third the economics would eventually work as long as we could rescale.

Now, in building cogent, We also had to be disciplined about where we deployed the network.

So in any utility, in the internet as utility, just like any other, there's a natural monopoly that should only be one.

But the reality is for the internet, there's already three there. There's this overlay on top of phone network, overlay on top of the mobile network, and the more successful networks tended to be the overlays on top of cable.

So you had to say, was my network good enough that I could overcome the disadvantage of not being an incumbent, having an existing plan.

And in our case, the way we did that is being very thoughtful about the endpoints on the network, effectively cream skimming the market going after the highest traffic locations. The news really fall into two major categories.

We connect to just under one billion square feet of office space in North America where we've thrived sixty percent of our revenues and that is as an end user ISP, but the average building we serve is five hundred and fifty thousand feet or the equivalent of about twelve football fields. So you wouldn't serve your own building here.

We actually do because we're here. Right. But, yes, this would not be a coaching building if it was not for us. Physically occupied. Right.

And then, secondly, we serve about thirteen hundred and fifty carrier control data centers in fifty countries around the world that are rule of supermarkets for bandwidth. And the important reason for our success is we ignore the millions of other buildings that we could, sir, but we've concluded that we can't generate a high return on capital when we factor in our sales and marketing costs.

So we're very selective.

It's you you said a few different things in the last twenty minutes to unpack.

One of the things that I struggle with is, focus.

And one of the things that I've enjoyed working with cogent is seeing the focus because I often I maybe see too many opportunities out there and actually had some good friends before I started, can't take say maybe you should just pick one. If you wanna make an Akamai or a cogent size thing, maybe you should not try to do five things at once. But, you know, is that a struggle? Do you see these other opportunities and say, Well, these other carriers think that they need to get at a CapEx job and making managed services or salespeople come in and say, hey, but we sell this or we sell this. Is is there any temptation to to break the model that's worked so well?

No. No. I Oscar Wild said I could resist everything but checked.

Right. And you do need to be disciplined. And and it's it's It's one of the biggest challenges for an entrepreneur because you start with a blank sheet of paper. You can literally do anything you want. Capital is fungible, and most businesses change course multiple times in their evolution.

Actually, of the seven companies I've run coaching is actually the only one It stuck to its original business model and did not pivot, in mid course.

The problems are also exacerbated when you take venture capital because you then have a board that is very focused on the next new thing. So they're constantly looking for another shiny object and distracting the entrepreneur.

Now on the flip side, the entrepreneur needs to be able to stay focused, but when things aren't working, know when to pivot. Don't need to go reality. Yeah. I mean, plus If you could get the messages from God, it's like, you know, it's like the the person is like, why couldn't you help me? And he's like, you could buy a lottery ticket, you know.

You know, maybe it's too simple minded, but I take the approach that when you go out to raise capital, the people you talk to, we're in the business of investing at capital.

And if you talk to fifty people who really wanna put capital or work, you all say it's a dumb idea, It probably is a dumb idea.

You know, I understand entrepreneurs need to be determined. They need to be focused.

They need to have the tenacity to plow through no, and I agree with all of that. But once enough people tell you something doesn't work, You probably should try it perfect. Now to be fair, I remember the the telecom cousin days and hearing your pitch for the vision and you know, we'll come back and talk about it. It was crazy times when, you know, sometimes timing favors your plans too, with what was happening macro.

But I thought you were a little crazy, you know, at first. And so I say that with respect and pleasure at seeing what cogent has done. I think that I'm not gonna I won't defend at least some of the VCs, you know, in this this hour, but I think that there are some people that definitely get caught up And it's true that VCs keep seeing the new hotness, the new hotness, the new hotness, but if, you know, there's some folks that focus on the fundamentals. Now it is true that in the private market today, most people would rather have you ignore profitability for growth that's generally true in VC.

And right now, you know, even gross margin, as you said, selling the dollars for fifty cents. So it'll be interesting to see what happens. I'm glad for all my friends in the private and public markets who are taking advantage of this market, but it is a little crazy when I do the math trying to figure out how everything, came together. So when you decided to start cogent, you did raise some outside money.

Was that how much so how much and what year was that? Okay. So Most of my career, I bootstrapped my companies.

When I started my previous company, we needed the credibility. The Spector arbitrage? Yeah. The arbitrage. Right. So it's called Pat Nat.

When I started Pathnet, I bootstrapped it. It quickly realized in dealing with major companies fortune one hundred companies, that they would not do business. With a non institutionally funded business. So I brought in external capital and grew that business by using other people's money.

When I sold my position in that business, put in a management team, and then literally came in here with Kimila, our old office Right. Lots from here on a, Saturday afternoon.

The entire coaching business plan that afternoon, in August of ninety nine, and realized that we needed to raise capital.

Now, if you can remember back, the world was on high, much like it is today.

We were actually a unicorn before companies were called to utilize.

We raised ultimately about five hundred million dollars in our seed round.

We're about there now. We're about there now. Okay. In PowerPoint, And at the time, cogent was myself, and I was paying two twenty four year olds out of my pocket.

One was an ex banker for Morgan Stanley. The other woman was Mhmm. From Anderson Consulting. Alex Sanchez, who both wanted to work in a startup before going to business.

School.

And, we had told our investors that to build the network that we contemplated, it would take about two billion dollars that five hundred million dollars was the down payment.

Sometimes it's better to be lucky that's smart.

We've raised the money. We started to hire a team. It couldn't spend the money fast enough.

The telecom market then imploded.

It was a worse depression in technology than the great recession was in the financial the bottom line hadn't raised a billion dollars of debt. It would have gone. It wouldn't have survived.

And, that that's true of virtually every. We got lucky. Again, we got lucky. You know, it was already Amazon reached six hundred million dollars in the convert, right, before the market blew up and wouldn't have survived within that. Yeah.

And we had, effectively, a pool of capital, a good business plan and a very small team.

We sat down with our investor base, and we literally laid out three past four. One was liquidation.

Dividing down the cash that we had raised minus what we had already spent, which was for a small fraction of the money. And when we looked on temperature, percent.

The second option would have been for the current investor based pledge incremental capital because there was no external sources available.

And then the third path, which is what we debated, and ultimately chose was to use our capital to buy broken businesses.

So a common mistake that many outsiders think of when they think of coaches. They think we were founded to buy up distress assets. That's actually not true.

We were founded to build an all IP over DWDM network, protected a layer three, using ethernet as an interface, serving the highest traffic location. So you trusted that price would be the disruptive factor that would ensure, as you said, scale. Just to do that work, you need two billion dollars, but it only works and it's profitable with growth but only at scale. That's correct.

How do you get this scale? Telecom is absolutely a business of scale. So in many industries, there's the concept of a backward bending supply curve. So there are discontinuities where you have effectively diseconomies of scale.

Marginal cost is always below average cost in telecom. So you had to get big facts. Yes.

And we thought we had the right addressable markets.

And in serving our customers, there were kind of two different, ways to show value.

For our corporate customers, it was not price. It was quality of service for price parody.

So we went to the market offering in two thousand, a hundred megabit non blocked and non oversubscribed DIA product on a month to month contract. Do we have to get access for those? To that customer base for a thousand dollars.

When AT1 was two thousand five hundred. So you paid less, you got more throughput, It was ring protected.

The building was pre wired, so it typically installed in one ninth the time, it was three times more reliable, and you got somewhere between thirty and sixty times the throughput for the same time.

Bills delivered on UPS trucks that you can't make out. None of that. It's just literally plugging in with an r j forty five.

To a fast Ethernet port.

And, that model worked well for our corporate customers, Our corporate business generates only a small fraction, a couple percent of our total traffic, even though it's sixty percent of revenues.

We have a second market segment that is totally different, which is selling bulk transit and data centers. Mhmm. In that market, there's only one dimension you compete on, and that is price.

We came to market at ten dollars a megabit when the average price for transit in a data center was three hundred dollars a megabit. And that had recently been higher. Yeah. It had been as high as five thousand dollars a megabit.

And we started at ten.

Today, you know, we sold sub-ten cents for many customers.

The prices are gonna keep coming down. Do you have a prediction? How much per year, per decade, for five years? So over the past, twenty years, for coach, the average price we sell at has fallen at about twenty three per percent per year. On the wholesale side. So this is measure services metered in a data center where you're buying by the megabit, prices are falling at twenty three percent a year.

The two technologies that allow that to occur are wave division multiplexing, and optically interface routing.

If you look at wave division multiplexing, That technology has improved since the mid eighties, when it was first deployed, actually, at the late last Whitter Olympics was the first deployment of a test, Kate. Mhmm. Commercial test of a four channel asynchronous wave division multiplex system, at roughly eighty percent per year compounded.

Now put that against Moore's law, which is only fifty five percent per year.

Price performance improvement.

And different than Moore's law was much further from the physical limitations.

So I think the transport side will continue to improve at this very rapid pace for decades to come.

The second technology is an optically interfaced router.

So prior to coaching, browsers had to be front ended. Mhmm. Usually with an ATM switch, so you had a Sonet Mox, you had an optical transport, terminal that talked to a sonnet box Right. That talked to an ATM switch that then talked to the router by stripping all of that complexity out and going directly in a land type architecture, to IP directly over DWDM, we could provide the price decline curves in those rollers, Now, the routing market has never been overly competitive.

It's always been effectively and duopoly. And the price per packet forward for the past thirty years has declined at roughly forty percent per year. So it's There are a couple discontinuities there around c twelve where everything went straight Ethernet. That's right.

Just go ahead and step functions, right, if you just grab that. The last five years, it's been awesome.

I think we'll continue to see those types of improvements.

If you look at the basics that are used in routers, they are typically still built on nineteen nanometer traces.

Today, new microprocessor fabs for being built on five or three nanometer traces.

Today, seven is relatively common in the market.

So there's a huge amount of catch up that routing has to do to compute. It's it's interesting because in the nineties, I used to say the router would have yet whatever four year old man Macintosh will have. You're right. It's actually much more. It's stretched much more versus, you know, the HPC chips and things like that and what people can do.

Yeah. And I think we will see a continuous improvement in routing technology for decades to come.

So with that, as long as the market for transit remains competitive, and yes, the competitive landscape has shrunk other carriers have to emphasize the product, but there's still enough diversity of suppliers that there's competitive attention, prices should continue to decline at twenty three percent per year, and we'll be talking, fractions of a cutting effect.

Probably over the next step. Yeah. No. It's it's definitely been interesting to see. And, you know, we work getting with a lot of these folks and everyone has their different take.

What's fascinating about cogent is probably the highest terabit her, I'll say backbone architect of anybody that I've seen. And I guess, is there is there a secret to running, you know, lean effector teams? Is it that focus that we don't bother them with other features, is it, is it, something about the culture? What is it that enables, you know, such a tight, clueful lean team?

So I think there's two major messages.

One, you have to be relentless in driving down the cost of an interface routed bit mile.

So an internet service provider sells internet access, but the reality is we produce routed bit miles connected to other That's our factory. So everything we do is designed to drive that cost down.

Two, it's about standardization.

So our network is about sixty thousand route miles terrestrially.

Spanning every continent.

In addition to the long haul network, there's about another seventeen thousand route miles about forty thousand fiber miles of metropolitan fiber in two hundred and fifteen markets about a thousand and twenty five rings.

Between those locations, we have just under a thousand nine hundred and ninety five optical transport nodes. And if you walked in to an amplifier site in the Ukraine in Finland and Sydney or in, you know, what you call it, Kansas, the rack faces, the diagrams will look identical. It's all about standard. Is it all one single vendor? One vendor. We are end to end a Cisco shop for routing and transport.

Now, Cisco has never been the absolute best in any of those technologies, but The benefit of getting an additional half a generation of technology advanced is equipped by the savings you get through standardization and operational simplicity. So there's a lot of, do it once replicated a cogent as opposed to bespoke design, and the product focus also is a big part of this. I mean, we basically sell three things. Right. So internet connectivity.

We sell VPN services on top of the internet. We spelt cell space and power and our fifty four data centers that we operate.

And that's it. Well, I didn't realize you had that many. Yeah. I've been to the old PSI net one, and, the current end. Yeah.

Yeah. Now, our biggest one is actually the old Earth link facility in Pasadena. Oh, interesting. I didn't know that wound up with you.

Yep. That that's, I think our largest have a pretty big one. I know you've recently relocated to Seattle. We have a big facility in Seattle.

We have been in New York. We have been in Madrid. We have been air as we have in Frankfort, Illinois around the world. We operate about six hundred thousand feet of rays for space in those fifty four centers with about seventy megawatts of power.

So you're not competing for the bitcoin mining market.

People tend to want to be connected where to it's it's to enable people to connect things to the network, and take time. It's it's for DRs, for corporate customers who wanna put a, you know, a backup facility, but don't want to go into a carrier neutral at higher prices.

Could be for some of our wholesale customers who look to co locate directly adjacent with our network. Mhmm. And these facilities also house the majority of our hubs. So we have roughly a hundred and ninety one cogent central offices.

In those wires. Super pops. Yeah. They're pops that they're basically they're the intersection of the backbone and multiple rings of fiber in a metro.

You go to the metro. Right. Yeah. So you you'll use the same ring for I'll call it backbone as you do for access to the buildings.

Absolutely. But it's all rigged. So it's not like it's stealing bandwidth from anything. There's I mean, you just get a better prism and you've got more bandwidth So one of the innovations that we did was our network is entirely ring architecture, there's no single points of failure, and we use wave- version multiplexing to allow each building to be a logical hub and spoke, but on a physical rank so you get the best So you don't have to break it out at the IP layer in that building.

That's correct. Basically, it's on the wave line. Right. Which building gets its own. Right. Pair of wavelengths in two directions, both clockwise and counter clockwise, and if a fiber cut occurs.

The restoration occurs through IP fast reroute Right. As we're able to redirect the traffic. And you're still up to both devices in that pop, you know, they each have their own wavelength. That's correct.

Right. And they have them in each direction. So so the network doesn't know that you're using the wavelengths in the clockwise direction where you're also using it in the calendar. You don't have any fancy learner glass, robots, you know, affecting mechanical smitches in town.

Yeah. I remember when some people called it. I won't I won't say the acronym, but, you know, or it wasn't only CWM, of course, but, you know, passive. And I remember, trading away with a big web company that starts with the g back when I was doing packet fabric one point o because they couldn't they had to put this opt up the WDM equipment in.

It was gonna take, I don't know, like, like, ten KW. I'm like, but it's optics. Why are you taking ten KW for a prissel? But they were good running down.

Right. Right. And you said, it's all the APOC stuff. It's all the protected, you know, protocols and all that, but you know, yeah, if you over by, I mean, yeah, if you're going to space, you have to be really careful and maybe, you know, if you want different protocols, but There's enough by her, or as Dave ran at above mentioned, QOS should be quantity of service.

Right. And that's always our model. There is no QLS prioritization.

Queuing. You saw you throw bandwidth at every problem, and all the problems disappear.

Yeah. No. It's it's funny because I remember when I came to above that, they was like, someone wants a mistake. All we deliver is Evernet.

Someone wants an OC. They could put it in their rack, but our endpoint is the Ethernet, and then you took it one step further. Like, make the Ethernet come to the, come to the custom. That's correct.

Yeah. Is pretty cool.

It's funny. What's the network like yours, there must be something down all the time. Is there ever a time when the circuit is down anywhere?

So with the breadth of network, we will typically have two to three fiber cuts, every day, somewhere in the world. Yeah. I mean, backhoe has just seemed to have a backhoe. Backhoe, It's almost like you got the binding rock directly to the fiber. But the regular architecture gives us a lot of resiliency.

We've actually never had a backbone outage in twenty years of operation.

We're very cautious about rolling out new code.

We have, again, a lot of redundancy built in the network. And a simple product style.

But, yeah, listen, endpoint customers can go down.

Typically we're running about zero point zero three of a percent of port seconds that are down on our network in a given week. We actually monitor for that being on it every week to try to improve it.

It almost all of those typically are the customer prem equipment, either failing. Right. Someone on plugging them. Right. I'm not saying we never have an outage. Someone does a fat finger, config.

But for the most part, the network is Very, very resilient. No. We've seen that as well. I mean, really, which I guess gets to the next topic in the nineties, if if COVID had hit and we had tried to all ship each other webcams and use the internet, I'll just say it wouldn't have worked very well. What it failed was too much massive oversubscription.

Remember, our local network is not oversubscribed and unblocked notice. No statistical multiplexing and are But you don't peer with people at every I mean, some people you peer more widely, but you don't peer in every metro.

With. So most of our interconnection is actually customer to customer.

So COVID today runs BGP sessions with about seven thousand six hundred ASs.

I don't think there's anybody who's ever even approached that number.

We're running roughly twenty five thousand. And the internet is your customer. You appear with many networks.

So, Yeah, when Contrin got started, we obviously had de minimis, Morica share.

And at the very beginning, we had to buy upstream, and we we're supplementing that with very open peering policy.

Today, we peer with less than twenty five networks globally. So you flipped it from open to be my customer? Well, I would say that the strategy has stayed the same, The difference has been the bar to be our peer has gone up as our scale has gone up.

And, you know, I think we are the most interconnected network.

So, obviously, we're running BGP sessions and over thirteen fifty endpoint locations, those data centers.

And we're pretty flexible on where we'll appear. We can actually period any one of those facilities if someone wants to peer or we sell transit, which is in fact how most of our interconnection occurs to that. Right. But there is gonna be some there is some oversubscription getting between networks, but it's fifty x less.

I don't know the exact number than it was in the nineties. You know, I mean, when I started at Akamai, I remember asking the Zen question. If all we're doing is optimizing, you know, around the congested paths that at some point what we have done all the optimization, the internet will suck equally for everybody, but I think we just got fortunate that, one, you know, things have evolved. And as you say, the costs have come down so much, and two, most most of the the networks that I've seen and and worked with had ordered enough in advancement, couldn't fit that even before the supply chain stuff started getting bad, they had either built or they had equipment ready.

So, you know, there was enough capacity pre built that when more things went online, you know, things, did okay. So how has that been for coaching?

So it may address the Oh, sure. Over subscription peaks. So, clearly, we don't dictate to our customers what size ports they buy. Right.

But we priced our service in such a way that most of our customers generally utilize twenty or thirty percent of their port capacity and kind of an average So you're willing to give people that kind of birth. That's correct. You can't buy five megabits and get a hundred gigabit for it, but right. Yeah.

We we try to be very flexible.

If we have in excess of a exhibit of capacity to other networks.

So very, very broad reach for the geography and and scale.

The twenty five networks we peer with, because these are non monetary agreements, they tend to be stingier than we are in terms of I saw that at above that too. We were like, well, why don't we just do ten x, not two x, the capacity. And we're like, no, it's okay. We're, you know, we're good.

We'll just have two x. Yeah. They kind of want to ration out that. Let's usually because their networks can't support it.

And at the end of the day, I think most telecom companies and cable companies won't admit, as the intermittent, the internet is their worst nightmare.

It drives up the load on their network by orders of magnitude and concurrently drives revenues down by orders of magnitude. To.

So if you were a phone company, you love selling for sixty four kilowatt hot circuit that people use, basically, fourteen hundred minutes a month at sixty four kilobits.

And Mike was great. So it would go for twenty five bucks. And they had no choice, you would break. Or conversely, if you're a cable company, you would take a six five megahertz channel, delivered video.

Well, correct, it was shared between all of the to users a symmetric line. Thank you charge for one, you know, eighty bucks a month, and mine was good. Now, the customers demanding hundreds of megabits of end user capacity. And the content is moving to the internet instead of necessarily from your network.

Sure. Your revenues are going down while your capital expense is going up. So legacy providers who still dominate the Internet, whether they be telephone, cable, or mobile phone. All.

Take dinner. I don't know what's if, but John Stakey could go to bed tonight at AT and T, I wake up in the morning and the Internet was gone. He'd be a happy name. I would say that I'm not gonna have any names.

Some Non eyeball carriers do still have that. Let's just double the ports and not, you know, four to ten x. Some eyeball networks do better than others or the IP engineering people.

I had Elliot Nelson on. I had to we had to edit out a few things that were said about telcos. This experience too. So, you know, sometimes the IP engineering team is is is doing the right thing, but I understand what you're saying.

Or if you took a rational person and made them the CEO of the company and they looked at the inputs and say what makes what helps my company and what is a threat, then yes. Right. But if you built the right network and you built it around the products that people want, you can actually make money as high as Well, I I think that there well, that as I had Elliot Nass on him is, like, why? Why?

He called me, I don't know, twenty It was like, I wanna start an ISP. You wanna help. I'm like, what who bit you? Like, what got it?

It's like, no, no, we're gonna make money. We're gonna it's a mid stock. And we're gonna make money. And, like, why not?

You know, and for him, he's been working on solving, like, billing challenges, which is one of the problems that you solved, you know, early on. For, but it's still a huge, you know, problem for a lot of companies. You know, yeah. I I I would say some again, some of the incumbents Some of those some of those folks do have very modern networks, you know, and IP sits at the sits underneath, and then Some of them are straddled with fiber plants where they've got, you know, three hundred DB loss from side to side because there's forty splices.

So, you know, yeah, it's an interesting market, and it's and also, as we talked about, like, what's how many products do most of those companies have? They've got consumer, they've got business, You got many different products. And again, it's, it can be hard to navigate. And it's interesting working with companies that have focus versus working with product companies, you know, and seeing that.

So with COVID, how did that how did that affect from the, I guess, I'll say the the people culture side and the business side? You know, so what's wrong with our coaching team?

We put their safety first. They have about eleven hundred employees and in March of twenty twenty, they all work remotely.

And they continued through October of twenty one to be remote in the US, and in some parts of the world, they are still remote.

As vaccines became available, we implemented a mandatory mandatory vaccine policy, and we have required employees to come back because there is a huge benefit for people collaborating face to face.

And the culture of the company is a interesting mix of mature processes with startup drive. That's really what we try to maintain. In our customer basis, there's really been two very different tales from COVID.

For our corporate customers, demand fell off. The access building using the office. They don't know what the future's gonna look like. They they know their employees are working from home. They upgraded their VPN concentration ports, but they're not willing to make massive rearchitecturing of their businesses and networks particularly migrate away from MPLS to a SD WAN or VPLS solution until they have clarity on what the new work environment is gonna look like.

For our wholesale business, we saw an acceleration. Right. As people stayed home, they had more time, and they strained more. So we have thousands and thousands of access networks around the world if I upstream, their consumers were spending more hours a day.

They needed more bits. We've sent that in the config data, you know. Yeah. And we've been accommodating them over a hundred PTTs around the world, for example, by the upstream from coaching.

And then on the content side, you know, the streamers and the application providers have seen unprecedented demand. There's been a wave of new entrants, whether it's Disney or Amazon that are broadening out the streaming market, And, you know, that has helped that business. It's actually, we've had the best two corners over the last two quarters in the company's right. And that host should I haven't I haven't read your your your quarterly file filings or the transcripts, but you consider that the same business, the the access business for service providers and the content.

Side that's all wholesale or you break this out? You traded that down. Okay. So it is that.

Okay. We we coined the term Netcentric, and it just means we're selling bulk bandwidth to someone who's reselling our bandwidth, either as an access provider, or what the context of the companies where Synchronet was never an option. That's correct. Their network is their business, delivers the revenue.

We have Similarly, we have traditional enterprise, and then we have Netcentric, you know, as well, which is. Yeah. That's all logical, dichotomy. Yeah.

Yeah. No. That makes sense. So what was the reaction to mandatory vaccination?

You know, we lost a few employees, not many. A few. We had, a lot of counseling to do, a lot of controlling, and it's you know, it's a combination of making sure coworkers are safe, making sure the work environment is safe, and quite honestly making sure employees are safe. And it's a balancing act between people's personal liberties and public health safety.

Right. And we also We are in a society Yeah. We're social animals. And we also had the way the benefits of getting people together versus the isolation of being on our own.

And we have a big sales organization, and that sales organization typically has turn it. Mhmm. Sales people come in, some very quantitative, some sixteen or something and to train their sales people.

You can't lift your car. Fair it or bus online and classroom training materials to be done remotely. There's no substitute for a mentor on the next desk. So Yeah.

You know, we just had to get people back to support that growth in sales organization. I'm told that I shouldn't say Osmotic learning because people don't know what I'm talking about, but it is funny. How just absorbs it? Yeah.

It's funny how but I try not to use dictionary words. You know, c o u eight speak sibley and clearly and everyone can understand you. But but yeah, I mean, it's it's interesting at networking, how and the many professions how much we are not that far from the middle ages of you know, by the ones and apprentices and, not about power structure and do my work for me for three, but know it is known in in the the game of Thrones. Oh, it is known.

Like, did you not know that?

Did you not know that that network doesn't have that or that that has, you know, if you turn on that protocol that way, it does that problem work. And there's ways of doing it constructively, which is not like you're an idiot, but we're like, hey, one might think this, but actually, So, yeah, that's it's definitely interesting.

I I I'll tell a story. I remember early on a coach. I had some exceeding with talented routing entry. If she could inform more about rallying than I get, I to serve myself a amateur hack, a mess, and, you know, maybe he was more of a classically trained person. I came into a problem and whiteboarded to Kronicksburg bridge problem.

They didn't know what the hell I was talking about, but they knew how to fix the problem. Right. So I'm thinking about more of the way whaler thing, solving the problem. You're thinking about, like, oh, I've seen this, and this is how I need to know.

Well, there is a lot of pattern matching, but that reminds me of a story. Probably why I don't teach. I had a great teacher was a doctor who got into image processing. And when I went to Temple University, the first day, and I met this this guy, both and I got ops staff who did networking.

And I was like, oh, AI image processing. That's the really interesting stuff. Networking. Yeah. Yeah.

And then, of course, I was, alright. Got into networking even more, but And so he taught a class on real time systems.

And he asked a question on the exam, why do tires appear to be going backwards? We had studied my quest and sampling rates and all that. And so, but a lot of people in the class thought it was unfair that he'd asked a question that we didn't didn't we never talked about tires. Right? And he was asking to make leap from computers to brains and your eyes sampling. And so, sometimes that that you know, from first principles. But again, the classic physics education of, let's, let's reason and understand.

Is it war with the practical real world constraints. Yeah. Uh-uh. Or as as, you know, again, we talk I talk with academics.

And I was like, you have to understand internet is a bunch of garage mechanics banging on routers with wrenches. And, like, yes, there is science here, but it is not primarily science. It is. It is definitely and are, like, current I wanna do car talk for the internet.

You know, it's like, what sound does your router make it? But you sort of see that. Right? It's like you log in or API, whatever you do.

And this thing is like that. You're like, uh-huh. I think that's something we need to formalize not just because of COVID, yet, for us as we strive to be the more diverse. The most diverse set of folks is earlier in career.

And, yeah, how do you get those people and how do you train them both about networking and then what level? Right now for KenTech, we are We gave up our office because we couldn't use it when the term was up, and we're doing a lot of collaboration, but, you know, we'll see how that goes. What what percentage of of the company is is the whole. And he said you mentioned you have multiple offices.

So, we actually have plenty of offices around the world, mostly sales rent. We also have you know, about ninety people in our field services organization that either can work in an office or some or so about they work from home. But here in DC, there's about two hundred of us out of the eleven hundred Mhmm. Probably another hundred or so in Northern Virginia.

So about three hundred of the Avenue in this area and the rest of the company is truly global. Singapore to stockholm to Seattle, to LA. I don't want to make you name the law.

Or ask, you know, when's the last time you visited all of the offices? You know, my goal pre COVID was to get to every office once a year. Oh, every year. Every year. And I did pretty well. I would typically get about thirty five of forty done in a year. Wow.

That was pretty much my pace for the past decade.

And then COVID hit and I've been doing office outside of DC in two years. I just last week was the first week I was at home for three months. It felt like three years ago. And it started in the middle of that was Kukan, which was mostly vacant.

Very safe with lots of space between spaces, between booths. And then, a couple weeks ago in Vegas for reinvent, it was, like, almost full. Even as omikron was, you know, was firing up. So you know, it's interesting.

Y'all is type one diabetes, so we try to stay safe. And, people are coughing on each other to stay away, and they have to decide whether you wanna go to masqueraded the world science fiction convention of all the people. So And we have pretty strict protocols here in the office, and really do try to keep people safe. So even even with mandatory vaccination, people a mask in the office.

Absolutely. If you're in an open space or if you're walking right up, mask, if you have a private office with a door off, you can take your mask, while you're at your desk. Mhmm. Perhaps and that masks all the time.

Awesome. I mean, I I can tell you that there are definitely we're doing an off-site in January, and then on the con head, we're still planning to do it. Number one, you know, in case we talk to people, they wanna feel safe. You know, be comfortable, that you're protecting that.

So two last questions, what percentage of of, coach and hires are, I would say, early in career.

You mentioned sales, you know, in engineering, energy prefer to hire senior people and bring them in, or is there a thought that come do do you try to wait one way or the other?

So it's kind of a tale of two different workforces.

Cogent's operations team.

Have average tenure of about twelve years. Right. We actually I do it. I do it longer.

All the people I know are longer. Yeah. I know people that twenty years. Yeah. I mean, we have very little turnover on when the operations are.

So we just don't do a lot of hiring. Make occasionally, people retire, someone moves off, and that, you know, will bring in Is the knowledge a traditional place for our technologists to come in? Typically, we're working for people who who have interest, but not necessarily the exact skills we need. And we'll train them.

There was some kind of engineering computer science background.

And then on the sales side, it's a very different sort.

They will, I would say, the average higher is in their late twenties.

We hire smaller people we have a huge shipment of seventy eight and seventy eight here in sales receipt. Number ten, we have sales reporting. I know a lot of people that started their career in sales and coaching. Yeah. But, you know, it's sales is brutal. It's very quantitative and It's very demand for calling, which is how our sales model is organized.

Is try the hardest job you could imagine. Recruiting number two. Sales number one. I think that's right. Yeah. And, yep, for that reason, people say they could do it, train them if you can.

And it's very hard. So there, that would charge probably late twenties.

Second or third job selling, but not necessarily It's not right out of college.

Not right at we do hire right at touch, but generally not, but seriously you sold cars, you sold insurance, you sold so, you know, life insurance, medical support, something. Yeah.

Or technology. Mhmm. And then we train the technology component of it, and, you know, some six eight and some down. But on the operation side, we have a very stable workforce.

Okay. Yeah. No. I I have definitely seen that, but It's as we get larger, that's just something we've been thinking about is is what is the what is it that it takes to train people, as we started and we just had a big network of people that had a long experience.

But if you higher from your network than, you know, you don't have diversity of of thought and opinion and wisdom and, and background. So we're working on that. I guess the last question, if you could go back to, taxi, Dave, earlier earlier on, any any, any lessons or advice?

Yeah. Question. I I think everyone kinda has to find their own path I think, you know, I'm often asked by salespeople what makes me successful under basic life skills, It's the tenacity to stick with something.

It's the flexibility to change when you need to change.

It's organizational skills.

You know, time is your most precious asset, and you have to organize your time, you have to set goals, And you don't always reach them. I mean, I start every day with a so a white notepad with to do that. I say I won't take a picture of it, but I see it. Yes.

Yeah. And, you know, that, you know, I've worn through. Uh-huh. And, you know, what I'm talking about is use index cards and now use the icons.

So now, I I I know I be more high-tech than that, but it's it's old fashioned.

It's just how people manage your time, focus on what you wanna get accomplished set realistic goals and successful comp. Mhmm. It's that simple.

Cool. Well, thank you very much. Thank you for sharing. Thank you for hosting, Abby. Absolutely.

Well, thank you for physically hosting me. I know the cogent story is fascinating, and you've got a lot of customers out there. And people wanna look at cogent. It's cogent dot com.

Any other ways are you with the social I'm not a very social guy.

People can reach out. I talk to customers all the time. Okay. So be your customer, the best way to get connected thank you very much. It's been a real pleasure. Thank you very much.

Got a guest?

Network AF is accepting guests for upcoming episodes. If you’d like to be on the podcast or refer a friend, reach out to networkaf@kentik.io.

About Network AF

Network AF is a journey of super-nerd proportions into the world of networking, cloud, and the internet. Avi Freedman, self-described internet plumber and podcast namesake, hosts top network engineering experts from around the world for in-depth, honest, and freewheeling banter on all-things-network — how-tos, best practices, biggest mistakes, war stories, hot takes, rants, and more.
We use cookies to deliver our services.
By using our website, you agree to the use of cookies as described in our Privacy Policy.