Kentik - Network Observability
More episodes
Network AF  |  Season 1 - Episode 18  |  June 21, 2022

Problem solving and pinball with Jay Adelson

Play now

 

Chairman and Co-Founder of Scorbit, Jay Adelson, sits down with Network AF host Avi Freedman to talk about his history as a serial entrepreneur. Jay founded Equinix, Revision3, Opsmatic, and was the CEO of Digg. Throughout the conversation Jay and Avi touch on problems founders encounter, and discuss their mutual joy for gaming.


Highlights of the conversation include:

  • Imposter syndrome and working at DEC
  • Server evolution, power efficiency, and presentation
  • How not to build a data center
  • Terminating a cross connect and turning off service for Australia by accident
  • Equinix during its hyper-growth phase
  • Richard Clarke and peering
  • The Forrest Gump of the Internet
  • Amazing that the internet and the human body work at all
  • Emergent behavior and the case of HD DVDs
  • An evolution of the anarchy started by the internet
  • Scorbit, gaming, and problem solving in pinball
  • Advice and lessons to a young Jay

Transcript

Hi. Welcome to network today.

Please to have my friend, Jay Edelson, who I've known for many decades, and used his services and, proud to have him as an investor in my projects with us. And, Jay, could you give us a quick intro? Tell us about Jay. Sure. Well, first of all, Avi, I'm I'm honored to be with you today. It is always an honor to be speaking to you and fun.

So I'm Jay. I I've been in the internet business since nineteen ninety three, give or take. When I was broke trying to be a a filmmaker and audio engineer, and a friend of mine, Jeff Rizzo, got me a job at Netcom.

Installing routers for people.

And that was a very long time ago. And then if you just sort of, like, hit me in the back of the head with a bottle and I wake up thirty years later, founded fourteen companies, most of them failures.

Few good ones, known for probably Equinix and Dig and revision three and couple other things in there. And and ended up kinda leaving that world and becoming a pinball professional.

And that is why I'm sitting in my office filled with pinball machines right now. So that's pinball design creation repair playing?

Well, you know there's an internet twist to this. Right? I mean, come on. There has to be an internet. Yes. Of course.

I'm really into retro stuff. And so I just thought it would be really fun to be able to connect to old arcade games, you know, mostly pinball, but basically anything, and push the scores and achievements like you do on Xbox Live.

And, didn't quite anticipate just how challenging a technical task that would be, and that was, like, seven years ago, which is crazy to think. But I mean, I still operate a venture fund, you know, the one that we invested in in Kentech with and and that, but I pretty much stepped out of that world in twenty seventeen.

To do this always, which is I know So you're IOT at the end. I'm IOT. I know we're we're an internet infrastructure and IoT investor.

Yes. And I remember learning that IoT is a terrible business.

And then I went and founded an IoT company. Well, it seems to be good doing okay for some Sarah and some others.

You know, that started really about the time of Kentech and, you know, I don't come from the skate of land and all that, but turns out there seems to be a good business in connecting fleets and home and stuff like that.

So I just happen to come from the other side of it.

Yeah, Gail, I don't have it all set up.

Gail does not prefer.

She doesn't love my hobby, of collecting vintage eight bit computers back when you could understand everything about the hardware software. If I were actually at home instead of having a background of at home, I'd I'd show you the commodore sixty four over there or pro. And get a Capero or an Acorn if I was talking to, to someone from the UK.

But she calls it Amos. I'll use the exam of stuff. That's awesome.

By the way, if you're interested in taking over my collection of Apple twos, I have everyone, every model, Oh, I Thirty two of them. I am very interested.

Oh, really. I have a warehouse Thanks to a friend of my not warehouse. Sorry. It's a storage unit.

It's actually it's effectively on the Ashburn campus. So, you know, the post office that, Equinix killed, kicked out. That's on the original Equinix campus there. Are in in not the original, but the the the I remember it had a it had a catwalk for people with guns.

Yeah. Yeah. So that that all got tore down in that. I don't know if it was DC eleven or something.

To the Guardian self storage there. And so a friend of mine gave me one of every not a spark not a not a sun one or no. There is a sun, two. And then every sun, three, and four, and spark, up to a four fifty.

I don't have, like, a e ten k.

Actually, no. There is a forty five hundred.

So, which used to be the the cheapest way to get a lot of ram used, but the power bill was not fun. So Now, are they operational?

In theory, I haven't fired them up, but I have a bit image of four one three u one and four one four with my custom kernel for Synflood, you know, for, like, Sin cookies that we had to do back when Penics was a customer who was being attacked. So Wow. You know, the fun old days. But you Yeah. Worked at deck.

I did. I worked in the network systems laboratory, which is sounds so like, I might know something about networking. Believe me, I don't. Like, the the thing about the NFL was it was like, it was, like, the Los Eisley or Moss Eisley Moss Eisley.

Like, can't You call that use that now. Richard Hive of bugs and flamers, you know, yes. You know, I I Or day or, you know, I You know, I I just couldn't believe I was there. Like, I felt like I felt like I was, like, somebody was gonna figure me out and that I was, like, imposter syndrome in the middle of this lab next to Paul Vixie, you know, and Steven Stewart and you know, and, of course, Ally Avery, Yeah.

And all of these guys who were really doing cool stuff, and I was so excited and they could tell they could tell, but I I've was hopelessly addicted to this idea of internet scale. And I it perhaps became a little too obsessed with it. To, my wife Brenda, She still makes a joke that if my phone rings in the middle of the night, she always says the same thing, which is tell him to reboot the port master.

The port master. The port master. Yes. Yeah. Bloomingston port master was like Yes.

Most early ISPs use something similar for all of these. That or if I were again, if I had an updated picture of my actual background at home, I have a a a micro annex a terminal server. I use the Zelle Logix Analytics, later living, later, wellfleet, I guess, bought them.

So, you know Yeah. And I Yeah. There was a lot of a lot of late nights, and I was an operations manager to be clear. I was like, some people have said, OJ founded packs.

That's not really true. I mean, I mean, it was operational as an exchange point for easily a year before. Well, I remember Well, we're we're getting ahead of ourselves because we've seen my folks. Yeah.

But, yeah, I I remember, when I first, well, well, okay. So who who did who did start it? Well, it was called Tabista connectivity? Or There were it's it's an interesting So so I would love for Paul or Steven to one day get into the really boring details of this because I'll tell you what I know.

Right? Was that when I was running network engineering and operations at Netcom, which I again have no idea how that happened. But I was operating there.

The deck network systems lab is in Palo Alto, and it was in a basement of this of what is currently now the packs.

And, Paul, who was part of the lab, working for Brian Reed, who was the director of the of the lab, had somehow convinced Bill Yunt, and a couple other folks involved with Barnett and Alternet, who had been exchanging traffic in the Stanford lab, computer lab, I believe. Because I remember Paul had showed me the, I think, the, the aims side No. No. He showed me the kicks.

He showed me the kicks. Not the aims side of May West. He showed me the commercial internet exchange, which was in some office park you know, because it was the secrets of peering and who if you go to the kicks, do you get That's right. No.

That's that's right. And I think that that you know, Paul is a visionary in a lot of ways. And I think that one of the things that he recognized around this time when he was you know, helping to, I think, administrate that interconnect. Mhmm.

So he figured out that there was there was a huge issue on where a place was and who was operating it because, you know, in the case of Stanford, I think they had a squirrel literally take out the the the data center and take out, like, half of the Western Seaboard of the internet for for some period of time. I mean, it's a it's a story that everyone tells in some form. I've never heard it, but I'll ask. But I mean, everyone's got a scroll within the power box story about their data center going down.

I hit the EPO button to exit. I mean, that would make a lot of sense. Right? And that so, yeah, you lean against something and it pops off.

I guess Stanford set up had never been set up for, you know, reliability.

Neither was the basement of the network systems lab.

But somehow, Paul convinced alternate and Barnett to move into that basement is correct. For people to keep track, so alternate was commercial.

Barnett was the Bay Area Regional Network that was They were all semi commercial to regional networks, but it was sort of the the government NSF, you know, access network. So Right. This is before the official Right. Or just no. There's just after the contract was handed, I think, to COA.

To the various carriers. Yeah. Yeah. Yeah. And, And he so so he moves it in there, and I think he he, you know, Deck was also a hardware manufacturer and manufactured network switches, and they made a a Fitty switch or a fiber, interconnect.

Mhmm. And so he had them I think I'm originally connecting on a Cisco like serial number one of some old Cisco. And then and then he in parallel, in parallel, he started the commercial internet exchange with Barb Duly.

And that initially, I think, was a lobbying organization designed to sort of speak on behalf. Policy side of it, right, and peering and, yeah. Exactly. Cause I don't think that, you know, at that time, it was it was anarchy.

It was the it was the it was the wild west. Right? And there wasn't a lot of regulation or oversight or thought given to how this stuff rolled out. And I think that Paul felt very strongly that, it should be community based managed exchange point.

And meanwhile, though he was doing this at digital, And so digital decided to productize it as as what business unit inside of the network systems laboratory. And when that happened, that's when Al Avery took over as the general manager, and he hired me to basically run operations.

And, yeah, I mean, by the way, I I'm I think that Paul was right on a lot of fronts in that, you know, there is challenges when you're commercially driven, right, because you're you know, you're on behalf of the the shareholders and that's an issue.

The the flip side, though, is community based organizations just ask all the European exchange point operators to, like, move anything one port over is, like, a year of debate and I just I remember because contemporaneously, Doug Humphrey had showed me a little bit before, a year before, like, the nineteen nineteen Galows basement, you know, May East where, you know, like, unit and psi routers were on a on a power strip on a two prong extension cord plugged into the wall.

And, you know, I I remember visiting packs of course, Ulta Vista was like there, and Ulta Vista was the cool search engine at the time to to use the deck hardware, which I was a fan of. I'm not a big VMS fan, but, you know, but, you know, I other than architecture, but, you know, more of a Unix guy. But I remember thinking, Wow. This is, like, real because, you know, going to kicks, going to May east, going to May west, and, you know, with no offense to our, you know, MFS, sisters, and brothers, you know, it was a little bit of a hack job.

It was, like, my, my, you know, my Kolo and Philly where I convinced MFS to give me a rack for every t one that I sold in Sunday. I own this whole data center. It was all DC power. And, you know, had only AC equipment.

And they're like, what are all these computer things that you're putting in our in our building, but it was, you know, it was like a room in an office building full of lawyers. And, you know, I remember going into packs. I was like, Oh, they only turn on because the guard shack the guard thing was upstairs. If you go downstairs, they only turn on the lights for the people that are there, so you can't really snoop quite as much.

There were set of rules we had to follow. And I was like, wow, this is getting real. Yeah. You know, I have to say.

I I remember that a lot of that feeling was planned.

And and thought through by Brian Reed.

Brian, who, you know, brilliant, you know, network engineer who who ran the the system laboratory there hired an art designer.

To come up with, like, colors and patterns and and, you know, fun lighting to try and make a very small space, a vote evoke some kind of, sense of, of safety and reliability and importance And and the thought there was while you're kind of, you know, building an empty exchange point and hoping people will come. Mhmm. And I think the idea was is that if you build it and you and you enforce these rules and you're in your, you know, think about it in the tradition of maybe the finance and, you know, the banking industry. But not as annoying as doing Colo in a CO.

Know, it's not the company, which was very annoying at the time. I mean, there there was a certain amount of, you know, lipstick on the pig, so to speak. You know, a a certain amount of it was, you know, dressing in front of what was a very roughly built data center that was incredibly effective giving the small it was five thousand square feet. Right.

Yeah. So So people who are, like, building their their first data center at two hundred and fifty thousand square feet. And how many many gigawatt? How many?

Oh, man. I I Two hundred KVA? I've I've less than that. I mean, I think we might have said that.

And I remember negotiating with for software people out there. Like, you can now have the equivalent of, like, twenty houses of power in one cabinet with or without, you know, immersive cooling.

But at the time, you know, it was not that much more than a few light bulbs per cabinet worth of, you know, heat and power that one was, dispersing. So, yeah, that was the internet. That was the point. I mean, by the way, if you want training on how not to build a data center, you know, one of the things you could do is take someone familiar building out points of presence like I was and put them in this, you know, spaghetti set up and tell them that the entire internet depended on its functionality.

Right? And, we had some great assistance.

There was a a contracting firm named MCM, and this this guy named Jim Brush, who looked like the prospector from, like, the toy story movies, like, exactly the same.

And he would walk in and basically explain how in the days that they invented electricity This was how they solve this problem for, you know, this or that. And I learned all the lingo and, learned basically how to talk to electrical engineers.

Mhmm. Not necessarily how to be one, but, you know, certainly how to understand that.

And I think that Alan and I began. That was when we started fantasizing over beers and cigars.

How to, you know, how could we do it if money was no object?

What would it look like? And and how could we how could we grow this thing? Because we were also friends with all of the network operators, and and they would complain about, you know, Tyson's corner, you know, a a parking garage and eighty one hundred boo in nineteen nineteen gallows. The, you know, yeah.

And there was one there was one exchange point. I believe somebody actually perished due to an electrical accident. I don't know if you knew about that. We won't we won't name any names, but but some of the folks watching your your, video will probably remember that I think it was right around, had to have been ninety eight or ninety nine There was a, an incident where a electrical technician and a carrier exchange point, touch the wrong thing and was not good.

And so, like, I went to the Pentagon app, and I remember Oh, yeah. You know, that was a little bit too real. That that was more like that was closer to the c o level of, you know, how how how things thought, but it was pretty big, you know, for what was there. It's arguably true that carriers knew more about compacting a lot of network in a small space. Better than anybody else.

I just think that, it was never a question of someone losing a lot of money Well, at least from a commercial internet standpoint until way later. Mhmm. And once once you got to that point where content eyeballs and the path between them kinda leveled out in importance in the ecosystem.

Then you kinda had to change the way you designed all that stuff, you know, but that was really fun. I remember being at a Nanogg unlike some people, I can't remember, like, yes, Nanogg twelve was in this city and, you know, whatever. And think it was you had announced, maybe Bill was up there, had announced Equinix. And I'm like, Equinox, they make serial port cards. You know, I was like, what? But But, you know, you mentioned rules, and so we do have to tell the story about the cutting off of the non Tulsa transit of Australia.

I have a friend. You know, I I I sometimes I definitely try to figure out what are all the ways to interpret the rules, you know, to create good humanity of myself. But, we knew one of the roles of packs was thou shall not run thine own cross connect.

Right? Now a lot of people in data centers now will will use passive optics to pay for a cross connect and make a cross connect in it or you know, packet fabric version one, Equinix got upset with me. And so we sure shut down, you know, because it was clear that we were trying to do multi data center, but when people figured out it was cheaper to do exchange in the building over it. This is before Openix started to drive the prices down.

You know? But so, my friend and, person who's less, I mean, he is aware of the rules. Feels less concerned about that Andrew Ku have run a cross connect between our we each had a cabinet, net access had a cabinet, and, and, and he aircaged within that cabinet Sorry. No.

No. No. Not cage. Relay rack.

Sorry. Relay rack. That's right. Not not even Kager Kevin, and I'm thinking, you know, modern.

And, and I was giving him to transit over, I think, actually, I was tunneling to make it look like I had, you know, at least lying to that location. And then he called me, and he said, Avi, you know, my transit's down. And I think it was funny in AGS plus at the time. And, yeah, I remember getting a call from you.

Hugely apologetic. He's like, Avi, I'm really torn here because, you know, you guys violated the policy, but you should never interrupt operating networks. We talk about these things. No.

Wait. Okay. So here's the untold part of this story. Okay. Alright. So to be clear, you're end you end up by by terminating this cross connect, you you definitely turn off a good a good portion of the continent of of Australia.

So Right. So, you know, probably not, a great decision.

So we had an operations manager at the time who worked for me, a guy named John Pedro, and almost anybody who was in the internet space back then knew who John was, a very sweet guy who worked at the packs and was responsible for, you know, sort of managing the day to day task work of the technicians who run the cross connects.

And he discovered this cross connect, which was surreptitiously run between two points and told me about it and asked me what I think what I thought he should do. And I said, well, I think you should email everyone and let him know and And, apparently, this went on for several months where Oh. Where John was emailing someone by the way, I don't know if it was Stuart or Andrew. Like, I don't know who who it was he was emailing. If he was doing it right, probably should have been me. We had no escalation process for this because we remember work at database.

And but but but the but the thing is is that I think it okay. This might be revisionist memory, but the way I remember it was we discovered it, and we said, hey, If you pay us whatever the cross connect price is per month, we're good. Just send us the money, you know, and and we'll be good. I have no idea how billing worked at digital. Like, I I can't remember.

I remember we had just invented these these terms of CNI and GNI, and INI and PNI.

And we and we were like inventing one. For, like, you know, within a cage kind of thing.

And I guess some time passed.

And and I don't know.

I it makes me remember when it happened because it was a middle of the night. I got a call.

Actually, I got a page. And you said reboot the port master?

Basically, like, you know, this is down, and this is a big deal because it affected a lot of other folks. So at the time, This was supporting satellite based bandwidth augmentation.

So people would buy their e ones in Australia, and they'd be full on the inbound side, but they have more outbound.

So we were selling, you know, eight meg circuits that were, you know, basically one way frame relay for people to get excess capacity for, you know, and they could make SSH and IRC go over terrestrial and use net and web caching go over, you know, satellite So, there were it was it was a lot of circuits were, you know, and for latency, it was satellite back, but this was supporting this is the transit for that satellite thing. So there were a lot of small networks. It wasn't affecting telstra, but there were a lot of small networks. Yeah. Yeah. And The other problem was how we did it because back then our cross connects were largely, I think, multimode fiber, if I recall, and he cut it with a pair of scissors. Yeah.

He didn't just pull it out. Yeah. I don't know. Like, he he would get angry sometimes. And This must have really made you have to understand. He was a very sweet guy, like, very peace, peaceful. He would calm you down.

Very well organized, not the kind of guy that would do something vindictively in any way, shape, or form. And so for him to cut it with a pair of scissors, which is the part I remember, implies a long history of battle. Prejudice. Yes.

Yeah.

You know, packs had a cable plant it had a paper system for knowing, you know, what was plugged into what it had all of these sort of, like, inherited, characteristics from the Paul Vixie, you know, And, also, the hard days. It's definitely, you know, real engineering, you know. I mean, scientific computing and business computing and, you know, it's, you know, there is a there is a, Well, here's the the the ultimate the ultimate thing about that whole operation is that it never went away. Like, like, the packs would later, you know, become acquired by Is that your data?

No. Before that, it wasn't above yet. Above it. Yeah. Sorry. I was there because the name was running a generic at the time.

Yeah. What what was the It was above net was acquired then, though. That by MFN. By MFN.

And then Right. And then MFN acquired, I think, or I don't know which order, but had the packs. Mhmm. And then later switching data acquired them.

Right.

Or acquired packs from from MFN maybe. I Yeah. That sounds right. If you look it up, And and then ironically, I I don't know, a decade later, maybe more.

It's like it's like, but with networks. Exactly. I mean, everything came around. Yes. And Equinix ended up owning Mhmm.

The packs and that it was aft I hadn't returned to the building the entire time until equinix acquired packs, like, I don't know how fifteen years later or something like that. And, and, despite the fact that the original deal when L and I left Pax, the deal we struck with Bill Strecker, who was the CTO of digital at the time, because digital was getting acquired by CompAC. CompAC. Yeah.

Was that if they were going to sell that we would get a right of first refusal, And, let's just say that, oh, and we agreed on a price. I think it was, like, thirty million or something like that. That's advantageous, you know, to fix a price. Yeah.

And and I think it's all to above net for, like, two or three times that. So we were out of the market. But but, yeah, came around So it's it's funny. Thank you.

Equinix.

Equinix is a a customer.

We all of, you know, we have some stuff in DRT, but for the most part, I'm an old server hugger. So as SaaS company, we make our own cloud in Equinix.

That's where I have my usenet where I bootstrapped, you know, cloud helix that, you know, can take on my usenet servers.

But, what was it like during hyper growth? Because, you know Yeah.

I mean, I I I would love to tell you that it was it's funny because because now looking back, across my career, I think, really only twice or three times I've been operating a business or founding cooperating a business that that had a hyper growth phase at all. Right? And and the Equinix one was brutal.

It was, uncomfortable.

We raised a just under a billion dollars It used to be a lot of money before crypto, before your content. Well, remember, like, so this is pre nasdaq crash, we raised a billion well, not a billion, but, like, eight hundred million across three years or four years, you know, to the credit of, Peter Van Camp who we brought in to run the business and and Keith Taylor and and Phil Kjellin, who was the CFO at the time. I mean, we did some really crazy financial deals.

And started an extremely aggressive build out with Bechtel Corporation to build out all these. You had you paid someone to build the data centers. Yeah. Yeah.

I mean, we he came up with a lot of original concepts, and we designed them. Had our own internal design team, Even hire John Pedro. Yes. See work for equity.

So yeah. Well, I remember that when things got tough, they stopped putting game rooms in the, you know, too. So Yeah. I think, you know, so I I I remember seeing you.

About the time that I knew that I couldn't stay because I was so I was so stressed out. I mean, So so first of all, you had the Nasdaq crash. And if and if having your entire net worth and the company's net worth wiped away, in a course of six months while your company's locked up from an IPO.

I mean, that that'll do it to anybody, but then throw on top of that that the entire community of customers were going out of business. Yep. And, Akamai, we discovered that, you know, Like, we were watching DS zero DSO day sales outstanding, and it wasn't that our customers had stopped paying, which is what we thought. I remember when Tim Weller was a CFO came in and he realized that our customers didn't exist anymore.

Like, half our customers just didn't exist. It's it's crazy. The amount of equipment that would go abandoned inside the Equinix data centers. And and so we had a bunch of construction projects around the world that were like one quarter away done, and we realized that the only way to survive as a company was to pull out of those twenty year lease agreements and stop construction and and, you know, lay off three quarters of our team.

And this is like all Probably two or three weeks after nine eleven?

So so you know, this is so hyper growth during that initial phase where, you know, I I I learned what a sales marketing kickoff was, you know, and yet, you know, and and and watching the contracts come in and It was like Cisco in the nineties when, you know, the fax machine was out of paper and people would you know, line out the door to shove checks in your pocket, you know, because it was it was it was a great, spoiling of engineers around the world because that was when we decided it was okay to spend a quarter million dollars on a on a box at the core of your network. You know, and I and and I and I think before that point, ISPss would do that to a certain extent you know, the the, you know, BFRs and whatever. But the but there was this sort of limitless bank account that existed on the part of our customers.

And sitting in the middle, you got to play with all this really fun stuff.

But then when it ended, and I had to basically cut the entire R and D team.

And we had some great guys, you know, Ted Hardy and, you know, Sean Donnlin and and, you know, Ian Cooper was was was there. And, you know, we had Dwayne Wessels working on casting, you know, back then. And we had this, like, crazy great fun you know, a team that all had to go at the same time.

And I remember seeing you because I think it was around the time Richard Clark Mhmm. Yes. Was running the cyber security.

What was he running? The What was the that it was the so he was the cybersecurity advisor for multiple presidents.

And So he got us involved when I was at Akamai with the National Communication System NCS before DHS, you know, was formed. And so, I think I went to the White House for something for some because everything was stressful. We were, like, under attack. We were at war. We were you know, everything bad was happening.

And, and I was on the road all the time. And I remember, like, I had pulled an all night or preparing for something. So I don't remember what government thing. And I'm waiting outside some office, and I hear your laugh. From behind the door.

Right? And you came out all, like, you know, tell me with I think it was Richard Clark, And I'm like, wow, Akamae got here first.

I can't win. There's no way I can win this. He's already here. He already is telling people that that, peering is overrated. Right?

And, I've always been a big fan of I know, but we were both you were you were correctly advising that we needed to to revise our our infrastructure in the sense that So that was the thing that I did, which is ironic because someone just asked me to Jobe Snyder's just asked us at Kentech if we'll support, SBGP. I think it's now called Secure BGP. My god, the nineties once it's protocols back. But we actually have fast computers so we can do path validation.

So for Richard Clark, I wrote something, which is like, We should filter the source addresses of our packets and we should secure the routing and we should use the government purchasing power, which lasted like two minutes until the lobbyists heard about it. And, you know, I was like, everyone's worried about the the authoritative name surface. That's not the problem. It's on any cast did recursive name servers, and we should support the infrastructure. And he he had a good heart for that.

Well, very yeah. And and And I think thanks to you called on a bunch of others.

I remember, well, and and Vince SURf and and Kathy Aronson, And, gosh, I I just remember sitting around a table being fully intimidating, intimidated, And realizing that, I just I I kinda used up my resilience.

You know what I mean? Yeah. Like, like there was a a certain sense of responsibility, and I think you probably felt the same thing where you know, we had kinda had fun building this thing, and I've been this really fun sort of rise.

And And you know what? This was pretty serious stuff.

And, and my, you know, I had three little kids.

That I wasn't seeing ever because I was traveling all the time, and I was, you know, in DC, half a time because remember, I lived in San Francisco back then. And then then right around that point, I moved to rural New York to try and escape.

That's pretty much when I, like, I think I went to my last ripe and my last Nanog for, like, a decade. It was right around that time.

And I and I was planning on getting a teaching certificate and becoming a public school teacher and just disappearing entirely. Interesting.

Yeah. That didn't work out as planned. But but, yeah, like, I remember, like, at, like, like, at that moment, I I'll never forget hearing your laugh and thinking, no, it's gotta be obvious on the other side of that door.

And it was It's funny because My brother, no, has a distinctive laugh. Also, you can tell where wherever he is on a floor.

And now he's running, as he's moved from running network, I think. Acamai to, well, cloud, but they call it compute because they partner with cloud providers, but after buying the node. But it was something I spent a long time at Akamai not getting them to in the startup cloud. But, I mean, I I I think there's a couple things that you know, I've definitely felt, imposter syndrome. And with some of the same people. Right? I mean, I was not a fan of the maps RBL, but my interview for, above it was, for various reasons writing a BGP demon.

To do optimized routing. There's some other stuff that didn't exist.

And then Dave Rand gave it to Paul Vixie who, like, Yeah. After he got it, sent me, like, five one line corrections to my three thousand line BGP demon. And, you know, to make it better, fix a bug, and make it better. And I was like, wow, you know, I'm not worthy.

That someone was like, Ali, who writes a PGP demand for an interview? You know, like, you know, like, you know, and by the way, let me just can we just stop right there for a second? You wrote what? I mean, look, I A lot of people have.

It's a it's a time honored way to take the internet down, mister, write your own BGP demand. It it's well, that's true. You know, at this next level. And so it's all in context.

Right? I mean, so, so, you know, you were I I'm gonna say, like, the forest company, are you saying you're the forest company, the internet? You just happen to be there when all these things happened around you? I know.

It's a little bit like that. I mean, I I went to tell my It's an Equinix.

Equinix, you know, in a lot of what is still there today, you know, in great culture is because of you and build other people that, you know, yeah, weren't there, you know, decided not for various reasons to be there for, you know, the whole stay. But, you know, made the mark. On the other hand, what I hear you saying that's to put words in your mouth is you do have to pay attention to what's in your head and not, you know, sometimes you know, if you need to change, you need to change. So I I think so.

And I and I think to some extent, I I recognized how big you know, I mean, now Equinix is a Fortune five hundred company. I mean, you know, it's a very different universe. It it's, I think, known by its employees to be a wonderful place to work. And, and they treat their people extremely well.

They've sort of come back from sort of a position of, you know, sort of too much strength to one of a little more humility despite their size, and and I think have, done a lot of soul searching as a business over the last fifteen, twenty years.

And and really is it's a different place and and a great place, but my strength was in the people part and the and the sort of the growth early, early stage of a business where you have to kind of accept a certain measure a more more risk and more sort of product development, you know, trials and failures.

And, and a certain amount of, mentoring. So so part of what I love to do, and I was so grateful that LA Avery gave me the opportunity to do this, you know, at Pax and then later as a co founder, I would find people who were phenomenally good at something way better than I was.

And I would say, hey. I'll I'll take care of you and make you VP of something.

But I have one request, and that is that you that you allow me to ask you stupid questions in front of your team so that I can learn.

And I love that. And then there become a point where that was, like, actually interfere to say foundational questions, not Well, I mean, I I always sort of presented it on my sleeve. Like, hey, you know, I you know, a classic example is, you know, I I manage coders all the time now. You know, like, I'm a lot of what I do is is small. Right? And so I end up contracting coders or hiring them directly.

And, I mean, I learned to code in college.

Right. I have no miles whatsoever.

And so I'll be like, hey, so I realized in the public Slack channel, you don't like me asking you stupid questions about syntax.

But do you mind if I ask, because I because I I love to do code review, and I love to learn about code because I have a a nerdy curiosity on how it all works or frameworks or Right. What have you. And so I I've I've slowly teach myself to code, you know, in the background in private. But if you wanna work with me, Sorry. You have to be willing to to to deal with. I found technologists to be better than doctors. I I've offered to pay a lot of doctors to, like, my questions.

And, you know, they're but but that's to say the doctors are really busy, you know, and multitask. That's so hard. So are developers in those mirrors. But I've been more frustrated.

And my father who is a doctor is like, I don't know. I just do what they tell me. It's like, well, I'm sorry. That's not that's not me.

I need to understand. If I don't understand, I can't. You know, that's always been my problem.

Like, I I was upset. I missed RSA this week because Nenon conflicted.

And that's part of what I do at a trade show is go around and try to figure out I'd I see what value is delivered. I understand the marketing. If I don't know what it actually does, I can't reason about it. That's right.

You know, the body is like the internet. You know, it's amazing that it works at all. Much less. It's not amazing that it breaks sometimes.

Amazing that it works at all. So, Yeah. Well, and all of the things that go wrong with the body, also go wrong with the internet.

I was that's that's also true. And it's a complex distributed system with ingotitor dependencies and the symptom that you're seeing is maybe not the organ or the thing that that that is the actual problem, you know, underneath.

But I remember thinking when you, you know, you had left, and I I don't know if it was a year or whatever. I mean, you popped up as Cio dig. And I remember someone I went to Hebrew school with. It started CD Now with his brother. Oh. And you know, sold it. And I remember selling sending a note to my ISP saying, I guess the real money's in content, not infrastructure.

And and so it was like, Is it though? But is it though?

I don't know. Well, you know, the jury's out. We'll have to wait and, tally up. But, I that was that was so much fun. Avi, I cannot I cannot tell you how how luxurious it is. To be a consumer of internet services versus a manager of them.

I gave one talk at Nanogg as the CEO of Dig.

Where the whole point of it was, wow. It looks different from over here. It looks so different.

And I I got to play with things that I never thought I would get to play with because I was finally, you know, building a a presence. And, you know, I mean, we had our versions of I I don't know what you call product innovation or, you know, web two dot o stuff that we were doing that was interesting. I was really obsessed with you know, egalitarian, you know, democratization. I mean, after all, I mean, neutrality, I was, as you know me. I mean, I was super big on this whole idea that, you know, conflicts of interest ruin ecosystems and And so, you know They're never that simple, you know.

You know, net neutrality. It's like, okay, well. Yeah. Need to have equal access that maybe the government paid for you to build your network.

So it's not really yours. On the other hand, if you can't filter DDoS, then the internet doesn't work. And You know, what is free speech was a little different in the nineties before people were doxxing and, you know, sending killers to people's houses. And so, like, it's you know, as much as you wanna be purest, the world has gotten definitely a lot more grayscale.

Well, this is right. This is right. You're you're a hundred percent correct. It's not simple.

And the complexity of of, you know, taking a vision you know, whether it be an architecture for a network or or a, or even a business model. Right? Any of that complexity, that you introduced to it, your vision has to compromise. It it just it it is a nature of how it works. Right? There's a purest sort of view, and there's components of it that compromise. And it's that act of introducing complexity to your simple vision.

That sort of that's kind of that is the the the the challenge, I think, the thing that sort of sucks the resilience out of you. Right? And so if you can manage to process that, there's this one time at dig when, I don't know if you would remember this, but, remember HD DVD?

Yeah. And so there was, like, Blu Ray was fighting for dominance over HD DVD. And, some users remember, so just for your for your viewers, so Dig was a was a website where you would submit a story in the form of, you could put a thumbnail on it, a title that you would write, a link and a description if you wanted to put a description. Right? It would pull metadata if it existed. Remember this is, like, two thousand and eight or two thousand and seven. So, you know, There's only so much metadata.

And then you'd submit this to, sort of a pooling area called upcoming, and people would would see that and vote on it by digging. And after a certain algorithm filter would would pass, it would go to the front page of the website And then because of the attention the front page got, it would be like a fire hose of traffic to whatever got there. So there's a huge competition. There's this thing called the washed out effect before that.

That's right. And then it became big. Yeah. That's exactly right. And, so what happened was you know, dig attracted a lot of, you know, network, people, like nerds, like me, hackers, people who love Kevin Rose and his sort of, like, hacker TV show and, and, and a whole lot of those kinds of folks.

And so it had a very technical bent to it. Although it was also, you know, there'd be links that we're completely sensational to. Right?

But one day somebody posted the encryption key for HD DVD. Right.

And when it this is, you know, after DM you know, DMCA, you know, type, you know, take down notices and and chilling effects was out there.

In the e f f. Need to use that provider at the time. So, you know, I had to deal with that. So you know that there is, like, a lot of complexity and and and debate over who owned the content and whether or not it should be pulled down, like, who can, like, if if if any Someone was just someone was just taken off of Google last week. I forgot to go, you know, by a DMCA complaint. It was back in, like, a day. But, you know, We were new to this, and we were popular.

And someone posted it, and it got so many votes. It went to the homepage.

So then we get the take down notice, and I meet with my my, my buddy, Kevin, about what to do. And we were like, oh, well, I guess we'll just take it down.

You know, it was sort of already lower and lower and lower and then it was off the homepage anyway. So we took it down, and the users didn't like that. So they reposted it again.

And this process started to escalate. So finally, they were posting it, like, once every two seconds faster than we could manually take it down. So we wrote some code to take it down. Right?

To filter for the To filter that. And then they started recording themselves on video playing guitar singing the code. So you couldn't filter it, right, programmatically. And it just started escalating.

It got to the point where they took down our website.

Effectively a DDoS attack. Right.

And then we posted on our blog the tie the title of the blog was the code from HD DVD, and we're like, you're right. It doesn't belong to us.

It belongs to you. Right?

And that was, like, a big awareness when you talk about that when I talk about the resilience being, you know, you you think that a simple idea like neutrality can be compromised. And then to some extent, it can be. But when you introduce some of these things, they also become religious.

Emergency. Yes. That's the word I was looking for. And, you know, there's a there's a science fiction series Enders game. And I don't know if you've read the book. There's a movie. But in the book, these two really bright kids become lock and demonstrations.

And there's this idea of this worldwide thing. But even there, you know, it's like there's a contrast because there's the theory that was not like he's met at the time or dig or anything or read it or anything of like rational discourse, like, you know, Socrates and, you know, arguing with people. And but it wasn't it wasn't you know, but that we've never seen that. On the other hand, the idea of, like, people smart enough to actually influence the masses and drive behavior. We're seeing that with influences. So That's right. It's it's a pretty, you know, as a company, whereas as not that kind of, you know, you have to have the sense of what is, and sometimes, yeah, it can get away from you, you know.

And it's sort of an evolution of the anarchy or the, you know, the internet the internet had the advantage of being an unregulated and anarchistic entity where I mean, you you would sometimes get de facto standardization way ahead of I ETF, you know, way ahead of, you know, John and RFC saying, you know, there was some kind of, official way of doing something because we just didn't have time to wait. Right? It was growing that fast.

It's funny because on the sidelines while I'm not, you know, operating network companies.

I am on the board of Megaport, and I And I do, of course, you you know I'm a network geek, and I always will be. So I'm, like, watching with amusement ethernet.

Oh, and these questions we argued about. What happens if there's two hundred milliseconds between peers? Do bad things happen? It was a question we had in the nineties, and now we know, you know. It is a computer science problem. Right? And you and you introduce real life and randomness and anarchy into these computer science models and things sometimes happen that you don't anticipate.

Well, I think that Cool.

It is.

And also, it's something that that that I'm gonna say good startups in my opinion learned to embrace at a micro level is, like, hey, we tried this and it didn't work.

Now it can be frustrating. We tried at a macro level and it doesn't work. But some of the best some great companies that that have created a lot of value. I'm not just saying economically, but for users, came from things that didn't work.

You know, Slack, I don't really understand. It just looks like you've checked. It doesn't crash. But clearly, people like it, and that came from a gaming or or, you know, often it was like a whole company that didn't work and then that taught the lessons.

You know, the biggest one that I struggle with is is focused because you want you have customers that want stuff. You wanna build it, but then, you know, that becomes difficult. Or you know, really what you see is when you're building a a company, you need to have you try to engineer what you want that to be, and most companies just become emergently copies of, you know, the people, the first employees, you know, and how they operate.

And that's been a really interesting time for people, you know, growing through COVID, where stuff that was undocumented and emergent and understood, you know, become and some companies designed for that, but very few before COVID, we're like, we are removed first. We don't have hybrid meetings. We're very thoughtful about this. And then, you know, like, actually letting people talk over the internet.

Thank god, works really well, but the human dynamics of it, you know, obviously is a little tricky. Oh, I I agree. A hundred percent. I mean, I I launched my most recent business in September of twenty twenty.

What does that mean? It's it's Scorebit, which is the the the the alternative vault. I I actually planned on launching it in March of twenty twenty. Mhmm.

You know, we have some, like, fun investors like Matthew Prince and You know? And, like, we have, like, a bunch of network guys who are involved in this tiny by the way, it's a small business. It's not an venture backed business. It's, you know, say small business.

Not by the way, I mean, I don't view lifestyle business as a negative thing. I know in Silicon Valley, it's viewed as a negative word, but, you know, I don't Yeah. I mean, this is I'm definitely building this. Let me put it in this way.

In certain markets and certain products, if you don't have a big pool of capital, to very quickly land grab and establish a first mover advantage, you can't succeed. And I'm not saying that that it's always good to do that. But, a lot of times, it makes a lot of sense.

In my, you know, I I decided to go after something retro and build a a network infrastructure for it largely because I knew that no one else was crazy enough to try it it would require years of R and D to make work. And then once it worked, my audience was this fascinatingly diverse international crowd. Of largely high net worth, but not all. It's true.

It's true because pinball tends to be Right. You know, of course, the Brutade started growing and all these arcades and stuff. The new employees have to read Armada and ready player one to get the eighties context. Oh, if if they haven't read it already, they're not allowed in the door.

But I but I'll tell you. It is interesting.

If you look at the the cast of characters, if we had the the tops playing cards for for Nano or whatever, and you and you just sort of dealt out a random ten of from your deck of cards, I guarantee most of those people would probably either own or regularly play at pinball machine.

Yeah. For whatever I was more dig dug to relax and Mhmm. Galaxian And I I preferred writing stuff than to play. And I think Zack's on on the, you know, it was like it's cool through Deepgram, but I didn't get into pinball for whatever reason. So And and I didn't get into it as much either.

I was I don't know about you, but the first time a networked game became available. The first one I remember that I played was there was a game called Spector seven on the Mac. And it was sort of battle zone, but multiplayer battle zone.

And it was, like, nineteen ninety four Apple Talk. I remember I ran meta verses.

Multi multi user dungeon games. But that was how I learned IPC and just distributed processing and all that. I was a it was an eighty by twenty four at Medifers. So, you know I mean, isn't and I think and I don't know how many of us fit this model.

But from a very young age, the the thing that kept me dialing that modem over and over again. And but it was, yeah, the connectivity with other people was just I I was a lot of ham. A lot of people that were hams that went into, you know, started ICs and stuff like that. I think that I do I do have fondness for gauntlet.

It needs more quarters badly. You know, it wasn't network, but you could play with your friends. So that was It there is But there is an interesting question, which is if you went back in time to nineteen eighty four or something or eighty two and he walked into an arcade and Zaxon was connected to a Zaxon somewhere else. Wow.

Yeah. That would have been the shit.

You know, someone else's, fader. Revotron was too intense to to to have any third access of other people you know, participating. But, yeah, that's true. Or even battle zone, you're right. You know?

I mean, I just feel like there's a there is a a nicely contained limited network problem that I've that I tried to solve. So on one hand, we had to have the latencies of an MMO.

Right. We're not doing really head to head because we don't really have to. It's a turn based world, but but the visualizations and the and the response time had to have MMO latencies.

And yet we had the volume that was so low that it's, you know, from a financial standpoint, you can't, you know, and I'm trying to buy eight hundred thousand servers and distribute them around the world. And yeah. So we're doing it serverless, the whole thing.

The whole thing. It is so much fun to do it this way. I mean, it's inefficient in some ways, but But I gotta say it is, like, thanks to folks like you who did all the hard SaaS work over the last ten years, we don't have to do anything anymore. I did give the Equinix just a few days ago because our force ten switch in our one rack that me and a bunch of friends still operate as V1 decided to kick the bucket.

I can hook you up with a user risk, that'll probably better than running your, of course. You know, I and we we should have shut this off long ago, but we we still host net beer. Remember tasting lime? Yeah.

It's still operational.

And, I had to bring Jeff, and it was Jeff Rizzo, and it was so funny because we were, like, walking to the data center and we hadn't been to, network has Yes. We run is basically in LA from, you know, if you're here, you know, it's like so far south in San Jose. I'm like, a resort. Yes.

But we went from a place where it was an apple orchard, like, the last time we were there. And now it's three story buildings surrounding us. It was really weird. Yeah.

But, yeah, I I do occasionally go to a data center, but But again, I I hopefully, it's all turnkey now. I could just press a button. That's right. It's all magic.

We only I mean, I can't say we only run our own infrastructure because of the economics because, you know, it's a nerding thing that I like. But we do, you know, the reason that it works because every time, you know, half the people that we hire or, like, we're doing what? We're making a cloud, aren't we a SaaS company? Why are we making a cloud?

It's like, look at all these other SaaS companies that make cloud because there are customers, and they use cloud, like we do. But not for always on stuff because that's really expensive.

But, yeah, you know, Idic, infinite diversity and infinite combinations. That's problem is that is enterprise cloud architecture, but that's that's that's a story for another time. So if you could go back Mhmm. And tell, tell young, young Jay maybe before Netcom.

Or whatever age, a couple things to, to think about do put into con you know, lessons to put into context. Do you have any idea what they would be?

Well, the first thing I would say to young Jay is run.

Go teach.

You you do end up doing what you love in the long run in whatever job it is that you end up, you know, paying the bills with.

You know, if you're if you love to teach, you'll end up being a teacher. If you love to, you know, create new things, But if I could go back and talk to Jay, I I I would probably tell him that you don't have to be so angry.

I I spent the Do you take it so personally or be so angry? It's it's really both because the, the problem is, and and I I I think a lot of investor types try and paint this picture that you need to be somewhat detached in order to be successful.

And, and but what really happens is the most successful founders are the ones that literally can't do that.

Their their people that they work with are their family.

They take every customer negative thing totally personally.

And, you know, I had not developed the ability, and I still have not. Fully to to really roll with that as a young entrepreneur. I was not prepared for what that would do to my life and my family and everything else. And I and I'm I'm proud that I survived it, but if I could give my former self some lessons, it would be around understanding that balance.

Even if I couldn't control it, just understanding it, because remember, you're You know, you've got all these mentors and investors who are telling you, you're you shouldn't feel a certain way.

I tried that in my so Gail and I would be married thirty years, but it's our thirty first anniversary of meeting at the Chicago World Science Kitchen Convention this year.

And I think I tried once or twice saying, okay, well, we've understood what the problem we were disagreeing don't be angry. And I discovered that doesn't work. It's like, don't be ticklish. Right?

Yeah. It's it's a it's a I think that taking it personal is okay. I think my my lesson would be taking it personal is okay. But it's what you do with it and how you channel it and and your willingness to express yourself.

Right?

I think also my early work was what came out of a lot of insecurity and controlling, you know, trying to sort of control my world.

And sort of coming to peace with what you can't control is is part of that. I mean, I'm I'm fifty one now.

Have you beat? Yeah. You know, my, my, I have three kids. I have a twenty four, twenty one, and nineteen year old. My twenty four year old was born two months before we founded Equinix. Oh my god. Right?

I mean, I think that the under the subtext of that lesson is stay home.

Right? You know, you you only get one time.

And now I'm at twenty eight years, by the way. So So, you know, I don't know how Brent tolerated all those years of it, but that is that is, like, the very like, if there's one shining thing about that.

It's the technical details don't really matter so much. You really just need to learn how to deal with that emotion. I think that is a a great Or you must be feeling it now at Kentech for Christ.

I I think I have a slightly different problem because I I mean, I've seen CEOs that are manic people that are manic that express it And either everything is always awesome or everything is horrible, and that can be really, damaging. You know, it could be exhausting for everyone around. Now that's not me. Generally, I'm like, okay.

Well, I fucked up. Let me not fuck up the same way. Let me, you know, experiences what you get when you didn't get what you want. But what you have to remember when you start a company, and especially if you're CEO and there's hundreds of people is, the words that come out of your mouth, people think are more important than you think.

You know? So if you're just talking and thinking off the cuff, then someone's like, oh, but the CEO said this. And I even saw this at Akamai, you know, where all of a sudden, three people will be working on a project because I was like, wouldn't it be cool to do this?

You know, it's like and the manager of a company was like, what did you ask someone? Did I ask anyone to do anything? Oh, wow. But you know, that's something that people remind me. It's like, hey, you know, when you talk to people, remember.

So it's not as much and it's, you know, focus. Again, Jason Limkin, I call him the Great Limcone, you know, he's a SaaS VC.

And he hits on, you know, you could critique his content and say, well, it's so repetitive, but you look at his content, which is pointing out what, you know, they tell CEOs as you as you grow your company, which is if you are only just getting sick of hearing yourself say it, then you haven't said it enough yet. That's good. And, you know, in the early days, it's cool to be, you know, holding everything in your head and thinking about all the possibilities, but at some point, especially when you have, you know, four hundred customers and two hundred people. Like, Not everyone can hold everything in their head and shouldn't, you know, and, you know, is, so you need to carefully limit focus.

And, you know, if you share everything, you know, it can be hard for people to see the patterns and get the details. So Yeah. Your company is not a routing protocol. Exactly.

Exactly. Well, I mean, except it is. I mean, sort of, I do have a distributed state.

So, It's my post pandemic. Maybe it is.

But I've I've seen that. I've worked with some companies where I could see that not everyone has the whole picture. I'm like, oh, I do that. I should, you know, you know, make sure to, you know, enable communication.

I think that's the word that, is The thing that's most important to engineer around companies besides some of the core cultural stuff is communication, which, know, I could be a better communicator. You're talking about Matthew Prince. He's a great communicator. So Yeah.

He's fantastic. I you know, the the the challenge right now is that I grew up and and enjoyed building businesses where it didn't, you know, they they could be businesses that maybe from the outside in looked a little boring But because the chemistry and the social part of coming to the office every day was fun and the people I love to work with when I would come I would drive to work every day with Jeff Rizzo and Matt Wood, and we would we would sit in the early Equinix days and, you know, and you know, we'd have visits with folks like Martin Levy and Yep. And, you know, it's like we were all kind of like seeing each other all the time and and and riffing off each other and feeling that, that energy.

And in in a in a modern context, And and, you know, this extends to not just network infrastructure companies, but but any company. I think that we lose so much of that which means that in order to overcome it, we have to over communicate. We have to, you know, have a morning stand up when it's really not everybody knows what to do already. Like, it's not really it's more just a social check-in point I think we'll we're still figuring that out in conferences.

Now things are getting back, but still how do you have the hallway track without you know, how do you do that? And then how do you do that in a company? Or, you know, build the next ding type thing or with what Elon wants to do without without facing the eternal September of usenet or the or the, you know, the each the encryption keys of DIG or the you know, all these things that happen as emergent behavior. So, at any company, you know, if you look at Apple, you know, what happens when people don't like policy.

You get American behaviors and companies too. In fact, the more you you stand on your principles, you know, the more, you know, it's reasonable for employees to say, hey, you know, if you believe this, then we believe this. So topics for future times, but Jay. Yeah.

Thank you so much. Thank you for It was my pleasure. For sharing.

It's so great to see your face. It really is. Great to see you. We need to get you to a Danog at some point.

So you can get Bob I'll do it. I'll do it. And you and I will stand back to back, like, you know, like, and, you know, with swords, you know, as the They're blue savers. Yes.

There you go. Perfect.

Cool. Well, thanks, and thanks, thanks, Jay, and thanks the audience for this.

Got a guest?

Network AF is accepting guests for upcoming episodes. If you’d like to be on the podcast or refer a friend, reach out to networkaf@kentik.io.

About Network AF

Network AF is a journey of super-nerd proportions into the world of networking, cloud, and the internet. Avi Freedman, self-described internet plumber and podcast namesake, hosts top network engineering experts from around the world for in-depth, honest, and freewheeling banter on all-things-network — how-tos, best practices, biggest mistakes, war stories, hot takes, rants, and more.
We use cookies to deliver our services.
By using our website, you agree to the use of cookies as described in our Privacy Policy.