Kentik - Network Observability
More episodes
Network AF  |  Season 1 - Episode 5  |  November 9, 2021

Building relationships as an internet analyst with Doug Madory

Play now

 

In this week's episode, you'll hear our host Avi and guest Doug Madory's conversation around internet analysis. Doug is the Director of Internet Analysis here at Kentik, with previous experience at Oracle and Dyn in the same role. Today he shares how he got into technology and his career in the Air Force. Doug later dives into what it's like building relationships with the press and working with them as an internet analyst. You'll get to hear about some of the essential stories highlighted throughout his career, as well. Listen now!

Transcript

Hi, everybody, and welcome to the Network AF. In this episode, we talked with Doug Madore, a friend of mine who runs internet analysis at Kintech, and did that at, Oracle, Diane and Renesys.

He talks about how he got into technology, his early career in the air force, both in technology and leadership, how he got into studying the internet, and we talk a lot about building relationships with press, some of the stories that he's covered, what went well, what got, what got more coverage than he thought, what got less. And generally tips on building relationships having a great professional career and, working with the press, overall.

Hi, everyone, and welcome to network a f. Today, we're with my friend, Doug Madory. Doug. Would you like to give a brief intro?

Sure. My name is Doug Madore. I'm the director of internet analysis at Kentic, and, I've been doing internet analysis for about twelve years now, starting with a company called Renesys. It was a small group that did a lot of, BGP analysis.

Worked in the internet and telecom industry.

I started off as a guy that would just write reports that nobody wanted to write. And they doing data QA, the people didn't really wanna do. And I was happy to happy to do it. Happy to be involved.

And kinda got into it. And so then, the data QA work was actually a window into analysis. Because they're kinda related identifying, what are the problems in the data. You have to kinda understand the data of how you might make a conclusion of something that's novel.

So, I worked there eventually started, writing my own blog posts, started to, answer media calls on my own. And, eventually, I kinda took that over from, one of the Renaissance's founders, Jim Kawi, as he made his way, out of the company and started his own next company.

And, yeah, I've tried to make the most of So we were acquired by Dine, the DNS service provider, in twenty fourteen, and then Dine was acquired. And at that point, we were rebranded as Dine Research.

Then a couple of years later, Dine was acquired by Oracle, and we were rebranded as Oracle Internet intelligence.

And through all that time, Yeah. I was writing a lot of blogs, going to conferences, and, yeah, doing getting everything I could out of our data, to tell stories, either with the media or the industry.

Studying the underbelly of the networks and turning it into interesting and relevant content. There's a lot of material there. Yeah.

So How did you get into technology, you know, generally?

I guess it was in my file household, growing up. So my father, worked at IBM. We grew up in Poughkeepsie, New York, where it's the headquarters of the mainframe operations of, of IBM and my dad worked on a mainframe. He, wrote a low level code, for mainframes for thirty some years.

And, so we always had, computers around the house. I wasn't, like, a a big hacky coder guy, but there were it was just it was just something that was always there. And we, I remember my older brother and I, we both had PC juniors when those came out in the eighties. They were they were trying to market computers to kids, and we had, we would play games on those.

And, my older brother was a little more into writing writing code, and, he would copy it out of the magazine. We would get the magazines where the code isn't, like, an article, and he would try to type it out of the magazine.

Okay. You've got one. I've got how that compendiums, yeah, typing basic programs in. Yeah. So I'd help him.

He was seven years older, so I was just kinda watching him. Grabbing things for them and stuff, and then we type the whole program in, report it onto. We have wait. We always have all these ten minute cassette tapes just how you store it, which is kind of amazing.

Think about, but we had a little tape recorder plugged into the Texas instruments, computer. He type all this in, record it onto a ten minute cassette tape, then we would that would be our long term story if you wanna play that back again.

And technologies come a long way.

And then, let's see. Yeah. Later on, I decided to follow my my father how to, did they, degree in double e. So I figured I'd do the same. I seem like he I liked the stuff that he was doing and, went on to, go to, do my undergrad at University of Virginia, in engineering school there.

Cool.

And, when did you start using computers and technology professionally?

I guess my, I guess, my first job out of college. So I, we didn't have a lot of money for school. And I guess, I never considered taking out a lot of debt. So the other was, I applied for and got a RTC scholarship from, the Air Force was lucky enough to get a full ride.

At that time, you could get a scholarship that whatever the bill came to they'd pay if you did the cut the best one. They've kind of put a they've kind of capped that given that, university prices have gotten so expensive. But, anyway, so it was all expenses paid, and then I needed to go in the Air Force after that. So But that, you know, my air force time was a, was a really good formative experience.

So after graduation, then I went to San Antonio, Texas was my first duty assignment at the Infranch Wharfur Center, and I was in the, when the team of guys, gals running the, IT infrastructure for the intelligence operation. And started off as a network engineer, needed to learn Cisco gear. And, they were a well funded organization. So we would go to classes and do start our way through the Cisco certification.

I I still really like the, that that whole curriculum, the CCNA, the the, all the certifications. So I went, through that.

For about a year. And then, then nobody wanted to do, unix administration in our group. Nobody, like, thought it was they didn't understand it. They didn't wanna have anything to do with it.

I was like, windows, windows background people. Yeah. And I I was like, really? Because I, like, I'd prefer to be there.

And they're like, really? And, do you wanna be in charge of it? And I was like, yes.

So, that was everybody was happy. And so I became the head of, a solaris administration for the information warfare center.

And in undergrad, we everything was Phoenix based, coding and, all the CS stuff. So that was very familiar territory.

Anyway, it says a very technical, a technical job. A good one as a first position in the air force. Did a good job there. I got rewarded with a command position. It was completely different, and was a difficult transition to be in charge of, fifty five airmen, in a tactical unit that's, has to be able to deploy on a moment's notice.

Just completely different skill set. But, it worked out. That's cool. So like me, you like the, system and distributed system aside of the house and the networking, but what, made you choose for the dark side of of networking versus, you know, staying in or getting into the the more sysadmin.

Now we say SRE and, you know, generally distributed system side.

Oh.

I don't I can't say it was real plan. So, after so after I force. I went to, did a master's degree at Dartmouth to try to get back into that. I spent three years as in in a more leadership management role.

And I kind of decided I I like to get back into the technical space, but I've been out of there for a little while. And I thought I'd be good to, brush up and get another credential in that space. So I came back to Dartmouth to try to, get, get back into, secular science. And, when I finished there.

I'd spent a year at a defense contractor, and then, became the head of security at Dartmouth City, medical center of the hospital system up here. And, yeah, that was, interesting. And, after a couple of years, I was like, alright. I'm ready to something else.

And they would so opportunity came open with, I knew one of the founders of, of Brennan was a, social friend of my adviser at Dartmouth I'd seen him at parties at my adviser's house before. He actually was recruiting me, in years prior, and I never really took him seriously, which is kind of, too bad. But, And then, yeah, and then I, and I I, you know, wrote them out. I was like, hey.

You're you're still looking for people that I met with those guys. And I was like, this is actually fascinating. I never never knew that such a such a type of work existed. It definitely wasn't on my road map.

And then I got in, when I started at Renaissance in two thousand nine, I there's a good line, and David Carr is the late immediate critic, for the New York Times, and he wrote a memoir, and he's had a colorful troubled life, but when he cleaned himself up, he got the job at the New York Times. He says he had a immigrants love for the place. Like, he he loved it more than, than the people who were there. And, I feel like, oh, that that characterizes me, run us a subject off.

We've done a bunch of different things. Like, I'm like, I've been all over, and this is way better. This is way cooler than anything I've done before.

And, yeah, the rest is history, I guess. So you still kinda doing similar stuff. You didn't think you're gonna grow up to be a detective.

Yeah. I mean, that's that's the that's that's the that's the challenge today in trying to advise kids. Right? Like, if I could talk to myself in twenty years, like, I would have no there's no way I could have predicted that such a such a job that the internet is such a big industry that you can have, special specialties of someone who's actually just, analyzing, this thing, this, technology that we're, all dependent on.

And I guess when I say it that way, it makes sense. But, it's it was hard to yeah. I never would know. Detective is a much better way of selling it than, you know, internet janitor or network proctologist, which some of my friends think of themselves as.

It's a tactic. I need a trench coat or something and a fedora. I think it's a big spike glass. And, yeah.

I mean, it's an interesting it's interesting because we get so disconnected from all the layers of things, but a couple weeks ago, I was down hugging the servers and putting routers in and forming opinions about which which operating systems I liked and hated, and it was refreshing because I live up here, and it's nice to remember the joy of you said the immigrants love, the actually putting hands on things and getting into it and seeing things. And I guess, you know, hopefully, things will keep changing, so it'll stay fresh and exciting.

So you didn't set out to study the internet, but that was just there was already a practice around it. Kinda fell into it. Yeah. So what what does that mean from, I guess, from, output side? Like, what are you finding? What are you hoping to find? You know, what kinds of things are you are you trying to surface and and hoping to find for the kinds of analysis, you know, like, what's your beat?

Yeah. I guess, there's one that's why I've got a public facing thing that maybe people, are might be more familiar with, since there's an internet shutdown or some sort of development on the internet, If we've got data, it can take down a lot of great data, to comment on these things, then I that's a that's something I'll, keep my ear out for. And, I'll try to put us in a good position where we can make an informed comment to inform the discourse, but, that's not all my time. That doesn't that's not a full time job. So the rest of the time.

Here, at Kentic, and we we developed a lot of We were in the space of, being a vendor to do internet monitoring, around, you know, BGP and, you know, We call it, Kenty calls it synthetic measurement. We call it performance monitoring.

And so we've got a I've got a lot of history there. And so I'm kinda working with the product managers, here at Kentyc.

To try to, share with, you know, some of our experiences and, what what worked well, and developing our next generation of synthetics, products.

And then, so there's a there's a product management component to it. And then there's also a sales thing of just, lead generation. How how can you, provide material that helps, the Salesforce, convey the value of the the products, based based on the data that we've got.

So there's a there's a lot of ways that you can use an analysis function And then, you know, like, I guess data QA, I I I haven't done that much in the in the form of data QA, but that's a potential area as well. And those the skills of an analysis function can help in all those different ways via, media, getting the brand out to the press. Fixing data errors, helping sales or helping product management. Mhmm. And, so I I see myself as trying to balance, those various needs.

On the public facing side, are people interested in sort of the technology underneath the, you know, the impact about, you know, things being down, the sort of the policy, you know, towards the right side. You know, as you talk to, well, I guess, the reporters that are the proxy for what people should be interested in, you know, where are you seeing the interest, you know, in terms of what's going on with the internet the digital infrastructure, you know, the applications that connect us all. Yeah. There's there's a couple of for the public side, there's there's a couple of I guess, broad buckets.

One is, news that's accessible to the masses. It's, like, so country access going offline or something like that. That's understandable by, most people without having a CCNA or a background in, computer networking.

And so in the in those on those stories that I'm, you know, defining what a CDN is or something, to a a reporter That's, that's one bucket of broadly accessible, maybe geopolitical things often fall into that category. Things people can relate to. And then there's another one that's, you know, like, the Nanog talks, I'm not giving Nanog talks on a country going offline.

Or or you see these are gory routing, auto routing incident, autopsies or some kind of, describing, you know, what what could go wrong and what how would we, you know, propose to fix things? That's more, technical. And on and those for those blogs. Like, I kinda have, in my head, a couple different audiences.

Am I writing for a broad audience or writing for, a kind of, an analog audience? Where I don't have to define what a prefix is, and I don't need to end up and I've kinda had a point where I kinda know if I have to define what a prefix is, there's some, like, like, That that's not the story for you. Like, I I'm not even gonna use that use some of these, this vernacular of the space. So, there's kind of two, maybe just joint, areas.

And then, I guess for the more technical side, my, What I've spent a lot of time doing is, trying to understand how can we, get more insight out of all this, BGP data that's going on and, that's getting passed around and, you know, what can we learn, when it goes wrong, what, you know, what caused it, what were the impacts.

You know, that is, that's something that we because of who we were, Renisys, having all the BGP data and analytical tools that was, that was what we were well suited to answer those questions. So I got to the habit of, a lot of practice. And for those that don't know, not August, the North American Network operator group, it's, the people that, connect things together in the US, although it's an international forum, and there's similar, similar globally. And Yeah.

And I guess what say, a a Nanog audience that includes ripe. I don't wanna leave anybody out. Oh, god. A little black neck, Yeah.

Afronac. Yeah. Yeah. All the different regions. I used that. He's a in a Royal Royal Nanog.

Yeah. It's our background. That's that's where we fell into it.

No, you know, it's interesting because often people come through and are like, it's time for a new protocol. BGPA is, it's like, well, do we understand really all the activity in the BGP thing before we go design, you know, a new one? Now, obviously, with cloud and everything going on and, you know, infrastructure as code and other people's infrastructure as code. We can't, you know, there are there is innovation in new protocols, but from a internet, underlying perspective, occasionally good at clouds, big companies say, oh, we don't run BGP. We have our own protocols. It's like, I assure you. You do run BGP.

To the internet, you do run BGP.

And, yeah, there's a lot of a lot of Those things are never going away. It's BGP version four, one instance, for the world forever. How do you well, there was an there was an EGP, and there might be something else, but we probably should understand it better before we go replace it. So How do you get from BGP updates to human rights?

You know, I've seen you do, you know, a bunch of articles that you know, as you said, shutdowns or or, you know, partial shutdowns, you know, how do you go from watching VGP updates and making a conclusion about how people are are behaving and oppressing folks?

Yeah. So I guess I got, started in that space, during the herbspring. So, in January twenty eleven, when was having their protests and outside of turn off the the internet.

Let's see. I remember, Jim Cowie, are one of the rest of the founders sound a message. Hey, if anybody's around, I need help on a evening, on a, like, a Wednesday evening or I think it was, and, and so I was I answered the call and, and so then we started kindly are. Well, we we had a because we sold these tools to the telecom industry, we had a really good picture of every country. This was just this is our product of understanding, like, the from a BBB standpoint, what's the topology of every country? How's that changed second by second?

What's normal? What's different? And, so those we had all the tools start digging into this. So, as soon as it went offline, then we started going by what went on and what what was the timing of everything, what what's still up, what parts are still, connected.

And, so I was kinda racing doing as much else, handing it over to Jim, who's kinda synthesizing the sensitive blogs, talked to the press, And then, yeah, then, you know, for that, for the herbspring, we had, you know, Libya, Bahrain, Syria, like, a bunch of countries had their own events that we were, trying to do some coverage on and Yeah.

Some sort of adverse event happening to the the internet, of those countries.

And, yeah, that that was a that was a high watermark as far as, like, media attention for, for Renaissanceisys.

Clearly, we were filling a need that, that was not out there of people who could analyze this. And so they I got people who knew the internet was down know, this was getting reported, but, at some level, that, you know, there was a desire for a more technical, analysis of exactly what was taking place. And so that we kind of, evolve to kind of fill that role of, being, you know, we're not we didn't see our role as, carrying being a flag carrier for human rights. But what we would do is, we had all these technical assets. We would inform inform the discourse and make sure we've got the right facts and then You know, there's a there's an implicit thing that we're kind of showing off that we've got disability. So that's good for the company, but we can, our contribution will be just objective technical analysis, make sure everybody gets the story ready because, so then I feel like I've been kind of doing that ever since.

And then, you know, by the end of that, I remember, we I'd spotted, Syria of, I think, in June twenty eleven, it was their first shutdown. I I'd spotted it with the it's kinda we started setting up more stuff to to look for these things back in twenty eleven. It's actually Egypt went offline. We had gotten a tip before it went offline, to look for it.

But I didn't have anything at the time running that would just alert me when when these things would happen. And then Well, that's that's interesting. You gotta tips that could actually have been from someone that saw what was happening and maybe was a little bit more liberitarian than where they're Yeah. There was an engineer who reached out and it was like, we've we've gotten, we've gotten an order to stand by to shut the Internet off tomorrow.

So we had a day's notice. Mhmm. But I don't know. Sometimes you hear those things and you don't know how much faith to put into it.

And so I went, okay, like, we'll see and then, sure enough that it did end up going going down. And then, Yeah. By and so then by the end of it, we were we had I had automated things that would, you know, clue me in when this stuff is happening. And And then also by the end of it, yeah, Jim was very, very gracious as a founder to share in the limelight with me and let me take, start taking taking calls, writing writing my own stuff.

Mhmm.

Yeah. I'll always, you know, be grateful, for that. See, I would've I never would've held it against him if he always, kept it for himself. His company. It's Oh, he's a great guy. I met I met him and Andy actually doing internet measurement things in two thousand.

And then when they were starting and tried to buy it when I was at Akamai, and, it didn't, but, no. It was fun to see them do that. We actually competed for BGP sessions briefly.

Because kentech sorry, Akamai at the time.

You know, we needed BGP for figuring out where we were allowed to send traffic to our ISP partners, but we also did analytics and went and asked people for BGP sessions. So we had about a couple of years of of of asking ISPs and figuring out who could give better, and then Redesys started giving people analytics on it, which was not a business that Akamai was in, although I wanted to be. It was not, it was not something that we decided to focus on and, certainly, the network group is not gonna go build that product. So, no, this is fun to see the success there.

So Well, did you walk into some pretty mature, relationships with journalists that Jim had or, you know, how did you go about I did not, inherit really. I mean, I so Once once, once we kinda got going into the twenty eleven, then, I started to, you know, write my own stuff because I had been a little bit of a understudy for him. Like, so he would travel around. He'd like to go to, obscure places.

Get for a station site. He could do his own analysis too. I don't wanna take anything away from him, but, you know, occasionally, I'd have stuff and I can help him, where I augment. So I was ghost writing a little bit, but, eventually I had enough.

I could start doing this on my own, and then it was helpful to have two people that would write these things.

Presented conferences.

But the, yeah, the media stuff, I think, you know, once they started to get into a rhythm, of, like, how, you know, what are how to look for a story that's, newsworthy, I think I started to just develop it on my own. I mean, there may have been a few that I, I think I got handed over to me, that were previous previously talking, Jim. What was what was most helpful in, you know, building those relationships?

Because a lot of companies go down the path of hiring PR agencies and doing pitches and, you know, but Yeah. The, the rest so this is a, you know, rest is like a thirty person company. This is that's a small, probably too small to have a full time PR person, but it needed PR.

So this was our this was our solution to it since we had all this data and tools. I guess, yeah, the, Definitely, the writing, I got a lot of, good feedback on how to how to assemble, a a story how to write a a blog, I think I think I'm okay at it. Mhmm. And then, and then also with the, whether it's the media or, conference presentations.

I know we're going to early nanog, giving a giving a talk, being very nervous for it. Kinda did a crappy job, but, every you gotta start somewhere. And, Well, Jim, Jim had done a lot of them and did a great job. Jim, you know, gave great Yeah.

Right. He he was he was fine. He's tough a fact to follow, but he would, be like, you know, listen, we've got the data. You know, you're presenting on our data.

They don't have the audience doesn't have your data.

And as long as you're, you know, caveat this, like, this is what we have. This is inclusive we're making with our stuff.

You're safe. You're safe there.

I I I even, you know, even when we put out a a tweet of, regarding something that can take, as as per can take data. Like, we're Like, I I wanna add that caveat, both for where it's coming from, but also just be like, you know, I can't rule out that we aren't, subject to biases and our data sources that's that's kind of implicit map caveat, but, that's that's gonna be true for anybody. But, anyway, but I guess, as far as, like, you talk to the media like, you know, the the lessons are that, obviously, you have to start up with. You gotta have something to say. You gotta have some sort of, something good, that If you don't have that, then you don't, but but but that's not the only thing you need. You need to be, you know, very timely.

You need to kind of say a quick, be responsive, journalists or time is very important, to they they need to make a they're usually if they're with their salt, then they can size something up pretty quick, and and work pretty fast. So you need to be able to, be responsible and and be able to assemble a, concise, summary or statement about something that took place. And then if you don't know, just say you don't know and maybe make a referral, and it and it's it's good for you. It's good for you.

If you don't know, like, tell them that ASAP, don't try to lead them on. Or or we're not the best source for this, but as you said, you know Yeah. Like, we we get we we get queries about Let me see if I can get you somebody that knows this in that space, because I know enough to know that I don't know what I'm talking about in that space. So, but I do know the people, so I make referrals then, you know, that's that's good.

And they'll keep coming to you if you, knowing that you'd be a, an honest broker, if, if, if they had another question. Because sometimes they, you know, they're not gonna they're not they're not in the space. They're not gonna know what are the parameters of what are the things that you're an expert at. They're just never gonna understand that.

So, they'll just kind of, trust that you'll, once you've established some trust, and I'll trust that you'll guide them in the right direction. And I try to and try to be a good person. Yeah. Well, and thank you for your referrals last year.

When you were still at Oracle on people looking for traffic, which wasn't a data source that you had. And we might have hired you anyway, but, you know, we was certainly, appreciated. And, yeah, generally try to do that. Or if it's an investor, I think that's a good lesson, you know, generally for people in building relationships.

It's certainly it's the Silicon Valley style. Doesn't mean you have to be there to do it, but You know? What is the Silicon Valley style? The Silicon Valley style is pay it forward, help people.

Okay.

And good things will happen.

You know, in the nineties, I was just really frustrated with how crappy the documentation about routing, and specifically internet routing was. So I did all those tutorials and helped people and Still, people are like, oh, I read that tutorial, and that was super helpful. And, you know, now there's a whole industry of content marketing that people do. Which is doing good doing good for e hole purposes, which I think is probably still okay.

It's the same thing if if, you know, if I talk to the investor says we'd like to talk to someone. Maybe they're not a good fit for kintech, but I'll tell them about other startups, try to make a recommendation. And you know what? Maybe at some point in the future.

Of those other startups will find someone that isn't the right the right fit for them and, you know, refer them, to us. And, you know, Kentik is now seven years old. So some of those things that you do early on. Eventually, maybe you talk to someone.

They're in their first or second job, and all of a sudden, They pop up and they're a leader or they're, you know, you know, they're they're a good fit. They've moved through here. I've had that too. I've had, especially at running since we have inquiries somebody who's kind of, you know, fledgling researcher trying to start out and be like, has a question.

I had spent some time with them. And then years later, this is a prominent person and I've really made it. And I was like, oh, good because I'm glad, you know, these, especially in the internet space. I don't know if it's unique, to our industry.

But, if you if there's a young person who's really into it Yeah. Yeah. I remember Nick Feamster when he was starting out, super voracious trying to understand everything and, you know, work with him. I think I was at Ocuba at the time, gave him access to resource.

And then, you know, in three years, he knew more than me. Okay. But it's a it's a side effect. If you keep someone something, there tends to be this this dynamic where they will for all the forever think that you are smarter and were accomplished, even though all you did was know something they didn't yet know, like, at one point.

But, but it's a I think it's a I I I think you're an embodiment of this as well, but just, you you can You can have a lot of success being a a decent human beings in the in the space. It's not required that you, it's probably better to to do that. You can I've got, like, a bunch of, people in the space. I feel like, there's a few, that they have less of a less of a name. I tried to help them out, and then we're we're still in touch. And, now they can help me out.

Okay? Yeah. No. Well, I certainly would advise that. Also, there's different people who have different styles. And, you know, some people are are, less social than others, but, you know, it certainly works well and I think helps, helps folks and careers if that's something, you know, that they're looking for. So be helpful.

Don't over pitch, you know, guide, even if even if it's not your own, you know, thing. So you talked a little bit about talking to journalists, you know, any other tips for you know, if you get approached, you know, talking to journalists as part of building those relationships.

One other thing that we use, like, that's usually important to have that sense for is, a deadline. So is this something that's gonna take a couple of weeks, or is this something that's they're trying to file a story this afternoon and and how you're gonna treat that are completely different.

The different levels of info and, Yeah. Like, everything. Everything. Like, if it's, if it's a afternoon, then this is the conversation, and there'll be no more conversation.

The one what the next few sentences are what going in the story or nothing. Right. So it'd be good to know that. And then if it's in a couple of weeks and, like, wow, then it's almost like you have infinite time of just to back check and develop, you know, history of something, it's a much more, a a bigger thing, but, that's, sometimes I overlook.

I've I've attended, over the course of renaissance and dying, not so much Oracle, some, media relations coaching, and it's been kind of funny because I've done, you know, like literally hundreds and hundreds of these, and so then we've had a couple where I felt like I probably had more, more experience than the person giving the coaching, but I'll I'll listen. And then I'd and I'd say, like, well, here's all these points You ought to mention the phone was, like, the time, the deadline thing. Like, everything you said is great. You know, one of the things that they'll they'll say, there's there is no off the record.

Everything you say, Micah, and it's just true, but you do wanna I think, one one thing, once you've established some trust with people, not gonna burn you.

Because if they burn you, then that's the end of this relationship, and it's mutually beneficial. So they're they're not out to get you, and that was, that was definitely the The mindset in Oracle was, we're journalists are out to try to, stump you and make you say something stupid or, I think that there's you know, investigative journalism, there's people that again it helps with context. If if if someone's talking to you because someone told them to talk to you, then that's that's a middle level of maybe you know, get better watch what you say. But that's why I'm asking about relationships because, yeah, if you have a relationship, then it's sort of the same with everybody. It doesn't mean You don't have to keep, you know, be aware of where everyone is, but, it's another reason why it's important to treat people, you know, respectfully and give you know, and, you can get. Personally, I just I am in awe of how It's like the narrative journalists.

You know, what they do with the facts and the explanation and, you know, weave a story. So You know, it's like, how do you give them things that, and contacts that help with that?

I I still I don't understand how what what comes out is so great often. You know, especially as you said, for the masses.

You know, so that's that's, Yes. There's also also another thing I'd learned, like we, you know, if you if you can put it in terms of this is the the hottest or the wettest, in the last decade or ever or the the something, some sort of superlative to put it in context, that's, that'll, that's something that they can kinda, latch onto, then that's, I, for example, I remember in, November twenty nineteen.

We were covering the internet shutdown in Iran, and we were We have been a long scooped by, net blocks, a dedicated outfit, to covering this. And, but, one thing I noticed, they kinda missed that this is, like, the biggest outage in Iran, like, ever. And that's what the headlines should be. And so we put put the if I think I put that on a tweet, I could I could tweet, but I couldn't vlog or talk to the person.

Somehow, I was allowed to that was my loop hole. I could I could tweet reporters could see the tweet, and they could quote the tweet. And there was a a couple headlines, like, you know, Oracle says it's the the, biggest, outage ever and and Iran. It doesn't notify you.

Yeah.

Not not on that day. I did that a fair bit of Akamai, the stuff that like, I think it'll fire me probably not, you know. So you, you know, when you when you pitch or you work on these stories, with journalists.

Do you have any things you spent a lot of time on and thought were really important that didn't get much reception or, you know, really in the reverse things that you didn't think were a big deal that wound up you know, blowing up any, you know, what what's been most successful? Yeah. Definitely the, the, on the case of one that I didn't expect to be big ended up being the biggest thing I've ever done was, the North Korean outage in December fourteen.

So to set the stage, this was, Sony pictures was put putting together this movie, the interview of a a comedic you know, assassination attempt against the North Korean leader.

I still haven't seen the movie.

And, I the North Greensboro upset about this, the hack Sony, and all this, you know, secrets came out of Sony that were, there's all these bombshells that came out and big news story. And then, president Obama at the time, you know, made some speech saying, at the time of our choosing, we're gonna get back, you know, in so many words. I'm, and we're gonna we have some sort of response.

And, you know, in that around that time, I've been working with, a researcher named Martin Williams, who is the author of North Korea tech blog, any recorder IDG, for a long time. And, he is a, probably, one of the foremost experts on open source tech information about North Korea. So he and I would always would often trade notes and collaborate on stuff. And so we were we were already kinda having a conversation.

Like, is there any kind of Another way to look at, you know, this tiny little internet of North Korea, I had theories. I wanted to run down with him and While we're having this conversation, then, I'm starting to get little alerts those alerts that I set up after Egypt and and there we're spraying So three and a half years or more, almost four years earlier, we're, we're, pinging me saying, you know, North Korea is very you know, the the routes are unstable. So BGP is a report by exception protocol. Everything's fine.

No. Things changing. There's no messages. Like, this, the way it should work.

If, the routes could stay up, but if you're receiving a lot of messages, it can. It's not always, but it can be an indication to some sort of problem, because otherwise, There should be no.

If you get a flood of messages on a prefix, something's going on. It's having it's having some sort of issue. So we were getting, alerts that that the routes were unstable. They were up, but they were unstable in, North Korea.

And so then we were talking. I was like, I wonder what's going on there. I don't know. We'll check check it out in the morning.

And then in the morning, I I come into work.

The four routes are down. And, I sent something over to, Martin again. And I was like, hey, It looks like North Korea is down. I was like, I I made some comment.

Like, I wouldn't be surprised if they're under some sort of attack or something, and, and then I I sent that off. And he said, why don't I publish that? I don't really go for it. And, and so then I, I went into a meeting for, like, two hours And then I came out and, Martin had published a story, and it was huge.

It was huge. And my phone was was just blowing up, and my, like, my email inbox was filling up. And, basically, North Street had been down, and, and that was I think I answered Nicole Pearl from New York Times, and I was tried to, in my mind, try to triage based on my my knowledge of the hierarchy of, of the media landscape. So if it's if it's TV news, that's, like, if it's national TV news, that's that's probably number one.

NPR is up there pretty high. Top shelf newspaper newspapers in, New York Times Wall Street Journal and, broadcast over print. While you think about it, like, they have they've only gotten seconds to, like, every second, is very costly to put something on the air. This is my this is Doug's Doug's philosophy.

I don't know. This is, but right. So if so if I was I ended up being on, like, ABC evening news, and so that's a thirty minute broadcast. Every second is, super valuable, whereas, can New York Times can always make another story, and there's endless, you know, internet space for the in the in the print paper, maybe they they've got some balance to it, but then they're just relying on reputation and stuff.

But once you get to some sort of a analog media, it's it's hard to it's hard to break in, there. And, anyway, so then, yeah, it was just getting just inundated, that we ended up being essentially the for that day, for that new cycle, the twenty four new york news cycle, for the world, that was the biggest story that Oh. It seemed like Obama made this kind of implicit threat against North Korea, and then their internet went down. And we were the main technical source, that everyone was citing and want to ask the same questions. And, you know, that one from a technical standpoint, we're talking about four slash twenty fours were withdrawn. Is essentially this is what took place. I mean, there was there was nine hours of it kind of flapping, and then eventually just went offline.

And, that it's not super deep from a technical standpoint of what took place. So it wasn't super interesting to me, but, the timing was right that it was this It wasn't a huge investigatory challenge.

And Not really. Four slash twenty fours is like the average in the days when people had offices and and and lease lines and such, which some still do. That's like a a a a a a small law firms worth of, you know, address space.

You know Yeah. I guess, I mean, like, I bet our company anybody in this audience, the company you work for has definitely has more than this, an address space. I'm sure. And, it's a thousand, like, a thousand addresses.

Anyways, It's kind of nutty how how tiny their thing is. But, yeah, so then that night, like, I was getting, yeah, it ended up being NBC News. They had recorded an interview for the seat this morning, the following day, and that was on NPR. All things considered.

And Yeah. That was that was the biggest media thing I ever did.

And, yeah, it just goes to show that it's just a little bit timing, and, you're not in complete control, and it's hard to know what what gets legs sometimes.

And then Anything on the other side that you did work on? And Yeah. I guess, let's see.

I know that, so one thing that's been a trend in the internet shutdown space is, internet shutdowns for, for for reasons to combat student cheating, just kind of a hard little hard to understand.

Not of a country. Of a country? Or Yeah. Yeah. I think in a country offline to try to combat, cheating on exams.

And to understand this, you kinda have you have to understand. And so, I've been involved in both Iraq and Syria have done this, We'll see. Siri was doing it this year. Iraq maybe has stopped, but, that so what's happening, like, in our in the case of Iraq, basic public education goes up to, like, the sixth grade, and then you take an exam. And if you don't score high enough on an exam, that's the end of your public education. So the stakes are actually super high.

The odds are already stacked against you as a kid in in Iraq. It's, your your prospects are you've had a lot of challenges to overcome. And if you if you're without your the rest of your high school public, a public education, it's even harder So, winds up happening is because the stakes are so high that parents will resort to anything, to try to make sure their kid, gets a high mark, and there's also just understanding that everyone else is cheating. Right.

And so your kid, even if he's smart, is not gonna make it, because everyone else cheating. Well, then you need to cheat too. So that's that's kind of devolved into that. And so what ends up happening is they, there's a a moment where they, publish the exams they, this global not the global, but the, the national backbone goes down.

While they're physically everything exams, what would happen in the past is as soon as this took place, all of a sudden on social media there'd be screenshots of every question and people would be solving all questions, and the kids would be trying to memorize all the exam all the answers. And by the time they sat for an exam within a couple hours, they'd have all the, maybe they'd have it written on their arm. I don't know if, but, the an the answers will all be out. The exam's compromised.

So, you know, Iraq started doing this when, when Isis had taken over, taking over Moscow. This is back in twenty fifteen, and they started shutting the internet down for student exams, and they had so many other issues. Anyway, this was one where, like, this is a hard story to sell, and hard for people to understand. I mean, I did get end up getting some some coverage on that. And then when Syria started doing following you. I had an insider inside Siri Telecom who was, very useful in trying to help me explain, explain to me what was going on.

I knew when I was writing the one on Syria.

I knew there was gonna be no.

It'd be hard to a story. I'm just doing this because of I just maybe it's your American media relationships, you know, maybe it's just too foreign, you know, because we don't all take the SATs at the same time, not that I understand the SATs are now vanishing, but, you know, we're not you know, the the concept of the national test, you know, just being, just being foreign. So So back end, also, I guess another example would be, of I mean, I guess I did we ended up getting a little bit of coverage on that, but the one that really just dropped like a stone was, I thought the opening of Myanmar from being a a closed country to, you know, rejoining the international community and doing all these political reforms the telecom sector was a really amazing story of how they, reform their, and liberalize their telecom market, and brought in these have this open auction at the Republic process.

They had a a couple of winners. They paid top dollar for the rights, ten year brights to, operate mobile operator, Sarah. And, and they had this, internet boom from, like, zero to sixty and just overnight, practically. It's an example that I later used, to talk about what could happen in Syria.

Obviously, Myanmar's gone a little bit sideways this year. So that's that's too bad, but a, so that I was, I had been in touch with a couple of reporters at Myanmar. We were kinda documenting us these things are happening. You know, there'd be a couple of outages.

People would suspect were government related. I don't know that we've I can't remember if there was, one that was a government directed outage versus just things breaking. But when Telenor came online, you know, we could see that in routing, and that was kind of a little milestone in there. But Meanmar was a tough sell, as far as, like, getting any coverage.

And then after a little while, I was like, alright. Well, I'm just doing this because I think just genius. Right? I'm just writing for Doug, at this point.

I think it's an important story. I'd like to have this, you know, documented and so It's just great. I'll keep doing that, you know. Yeah.

Because then later, then later, then, you know, and, with the had the coup that began on the first of February this year, and then like, everything. They had, total shutdowns, mobile networks, censorship. They had to even have a a beach fee leak, a la, the pack stand YouTube. I think they have they leaked out a a Twitter prefix, the, disrupted Twitter in South Asia for a little bit.

Just all the above. Everything we've seen all took place within a couple months in Myanmar. And then, I had a a bit of background, at that time, it's having, you know, known the landscape, and I still had a few contacts, in the in the country to help, tell, you know, whatever I could help inform that story.

So, yeah, that was definitely interesting. Things don't always happen exactly as you predict, but this no way to find out, but to actually do it. I've had experience with, Pretty cool experiences with different, as I mentioned, sort of people narrative journalists, people that are trying to tell a story, George Gilder, as he was doing his telecosm thing, had a friend and, you know, he actually joined us for dinner and then all these observations and stuff and and some quotes.

I was trying to It's a skill. It's a skill for sure. PGP. It was actually PGP, and, you know, it was just the time that, that, people are saying, oh, the internet's gonna die.

I said, I was trying to explain, like, log base scaling. Like, we didn't have a route for every IP address. And he sort of made fun of me for being a nerd, but a nice way. It was good.

And Simpson Garfinkel did this article on on Haven Co in sealand, and I was like, welcome to sealand. Now, bugger off. But it was, like, good. And, and then Charles Fishman for the Atlantic.

And it's like, I talk with him. And then at the other end is this, like, awesomeness that is, like, did I say that?

If you had any, you know, experiences with, you know, where you gave an interview and you're happy with the result, but it wasn't exactly You know, like, what was the best or most surprising interview, you know, you've had? Well, let's see. A few little I'll I'm gonna check up a little ones now. I'm gonna give you bigger example, but I know that I had, I did have one. I will mention the publication. I had one thing. I think it was, Syria or some other country.

They modified my quote to make it more, efficient. Yeah. And, I wrote back, and I was, like, it was, like, it was basically something along. I've never seen anything like this or something like that, but that's not true. I was just I've I have seen this something like this, but this is just happening today.

And I wrote back. And I was like, I was like, I that's not what I said. I mean, we were it was the email based, you know, conversations. We could like, it wasn't like I, I was like, I, I think it's right here. And the and the guy was like, there he was like, oh, the editor, you know, punched it up a little. I was like, that makes me sound like I don't know what I'm talking about, here. So that, I've had a couple of, yeah, then I was kinda leery after that with, with that particular individual.

I have another one that was a there was a style. There was a it was a publication during the Syrian Civil War called Syria deeply that would was just dedicated to the, series of war, with just any fact, they kinda, I think, yeah, there was, like, another concept journalist Uh-huh. Journalism concept or just try to bring every every stitch of everything into this one source. And I gave an interview there, and I, I had known the journalist from, she was at a previous publication.

And this was just surprising when it got printed, it was, It was basically just a verbatim, of our, transcript of our conversation was the with the interview, it was usually kinda cleaning up a little bit. And so it was like everything warts and all, like, um, s and us, just, like, it was it read, like, it was literally just the words, like, they just had a, to put it into a piece of software and just poop this thing out and I think, like, I would have maybe been a little care more careful or something if now I said it, that was a surprise. But, On that positive side, though, I I am in twenty I think twenty fifteen, I got asked to, join this, investigative journalist team from New York Times.

So it was going into, trying to study online gambling at the time. And, that was one where, we started, they asked for some assistance. They're basically trying to track down these, you know, where were these things getting hosted? At the time, the law has been changed.

But at the time, it was illegal to do sports betting, inside of the US, unless you're in Vegas or at, like, Atlantic City, there's a couple carve outs, but you couldn't just online place bets. Having said that, everybody's doing So then, in case we just move offshore.

And, but, the understanding was that all these things had just moved offshore just being outside the borders of the United States, but in fact, it really was. And it was still in the US, using a couple of, you know, a a variety of things were happening.

So I was basically tracing, you know, out, like, where are these websites served up from and, and trying to explain how the internet works. And, they actually had me down, to Manhattan to the darn times building and I've had a conference room. I was it's definitely a a life highlight, with their, like, whiteboards and So I had a so I had a whiteboard and a afternoon and a co a conference room in the New York Times, a building and, had the the investigative team, a couple guys from sports journalism, and then a couple folks from, front lines, which was ended up doing a documentary on it, which I got the interview done.

Anyway, in the end, it was, like, I'm amazing. Like, I'm a huge fan of, the front line, and it usually deals with, like, really, really grave, important things like that. I think the episode before me was, like, about infiltrating Isis. And the next thing I was, like, you know, something about Yeah.

They're all, like, really big, the biggest issues. And then there's, like, yeah, online gambling was, was in there too.

But that was a highlight. So I'm drawing on the, dryer's board. Like, how how's how's why do why would I know anything here? How do we know does BGP work and how would I know what's going to happen and how does DNS work and how would I know, and and could I be wrong?

Like, what what what kind of went through all that? Like, I wanna just trying to suss all that stuff out. And, after I left one of the one of the guys on the team was like, you know, the guy in the back was, and I was like, no. I I didn't, catch his name.

And he said, oh, older guy in the back of it. I was like, no. I don't know. He's like, that's a little Bergman.

And, so you know, who that is? And I was like, no, I don't. And he said, well, he's the executive producer of Peter's front line, but he also in the movie, the insider, this is this this is, the guy who, produced that story, and he's a legend in the the business. I was like, no.

We got to sit through the explanation of BGP Internet, I was, that was pretty neat.

Anyway, yes, than that being a a documentary.

And, yeah, so that was it was a bunch of a bunch of work, similar stuff explaining how's the internet work and how's it relate to the story. Of course, a couple of years later, they they modify the law. So the whole issue was kind of changed.

That was a that was that was definitely a a career highlight.

It sounds awesome. So what are you looking forward to now either? Technology, I mean, besides COVID being over and everyone being healthy and, you know, in the professionally, what are you looking forward to learning or doing or working on over the next, you know, year or so.

So I guess I'm I'm spending a a fair amount of time trying to I I am fortunate in joining Kentic at the time that it is for, the time the stage that it's at where it's got Lots of data, I've got some pretty good tools I can use to answer questions that are really unique. And so right from day one, I was in a position where I could start, try to pick up where I left it off, because I had been, a little bit limited by, Oracle's, conservative policies around.

Working with the press.

So I didn't know. I even, you know, I was talking to you about maybe, joining Kentic last year. I I feel like I was always trying to be careful what I can promise because I I don't know. Like, some of these guys I haven't worked with in a long time, and people, people leave the industry, and they move on.

They find other sources. I I I I I think I can gets going again, but I don't know, and sure enough, like, it looks like, it never stopped. And so, I guess I'm right now trying to, build out more capability to, alert on more, types of, you know, things that would be interesting both to both for both for news media and then also for our clients. You know, a lot of the the the we would collect the analysis gun that we would use, to to, do stories.

That's that all that capability is also useful for, for for customers. So we would I would also be looking out for anybody who has kind of required a white glove treatment, then I was definitely watching anything that could have, potentially affect them, and then occasionally why we're collecting all this data and looking at it is, right, for Yeah. And then we would get questions, like, like, like, we we think you guys would be in a position to answer this. Would you would somebody answer this and plus we're paying you a lot of money.

So you kinda can't say no. And that was basically, Jim would always take those because, like, who else was gonna do that. And then, when I got proficient, then we could share that task. I I guess I wouldn't say I did every until he moved on.

Then I could I could help out. And that was, that was a question for somebody would ask something and be like, alright. You just gotta pay a lot of money. You're gonna pay us a little more for this.

And we wanna make sure they're feel like they're taken care of.

I'll do it. I'll I'll I'll put together some analysis on this. So, there's, probably the majority of my work of day to day work never has never seen the light of day because it's more direct for, our our paying customers. But That's, that's that's the way it works.

So I I guess I'm I'm trying I I'm always continuing to try to build out, more ability to, you know, analyze, analyze stuff, I think, right now. But we've gotta I I'm I'm able to get a lot out of all the all the the Netflow that we've got, which is a really unique, something I've I've never had before. We always had, BGP and Trace route, as data sources, and those are can describe potential paths, but I don't know if, we'd I do a presentation about a routing leak, and on at the back of my head, I'd be like, I don't even know if a single packet got impacted by this. I just don't know.

But here, I do know. At least I've got a pretty good sense because this, if we missed it, then maybe a single person. An aggregate basis from the people that have For sure. Yeah.

Yeah. Yeah. Like, let's be, clear on that.

I'm just, but, like, if there's a, you know, a routing, leak or something, and we can, I can begin to get a sense for, like, well, we're, you know, who who likely was mostly affected? And I don't need to rely on just how much did different routes get propagated, then you can just look at, you know, aggregate flows, to get a sense. So that's cool. And I guess, the next phase is is, you know, we have a lot of capability now in the synthetic space of running a lot of, measurements, performance measurements. And I'm trying to, and I'll build up more, capability there to be able to, tell stories in that space as well, and Yeah. That's that's what I'm spending on, a bit of my free time doing. So looking back, any advice you'd give young dog at various points, starting out, you know, either entering the air force or, you know, leaving.

Yeah. Less than I learned, that I guess I could have learned earlier, of, of, their relationships can be really really useful. So the story I tell that, illustrates this is when, you know, after So I got to, my second assignment in Aviano, Italy, in August, twenty eleven. So September eleventh, which we're coming up on the anniversary tomorrow.

That was the following month. And then the rest of my time there being at a mobile unit that you know, task to be able to deploy a moment's notice. Life was pretty crazy for the rest next, you know, three years.

And, at times, we had stuff we needed to, get out the door, like, send to, Pakistan or something, like, really quick. And, we were so I was socially friendly with the other, lieutenants on the base, and and they kinda call this, in the space of, like, the Lieutenant Matthews. These are all the young officers often will go socially, go party together or do something together. But then this is they also represent, kind of the the lowest officer, but the highest there's a lot of power, the, even the even the most junior officer can have.

And, and so it's a great network. These that level that layer often knows each other really well. So we had something that needed to go. I had you guys, stuck in the logistics line trying to get stuff onto a plane.

And I knew the lieutenant who was in charge. And so I called him up. I was like, Brendan, like, we got yeah. I need your help.

And he's like, I'm on it. And so then, he went and personally saw to it to get it on the plane. And I was like, oh, it's it's helpful. Like, that was, that cost me nothing here.

That was just the fact that and and he was, you know, we were doing the right thing, but, but those relationships can be, super useful.

Also, like, with the relationships, There's a lot of people, especially in our space, in our industry.

There's a lot of people who are really here because they really like it. And if you have a quest they will they would love to answer your question.

And, and so, that's, maybe a untapped resource because sometimes you can you can sit and read something and, and that's sometimes you have to do that. We have to struggle with it a little bit to get to understand it.

But, but you can also ask someone. And, you'd be surprised sometimes how how much someone jumps on the chance to answer your question just because they are really into it. Maybe they wanna show off that they know it or prove to themselves that they know it, but, you know, if you ask it from a, position of humility, I feel like, hey, I I don't I don't get this to you. Can you explain this to me?

Yeah, people are, will will be very helpful.

And, anyway, I think those are things that I kinda picked up along the way, but Mhmm.

Yeah. I could've I guess I could've learned that a little earlier. Don't be shy as long as you're genuinely humble about it.

Yeah. Trying to build you a I mean, don't, yeah, don't don't have. I guess don't ask, someone to explain, you know. Right. You have to have done a little bit of work on your your end. Yeah.

Like, you gotta know what switch you gotta ask so much, but I found that in college, you know, I went to Tump University. It was a commuter school. So, the professors were all outstanding.

But, you know, there's a lot of the students that just weren't in the position to stay up till two AM just playing around with stuff because you know, their families and other. I was very fortunate.

And so, you know, I found, you know, that there was a lot of receptivity to If you are poking at something that's actually the area that they're, that they're working in, people get really passionate and excited about talking about it. And especially if you are interested in doing some of the work, you know, professors are always looking for people to help, you know, study and do work. And I'd like to think I've been that for someone else.

If if somebody gets me on the right moment, like, especially if I'm in person, I could go for hours. It's how much time they have. I could I could, you know, explain stuff. Yeah.

Hopefully, hopefully, they become someone that can, can help others and and maybe you back. So Definitely. Yeah. Definitely.

Very cool. Well, thanks, Doug, for being on, and maybe we'll have you on again to talk about, evolution over the next few years as as things are going. And that'd be great. Yeah.

Alright. Thanks a lot, Abby.

Got a guest?

Network AF is accepting guests for upcoming episodes. If you’d like to be on the podcast or refer a friend, reach out to networkaf@kentik.io.

About Network AF

Network AF is a journey of super-nerd proportions into the world of networking, cloud, and the internet. Avi Freedman, self-described internet plumber and podcast namesake, hosts top network engineering experts from around the world for in-depth, honest, and freewheeling banter on all-things-network — how-tos, best practices, biggest mistakes, war stories, hot takes, rants, and more.
We use cookies to deliver our services.
By using our website, you agree to the use of cookies as described in our Privacy Policy.