Exploring Submarine Telecommunication Cable Technology, Cuts, and Risks with Doug Madory

Submarine telecommunication cables have been in the news a lot lately with recent cable cuts in several parts of the world, most notably in the Red Sea recently in the spring of 2024. Now this kind of activity isn't good for global communications, but it has given rise to a new interest in how submarine cables actually connect the world and give us the Internet that we have today. Now there are a little over five hundred active submarine telecom cables right now with new ones coming online soon, running longer distances with greater capacity and connecting more geographic areas than ever before.

So it goes without saying that they are also a growing attack surface for bad actors in the world.

My guest today is Doug Madory, no stranger to the podcast and certainly no stranger to Internet measurement and the analysis of submarine cable activity around the world.

In today's episode, we'll explore what happened in the Red Sea. We'll discuss a little bit about how submarine cables function, how they're repaired, and we'll talk about some of the risks we're seeing today as they relate to submarine telecommunication cables.

My name is Phillip Gervasi, and this is Telemetry Now.

Doug, thank you for joining me again. It's been a little while since you've been on the podcast, but, certainly, I'm really glad to have you back on. I read your blog post on the cable cuts in the Red Sea recently. And, of course, I've been keeping track of all of that that's been going on, that and other events as well, simply because of my interest and, of course, Kentik's interest in, in monitoring Internet traffic globally.

Very interesting. So, I'm really glad to have you on. I would like to start off with a little bit of we already know your background. The audience knows Doug Madory. But I'd get I'd like to get a little bit of your background specifically in what you've been doing in monitoring this particular activity, cable cuts or or global Internet activity in general. And then I'd like to get into a little background about how these submarine cables are are laid and repaired and and just out there in the world.

Yeah. I've been doing a, analysis on, the impacts of submarine cables for, quite a quite a number of years. You're going back to my days, Renesys looking at, yeah, how do how do we see, submarine cable activations and, and cuts? What are the impacts?

Typically, when there's a submarine cable cut, it's a pretty violent event, from an Internet routing standpoint. There's a lot of reaction, to it. One thing that makes this challenging is that packets aren't labeled, what submarine cable they're on or these routes. There's no most of it has to be inferred.

And, and as years have gone by and we've added additional submarine cables in different parts of the world, that inference becomes more challenging because there's a lot, a lot more options. Actually, it's a good thing. It's, the whole system is more resilient, but as an analyst, it becomes a a more challenging prospect to try to under tease apart which cable, caused which event.

But, yeah. So then some some highlights, some for cable activations, the the Alba one cable to Cuba, the Kirk Strait cable to Crimea were ones that I spotted active more recently was the Saks cable, the South Atlantic, cable system that goes from Angola to Brazil, and then just this past year, the the spur that was activated to Saint Helena in the South Atlantic. Those are some examples of, submarine cable activations.

I was the first to publish, some Internet measurement data confirming their, their use.

And and I remember all your publications related to those. Very interesting. Has that been, a major focus of yours specifically because some of the activity that's been happening both with activations and with cable cuts in recent days?

Yeah. So the the stories right now are about cable cuts. That's the opposite, you know, the opposite, flip side of the coin here.

And, so we've got two major events, the I've been digging into, that are worth worth talking about, of of cable cuts. And there are they occurred under very different, circumstances, but they did also kinda occur on the opposite sides of the African continent.

And so the fur the first one was the one that took place in the Red Sea, on, let's see. I think February twenty fourth.

And, and so the background of what's going on there, we have, in Yemen, there's been a, an ongoing civil war, where the, Houthi group has kinda taken, Sanaa, the capital of Yemen.

They've taken over most of the telecommunication infrastructure and, and in, in retaliation for the war in Gaza, they've been launching missiles, in the direction of Israel. And now most recently since, October, they've been firing at, passing ships, that have some association with Israel, United States, or Great Britain.

And, there was some concern over there was a posting on a Telegram channel of whether they were gonna directly at, sever, submarine cable submarine cables going by Yemen because if they're Okay. If you can, if you don't have the, global submarine cable map memorized like I do, you know, Yemen is on the, the bottom of the Arabian Peninsula, and that's where all the, submarine cables that connect Europe and Asia have to traverse the Red Sea, and they go right by, the straight between, Djibouti and Yemen.

And, so they these cables are within their, their range. They're mostly they're often within their territorial waters. And, so there's some concern that they might do something directly, to one of the cables. How they do that, it's a little unclear.

But what ended up happening is, there's been ships that have been shot at, and a couple that have been disabled, including this one ship, the, Ruby Marr, that was hit with a missile.

Thankfully, the crew, was able to, get, rescued off of the ship, safely. They dropped an anchor, left the ship, and the ship began taking on water but didn't sink, for a couple of weeks. And, while it was, taking on water and listing, it started drifting And that anchor that they had sunk had dropped then, started to cross the, the sea floor snagging cables and, and cut, three of the cables that pass through the Red Sea.

The Red Sea is a problematic area because it is a, there's a there's a couple of factors that make it, problematic for submarine cables. And I've published analyses of submarine cable cuts in the Red Sea, a few times in the past as well. And so you have fairly shallow water, for an open body of water, like a large body of water like that. So it's just like only only three hundred meters or something.

And, it's also heavily trafficked, for ships passing through. You have a lot of ships that are queuing up to wait their turn to come through the Suez Canal. So there's a lot of, risk from anchors just in general. Right.

There's also, you know, you have this high concentration of submarine cables that have to, go between, Europe and, and Asia. You know, we could talk about why that's the case, but it's, that is just there's economies of scale that make that the optimal route.

And then those, sometimes they end up getting laid near each other because that's the optimal path, due to the, topographical features on the seafloor. And also, you know, sometimes these cables try to limit how much time they spend in the cable in the territorial waters of, nearby Right. Countries. So in the event that they had to do a repair, then, they'd like to be able to not have to deal with, the local government, the permitting process that can sometimes slow down a repair process.

But those factors then put a bunch of cables in a close, space, shallow water with lots of anchors swinging by. So that's a that's a, a very threatening environment. So then what's unique now is that you've got a hostile actor shooting at ships. Like, if it wasn't already a difficult situation before, now, you know, the the ship's been hit.

It finally sank. It took it down a couple of cables, before it was gone, and, we're not out of the water. This is, this could happen again.

The situation is not resolved. But, anyway, so that's the that's the Red Sea.

Right. And that's a that's an important one to consider, because there's a couple of different things to think about here. Number one, it is a high traffic area. But when you think about how, for our audience's sake, submarine cables are laid, they're typically going to make landfall near some sort of population center, though not necessarily, like, you know, the middle of Manhattan, something like that, but it's gonna be where there are data centers, some sort of a cable landing station, some sort of infrastructure to go from wet to dry and then bring that signal on land and do something with it.

And, of course, service providers can then, go go with it from there. And so, it's inevitable that you're going to have that last few miles before landfall be of course, it's in shallow water because you're in the last few miles before landfall, and there's there are protection zones. But it's inevitable that you're gonna have these I say inevitable, but it's highly likely that you're gonna have incidents occur with that that are unintentional with ships' anchors being dragged or dredged, whatever the word is, along the, the ocean floor in those shallower areas, or fishing trawlers. And and those are indeed probably the most common reasons that we have cable faults out there.

Now there are natural disasters and natural occurrences that that are also a cause of cable faults, which we're gonna get into as well when we talk about a different cable cut, later on. But it's something to think about that this is not, like, an isolated incident. Cable cuts are happening all the time, really. And many of the cables that are out there, there are over five hundred active, in the world right now, are sometimes often, maybe not often, but sometimes, they're they're, put into production in pairs or even in in triple triples, triplicates, whatever, into three Three cables.

So you have those that redundancy.

Now, of course, there's a cost associated with that, so not all cables are. Now you're talking about a very narrow body of water and shallow and geopolitically unstable area and missiles in the air. So you see there seems to be a, all of these things conspiring together to make it a very dangerous place to put these cables. Can you talk a little bit about why they're they are there and not somewhere else that might be a little safer?

Yeah. I mean, the same it's the same, reason that, that's a that's a common route a trade route, why the Suez Canal was constructed to to link the economies of Europe and Asia. That is the, the fastest way to route a a connection, between those geographies. And one might ask, why why can't we do this over land?

Because then you wouldn't have to deal with the water and the ships. And it turns out, actually feeding a line at the back of a ship is an easier, proposition than dealing with, all the the things that come along with, running a line, through crossing borders and across, terrestrial, distances. So it may be easier to fix a a terrestrial cut, but, there's a lot of lot of examples of why, it's easier to just run a a cable out in the ocean, than to try to, get a right away between two, destinations on on land. I would I would add to your, you know, your you said a couple of things here about, you know, obviously, you're gonna so what you wanna land a, a cable near a, you know, a population center or some sort of, hub of connectivity.

There's also other rationales where, you know, once you go into, a country's territorial waters, that country may have, requirements, for you to land the cable there because you're in the water and they have a say. Interesting. And so if you were to pull up a map and look at Djibouti and say, wow. Djibouti must be the most popular place, along this route for some reason.

Well, it turns out that they, this is a requirement by their government. If you're gonna pass through you can't you can't get through, the Red Sea without coming through their waters, and they want you to land, and, you know, there's a, I'm sure there's some, some fees they can collect, and there's some, some advantages to that connectivity. But, so there's there's some things there are kinda non, non obvious reasons why cables are landing in certain places. But and so that I think that this, these cable cuts, this particular incident has kinda brought up the, you know, resurface this concern around, our reliance on this particular route through the Red Sea.

It doesn't really go through the Suez Canal. There's cables that they typically come up on land and they go parallel to the, Suez, and there's this project called Trans Egypt, telecom Egypt, does where they run, in a in a mesh that goes across the the the land that pops out on over on Alexandria and the Mediterranean.

And, and so then but this this route, this has been an issue.

I I started going to submarine cable conferences maybe twelve or thirteen years ago, and it was always a topic of, like, how do we diversify this route and not have Right.

Not just have all the same cables, same going the same route. And, yeah, over the years, the other projects just haven't caught hold. There was a E peg that went through Iran, and, there was some complications around that. It was a jihadi cable that went through Syria and this and this civil war there, ended that, proposition.

I guess, there are some there's a I'm trying to think. Maybe GBI has got a route through Iraq.

They just, you see it for cost purposes.

It's just cheaper to use, the submarine cables. And this is a commodity business, moving traffic at the wholesale level that you would you would see on a submarine cable. So, it's very it's a very price, you know, sensitive thing. So the cheapest route is to do what everybody else is doing, which ends up pushing down the price and ends up being a submarine cable going through the Red Sea.

Yeah. Ultimately, we're we're leasing out, slices of bandwidth and and capacity, to to various folks on individual cables that might have multiple pairs of fibers, to increase their capacity and serve multiple customers and and therefore make some money.

But, you know, here in this particular scenario, the one that we've been discussing so far in the Red Sea and with this ship going down, you know, this is, this is an interesting one because we like to say, oh, this is, this is governments that are taking out submarine cables and there's war and all these things. But really when it comes down to it, this really is an example of a ship with an anchor accidentally taking out this cable. That was not intentional.

Though that was the initial, theory, wasn't it, that, this was, you know Yeah. I think, initially, there was a bit of a, a rush to Yeah.

That makes sense.

I mean, I I wanna jump in the same conclusion.

In in the in the in real I mean, it it it was the Houthis end up cutting the cables. It just was that they shot a ship that dragged an anchor, so there's a a couple more steps in the process. So, ultimately, you know, the folks firing the missiles are the ones responsible for these cables. That's Right.

So what kind of effect does that have on Internet connectivity, in that geographic region? I mean, are these the three cables? And I know from from reading your blog post that one of them was already sort of out of service anyway. But, ultimately, are these three cables and, the source of Internet connectivity for that entire region, or are these backbone between, you know, two different providers or two different areas?

Yeah. I would say they're very important.

Thankfully, there are other cables, and this goes back to maybe the conversation like, what I was saying at the, outset of just, as an analyst who's been trying to tease apart submarine cable incidents, for more than a decade.

And and that it gets a little harder, as the more cables arrive.

We still have, like, c m v five. There's other, major cables going along this route that can, take up the slack. So you've lost some capacity, and I think the you know, they can try to light some dark fiber or get some routes that weren't in use or weren't used at their full capacity. There's enough slack in the system that, with after a a couple of days, the engineers of the telecoms in this, along this path could find alternate meet alternate sources of of capacity to keep the traffic flowing.

There's not an infinite number of cables. So if this were to keep happening, if we'd have another, incident of this magnitude, it could be really, very problematic.

But, yeah. Right. At the moment, I I think people have been able to shift, their transit. And I guess, so then I I when I started looking at this, I I started, you know, look at the country level impacts, not just do we see a drop in traffic to a country, which I would classify as kind of a first order effect. But also looking at for the providers in the affected countries, how do their, telecoms shift, going from one transit source to another? That's kind of a second order thing, a little bit more complicated, but it's something that we have got the tools to do, here. And I I find that kind of interesting because then you can get a little bit richer, into the impacts.

And so, yeah, there's a I mean, I I guess once I started doing that, I started realizing, well, I think I could figure out which, the timings of, the cable the specific cables going down because they seem like they failed at different times.

And it was around this time that I joined up with Wire Magazine who was doing a, an investigation into this using satellite imagery and AIS, which is like tracking, vessel tracking, data to try to, use that to confirm the theory of the Ruby Mar, which I think most most people, in the industry and the submarine cable industry had settled on this being, likely true.

May there may be some other facts that need to be the surface to completely confirm it. But, anyway, so I joined with, Wired, and what I was sharing with them was looking at, I think, we could see, for one you know, so there's three cables. There's the EIG cable, the AE one cable, both of those go between Europe and Asia, and then CECOM, which is primarily a cable to provide service into the into East Africa, all the way down to South Africa.

And so, I just know from doing this for a long time. In fact, maybe going back to two thousand twelve and there was another three submarine cables, cuts due to an anchor in the Red Sea, that, you know, Seacom is, owned by is one of the owners is Tata Tata, the global telecom giant, out of India. And so, typically, they the the owner likes to use their cable to provide their service. Obviously, that's why they have this bought the cable, and so that's usually a good, you know, tip-off, when you're trying to infer, the timings of the cable cuts.

So what I saw was, you know, a bunch of East African providers losing Tata service at the same time. And I was like, alright. Okay. That's probably, it's got the CECOM.

And then and then about six or seven minutes after that, then seeing impacts, along the path of eighty one. As I said before, it wasn't public information, but, a a EIG had already suffered a cable cut. In fact, they were in the process of trying to get a ship out to repair it when the cut, EIG a second time, and, and they're trying to get that, squared away as well. So, yeah.

So then in the blog post, I that I we kinda published simultaneously with Wired magazine. I tried to, you know, present some of the, the analysis and the data that I did, relying on either BGB data or, some of the aggregate net flow that we have, at, at Kentik to look at, to tease apart the impacts of the different cable cuts. And, yeah, I find that interesting, and it's in it's also interesting that these cuts I guess, I can't see the third cut, so I don't know what the timing of that one, because there was no operational Internet impact. But, but the other two were within, you know, within ten minutes of each other, which is, different than the the cable cuts in West Africa.

So then what data are you looking at in order to see, what's going on? I mean, I guess that's the question for Internet measurement as a whole, but in the case specifically of what's been going on in the Red Sea. And then how were you able to determine the timing of those cuts? What were you able to see in the data?

Yeah. Like I said, I, I would look at transit for the big providers in countries along the path. So, I guess, for one, we could see a little dip in traffic, for a handful of countries in East Africa. Also, it was visible in IOTA, the Georgia Tech, Internet outage detection service.

And, so that's a that's a public tool.

And so that, was another clue to say, like, alright. Well, this is probably when, the, probably, CECOM, had to cut. And so we can look at transit in BGP. So in BGP data, we'll see BGP adjacencies and see, you know, the basically, look at the Internet.

How does the Internet get to, SimbaNet, and through what transit provider and how does that change through time? And so when you do that, then you can see there's those things are typically very stable, like there's no movement. And so when there's but then when there is a change, the beauty of DGP data is that you can get down to the one second, you know, level of granularity to really pinpoint when something's changing. And, and so those features, those, yeah, the the changes, kinda give away, when there's something that something something has perturbed the system.

And in this case, we already kinda know, that we're looking for, impacts of the submarine cable cut, ought to look a certain way. Here we are seeing train drivers dropping away or routes, going to getting getting completely withdrawn.

So I, I guess I've been doing BGP analysis for, like, almost fifteen years now. So that's definitely one of the first things I go to. I know how to, go for that data and look for, the impacts. We've got some pretty good tools here, that I can do this en masse.

And then, yeah, then, yeah, again, within the net net flow, I guess, is the other thing that we've got, to, have a giveaway of, impacts to traffic. Again, things ought to follow some sort of predictable diurnal, you know, slope throughout the day. And when there's a aberration from that, that curve, then there's usually a reason. And, anyways, that's that's another giveaway.

Right. Now what about the, there was another cable cut that happened not quite at the same time, but, in early March of, of last year in West Africa or off the coast of West Africa, the main one cable.

That's something that you were able to see as well, right, the data?

Yeah. So there's in that case, there's four cables. So there's a lot of cable broken cables, within a couple of weeks.

In that case, so this is a, is a different circumstance, and there was four cables cut on March fourteenth.

And so let me see if I can recall them. The yeah. You mentioned the main one cable. There's a wax cable, sat three and ace, are the a four.

And so these were cut, not due to a vessel, drifting after being hit with a missile, but, again, the, the the prevailing theory within the industry is that there was an undersea rock slide, off the coast of, Cote d'Ivoire, and there is a, there is a, I didn't know this about the geography, under the under the water there, but there is a, I'll I'll get the French term wrong, but it's transomphon. Like, it's a it's a bottomless pit. There's like a hole, in the sea floor that goes down a couple of kilometers. Yeah. And there's, the the ground around it kind of, is on a slope down towards this, bottomless pit. And, so the belief is that, somewhere along the slope, triggered a rock slide and, and I over the span of about five and a half hours, then these four cables all suffered, catastrophic failures.

And so then, it's an interesting contrast to the Red Sea cable cuts where you've got again, I only know two of the three, with failed within, six or seven minutes.

In the case of Cote d'Ivoire or West Africa cable cuts, You've got four that failed in the course of five and a half hours. Mhmm. And it it likely has to do with, what's going on, under the sea. And I talked to somebody who had been involved in, laying of one of the not not one of those four, but another one of the major cables off the the West Coast of Africa. I try to tease apart because I don't have a I don't have a lot of experience on the sea floor to understand, you know, how how how like, physics is a little different down there of just how things happen. And, so in the case of, you know, you've got a rock slide or undersea landslide that takes out a cable, why would it be that it takes many hours to get all these cables to break? And there's like a few, you know, theories from people who, are more experienced than I am, who work with this stuff.

One is that, in a case like that, the brakes can get complicated. So you can have, what they call a shunt fault. So if some water breaks through the cable and interferes with the electrical system, so there's, you know, thousands of volts going across these cables to power the repeaters.

And they can they can survive a little bit, a break at one point, into kind of a limp mode, if you will. But then if anything else happens and they're gone. And and so you can have kind of, degrading, health of the cable over the time. And so I'm imagining something where you've got, you know, a rock slide, you've got weight sitting on a cable and it it's strong enough to take it for a little while but not forever.

And if it's certainly certainly if it's suspended, if there's a suspension in the cable with weight on it, then over time, that's gonna be too much and it's gonna, cause it to break. But it is interesting, detail that is a a different attribute between what took place in the Red Sea just a couple of weeks prior. I would add one more thing too is that and so I there was a, this isn't the first time in the past just in the past year that there was a, undersea landslide taking out submarine cables in the West Coast of Africa. So in August, there was another incident and I, we published some analysis the going digging into, the undersea cable, undersea landslides that took out the subsea cables in the Congo Canyon.

So in, the Congo River comes out of a West Coast of the West Coast of Africa, and for the last, I don't know, million years, it's been it's been carving a a groove into the the sea floor of the Atlantic.

And, so it's created the largest undersea canyon in the world, and they go the canyon goes east west, and the cables all have to go north south. So they have to kinda, cut through the the cable, and it's just a precarious place for anything under sea, especially submarine cables.

So we just had one, in August, that took out, a couple other cables, and, and here we are again with another undersea event. And in that case, the two cables that were broken, I went in and looked at the timing of those, and they were about ten hours apart. So it was even bigger difference. So it's interesting to just put your put yourself in, in your imagination on the sea floor trying to imagine what, how these things are actually getting severed.

But, yeah. So they're in the process of of patching these things back together.

But, And it does take time, certainly.

You you are talking about the deploying repair ships. Now for the audience's sake, you know, we are talking about fiber optic cables. And so we're talking about a core of fiber strands. It's very, very small.

Everything else around that cable, which can be the size of a garden hose or perhaps a little bit larger, depending on, you know, where it's laid and things like that, is protective sheathing. However, one of those layers is going to be the copper mechanism that provides power to the cable. These are active cables, like, and so, I what are we talking about? Something like upwards of twenty thousand volts, coming into the cable from both ends, from both both cable landing stations since they they are home run on both ends.

And so, you know, there's there's there's various types of mesh and steel mesh and other types of sheathing to protect from this kind of thing. And, believe it or not, something that I've read in the literature and in talking to some folks over at Telegeography in in particular is that it it's it's safer generally in the deepest parts of the ocean because there's less stuff going on. You don't have ships' anchors and, and as much movement going on. And and these cables can be submerged down to seven, eight thousand meters in the deepest parts of the ocean. So it's very it's very interesting, what's going on as far as the technology.

And then on at the cable landing stations, when we're talking about rerouting and things like that, this is a this is a manual process. I mean, there might be some automation involved. What I mean is that there is a process, a workflow involved in in moving traffic from one physical cable to another physical cable at the termination points of the cable annexations.

So, you know, with over five hundred cables out there and and natural disasters occurring and rock slides happening and geopolitical instability, there are cable faults going on all the time. All the time. So I would like to move over now, Doug, to recovery and how that works. So, what who who's fixing the cables? I mean, is it just Seacom send the boat out? We have helicopters and divers. How does this work?

Yeah. So there's a, a small fleet of specialized ships that go out. They have to send a submersible down or or try to snag the, the cable off the sea floor, pull it on the deck, and they've got a crew there to then, you know, assess the, what's wrong with the cable, make a fix, and then, gently push it back out the back of the the ship.

You know, you in the case of, the Red Sea, I mentioned in my closing of that blog post, that the, minister telecommunications for Yemen, this is the Houthi government that's in control and Sanaa.

I guess he's active on Twitter, and, and posted that, just a reminder that any cable ships that we're gonna come in, needed to, get permits from their government and pay them, for access to the waters, to their territorial waters, to do the repairs and emphasize that this was for, their the ship's safety.

And I, I couldn't help but kind of smiled to myself, and I was like, yeah. Maritime safety is definitely the passion of these guys.

But, yeah. So then, in that in the case of, the Cote d'Ivoire, cuts, those are estimated to be about three kilometers deep. So this is pretty deep. In fact, when that depth was, first surf surfaced, so to speak, then that's that's outside of the range of an anchor.

Mhmm. It's, so there had to be something you know, seismic landslide kinda thing on the on the sea floor.

And, and those typically end up being harder to, to deal with. The the the greater the depth, the more, the increased difficulty.

But, like I said, they've, I think they've already got sat three for whatever reason, which is the oldest and probably the the least, impactful cable was seemed to be the first repaired. Maybe it was just the easiest to get to.

Yeah. The there's a there's a, there's also a it's all worked out ahead of time, through contracting of who are the ships that have the are, you know, assigned to repair what cables. So this is, part of there's a process, when a cable goes into existence and operation who what what ships are maintaining it. There's a prioritization order.

And so, you know, based on who owns the cable, then that they dictate which, cable or pair ship needs to come out and fix it Right. And stop whatever it was doing, prior because there these ships are not idle, really. They're they're always, working on some sort of, maintenance until they get called away to do Yep. A repair.

And, and so then a lot of the time, that, you wait to repair a cable, some of it is just the time it takes for a ship to sail from one part of the world to another.

The fact that we had a bunch of cable cuts at the same time meant that, you know, the, there's a whole bunch of work that needed to be done. I I guess we had I guess it wasn't so much of an issue that, I think there was some concern there would be a log jam, for, cable or parachutes. I think we've got a sufficient number, and I don't know that that was an issue. And then depending on the country or the territorial waters, you've got permitting processes that Right.

Yeah. Can be kinda headaches for, the folks Yeah. Can be kinda headaches for, the folks who do, the the repair work. Sometimes it ends up being a bureaucratic process.

Yeah. The the real life of, the reality of undersea cables is a little bit more complex than the fiber optic you run-in your data center. You know, we're talking about path diversity, and I need to connect to two different providers outside of my campus. That's that's fine.

That's easier said than done when you're talking about transcontinental sub sea cables, multiple countries, multiple companies, web scale providers, Google, Meta, Microsoft. These companies have a vested interest as well. In fact, they're they're just putting out their own cables or rather when I say putting out, they're investing in the construction and the installation of their own cables. So it is much more complex than what we would do in our data center.

And so, we are talking about, permitting. We're talking about geopolitical instability. We are talking about doing environmental impact studies. And then as far as repair is concerned, you mentioned that we have enough ships, but let's put a number out there.

It's it's in the dozens. I mean, there's, like, sixty, eighty ships that we have available in the world to do to do these types of repairs. Is that right or am I way off?

I'm not sure what the number is, to be honest.

I know I It's not a huge number.

I know that.

A few years ago, I remember seeing a figure of, like, forty something, but that's from maybe twenty fifteen. So there's probably more, but, it's not you know, it's less than a hundred. So and these are Yeah. Really specialized ships. They don't this is all they do. Yeah.

And it's interesting when you when you think about how how are they figuring out where the fault is. You wanna narrow it down, so you send the ship to the right location. Right? You're not just gonna go trace this this thing like you, you know, run a traceroute command, but you sort of are. You sort of are. You're using, OTDR, optical time domain reflectometer.

So, basically, you're sending I'm gonna, gross oversimplification is you're sending an electrical impulse or or a a network, you know, like a like a ping, right, down the line to see where you know? And and each repeater is gonna send back a response, and you have repeaters every fifty, seventy, ninety kilometers depending on what's going on, but around there. And so when you stop getting that response, you can narrow down where the fault is. And so you can't narrow down to, like, you know, a square inch, but you can narrow it down to enough of a a, you know, a certain number of kilometers that you can send the ship to that location, pull it up, and and get to work.

Yeah. No. I think it's I think it's actually quite analogous to ODTR that you would use in an in a building.

That it basically blasts a light signal and listen for the, reflection, and then they can use the just do a bit of math and figure out how far down the line. So I think on one of the cable, notices, somebody shared with me, you know, they mentioned that they're, in the process of getting somebody to run some ODTR tests, from the nearby cable landing stations to just isolate exactly where the, get a good estimate of where the cable break is. But it's, it's very analogous to what someone, just working at fiber optics, on land would do to identify where the break is.

Except for the fact that you need seven hundred and twelve permits from international bodies and you have to worry about missiles being fired upon your repair ship. That part may be a little different. And therein lies, you know, some of the complexity. Right? We are we are talking about the, this this interesting blend of, of technology and advancing technology and a desire for more bandwidth. We are in networking, so we always want as much bandwidth as we can get, But also, how how this interplays with multiple countries vying for control and vying for connectivity and, and some sort of say into this entire tapestry of of submarine cables that we have.

The implications of these, though, are significant. Right? So so what other parts of the world would you consider sensitive beyond the Red Sea? And and and I don't mean in the ambiguous sense of, like, anything that's close to land because a ship can hit it. What other parts of the world have you seen as issues or potential issues for submarine cable cuts?

Sure. So in Asia, there's a the Luzon Strait. There's a couple of, bodies of water where it's really hard to avoid, going through certain certain spots.

I think if you bring up a submarine cable map, you can see where those are. But in, you know, Southeast Asia, there's a couple of spots. And then you've got this added complication too of, again, back to the permitting and the bureaucratic process. So Indonesia has a bad reputation for being really slow on, the permitting process. And so you've got cables that are, you know, the big cases have been cables have been broken. The the consortium's, you know, losing a million dollars a day or something. There's some very large amount of money to get this cable back, up and running, and, and they have to wait, you know, for a month for someone to stamp a form.

And then you've got also, you know, the South China Sea is another geopolitical hot spot. We don't have a cable cut there, but that is another place where you can have a a very strong geopolitical disagreement over who, is, you know, whose water, the cable is in. And so I think these days, the cables going into that part of the world tried are trying to avoid as many of those situations as possible because we get a break and, two countries vehemently disagree over whose, whose water this is, it becomes no longer a technical issue. It's a, portability.

You mentioned the word consortium. Who who are you talking about?

So a lot of these cables are, are put together with a lot of, many companies. Often, all of the so like take, the CME WE, series of cables.

Those are I'm not sure the number, maybe twenty or thirty, telecoms have some sort part ownership of the cable, and together they form a consortium, which is kind of like a, you know, a a legal, construct, for them to, share in the governance and the costs and the profits of the cable, and that there's a whole science into how that gets done. There's typically, you know, one, one or two major telecoms that are that kinda lead the effort, and the others are there because they have, there's spurs that come out for cable landings in those countries, and then the incumbents of those countries wanna, be a part of the cable. They'd be a part of the consortium.

So that's that's where that term comes. Not everyone not every cable is is that way.

The other model is that you've got a single ownership cable. So you mentioned, like, Google and Meta. So like the Google cables, the these are submarine cables that Google is putting the money up for, and building, you know, for their use, but they also will sell, capacity like another submarine cable to offset the cost.

And, in those cases, those are not consortium. They are the owner, but then they can, you know, share the costs with the other people using the, the cable. But then that's a that's a different, model where, yeah, I I'm I'm sure, people have different opinions of what's a more efficient, way to do do it.

And by cost, I mean, we are it depend depends on the length of the cable, how many spurs there are, how many, you know, basically, how many kilometers long because that does determine installation. It determines the amount of physical material. It depends, how many how many fibers are you actually running. You know, how many channels do you have?

Yes. The the cable itself, the fabrication of the cable itself is the main cost. Yep. That's far away more. And then the, you know, the path that you choose to, lay your cable, you know, how how direct you can be from one place to another, how can you avoid different, undersea threats or features.

If you have to, you know, go around something that you're adding costs. So there's a lot of there's a lot of considerations, and then you'd like to have something that the was a more direct path for lower latency.

You might wanna be outside of some territorial waters like we talked about earlier, but that may come at a cost of a greater distance of the cable.

So these are all factors.

Right.

It's a complex endeavor.

Sure. And then the nontechnical factors like all the lawyers that you need.

You need yeah. There's a whole there's a whole world of specialists that do different parts of this process, of Environmental impact studies Yeah. That probably cost millions of dollars.

So the overall cost of a cable could easily, very quickly, hundreds of millions of dollars.

That's a normal that's a normal cost.

Yeah. Four, five hundred million dollars to do a transoceanic cable is a is a is a normal cost. And so, it's a it's it is important to find these ways to be efficient to to, maintain their, usefulness for as long as possible, which from what I understand in the research, is a couple of decades. Twenty, twenty five years is the expected lifespan of a cable. So which is why we have these repair ships, and and we do as much as we can. These organizations will deploy cables, sometimes in pairs for that for that reason.

Now I know that the vast majority of Internet traffic and when I say Internet traffic, I don't mean, like, Doug, you and me clicking on a thing and then going to a website, because a lot of that stuff today is delivered much more locally.

CDNs and and, you know, Microsoft is gonna put pops regionally. And so when I'm going somewhere, that that information is collocated around the world. So I'm not going across submarine cables constantly, like me personally as an end user. But there's a tremendous amount of Internet traffic, backhaul traffic, data center traffic among the cloud providers, among the, web scale companies, all of that, and perhaps some of our traffic if we're if we're on Saint Helena Island, right, where all of our traffic is leaving the island, is all traversing submarine cables, and the and only a very tiny percentage is traversing anything else, and specifically satellite.

But I'm bringing this up because I just watched a documentary on Starlink, and the latency that they offer, the bandwidth that they offer, and the fact that you can get connection in seven continents, which for our listeners that don't know, are all the continents, Surely, that has to be a viable option, as a backup when cables go out and and a region is completely offline or maybe as a primary connectivity type? What are your thoughts on that?

Yeah. The Starlink is a very impressive technology. It's very, cutting edge as far as a satellite based Internet service, which I might add, you know, what did exist decades prior to Starlink and Elon Musk. I mean, this didn't it didn't first appear, but but, Starlink is a superior, service, by a lot of metrics.

Well, it's the difference between low Earth orbit and and and geostationary.

Yeah.

So we've had geostationary for quite a while.

And, and so then the next, you know, the the three, you know, categories here, you've got Geo, MEO, and LEO. And so in in the MEO category, the one company that was able to do this that's still in existence is o three b. And so in the, it's a interesting, you know, transition between the three. So with geostationary, you could just put one satellite on one place. It's gonna stay above a certain part of the, world, you know, for the lifespan of the satellite. With, MEO, so, this is between, GEO and LEO, then you can cover the world with you're kind of it's like a, compromise between the two approaches with about, I I don't know, like sixty or seventy satellites all in motion.

And and so then the, because the the satellites are still kinda far away, you need a a dish or something, that can, receive the signal, but also that satellite's moving. So the dish has to be able to move. So the the ground stations for o three b service, which I believe is the only, MEO, service that's that's in use, usually have multiple dishes.

They're tracking the satellites as they go by and they're handing off, which is a this was a technical feat when this first came, you know, a number of years ago, where they're they're handing off the signal from one dish to the other as one satellite goes over and then is out of sight and another one comes over.

And so then the next phase down is like we're talking about with the Starlink. And so these are what they call constellations of thousands of satellites all in low Earth orbit, where you you need a a fairly small, piece of equipment that doesn't have to be tracking satellites through the sky, and moving.

It's, it's far simpler technology. So it's it's a it's a very cool thing. We see it, you know, in a lot of geopolitical stories, the war in Ukraine. Exactly. Every time there's a a shutdown of Internet in a authoritarian regime country, you'll see people tweeting at Elon asking him to use Starlink to turn the service back on.

So it's a it's a hot topic for a good reason. But the I guess the one thing that anybody in this business would tell you is that the, you can't, the capacity is just not ever gonna be the same. You cannot have, in fact, the the entire capacity the capacity of the entire Starlink, constellation is less than what you would get on a submarine cable.

On the singles at marine cable.

Right. So, it's just you just it's hard to replace, the the, what's available via a fiber optic. I guess the the other piece too is that oh, let's see. I got yeah. At one point, this is this has changed through time, but at one point, because, with LEO, you've got lots of satellites fairly close to the Earth.

You had to, bounce back down. You're kinda relaying off the satellite to something in in the general vicinity, on on planet Earth of, where you're at, because they at one point, there wasn't this ISL, so inter satellite links, which now is, again, another another piece of innovation from, from Starlink of being able to, go you you beam up to a satellite and then you can ping pong between satellites to get down to your other location. It's, it's it's it's really, as as a technical person in this space, it's kind of amazing. I remember talking to one of the engineers who was involved in this, at a Nanog in the past, and just he was just commenting on how they had to invent some new routing protocols of how do you, you know, at least when we do routing, the routers stay in the same place.

If you're if you're if you're com clauses were all rotating around and you had to kinda constantly make decisions, on a per packet basis based on which satellite or which closet or, you know, switch rack is near you, That's a lot of comp complexity, and it seems like they so the the the challenge was could you do LEO across the Atlantic Ocean? Because if you if you had to reflect down to a local, ground station that was, you wouldn't you wouldn't it couldn't be one. And, it seems like they've been able to use ISL then to extend the range of, the Leo constellation.

It's pretty neat. I mean, it's still as a business, it's still unclear.

A lot of these projects have not have have failed, and gone up to bankruptcy. And it's, in and as amazing as Starlink is, there's no guarantees, for that project either. But, Right. It's, it's still pretty cool.

Pretty cool. And, I think it's clear based on, you know, what you just said and then and, what we know about Starlink and and then others that are coming up like Kyper from Amazon is that, ultimately, it just doesn't provide the same, It's awesome for remote connectivity and for a lot of use cases. I've I've been looking at buying one for, you know, bringing camping. You know?

It's a little bit expensive to have a monthly payment for just camping a few times a year, but, nevertheless, that's something I've been checking out as a as a cool option for that specific use case. But for Internet backhaul and for global Internet access, on that kind of a scale, global scale, not not really there yet. You really can't compare with fibers in a in a tube, under the water right now. The the two Africa cable itself, I think, is, almost two hundred terabits per second capacity.

Okay.

That's a lot.

Could be.

Yeah. That's a lot of that's a lot of Starlink satellites floating around.

It's not gonna be.

On the topic of Starlink, you know, with with the ongoing war in Ukraine, so Starlink's been a big topic, or subplot to that, conflict and the connectivity of the country. And, you know, at one point, there was, you know, some headlines saying, Starling's keeping the whole country connected, which is not true. I mean, the most of the country is connected again through the fiber optic lines going out west, from the country that were in existence prior to the, invasion, from Russia. But, but it it can play a critical role of reestablishing, connectivity in places places where, it's been lost or in this military context. Again, as someone who did tactical comm, I get why this would be super useful. You've got units on the move. They can get this, terrific Internet service, through a a lightweight piece of equipment.

So I think that's that's where that role, has played.

On on occasion, it's been kinda clumsily, rounded up to, oh, the whole country is, running through a satellite, or or through a a Starlink constellation. That's not really true.

So, Doug, I'd like to, I'd like to move on to, a story that I heard, if you don't mind sharing.

Okay.

You you, at one point, almost got a piece, physical piece of a submarine cable.

I have always wanted a piece. I've been doing a lot of research into this area for quite a while, and I would love to get my hands on one. I've checked eBay.

You really can't buy just a chunk of submarine cable, but you almost did. Can you tell us about that?

Yeah. I I was at a cable conference, a number of years ago. I was on a panel with, with, a gentleman, from another, Yeah. I don't know.

I don't know if I wanna name any names, but, terrific guy. He was working with a Chinese cable manufacturing company, and, they had a display one of the Submarine Network's World in Singapore a few years ago. And I was like I was like, oh, like, you know, how do I get one of those samples? Because at at these conferences and all the booths, there's all these, you know, little, mounted, segments of cable.

And I, he's like, oh, I can get you on now. I've known you for a number of years now, and I was like, okay. So we walked over there and then the the the guys in the booth were like, get out of here.

Like anyway, I I thought that guy was in charge. He was the boss, but it turns out he did was not. And it was the folks, in the, in the booth. So, yeah, I'm still still waiting for that, the piece of cable, one day. I I I make the joke that, you know, for some of these older defunct cables, it might be worth it just to pull it up out of the ocean and cut it up into, twelve inch segments, mount those on wood and sell those for, you know, a couple hundred bucks. Or I could just drive down, you know, the highway to the Subcom, plant in Portsmouth, New Hampshire and, just rummage around in the, the trash outside the plant and see if I can find, some some pieces that they've discarded.

Yeah. Because they're probably discarding transatlantic submarine cables in their outside centers for the local trash folks to pick up. It would be Barnacles.

One of those behind my desk, you know, in Zoom calls in my background and and Somebody's gonna somebody's gonna wise up and, satisfy this market need.

Absolutely.

Well, this has been a really great, podcast. I love talking about submarine cables, of course, in general, but also now, tying it to what's been going on in the world, some really important events. So thank you very much for your analysis, of course, and what you do, but also for joining me today and talking to me. So, before we end, though, of course, as always, how can folks reach out to you? If they have a question, which some might on this particular topic, of course, how can they find you online?

Yeah. I'm on Twitter, now X, and, LinkedIn. Those are probably the best places to to find me. Great.

And and and for the audience's sake, you're gonna find a lot of Doug's writing on the Kentik blog. So I recommend that you scroll through there, search for his name. You can search for my name as if you want as well, but search for his name and see his work there. You can also find me on Twitter, still very active there, network underscore Phil.

You can search my name in LinkedIn, and, you can find me online other places. Now if you have an idea for an episode or if you'd like to be a guest on telemetry now, please reach out at telemetrynow@kentik.com. So for now, thanks very much for listening. Bye bye.