Connecting the world: Undersea cables with Alan Mauldin

Phillip Gervasi: How does the massive amount of internet traffic really get around the world so fast, so reliably, pretty much regardless of what country you're in? Now, the answer isn't satellites, not even close. Under our oceans is an entire ecosystem of undersea fiber optic cable technology crisscrossing thousands of miles back and forth beneath waves, moving huge amounts of data from data center to data center, country to country, continent to continent. So how does that work? Who owns those cables? Who repairs them? Who pays for them? And not only that, how do they work and how much data can these submarine cables actually move? With us today is Alan Mauldin, Research Director at TeleGeography and a subject matter expert on the submarine cables, making the global internet possible. Also, joining again today is Doug Madory, Director of Internet Analysis at Kentik. Now, in this episode, we'll be answering those questions and more, and we'll even get into some of the geopolitical nuance of submarine cables. My name is Phillip Gervasi, and this is Telemetry Now. Alan, thank you for joining us today, really appreciate it. Just having spoken to Doug quite a bit about what you do, and then of course, the fact that everybody in our industry seems to be enamored when we look at submarine cable maps. Am I right? It really is cool to talk to you today. Can you give our audience just a little bit of your background, what you do for TeleGeography, of course, but your background overall as somebody in this field?

Alan Mauldin: Thanks, Phil. I'm really happy to be here today and talk to you guys. So I'm a director of research at TeleGeography. We're a market research company focused really on the international telecom industry. So we're gathering data on the supply domain of bandwidth globally on the submarine cables, the terrestrial networks, the IP backbones, data centers. Looking at the pricing of various services as well, from wavelengths to IP transit to enterprise services as well. So covering a wide range of research areas at TeleGeography.

Phillip Gervasi: Excellent. Thank you, Alan. Really glad to have you on today. And Doug, also great to have you on and returning to us as well. Can you explain to us a little bit about your own professional background, of course. But also what your experience is working with Alan and TeleGeography from the perspective in the context of internet analysis?

Doug Madory: Yeah, so I've been doing internet measurement and internet analysis since 2009, back when I was with Renesys. And one of the initial requests I had from my boss at the time was, " Hey, we always get asked about submarine cable, this or that. Can you see if you can figure out, can we see submarine cable events, activations, breaks in our measurement data?" And so, I started to try to get smart on the field and follow headlines. I was subscriber to at least all the geography free stuff of just announcements about new cables. And I was like, " All right, can I see an activation?" And once I started to get kind of good at that, then it led to a couple of discoveries, one of those being activation of the ALBA- 1 submarine cable to Cuba, which we just celebrated the 10- year anniversary last month of that activation. So anyway, so that's why I got into submarine cable stuff. It's not that easy to map internet measurement data to submarine cables because things aren't labeled. You have to kind of make a lot of inferences.

Phillip Gervasi: Oh, is that right?

Doug Madory: Yeah. Occasionally, there is a submarine cable owned by a telecom that does an IP service that is strictly using that cable. So in the case of Angola Cables or C- COM or there's a few. There's an IP transit service that also goes over the cable, but then there's lots of other stuff that goes over the cable too that you'd have to infer who is using it based on outages. Over the years with the additional cables, it's become harder. I mean, it's better for all of us. It's better for the internet, a lot more resilience. But as an analyst, trying to make these inferences, it becomes more difficult. But it's a bit of an art trying to suss out a submarine cable event in internet measurement data.

Phillip Gervasi: Which is why you work with folks like Alan and others, I'm sure.

Doug Madory: Yeah. So then I started getting into this and then I got onto Alan and Tim's radar. I think we crossed paths at a NANOG Conference and we started chatting. And then I got invited to one of the summary cable conferences that Alan is on the program committee for, and he and I would kind of kick off for a couple of years there, the two- day conference in Singapore. I mean, as it's fascinating, Phil, you were mentioning this is just a topic of fascination to people in this field, and even not in this field. Even as somebody who's been in it for a while, it's still kind of amazing that this is how it all works. But then getting to speak at a bunch of these conferences and meet the people who are involved in the industry, it's an interesting place unto itself, the submarine cable industry.

Phillip Gervasi: What are some of those conferences? Cause obviously I'm very familiar with NANOG and some of the other conferences out there. But I'm really not that familiar with the submarine cable conferences, if that's what they're even called.

Doug Madory: Alan, do you want to take this?

Alan Mauldin: I mean, there's a few. I'd say one is Submarine Networks World in Singapore each year. One in London as well in the spring. But also, just the major telecom events have a heavy summer cable focus like the PTC Conference in Hawaii in January. And other of their regional capacity events, whether it's in Dubai or Brazil, often have a heavy focus on subsea cables because cables are just one part of the overall system of how the world connects. So there's things that need to be discussed alongside cables, whether it's data centers, terrestrial networks, all these things play a role in getting the world together.

Phillip Gervasi: Yeah, in fact, just this morning before I left the house, I have some workers in my house remodeling, and one of them was framing a new closet for me, and he was asking me about what I did for a living. And I was talking about being a network engineer, which I'm really not anymore because I work in technical marketing, but whatever. I always say network engineer anyway, and we're talking about that stuff and he starts bringing up internet stuff and websites. " Hey, could you design my website?" I'm like, " No, that's not what network engineers do."

Alan Mauldin: Fix my printer.

Phillip Gervasi: Yeah, right. Standard answer, can you fix my wireless printer? But then I got it in my head to say, " Hey, let me show you this picture." Because I knew we were going to be recording this podcast this morning. So I queue up a submarine cable map on my phone and I show it to him. I'm like, " Here's what the submarine cables look like that carry internet traffic around the world." And he's looking at it like, " What?" And he was so enamored with that picture on my phone, pinching and zooming and looking in and zooming in on all the connections between North America and Europe primarily. But then kind of looking around. He even noticed those single little blue and orange lines that would go across the Indian Ocean to one island, which I assume is some island nation, really cool. And then to him, his response was, " I thought it was all satellites." And that's probably a common misconception for a lay person. But the reality is that satellites have virtually nothing to do with moving global internet traffic around the world, that it's almost entirely physical media fiber optic undersea cables that do that for us, right?

Alan Mauldin: Yeah, that's absolutely right. I mean, the vast majority of all intercontinental communications is going over submarine cables. It's not going over satellites. Satellites do a great job of providing an alternate end user access technology to reach remote places, underserved areas, airplane ships, things like this. But they are in no way capable of carrying the vlog data that's being carried by a submarine cable. I think one fiber pair on one cable is the same capacity as the entire starlink system.

Phillip Gervasi: Oh, is that right?

Alan Mauldin: So you can just think about how that's a very high capacity satellite system, and it's very useful for linking people remotely to the internet. But it could not carry the data that's being sent between continents around the world at all. And it wasn't even designed to do that, that's in the point of starlink anyways. Cables have a very important role, and that's going to be that way for the future thus far, we can tell. We're seeing more money going into new cable construction over the next three years. Over$ 10 billion is going into new cables, really all over the world. You're seeing cables in the Atlantic, Pacific, South America, Africa's getting two massive new cables installed during this year and next year. So really a boom time right now for submarine cables around the world.

Doug Madory: Alan, what's the cause for that? Is it just capacity or is there more resilience or lower latencies between particular locations or what's the driving factor?

Alan Mauldin: The driving factor for new cables, there's many factors, of course, that drive new cables. I mean, the most obvious is demand growth. Whenever you have demand for bandwidth internationally growing 30, 40% between most areas, that's almost doubling every two years. So there's a need over time to just build new cables to meet that demand. There was many cables built in the telecom boom like the late'90s or early 2000s. These cables, many of them are still on surface. They're doing a great job of providing a high level of bandwidth, but they can't scale much anymore. So they're going to be turned off actually probably in the next four or five years. Some of them have already been turned off. We're going to see a couple of turned off this year. I mean, the Japan, US cables going to be turned off in July of this year, for example. So when you have older cables going out of service, you got new ones coming in, you got to have more than one or two. You have to have diversity because no one knows cables exist partially because when they break, no one notices, right? Because there's so many that are in place. All that just happen all the time globally, but you don't hear about it. So building not just 1 or 2, but 3, 4, 5 high capacity cables to help meet the demand going forward is a very important part of what's happening right now. Lower latency, as you mentioned, it's a desirable trait always for cables to follow a more direct path to the extent it's possible to do that. But latency alone does not drive new cable construction really. The other, I guess, big angle to bring out, which is I think people are probably pretty aware of this, the main companies building or funding, a lot of the new cables now are the hyperscalers, content providers, cloud provider, whatever you want to call them. It's Google and Meta, Amazon and Microsoft. Those four companies are heavily involved in cables all over the world, have been for some time, and are really trying to link together their data centers and their large user bases globally with their own fiber. They have far more demand than telecom carriers do, internet backbone providers do for capacity. So now they don't have their own cables that other companies have their own cables. Cables are a shared resource. Everybody's using the same cable as though there's like just a cable for the internet backbones, there's a cable for the content providers. Everybody's using the same cables. So everybody benefits from the investments by content providers in the submarine cables that we're seeing deployed around the world.

Phillip Gervasi: So it sounds like these aren't owned by a single private company. It sounds more like a cooperative of companies of investment and then maybe also nations as well, investing for the public good.

Alan Mauldin: Yeah. Most often these cables are built by multiple parties. They aren't just built by one company in most cases. Google has built some cables of which they are the sole owner of, but then other companies do get access to capacity on those cables. So Orange can provide you capacity on the cable between France, United States. So there is the ability for all types of users to get access to these cables, even though Google was the sole builder of the cable.

Doug Madory: There's also a lot of... Again, from doing analysis on this over the years and how much more difficult it is to tease out individual submarine cable incidents, and that's a good thing. But there was a time where I think an Middle East Africa, Tata, you could map, we knew what cables they were investors or owners of, and they would be using those. And if that cable went down, then the service was off. And that's not a scenario that really happens anymore, because they're going to buy capacity on the competing. They may own one cable, but they're going to buy some backup capacity on the competing cable.

Alan Mauldin: Absolutely.

Doug Madory: And so, then when it goes down, it's hard to find the impact, which again, is a good thing. But it wasn't always that way. There used to be a little bit more one- to- one as far as a telecom owning a cable. They would be just using that one cable. That's the end of the arrangement.

Alan Mauldin: That's right. Absolutely.

Phillip Gervasi: So Alan, can you tell us a little bit more about the underlying technology here? I mean, I'm familiar with fiber optics in a broad sense wavelengths and the different types of fiber and that sort of thing. But ultimately, I know that for very long distances, you have the potential for latency problems and for issues with power, just getting that signal down of very long run. And in this case, we're talking about thousands of kilometers and miles of course. And also from a traditional network engineer's perspective, the concern is usually with latency because that directly affects an application's performance. But what you mentioned wasn't latency, what you mentioned was volume. So it sounds like the concern here for you and for the undersea cable industry is moving massive amounts of data and not necessarily with latency.

Alan Mauldin: So the cables that are going long distances have repeaters in them that boost the signal in them. So because of that, they're powered. And there are limits to how many fiber repairs you can put in a single cable across the ocean. You can't put 192 fiber repairs, at least not yet in a long haul cable across the ocean. So currently, what we're seeing is a shift in the industry from trying to boost the capacity carried by a single optical fiber pair to seeing more fiber pairs in the cable. So from the past, you'd have 4 to 6 fiber pairs in a cable, but now we're seeing 12, 16, and even 24 coming. So the newest or the latest greatest cable in the Atlantic coming next year being built by Meta is going to have 24 repairs. And they say it's going to be capable of carrying 500 turbids per second of capacity. So half a petabyte on one cable, is the future design capacity of the system? So we're going to see that going forward, more fiber repairs to boost the capacity. There's other things that are being considered as ways to boost the capacity as well, whether it's multicore fiber or C plus L, things like this that are being looked at as well. But there's constant efforts to try to boost the capacity of the cable because once the cable is in the water, you can change the things at the shore ends, the SLTE, the terminal equipment. You can try to improve things there, but you can't really change the fibers. You can't change the repeaters, at least not very easily. So you want to ensure you're doing the best job you can on day one to get the highest capacity system in the water, because cables are designed to last for a minimum of 25 years. So-

Phillip Gervasi: Oh, is that right?

Alan Mauldin: ...it's a long- term investment.

Phillip Gervasi: So then explain why a cable would be considered old and then retired. You mentioned that earlier. Is it just a capacity problem or physically the cable's deteriorating because some megalodon chewed on it under the water?

Alan Mauldin: Are you implying that sharks eat cables, Phil?

Phillip Gervasi: Oh, I'm hoping they do and that we have video of that.

Alan Mauldin: That is a myth.

Phillip Gervasi: Is it really a myth?

Alan Mauldin: That's a myth.

Phillip Gervasi: Really?

Alan Mauldin: Myth, absolutely it is. Sharks don't bite cables. I mean-

Phillip Gervasi: Sharks don't like cables.

Alan Mauldin: ...years ago, there was a few cases of this happening, but in the last 25 years, there's no cases of any animals biting cables. That cause of fault really is fishing, it's anchors, and there's things like earthquakes and volcanoes like the Tonga thing last year. So why are cables turn turned off? Well, they do have the design life of 25 years, but it's an economic issue as well, not just the cable lifespan. If their capacity can't scale enough to offer a comparable unit cost per turbid of capacity compared to the newer cables, it becomes very expensive to operate. Because with the cable, you are paying a lot of money for the upfront cost to install it, but you're also paying some ONM and some opex every year just to maintain the cable. You have a fleet of ships on standby to go out and fix the cable if it breaks. You have to have that right? And so, if your cable is carrying five terabytes and the other guys cables are carrying a couple of 100 terabytes, your five terabytes are very expensive. So that can make it a candidate for being turned off purely because it's a very expensive cable to run. Technical issues can exist if there's a lot of fault in the cable over the life of the cable, that can impair the ability to upgrade the cable. If the cable also let's say it's very fault prone, maybe you're inclined to just turn the cable off instead of having to constantly deal with having to pay to repair the cable as well. But there's many factors that come into this when you turn off a cable, you have to get all the owners to agree to it as well.

Phillip Gervasi: Yeah. You mentioned that earlier. Sometimes there are, or often there are multiple private and even public investors and owners with an ownership stake in these undersea cables. So my question is, who owns the boats? Who owns the repair and maintenance obligation? And then who makes that decision about decommissioning a cable?

Alan Mauldin: Depends on each cable being different. But I think it's fair to say that different parties could have different values that they are getting from the cable. So some party may think, well, this is one of my three cables that come to my country. For me, it's really important to have this cable. I don't want to turn it off. But somebody else in the cable consortium maybe has 10 cables and they're like, " This is an old one, let's turn it off. I'm tired of paying for it." There could be one party in a consortium who makes a lot of money on the back haul or the cross connects in the cable station. So they don't want to turn off that money sticker, they want to keep that go for their own selfish reasons. So there's a lot of things that go into turning cables off, but we are seeing it happen slowly. As I mentioned with the couple this year, and there's several the last couple of years. So it's the map of cables that we see on our cable map is changing slowly with new ones being added all the time. But also some lines do go away. We don't show cables that have been turned off on our map.

Doug Madory: But Alan, this is better than me. This is a more your industry. But just from being around the space, there's an interesting ecosystem of the people in the submarine cable industry where they're the fabrication companies that fabricate the cable, which may be different than the installer, the people who are going to put it in the ships, in the water. And then there's these various different types of consultants that will help you with the international permitting. There's a bunch of legal things. There's also a business case that you have to... Maybe it's less an issue when you've got these hyperscalers just footing the bill for everything. But I know that when I first started attending these things, it seemed like half the conversations were just around all the economics and making sure you can get this right. Because as you mentioned, there's a big upfront cost, there's a fair amount of risk. The thing could break and now you have to pay to fix it, and you're not making money when it's down. So it could be a risky endeavor that you need to make sure you understand the economics of it.

Alan Mauldin: It is, for sure.

Doug Madory: Anyway, so have I left anybody out? There's like a variety of different types of people involved in making a cable a reality.

Alan Mauldin: That's a very fair coverage of it. But one thing you said about the, does the business side not matter anymore because you have content guys putting the bill? I would disagree somewhat. I mean, because the cables need to be viable, I think. And they do involve parties who aren't these content providers. And so, they have to look at what are their internal demand requirements to serve maybe their broadband customer base, their enterprise customers. Are they going to try in wholesale capacity? At what rates can they sell that at? So we do a lot of work at TeleGeography trying to help companies who want to build new cables and to do their commercial side business plans for them to help look at what are the revenue prospects for your cable? Sure, 25 years is a long life for a cable, but you're hoping that in the next five, six years that it's the right choice and is able to pay for itself. And there's a lot of risk, demand growth fluctuates. Price erosion also goes up and down. There's new competing cables coming online that can take market share from your cable. Lots of variables here that need to be considered when looking at building a new cable.

Phillip Gervasi: Alan, you mentioned that there has been a steady growth in the demand for undersea cable capacity over the years with some ebbs and flows, but ultimately resulting in more cable being laid, including this new one that you mentioned that's crossing the Atlantic. That's a very high capacity. But I have to imagine that this could pose an issue when new cable is being laid in between and among countries that are not necessarily friendly with each other.

Alan Mauldin: Yes. Here we go. I know Doug has written a lot in blog about some of these issues with Cuba and the US recently. Just for some background for the audience, I guess. So a few years back there was plans to develop a new cable between the mainland of the US and Hong Kong, Philippines, and Taiwan. The US government blocked that cable from being licensed, claiming that one, there was some concerns about one of the parties, the ownership having some Chinese government ties. That was one angle. The other angle was that the fact that it just went to Hong Kong, that it went to Chinese territory, that was a problem somehow. And so, a few other cables that had also been planned to go between Hong Kong and the US were then canceled subsequently and are trying to be rerouted or redesigned to avoid this path. And so, that happened to me, it was kind of interesting. I can't speak to the national security threats and this type of thing. But I can say that there are currently cables that exist, that are in service between China and the United States right now. So that already exists, those are in place. Those are still active. Also, data is going between China and the US all the time on cables that do not link the countries directly. Data can go via Japan, which is what the vast majority of it does actually. So by blocking these cables, that doesn't really do anything to change the fact that data is going to still flow between the two countries. That was kind of an interesting point there. And also, recently, the thing that Doug wrote about was that government had blocked the plans to build a branch from a cable that goes to the US branch to Cuba. And so, what does that do? Well, Cuba already has two cables, they're going to have one more soon. It doesn't mean that Cuba can't communicate with the United States. It just means that the data's going to go via a third country. So I'm not really sure what's being accomplished by these plans to block these cables. It just leads to a rerouting of the demand among different places. And I don't see how it creates any additional security for the United States by doing this. But again, I'm not an expert in national security.

Doug Madory: So Alan, maybe I'll jump in here.

Alan Mauldin: Please do.

Doug Madory: So I wrote a piece last month just in anticipation of the 10- year anniversary of the ALBA- 1 submarine cable to Cuba. So just maybe to recap why this is an interesting thing, why it was interesting 10 years ago was that Cuba had been left. There's US embargo against Cuba, it continues to this day. And they had been left out of every submarine cable project in the Caribbean. And if you were to pull up Alan's submarinecablemap. com, you can see there's a big hole in all the wiring where Cuba and Cuba's a big island in the Caribbean. And so, they were stuck on stationary satellite, which is high latency, low capacity, expensive in comparison to a submarine cable link. And so, it's bad on every front. And so, in order to get connected, then the Venezuelan government, so it was Hugo Chávez at the time said they put up the money because nobody else would want to touch it. You don't want to cross paths with the embargo. And so, they built a cable from Cuba to both Jamaica and to Venezuela. And that was the ALBA- 1 cable. Now that was supposedly finished in 2011, but there was no evidence that it had been used or made any difference for a couple of years. And that was around the time that I was trying to get smart on submarine cables and internet measurement. And I just came into these blogs of people speculating, Well, maybe it's only being used in this way, or Maybe it never was built and everybody just stole money or something. There's all these theories going around and Cuba is a bit of a fascination. There's definitely an audience within the United States that's very fascinated with these Cuban related issues. And so, then I got into it and I was like, All right, we'll set something up in my Renesys staff to alert me when we see a new connection into Cuba. And it was like two years later, or 18 months later, I got an email that was an automated thing that I had set up showing this new connection. And sure enough, that ended up being the activation of this cable. But anyway, so then fast- forward to recently, so that cable has been really the one, there's a couple to the Guantanamo Bay. But for the people of Cuba, there's just the one cable that they've been relying on. ARCOS is another Caribbean cable that lands on a number of countries, including the United States. They had applied to get permission to build a little spur over to Cuba. And then in December, the Department of Justice national security team said that the risks were too high and they should be rejected. Now, in that, I kind of got into this in the blog and part of their rationale was some things that also, I know a thing or two about. One is like this would enable them to do BGB hijacks against the United States, and if there was a cable that connected both the Cuba and the United States. And I was like, " Well, the two things have nothing to do with each other." You can-

Alan Mauldin: Exactly.

Doug Madory: ... usually hijack withouthaving a direct submarine cable. And even they were indirectly citing stuff I had written in their rationale. And I was like, "All right, well, I feel like I know a thing or two about this and that." You can do whatever you want, there's always political calculations. And-

Alan Mauldin: Absolutely, yeah.

Doug Madory: ...those folks are going to make their decisions and that's their thing. But let's not make that there's a technical rationale because I don't buy that and I do know something about it. But anyway, that was my soapbox on the Cuba scenario that I wrote up last month. But I don't know, there's not a lot of incentive. Cuba is an adversary in the United States, there's a third hearing government. There's a lot of bad stuff going on in that country. So it is tricky to engage with in the same way that we're trying to figure out how to engage with Iran and what we want to support the people. We want to support internet connectivity and an open internet. But at the same time, you've got an authoritarian regime that may benefit from that, and you have to weigh those costs and then that's where politicians will make decisions. But anyway, it's a tricky area.

Alan Mauldin: I wanted to go a bit more into the China issue because as I said, one of the issues with denying direct cables between China and the US, it doesn't stop the data from flowing between the countries. But one indirect side effect can be that it does decrease the role of Hong Kong perhaps, or China as being a hub within Asia for other countries seeking to connect to China to get access to the rest of the world. So what we've been seeing since those cables have been blocked is everybody's clamoring for what's going to be the next big hub in Asia. Is it the Philippines? Is it going to be Indonesia, Thailand, whatever? Because you have Japan up north and Singapore down south as the two key hubs in Asia right now. We're seeing several new cables that are being built from Singapore directly to the United States as well right now. So you have these two big spans, the Northern Transpac, Japan to the US and South. You're going to have Singapore to the US. And so, that leaves, the question is how is China going to connect? Well, they're going to have their own cables that emanate out of Hong Kong to their countries within their region. But by denying these direct cables, it does make other countries want to use more Singapore and Japan as their main points, their hubs to connect the rest of the world. We're also seeing within Asia, at least one be more planned cables that are Intra- Asian cables that traditionally would go from Japan to Singapore and land in Hong Kong, other spots through the South China Sea. There's a new cable plan called Apricot, it's going to be planned to inter service next year that goes east of the Philippines. So it is a Japan to Singapore, east of Philippines, hitting a few more spots, Guam among them, which seeks to avoid going through the South China Sea and any waters that would be part of the nine- dash line. There's been so many delays with cables that are going in that part of the world. So there's this new cable, the Southeast Asia- Japan Cable 2 that was announced in 2018 to be built by any NEC by consortium of parties including Chinese carriers between Japan and Singapore going to Hong Kong as well. It still hasn't entered service yet, it's been five years. It's mainly laid except for the part that goes through Chinese waters waiting permits and we hear that it could be ready for service next year. But we heard that two years ago that it would be the next year it begins service and it's the next year it's going to be in service. So there's been so many delays. I think people have tried to rethink their network strategy of where they want to risk putting cables to do things that are going to have a higher likelihood of happening as soon as possible. You can't afford some of these risks because the delays to new cable construction, it can really set you back in terms of, I mean, it's expensive having the delays for one. But also if your strategy is to have this new capacity online to meet your demand and it's not there when you need it can be really bad for your network planning.

Phillip Gervasi: Has there been anything that has changed in the undersea cable technology or landscape with regard to the proliferation of public cloud technology, and how data is moving around the world differently than it used to, especially in large quantities?

Alan Mauldin: I think I would go back to what I said earlier about the shifting ownership of these cables and having the companies that are the made cloud companies, owning the cables and having a direct stake in them, not just as owning capacity in them. They want to increasingly have a role in where the cables go and how they are designed. And they're also helping to push the envelope in terms of the capacity of the cables, because for them, cables are a cost. They don't earn enough cables, it's a cost for them. So their entire goal is to let's lower the cost per terabyte as much as possible. So that's been a big change in having just the cloud growth does help to fuel these kinds of hidden parts of the network, the subsea cables that are how the cloud regions are linked together via the cables.

Doug Madory: Alan, one interesting dynamic, it's interesting to hear all discussion about the demand, and that's really cool. I mean, at one point, and this is dating myself maybe 2014 or something, you guys were tracking, I don't know, maybe a bit of a lull in up until the right graph of summary cable demand. And part of the rationale was there was a time when domestic connectivity in a lot of countries was not what it is today. And so, you'd have this crazy hair pinning, the Middle East is a big example. Asia was also another one where two service providers in one country would exchange their traffic in London or something really far away. And so, they were kind of artificially pushing up the demand on submarine cables. And as that got resolved, as there was more domestic connectivity and these guys were exchanging traffic in the country, then it was something that would just maybe take the foot off the pedal, maybe just slightly. And then there's also some of the OTT services being able to... All the innovation around caching where stuff isn't just going off, isn't always an international connection to services, say Netflix or something. You've got local cashes serving up, it becomes a far more efficient system of how do you deliver traffic. That too seemed like there was a couple of factors there, but it sounds like that was just a little hesitation and then it just went right back into unbounded growth.

Alan Mauldin: I think kind of what you're getting at also, I mean, what you said is absolutely true. I think one thing that we did see in our data was... So we gathered data each year from the major internet back operators about their international network, the capacity they have between various cities. And I think we first saw some really slow growth in the Atlantic, and I forget the year, maybe it was 2012, 13 around or 14 around there. And it was kind of strange because it had been a pretty solid clip for many years. And what was happening was at that point in time, it was the content providers who started to stop using or having the internet cable providers carry from their traffic and they started to deploy their own private networks across the Atlantic. So this was first done by them acquiring wavelengths capacity across different cables. Then it came to them acquiring entire fiber pairs on older cables. And then it came with to them investing in new cables. And so, we started to see this big shift happening. And you've seen it in other markets around the world where we've seen this kind of lull sometimes in the biband operators, their traffic or their capacity they deployed, it hasn't grown for a few years and we're like, " What's going on?" They're like, " Well, the content guys moved in, they put all their stuff locally and now they're have their own transport going back to their data center in Europe. So we don't need to have as much ourselves." And that still happens because the cloud providers, they haven't deployed their backbone everywhere in the world yet. But whenever they do, there is this kind of a transition period where you see some shifting from the demand, from the major carrier networks onto the content power networks for the long haul portion.

Doug Madory: But your point is that that traffic is always there. It's just your collection methodology may not capture it.

Alan Mauldin: Well, we do track the content growth as well. So in our overall demand data is definitely showing all categories of demand. But your point was, I think a good one about there was some things that were international that didn't need to be, that things were hair pinning in London. And I think that's being resolved as you see more local internet exchanges globally and people agreed to create a pair. But there are some parts, the world worldwide carriers who don't want to pair with each other because they don't get along. And so, there is data exchange maybe in a third country.

Doug Madory: Yeah. Actually, there's a rational argument, why that takes place. Like you said, even a powerful incumbent doesn't want to appear domestically with its private competitors. And if they have to send their traffic out of the country, too bad for them because that hurts them or that hurts the incumbent. And then you end up with this weird, when we look at trace routes in the Middle East, we'll see too Saudi providers exchanging traffic in London or New York City. Middle East is a good example, but I mean, Mexico is too, that's our neighbor. And that's another scenario where you have a very dominant incumbent that exchanges a lot of their traffic in Texas and Los Angeles. A lot of it ends up having come back into the US to go back into Mexico.

Phillip Gervasi: So is there any built in logic, any intelligence in the undersea cable infrastructure? I mean, I'm just thinking in terms of how one cable may touch many different regions or many different countries and you might want to be strategic in where data can go, for example.

Doug Madory: There's not a lot of logic in the submarine cable itself. I know I remember at one point seeing people propose like, " Oh, what if the submarine cable itself was routing channels?" I was like, " Ah, not sure that that's going to work." So another misconception is that when you see a submarine cable, you pick one off of submarinecablemap. com and you see it land in these 11 countries or whatever it is. The one interpretation is it's like a bus, everybody's got access to everything kind of thing. And then I feel like if you can only show so much on that graphic. But in reality, you've got pairs that are going between different cable landing stations that aren't necessarily traversing the entire cable. So when there was a break or something breaks, it's not obvious sometimes where the impacts are going to show up. But I mean, I've only just learned this just from trying to understand from a data standpoint.

Alan Mauldin: Yeah, so I think what you're saying... Actually, there's two points here to make. One is you're right, the fiber repair mapping of a cable can really vary. There can be express fibers that don't hit every spot in the cable that you see. This happens a lot with cables on the coast of Africa or cables between Europe and Asia. So some fibers just drop in certain places and the capacity of those branches might have a totally different capacity than the major trunk of the cable. So it's very imbalanced in some ways. But I think your point's a good one. I think we're getting at also is that just because a cable might hit, let's say every country, it's how the operators choose to deploy their backbones and maybe their IP routers are just, let's say in the Dakar and London. They're not in every spot along the way, so they can't directly access Nigeria from Dakar. They have to go to London first to then send data to local. So that's the other issue.

Doug Madory: Oh yeah, I guess maybe that's a better example of, let's take like one of the submarine cables. You're going to have a landing in Mumbai and Karachi, which are adjacent on a submarine cable map. But you can't just go onto the cable and then go to the next, like you would on a subway, just get off at the next stop. You may have to go all the way to the end and come all the way back to go to the next stop.

Alan Mauldin: Yeah. The first question, is it possible to go direct? And if it is, you have to then have a carrier who will deploy that link and make it actually inner service.

Doug Madory: That's right. It's physically possible, layer one, and then there's a layer three or something, another layer that also has to make it happen. So for anybody who's followed internet connectivity in the country of Vietnam over the past decade, you know that they have a lot of submarine cable issues. In fact, when every time I see a headline that Vietnam's internet is crippled due to some submarine cable outage, I have to check the date to make sure this is the new one or this is the last one or the one before that, it's multiple times a year. Was there just some poor mapping planning of cables? What's the explanation for the woes of Vietnam's international connectivity?

Alan Mauldin: I think the heavy reason for that is this, the cables that go into Vietnam are in waters that are very shallow and are heavily fished, and there are lots of anchor issues as well. Vietnam does get the majority of its connectivity from international submarine cables. They do have terrestrial fiber as well, which does meet some of the requirements. But most recently you're right, four of the five cables were out for at some length of time, and they do have new cables planned. And one of those cables I mentioned earlier is the SJC cable that's been delayed for five years now or whatever. So they should have some new connectivity in the next couple of years with some new higher capacity cable, which will certainly help to augment what they currently have. Hopefully they're going to be more reliable, though.

Doug Madory: I don't know. If it's just shallow waters anchors and fishing, then could the new cable be more resilient to that?

Alan Mauldin: I mean, I would have to... I'm not aware of the exact routing of the cable, the burial depth of the cable, the armoring type on the cable. But all these measures are routinely used to help guard cables, you have more armory on the cables that are in the shallow waters. You bury it where you can, two to even three meters deep so that anchors can't really touch them, hopefully.

Doug Madory: I would imagine this is probably foremost on the mines of the people installing whatever's the next cable that they need to get. These are dangerous waters for submarine cables.

Alan Mauldin: For sure. And I think one of the best things to help to improve the quality of the internet is just to have more cables safety in numbers. That is a strategy that seems to work in many parts of the world.

Phillip Gervasi: Alan, I think I'm going to stop us here. We are at time, but I do want to say thank you very much for joining and talking about your experience and knowledge and undersea cables with us today, very interesting. As someone who has been a traditional network engineer for many years, my perspective is just much more local. So it's very interesting to hear from you what goes on when I configure routing at the local level and then send traffic across the entire world to access information somewhere else. Really interesting to hear. Now, for our audience, I will put a link to the TeleGeography website and to the submarine cable map. It's thoroughly interesting to look at, pinch and zoom and kind of check out what's going on there. Now, Alan, if folks have any questions for you or would like to learn more or have a comment, how can they reach out to you?

Alan Mauldin: For me personally, the best way would be via LinkedIn. But also, I would suggest going to our blog, which is blog. telegeography. com. We have a wide variety of things we post there with the latest presentations or thoughts we have on different things happening in the industry.

Phillip Gervasi: Excellent, thanks. And Doug, how about you?

Doug Madory: Let's see. I'm on Twitter @ DougMadory. I'm also on Mastodon now @ DougMadory again. And then LinkedIn is another good way to reach out to me.

Phillip Gervasi: Great. Thanks Doug. And you can find me on Twitter @ network_phil, still very active there. And you can search my name Philip Gervasi on LinkedIn. You can also follow Telemetry Now on Twitter and LinkedIn as well. Now, if you have an idea for a show or you'd like to be a guest, we'd love to hear from you. Email us at telemetrynow @ kentik. com. So until next time, thanks for listening. Bye- bye.