Ukraine’s Wartime Internet from the Inside
Summary
It has now been over a year since Russian forces invaded its neighbor to the west leading to the largest conflict in Europe since World War II. Kentik’s Doug Madory reviews what has happened with internal connectivity within Ukraine over the course of the war in this analysis done for a collaboration with the Wall Street Journal.
This February marked a grim milestone in the ongoing war in Ukraine. It has now been over a year since Russian forces invaded its neighbor to the west leading to the largest conflict in Europe since World War II.
In the past year, we have used Kentik’s unique datasets to show some of the conflict’s impacts on Ukraine’s external internet connectivity, ranging from DDoS attacks and large outages, to the rerouting of internet service in the southern region of Kherson.
This blog post contains analysis done for a collaboration with the Wall Street Journal (pictured below) using a novel data source that allows us to explore connectivity inside Ukraine: the Ark dataset from the Center for Applied Internet Data Analysis (CAIDA).
Domestic measurements using the Ark dataset
Based at the University of California San Diego, the Center for Applied Internet Data Analysis (CAIDA) is a leader in the academic field of internet measurement. Among their numerous measurement projects is the Archipelago Measurement Infrastructure, or Ark, for short. Ark consists of servers located around the world continuously performing traceroutes to randomly selected IP addresses.
One of those Ark servers is located in Kyiv. For the past year, it has been dutifully performing measurements to IP addresses around the world, including destinations within Ukraine and Russia. CAIDA graciously provided the data generated from this server to Kentik for the following analysis.
The data gives us a unique view into the internal connectivity within Ukraine over the course of the war — at least from the perspective of this one important internet connection in Kyiv.
Arguably the most dramatic development that appears in the data was the rerouting of internet service to Kherson through Russia. To analyze this development, we extracted the traceroutes performed by the Ark server in Kyiv to IP address space originated by the ASes of Kherson.
The data from these traceroutes is plotted below by the overall latency (y-axis), time of measurement (x-axis), and AS origin of the last responding hop (color).
There is a clear point when the latencies increase due to the Russian rerouting at the beginning of June 2022. As one would expect, this aligns with the timing observable in BGP. The above graphic also illustrates the result of the Ukrainian liberation effort in Kherson. Ukrainians have recaptured half of the region, and we see a portion of the traceroutes reverting to a lower latency as those networks restore their Ukrainian transit connections. A few providers in the region of Kherson are still on Russian transit, presumably in the territory that is still under Russian control.
If we isolate the measurements to one particular ASN in Kherson, AS21151 (Ukrcom), the traceroutes tell a clear story of the network’s transition from Ukrainian to Russian transit and back again.
If we convert the individual traceroutes used in the above graphic into a readable version, we can see how the internet path between Kyiv and Kherson changed hop-by-hop.
Before the invasion, packets could get from Kyiv to Kherson in as little as 11ms:
Traceroute from 88.81.224.210 to 185.43.224.143 at 2022-01-20 00:34:58 (COMPLETED):
1. 88.81.224.209 PJSC Datagroup 3326 88.81.224.209.ipv4.datagroup.ua. 0.999
2. 185.1.62.22 UBNIX-GLOBAL-IX 199524 uarnet-ix.giganet.ua. 0.724
3. 194.44.100.254 http://www.uar.net 3255 14.859
4. 193.109.128.78 Ukrcom Ltd 21151 12.551
5. 193.109.128.182 Ukrcom Ltd 21151 193-109-128-182.ukrcom.kherson.ua. 11.824
6. 185.43.224.143 Ukrcom Ltd 21151 185-43-224-143.ukrcom.kherson.ua. 11.883
During the Russian occupation, traceroutes needed to leave the country to reach Kherson. In the example below, this traceroute travels from Kyiv to Vienna win to Frankfurt ffm to Moscow 12389 to Simferopol, Crimea smfl and finally to Kherson 21151.
Traceroute from 88.81.224.210 to 194.79.22.56 at 2022-06-13 18:52:08 (GAPLIMIT):
1. 88.81.224.209 PJSC Datagroup 3326 88.81.224.209.ipv4.datagroup.ua. 0.504
2. 88.81.244.145 PJSC Datagroup 3326 xe-1-0-2.2.at.mx02.iev2.core.as3326.net. 0.62
3. 62.115.189.58 TELIANET 1299 kiev-b1-link.ip.twelve99.net. 0.825
4. 62.115.123.130 TELIANET 1299 win-bb4-link.ip.twelve99.net. 32.405
5. 62.115.138.22 TELIANET 1299 ffm-bb2-link.ip.twelve99.net. 31.878
6. 62.115.151.97 TELIANET 1299 rostelecom-ic319651-ffm-b11.ip.twelve99-cust.net. 64.392
7. 87.226.183.91 PJSC Rostelecom 12389 46.853
8. 185.64.45.207 Miranda-Media Ltd 201776 76.742
9. 31.40.132.165 Osipenko Alexander Nikolaevich 201776 ae20-13623.smfl-04-bpe1.miranda-media.net. 94.087
10. 193.109.128.17 Ukrcom Ltd 21151 193-109-128-17.ukrcom.kherson.ua. 103.082
11. 193.109.128.78 Ukrcom Ltd 21151 105.668
12. 194.79.22.26 Ukrcom Ltd 21151 194-79-22-26.ukrcom.kherson.ua. 95.683
As a consequence of the greatly increased geographic distance traveled, the overall latencies from Kyiv to Kherson jumped up to over 70ms - greater than a round-trip time across the Atlantic Ocean.
Following the liberation of Kherson, traceroutes revert to a shorter, more direct path:
Traceroute from 88.81.224.210 to 194.79.22.43 at 2022-12-06 22:50:52 (GAPLIMIT):
1. 88.81.224.209 PJSC Datagroup 3326 88.81.224.209.ipv4.datagroup.ua. 18.287
2. 185.1.213.30 UA-EUROLINE-20210701 None compnetua.1-ix.net. 0.41
3. 193.109.128.137 Ukrcom Ltd 21151 du-137.ukrcom.kherson.ua. 10.254
4. 193.109.128.78 Ukrcom Ltd 21151 10.253
5. 193.109.128.82 Ukrcom Ltd 21151 nat-polar.ukrcom.kherson.ua. 9.983
Not every Kherson AS switched back to Ukrainian transit. Russia still occupies half of the region, and a few ASes operate in that half. RubinTelecom (AS49465) is one of them and remains on Russian transit. It also suffered extended outages.
Connectivity to Donbas
Measurements to Russian-held Donetsk and Luhansk also exhibited clear changes in latency and path. We can’t be sure whether these changes were due to a technical failure along a more direct path or an administrative disabling of the link. On February 17, 2022 — one full week before the invasion — we saw latencies to Luhansk double from 35ms to 70ms.
Looking at the individual traceroutes shows a clear change in path. Initially, the measurements heading to Russian-held Luhansk headed directly to the Moscow Internet Exchange (MSK-IX) by way of Kharkiv on Ukraine’s northeast border with Russia, as illustrated in the traceroute below:
Traceroute from 88.81.224.210 to 176.113.251.26 at 2022-02-14 19:39:45 (COMPLETED):
1. 88.81.224.209 PJSC Datagroup 3326 88.81.224.209.ipv4.datagroup.ua. 0.244
2. 88.81.244.175 PJSC Datagroup 3326 xe-11-1-2.2.at.mx01.iev1.core.as3326.net. 31.632
3. 88.81.244.1 PJSC Datagroup 3326 xe-0-1-0.2.at.mx01.hrk1.core.as3326.net. 35.792
4. 195.208.210.92 MSK-IX None 42.23
5. 193.228.160.179 Telematika LLC 43201 33.902
6. 194.31.154.3 Luganet 39728 pool.luganet.ru. 34.875
7. 176.113.251.26 Luganet 39728 35.3
After the change, traceroutes had to head west to DECIX in Frankfurt before getting carried to Moscow in Rostelecom and then on to Luhansk in Eastern Ukraine.
Traceroute from 88.81.224.210 to 176.113.242.30 at 2022-02-17 23:44:13 (COMPLETED):
1. 88.81.224.209 PJSC Datagroup 3326 88.81.224.209.ipv4.datagroup.ua. 0.583
2. 88.81.245.221 PJSC Datagroup 3326 xe-3-3-1.2.at.mx01.fra1.core.as3326.net. 30.813
3. 80.81.194.31 DE-CIX Management 6695 frkt-ar2.intl.ip.rostelecom.ru. 32.264
4. 188.128.126.238 PJSC Rostelecom 12389 94.839
5. 178.35.228.91 PJSC Rostelecom 12389 67.917
6. 193.228.160.179 Telematika LLC 43201 94.294
7. 91.217.5.131 Luganet 39728 pool.luga.net.ua. 67.113
8. 176.113.242.30 Luganet 39728 69.986
Datagroup’s connection to Russia via Kharkiv appears to have been disabled ahead of the invasion, perhaps as a measure to thwart cyberattacks from Russia.
In mid-March, traceroutes from Kyiv to Russian-held Donetsk exhibited a similar leap in latency, from as low as 34ms to over 80ms:
Connectivity to Russia
Below is an illustration of the traceroutes from the Ark server in Kyiv through Russian state telecom, Rostelecom (AS12389), to any global destination. These are traceroutes that received responses from the target addresses (i.e., status = COMPLETED) and, again, are plotted by the overall latency (y-axis), time of measurement (x-axis), and AS origin of the target IP (color).
The chart above reveals that traceroutes destined for address space originated by Rostelecom (AS12389, AS42610, AS25490, etc.) and many other Russian networks became unreachable. But the traceroutes didn’t begin and then stop at a certain point. Instead, beginning around 09:00 UTC on February 28, 2022, Datagroup appeared to simply no longer carry any routes from many Russian ASNs in its routing table.
The Ark server did, however, continue to successfully run traceroutes through Rostelecom on to other countries: notably Kazakhstan and Iran. In the graphic above, traceroutes to Kazakh Telecom (AS9198) appear in green, while traceroutes to Iran’s Information Technology Company (AS58224) appear in purple. This is to be expected as both countries normally utilize Russian transit.
The lower band of measurements in yellow and dark gray represent successful traceroutes into Russian-held Donbas and Crimea — perhaps demonstrating a reluctance to block traffic to any parts of Ukraine, even if occupied by Russian forces.
Conclusion
The analysis above only scratches the surface of what is contained in the measurement data that was produced by CAIDA’s Ark server in Kyiv. Traceroutes performed from inside Ukraine can reveal changes in latency and path for domestic traffic that isn’t possible to observe from outside of the country.
In the case of Kherson, the internal view provided by Ark revealed the performance impact of forcibly re-routing domestic traffic through an external country. Of course, the issue of increased latency pales in comparison to the security concerns that arise from having one’s communications re-routed through an invading country.
Recall Skynet (formerly Kherson Telecom) was the first ISP in Kherson to begin using Russian transit in May of 2022. It was an action that the CEO defended on social media. After the Ukrainian counteroffensive in the fall of 2022 recaptured much of the region of Kherson and ISPs there started using Ukrainian transit again, Skynet went dark. Since then, it hasn’t appeared in the global routing table.
The war in Ukraine has devastated the country, and its telecommunications infrastructure, while still operational, has paid a heavy price. For most of Ukraine, the Russian-held Donbas got further away, both in terms of internet distance and national cohesion, while much of the Russian internet became unreachable.
Ukrainian telecommunications technicians are continuing to face unforgiving challenges while working to keep Ukriane connected to the outside world and, of course, from the inside.
