How to Perform a Forensic Analysis After a Security Breach

Kentik's visibility into public cloud provides both a macro view of how your cloud traffic is moving among and within regions but also a powerful tool for granular visibility for very specific traffic moving among regions, countries, data centers, and down to specific applications and endpoints. Having a single view of all these cloud components, including their connections to on-prem resources means that engineers have the ability to identify compliance issues, spot the inefficient movement of traffic among cloud resources, troubleshoot application performance problems, and even perform a forensics analysis after a security incident. So let's take a look at an example, a hypothetical scenario in which our security tool alerted us to a possible security incident. So first, we need to verify that this is indeed a likely security breach and not a false positive, and then we'll need to analyze how the attacker gained entry, what specific devices were compromised, and the extent of the data exfiltration. So to start things off, we got an alert from our security tool about a suspected attack. Our public-facing hosts in our Azure East US region was being hammered on port 22 for a short amount of time. Now a spike is gonna show up as anomalous behavior in any SIM, so that's a pretty standard alert. But what we need to do now is figure out what happened, and this is where Kentik's ability as a forensics tool comes in. Now the Kentik Map, which you see on your screen, is a great way to see an overview of all of your sites, on-premises private data centers and branch offices, your internet connections, and all of your public cloud resources. Seeing this data overlaid on a map, it allows us to see clearly how traffic is crossing national boundaries and that has very important compliance implications, especially for highly regulated industries. So for our scenario, we know which host was compromised. We know it's in Azure in our East US region, and from our SIM, we know the IP address 10.170.50.5. So we have a place to start. Now there's a few things we wanna figure out here. What's the source of the attack by IP address? Where is the attacker in the world geographically? We wanna know if there's any other compromised devices, especially internal devices that don't have a public internet connection. And lastly, we wanna know if there's any evidence of data exfiltration. So I'll start by drilling down into our East US site and see if there's anything that we can see there that can help start us off. So zooming in, we can see what resources we have in that location, and we can see Azure East US on the list. Clicking on that, we can get our quick view traffic summary for the last few weeks, which shows us a spike in traffic that does look pretty abnormal, so we're heading in the right direction. Now if we wanted to look at how traffic was moving for all of our Azure instances, all of our regions. We can go to the top of our map, select Azure, and then view the topology from there. And that's including, on-premises sites, our ExpressRoute circuits and of course all of our regions. So scrolling down, we can hone in on our host subnet here in East US and get a quick glance of the traffic there. So here we can see the spike. Can get various traffic metrics. We can see information regarding our network security groups and so on. We can also look at the connections that this subnet is making right here. And we can also look at the connections that we're making in and out of the entire region as well. So I'll go back a screen and select view topology to look into our entire Azure region. I'm gonna expand the time range to the last two weeks and notice here the spike in traffic, which lasted only a very short time. So I wanna drill down and explore the underlying data. What happened? What I'm gonna do now is open up Data Explorer from the main menu. I'll start by running a query to see what was hitting our host a little over a week ago. I wanna track down this IP. From the data sources, I'll select Azure since our host is in Azure and we can use those logs. But notice that we can select multiple data sources or even all if we want. From the dimensions menu, I'll select source IP, source port, destination IP, the destination region in Azure I wanna see the firewall action and I also want the source country from our geolocation option because I wanna know if data is leaving our Azure region across international boundaries, and I also wanna know the application. Now I'll also modify the filter so we can narrow down the scope to just our Azure host. I'll change my filter so the destination host's address is 10.170.50.5. And this is our inbound destination, and we're gonna leave the source blank to capture everything. This should get us a good picture of what outside IP addresses are trying to make connections to our host, at least in that time range. So let's change the time range to the last few weeks since the attack happened over a week ago and thankfully Kentik allows us to go back in time when we query the data. Okay. So we see here basically a lot of connection attempts to our host over SSH. Now this is a public facing host in Azure, so it's understandable that we see all allow statements through the firewall, but almost every single one of these has almost no data transfer. So to me, I would suspect that these are likely TCP resets after failed login attempts. Now we do have the IP address here of what I assume is our attacker, and we also see that we have the source country, Singapore, in this scenario. So we are making progress. And we also have one float here in particular with a bunch more traffic So we have evidence of suspicious activity, which does support the security alert. And, tracking down network activity this way gives us a point in time that we can then use to isolate the logs locally on the host itself to look at the actual login attempts. So, basically, our host was targeted over the internet on port 22 over and over again using different source ports, which are SIM-identified as an indicator of compromise and when we confirmed it here presumably using different login credentials until finally the attacker was able to gain entry. Now SSH is extremely light traffic. So to me, this suggests that there's probably some tunneling activity happening here, maybe with SCP or SFTP or some other data transfer protocol, that you can tunnel within an SSH session. So now that I know the attacker's IP address and geographic location and the most likely method of getting in I want to drill down on our compromised host to see if it talked to anything internally that may also have then been compromised. Looking at lateral movement is one of the areas of visibility where Kentik really shines. Let's change our filter to look at source and destination IP and application but this time we'll filter with the source being our host at 10.170.50.5. And the destination, I'll leave blank, except I will exclude our attacker's IP so that we only see what our host talked to laterally within our own network. Now we could also apply a traffic profile to isolate only outside-to-cloud traffic and cloud-to-outside traffic. So we can apply our filter here and run the query. And we can see that there are a bunch of connections happening. Notice that we have a lot of connections made to this one inside host at 10.170.100.10, all with very minimal data transfer on a variety of ports. Now to me, this is definitely indicative of a scan of some type. It looks like our Azure host on the outside is trying to make a connection to something internally. And we can see here a significant amount of MySQL data transfer right at the top. So let's do one more thing and change the filters to focus on just these IPs and MySQL and SSH traffic in a time series to compare them. Basically, I wanna isolate what I believe to be our data exfiltration activity. So editing the filters, my first filter rule isolate the lateral traffic within Azure. So that should be MySQL. And my second filter is gonna isolate the traffic between my Azure host and the out side and then the attacker, which should be a similar amount of SSH traffic. And for this visualization, I can also enable bidirectional mode to more easily see the correlation on the graph. And if you haven't noticed yet, you can filter any way that is relevant to you. Remember that the underlying database here is all of the telemetry that's ingested into the system. And so through Data Explorer and the Kenik platform in general, you have the unbounded ability to explore all of the data that's in the system. And so we can see here very clearly that the spike for both MySQL inside and SSH Outbound happen at the exact same time. And the two are very close in the actual amount of traffic. So let's review what we did in this demo. We were able to start by confirming our alert from our security tool was valid and not a false positive. We verified that an attacker scanned our outside host and identified port 22 as open. Now we assume that the attacker then initiated a brute-force dictionary attack as evidenced by the many SSH connections with minimal to zero traffic. Followed by a successful attempt with significantly more traffic. Then with this open connection, our attacker was able to scan and identify an internal host running MySQL. We saw the data transfer internally, and then we saw the data transfer back out to the attacker. Now at this point, I'd hand this off to our security team to look into system logs themselves. Probably revisit SSH credential policy on our outside host and MySQL credentials on the inside host. And depending on what they find, I might also recommend implementing two-factor authentication and something like fail to ban to prevent brute force attacks. Ultimately, visibility is a cornerstone of effective cybersecurity and Kentik's data-driven approach to network observability provides engineers the tools that they need to make informed decisions to remediate problems and improve their security posture. For more information, visit kentik.com/security, and you can always speak to your local Kentik representative as well.