Network observability is a core part of systems administration. No matter your line of business, and no matter your number of users, you need to know what’s happening on your network. There are a number of good reasons to invest in better network observability. If and when you experience a network outage, high-quality observability means you’ll diagnose the issue more quickly. Network observability gives you a tool to detect malicious intruders in your environment. Regularly inspecting network traffic can even help you identify network bottlenecks and improve the day-to-day experience for end-users on your network.
In this post, we’ll talk about one of the most popular means of network observation: port mirroring. We’ll dive into what it is, how it works, what it’s good at, and the drawbacks as well as best practices in using port mirroring.
The concept behind port mirroring is quite simple. When you configure a switch, you reserve one port. Then you configure the switch to “mirror” all traffic that passes through to that reserved port. Whenever the switch processes a packet, it makes a copy and sends it to whatever is connected to the aforementioned port. Usually, this will be some kind of dedicated system set up to monitor the traffic on that switch. SPAN (Switched Port Analyzer) is a Cisco-specific way of handling port mirroring. For the purposes of our discussion, we can use these terms interchangeably, but you should keep in mind that every network vendor provides some sort of port mirroring.
As you dig into SPAN configurations, it’s important to understand what SPAN setups can and can’t do. For starters, they can’t tell you about any traffic that doesn’t route through the switch you’re configuring. That just makes sense, right? When you’re configuring SPAN, it’s also essential to understand how network traffic passes through your network. If you configure SPAN on the wrong switch within your topography, you’re going to wind up missing packets that you want to see. Fortunately, specific SPAN implementations don’t mean that you’re confined to a single physical switch.
RSPAN takes our SPAN configuration from earlier and works across a dedicated VLAN tunnel. Traffic going from one switch to another moves along a dedicated tunnel. When you configure your switch, you dedicate a VLAN (one or more ports) as an RSPAN VLAN. Now all traffic that passes along switches within that tunnel will be copied to the RSPAN VLAN. Much like a traditional SPAN configuration, the switch copies all traffic. The important thing to know about RSPAN is that all the switches involved need to be on the same physical network. RSPAN is an OSI Layer 2 configuration. It doesn’t support routing traffic through Layer 3.
If you read the previous paragraph, you can probably guess why ERSPAN exists. While RSPAN only supports Layer 2 routing, ERSPAN supports Layer 3. When you enable ERSPAN, you gain the ability to route mirrored traffic across multiple physical networks. This provides a real benefit for organizations with multiple geographically distributed network environments. Unfortunately, ERSPAN is a Cisco-proprietary feature. It’s only available on certain models. Those limits remove a lot of choices if ERSPAN is a feature your team needs.
Let’s start with the most obvious benefit of port mirroring: the functionality is available on your switch. Compared to something like a network tap, port mirroring is easy and cheap to configure. You don’t need any additional hardware. This makes port mirroring particularly valuable when your network configuration is constrained by physical space or when you might only need to monitor a VLAN for a short period of time. Instead of needing to get into a cage and physically install or remove hardware, you’ll just need to modify your switch configuration. This ease of configuration and lack of up-front cost makes port mirroring an attractive proposition for organizations taking their first steps toward network observability.
As an additional bonus, port mirroring is effectively invisible on your network. If you introduce a dedicated network tap, that’s another device you need to maintain and support. While dedicated network taps rarely fail, they’re still a potential point of failure. Enabling port mirroring on a switch makes it no more likely to fail than any other switch. And, as noted, port mirroring works across multiple switches. A device like a network tap requires installation connected to every switch you want to monitor. But by far, the biggest benefit to port mirroring is that it’s so simple and quick to set up.
While port mirroring is cheaper and quicker to set up, it does carry some real drawbacks. The most significant is that the switch will treat mirrored traffic as a lower priority. While the CPU overhead of copying any individual packet and mirroring it to a port or VLAN is low, those costs add up. So the switch treats each mirrored packet as a lower priority than normal network traffic. During low-to-medium traffic periods, this isn’t a big deal. The switch capably handles both the normal traffic and the mirrored traffic. However, when traffic flow loads up, things can get hairy. The switch will drop mirrored packets first. This means that you’re most likely to lose some network observability during the time when you need it the most.
Moreover, the reduced priority of mirrored packets makes some network observability goals more difficult to achieve. For instance, if your network observability goals include things like reducing network jitter or latency, port mirroring might not be a good fit for your goals. This is because the delay in delivering mirrored packets can make those time-sensitive issues more difficult to detect and resolve.
Another downside to port mirroring is that it requires resources on physical or virtual appliances. These can be costly from a hardware and (in the case of commercial solutions) software licensing point of view. As a result, in most cases, it is only fiscally feasible to deploy forms of port mirroring at selected points in the network.
A cloud-friendly and highly scalable model combines the deployment of lightweight host-based monitoring agents that export packet capture statistics gathered on servers and open source proxy servers.
If you’re jumping into port mirroring for the first time, here are some best practices you want to keep in mind:
Port mirroring is a great low-cost option for tapping into a source of network telemetry. Especially if you’re just starting your journey into high-quality network observability, it’s easy to configure and enable out of the gate. There are no big up-front costs, and enabling port mirroring is something you can do on your existing hardware today. That doesn’t mean you should just jump in with both feet. Before you get started, you want to intimately understand how your network traffic flows. Where is your most critical traffic coming from? Where is it going? Once you answer those questions, it’s easier to know where to set up port mirroring in your existing network.
It’s also critical to understand that port mirroring isn’t a silver bullet. As we noted, while it has some real upsides, it also comes with some real downsides. Given that, you want to use port mirroring as just one tool in your toolbelt. Unless your network is very small, don’t expect that setting up port mirroring will solve all your problems. You can’t just set it and forget it. Good networks, like gardens, require tending and maintenance to flourish. If you’re interested in learning how to tend your network so that it runs its best all the time, we’d be happy to show you how we can help.