What are VPC Flow Logs?

VPC Flow Logs are a feature provided by cloud providers like Amazon Web Services (AWS) that allows users to capture information about the IP traffic going to and from network interfaces within their Virtual Private Cloud (VPC). VPC Flow Logs are used to gain insight into network traffic patterns, monitor network performance, and enhance security by detecting unusual traffic behavior. In essence, VPC Flow Logs (also known as cloud flow logs) serve as a valuable tool for NetOps professionals and cloud architects working in cloud, hybrid cloud, and multicloud environments.

In cloud computing and cloud networking, flow logs are equivalent to the network flow records (e.g., NetFlow, sFlow, etc.) generated by devices in physical networks. Various cloud network components such as a VPC, a subnet, a network interface, or a transit gateway, can generate flow logs. These logs can be published to a network storage location (such as an Amazon S3 bucket) at various intervals. The logs can then be ingested into monitoring and observability solutions (such as the Kentik Network Observability Cloud) for further analysis and visualization.

Understanding VPC Flow Logs Across Major Cloud Providers

VPC Flow Logs are a crucial feature for monitoring and analyzing network traffic across various cloud platforms, including Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. Although each platform has its terminology and specifics, the underlying concept remains the same.

What are VPC Flow Logs used for?

VPC Flow Logs are a network monitoring tool that captures and records information about IP traffic between network interfaces within a virtual private cloud (VPC). Depending on the platform, the logs can be published to different destinations, such as log management or storage services. (See “A Step-by-Step Guide to Writing VPC Flow Logs to an S3 Bucket” for a tutorial on exporting flow logs in AWS.)

Once published, you can analyze the flow logs for various purposes, including network optimization, network performance monitoring, security analysis, and troubleshooting.

Key Cloud Flow Log Features and Capabilities

While implementation details may differ among cloud providers, several core features and capabilities are common across AWS, GCP, Azure, and other virtual private cloud providers:

• Scope: VPC Flow Logs can be created for different levels of granularity, such as VPC, subnet, or network interface. This flexibility allows you to monitor traffic according to your specific needs.
• Traffic Types: You can configure VPC Flow Logs to capture different traffic types, including accepted, rejected, or all traffic, which enables more focused analysis and monitoring.
• Performance: Collecting and generating VPC Flow Logs does not affect network performance or latency, as the logs are generated outside the network traffic path.
• Log Destinations: VPC Flow Logs can be published to various destinations depending on the cloud provider, such as log management services (e.g., Amazon CloudWatch Logs, Google Cloud Logging) or storage services (e.g., Amazon S3, Google Cloud Storage).
• Log Format: Flow logs are typically stored in a structured format, such as JSON, containing fields that describe the traffic flow, including source and destination IP addresses, ports, protocols, and more.

What do VPC Flow Logs Show?

A flow log consists of a set of records about the flows that either originated or ended in a given Virtual Private Cloud, with each record composed of a set of fields that provide information about a single flow.

For example, in the default AWS flow log format, each line of the log is a space-separated string with fields that describe an individual flow. These fields include the flow log version number, account ID, interface ID, source and destination IP address, source and destination port, the network protocol, number of packets and bytes transmitted, the start and end times for the flow, the specific flow action, and a log status indicator.

The format of VPC flow logs may vary depending on the cloud provider, and it is also possible to create custom log formats.

Use Cases for VPC Flow Logs

VPC Flow Logs can be used for a variety of purposes, including:

• Network Monitoring: Flow logs provide real-time visibility into network performance, enabling you to monitor traffic levels and bandwidth consumption and diagnose potential issues. Network monitoring and network observability solutions like Kentik Cloud can provide real-time visibility into VPC network traffic that is otherwise hard to understand and visualize.
• Usage Monitoring and Optimization: Analyzing flow logs can help identify network usage patterns, top talkers, and cross-region traffic, which can inform decisions for network optimization, capacity planning, and cost reduction.
• Compliance: Flow logs can be used to verify network isolation and ensure compliance with enterprise access rules and regulatory requirements.
• Network Forensics and Security Analysis: By examining network flows, you can detect compromised IPs, investigate security incidents, and integrate flow logs with security information and event management (SIEM) solutions for real-time security analysis.
• Troubleshooting: VPC Flow Logs can help diagnose overly restrictive security group rules or network access control list (ACL) configurations that may be causing connectivity issues.

VPC Flow Logs are an essential tool for network monitoring and analysis across major cloud platforms like AWS, GCP, and Azure. They provide valuable insights into network traffic patterns, performance, and security, helping organizations optimize their cloud environments and maintain a secure and efficient network infrastructure.

Using VPC Flow Logs with Kentik’s Solutions

Kentik provides VPC Flow Log solutions for various cloud platforms, empowering IT professionals with a unified view of all network traffic across infrastructures and between workloads, services, and dependencies. Supported cloud platforms include Azure, AWS, and Google Cloud.

Kentik’s solutions integrate with the respective cloud platforms to provide granular, context-rich visibility and insights into network traffic. By streaming flow log data to Kentik’s network observability and analytics-as-a-service platform in real-time, network operations, security operations, DevOps, site reliability engineering (SRE), and executive teams can gain powerful insights for managing and optimizing their cloud infrastructure.

Azure NSG Flow Logs for Kentik

By integrating with Azure’s Network Security Group (NSG) Flow Logs, Kentik allows cloud ops teams to visualize traffic flows, understand service dependencies in hybrid and multi-cloud environments, and utilize a data-driven approach to cloud infrastructure planning, growth, and cost management. Learn more about Azure NSG Flow Logs for Kentik.

AWS VPC Flow Logs for Kentik

Kentik integrates with AWS VPC Flow Logs to provide granular details of all network activity within AWS VPCs without having to instrument instances or services individually, offering powerful insights for teams across the organization. Learn more about AWS VPC Flow Logs for Kentik.

Google Cloud VPC Flow Logs for Kentik

Kentik works with Google Cloud VPC Flow Logs to provide granular details of all network activity between VMs within GCP projects, offering powerful real-time insights to teams across the organization. Learn more about Google Cloud VPC Flow Logs for Kentik.

Kentik Firehose: Use Cases for VPC Flow Logs

Kentik Firehose is a powerful solution that offers numerous applications and benefits for VPC Flow Log users. By providing enriched network observability data, including flow records, streaming telemetry, SNMP, device configurations, and performance metrics, Firehose allows organizations to comprehensively understand their network dynamics and context.

For VPC Flow Log users, Kentik Firehose can export enriched traffic data, including VPC flow logs from all major public clouds, enabling them to have full access to this data for unique use cases. Integrating Kentik data with other analytic systems, messaging queues, time-series databases, or data lakes becomes seamless with the help of KTranslate.

Discover How Kentik Improves Cloud, Hybrid Cloud, and Multicloud Networks

By providing comprehensive visibility and actionable insights into network traffic, performance, and costs, Kentik enables organizations to make data-driven decisions and improve their management of cloud networks. To get started, request a demo or sign up for a free trial today.