AWS Certified Advanced Networking Series: VPC Pt7 (Flow Logs)

8 min readJan 27, 2021

This time I writting about Flow Logs which is used for traffic monitoring & analyzing purposes.

VPC Flow Logs

This is a feature to capture information about the IP traffic going to and from network interfaces in the VPC. Flow log data can be published to Amazon CloudWatch Logs or Amazon S3. Stored data can be retrieved and viewed.

Flow logs can help with: -

• Diagnosing overly restrictive security group rules
• Monitoring the traffic that is reaching your instance
• Determining the direction of the traffic to and from the network interfaces

Flow log data is collected outside of the path of the network traffic, and therefore does not affect network throughput or latency. Flow logs can be created or deleted without any risk of impact to network performance.

Flow logs basics

Flow logs can be created for a VPC, a subnet, or a network interface. If a flow log is created for a subnet or VPC, each network interface in that subnet or VPC is monitored. Flow log data for a monitored network interface is recorded as flow log records, which are log events consisting of fields that describe the traffic flow.

To create a flow log, specify:
• The resource for which to create the flow log
• The type of traffic to capture (accepted traffic, rejected traffic, or all traffic)
• The destinations to publish the flow log data

Tags can be applied for flow logs and each tags consists of a key and an optional value, both of which the account holder defines. Tags can help to organize the flow logs, for example by purpose or owner. After a flow log is created it can take several minutes to begin collecting and publishing data to the chosen destinations. Flow logs do not capture real-time log streams for the network interfaces.

When more instances are launched into the subnet after flow logs are created for the VPC or subnet a new log stream (for CloudWatch Logs) or log file object (for Amazon S3) is created for each new network interface. This occurs as soon as any network traffic is recorded for that network interface.

Flow logs can be created for network interfaces that are created by other AWS services, such as:

• Elastic Load Balancing
• Amazon RDS
• Amazon ElastiCache
• Amazon Redshift
• Amazon WorkSpaces
• NAT gateways
• Transit gateways

Regardless of the type of network interface, use Amazon EC2 console or the Amazon EC2 API to create a flow log for a network interface.

Flow logs can be deleted if no longer required. Deleting a flow log disables the flow log service for the resource, and no new flow log records are created or published to CloudWatch Logs or Amazon S3. Deleting the flow log does not delete any existing flow log records or log streams (for CloudWatch Logs) or log file objects (for Amazon S3) for a network interface. To delete an existing log stream, use the CloudWatch Logs console. To delete existing log file objects, use the Amazon S3 console. After deleting a flow log, it can take several minutes to stop collecting data.

Flow log records

· It represents a network flow in the VPC. By default, each record captures a network internet protocol (IP) traffic flow (characterized by a 5-tuple on a per network interface basis) that occurs within an aggregation interval, also referred to as a capture window.

· By default, the record includes values for the different components of the IP flow, including the source, destination, and protocol.

· When creating a flow log, use the default format for the flow log record, or specify a custom format.

Aggregation interval

This is the period of time during which a particular flow is captured and aggregated into a flow log record.

By default, the maximum aggregation interval is 10 minutes.

flow log can be created optionally by specifying a maximum aggregation interval of 1 minute. Flow logs with a maximum aggregation interval of 1 minute produce a higher volume of flow log records than flow logs with a maximum aggregation interval of 10 minutes.

When a network interface is attached to a Nitro-based instance, the aggregation interval is always 1 minute or less, regardless of the specified maximum aggregation interval.

After data is captured within an aggregation interval, it takes additional time to process and publish the data to CloudWatch Logs or Amazon S3. This additional time could be up to 5 minutes to publish to CloudWatch Logs, and up to 10 minutes to publish to Amazon S3.

Default format

By default, the log line format for a flow log record is a space-separated string that has the following set of fields in the following order.

The default format captures only a subset of all of the available fields for a flow log record. To capture all available fields or a different subset of fields, specify a custom format. Default format cannot be customized or changed.

Custom format

Flow log record can be specified in custom format. For a custom format, specify which fields to return in the flow log record, and the order in which it should appear. This enables it to create flow logs that are specific to the needs and to omit fields that are not relevant.

A custom format can also help to reduce the need for separate processes to extract specific information from published flow logs.

At least need to specify one from the available flow log fields.

Available fields

The following table describes all of the available fields for a flow log record. The Version column indicates the VPC Flow Logs version in which the field was introduced.

Flow log record examples

Accepted and rejected Traffic

The following are examples of default flow log records.
In this example, SSH traffic (destination port 22, TCP protocol) to network interface eni-1235b8ca123456789 in account 123456789010 was allowed.

In this example, RDP traffic (destination port 3389, TCP protocol) to network interface eni-1235b8ca123456789 in account 123456789010 was rejected.

No data and skipped records

The following are examples of default flow log records. In this example, no data was recorded during the aggregation interval.

In this example, records were skipped during the aggregation interval.

Security group and network ACL rules

If flow logs are used to diagnose overly restrictive or permissive security group rules or network ACL rules, be aware of the statefulness of these resources. Security groups are stateful — this means that responses to allowed traffic are also allowed, even if the rules in the security group do not permit it. Conversely, network ACLs are stateless, therefore responses to allowed traffic are subject to network ACL rules.

For example, you use the ping command from your home computer (IP address is 203.0.113.12) to your instance (the network interface’s private IP address is 172.31.16.139). Your security group’s inbound rules allow ICMP traffic but the outbound rules do not allow ICMP traffic. Because security groups are stateful, the response ping from your instance is allowed. Your network ACL permits inbound ICMP traffic but does not permit outbound ICMP traffic. Because network ACLs are stateless, the response ping is dropped and does not reach your home computer. In a default flow log, this is displayed as two flow log records:
• An ACCEPT record for the originating ping that was allowed by both the network ACL and the security
group, and therefore was allowed to reach your instance.
• A REJECT record for the response ping that the network ACL denied

If the network ACL permits outbound ICMP traffic, the flow log displays two ACCEPT records (one for the originating ping and one for the response ping). If your security group denies inbound ICMP traffic, the flow log displays a single REJECT record, because the traffic was not permitted to reach your instance.

IPv6 traffic

The following is an example of a default flow log record. In the example, SSH traffic (port 22) from IPv6 address 2001:db8:1234:a100:8d6e:3477:df66:f105 to network interface eni-1235b8ca123456789 in account 123456789010 was allowed.

TCP flag sequence

The following is an example of a custom flow log that captures the following fields in the following order.

The tcp-flags field can help to identify the direction of the traffic, for example, which server initiated the connection. In the following records (starting at 7:47:55 PM and ending at 7:48:53 PM), two connections were started by a client to a server running on port 5001. Two SYN flags (2) were received by server from the client from different source ports on the client (43416 and 43418). For each SYN, a SYN-ACK was sent from the server to the client (18) on the corresponding port.

In the second aggregation interval, one of the connections that was established during the previous flow is now closed. The client sent a FIN flag (1) to the server for the connection on port 43418. The server sent a FIN to the client on port 43418.

For short connections (for example, a few seconds) that are opened and closed within a single aggregation interval, the flags might be set on the same line in the flow log record for traffic flow in the same direction. In the following example, the connection is established and finished within the same aggregation interval. In the first line, the TCP flag value is 3, which indicates that there was a SYN and a FIN message sent from the client to the server. In the second line, the TCP flag value is 19, which indicates that there was SYN-ACK and a FIN message sent from the server to the client.