Table of Contents

Overview
#

The second part of this blog series will focus on the logging capabilities of the AWS and what different services we can use to collect the logs from multiple services like EC2, IAM, S3 buckets, VPC etc…

Logs Available in AWS
#

CloudTrail Logs
VPC Flows Logs
S3 Server Access Logs
Route 53 Logs
AWS Lambda Logging
AWS Elastic Load Balancing (ELB) Access Logs and more

Details Available in Different logs
#

Service	Details Stored	Use Cases
CloudTrail	Records AWS API calls, including user identity, time, source IP, request parameters, and response elements.	Security analysis, resource change tracking, compliance auditing.
VPC Flow Logs	Captures information about IP traffic going to and from network interfaces, including source and destination IP, port, protocol, and traffic acceptance/rejection.	Network monitoring, troubleshooting, security analysis.
S3 Access Logs	Logs detailed information about requests made to S3 buckets, including requester, bucket name, request time, request action, response status, and error codes.	Access auditing, security analysis, usage monitoring.
Route 53 Logs	Captures DNS query logs for hosted zones, including the domain name queried, query type, and response.	DNS traffic analysis, security monitoring, troubleshooting DNS issues.
AWS Lambda Logging	Logs function execution details, including start time, end time, and result. Integrated with CloudWatch Logs for custom logging.	Function execution monitoring, debugging, performance analysis.
ELB Access Logs	Logs detailed information about requests sent to load balancers, including request time, client IP, latencies, request paths, and server responses.	Traffic analysis, performance monitoring, troubleshooting application issues.

How To Configure Services To Collect Logs ?
#

This blog will you detail explanation of steps involved in configuring multiple service to collect logs.

Services covered in blog:

CloudTrail Logs
VPC Flow Logs
S3 Server Access Logs

AWS CloudTrail
#

AWS CloudTrail is an AWS service that helps you enable operational and risk auditing, governance, and compliance of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs. Source

Configuration of AWS CloudTrail
#

Open CloudTrail Service in AWS and click on Create Trail button
Fill the details like Trail name, s3 bucket details where trail logs will be stored.

There are are additional options which can be enabled when required, like Log File Validation, SNS Notification delivery.

Logs can be transferred to the CloudWatch, But then it will increase steps in setup as more details like Log Group Name which will be used in CloudWatch to check the logs and the IAM role that will be used to send the logs to the CloudTrail should be filled and roles should be created.

Follow AWS documentation for setup : Sending events to CloudWatch Logs

CloudTrail provides multiple log events :
- Management Events: Tracks control plane operations that manage AWS resources, such as creating or deleting instances.
- Data Events: Tracks data plane operations that interact with the content of AWS resources, such as reading or writing to an S3 bucket
- Insights Events: Provides automated insights into unusual operational patterns in your AWS account, helping identify and respond to anomalies.

Data Events have three data event source S3, Lambda and DynamoDB. Select what type of activity like read or write to be collected.
- Data Events have extra cost when enabled Refer to CloudTrail Pricing Documentation
- AWS Documentation on Data Events

AWS CloudTrail Insights helps AWS users identify and respond to unusual activity associated with API calls and API error rates by continuously analyzing CloudTrail management events. CloudTrail Insights events analyzes your normal patterns of API call volume and API error rates, also called the baseline, and generates CloudTrail Insights events when the call volume or error rates are outside normal patterns. CloudTrail Insights events on API call volume are generated for write management APIs, and CloudTrail Insights events on API error rate are generated for both read and write management APIs. CloudTrail Insights events are only logged when unusual activity occurs. [ From AWS Documentation ]

Finally review all the settings and check if Multi-region Trail is Yes or not. If Multi-Region Trail is No then it will only collect the logs from the region in which trail is configured.
- For more details on Multi-Region Trail Refer : Receiving CloudTrail log files from multiple Regions

Configured S3 bucket will start collecting all the logs and each region will have their separate directory.

VPC Flow Logs
#

Let’s understand what is VPC before configuring the flow logs.

Amazon Virtual Private Cloud (VPC) allows to launch AWS resources in a logically isolated virtual network. It offers complete control over network environment, including selection of IP address ranges, creation of subnets, and configuration of route tables and gateways. VPC enhances security and scalability, making it a fundamental building block for deploying secure and resilient applications on AWS.

To learn more about vpc concepts like vpc peering, Security Groups, Network ACL ( NACLs), Route table and how it can be configured properly, Refer Amazon Virtual Private Cloud

VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your VPC. Flow log data can be published to the following locations: Amazon CloudWatch Logs, Amazon S3, or Amazon Data Firehose. After you create a flow log, you can retrieve and view the flow log records in the log group, bucket, or delivery stream that you configured. Source

Configuration of VPC Flow Logs
#

Select VPC for which flow logs will be configured
Under details section there is Flow Logs options as shown in below image.
Click on create flow logs and give name to flow logs
- Select which traffic logs required accept, rejected or both
- Choose Maximum aggregation time: It means that at what frequency you want to collect details like bytes, packets and these interval will be reflected in the start and end time in vpc flow logs.

There are multiple service where logs from vpc can be stored and later used for the analysis the simple way is storing logs in S3 bucket if these logs will later be ingested in any SIEM or will be used for manual analysis.
- To send logs to s3 bucket specify destination bucket ARN.

If CloudWatch is selected then follow same approach as done in CloudTrail, select a log group and IAM Role.

AWS by default have few fields in logs but there are many more, those can be selected by using custom format option and then select from the dropdown in the order you want them in logs.

There are more options which can be used if logs will be used by other tools those require any different format otherwise by default logs will be stored in a txt file in a gz format with space as delimiter.
- Partition Logs by Time: It something that will config when the collected flow logs will be pushed to the destination.

Destination S3 bucket will start collecting all the flow logs.

Sample Logs for a VPC

To Understand VPC Flow Fields Refer AWS VPC Flow Log Records

S3 Server Access Logs
#

Server access logging provides detailed records for the requests that are made to a bucket. Server access logs are useful for many applications. For example, access log information can be useful in security and access audits. Source

Configuration of S3 Server Access Logs
#

Before Configuring S3 server access logs let’s understand some points which should be in mind
- S3 buckets server access logs should configured per bucket
- Logs are stored in txt format with space as delimiter
Choose the bucket for which S3 Server Access log will configured and got to the properties
There are two options as shown in below image, either setup a CloudTrail or S3 Server Access Logging.
Click on Edit button on the Server Access logging, choose enable option as by default it is disable, specify the destination S3 bucket details.

There are two formats in which logs are stored in the destination S3 bucket.
- The first format will store all the buckets logs under same folder but second format create a separate folder for each bucket and store logs in their respective bucket folder.