↓Skip to main content
  1. Posts/

[ AWS Threat Detection Part - 3 ] Detecting Attacks in AWS using CloudTrail Logs - Chapter 1

·1290 words·7 mins·
TheDeadThinker
Author
TheDeadThinker
Table of Contents

Overview
#

This part of AWS Threat Detection series will focus on how CloudTrail logs can be used to detect the attacks in AWS environment, how those can be mapped to the MITRE Attack Framework and try to understand the process of investigation rather than focusing specific attacks.

Initial Setup
#

If you also want to practice while reading the blog check the EVIL CORP Breach lab, all the logs are from this lab.

I will be using Splunk SIEM as I am more comfortable with it you can use any SIEM of your choice or any text editor if you are comfortable using that.

There are already multiple videos and blogs regarding how to setup SIEM and ingest cloudtrail logs into it.

Ingest CloudTrail Logs into Graylog

Splunk AWS Add-on : Ingestion of AWS Cloudtrail data in Splunk

0xdf: Splunk Docker Container for CloudTrail Analysis

Understanding The Situation
#

In large AWS environment there can be millions of logs per day, therefore whenever we investigate any breach or incident we should not directly start investigating the logs, first collect as much information you can from Devops, Cloud, Developers or anyone using cloud resource for any task.

We can start with questions like-

  • What are different users responsibilities on cloud
  • How many user have access to root credentials
  • When the Team identified the incident
  • Details of roles and policies in the AWS
  • Does Team have assets list hosted on AWS
  • Does team have network diagram etc…

Once you think you collected the sufficient information and have a rough time period when breach was detected then we can start investigating with this information and have a conversation with different teams when you identified something that needs some description or clarification.

Understanding the Events
#

In our case we have information that we have to investigate the logs for 3 days and For investigation ForensicInvestigation user is created by the DFIR Team.

Time Period: 3 July 2024 - 5 July 2024

In CloudTrail Logs if data events is also enabled during configuration of cloudTrail then there will be lot of events for the logs saved or sent to the s3 bucket.

In below image it is clear that out of 5.5K log 4.9k are having the useragent and useridentity as cloudtrail.amazonaws.com which is related to the cloudtrail service check the document from AWS

index="tb-cloudtrail"  | fillnull userAgent | stats count by userAgent | sort - count

We can check the events related to the cloudtrail.amazonaws.com and we found that major are either PutObject or GenerateDataKey [if KMS is enabled during configuration].

The GenerateDataKey operation is a function provided by AWS Key Management Service (KMS) that generates a unique symmetric data encryption key.

index="tb-cloudtrail"  | fillnull userAgent | search userAgent="cloudtrail.amazonaws.com" | stats count by eventName

We can exclude them as of now and focus on other logs, after excluding it can be observed that log count reduced to 500, which will be lot more higher even in small AWS environment than this. Therefore still we can’t manually check each event one by one to find the suspicious API calls.

The approach we can follow here is that combine logs on hourly or daily basis depending on logs quantity. We have only 3 days logs and count is also less than 1K we can go with the hourly basis.

index="tb-cloudtrail"  
| fillnull value=null userAgent userIdentity.invokedBy userIdentity.arn 
| search userIdentity.invokedBy!="cloudtrail.amazonaws.com"  
| bin span=1h _time as AggTime
| eval AggTime=strftime(AggTime, "%m/%d/%Y:%H:%M:%S")
| stats values(eventName) values(userIdentity.arn) by AggTime | sort - _time

We have grouped events based on hours and it can help us to identify which specific hour or period to target for further analysis. but there are lot of repetitive task like PutObject, ListGroups etc, it can either be AWS service logs or attacker using some automated script or tools [ But it’s very less likely that it will be attacker because performing same task multiple time will generate lot of evidence in logs ]

AWSServiceRoleForAmazonSSM/SSMInventory is a default service-linked role created by AWS for the AWS Systems Manager (SSM) Inventory feature. Service-linked roles are predefined by AWS and are designed to enable AWS services to perform actions on your behalf with the necessary permissions.

AWS Systems Manager (SSM) using ListGroups API can be part of its inventory collection, compliance management, or other administrative tasks.

We can exclude arn:aws:sts::339712839666:assumed-role/AWSServiceRoleForAmazonSSM/SSMInventory and null from userIdentity.arn. For null we will check later.

index="tb-cloudtrail"  userIdentity.arn="arn:aws:sts::339712839666:assumed-role/AWSServiceRoleForAmazonSSM/SSMInventory"
| fillnull value=null userAgent userIdentity.invokedBy userIdentity.arn 
| search userIdentity.invokedBy!="cloudtrail.amazonaws.com"  
| bin span=1h _time as AggTime
| eval AggTime=strftime(AggTime, "%m/%d/%Y:%H:%M:%S")
| stats values(eventName) values(userIdentity.arn) by AggTime | sort - _time

Checking more logs we can observe that around 4 Jul 2024 12:00 there are activities from two new arn one of them is root user and another is a role assumed by a instance.

arn:aws:sts::339712839666:assumed-role/EC2RoleForServers/i-0003a74a081eb21bb
arn:aws:iam::339712839666:root

The root user is only performing the read operation not any write operation, but EC2RoleForServers/i-0003a74a081eb21bb is performing activity like creating keypairs, security groups and checking for images this is probably that attacker is trying to run new instance or it can be normal activity let’s confirm it.

index="tb-cloudtrail"   
| fillnull value=null userAgent userIdentity.invokedBy userIdentity.arn  
| search userIdentity.invokedBy!="cloudtrail.amazonaws.com"   userIdentity.arn!="arn:aws:sts::339712839666:assumed-role/AWSServiceRoleForAmazonSSM/SSMInventory"   userIdentity.arn!="null"
| bin span=10m _time as AggTime 
| eval AggTime=strftime(AggTime, "%m/%d/%Y:%H:%M:%S")  
| stats  values(eventName)  values(userIdentity.arn) by  AggTime  | sort  by _time

Excluding root and AWSServiceRoleForAmazonSSM from logs to focus more on the new identified user and role to confirm if need further investigation or not.

index="tb-cloudtrail"   
| fillnull value=null userAgent userIdentity.invokedBy userIdentity.arn  
| search userIdentity.invokedBy!="cloudtrail.amazonaws.com"   userIdentity.arn!="arn:aws:sts::339712839666:assumed-role/AWSServiceRoleForAmazonSSM/SSMInventory"   "userIdentity.arn"!="arn:aws:iam::339712839666:root" 
| bin span=1h _time as AggTime 
| eval AggTime=strftime(AggTime, "%m/%d/%Y:%H:%M:%S")  
| stats  values(eventName)  values(userAgent) by userIdentity.arn AggTime  | sort  by AggTime

But before going with the EC2RoleForServer we have to remember that we excluded null ARN and it can lead to missing evidences that can help us to identify suspicious activity.

The null ARN have activities during same hour 4 July 2024 12:00, events includes GetObject and assume role and these can lead that a role is assumed or objects downloaded from s3 bucket.

Checking for the GetObject for the ARN value as null as it’s useragent is interesting and after investigating, it is clear that network diagram are downloaded from the betadocumentsv2 bucket and useragent shows it is from Windows 10 [ if useragent is not spoofed ].

index="tb-cloudtrail"   
| fillnull value=null userAgent userIdentity.invokedBy userIdentity.arn  
| search userIdentity.invokedBy!="cloudtrail.amazonaws.com"   userIdentity.arn!="arn:aws:sts::339712839666:assumed-role/AWSServiceRoleForAmazonSSM/SSMInventory"   "userIdentity.arn"!="arn:aws:iam::339712839666:root" 
| search userIdentity.arn="null" | stats values(sourceIPAddress) values(userAgent) by eventName


index="tb-cloudtrail"   eventName=GetObject
| fillnull value=null userAgent userIdentity.invokedBy userIdentity.arn  
| table _time requestParameters.bucketName requestParameters.key sourceIPAddress userAgent

Checking for all events from IP 36.50.238.1 which identified from previous step only have GetObject.

index="tb-cloudtrail"   sourceIPAddress="36.50.238.1" | stats count by _time eventName

Excluding null also now shows that there are total three ARN we identified out of which EC2RoleForServers we already aware of, the developer user is bit suspicious as it using the ip same as aws:ec2-instance/i-0fcc1a99214f7dcdc and performing events like CreateUser, CreateAccessKey which can be used for persistence.

arn:aws:sts::339712839666:assumed-role/EC2RoleForServers/i-0003a74a081eb21bb
arn:aws:sts::339712839666:assumed-role/aws:ec2-instance/i-0fcc1a99214f7dcdc
arn:aws:iam::339712839666:user/developer

index="tb-cloudtrail"   
| fillnull value=null userAgent userIdentity.invokedBy userIdentity.arn  
| search userIdentity.invokedBy!="cloudtrail.amazonaws.com"   userIdentity.arn!="arn:aws:sts::339712839666:assumed-role/AWSServiceRoleForAmazonSSM/SSMInventory"   "userIdentity.arn"!="arn:aws:iam::339712839666:root" 
| search userIdentity.arn="null" | stats values(sourceIPAddress) values(userAgent) by eventName


index="tb-cloudtrail"   
| fillnull value=null userAgent userIdentity.invokedBy userIdentity.arn  
| search userIdentity.invokedBy!="cloudtrail.amazonaws.com"   userIdentity.arn!="arn:aws:sts::339712839666:assumed-role/AWSServiceRoleForAmazonSSM/SSMInventory"   "userIdentity.arn"!="arn:aws:iam::339712839666:root" 
| search userIdentity.arn="null" | stats values(sourceIPAddress) values(userAgent) by eventName _time | sort by _time

Conclusion
#

The blog has already become too long and lot of information to understand and practice. Let’s summarize the information we collected till now. This investigation will continue in chapter 2 of CloudTrail log analysis.

  • There are three suspicious ARN identified
arn:aws:sts::339712839666:assumed-role/EC2RoleForServers/i-0003a74a081eb21bb
arn:aws:sts::339712839666:assumed-role/aws:ec2-instance/i-0fcc1a99214f7dcdc
arn:aws:iam::339712839666:user/developer
  • Multiple IPs identified that need further investigation
36.50.238.1
18.188.58.36 // AWS IP
18.224.8.62  // AWS IP 
188.214.122.83
  • Objects downloaded from the betadocumentsv2 s3 bucket
network diagram - version 1.png
network diagram - version 0.1.png
  • Nearest time when suspicious events started 4 Jul 2024 12:00