A Deep Dive into AWS CloudTrail

[Music] welcome to cloudera note plus your source for exclusive online videos and online events for aws professionals my name is andreas and today i demonstrate how to collect and analyze audit logs with cloudtrail i will start with some basics and we will dive into the very details later did you enable cloudtrail for your aws accounts already you should not only for being able to answer future questions about security incidents but also to answer day-to-day questions of an account administrator i will start with two demos first i will show you how to analyze cloudtrail logs with cloudwatch insights and then we do a little bit more advanced stuff and analyze the cloudwatch cloudtrail logs with athena which is a bit a little bit more complicated but also more powerful and then we talk about some details of cloudtrail that you need to know about we talk about blind spots we talk about extensive costs when configured not correctly and last but not least i will show you how to do real-time alerts with cloudtrail based on the sys aws foundation's security standard let's start with analyzing cloudtrail audit logs with the help of cloudwatch log insights so this is the aws management console and i've configured cloudtrail here and basically the most important thing is i'm sending the audit logs to an s3 bucket but also to a cloudwatch log group so that's what happens here so let's jump to cloudwatch logs and i'm using what is called logs insights to query the data because it gives me a more advanced user interface and more search and analyzing capabilities so what you get here so i've selected the log group here which contains the cloudtrail audit logs and if we do a query you get just yeah a lot of log messages basically in here with a lot of details about the request to aws apis and what's interesting if you want to analyze the data is you can show the fields so the data fields of all those log messages and that helps to see what's in there what's in the structured data to basically answer your questions okay the first question that i want to answer with the help of a query is which aws accounts are which regions are in use within that aws account so that's the first question i want to answer and i've prepared a query for that that i'm now pasting in here so what i am doing is i'm using an aggregator function i'm counting by the aws region so what does that mean that means um every every audit log has an aws region field and we are now counting that yeah to basically get a summary of all the regions used within the account so let me run that query um takes a little while we already see some results popping up here so what you can see here this is the count the number of basically api requests that cloudtray was collecting for the different regions so we're using 12 different regions here um and you can even jump to visualization to see which regions are used or at least have the most api requests which is you vest one in in our example here okay so that's that's the first one that can be interesting to find out which regions are unused um let's do the next one and by the way i will put all those queries into the video show notes um so that you can run those queries on your own if you want to or modify them a little bit maybe even better so the next one is a little bit different i'm filtering i'm adding a filter and i filter for user identity type i am user so i want to only get audit logs from i am users not from i am roles or what what have you but only for im users why because many organizations have the requirement that imuses are not used at all or only in very rare circumstances and this is a good way to find out if someone in this account uses imuses again i'm using um the aggregator function i'm counting the user identity um the the amazon resource names of those um to find out which users are in use here so let's run that query and see what the result is uh from that so the result is there is one user in this account github with excluder node.io which is in use um this is used for as as the name implies for for github actions and this is the only i'm user in this account okay so that makes sense from my perspective okay the next thing um the next query is a little bit different i'm using a filter again and we are filtering by the event source and the event source is signing amazon aws.com and basically what this gives us is who logged into the aws management console so that might be an interesting question to answer as well so let's run this query and what we get back is a list with in this case this is i am rolls or it is assumed i'm rules to be to be more precise so you can see this is michael and me and stefan so we are logging into this aws account and you can see that here okay so that's interesting i think to answer i would say um let's go to a little bit different thing so that might be something interesting after security incidents so or you want to know which i am policies in my account have been changed so this is an interesting question so who modified basically um the authorization part of your cloud infrastructure and the query i'm doing for that is a little bit different because what i do is i have a more comp complicated filter i filter again by event source which is identity and access management iom.amazonaws.com and now i'm listing different event names so attach group policy a tetral policy create policy create policy version so those are all the im actions that are basically allowing you to modify or to attach an iron policy to a role user or group so i want to know about all those changes okay let's run this so what we get back here now is we don't get back um aggregated data or something we just get back to events okay so one event or two events are popping up here so those are the events where someone changed um an im policy basically put a role policies on inline policy and you can now dive into that see basically what happened so exactly which iron policy was modified and stuff like that so this is really for um yeah debugging or um having a look into what was going on after security incident the next query is similar to that i want to detect changes to security groups so did someone modify the firewall rules security groups so the query is similar uh the only thing that's different is the event source which is easy to in that example and then of course the event names which are authorizing security egress and ingress and also revoking security group egress or ingress so that's very similar and if i run that query i get the events showing me who changed the security group in which way so that's basically the question that i can answer with those information so in that case um let's see an assumed role so specific i'm role was changing and it did a change to the security group and opened port 3306 um so that is the information that we get from here which is i would say quite interesting okay um one more thing um if someone accidentally leaks their aws credentials for example the classic way to do so is on github how do you find out if someone was using those credentials to access resources or modify resources in your aws accounts and there's a query for that basically you can filter the access key id and put in the access key that was leaked and then run that query and what you get back is yeah was someone using that access key within the specific specified time frame and that could um yeah that could give you a feeling how bad the situation is that you're in and which resources have been modified or even which data might have been leaked so it seems like no one was using that leaked access key in my example so i'm happy about that the next demo is about querying the same audit logs but those that are not stored in cloudwatch logs but on s3 so why is that an interesting thing because um you remember that i've configured my cloudtray logs to go to s3 and to cloudwatch logs and when you do the s3 option you can basically have one s3 bucket that collects the audit locks from all your aws accounts so you can also analyze them in one central place we'll talk about that later but that is why athena is a good way to analyze uh those cloudtrail locks as well when they are stored on f3 so let's have a look into that so the first thing when using athena is you need to create a table what athena is doing is it analyzes structured data in our case it's json data stored on f3 ad hoc so you can run queries with a sql like syntax on data that is stored on f3 and athena will just crawl through all that data and analyze it so that's what's possible uh with athena and cloudtrail locks as well um you will find all the code in the show notes so check that out um i've extended what aws has in their documentation a little bit and we'll talk about that so what this first query is doing is it creates a table so you can use that in your own account to create a table with for athena being able to analyze your cloudtrail data so all that is the structure of the data so you need to define the structure of the json data that is coming in so you can just keep it as it is that's what comes from dws documentation what is interesting is the partitioned part so cloudtrail stores your data on a three in a certain format they use a certain key schema to partition the data so for example you will find a timestamp the account id and even the ews region within the path or the key basically of the objects and what we do here is we use partition protection which allows you to crawl through all the data without needing to add partitions manually for your table which is normally a thing you need to do with athena so this is a really cool trick i think in my opinion so what i've been doing is i've configured that for the s3 bucket here the three bucket is called wittek's cloudtrail where we store all the logs and then i have three protections for the partitions one is the timestamp then we have the account and the region and what you can see here is um athena will automatically try to find all the json files within that structure the timestamp it starts from the beginning of 2020 until now um the accounts i have an enum defined here and here it there goes in a list of all the aws accounts that that we own and the same is for i have another enum for the region where i paste in all the region identifiers that we use so by doing that athena is just crawling through all that data automatically without anything of any partitions management needed so that's very helpful um so i highly recommend to do that for cloudtrail data okay you will find the details in the show notes as i promised and now let's create the table let's do that again because yeah this is always a shocker so basically the first time you click the button doesn't work okay so here's the table um the cloudtrail logs um partitioned table and and now we can query the data so let's do a little one let's do a preview so we're running the data and it's basically the same data that we had in cloudwatch logs but now it's stored in f3 and so it runs the query takes maybe a little bit longer than with cloudwatch logs insides um but should return a value soon yeah so now we get back um this is the cloudtrail data from all aws accounts that we own aggregated and of course we want to dive into a little bit more detail have a more advanced query than that okay so let's do that i've prepared a query here already and this is the same query that we have been doing before we are querying the changes of security group so who made changes to a security group and it's basically very similar query now it's sql style so we have select from default cloudtrail logs so this is our table then we have the event source ec2 same as we had before and then event name in and we have a list of the event names authorized security egress and so on you know that already and i'm also filtering by account and by timestamp to make use of those partitions of the data and so that athena doesn't have to go through all um the data and i'm also limiting everything to 50 results um also to make sure we're not crawling through too many data okay let's run the query and the same problem i have to click the button twice and i don't know why this doesn't get fixed okay so now we have the events and now it says this is the user it's again an iron roll and um it revoked security group increase um and so on so basically that's the same data that we have in here um i agree maybe the user interface is a little bit clumsy but you get the same data out of it and i think it's really cool to use athena because you can query not only one aws account but all your aws accounts given the fact that you store all your cloud trail data in one s3 bucket which is very helpful i would say next let's dive a little bit into the technical details of cloudtrail and tips and tricks um that i i want to share with you okay first of all the question is how do you configure cloudtrail in your ews accounts and this is my recommended way of doing things so first of all i would encourage you to create one cloudtrail trail within the management account of your aws organization and what you can do there is you can enable an organization-wide trail that means this trail will collect all the audit logs from all the accounts that belong to this or aws organization to your organization this makes it much more simpler to manage cloud share because you don't have to add a cloudtrail trail into each and every aws account it just collects all the data with only one cloud trailer and as a plus the member accounts cannot remove that cloudtrader they cannot deactivate that in any way you don't have to protect that in any way so that's what i think is very helpful and then i would recommend that my personal thing i would send um the cloudtrail logs to a cloudwatch log group in the same account so in the management account but i will also have a separate aws account in the organization i often call it the vault account um where basically this is a very high security level account in your organization and there is a security and s3 bucket in there to um yeah to send the cloud watch logs in so that allows you to make sure that no one can basically modify those cloud tray locks later or it's at least very hard to do so so that's my recommended way of configuring cloudtrail next i want to talk about the blind spot because in general cloudtrade captures so-called management events so some also say the control plane events of almost all aws services 99 but the problem is there are blind spots as well so let me tell you a story so i was working for consulting client of us and someone from their team leaked their aws credentials and then we got an email from aws they basically deactivate those credentials and they told us we need to go through a list of things to make sure everything in the account is fine and i was going through the cloudtrail logs and what i wanted to answer is the question did someone use those credentials that were leaked on github to access our sensitive data stored in an f3 bucket and what i found out is that unfortunately cloudtrail cannot answer that question or the logs cannot answer the question by default why because by default cloud does not capture data events so it does not capture reading and writing to us three buckets for example and this is a bummer by the way the same is true for many other aws services that have an api that you can use to access and modify the data so for example dynamodb and so on so the thing is cloudtrade supports data events for s3 and lambda but you have to enable this separately we'll talk about that in a second but then there are a lot of other services out there so dynamodb sqs sns rare do not even have the option to collect those audit docs they're just missing so i think that is a problem because you have no way to get an audit log for example the data access in your dynamodb tables okay that's something you have to be aware of so cloudtrail covers the management events and only some data events that's important but then the problem is there's another problem with the data events and that's what we will have a look into next the problem with cloudtrail data events is extensive costs and i made a pricing calculator to show you the problem so if you enable cloudtrail data events for f3 or lambda you will pay a fee for every data event that cloudtrail captures and this is significant so um take a look at this one here the extra charge so um basically when writing data to s3 you pay a premium for enabling cloud data events of about 20 and when reading data the premium is 250 roundabout so that's makes a really huge effect so that means if you have data analytics workloads for example running in a car in an account where you go through terabytes and very very many many small files in the three buckets to crunch the data enabling cloudtrail data events for a3 is not a good idea at least you have to exclude those buckets and that's a bummer same is true for lambda also a huge extra charge on the lambda invocation costs by enabling those data events so i recommend it to enable data events but you have to be aware of the costs and that might be no problem at all in your scenario but it could have a huge impact on you in your 8ws build so be very careful with that there's one more thing that i want to show you and this is about getting real-time alerts um based on the cloudtrail audit logs in your account and um one one reason to do so is the cis aws foundations standard um for example if you have um security hub enabled in your account you will get findings out of that so this is just a recommendation or security best practices for you aws account and they define how to do real-time alerts um based on cloudtrail logs as well so how does that work so basically it's it's not too complicated you have cloudtrail which sends the log messages to a cloudwatch log group where you define a metric filter and the metric filter basically analyzes the log messages in a similar way that we did in the first two demos and sends data to a cloudwatch metric um basically a counter and then you can define a cloudwatch alarm based on that metric that notifies you in certain circumstances so that's how it works and we have a cloud formation template that you can use to set things up i will link that in the show notes and basically i want to show you the results of that in an aws account okay so this is the cloudwatch log group where cloudtray sends all the audit logs to and what you can see here is there are 13 metric filters defined for that log group let's dive into that and here you find the metric filters that basically are defined by the cis aws benchmark and we've just um implemented them here so for example this is um a filter pattern that searches for did someone create or modify a cloudtrail in the account so of course that's the security relevant information and whenever that happens it sends a magic value of one to a cloudwatch metric and there's then a cloudwatch alarm that will notify us about that problem so that is how you can do things so there is um there is um yeah this metric that counts uh for that so let's quickly jump to um the cloud watch alarms so in the alarm section we will find um a lot of those um cloudwatch alarms for example and this one here goes to um count of management console uh authentication failures so there's an alarm for that and it basically says whenever there is a data point in here whenever there's a metric value higher than zero notify me so that's basically the way that works that can be helpful to give you real-time insights my experience with those metric filters defined in the cis standard is you need to do them to get green check checkboxes on your security benchmark but the problem is many of them are not very relevant many of them produce a lot of false positives one example is with multi-factor authentication if you use aws sso sso manages mfa um but basically the cloud events do not know about that you will get a lot of false positives in that example so yeah take them with a grain of salt um but yeah it's a security best practice that many of you have to implement and that's how it works and what we do is we use some of them a subset of them and deactivate that one that doesn't make sense in certain scenarios you can reach out to us and the community about this topic or other aws related questions at any time so visit community.cloudonow.io and ask your questions we are looking forward to hear from you thanks a lot for watching don't forget to rate this video if you learned something new your feedback helps us to produce relevant videos so reach out to us via email or twitter you will find the details in the video show notes we are back in one week thanks a lot for your support

Channel: cloudonaut

Views: 431

Keywords: aws, amazon web service, cloudonaut, cloud, cloudcomputing, cloud computing, aws training, aws cloud, aws tutorial, aws tutorial for beginners, amazon aws tutorial, AWS Cloudtrail, aws cloudtrail configuration, aws cloudtrail demo, aws cloudtrail setup, aws cloudtrail vs cloudwatch, aws cloudtrail vs athena, cloudtrail, cloudtrail aws, cloudtrail setup, how to configure cloudtrail, Querying Audit Logs with CloudWatch, Querying Audit Logs with Athena, CloudTrail Data Events

Id: -a3EX758Mq8

Length: 26min 4sec (1564 seconds)

Published: Sun Nov 07 2021