AWS - CloudWatch Logs

this is Shaitan from cabbage also in this particular assignment or an exercise we are going to see cloud watch logs what we want to do is we want to use cloud watch for collecting logs from ec2 instance so we will create some dummy logs on ec2 instance as an application logs and then we will see how to stream those logs from ec2 machine to cloud watch dashboard and after we get those logs we can even create log filters and define the alarms based on say certain text in the log messages for example many times in your application you will find there is an fatal error or there is a critical error message or error number so you can have filters for those kind of strings and you can generate alarms so that whenever there is an any error in your log message you are notified say over email immediately okay so for doing this exercise what we need to do is we need to have one easy to machine and on that machine we need a cloud watch agent so AWS provides agent package which you can install depending on your OS version I am going to use Amazon Linux and then you need to configure the agent configuration files and define which log files from this ec2 should be streamed to the cloud watch so we will see how to do those configurations one more thing is like if you see here again logs Arbit getting streamed from ec2 to cloud watch now these are two different AWS services again and I am role comes into the picture where cloud watch agent should have permission to send these logs to the cloud watch service so for that we would require and cloud watch now I am role which will assign to ec2 instance okay so at high level these are the steps which I have listed here first thing we need to create I am role and then we launch ec2 instance attached to that attached I am rolled to that instance then we log into our ec2 instance and install AWS logs agent and then we will configure this two files AWS CLI dot count an AWS logs con and configure which application logs we want to stream to the cloud watch dashboard and then we'll start AWS locks service after we see our logs in the cloud watch we will create cloud watch logs metrics based on certain strings for example an error and we will see how to create an alarm based on that string ok so let's do our exercise so I am into my AWS console and the first thing I need to go - I am dashboard and create one I am role so let us do that so I am policy I have already created here in this presentation so I'll just copy it as it is and I will go to my I am dashboard I will create a policy I am directly putting Jason as a text okay review policy I will give name watch logs okay our policy has been created the next thing we should create I am role for our ec2 instance so we will create a new role this time for ec2 instance I select that and in permission I will select the cloud watch policy that we have just created and that role I can give cloud watch Long's role and I created their own okay so we are done with the I am settings let's go to over ec2 instance and launch one ec2 instance so I am selecting the regular Amazon Linux all the configurations are default I am launching it in default VP see I want to connect to this instance over SSH and I will just open SSH port for myself and launch and I am launching this instance okay so my instance has been launched if you see there so the next step is I need to attach this instance and I am role shall go here instance settings and attach or replace role if you remember our role name was Cloud watch dogs rule okay so next let's log into this ec2 instance over SSH and then just configure the it obvious logs agent okay so I am inside this instance the next step is to install the cloud watch agent and for that you can just run this command yum install AWS logs okay so AWS logs agent has been installed and as stated earlier we need to configure some files here so that AWS logs agent can stream those logs so there are two files essentially one is ET CWS log AWS CLI dot-com so I am just editing this file with sudo permission and we just need to have a write region where all these logs are going to be streamed so ap south one that is my Mumbai region and important file is this it is say the blaze log AWS logs dot-com which contains the configurations which we need to set so you can just go and delete all the content from this log file and I have already given this sample content for this log file and let's go over through it now if you say this file important section is this this is a section which can be any name in which region we are operating and which file it has to monitor so right now I just gave this dummy file slash where log application dot log right now on this system I don't have this log file but for this demo purpose we need to create this log file then only we can see those logs there and then in there is a log group under which all the logs will be streamed and as you know in cloud watch logs there are locked streams so in one log group there could be multiple log stream so by default the L'Occitane name would be same as this particular ec2 instance ID so that if we are gathering logs from multiple in situ instances we can identify or differentiate between logs from different easy to instances okay so that's what we wanted to do as settings save this file and as a last step just start AWS logs service okay on this system aw slogs is not the name of the service its AWS logs D so I just started it okay everything is done so ideally we should see log files on the cloud watch logs dashboard but if you go to the cloud watch and go to the logs you will see there is no logs as of now that's because we don't really have the log file actually been created there so let's just go to where log let me change myself to sudo and if you see there is no application dot log so let's just create one file I'm just saying Eco hi this is dummy log data and I can just redirect it to application dot log okay now if you see I might have one file here and if you just do cat I have the content in this file so if everything is working if our crowd watch logs agent is working we should refresh this page and we should see a log group with application logs and there should be one log stream with instance ID and it should populate the data that we had just appended to our log file okay that's good let's just verify if it really streams almost real-time let's just try to obtain more data to this log file say message to redirect to application Tork log similarly let's put more data our log message three redirect to application dot log and all that if you just see this application log it has three messages let's go back to cloud watch dashboard and refresh this it got three messages that means whatever we are writing to the file is getting updated here in the dashboard so that's what we wanted to do now the next part is to a cloud watch locks filter so let's go to the cloud watch lock select this log group and create a filter now you can give any pattern but typically you'll see in real world there would be like if there is an error message then we should have an alarm and then send a notification so I'm just putting the pattern as error is the pattern it won't find any entry as of now because there is no error and then we assign it a metric and we can say name of the metric itself is a app error something and we are creating the filter now we have this filter we can just go ahead and create an alarm based on that and we say if error is the name of the alarm and if whenever number of count is greater than say zero then you should send an alarm okay and then create an alarm okay so we have created an alarm for the metric filter but as you see right now we don't have any error messages and that's where the there is no data to create an alarm so let's just simulate this problem so let's say echo error there is problem with the application just keep redirecting it to the same file and then maybe there are some more errors later on innovator okay so if you see this file it has no additional error messages let me just restart this and if we just look at application dot log it has this error messages so cloud watch first let's see if cloud watch logs captured that okay so it got this and if everything works we should also get an email okay just let's wait for some time till we get some email okay so let's go here and check ok I'm in my email and if you read this it says there is one data point crossed that means it got an error so that's what we wanted to do when there is certain messages any kind of string pattern we found in the log files we should have been notified why are the SNS notification and that's what we have received so if you could do up to this point it's good and this is what we wanted to do thanks for watching if you like this video just like and share thank you

Channel: AWS Training Center

Views: 45,179

Rating: 4.9165506 out of 5

Keywords: cloudwatch logs, aws, stream logs, logs filter

Id: F4IE69V-iuw

Length: 13min 11sec (791 seconds)

Published: Mon Mar 11 2019