Understanding Logging: Containers & Microservices

now i've been getting quite a lot of questions about logging for containers and kubernetes people talk about the efk stack they talk about the elk stack for many folks who are starting out and new to the technology there's quite a lot to understand about these technologies and why and how they're used this is a movie about centralized logging in traditional systems applications write logs to files this can be troublesome when the disk gets full log collectors can be used to ship logs away from the disk but this adds extra engineering overheads to install and maintain applications can also be programmed to send logs directly out of the system somewhere else this avoids writing to disk but results in a lot of duplicate code in each of the applications for pushing logs out of the system so i thought this would be a good time to start a series about logging now with logging there are two categories that i want to discuss before looking at docker and kubernetes number one is standardized logging number two is centralized logging i like to keep my applications as simple as possible applications should have very little knowledge about where the logs go if your company decides to use log system a you don't want code about log system a in every application that you build that means if your company wants to change from log system a to b you have to rewrite all your applications if log system a changes their api or sdk you have to change all your applications to improve all of this in linux process is right to standard out standard out is the default file descriptor that processors can write to instead of writing logs to a file it goes to standard out this prevents the disk from getting full and provides simple standardized logic in our apps to send logs to standard out when processors run in containers the container runtime gathers all these standard out logs and manages them for us the container runtime takes care of the life cycle of these logs so let me demonstrate this if i run a container let's say i say docker run redis and i run a redis container look what happens now because i'm running the container with the minus i t flag docker picks up all the standard outlooks and prints it to the terminal so you can see all the redis logs are shown here and if i open up another terminal and i say docker ps you can see we have radius running i can go ahead and grab the container id and i can say docker logs with the container id and i can get the same logs so we can see the container runtime collects all the standard out logs for each container now remember a container is just the process so if i ran redis on top of linux without a container i would see exactly the same thing this is because if we look at the documentation by default docker log shows the command's output just as it would have appeared if the command ran interactively in the terminal so whether it's running in a container or not it's going to behave the same docker picks up all the locks from standard out and standard error so if docker is by default writing all the container locks to stand it out how do we access those so if we take a look at the docker host and we ran docker system info if we scroll down we can see that docker root directory is set to var lib docker this is the directory that docker stores all its data including logs so if i say docker ps and i grab the id of the redis container and then let's try to access that docker directory by running a container and mounting var lib docker into the container so if i go ahead and run that and i say alice var lib docker we can see that we have all of these folders inside of that directory and if i say alice on the varlip docket containers directory you can see we have all these container ids and if we say alice on the varlip docker containers directory and we use the folder of that container id of redis you can see we have all this data inside of that folder and interestingly we can see that there's a big text file which starts with the container id json.log so if i take that file if i say cat var lib docker containers that folder and that file we can see it'll just print out the docker logs so the standard out logs of redis is being stored on the container host in that directory now internally docker writes its logs in json format and we can confirm this by saying docker system info and if we take a look at the logging driver we can see it's json file now it is possible to configure docker and change the default logging driver so we can tell docker to write logs to a file to an external system or a database but i would always recommend to probably keep the default setting and not have to poke the docker host if you're running in a system like kubernetes you would prefer the docker host to have its default setting now ideally we'd like to keep things very simple so docker can just write to json file and take care of rotating the logs and maintaining the file system and remember if you run a system like kubernetes we're going to have many docker hosts so we don't want to have to go ahead and configure that and remember if you're running a container orchestrator like kubernetes you're probably not going to want to interfere with the default settings of the docker hosts so i mentioned that we should keep our application simple by just writing logs to stand it out i also mentioned that we should keep our docker host simple by keeping the default setting and just pick up the logs from standard out this makes a great and effective standard for standardized logging which is very simple but what if we ran many containers on many hosts like in a system like kubernetes how can we consume and bring all these logs to a central place to achieve centralized logging centralized logging is all about being able to grab the logs easily from all over a distributed system and bring it together to a single place where it can be analyzed with apps writing to standard out and docker reading standard out we can configure software like fluentd to collect logs from the container host and ship it out before the host cleans it up fluentd can mount the host system folder it can read logs and send it to external systems it has a plugin system so you can send logs almost anywhere so to showcase centralized logging i have a github repo and everything i'm doing in the series is in the monitoring logging folder and for this introduction i have a fluentd folder and inside there i have a basic demo now in the basic demo i have a simple docker compose file which is basically going to run flue and d if i change my directory to this folder i can say docker compose up and it will start fluendy now what i'm doing in this docker compose is i created a volume and i'm mounting the docker host var lib docker containers into a directory inside the fluency container i am also mounting a very simple fluency configuration file telling fluentd how to collect logs if we take a look at this configuration it's very simple we're just providing a source of where to gather logs from and where to output the logs so you can see here we're giving it the path to the containers folder that i showed you earlier and we're telling it to pick up anything with the container id json.log so fluentd will pick up all the logs from the docker containers on this host and then we have a simple output of writing it to an output folder and i've mounted the folder here so we can see the logs being collected by fluentd so this is a very simple log that for demo purposes shows us how to collect all the container logs on the system and send it to fluentd so fluentd will monitor the docker host on var lib docker and pick up any container logs that any container writes and for demo purpose it'll just dump it all to a file that we can see here so while fluentd is running let's go ahead and open up another terminal and start a redis container and if i run that we can see the logs are writing to standard out as it usually would and if you give it some time you can see in our logs folder here fluently has started to collect all the logs and starting to write it out to the file system here and if we give it some time we can see eventually the buffer starts filling up and fluently starts writing the logs to the buffer you can see here it's collected all the reddest logs in json format so this is a very simple demonstration of standardized and centralized logging and why it's important in container environments before you start taking a look at kubernetes and if you like this video be sure to give it a thumbs up because in the next video i'll be taking a look at a deep dive into the logging ecosystem we'll be taking a much deeper look at fluentd the elk stack and the efk stack we're going to take a look at how to set all of the stuff up locally understand the fundamental concepts and then finally run it all on top of kubernetes so also remember to check out the links down below to the community page and if you want to support the channel further be sure to click the join button below and become a member to gain extra perks on the community server and as always thanks for watching and until next time peace [Music]

Channel: That DevOps Guy

Views: 11,977

Rating: 4.9571209 out of 5

Keywords: devops, infrastructure, as, code, azure, aks, kubernetes, k8s, cloud, training, course, cloudnative, az, github, development, deployment, containers, docker, messge, broker, aws, amazon, web, services, google, gcp, fluentd, logging, monitoring, prometheus, microservices

Id: MMVdkzeQ848

Length: 8min 51sec (531 seconds)

Published: Mon Sep 21 2020