Linux Control Groups (cgroups) demonstration (George Magklaras)

hi I'm George Maharis in this video I shall be talking about linux control groups it's not a new technology but in light of reviewing the Red Hat Enterprise Linux version 7 and the resource management with Linux containers one of the key technologies that containers use is that of control groups therefore I'm making this video if you are a sysadmin or you are in the DevOps arena you have seen situation like the one in the picture here where a pretty hefty system in this particular case a 32 core quarters of terabyte of RAM system can be easily overwhelmed by heavy and a symmetric utilization of course one way to solve this situation is to apply you limits but there are smarter ways that help you tag the processes and therefore make a more refined resource management model in that is a Linux control group or C group so essentially a C group is a way to administer a number of resources and set resource limits to them by using certain resource controllers or C group subsystems as they're called so you can administer CPU memory network bandwidth and block i/o resources amongst a hierarchy of processes a C group hierarchy is something that you have to get used to if you are not familiar with that concept before as I said C groups are not is not a new concept but essentially very few system administrators actually employ it in production environments of our so it gets a certain getting used to before you are confident to actually deploy them in production so let's go ahead and see how system D the core 2 employs a C group hierarchy if you are not familiar with system D you can watch a video that I made that explains it basic usage the link will or should pop out on your screen right now please make sure you review that video if and only if you are not familiar with the system D because you need that knowledge to get further into this particular video so going back to the C group hierarchies we have first of all slices slices which are actually underlined here and of course they are in italics because in themselves they don't actually contain any processes but they provide the blueprint for organizing a hierarchy around processes and in particular a slice might contain a service or a scope a service is of course a hierarchical group ordered process set of processes that actually refers to system services or things that are started via systemd via permanent unit files a scope is a little bit different is a hierarchy of processes that can actually start a start or create processes dynamically so we are talking about transient processes things like user sessions or virtual machines that might fire up or containers outside the boot time this will be organized in a scope which of course will be contained inside a slice slices and the blueprints for actually holding a hierarchy of processes have they they are named in a way that can hint the hierarchy so for example we can see here that the slice name - instance should be a child of name dot slice and of course the - dot slice is the root slice there are four default slices that you should see easily on most them's that employee see groups of course the root slice which is the mother of all slices sitting at the very top of the C group tree and then we have three other categories of slices the system dot slice which should by default contain all system services so things like the blue tooth service the httpd service the sshd daemons should actually go by default under the system dot slice the user slice is of course the default place for all user sessions so processes like sshd in some systems and the graphics display manager related processes should actually appear under user dot slices and finally we might have a specific slice for virtual machines and containers the machine dot slice if you don't understand what all these things mean and it seems difficult don't worry there will be plenty of examples in this video to demonstrate the concepts before we go on the command line on on rail seven terminal here is a review of the key group commands the most important C group commands that we're going to use at the very top we're going to see the system DC GLS and the system D cg top these are two commands that one may employ to in order to display the hierarchy the C group hierarchy of processes in particular the first one system v c GLS and the system D - C G top is of course the equivalent of top but C group-oriented so you can see for example how much CPU and how much RAM is occupied by a particular slice the system D run and - run is a very useful command that actually illustrates how we can start transit processes and create scopes of course the system city command and in particular the systemctl set property which is actually used to apply the limits to AC group a hierarchical process is controlled by AC group so this is the command that you use to use a particular C group sub controller to limit CPU memory bandwidth and a number of other resources okay so let's go now to the command line here we have a ready configured rail seven system and we are going to start by actually demonstrating the hierarchy the sync group hierarchy of processes traditionally you would do something like top or ps3 to actually view the tree of processes running in a system but if you want to see the control groups what you should do is invoke the system CTLs and as you can see system D does see GL s gives you the C group hierarchy of processes we start of course by the root slice which is of course the system D process itself and then we see the various different slices default slices that we talked about earlier on in the slide so the user slice in this particular case might contain a sub slice so user - 1000 dot slice should be a child of user dot slice and in itself it might create a using session and the user session itself creates a number of processes associated with a user activity in this particular case this slice represents me logging in into the system so this is my cell this is the secure cell demon processes that are actually forked to handle the shell and this is the command that I just executed to illustrate the system D - C GLS on the system dot slice we can actually see a number of services pretty much like we see services on a system V based older system so for example here is a web server contained as a service so this is the HTTP d dot service forking a number of web service in the foreground here and here is the system getting slice which of course is the Getty dy service the firewall service and you get the idea of what this does if we do System d cg top you can see exactly what i said earlier on in the slide so you see all the different slices and the processes that they have the number of tasks that they have the percent of the cpu they occupy the memory and the input/output per second ok let's move on to actually demonstrate how to run a transient process with system D ok in order to run a transient process something that's not started by a permanent unit file we still have to invoke system D and the command is system D - run we give it a unit the transit unit name and then we give it a slice name under which the process hierarchy should appear so essentially what I'm going to do here is choose a common command like top - B to run top in batch mode and then place it under the system process hierarchy so that we can actually see it start it and stop it with system D so let's go ahead and do system D - run the unit should be something like top test and the slice should be something like geerios dot slice so I could actually put that under the system slice or any other slice in the system but for the purposes of making the illustration of this transient command easier I choose to make my own slice called George's dot slice and then of course the command which should be the very last thing is top - B if I do that then I get a confirmation running as unit top test dot service so if I now do systemctl status and then pass it the slice I could actually see that there is a process in this particular case PID 5000 417 and there is my top - be running there if I want to display the processed hierarchy again I would do system D C - C GLS and under the very top slice here giorgos dot slice I see a service called top test service and then under this service I actually see the slash bin slash top command running so pretty much I can do whatever I want with this service right now I can for example go and do systemctl stop top test dot service and then do a status to verify that lets status has that the the actual service has stopped and actually start it again but as you can see this is actually a transient process we don't have a permanent file so essentially this process since it stopped it has now disappeared from the system I could try and do something like this and yet as you can see there's nothing I can do to bring back that process okay so essentially this process has now disappeared since it was a transient process in the next example we're going to see how to actually make a proper unit file to start a proper service and apply resource limits to it okay let's proceed now to the last demo of this video which is a demonstration of how to make a service and apply AC group limit to it so what we're going to do we are going to make our own service and place its unit target file under the user lip system the system directory then we're going to reload system D in an attempt to load the unit file attempt to start and stop the service and see that if everything goes well then demonstrate the very essence of AC group administration which is how to apply in this particular case a memory limit to that service and see it taking effect so let's go to the system and let's navigate to my home directory on this system here we can see two files one of them which is a single executable shell file and what it does it calls periodically every two seconds the mem hog program the main cog application is a simple application where for example it hugs the the memory and it actually sends some test buttons and it occupies the memory it useful for some system tests and some for the purposes of stress testing the system but what we do in this particular case we mem hog a two gigabyte RAM memory chunk we sleep for two seconds and then we do the same thing over and over again so that's going to be our small test service and then let's have a look at the second file which is the actual unit file this is what a basic unit file looks like so you have the unit service and install entries under the unit section you have a basic description in the service section which is the most essential one you have the exact start and exact stop directives essentially instructing system B what to execute to start and stop the application and then this is the target or in the previous life of system V the run level dependency here in this particular system we are executing this under the multi - user dot target or what used to be known as run level 3 so let's go ahead and actually perform the first step and actually copy the mem hog test dot service file under user Lib system D system there was a previous incarnation of that service there so that but that's what I would do typically on the first time so going under that directory now and actually checking that everything is ok let's see mem hook and yes we are here at that point let's try to reload system D in an attempt to force the systemd demon to reload all the config files the unit files and then the next thing we should do is systemctl status mem hog test dot service and here we can actually see that the mem Hawk service is loaded which means that system D has successfully parts the unit file but is actually very inactive because we haven't started it so I can go ahead and do systemctl enable mem hog test dot service this command should actually start it automatically so that it's like a proper service every time the system boots and then this command system CDL start mem hog dest dot service should actually add them to start the service and after clearing the screen what we can do is systemctl status mem hog test service and as you can see this is the the journal entry here and essentially we have now started under the system slice mmm hog test dot service that actually runs in the background and it's actually executing now we can do a systemctl stop mem hog test dot service and then again do a status and again we see here that the service was stopped probably in that particular case now we are confident that we have a service that is installed properly so it was loaded and starts and stops properly what we need to do now is to actually demonstrate how we can start applying resource limits to this process and service so under normal circumstances every cycle execution cycle of the memoed test service if you remember the script should actually give us something like two gig ram chunk so let's say that we don't want that and we need to actually execute something or apply something that will limit the amount of memory to 1.2 gigs to do that we will need to apply the systemctl set property command and let's say that we want to make this change on the runtime process I'll explain what this means in a minute the mem hog test dot service is the service that we are actually interested in limiting and we will apply in this particular case a memory limit of 1200 mix which is a little bit under 1.2 kicks so if we do that we can actually not see anything happening yet but what we have done here is we set to systemctl go to the mem hog test service and apply at run time ma this particular memory limit here the run the - - run times which what does it applies this limit temporarily in other words if I go now and reboot the system and the system will of course automatically restart the mem hog test dot service this limit will not be in effect because it was not written on a file and it was not made permanent so if I now attempt to actually art the service again so we do a system CTL start mem Hawk desktop service and I do a status I actually see that the systemctl daemon reload is recommended here so I'm going to go ahead and do that because every time we change limits this limits need to become effective in system D so let's try and stop and start again the mem hawk dest dot service and status and this particular memory service is actually running at the moment let me clear the screen now because we had too much information but if we wait a bit and then do again a system CTL status mem Hawk desktop service we're going to see that eventually the service fails to start because the reason it fails and the reason it fails to start is that now it tries to acquire more memory than the 1.2 gigabyte limit that we have just placed so that in effect demonstrates very easily how one can actually create a hierarchy of processes and apply a memory limit so this is something that you can typically do with a Linux control group thanks for watching

Channel: Georgios Magklaras

Views: 26,689

Rating: 4.9298244 out of 5

Keywords: GNU/Linux (Operating System), Linux Kernel (Software), cgroup, cgroups, Linux Control Groups, systemd, systemctl, Red Hat Enterprise Linux (Operating System), RHEL, Demo

Id: 81j1WF5xEZc

Length: 24min 7sec (1447 seconds)

Published: Tue Jul 22 2014