Linux Monitoring and Logging | HackerSploit Linux Security

hello everyone welcome back to the linux linux server security series in this video we are going to be taking a look at logging and system monitoring on linux now it's it's very important to understand the context in which we'll be looking at this because uh when we talk about things like logging on linux uh this uh this can be quite a complicated an advanced topic uh given the various uh the various forms of logs that are going to be collected on a linux system so we are primarily going to be interested with the security logs or logs related to security events on the system uh um we are then going to be taking a look at monitoring your system so uh analyzing the various processes that are currently running on the system i will take a look at how to monitor user activity on a system and how to view what each user is doing so on and so forth so to begin with uh we have already taken a look at on a very basic level how our logs are used on linux in regards to security events for example on the youtube series and on the lynnode series we covered how to use fail to ban for brute force protection and i actually mentioned explicitly that fail to ban utilizes the auth log under the log directory right over here so again if i list out the default log directory and that is var log and i'm just going to list that out you can see we have plenty of logs here and as i said linux generates a lot of logs um you know various services generate their own logs so on and so forth so you know you have a kernel log you have your mail log syslog if you have that installed now we're not going to be covering syslog and again you know the various log analyzers that you can use because uh that is something that will require its own video however when it comes down to security we can cover the most important uh so number one is the auth log now the auth log contains information uh about you know various security related events uh number one it it it involves authentication attempts failed authentication attempts and it logs important information like the service uh the service in in in question so for example if there's a failed authentication attempt via ssh it lists that out and it actually it actually lists out the time and the ip address that made that failed authentication attempt and that's why tools like fail to ban can actually come in place and monitor this and say okay within the space of two minutes we received more than five failed authentication attempts if that's the case take the ip address and add it to uh or create a firewall rule to block that ip for a certain amount of time so it really is very simple to understand it that way so if we display the contents of auth.log here so i'll just do that right now so cat and i'll say var log and we'll then say auth.log and we hit enter you can see we have a ton of information here that might seem a little bit confusing but i'll try and explain it as best as possible um so when you take a look at a log file you should you should have an idea of what you're looking for now we talk about the auth.log file as i said it's related to authentication attempts in this case you can see it lists out the time and date uh the the actual host in this case localhost it then lists all the the daemon that you're connecting to in this case we have login d right so systemd logind and then we have sshd which is the ssh daemon and then you can go down here we have a cron we also have sudo where there was an authentication attempt made and you can see it gives you the command that that was actually used here um so again you can go through all of these logs and and monitor and you can look for exactly what you're looking for you can search for exactly what you're looking for uh using various utilities on linux uh one of these utilities is grep as i've already mentioned so if we're looking for ssh authentication attempts for example we can say cat the auth.log file and then use grep right and we can say with grep i want you to look for a pattern and i want you to match this particular pattern so we can say sshd for example and hit enter and that is only going to show up the ssh authentication attempts or any logs associated with ssh in this case you can see it matches them out and now we have a much clearer picture or a much clearer set of data that we can deal with so for example we can see that on october 1st uh at the given time uh you can see the server is the server starts listing on port 22 and then there was a disconnect here and we then add a pam authentication attempt and that looks like there was a failure right and then there's a failed password entered right over here you can see in it that that's my ip the reason i did that was to actually show you this so before i logged in successfully i logged in with an incorrect password so that there would be failed authentication attempts so that was these were these two sessions right over here and then the connection is closed uh and then you can see i log in with the correct password and it tells me accepted password and it then gives me a session and then there was a disconnect from another ip it looks like and there was another disconnect right over here so it looks like we had another ip trying to connect uh but that being said that's um that's how you can look for uh for particular pieces of data you can use the grep utility and of course it doesn't have to be just ssh authentication attempts it could be any other service that yeah that utilizes authentication um so that's the uh that is the auth.log file as i said you can look for failed authentication attempts so again i can search for uh authentication failure so i can say authentication failure within this file and sorry that is an incorrect spelling there so fail authentication failure and it lists out the various authentication failures right over here and it gives me the information regarding that so this as i said is typically our services like fail to ban uh work they work by looking for particular expressions within the um within the particular log file and if it finds that uh that particular expression it'll then look for the ip address or the value of our host and then it will add that to the firewall rule to to actually block that ip for a certain period of time so again these tools all utilize the log files um so that is how to list authentication attempts now let's take a look at how to list all login and log out attempts and again the log file that is responsible for that if i can just list out the logs one more time so far log and i'll just hit enter the file we're looking for is a file called wtmp now if we try and get the contents of um var log and wtmp uh you can see that it's going to give us a bunch of uh you know random data here that is really isn't readable to the user and that's because you have to actually use a utility called last so if we open up the man page and we use and we search for the last command the last command allows you to show a listing of the last logged in users so in our case we want to specify the we want to display the host name um we also want to display the ip address and let's see what else do we want to display and the full times so the login and logout times so if we want to do that we say last and then we say a i f right so that was capital f and we hit enter so it's going to tell us right over here that we logged in root and we are still logged in and it gives us the ip so this was the user um the date at which the user logged in whether they're still logged in or logged out and the ip through which they log in so for example if i just get out of here and i log in one more time and i'll just enter my password in here so there we go and then my password and uh i then try and display and write that command again you can see it's going to tell me right over here there was a log off right uh this was the login time and this was the log out time and it gives you a duration right over there and it also gives you the ip so very very useful um very very useful utility you can also use we can also take a look at the btmp file the btmp file and the wtmp file as we've just taken a look at these are all binary files which means you need to use the last command to to actually utilize them uh so um when we use the uh when we're taking a look at the other files so if i let me just list out the logs one more so var log and i'll hit enter if we take a look at the the btmp file uh the btmp file is similar to the wtmp file um it essentially lists out all the um the bad um the bad login attempts or the failed login attempts so again we to you to actually utilize this we'll be using the last command uh at the last b command so if i list out last b here and i hit enter you can see uh last b shows uh in this case we're using last b so last b shows a uh listing of last logged in users in this case we want to display the host last um the host last so this will display the host name we also want to specify the dns so we'll display dns and then we'll also display the full times so again we'll say last b and then adf so a d f and then we hit enter and that is going to display all the last logins but what's more important is we need to actually list out the user so we specify the user root and we hit enter now and you can now see it lists out the last bad login attempts in this case uh these were the attempts that were made uh when i had actually set up the server initially uh and as you as you recall i had actually done this intentionally uh so that i can actually demonstrate this and show you that this does work so again uh if i if i if i log out and i log in and i and i try and use uh an incorrect password and i use the last b command this will display all the failed login attempts or bad login attempts for that particular user uh right okay when we talk let's move on to the to some more information if you want to find out a user's last login time or when they were last logged in which again can be very important from a security perspective we can use the last log command so if i clear this and we search for last log and i hit enter you can see last log will report the most recent login of all users of a given user so we can use and specify the user here so in this case i can say last log user root and you can see it tells me the last time i was logged in was thursday october 1st and it gives us the time there we because i have another user here so if i say cat etsy password right i can hit enter um we can see that i also added another user called alexis if i try and log in in another tab here uh let's see if we can actually log in sorry with the user alexis um i actually added this user to the server so i'll log in with the user alexis here it should actually prompt me to enter a password is that the correct ip address for the server no it isn't sorry let me just specify that one more time that was an incorrect server ip so i'll enter in the password for the user lexis and um again if i just go back in here and display that one more time i can now use the user alexis and that will display when alexis was last logged in in this case it gives us the time and the ip address from which the user logged in right so that is again how you can list in the when exactly the users logged in last and you can specify the user you're interested in but what if you want to find out what users are currently logged in so we currently have about two users logged in i'll just change this to bash right and if we if we want to display that we're going to be using the who command all right so the who command essentially displays who is logged onto the system and again if i just clear that out and we're now moving on to the system monitoring uh side of things so if we say who and i hit enter you can see it's going to tell me alexis is logged on and the user root is logged on it gives us uh the actual date and time and the ip address right so uh that is how to view uh all the users that are currently logged in now let's talk about monitoring system processes um and again there's tons of utilities you can use to do this if you if you want to monitor your resource consumption uh your network consumption uh any processes that might be taking too much uh that might be consuming too many resources it's a vitally important as a as someone who is interested in securing their server to understand what's running on their server so to do this the typical utility that would come with any linux distribution is a utility called top and top is a is a very very good utility because it allows you to sift through all your system uh processes and the one problem with it is that it does not display all of this information in a human readable format because you can actually see that it really doesn't make that much sense because even in terms of the free memory you can see it tells me there's about 981 megabytes of memory available 183 is free so again yeah that might make sense and then of course it sorts out uh the it sorts out all the processes here from their process id and you can scroll down and yeah so it's a very good utility that way but another great utility is h-top right so make sure you have h-stop installed so to do that i'll just say sudo apt-install htop and hit enter it should come pre-packaged with most linux distributions so for it h-top h-top is much more clearer in regards to uh in regards to what's going on on the system uh secondly you can utilize advanced things like the search uh the search filter or you can actu you can use search options or specify the search of a search filter you can also specify um a tree the tree listing so for example if i list out the tree listing here you can see it actually lists out the tree of processes and you can actually see the commands that have been executed right over there or that are being run in the background right so i'll just get rid of that so go back into sorted mode you can also sort it by if we use f6 you can see you can sort it by various various options that might be important to you so for example we can sort it by the amount of ram being consumed i hit enter that's going to sort it by the highest the the process that is consuming the highest amount of ram in this case you can see it's a utility called glances and i'll get to glances in a second i can also say i want you to sort it by the highest amount of cpu consumption so i can hit enter force that or i can use priority or the user so htop is very advanced that way so i can then hit enter and it's going to sort it by the highest amount of cpu being consumed and it lists out the process id the user responsible and then of course all the other resources that you can then take a look at there of course you have the ability to kill processes or to search for them so if i want to search for uh let's see i use f3 and i type in uh glances right i hit enter and you can see it locates glances for uh for me if i want to kill it i hit f9 and then i can send the a particular signal so for example if i want to kill it i can send the kill signal uh if i want to stop stop it i can send the stop signal and if you're familiar with linux signals that should be perfectly simple for you to understand um so again it's a very very uh useful utility that you can use to monitor your system processes and to understand what's going on in your system you can search for processes via process id uh you can list it out based on the tree or you can actually list out the tree uh the you can actually list out your processes in the form of a tree uh you then have the ability to kill uh processes so on and so forth um now let's take a look at glances uh glances was the other utility that again is still used used or utilized primarily because it displays network activity so again to install it we say sudo apt install glances and we hit enter right and it's already installed so if i open up glances in the man pages here you can see the the basic description is that it is an eye on your system uh glass is a cross-platform uh curses based monitoring uh tool which aims to present a maximum of information in a minimum of space ideally to fit in a classical 80 by 24 terminal and again the goal of glance is to give you as much information as possible so if i type in glances right over here and hit enter you can see glances is much more it gives us much more information than we would typically have with h-top for example but again this gives you a glance of your system h-stop is really uh to do with managing your processes so in this case you can see we have the cpu usage memory usage swap usage as for the um as for the other options it gives us our public and private ip if we have one we then have the uptime as for the load we can see it's running on one core uh we we have the swap uh sorry the swap memory usage right over here the total allocated is 512 megabytes so on and so forth and then of course the most important is the network traffic so you can actually monitor incoming and outcoming traffic based on your interface so for example if i start downloading packages so i'll just try and update my repositories here so if i go into the user lexis which actually can do so i'll just switch to root here and then i'll try and say sudo apt update right and i hit enter and if i go into here you can now see the uh the traffic will increase so you can see we have download and upload uh and they both increase and reduce with the amount of traffic being consumed so this can be a great way of monitoring if there's anything weird or suspicious going on uh in regards to network traffic and then of course it lists out all your processes and it lists out the cpu consumption from the highest first and the memory consumption in this case um you can you can actually go through the processes uh so again when it comes down to process management uh you can actually use each top to again manage them in regards to searching for them killing them stopping them halting them but glances gives you an overview of your system and lists out as much information and as much useful information as possible all right uh that being said let's move on to the uh to some other commands and utilities that we can use and uh the last one that i actually wanted to show you is the ability to list out all the logged in users uh and when i say logged in users we of course used the you who command however the who command doesn't give us a lot of context as to what exactly is going on on that other user session so i'll switch back to alexis here and i'll just go into bash and let's say i want to run glances right so i'll just hit glances on the user lexis and let's say alexis is doing that uh the tool i want to install is called who watch so to install who watch i'll say sudo apt install cool watch and i'll hit enter right and i already have that installed so if i open up the man pages so who watch right over here and i hit enter so you can see who watch tells us this is a console interactive process and users monitoring tool so who watch is a console again i'll just skip that it displays information about the users currently logged on to the machine in real time uh now the keyword here is information so when it say information it will list out the users that are currently on the system uh that are currently logged onto the system irregardless of whether it is uh as as it says right over here uh via ssh or telnet or actually physically accessing it through a uh physical terminal session as opposed to a remote connection so you also have the ability to kill their process that they're running so it really is a very very very cool and advanced tool that allows you to manage uh what users are doing on the system so again you can take a look at the the commands right over here um if i just list that out so um you can and you can use the enter key um to actually view the selected users uh process tree and then that will list out uh the processes or you can use the t option to list all system processes and then you have the ability to scroll up up and down and then control i will send the end signal we then have the control k to send a kill process so we can actually experiment with this so to do that we'll say who watch hit enter you can see it now displays the two users that are currently logged on and it does it sorts it out really well it says zero local zero telnet and two ssh and then if it's if there's another session uh now when we refer to other sessions these could be reverse shells set up by attackers uh you know either through netcat or through any other scripting language they could have gained access through the web server so this is a great utility for monitoring who has sessions on the system either locally or remotely so again i can just hit enter for the user lexis and you can see it lists out the processes that are currently running so in this case you can see bash the commands i run were bash so it logged me in with a default uh bound shell i then said i want a bash shell i switched to the user root and then i went back to the user alexis and uh spawned my bash shell again and then i run the command glances uh so again i can go to that particular process you can see it's currently running uh right over here and then i can say control k and that should have killed it um so there we are you can actually see it is actually killed and it gives us a terminal session right over here which is pretty cool so you can you can actually remotely kill processes that other users are running for various reasons if you're a system administrator that is very very helpful so again if i run another command like let's say h top right and of course i'm just running very basic commands here to demonstrate what i'm talking about you can see in real time it actually displays uh what is being run by that user i can hit ctrl k and that will actually terminate that for us so um wait one second let me just open that up again h top and i'll open up the user lexis here and we'll go back in there and we'll say control k and that should close up h stop for us and there we are h stop is now killed and i'll just clear the terminal there so again a very very useful utility for monitoring what our users are doing on the system how they're logged in it can reveal a lot about your system i've actually used this tool many times to spot reverse connections uh that were established through uh various vulnerabilities and services so i really recommend that you use who watch when monitoring your systems uh that's pretty much it regarding uh you know security monitoring your system and logging uh when it comes down to security as i said these are uh these are very advanced topics especially the the the topic or the concept of logging primarily because linux generates a lot of logs and as we know within security logs are very important and you can actually use various tools or log analyzers to analyze these logs and make sense of what of what all of these logs mean on a larger scale and that's why you have various uh you know seam tools uh like uh like ibm qrada that actually allows you to use syslog servers and then you can then pull all of these logs into one centralized location and then process them and try and make sense of all of this that being said that's going to be it for this video thank you very much for watching and i'll be seeing you in the next video [Music] you

Channel: Linode

Views: 22,473

Keywords: linode, linux, cloud computing, alternative cloud, linux server, open source, sysadmin, linux admin, linux monitoring, linux logging, linux logs, linux who command, linux whowatch command, linux security, auth log, linux glances, linux htop, linux for beginners 2021, linux operating system, linux commands, linux security tutorial, linux security course, linux security hardening, linux tutorial, linux tutorial for beginners, linux server security, securing linux server

Id: kZ5LhS6fThM

Length: 24min 55sec (1495 seconds)

Published: Mon Apr 26 2021