Graylog: Your Comprehensive Guide to Getting Started Open Source Log Management

foreign [Music] systems and we're going to talk about centralized logging with greylog if you're not familiar with gray log it is a free and open source logging platform that is absolutely amazing at consolidating all of your log data and then creating correlations queries and even alerts off of all that log data and having it all in one place makes this really easy we're going to talk about how to get started with deploying it from scratch all the way to setting up extractors managing streams building your indexes and making sure you understand what all that means I've got some visuals I've also linked to the greylog documentation for anything that goes out of scope of here hey there's a lot more that can be done this is one of those platforms that you can just keep tinkering with and keep expanding on and really create some interesting correlation data between all of your logs or even parse it and create alerts in some very interesting ways we're going to be covering all that from the alert system to the extractors to the streams and what all that means now this video is not sponsored by greylog but I did do a video on greylog 4 which many of them send me a shirt which by the way I poked a hole in it so someone if greylog is watch watching I wear an extra large my address is uh you know over on my website now even though this is not sponsored by greylog if you would like to support this channel there's some affiliate links down below or you can hire us for a project head over to my website launchssystems.com there's a higher spot in the top let us know what we can help you with and we do Consulting for many of the things that we talk about on this channel now let's jump right over to greylog we're going to walk through exactly how to get it to play with Docker everything's going to be on my GitHub and because this video is static but my GitHub is dynamic and I keep updating it you always find the latest extractors that I'm using on there and the latest installer if any changes come after this video to the installer the latest version will be there everything's linked down below and time index down below so you can jump to the part that's most relevant to you so let's jump over and start getting gray log installed now I'm doing this demonstration on Ubuntu 2204 but it's Docker so it should work with really any distribution that supports Docker one thing I will make note of is if you install Docker as part of the install for Ubuntu at least in version 2204 it wants to install it as a a snap if you install it as a snap instead of like aptkit install as a package well it's going to work differently and the problems you'll run into go out of the scope of this video so there's important reasons not to install it as snap unless you want to solve the other issues that may come with that particular challenge now the next thing we need to do here is make sure we have the time zone set correctly sudo time control set time zone UTC it's actually should have been the default for setup on here but I do really recommend that everything you've done in UTC or you'll have to do a little time skewing and mess around with this in terms of gray log it's important that your logging server have everything in the same time zone once you've said that the next thing to do is install Docker so we'll do a sudo apt-get install Docker compose and all the commands are actually in a forum post link down below so you don't have to try and copy any of these off the screen you can just copy and paste them out of the Forum post now an important step is making sure that the local user has the ability to run Docker so we're going to use sudo user mod Dash JG to add the group Docker with this particular user once it's added just exit and go back in if you don't do that or if you want to restart that would solve this as well you won't have permission to run Docker until you have those permissions applied by exiting and going back in yes there's other ways to do it that's just the quick way next we're going to get clone right from my GitHub the latest version of the docker compose now before we go into customizing it which we're going to do in a moment you may want to generate a new password the default password is going to be admin and the default user is admin probably not the best to leave those so you should probably come up with a better one and this is how you do it you run Echo dash n your password pipe sha sum-a 256. whatever your password you want it to be your password one two three four you know make it a little bit better this will create the shots some of that you just copy this and there's a spot we'll cover inside the docker compose file where you would paste that in now let's look at the docker compose file now this defines the networks right here at the top so we're going to call it graynet driver Bridge that's fine unless you have some special use case where you want to do something different the next is the volumes Docker has application data and volume data stored separately that way the applications can be updated and changed and are always pointing back to the same local storage if we have the mongodata log data and gray log data is the database used not for storing logs but for storing configuration inside of prelog so it's not that big and we don't mind keeping it local log data on the other hand we're using open search in this setup and log data can get very big so we'll talk in a moment about another way to configure it but this keeps it all on this local VM an ideal situation is to have a shared Mount that you're going to mount and put that on there we'll cover that when we get down to the open search settings gray log data driver local this is not much data but these are some of the settings for gray log itself then we'll go down here and this just pulls the image from manga go Docker and we have each one of these set to restart unless stopped what this does is once we start this up it's going to keep these Services up and running even starting back up on restart of the server itself so it'll always be ready if you have to reboot it now in the open search is where we're going to do a little bit more customization this is where I have it pointing at just log data but I could also have it pointing at last slash Mount slash log data if I had a mount somewhere and this is something that you may want to take into consideration because ideally if you have terabytes of data you don't want them to stick that in a VM that's not good storage design so my production system just has this volume pointing at another Mount now if you do that you don't need to define the volume up at the top but it doesn't matter if you leave that in it's not really using it for anything but this is where you would change the part before the colon to where you want that mount to be if it's external now we're going to scroll down a little further and leave all the ports and everything else the same it's attaching to the same gray net same restart policy and let's go down to the password part so gray log root password now they say root password we're not talking about the virtual machine itself here they're talking about the admin password so this is where you would paste that password in right now by default it's got the password admin and this is also something you may want to change as well gray log password secret so these are a couple more things there's more in the documentation about customizing this but at least change something better than admin admin because even once you log in you cannot change the admin password it always pulls whatever hash you have here and that's what's going to check for the password now as we go down it's going to be bound to Port 9000 that's fine there's not a https or SSL sort on this uh going and putting a reverse proxy is not a bad idea on here or tying it to your existing reverse proxy it goes out of scope of this video next is setting the time zones yes I have it in here twice because it solves some problem I was having and I can't remember which one of these is the one that's supposed to be but if you put them both in it solves all the little quirks I was running into so someone has a better answer for which one of these is the right one leave it in the comments down below or just leave both of them in there greylog email transport this is something that specifically if you're installing a great log with Docker you don't want to customize any of the server configuration files you want to customize the docker compose file to put all this in here I have it mostly pre-filled out here using Duo circles mailhop I'm not affiliated with them but they do offer free mail service I know it works for gay log so I threw it in here but throw in whatever mail server that you have and want to have set up in here so transport protocol set to SMTP the interface URL this is actually the IP address of this server that we're doing the demo on but customize this to even a fully qualified domain if you have to this is going to be where it points to when you have an email so you can click on it and get right back to that gray log Port this is all part of the customization it is not required for greylog to work that you have email because it actually supports other alert methods such as webhook but this is if you want to use email configure all this in here or you can just eliminate all this part to all together if you're not going to use it scroll down a little further a few of the networks service started service depends on so it says not to start gray log until we know the conditions of service started are met for both and open search and then we have the ports defined here defining ports is really easy so if we wanted to Define them and this is my methodology for doing it I like to create a different port for any type of server that's coming in this just keeps it easy and for example if we said 15 16 and we said 15 16 again slash UDP say those are my true Nas systems so everything true Nas is going to go to that Port that way all the tools and all the extractors and all the customization I put to the input when we get to that later in the video you see you'll tie that to a specific Port now there are other ways to do this you can't have everything going to one port and then separate and track by hostname and land things differently but I prefer this method right here and so that's what we're going to go with and it's how I run my main servers so once this is all customized we'll just go ahead and close all this save it we're going to just Docker compose up and this is going to go and get the latest versions of all of the docker images pull them down extract them and kick it off and running it's going to take probably just a couple minutes to get this set up we'll jump ahead real quick to this up and running all right gray logs up and running so we can log in with our admin admin because I didn't change that and we're in we have Ray log up and running on this system now we're just going to log back out well it doesn't have to I'm just going to shut it down into shorts I want to cover how to do that so it's up and running but I would see if we hit Ctrl C or anything on here or leave this session it's going to stop so we hit this it stopped it but before we put it in Demon mode Let's talk about a couple commands to help clean things up in case you need to do some for their customization so first we just want to stop it now we can start it up again with Docker compose up Dash D what this does is throws it in the background so you don't have to leave this session open in order for it to work so now it's up and running in the background and it doesn't take long to up and run because it already pulled all the images now if we want to take it down we can just go Docker compose down and with the Dash D running in D mode it will automatically start with the system restarts so now this brings it down now the reason you may want to bring it down is you want it down while you edit the docker compose file so if we want to go back in add some ports change some settings change that admin password that I had forgot or put those mail server settings in this will go ahead and give us the opportunity to do so but if you wanted to start over because you broke everything you can actually do a Docker compose down Dash B and careful with this command because this will destroy all the volume data and it's now erased and released all those volumes now the docker images for the application are still there so if we do a Docker compose up again we'll do a Dash D you can see it's creating the volumes again so all the data containers will be recreated and now it's back up and running with fresh data in there this is a good way to start over completely if something has gone wrong you've messed up the configuration you've locked yourself out of it and want to just clear everything that's all you have to do but Dr Cabos up to SD we'll leave it like this running right now because now we want to get to the customization of the web interface and show you how to actually get your logs imported in here now before we go through and set up and configure gray log I wanted to cover real quick the flow of data and how it gets parsed inside the gray lock system and how it lands to its different indexes so we're going to start here at the server or device that's sending the data to the defined input port and type TCP UDP and then inside there you actually Define what type of data is coming through there then we have the extractors that are attached to that input and you don't have to have an extractor but if you do have an extractor an extractor matches certain data it's going to parse them into different fields if you don't have any parse data and you can actually mix this where the match is only where certain things with the parser but not all of the data the structured data goes through into the fields but the unstructured data gets stored it's just a long string or a message so it's not a requirement that you have it you can still search unstructured data but it's even better to have structured data so you can create statistics around it then from there we're going to go to a stream Rule and if it matches the stream rule it will then go into the defined index that you have or the default index that you have now we'll cover how to create these indexes as well but it's really important to understand how this data goes through there's even more elaborate things that are not on here such as the way you can pipeline data and process it in different ways that is all through the gray log documentation but we're going to walk through this process to get you started with the basics to get it up and running and start capturing some structured data we're going to use PF sensor as our example here now very quickly I'll show you how to turn on logging for PF sense just so you understand how it works you're going to go into the log settings we're going to put it at Port 1514 and we're going to put the IP address of the server now whatever report you may Define may be different but in this one we Define 1514 as UDP and then we just want to send everything that's coming into this PF sense to that particular logging server we're going to leave the log message format at the default BSD RFC 3164 so that's all the default for psense and we'll go ahead and build off of this as the example now we're going to log into gray log and this is the main interface where you'll see your most recent logs but we haven't configured any inputs yet so there's no logs matter of fact there's a warning right here that says this is a node where there are no running inputs so we need to create some inputs before we do that I would recommend creating a new user you go here to users and teams you can create a new user then first name last name Etc set any of the specifics set the time zone for this specific user and then of course set the roles such as admin now you can just use a system as admin but I'd recommend having a separate user but for Simplicity in this demo we're not kind of bothered with that step we are going to go right over here to system and then inputs and create our first input to create an input we're going to go and choose the input type there's a lot of different options in here but we want syslog UDP launch new input now the option to set this globally or on a specific node depends on if you've done a larger install with multiple nodes that is something you can do but autoscope of this particular video and we're going to use this as a PF sense input for our example here bind address just leave it at this unless you have specific IPS that you've set inside your gray log setup to bind it to but we're fine with binding it to the single IP that the system has and this if you remember from the docker config was Port 1514 it's also where we told pfSense to send these logs you can leave all of this down here at defaults I go ahead and store the full message and we launch the input and when she launched the input you can see how many messages are coming in right now it's empty oh cool we have some type of message coming in if we click on show receive message we can confirm that yes we have messages data is coming in as unstructured because we haven't added an extractor it's all just all on one line right here so Source was filter log message is here the full message as it was stored was right here as it came from pfSense so let's go ahead and put an extractor in here so we're going to go back over to our inputs and we want to manage extractors now they have a wizard essentially that will help you create extractors and then you have the message ID and recent message where you can just say load this message and then build a extractor from here this is a little bit more in depth this is going to be out of scope of this video but I will show you how to import existing ones they do have instructions on how to build these extractors within the system and how to pull the messages one thing I will note that can be a little confusing so if you want to load a specific message and let's go back over here to load a message for you the message stream it wants is first this number here is going to be the message ID and then we have this here where it says stored in index gray log underscore zero so you can look at these messages and to our inputs and manage extractors create extractor message ID and then we would put here gray log underscore zero this can slightly confusing things if you don't know exactly how that's stored and what they're asking for so you can load a specific message to build an extractor for it we're going to make this really simple and we're just going to import the extractors that I've already created so here's the Json file it's asking for and we're going to switch over to my GitHub we already have a PF sense extractor set up so we'll go ahead and click on this one it's going to be the latest whenever you're watching this and we're just going to click copy and then paste and now we have all of this in here we say add extractor to input now when we go back we see all the different extractors I have one for icmp Sierra cat alerts openvpn filter log mpf sense filter log UDP so TCP filters UDP filters if you feel you want more as you can look through these and see how they're configured and then start creating your own from this now to see how the differences between the extractors and we just need a little more data so we have the extractor but now we have some more data that's come in so let's go ahead and look at what the extracted versions versus the first versions look like so if we scroll down and look at the First Data that came in it's all just on one line as a message we look at the more recent data it's all broke down to data length destination IP destination Port Direction here's the raw data then it has the flags ID IP version Etc so now it's all broke down into very specific Fields this allows you to do things such as add this to the query and now we're specifically querying for any filter that matches this specific Source IP matter of fact because these are now structured tables in here we can do things like look for Source IP or Source port and then use these as filters and it tells us how many it's found so far for these one is unstructured you can still find it but your parsing data just as text as opposed to as X very implicitly listed ports IPS Etc Now by default gray log is going to put all the data into one main index let's create a new one just for pfSense so we create an index set pfSense and we'll just fill these out to keep it simple now you can get granular here and control the different options for exactly how you want these to be broke apart there's some guidelines within the documentation for this there's also the more important part is the guidelines for how many days you want the rotation to do or even if we change this to M it would be month so we want to rotate once a month or every two months three months and then how many of these deleted indexes do you want to keep or do you want to do nothing with them and just let it grow forever which is a terrible idea now besides a rotation strategy of index time it can also say index size or even index message count so you can change it to be a certain size if we say we don't want to exceed this this helps with logs that well can get kind of unruly when you're doing something like turning on debugging where you aren't worried as much about the retention in terms of date but you want to retain only so much data because of some type of size constraint you have so these are the fine tuning you can do for any of these a lot of time time is going to be a lot of people like and there's more guidelines within a gray log documentation for all the different ways you can do this but we'll leave it for now of rotate each day and we'll say delete the index after a number of 30 days of rotation and we're going to create this index set now creating a set does not put any data in it all the data is still going over here to the default index sent but you can change and say set as default if you want it to default to all data in here but we're going to go ahead and show you how to create a stream to push the data specifically into this PF sense index and to do that we have to go back over to our local inputs now you don't get to choose within the input where the destination is in here this is what the stream is going to be for so we're going to go ahead and click show receive messages because I want this piece of information right here that says GL2 Source input and then has this ID this is the ID for the input that way when we're creating streams and we'll just make sure we've copied this so we'll go and stream and we want to create a stream and this is just going to be our PF sense what's going to be routed in here PF sense does it wrap to the default index no remove matches from the default stream this is going to create a rule so first we create the stream and then we create a rule for it that says these messages have to go into this particular index so we're going ahead and add a stream Rule and that's where we put in that GL2 source so it says GL2 Source input and the value is going to be that ID that we matched so GL2 Source input match exactly and then the value of this and hit create rule then you can say I'm done and start stream now what this is doing is using this specific index as this stream rule hits and the data is going to flow hit the stream Rule and go into that index so if we go back over here to our search we can see all events default stream or let's look at the pfSense one and get some data in here and we can see data is already flowing into here so this allows you to filter for that very specific new index that we created and once again the data is all structured it's all going into that index if we wanted to purge out of the other indexes we could go in there and delete things if you want but I'm not worried about it they'll fall off with the default indexes or you can go through here and control those settings manually back in the index set now I moved over to our production system because I want to show you how the alert systems work so we're going to go over here to alerts and we have like for example tracking openvpn logins tracking pfSense web interface logins I have different triggers for different things so I have an idea of what's going on and have these event definitions to find before you can define an event definition you have to say where you're going to send it and that starts in here notifications now I have a basic email notice basic SSH login email notice and a LTS slack VPN notice the reason each of these are different well let's actually edit the slack VPN notice here and we'll jump right to the part where you customize the message because this is specifically a VPN notification I only have it pulling the VPN variables that I care about so this right here and I'll throw this also in the Forum post link so you can see exactly how to put these together but they do give you a default template whenever you create a new alert that you can start from and then reduce the amount of data that you want to narrow or increase it to the exactly the right one you want for me I only really care about the username what IP address they logged into and what time they did and it just sends a really simple slack notification to let me know exactly which user logged in for my company now creating a notification itself is really simple so we'll just call this test give a description a test and you have a few options pagerduty slack notification teams email or using webhook with HTTP so if we were to do for example another slack notification or email provided you'd set the email up in the system all these come with these really basic templates that are set up here that I well reduced the amount because I just didn't want that much information sent to an email but as I said these are really cool because of the way they can be customized you can find some help also in the greylog forums for different ideas that people have had for doing this and and you can also execute the test notifications in here when you're choosing the notification type and you choose something like email it will automatically for example send it to one of the users or you can type in an email address so it can send it to me or you can manually type in an email address here or multiple email addresses as needed if you need multiple people sent notices for these things now let's talk about the actual triggering of those notices once you have it defined and you first before you define the notification you have to understand what data you're looking for so let's filter something and we did process cerakata and Source IP an internal IP of 192.168 4.104 I have the cerakata data piped into my syslog of my PO sense and then I have that extractor that we put in that allows the parsing of this and this is what it looks like so we can see the alert was a alert message was cerakata TLS invalid cert alright and we can go down we can see the process ID of cerakata so we matched on that up here as processarakata and we see the source IP is that and let's say we want to create alert based on that so now that we know I can see that there's event definitions and we're just going to go ahead and copy this we'll go back over to alerts and let's go ahead and Define a new event so create event definition and we can give the title bad TLS or 104 get more descriptive down here 192 1684.104 went somewhere without a TLS certificate all right we'll go next what's the condition type filter and aggregation all right what is the search query that we would like to alert on and that search query we'll put to this and we can go more granular because we don't want the system doing too much work so we'll say yeah this is specifically going to be found in the pfSense stream right now it's going to confined any we could load a message ID to make sure it matches so process and Source ID filter aggregation search within the last five minutes let's for search for an Alaska uh let's say eight hours Let's look back hey look we found some of these that are hitting and that's what this is telling you is when you search back does it find any conditions that are met for this execute how often do you want this to run filter has results aggregation research from a threshold so you can keep getting a little bit more fine-grained down here we'll just go we're fine with it hitting on there then we would go next I'm not worried about any custom Fields add notification this adds the defined notifications that we have so you can create a new one right from here going back to that process or we can just pick one of the existing ones basic email notice I have set up to just send me a dump of all the data that comes hit next and the system's just confirming how you set up these alerts and then you can create the event definition and it'll go to alerts and perform the actions and the tasks that you defined now this was enough to get you started with greylog get you extracting data parsing in the fields but there's so much more that you can do you can even pull in external data sources you can use this to consolidate all of your logs for all of your services I do and then start creating different extractors for different things and as I said I'll keep my extractors up to date or any new ones I may add in the future or maybe you're watching the future and there's even more available now on my GitHub leave your thoughts and comments Down Below on what you like or don't like about greylogger other things you'd like me to do a tutorial on especially around greylog also like And subscribe it really helps out the channel and uh head over to our forum for a more in-depth discussion for this or any other topics on my channel and thank you [Music] foreign [Music]

Channel: Lawrence Systems

Views: 121,597

Keywords: LawrenceSystems, centralised logging, open source log analysis, graylog, graylog tutorial for beginners, graylog docker, graylog dashboard, open source, syslog, syslog server, syslog protocol, log management, log management process, log management tool, log management server, log management system, log management system design

Id: DwYwrADwCmg

Length: 27min 14sec (1634 seconds)

Published: Tue May 09 2023