Azure Sentinel Lab Series | Setup Syslog Collector and install Azure Sentinel Agent | EP1

hey guys so this is going to be an azure sentinel lab series it's going to be a little bit advanced but if you stick with me we're going to set up a syslog collector you know on a linux box from scratch you're going to understand how syslog works you're going to understand how to filter you're going to understand how sentinel logs are sent to sentinel and understand logs work and then from there once you get the logs to sentinel we're going to understand queries we're going to build some workbooks we're going to build some dashboards some notebooks we're going to build some rules we're going to have those rules trigger automation to open up a servicenow ticket or call up an api enrich some data and have and show you the power of sentinel in this lab series how does that sound so um again this is going to be a little advanced but i'm going to try to explain as best as i can we do have a become an azure sentinel training uh level four it's a really great uh blog so i'll include the link in the description so you can learn more about sentinel i'm not really gonna just go too deep into explaining how sentinel you know over overall works there's a lot of videos and guides around that this is really just setting up a lab and getting our hands dirty and and just laughing out okay so you're by the time you're done you would be able to build queries dashboard workbooks and all this without leveraging a template at all right you're gonna know how it works but before i get started make sure you know kql so check out my kql series it's about four videos um and um i plan to add more later but the four videos should get you enough information or enough foundational knowledge to be ready to understand a lot of the content here in this training before i get started in sentinel i wanted to take the time to teach you about logs i think it's important because in order to stand sentinel you got to understand logs you got to understand the data you don't want to just send all your logs into sentinel and then just expect it to work you know you have to understand kind of the data and what you want to do with the data a scenario would be let's say you're getting a suspicious travel you know maybe azure identity protection notified you that they're suspicious there's a suspicious user but you don't want to do an alert yet you don't want to engage your team yet so you want sentinel to go out and find out more about a user and then from there have that data already available so that when you open up a ticket the ticket already has uh enough data that the analyst can work with that he doesn't have to go out and find himself so you're enriching the data with other data points so centono can do that for you because you're ingesting it from your third parties you're ingesting it from your microsoft products right all this data is coming in and you're trying to correlate that data and do automation and orchestration with it okay so before we get into sentinel we want to understand the logs syslog is a very common format for logs also very common is ceph logs ceflogs and syslog all kind of flow in the same port usually on five one four and here's a here's a question is syslog tcp or udp it's a trick question uh it's both the default format you know what protocol uses is udp udp is not is connectionless so it's not going to verify if the packet made it to the target you know the syslog collector you don't always want to expect that the log if the log isn't there it doesn't mean it didn't happen because maybe the log never got to the collector and never got to sentinel so if you want to make sure the log gets there you want to do tcp okay say let's just lock via tcp but like again by default this udp so just keep that in mind but for this scenario we're just going to go with the defaults we're going to try to keep it basic and give you an understanding of syslog okay and i'm gonna show you some examples if you wanna follow along spin up a linux box whether it's a docker whether it's a ubuntu vm whether it's in azure or aws get a get up linux box okay and just get just get access to the box and then i'm gonna show you how you can get it working all right this is the linux uh shell prompt with a weather ubuntu weather centos the commands are pretty much the same and you may have to change it depending on your you're going to want to do app install or syslog first you got to make sure it's installed make sure it's installed right and then once it's installed you're going to do netstat dash ano or and that will show you if we're listening on the port syslog is port 514. it's not on there right so we got to go enable it so you're going to go to see you're going to nano sc rsyslog.com okay and then you're going to want to delete some of these comments to make it look like this it'll go it'll look like this right and you're going to want to delete those so that now we're listening to port 514 and 514 for udp and tcp okay this will let you know that it's listening on that port syslog will listen to the port but we have to make sure it the packet comes in so how do we do that okay well let me get out of here you want to press control x okay and then if it says you want to say push y for yes i push an end for none but no and let's do a tcp dump but in but let's do a ifconfig to just look at the interface we want to make sure we know what interface we have ethernet zero and then you just do tcp dump and then you just say i eat zero port 514 i don't have tcp dump installed so let's do app install tcp dump all right let me do app update so first i'm do app update and then we're going to do the app install i don't really know the packages let's try tcp dump oh there yeah there you go so let's download tcp dump so we can listen on the packets tcp dump is a way to just see the packet flow the network packets at a at a network level okay of just what's coming in okay so you can do tcp dump dash i e0 port 514 that will say we are going to listen on port 514 on ethernet zero okay what you can do is you have another linux box and you can send logs to it so what we're going to do is i'm going to go to my another linux box let me just do my ansible box so we have another box and then we're going to do logger we're going to put the ip address of that you know whatever that when you set up the vm you're going to find out what the ip address is that picked up and then you're going to do logger 10.60.0.31 you want to put dash dash server to identify the server port 514. i don't think you have to do it but i'm just going to do it anyways and then we're going to do the message we'll say this is a test and as you can see on the bottom here the message came through it came through came from ip and then it said user notice so the message came through you don't really see the contents of the message but it did it did come through to know where it came through you have to go to cd var log so let's do uh let's tail it it didn't come in hmm why didn't it come in so we know that it came through the machine now if it didn't come through the machine for you um and you did a tcp dump and it's still not coming in then you may want to check your firewall rules and on the machine on the machine and see if the firewall's blocking it but you know with that tcpw you're going to get some information of did the packet actually reach reach it right so let's do netstat dash t upl and again oh remember if you do netstat that's ano it says we're not listening to port so we have to start syslog i forgot i turned it off to you know have it not there but we do r service r syslog start okay so now that we start this lock let's go around the netstat again and now we're listening to port 514 so the packet came in but it had nowhere to go our syslog wasn't listening to it so now it's listening let's do a tcp dump again tcp dump interface ethernet zero maybe you can just do with the port 514 yeah that works too so listen it'll listen to all interfaces all right let's do it again all right the packet came in um it's a notice user notice we're going to get to that because what happens is if you don't provide a facility or and severity then it might not populate because our syslog will only log what you tell it to log and by default some stuff is not turned on entailed five that's just the last five tail let's look at the the end of the logs there you go so it came from so if you look at it you can kind of see the structure it's the time stamp then it's the host then it's the user and then it's the message okay that's the standard syslog very basic i'm not going to go too much into it but we know the log is coming in okay now now that we know the log is coming in hopefully you're at this point where the log is coming in again if you're not at this point check if your firewall is turned on and you may need to open up the port to so that the linux box can listen to that port you may just have to you know enable port you know enable uh that port for syslog so we can monitor it but that tcp doesn't be very helpful because that packet needs to come in so that rsys log can pick it up now i'm going to explain our syslog a little bit further and there's also syslog ng but i've been used to rsyslock so that's what i'm going to use it's it's pretty common and it's you know turned on by default in most oss we're gonna go and um i'm gonna show you comp real quick because i want you to know why it's logging so if you look here and we're just going down we're going down um we're going to go to the rolls here we go so i want to log user dot star remember it was user.notice when we did the logging user.notice that's by default if i say user dot there's different severity levels so it'll go by that level and anything higher than a level so star means any level right so we want auth auth priv to go here we want cron to go here damon logs kern logs lpr mail user logs here's another thing to notice star dot star is going to var log syslog so it's being stored in two different locations so it'll first hit this one because it's the star dot star it'll store it in var syslog and then user it'll also start here so it's going to store it in two different locations unless you say unless you have sort of an indicator says i want you to stop after this so let's go what was it again it was in var log user so let's look at that tail uh user.log there you go so it showed up in two different locations why is that helpful or why do you need to know that you can parse your logs and store them in different files so imagine that you have logs coming from different sources you can put them in different files you can pull from files and then send it to different remote collectors you can parse the logs you can do regular expression on the logs you can call the logs you can pull only what you want and send it to the collector or do whatever you want does that make sense a lot of power and understanding when the log comes in we want to do something with it what happens if we go here and let's comment out this and we're going to leave this one remember i did control x yes to save whenever we change our configuration our syslog will not pick it up automatically you're going to have to restart that service service our syslog restart okay now to restart it it picks it up the new configuration file and let's test it out so we're going to here's another cool tip we're going to tail dash f f means just you know keep going and we're going to do messages we're going to do logger this is a test and it comes in shouldn't have come in star dot info notice and warn these are all oh it has a catch-all as well let me clear this catch-all see so there was another catch-all right here so let me comment that out let's restart syslog tail that as well this is a test five it's not coming in anymore all right number five let's go look at the user logs and they came into user logs so you're controlling where the log is going okay this is important to know because you know you can send all your logs to collector but you don't have to collect everything you can stop it here or it's better to actually stop it and call it at the sender so if your device can kind of check mark and turn on and turn off logs of what you want to send that's better because now you're not using resources on the collector to go you know process that does this all make sense okay so let's go back to nano let's scope it even further i'm going to do something custom we're going to do a new user dot warn and we're going to send it to var log messages actually we'll put it in the syslog folder okay we got to restart our syslog let's tail dash f and we're going to say this number five it's not coming in why is it not coming in why is it not coming in think about it why is it not coming in let's go let's see if it came in though they come in there it is see it's coming in well why is that going to syslog look at the configuration file again remember i said it has to be that level and higher user dot the level is um you know there's error levels or there's different severity levels a info or notice is lower than warning so warning and higher so i actually have to tag warning as a priority let's tell syslog and we're gonna do let me do a dash that's help if you need to know you know command you might not have to go in docs you might just do help and get what you need but there's a priority there it is the priority logger server 10.6031 port 514 and then we're going to do priority user dot error user error let's try again maybe it didn't like parentheses test user error test message there it comes so it made it to the syslog collector and we did user error if i do user info it won't come in if i do warn will it come in yes if i do notice will it come in notice is loaded lower than worn it won't come in but all these messages still showed up in the user log see does that make sense so now that it's sent to syslog or stored in you can kind of control syslog now that you know that let's start logging the sentinel so if you want to log in to centennial you're going to need what they call an oms agent the oms agent is microsoft's agent that will take whatever logs by our syslog or syslog ng and take it and send it up to sentinel the configuration is also within our syslog so imagine that the packet comes in on 514 when the packet comes in 514 and then rsyslog will listen to the packet it will go through the rules oms agent adds a rule and the rule says if it's this level or this user level or this facility or the severity send it to sentinel so it's separate from what you have configured and i'll show you so first you are going to want to go into sentinel and then go to your data connectors you're going to want to go into syslog open the connector page and then you're going to want to open your workspace agent configuration remember sentinel is built on log analytics okay and then you go to syslog you're going to have to turn on what you want to listen to by turning it on here it pushes it down to the agent or if you don't want to configure ear you can customize the agent to send what you want so you may have a collector send whatever it wants and you may have another collector send whatever it wants right whatever you decide you can have it globally managed here or you can custom manage it per collector whatever you decide but you can see all the facility names we were talking about like user and then you check mark the level that you want to send so i'm sending everything to sentinel this is going to be separate from what you have already configured so you may store it in the message folder you may store it in a log but another rule is going to send that over to sentinel okay so what we're going to do is we're going to go back into the syslog collector all right so the collector is our syslog.d ls here we go so the 95 oms agent is usually the syslog collector and if you want to turn on step locks it'll add this other role imagine that that global rule the rsyslog.conf is sort of a global rule and then 95 oms agent is another rule that will trigger and then security config is just another role so you're just kind of separating roles into different files so you can kind of organize it so we're going to look at the 95 on this agent show you what it looks like and as you can see whatever you turn on is going to show up here okay if you notice i commented out everything and i'm doing a custom one i'm doing a custom one the reason i'm doing custom one is because i want to show you exactly that i'm not sending anything but anything with the word test so i'm filtering if the message contains test then send it to this port this port is a special port that the oms agent is configured to listen to so remember our syslog is listening to the 514 so the basically the our sys the syslog packet will come on the 514 it will then come into the uh our syslog we'll listen to that port 514 pick up the packet go through the rolls and when it goes to the rules this roll sends it to 25224 so it takes that and sends it to another port where then you have the ole miss agent listening to that port and then sends it all it sends it on through does that make sense ceph if we look at the ceph we have if raw message contains ceph or asa then send it to 25226 if you do netstat ano you can see syslog is listening here and then the oms agent is listening to here and then sef is listening over here see so we have everything listening for different purposes different formats okay let's do a tcp dump on port two five two two four and remember we're listening for test so nothing our dislikes are gonna process the rule but only send test and if it doesn't have tests it won't come through so let's do logger dash that server 1060.0.31 port 514 priority user error error alert doesn't come in doesn't come in but if you do test alert still doesn't come in maybe then show up there but maybe it should be sentinel um let's check we will do syslog we'll pipe it we'll do uh um let's do last 30 minutes and we're going to limit by 10 and we're going to sort by time generated did that come in oh there it came in so it didn't show up in tcp dump but oh well it did come in though so error test alert came in see says log magic error alert test does that make sense let's do one that doesn't have tests let's do one that doesn't have tests so let's do it again we're going to do logger test alert 1 test alert 2 then we're gonna take off test we're gonna do three then oh whoa whoa whoa whoa whoa i did two five five two four i gotta go back to five one four all right let's do it again alert test one alert test two we're gonna remove test we'll go three we'll go test four test five so what we should be seeing is we're gonna see one two four five we're not going to see the other one because we don't have a rule to catch that and send up the sentinel let's limit it by five all right one two four five perfect so this is how you can kind of filter syslog maybe you only want to send certain messages this is how you can call data at the collector before it goes to sentinel so you don't have to pay for it right you only want to send logs you're going to use if you're not going to use it why you're going to send it right you don't want to hoard that data you want it you want to have data that you can use and you're going to do something with it okay so does that make sense everyone the again we're going to have two different files if i go and edit the 95 olemiss agent and i go turn on all of these back on then now if it comes with any of these types that's being pushed by the global policies it's going to come in okay and right here if this rule comes in what's going to happen if i have two roles that match do you think a double entry will happen huh let's see um now that i have that configured what do we need to do we need to restart our syslog d now we're gonna let's just do port 504 okay and then we're gonna do the same thing we're gonna do test one two and then we're gonna do alert let's do user dot error just to make sure it comes through priority user.error alert3 and then we'll do test four five so we know all the logs came in it went through four and four tcp dump shows the log flowing in our list actually gonna gonna listen to the packet go through the rules and if it matches then it sends it up to sentinel okay does that all make sense now let's go check sentinel out limit five all right there we go um okay so it did come in see double messages double messages are coming in now here's the here's a curious thing ah the test came in right let's do um limit 20 because uh it was double messages so it was more than five see one came in one one two two three four four five five so double messages came in because i had that i had that catch and then three and then four four five five does that make sense because you didn't have a double message because it didn't match test so it didn't activate the other one okay okay if you look at the configuration file again it activated um probably this one user notice user notice usually notice you to notice and if it has a test then it's a double entry so be sure you're aware of this if you turn on set vlogs sefflock still comes in syslog right it may be a double message that's why you're going to look at the configuration whenever you're loading your oms agent and you're installing it make sure you are aware of that okay let's run through the installer just so you know data connectors again um we're gonna go to syslog we're gonna open the connector page now when you download this agent you're gonna keep in mind your workspace id and primary key okay all right so um here's the agent installer so you want to get this w get copy that message and then copy your workspace and primary key but here's the cool thing it already comes pre-loaded this if you copy here and click the copy button it already has your message for you then you go to your collector and then just paste that message and just run it it's gonna automatically configure the policy set it all up provision it turn it on it should work and it should work for you know ubuntu or centos right if it doesn't there's a troubleshooting tool you can run to kind of troubleshoot it okay so again see it's starting the sweet it's installing it's gonna turn it on and then it tells you you can run the troubleshooter if you need help to run the troubleshooter just do a dot opt microsoft actually sorry opt microsoft ole miss agent ben troubleshooter okay and it runs a troubleshooter you may and you require python and things like that you're going to require certain if you want to install it you might have to install python3 and and certain uh so remember just go app install and those dependencies or yum install those dependencies okay but you know once the agent is installed and you get our system working good oms agent just really is just adding extra rules here for that okay if you're going to send set logs uh let me show you real quick yeah let me do step logs so if you look at ceflogs if raw message contains ceph right or asa then send it over to that port that port will know it's in the cef format and then parse it and stick it in so here's an example okay let me think of an example um i actually have it in jupiter notebooks i'll show you an example so i'm making a jupiter notebook that you know kind of just kind of an instruction guide but for the set vlogs i'm i'm parsing i'm going to grab some events and it's coming from a log file this log file right so let's do the step sample log so we're going to do ceph sample log we have four entries and now let me see what the entries are so here's here's the entries so ceph is going to be a different format right so as you can see it's a different format it's seph microsoft ata blah blah blah it's a different format okay all right but it's coming in through a different format right and the reason it's coming in the different format is because ceph if it's coming to be a syslog everything's in that message content but if you send it via cef you can specify the columns with key value pairs and just have those columns populate it can already be pre-populated and normalized um and structured by the time it comes in which is going to be helpful because then you can kind of you don't have to parse the syslog and then query it uh for a column it already comes and and and placed in the right column let's just do the tcp dump just to see what comes in all right so those logs is going to come in logger we're going to log to that port it's the same thing i'm just variables and then the priority is local warm four tag cef right and then we just have to match the format okay i'm not i'm you know i'll go into this later but i just want to show you ceph so we send this through and it doesn't come in formatted loss for each object logger let's do this there we go so this the logs came in four entries sentinel and go to common security logs and microsoft ata logs yep yep logs very five here's a message device activity does that make sense guys so it's now coming in and it's coming in structured activity application protocol external id see it's coming in in in in the right format okay so with ceph it comes in kind of structure does that make sense guys so what do we talk about you now know about syslog you now know about what porter comes in you now know that the flow is the packet comes in 514 our syslog will listen for it it'll run through the rules it's global you know it's global configuration file or specific rule files that has been configured by whatever app and then it'll roll through those rules and log whatever needs to be logged it could log it could log it to a file it can log it um it can send it remotely to another collector it can send it to another port it can take a file and then if you store something in a file you can pull from that file and store it somewhere else you can group it into different files a lot of things you can do with syslog and but now that the logs are coming to the collector you are kind of deciding what you want to send to sentinel and again you can configure that in the data connectors and you know set up the facilities and severity or you can just configure manually in the collector all right so go play around um take the kql knowledge that you learned in the other examples because now that it's coming in it's coming in either either via syslog you do count right in the last hour right it's going to be a syslog or it might be your common security logs if it's uh if it's going to be you know ceph logs so when it comes in it'll show up here syslog or common security logs now here's another thing i want to caveat if you're sending ceph logs over and then sending it to sentinel the set logs might already have additional timestamps or might not match the right format so you do have to make sure it's in the right format otherwise it doesn't know how to parse correctly okay um it has to be in the right format right and there's different set versions there's different ceph law syslog rfc you know formats so you just have to make sure you're seeing the right format or if it's not you parse it and don't have duplicate entries you want to just cut that up because it's going to cost you more money it could mean the difference between a thousand dollar bill or a 500 bill just because you duplicated entries right with your staff logs and syslogs so don't do that and save some money all right so this is sentinel hopefully this was helpful um i know i went to syslog i know there might be some commands that you might not know so just learn you know figure out you know what commands i kind of did um you know if you want a more in-depth linux tutorial i could give that as well if you want me to dive deeper into our syslog and actually go into configuration files uh it gets pretty it gets pretty deep with our syslog and let me know what you want to want to see outside of sentinel but in this series we're going to after this we're going to start building queries we're going to build our own workbooks we're going to build some dashboards we're going to build some charts from this data that we're going to make okay and so it's going to be fun so uh please like please subscribe if you don't i don't know if you i don't know what you like and i might refocus on another topic that has more interest right so if you're interested in this and you like it please like subscribe and let me know you like this content thank you everyone have a good day and stay safe you

Channel: TeachJing

Views: 30,242

Keywords: Teaching, TeachJing, Rsyslog, Syslog-ng, syslog server, Azure Sentinel Syslog, Azure Sentinel Syslog Connector, Azure Sentinel Tutorial, Azure Sentinel Lab, azure sentinel demo, microsoft security, Azure Sentinel, azure sentinel setup, syslog server linux, syslog, solarwinds, rsyslog server

Id: KJbIH7egdVI

Length: 38min 40sec (2320 seconds)

Published: Mon Feb 22 2021