Windows Event and Logging Demystified: IT Admin Edition

foreign [Music] [Applause] [Music] ER pool and this channel is dedicated to it students I.T professionals and anyone who enjoys learning technical subjects [Music] thank you now I don't know about you but I never had any formal training on the Windows Event Viewer yet alone understanding the windows logging system or even understanding what Trace logs were and event logs were most of us in the late 90s just simply had to jump in and try to figure it out that's not a good way of learning an operating system but unfortunately there was very little training at that point in time so my goal is to introduce you to the windows tracing architecture which creates Trace logs and event logs really understand what's going on with the windows logging system and how to use it more effectively how to find better lookup websites for looking up events and sources help you solve your problems faster I'm going to demo a lot in this presentation and I'm also going to show you some very effective third-party tools that are portable apps that really can help you with event logs now the windows logging architecture is built on top of what is known as etw event tracing for Windows if you look at the picture on the screen you see event providers those are typically modules of software primarily a dll in a process one of its functions is to look at how code is executed and determine whether a success or failure happens or something that should be alerted it then captures bits of information and then puts that into a binary file on your hard drive that's a log file now etw which is the foundation for all logging produces two types of log files one is Trace logs and the other is event logs Trace logs are used primarily for developers so as an ID Pro I don't look at Trace logs very much but the other log is called the event logs and those I do want to look at now you may not be aware of it but Windows logs a lot of stuff and you have a ton of log files you probably didn't know you had now one group of those log files end in the extension ETL and those are Trace files you can look at those but they're primarily for developers and in many cases this creation of log files tracing is not turned on until you put debugging on now the other one is event logging and those produce files that are have an extension of dot evtx so anytime you have a file with evtx those are event logs now presently I'm using my video editor and I'm using a tool called everything it's a file search program and it's 10 times better than Windows search but I'm going to use it to find ETL files so I'm just simply going to say look on my hard drive star.etl and you can see just like instant it has looked on my hard drive and has found every location there is an ETL file that's a trace file if you look down here at the bottom it says I have over 1411 ETL files guys that's a lot of Trace files you didn't know you had now I'm going to ask everything to look for evtx files event log files and again you can see it's just instant if I did this with Windows search we'd be here a long time and I have to do a lot of video magic but you can see just instantly all those evtx files notice something interesting most of them are in one directory that was not true we look for Trace logs they were everywhere but in event logs they're primarily in one location Windows system 32 when EVT and logs now I'm at nurse office website and I'm going to go to the pre-release tools where he has a nice portable app that allows us to look at providers in the windows environment that generate Trace files so let's take a look I'm going to go to his main page come to pre-release tools and slide down to where I see event log provider View and you can download and extract it and for me I just simply put it in a never share or a flash drive and you can run it anywhere on the network so here I have event log providers view running right now on my video server and you can see under provider name you can see the modules of software that are generating log files and then over message file one column you can actually see the dll that's responsible for generating those logs interestingly enough the the dll is called IO log msg.dll go figure now just because as I scroll through here you can see there's a lot of providers and a lot of potential log files that could be generated now just because they're in place they're in the registry they're on your system doesn't mean they're active and in use I'm going to quickly wrap up our talk about Trace logs because we really want to move to event logs but there's two categories of Trace logs analytic and those are typically used for performance evaluation so if you want to look at what is impacting your boot up or your shutdown or a particular application's performance you're going to use analytic Trace logs there's another category of Trace laws called debug those are primarily for developers and the debug channel is disabled by default so Mr vample if I want to look at a trace file how would I view that well you can use Windows performance analyzer which is a simple download and install you can use the SVS Trace view the wpp software Tracy which is used for kernel mode drivers event tracing for Windows etw Windows Event Viewer which you use for event logs you can actually open up a trace log with that tool and of course Powershell I'm going to take Windows Event Viewer and open up a trace file I'm looking at my everything search engine and I can see I've got a trace file called Boot perfdiagnostic logger.etl and it's in the windows wdi log files it's pretty good size so let's go take a look at that one so I'm going to go to Event Viewer and I'm going to right Mouse click and open a saved file because it is a save file and I'm going to browse for it go to my C drive Windows system 32 wdi and open up log files and there's the log file you can say it's quite large so I'm going to go ahead and open it and it's going to ask me to convert it so we're going to go ahead and convert that file it's quite large and here it is converting that Trace file into an event log or something that event love viewer can look at and then it's going to want me to save it and I'm going to save it under Event Viewer save logs which is right over here you see a section in the navigation let's just save logs and I'm just going to leave it with its original name and there it is I've opened up that Trace file and I can actually open up not that that means a lot to me but that's how you would open up a trace file with Event Viewer now back to event logs event logs are binary files in Linux and Unix they're typically text files but in Windows they are binary files they're primarily located in the windows system 32 when EVT backslash logs directory and event logs are organized into channels so we're going to turn again to nerfsoft he's got a portable app that allows us to look at event log channels now back to nerfsoft's website slide down to you get to system tools in the menu on the left hand side once you get to that you'll just slide the page down to where you get event log channel view which is a simple portable app utility and above it is another event log view it's called Full event log View take this one full event log view extract it and put it in the same folder as the one below and when you do you get to use two tools together so it's a very handy way of having both of these tools put them in the same directory and they'll launch and work together now here's what I've done I've got a network share and I've put the event log channel view utility in the same directory that I put the full event log view utility put them in the same directory and they'll work together now I'm on a domain controller and I'm looking at my event log channels and if you scroll down you can see there's a lot of channels and if you're kind of puzzled as to where these channels are in relationship to the Windows Event Viewer let's take a look I've launched my Event Viewer on my domain controller but as you start opening up you'll see application security setup all of these are what we call General logs and if you open up the applications and services log you begin to see a lot of additional logs these are channels if I open up Microsoft and I just come down to Windows it just goes on and on and on so you have many event log channels now nerfsoft's Event Channel log viewer allows you to see all the active and you can see it by the Green Dot beside the channel That means it's active you can actually disable it come up here hit the red button and you actually now have disabled that particular channel from logging so here you can see it's no longer green it's now red that's amazing you can now actually turn on or turn off any type of logging channel that you so desire that's not necessarily helpful in every situation but it's pretty cool I can also come back I'll go ahead and turn that back on if I go up to options I have an option of hide Channels with zero events so I'm going to uncheck that and you'll see we have a lot of inactive event log channels it's refreshing and now you can see all the channels that are on this domain controller the ones that have the Red Dot are off they're disabled and if you look at them it makes perfect sense some of these have to do with media Foundation device proxy that may have to do with a role or a service that I could add to this server it's not on so those vent log channels are disabled now each of these channels have their own event log if you go and look at file name column you can see the name of each of these event logs now I'm going to scroll over and we'll take a look at what additional information we can see first of all we can see the maximum size of that log has the file reached its maximum size no or yes we can see is it a debug operational admin is it analytic now under the column Channel isolation it lists application and system and custom those have to do with the security settings so why they call it Channel isolation I have no idea that is all about security now this is a good time to pop back in and talk briefly about properties of logs there are many ways that we can save log files we can do what's known as circular in other words once a log file fills up we can begin overriding the oldest log entries that's normally what we call overwrite then we can do archive where the log file fills up reaches its maximum size and then it's saved and then we create a new one and we continue on capturing logs then we also have do not overwrite and that allows us to once the log file fills to its fullest size it then stops it no longer captures events now in the event log channel view you can see it talks about has a column called retention mode if I right Mouse click on any channel I can see that I can look at that retention mode is it set to overwrite is it set to Archive is it said do not overwrite and I actually can change it in the right click menu options I can choose a new channel and I can also set the maximum file size so here I can set how large I want the file that's capturing events to be in this column I can see what channels actually have events captured which channels have nothing another nice feature is if I want to capture all the events on a channel I can also save them right Mouse click save selected items and then I can save them as a text file I can save them as a common delimited at HTML or even a Json file this says very very nice especially if you need to generate a report or include it in a compliance report you can just easily export this data out and attach it to the port that's going to somebody now I also have the ability to go to any channel I'm going to go to the system channel right Mouse click and notice I can launch the full event log view utility the other tool that we put in that same directory so I can select the channel and say Now launch this second utility so we'll go ahead and do that now I'm looking at strictly the system Channel and I'm only looking at seven days so by default the full event log view pulls up seven days worth of events if I want to change that I simply go to options and hit F9 or click the advanced options and here's the beauty of this tool I can choose the type of events I want to look at for seven days or in this case two days or two seconds two minutes two hours or from this date and this time from this date to this time or I can say show me all events or I've got a common delaminated text file that says only capture these events or these event IDs very powerful show me all providers or show me only the providers that I'm going to show you in this comment eliminated text file that I created so it's extremely powerful Search tool that allows you to go through a lot of logs and just filter out what you're looking for it has a lower pane down here so if I click on this error you can see information shows up in the lower pane this is really easy because it's text so I can take it copy and paste it put it into Google and maybe research that event now under the full event log view under Options under the lower pane if I want to look at the event itself I can display that event in XML or as as you see in here I've got it displayed in XML or if I want that error event displayed in data and description so I'll get both data and description so a lot of flexibility in how you can use this tool again if I'm trying to save this particular event I can save the selected item and I can save it in many formats text file comma deliminated text file HTML Json and even raw XML so very flexible if you want to save this in another data format now I will spend some time looking at Event Viewer and what you can do with that and I will definitely show you some of the limitations and some of the real pluses by using Event Viewer rather than than these third-party tools again a nice diagram showing us the big picture application since its event the event logging service plays a role in writing that event to a file on the hard drive if you'll notice the services we have the Windows Event log that's the service that's part of this process you'll notice you have another event log service called Windows Event collector that one is right now set to manual but if you decide to make this workstation a centralized collector of all events from all your critical servers you can put those events to one server or one workstation when that happens that workstation becomes a collector and then that event collector service will be on so who generates events applications Windows operating systems Services Windows local security Authority that's who does the security events and also Key Management Service there are channels that are installed by Windows when it installs one is the application security setup system and forward events these cannot be deleted and they're independent of any existing publisher when we look at the general logs under Windows log under Event Viewer we'll see the application log this is for all user level applications this channel is not secure and is open to any application events that need to go into this log this is very different from the next one which is a system this channel is used by applications running under the system service accounts installed system Services drivers or a component or application that has events related to the health of the computer system now the security log is controlled entirely by the windows local security Authority user events May appear as audits if supported by the underlying application now this is one aspect of Event Viewer that I'm not going to get into because this part of auditing and security is a whole another topic I'll get into that later now the setup log contains messages generated by installing or upgrading the Windows operating system now if you have a member server and you promote it to say a DC a lot of those events go into that setup now forwarding events are when your workstation or your server becomes a collector and you're pushing all the events from around your network to one device they will show up under forwarded events now events fall into different types we have critical they're typically things that you need immediate attention by the administrator there's also event types that are error they indicate a problem but the category does not require immediate attention there's also events that are warnings events that provide forewarning of potential problems and then information events describe successful operation of an application and I'll show you the value of an information event as we continue then we have success audit failure audit these are specific to auditing events that happen in the security law if you look at my Event Viewer for my video editor you can look down here and see many channels that are involved on my video editor it really depends on what you have installed let's take a look at my domain controller you can see that some of my channels look very different on my domain controller than it does on my video editor and that makes perfect sense now when it comes to event log viewers monitoring tools and analyzers there is just this huge available Suite of tools out there from third-party vendors obviously you can use Event Viewer you can use Powershell Microsoft Azure has some of this monitoring analyzing capability there are a lot of third-party tools that are simply portable like I've showed you with Nerfs off there's also analyzers monitors and viewers of event logs that have an agent based and then they push everything into a cloud dashboard and then there's of course fully installable viewers monitors and analyzers of event logs now let's turn our attention back to Event Viewer the utility itself it has many drawbacks very frustrating drawbacks and yet it has some really handy features now obviously if you've got a large server room or a data center you're not going to use this tool you're going to be using one of those tools we just talked about but for smaller environments and especially workstation troubleshooting this is really handy now at my workstation I can come right up to Event Viewer if I don't want to troubleshoot my local workstation I can rest right Mouse click and connect to another computer so I'm going to pull that over and go in and I'm going to go into my domain controller and in just seconds I've went right into my domain controller and I'm using my Event Viewer to actually analyze its events now you do have to set up a group policy and you do have to bring down some elements of your firewall that is in the notes so just download the video notes and you can see exactly what you have to do to get that done now I can just go from any server in my environment or any workstation in my environment once I have done that so as we look at the navigation of this utility we can see the very top the summary of administrative events and it gives us up to seven days worth of events which is really handy it's a quick little dashboard that gives you a quick view of say critical events in the last seven days so I have had a critical event in the last 24 hours and so I can dig in and take a look at that down at the very bottom in the center section of Event Viewer is log summary and we can look at each of the logs that represent the channels and we can see that the retention policy and we can look at their size so we can take a quick look at our log files if we want to look at them here so here's an aggravation of Event Viewer it calls the center section called recently viewed nodes why are we using the term nodes these are channels so if I come and I open up Key Management Service and I go back there's nothing there it will show that I went into Key Management Service as my recently viewed why not call it channels why call it nodes you know don't you love Microsoft so again I can go to the navigation pane I can click on system and there's all my events I can double click on any one of these and it pops information about it now this is one of the aggravations one you can't copy and paste out of this into Google you can't copy and paste it out of anything to anything so this is an aggravation you have to either sit down and retype it into Google exactly if you're trying to search for that type of general information about that event very poor why we can't copy and paste the other ridiculous issue is the event log online help that sounds exciting which is not help at all so having a very poor interface here and absolutely useless online help is some of the real drawbacks to Event Viewer now if you choose an event let's say I wanted to choose and save this event you do have over here on the right hand side I can save selected events and again I can give it a name and I can save it as an XML file a text or tabbed eliminated or a comma separated file remember I told you I would show you something very interesting that you could do with something like an information event let's say you have an information event that when this happens you want to run a script well you can click on that information or any of it you can do this but just to show you that information events can be very helpful so on this one it tells me that something happened and when this happens I want to run a script so I can come over here and say attach a task to this event so anytime this event happens I I want to go to task scheduler and I want you to automatically run a Powershell script a batch file whatever that is a powerful feature also on the left hand side you can clear a log so you can come up and clear the log you can also create custom views so let me create a custom view in the last 24 hours I want to create a custom view of any critical error having to do with my backup agent so I'm going to open this up and I run veeams backup agent so anything that is critical from the veeam backup agent I want you to show me and so I'm going to say okay and I'm going to call this my it's going to say under custom views and here you can see now I have my veeam backup events and it will indicate anytime I have a critical event with veeam and I can look at that and say the system has rebooted without cleanly shutting down first so those are things I want to know about my system and I can do that very easily and create a custom view in Event Viewer if I'm in the system event and I want to find an event ID I can just quickly type in that and it quickly searches my Event Viewer now if you remember in the beginning of this video we talked about traces where we have analytic and debug traces on our system they're not normally shown in Event Viewer but you can come over to view on the left hand side and say I do want to see those analytic and debug logs and you click that and more things show up over here that you can view and here I've opened up some of these Microsoft NS drivers and you can see I have some additional logging this is really Tracy that I can now view an Event Viewer now there is one custom view that is always created under the custom view you will always find the administrative events and that primarily is for a quick way of seeing critical warning and error messages quickly populate for a quick look at that workstation so that is a custom view that's almost always on each event you are pre-set up now let's talk about troubleshooting the whole purpose of Event Viewer is to help you with troubleshooting so let's look at some practical elements of troubleshooting it was actually this problem I was working on a DNS server service event error my gut told me that it was probably a misconfiguration having just built this home domain I probably needed to do some tune-ups and cleanups and configurations that I had not done and I saw this and I began to dive in to troubleshoot this problem one of the things you want to do when you're dealing with Event Viewer and troubleshooting is make sure when you're doing your Google searches that you include the event ID number and the source name in this case it would be event ID 4015 and Source DNS Dash server Dash service make sure you put all of that in there in your Google search it's going to give you a lot more effective results when I'm dealing with an event that is related to a service one of the first things that I want to do is go look at that service and look at its dependencies what other additional services are critical to the one that I'm troubleshooting in this case DNS server so I want to make sure that I look up event IDs related to active directory domain Services accelerate function driver for windsock RPC tcpip protocol driver I want to look up some of those so that if those show up in my hunting down this problem it could be these dependent services that are possibly causing the problem in the service that I am working on I'm also going to go see if I can find as many related event IDs that are part of DNS server service so I'm going to try to find those and create maybe a text file that I can use to pop in as I'm looking for related event IDs specific for DNS server services and see if any of those pop up in my search maybe three weeks ago two days ago 24 hours ago in addition to the one that I'm looking at so here I put in my text file various event IDs separated by commas and I can throw that into nerfsoft's tool and say while you're looking for the event ID that I'm struggling with check these also and I can expand my research from 24 hours to maybe two days to three days to see if any of these pop up related to my DNS problem a few days before a few days after I am attempting to filter out all the unnecessary events and try to focus on events that are related to my specific problem there's over 24 000 events in my system event log I could care less about them I'm looking for any related appropriate event for my DNS server service so here's an example of Google search I put my event ID then I put my my source name DNS Dash server Dash service when you're doing your searches be accurate put in the proper name if it's got dashes put in dashes be deliberate in putting proper information in your Google search you're more likely to get accurate information in your results as a matter of fact this result solved my problem now the best website I found for searching events if you just want to go to that site and do your own research this is the site you want to go to and they include events for Windows Cisco antivirus Veritas open manage VMware and others so it's not just Windows when you're troubleshooting event logs analyze things that happen but before and after to determine is there any relationship look for that chain of events that are related and showing some kind of Define sequence learn to ignore non-related events filter out the noise is a big step in finding the right events to analyze and thoughtfully Implement Corrections when you have solved the problem those events should go away avoid the problem of solving one thing and breaking another how many have done that one and if you find yourself frustrated and feeling stupid sorting through 24 000 events and not seeing the problem relax it's just part of the job there's not a tech that hasn't been there so it's experience it's it is hard work it is getting in there and working to try to solve the problem but don't beat yourself up about it of trying to find problems with events can be a real pain when things are not broken one of the most important reasons to get into your event log is to find misconfigurations if you're listening to security blogs or security podcast you're you're realizing that one of the number one sources of security breaches is misconfigurations and Event Viewer is very good about helping you find misconfigurations if you're watching this at this point in the video you are a hardcore technology person ninety percent of the people who are on YouTube who watch a video that I create are gone in three minutes so the fact that you're watching me right now tells me you're pretty hardcore and you're the very reason we do all the work all the video editing all the preparation is because of you you're the person we're after you want to learn you want to understand and you're willing to watch 25 minutes 30 minutes of just geek stuff and we really really appreciate you one way that you can help us tremendously is support us by liking a video and subscribing it's simple two clicks and it doesn't cost you anything and it really really helps us if you can join that's great it really does help us it's two dollars and something and a month that's a cup of coffee a month we really really appreciate it but it's more important if you can like And subscribe and it's the best way of already in this channel [Music] thank you [Music]

Channel: TechsavvyProductions

Views: 14,089

Keywords:

Id: AWjFAMOJS58

Length: 36min 37sec (2197 seconds)

Published: Tue Jan 24 2023