Log Analysis with Splunk | How to use Splunk to analyse a Real time Log | Splunk Use Cases | Edureka

[Music] monitor explore analyze and troubleshoot your logs with one platform which is none other than splunk hello everyone i'm manika from edureka welcome you all to the session on log analysis with splunk so before starting the video let me address the agenda of today's session so i will begin with introducing log analysis and monitoring where i will discuss what is log analysis and what is log monitoring moving ahead i will discuss the need of log analysis and monitoring then i will help you brush up with what is splunk and later i will tell you the advantages of using splunk for log analysis and monitoring we will move ahead by understanding how splunk works and the steps of log analysis and monitoring with splunk then i will discuss some of the use cases of splunk log analysis and monitoring then i will discuss some of the use cases of splunk lock analysis and monitoring and before wrapping up the session i will show you the installation along with a small demo of how to use splunk for log analysis and monitoring meanwhile subscribe to our channel and hit the bell icon to never miss an update and also if you are looking for splunk training and certification check out the link given in the description box so let's begin without further ado starting with introduction to log analysis and monitoring so answering what is log analysis it is the process of interpreting computer generated records called logs it involves a large amount of data depending on the scope of technology included in the evaluation and it gives visibility into the performance and health of iit infrastructure and application stacks it can also be referred as the process of reviewing and understanding logs to obtain valuable insights so this process allows organizations to analyze their logs in order to obtain knowledge that they wouldn't be able to obtain otherwise and then use such knowledge to their advantage not only by improving their decision making process but also in a variety of different ways it is a branch of data analysis which involves drawing insights from log files it's a staple in the iit industry where almost every product and service generates massive logs for a variety of processes logs can contain a variety of information about how a digital product or service is used so the applications of log analysis are endless examples of logs might include sign in and sign out requests on a website transactions made on a currency exchange calls made to an informational api and various other industry specific actions log analysis gives visibility into the performance and health of it infrastructure and application stacks through the review and interpretation of logs that are generated by network operating systems applications servers and other hardware and software components now let's understand what is log monitoring it can be referred as the act of reviewing collected logs as they are recorded it involves aggregating log files and providing alerts or notifications for particular log messages and events it involves the assistance of log management software to understand log monitoring it is vital to understand the process of logging in general logging is the practice of recording log messages to a file and log messages are recorded for the operating system of a machine and are typically for each application that runs on the machine for instance let's take application login with each event that occurs throughout the use of the application messages are logged to give a developer or administrator a view into how the application is being utilized log monitoring is the process by which we observe log messages often through real time processing and parsing of these files this is easily completed with the assistance of log management software log files are ingested by the log management software where they can be parsed in an effort to allow the developer or whoever may later need to analyze the log files to gain some insight into potential issues within the system or application now let's talk about log analysis and log monitoring together so both are crucial parts of log management and related in many capacities but by definition the two actually have different core meanings log monitoring is the process of collecting the information and alerting when a potential issue is involved while log analysis is the evaluation of that information to mitigate issues or improve existing processes also log monitoring typically involves the assistance of log management software log management software can be configured to listen for specific application related events and alert the proper people within a development organization when such an event occurs among other benefits log analysis on the other hand is a process typically performed by the developers or other id folks within an organization for various reasons often related to troubleshooting issues within a system or application collected logs are used to diagnose and resolve issues within an application that's a general summary of the difference between log analysis and log monitoring now let's talk about the need of log analysis and monitoring log analysis is an extremely valuable skill for tech companies that collect plenty of logs its applications are almost endless allowing analysts to monitor audit and debug their offerings while there are a variety of techniques that may be used for log analysis including normalization pattern recognition and correlation analysis there are dozens of tools on the market both free and paid ready to help so some of the benefits of log monitoring and analysis include compliance security enhancements efficiency high availability sales and marketing effectiveness so talking about compliance many government or regulatory bodies require organizations to demonstrate their compliance with the myriad of regulations that impact nearly every entity log file analysis can demonstrate that hipaa pci or gdpr or other regulations mandates are in fact being met by the organizations discussing about security enhancements as cyber crime becomes increasingly organized the need for stronger counter measures also grows event log analysis provides powerful tools for taking proactive measures and enables forensic examinations after the fact if a breach or data loss does occur a log analysis framework helps improve efficiency across the organization id resources in every department can share a single log repository an analysis of an organization's data especially the log data can help spot errors or trends in every business unit and department enabling rapid remediation talking about high availability so timely action that occurs based on information uncovered by log analysis can prevent an issue from causing downtime talking about the sales and marketing effectiveness by tracking mattresses such as traffic volume and the pages that customers visit log analysis can help sales and marketing professionals understand what programs are effective and what should be changed traffic patterns can also help with retooling an organization's website to make it easier for users to navigate to the most frequently accessed information now quickly see what is splunk so it is a software which processes and brings out insight from machine data and other forms of big data it is one stop solution as it automatically pulls data from various sources and accepts data in any format it is one of the easiest tools to install and it allows functionalities like searching analyzing reporting monitoring as well as visualizing machine data as we know splunk is a horizontal technology used for application management security and compliance as well as business and web analytics so there are basically three categories of the product available which are splunk enterprise splunk cloud and splunk line so splunk enterprise is used by companies which have large id infrastructure and idea-driven business it helps in gathering and analyzing the data from websites applications devices systems and sensors etc and it can be installed on the local machine splunk cloud is the cloud hosted platform with same features as the enterprise version it can be availed from splunk itself or through the aws cloud platform and splunk light allows search report and alert on all the log data in real time from one place it has limited features and functionalities as compared to the other two versions now let me tell you the reasons why one should use splunk for log analysis and monitoring as it analyzes the aggregate of logs from a big service cluster it generates report and alerts for the desired search it provides enhanced guis that are the graphical user interfaces and real-time visibility in dashboard in various formats it does not require other dependent services like database it monitors aws infrastructure also it uploads and indexes log data from a local pc to splunk directly other than these there are some more reasons which include that it finds real-time logs and with faster speed it provides quick results by reducing the time to troubleshoot and resolve issues it works like a monitoring reporting and analysis tool and provides insights and it requires minimum hardware resources it is easy to set up and low cost maintenance and it accepts any data type including csv json log formats etc moving ahead with how splunk works so splunk enterprise monitors and indexes the file or directory as new data appears you can also specify a mounted or shared directory including network file systems as long as splunk enterprise can read from the directory if the specified directory contains subdirectories the monitor process recursively examines them for a new file as long as the directories can be read you can include or exclude files or directories from being read by using whitelist and blacklist if you disable or delete a monitor input splunk enterprise does not stop indexing the files input references it only stops checking those files again and you can specify the path to a file or directory and the monitor processor consumes any new data written to that file or directory this is how you can monitor live application logs such as those coming from web access logs java to platform or dotnet applications and so on splunk lets you search alert report and monitor all your logs from one location in real time splunk indexes all your id data including custom application logs and multi-line logs across virtual and non-virtual environments without the need for custom parsers or connectors you can troubleshoot application outages investigate security incidents and demonstrate compliance in minutes not hours or days now let's just have a look on the steps of splunk log analysis and monitoring so it starts with installing spunk server then setting up index to store data then creating a listener to receive data then installing splunk universal forwarder then setting up forward server and monitor and then searching and viewing the reports and then collect the matrices so these steps can vary according to the requirement and we will see this in the demo part now let me discuss some of the use cases for log analysis so log analysis serves several different purposes like to comply with internal security policies and outside regulations and audits to understand and respond to data breaches and other security incidents to troubleshoot systems computers or networks to understand the behaviors of your users and to conduct forensics in the event of an investigation some organizations are required to conduct log analysis if they want to be certified as fully compliant to regulations however log analysis also helps companies save time when trying to diagnose problems resolve issues or manage their infrastructure or applications today in demo we will see the log analysis and monitoring of windows logs so let's discuss about it for a while so windows generates log data during the course of its operations the windows event log service handles nearly all of this communication it gathers log data that installed applications services and system processes publish and places the log data into event log channels programs such as microsoft event viewer subscribe to these locked channels to display events that have occurred on the system you can monitor event log channels and files that are not on the local machine and we can collect logs from remote machines also the event log monitor runs once for every event log input that we define to monitor windows event log channels in splunk cloud we can use a splunk universal or heavy forwarder to collect the data and forward it to our cloud deployment as a best practice we can use the splunk add-on for windows to simplify the process of getting data into splunk cloud now here comes the question that why to monitor event logs so windows event logs are the core metric of windows machine operations if there is a problem with your windows system the event log service has logged it the splunk platform indexing searching and reporting capabilities make your logs accessible now let me show you the installation of splunk enterprise so let me show you the installation part from scratch so what we can do is go to my browser and simply we will search splunk or splunk installation or splunk download anything i'm just giving you the smallest one i'm giving this splunk so you can get the link splunk download now and all now here you can see you are at the home page of our splunk now here what you can see is you have this form if you are doing it for the first time you have to sign up you have to register for it you have to enter the details in this form like first name last name if you're working somewhere the job title and your email address phone number company wherever you are working then the country your postal code your username password and then you will have to check this thing to agree with the terms and conditions of splunk and also you have to check this one where you have to agree to the privacy policies of splunk those you can go through once if you want and then you can just click on create your account and your account will be created then you can log in to your account from here or some year and then you can install it you can download splunk enterprise okay so i already have an account because i've already registered for it so i'll just log into my account so i'll click on login and i'll give my username and then i'll give my password and with this i'll click on login now once i'll get logged in then i'll an option to download the splunk enterprise now here as you can see multiple options like multiple versions will be present so for now you can see for windows we have this windows 10 windows server 2016 and 2019 so msi file will be available for linux we have the other ones for mac os we have other ones now as i am using windows right now so i will download the windows 64-bit so i'll click on just download now so my software will be downloaded and once that will be downloaded it will be present in my downloads folder and from there i can install it in my system so let's just wait for it it will get downloaded and one more thing like you are downloading splunk enterprise 8.2.0 for windows okay so this is the version 8.2 point and there are multiple past versions available according to your windows like it is for windows 10 if you are using windows 8 windows 7 windows 8.1 so accordingly you can download the past versions that will be present i'll just show you maybe you can just search splunk old versions maybe so older splunk releases i'll just go to the first link and here now you can find so for windows 8.1 this is the 8.1.4 and we are downloading 8.2.0 latest version we are downloading so these are the older versions you can see here for windows 10 windows 10 and you can find for windows 8 as well we'll scroll through it and we can find you so you can find for windows 8.1 for 10 as well and there will be some for windows 7 also according to your system capabilities your system configurations you can download it and install it so let's just see if our splunk is downloaded so it is almost downloaded i will just go to our this one so it is downloaded will show in folder it will be present in our downloads folder so here you can see our msi file is there now we will install it so simply will click on install and as i have mentioned in the starting of the tutorial that splunk installation is one of the easiest thing so it is like a normal software installation now here you can see this is the installer splunk enterprise installer will check this box to accept the license agreement and we can just view the license agreement also if you want and you can see the default installation options over here like this enterprise will be installed under our c drive under the program files and under splunks folder and then we will run splunk enterprise as local system account and start menu shortcut will also be created with this and if you want to customize these options you can just click on customize options and then you can customize it so i'm going with the default settings only so i'll just click on next and now we have to give some username and password you can create it over here so let me just give me some username and let me give password and we'll click on next now click install to begin the installation and here i have checked this thing it is by default check if you want you can uncheck it if you don't want any start menu shortcut for splunk enterprise i wanted so i kept it as check now i click on install so it will take some time like installing other softwares also it is the easiest tool to install like any other normal software let me just give the password and meanwhile let me tell you that splunk is the tool which has been used by 91 out of 100 fortune organizations and as we know splunk enterprise splunk cloud and splunk light is available for splunk enterprise there is 60 days free trial for splunk cloud we have 14 days free trial and for splunk light also we have some 60 days free trial so splunk enterprise will be run at our local machine and splunk cloud we can just access it on web as a cloud cloud service now it is copying the new files so once it will be completed we'll click on finish and then our splunk installation part will be done and then we can have our splunk enterprise ready to use which we will see in the demo part once we have our splunk enterprise installed we can start using the different features provided by it and there will be multiple apps like add-on apps also which we can install through splunk enterprise itself and we can make use of splunk dashboards we can make use of splunk enterprise security the search app and multiple other apps which will be available once will be logged into our splunk enterprise so let's just get it installed and then we'll log into it and then we'll see the features so as it is getting installed let me tell you some of the products of a splunk like the splunk enterprise splunk cloud splunk data stream processor splunk iit service intelligence splunk on call splunk insights for aws cloud monitoring and splunk enterprise security splunk user behavior analytics splunk phantom splunk infrastructure monitoring and splunk apm so these are some other products also available so splunk has a wide variety of products but we know the major categories of splunk products are the splunk enterprise spun cloud and splunk light and let me tell you one interesting thing that there is something called as splunk base so splunk base is a community hosted by splunk where users can go to find apps and add-ons for splunk which can improve the functionality and usefulness of splunk as well as provide a quick and easy interface for specific use cases or vendor products so as of 2019 there were more than 2000 apps available on the framework and integrations on splunk base include the splunk app for new relic the four scout extended module for splunk and splunk app for aws so with this our splunk enterprise is installed as you can see over here this splunk enterprise was successfully installed and see launch browser with splunk enterprise so i'll keep this one checked so what will happen is we will get a browser with the splunk enterprise login let me just click on finish and now i'll get the browser where i will be having splunk enterprise so this will be your url where you can access this splunk now here to login as you can see over here we have to give our username and the password and if you are signing in for the first time and you kind of forgot your username so what you can give is you can give your username as admin and the password which we have created while installing the instance so as i remember my username and the password so i will give my username and password which i had created while installing the instance and i will sign in and that's it with this we have entered to our home page of splunk 8.2.0 or the splunk enterprise now here you can see multiple things like overview of splunk software has been given to you you can just read it out and different links are present you can just go through these hyperlinks and you can read about it so let me just click on got it now here you can see multiple apps present the administrator messages settings activity hell fine and apps over here search and reporting python upgrade readiness app splunk essentials for cloud and enterprise 8.2 splunk secure gateway and these are the things you can just go through these things you can read about it and you can start using it also now let me show you the demo so let me start so before starting with searching and reporting we need the data and here we are going to analyze and monitor our windows locks so we have to give it as a data input so we'll go to this settings tab and then in this data part we will go to data inputs so here we will specify from where we want to take the data that is going to be our windows log data that we will choose i'll show you that so let it get loaded and then we will specify the details which it will ask it's taking a little time to load it yeah here it comes so here you can see the local inputs and here you can see it's written like we can set up the data inputs from files and directories network ports all these things which are available over here and if you want to set up forwarding and receiving between two splunk instances we can go for forwarding and receiving and i will show you how to use that forwarding and receiving but for the local machine because i'll be working with the local machine the data will be coming from my local machine itself and i'll be using it in a single splunk instance which was my instance so we'll click on local event log collection you can see over here remote event log collection files and directories local performance monitoring remote performance monitoring and so many and here after scrolling down you can see here forwarded inputs also so here you can find windows event logs files and directories similarly we have for the forwarded inputs this is for our two instances we are working on multiple machines or multiple instances so we will be working with the local inputs before local event clock collection let me show you with the files and directory okay then we'll go with this local event collection so starting with files and directories we'll click on it and then we can index any of the local files and we can monitor the entire file or the directory part so we'll create a new file so here you can see multiple files and multiple data sources are available so we will click on new local file and directory so that we can define the source type also and we want the windows logs data so we will be creating a new file or the directory so it is getting loaded and then we will have to specify our details now here it comes so we have selected files and directories now we can browse what files we have to browse so in c drive let me go with the program files we want windows so we'll go to windows and then here we can select any of the things what we want to monitor so let's just look at this maybe we'll go for logs now here it comes multiple logs like logs for multiple processes and all so we will go for system restore or maybe we will go for some other thing because i've already created one for system restore so let's go for setting sync this one so we will select this one okay i think it didn't get selected maybe let's just go for this system restore itself and here multiple tasks are there so let's just take this task file and select it now here it comes and we have selected continuously monitor and we will index also but index once we are not using we have to continuously monitor the log data now we'll have to move ahead with the steps so we will set the source type for that we will click on next and we will take the source type as default if you want we can just edit these things like what even breaks we want like after each line or any regular expression like for any particular pattern we want to break the even so we have kept it as automatic similarly for timestamp like after some seconds or hours or something we are just continuously going to monitor it and here we can specify some advanced filters kind of thing so we'll keep it as it is and we'll go to next now we have to save the source type so we'll simply give it as windows we had given windows restore i'll simply write windows restore now description is not mandatory so i'll just simply save it like this and here it comes so the app context and host so i'll keep it okay app context so basically we are going to perform search and reporting thing so that's why we will be using the search and reporting app so this context we have entered as search and reporting so we can keep it as constant value regular expression on path a segment in path so we will keep it as it is constant value and for host field value i have kept my host name which is by default was filled and we will keep the index as default and there are multiple indexes present like history main and summary and we can create a new index as well but i am keeping it as default okay let me just create some now i'll give it as may be demo1 i'll keep it for events and with the same settings which are autofill i'll go for save now comes the part to review now we will see so input type is file monitor source path is this and we have to continuously monitor is and the source type is windows restore app context we have given search and host is this and the indexes we have created new one that is demo one now we click on submit so file input has been created successfully now we can start searching we can extract the fields we can add more data we can download multiple applications we can build dashboards as well and now here you can see the examples and tutorials learn more about fields so these are our kind of documentation so whenever you have some doubt you don't know how to proceed further you can just go through this documentation as well now let me show you from scratch so the data has been added the source has been added now we'll go to our home page splunk enterprise and then we will work with the forwarder and the receiver part although we are going to work with our local machine but still so in this forwarding and receiving we just clicked settings and from there we have clicked on forwarding and receiving so let it get loaded and then we will specify the receiver and the forwarder so basically we are going to work on the local machine we will specify for our local host so these are forwarding defaults we will configure the forwarding part and as you will see here the field where we have to define the port and all there we will add the codes now so this one is already there and we can clone it also and we can create new forwarding host so let's just create new forwarding host and these are multiple tabs in which we can change the settings so here you can see enter host port to forward data to and data will be auto load balanced to each host and port and first we will give the host then a colon and then the port or ip because i am dealing with my local machine i'll just give it as local host or the ip you can define and then the colon and then i can give triple nine seven we'll just save it this localhost forward server already present yeah right so i'll just go back and what i'll do is we'll just simply delete that and i'll show you from scratch so whenever you are already having some forwarder or server so we can make use of that also which is to show you and just do it once again so we'll click on configure forwarding yeah here so we'll just delete it once it gets deleted then we will be able to create new forwarding host or set up new forwarding host so here you can see deleted localhost triple line 7 from the system now we'll create new one new forwarding host and then we can define localhost now i'll click on save now here you can see it got successfully saved in launcher now here you can see this host again because i have saved it again now let's just go back for receiving part now we will configure the receiving so for the receiver see already i have this receiver enabled so i'll just kind of delete it and do it again otherwise i'll get that error again so i want to delete this it will get deleted then we will be able to save new receiving 4 for the receiver there is some error awkward while attempting to remove triple line 7. let me just show you here new receiving port if we write so for example we can write here triple nine seven we receive data on tcp port nine seven so if i will give triple nine seven over here and if i will click on save that didn't get deleted i will get the error again but anyways it is helpful for us because we have the forwarder already and then we'll have the receiver present as you can see here something called as listen on this board so we have seen in the steps that we have to set up the listener so this is the thing yeah now actually it is not responding let me try it one more time it's not responding now it comes so encounter the following year while trying to save that is failed to create because configuration for port triple nine seven already exists so we'll just go back and we'll click on receive data and then we will see if that is present so it didn't get delete that's why it is already present so now you know how to set up the receiver or the listener and how to set up the forwarder now we'll move ahead we'll go to the home page now we'll make use of search and reporting app now if we'll click on data summary right now we are not getting anything or it is taking some time to load i'll show you that later now what we can do is we'll click on our search history yeah here you can see so these are the searches which i have already performed so now what we will do is we will search for our index so what index we had created index equal and we will get these suggestions as well as you can see in the suggestions we have index demo one we had created this index so we will click on it we'll just search it so there are actually zero events let me just check in the indexes we'll click on this indexes part and then we'll see here whatever the indexes are already there and the indexes which we have created will be present these are basically the splunks indexes itself now for our index that is demo demo one actually yeah okay event count is zero so it didn't get any event till now so i have one more index which already has some events that is sample index which i had created earlier so i'll just click on this and then in this also we do not have any of these so it takes some time to get loaded maybe we will wait for some time if it gets loaded or not till then we will check for main index and as you can see here the events the earliest events and the latest events happen now you can see the latest event happened was five minutes ago some audit has happened so the index name is ordered so it will be related to some audit thing now let's just go to our search here we will go to search and reporting so we'll see for our index main now we'll just click on search here you can see 313 events have happened now you can see here how many events have happened at what time now time now here you can see the details of the timestamp is here and the even what the log name is application even code is zero event type is four computer name is this my host name and there are multiple lines as well now here you can see something called as type type information date hour host source that is windows event log and application of the source type is windows event log now here the selected fields are data hour host source source type and type and if you want we can just select the field so here are some interesting fields available and we can choose from them and it will go to our selected fields so let's just go for category okay click on yes now here you can see it got added to the selected fields now earlier category was not present over here but now category is present now i'll go to this one wait a minute now here you can see all the other fields as well okay now what we will do is we'll go to the patterns because see here less than 5000 events may produce poor patterns so for larger time range we can define the larger time range from here like today or month to date yesterday previous week last 30 days etc so we will keep it as it is or maybe we will go for previous week i think still the same notification will come because it is not the larger one so let it be and then we'll go to statistics so my search is not generating any statistic or visualization result so what we will do is maybe we will define some more filters so as you can see with the help of pipes and these things we can create our search query that is known as spl so let me give as this will see the fields which were already there we'll check in the fields yeah so we have something called as type and we have source type we will define type may be information we have something called as information as information and then with the help of pipe we can give source type as here you can see this one whether it comes or not it's not coming so i will give it as windows event log let's see what we get source type is not there okay let me just go over type only because i have not provided properly yeah now here you can see and even the format of visualization we can change now we'll go to visualization okay still i'm not getting any result so we didn't get anything now what we'll do is we will specify one query so what we can do is we can just get the count of the stats so what we can do is stats count yeah here you can see in the suggestion we'll get stats count then by now here we can choose data all right we had that field in selected fields so we'll click on data now we'll search and then you can see in the events the patterns yeah here in statistics we have got the records so according to the date hour this is the count of our stats or the logs now we will click on visualization so this is how we can visualize it in different types like this is the column chart we can make use of line chart we can make use of scatter chart or we can make use of bubble chart so let's use pie chart now what we can do is we can save it as a report we can create alert also we can add it to some existing dashboard and we can create a new dashboard so we'll just create a new dashboard and then we will give some name right demo dashboard demo dashboard one so we'll keep it as private now as there are two kinds of dashboards available the classic dashboard and the other one is dashboard app which is available under this dashboard studio it is a new feature of splunk so we'll just go with classic dashboard and then we will save this visualization as a panel in the dashboard because the dashboard is made up of panels so let's just give some demo one so we have used the pie chart and we'll click on save to dashboard the panel has been created and added to demo dashboard one now we can view the dashboard now here it comes so this panel has been added to this dashboard we can edit it also we can add panel we can add input and we can make the theme dark so if you want to add some panel let me just clone from some dashboard so we have these dashboards available so i have one more dashboard with sample one and in this we have sample panel one so the data will come into this yeah so we have this one column chart so we will add this panel to this dashboard now we have two panels in this dashboard which have some data so this we have created for our windows logs so it actually contains the stats by date hour so it depends on what visualization you want to use in your dashboard to monitor or to analyze and monitoring will happen as we have selected continuously monitoring so it will be monitored continuously any different activity encountered or any change of patterns will be observed then some alerts can also be triggered let me show you for add input so you can kind of go for maybe time so you can go for previous way maybe okay now it has been connected to time of previous week so this is actually about dashboards you can play around with the dashboards how you want to visualize the data and the logs now here are the alerts let me just save it we can save it as dashboard so let me just save it because we already have this dashboard and we have just kind of edited it we have added one panel to it and we can change it to dark theme now to get that dark theme we need to refresh it also so when i will click on save now here you can see you have changed the dashboard theme so to see this change you must refresh the page so we'll just click on refresh then that dark theme will be applied and visible and afterwards we can get that dashboard as we have created yeah here in the dark theme and we can export it to pdf or print so we can have pdf or any format as hard copy and you can use it for your reports and you can analyze the logs or the data we are learning about log analysis and monitoring that monitoring thing is happening continuously by the splunk itself and we'll get the alerts whenever any activity will be observed and we can set some custom alerts as well and this is for analysis we can create the dashboards we can create the reports also so as we have created the dashboard similarly the search result can be saved as report also so maybe we have created some sample report or maybe we'll create some new one so we'll go to search again we'll make use of search app and then again we'll get some data so we'll go to search history and simply will give our previous search here so this is the one we'll add to search and we'll search it or maybe this one is for 24 hours earlier we had made use of previous week data now we will make use of last 24 hours so now the stats are less right records our rows are less now we can save it as report we can save it as alert also to existing dashboard and new dashboard we have seen for the dashboard just create a report we will give some title to it maybe demo report ring time range picker is there and we'll click on save we can change the additional settings like permissions schedule acceleration embed and all we'll just go with the basic ones and we'll click on view here it is this is the report available now we can add this report also to the dashboard or we can edit this in the description we can edit all these things here you can see the creator is created by search app we had made use of search and reporting so we have created it through search and other things like schedule expansions acceleration permissions etc are not changed okay they are the default ones and if you want we can add this report to our dashboard so for adding it to the dashboard we should make it as a dashboard panel so dashboard panel can contain any module or any report or anything but the dashboard contains only the panel anything which is saved to the dashboard will be in the panel format so maybe we'll give demo too and simply we will click on save now we can view the dashboard so we have created one more dashboard in which we have saved that report as a panel and we can make use of this thing for log analysis whatever the data we have got we can get it we can export it to pdf or print something we'll just go to dashboards and then we will get our dashboard which we have created earlier the demo dashboard get this one then what we will do is we will add that report which we have generated right now to this dashboard here it comes so you click on edit and i will click on add panel and then say new new from report we can clone it from dashboard so earlier we had cloned it from dashboard now we'll go for the report so we have these reports available so we'll just go for demo report now we will add it to the dashboard now here data will come and also we can give the name also like the title of the panels so this we can give for report of last 24 hours here we have so this is how our dashboard looks and we can print it also or we can make it as a pdf and we can perform the log analysis so i'll just click on save now just go to our home page now here i'll show you some more things like here you can see this red marker so this is health is red so for the health of my splunk instance is red that means here you can see so the green thing is that it is proper okay it is functioning properly health is good yellow means it is experiencing some problem and red is that it has some severe issues so what severe issues it might have that you will get here so i am getting two serious issues from here that are ingestion latency and tail reader so when i'll click on it so what root causes events from tracker.log have not been seen for the last 5250 seconds which is more than the red threshold that is 210 seconds so this typically occurs when indexing or forwarding or failing behind or our blog and here is the tail reader so the monitor input cannot produce data because splunk these processing cues are full this will be caused by inadequate indexing or forwarding rate or a sudden burst of incoming data so as i was using splunk already for monitoring thing so i was getting the data continuously i was getting the logs so that is why i have not changed the threshold as you can see in the ingestion latency whenever the data is ingested so the default one is 210 seconds it is more than the time is 5250 seconds from these many seconds the events from tracker.log have not been seen which is 210 seconds more than our red threshold so we can change these thresholds also so i have not changed it i kept it at default so that is why i am getting this ingestion latency so i am not getting those logs for my monitoring thing and then the tail reader we got so it might because of sudden burst of incoming data maybe there might be some operations the services the performance has been increased some processes have increased on windows as i might have chosen some logs which are related to some processing or some system restore or something wherever the data increased because of that process increase that's why the monitor input cannot produce data maybe the processing queues are full so these things we can manage we can change the thresholds and we can manage the data also maybe i would have created some indexes to monitor in which i have kept for like smaller time ranges like for each second or each minute so that might cause these problems so according to the requirement we can change this we can manage it and we can make use of splunk properly so whenever i'll get your green thing that means my feature health is all right now here you can see the messages so we have got these messages now skipping indexing of internal audit events because the downstream queue is not accepting data and here the tcp output processor has paused the data flow forwarding to host destination so all these things happen and as you can see here we are getting the messages so all these messages are kind of alerts which are basically for splunk itself we can do it for other logs as well for windows for an application or for any website or something here what you are seeing is it is for splunk and it has been already configured through splunk itself so similarly we can create some alerts for our application our website whenever we see any kind of activity for that we can customize our alerts and we can analyze it with the help of dashboards and reports etc and now let's see some of the dashboards which have already been created by splunk for analysis and monitoring thing so we will just go to any of the applications and then we'll go to dashboards or the reports so i'll click on the dashboards yeah here maybe integrity check of installed file so this is the dashboard which is already present over here so let's see if any data is there yeah here it is so status of file integrity check is here so this is being used by splunk for analysis and monitoring itself we'll go to more dashboards into end web socket test job details dashboard request racing orphan schedule searches reports alerts etc are there right and as you can see the owner of these are like nobody and the owner wherever my username is there that means i have created these dashboards and the sharing thing is there what type of dashboard is it classic one or the new one the dashboard app or the studio one and reports also there will be some pre-existing reports that will be basically for the splunk itself so errors in the last 24 hours errors in the last hour license usage data cube as we know splunk is the licensed product messages by minute last three hours so let's just check you click on custom maybe you can give it one so there are no such messages zero events here we can check the alerts so as you can see over here alert set a condition that triggers an action such as sending an email that contains the results of the triggering search to a list of people so here we do not have any of the alerts but we can make use of alerts so this is a kind of action after the log analysis or monitoring we will be monitoring the logs we will be analyzing it then we can set up the alerts accordingly that will be a kind of solution to that so we'll come back to our home page so let me just summarize what we did so what we did is actually we kind of provided the data input through our windows logs then we have indexed that data those logs and then we have configured the forwarder and the receiver or the listener and then we perform the search with the help of search and reporting application and then we created the reports the dashboards we made use of search query and then we created those reports and dashboards to analyze and we chose that continuously monitoring while indexing so that the logs can be monitored and with the help of dashboards and reports we can analyze those blocks according to the data we want based on the fields whatever fields we have selected so according to that we can monitor the logs i hope this was clear to you all you can just explore more and you can take most out of splunk for your applications websites or even windows data linux data or whatever is required and before ending this demo let me show you for that local data also so we'll just go to data inputs so we have seen for the files and directories for windows we have checked the logs for windows we have checked from the files and directories which are present in my system so here we have seen for files and directories now let's just look at this local event log collection and remote even draw collection is for for remote hosts it will collect the event logs from this machine so now here you can see available logs will be from application security setup system forwarded events direct show filter graph and multiple logs are available here now so selected logs are application so let's just take security one also we'll just take one the application if we can click on this so that it will go back to the available log so we will select our application log again and then similarly as we have done earlier if we can choose any of the indexes so we had demo one we have created this index so let's just take the default one and we'll just save it successfully updated localhost now here you can see this one localhost and logs our application localhost default enabled so right now we have created this one now again we can create new event log collection here we have come so we are checking for local event logs so we have come to this page add data as we have performed this thing earlier for files and directories similarly now because we do not want to upload any of the files and directories we do not want any logs data from any of the files and directories we have already selected the local event log so we just have to give the event your collection name whatever we want to name it we can just name it and then we can find the logs and then we'll just click on next next like that the same process then we can create the reports we can create the dashboards and we can analyze our data we'll go to home page we'll go to our search and reporting app and then i'll just show you the data summary we'll go to the search and reporting app as we have indexed our local event logs now here if we'll click on data summary earlier we were not getting any of the data but now some data will come let's just wait for some time it takes some time to reflect over here yeah here it comes see there are five hosts four sources and source types now here you can see my host name right then these are the sources like windows event log windows even log from application system application the source types you can see count is also there these many logs have been recorded update and their time is also given when the last update happened now i hope you are able to understand this and now you can perform log monitoring and analysis with the help of splunk for whatever logs you want to so i will in this demo here with the home page of splunk enterprise with this we have come to the end of the session hope it was useful for you and you have enjoyed it do not forget to like the video and subscribe the channel thank you i hope you have enjoyed listening to this video please be kind enough to like it and you can comment any of your doubts and queries and we will reply them at the earliest do look out for more videos in our playlist and subscribe to edureka channel to learn more happy learning you

Channel: edureka!

Views: 8,658

Rating: 4.8305087 out of 5

Keywords: yt:cc=on, log analysis with splunk, splunk log monitoring, windows event logs analysis using splunk, splunk tutorial for beginners, splunk tutorial, splunk architecture, splunk training, splunk dashboard tutorial, splunk edureka, edureka, what is splunk, splunk log analysis, splunk log collection, splunk administration, real time log analytics using splunk, basic searching log file, log analytics in splunk, splunk log file, edureka splunk tutorial, edureka splunk

Id: OBHtu285aqE

Length: 57min 6sec (3426 seconds)

Published: Mon Jul 05 2021