Splunk Configuration Files : Timestamp extraction using props.conf

okay today we will discuss about time stamp extraction using props dot confine okay so time stub extraction plays a crucial role in Splunk events because for each and every event whatever the time we are extra time we are extracting it's basically becoming the underscore time value of that event right and that basically says what time that event occurs so that understood time using the underscore time basically we can do lot of analysis like based on the time like when that event occurs and for a particular time how how many event occur something like that so underscore time is very much important in Splunk so in this video we will see how for each and every event we can extract times so it's basically a good practice whenever in any application generating event right there should be a timestamp associated with that event as well so that that that time can be extracted by by these configurations so mainly we'll be discussing some of these configurations today so as you may have seen before in my previous video we talked about index sorry you talked about line breaking of the event right mmm so today we'll see from that those events how we can extract time so while doing that I think we discussed some of the configurations like this time prefix max time look-ahead time format right so we will try to see the similar kind of example today so I have taken this data I think I have used this in my previous video as well this this logs okay and if you see each and every log file have this time format we have the GMT and after that the whole time format we'll see how to do that okay so to do that first we will talk about data one file do we just know I have shown to you okay so let us let us first line break this event using using our this one break only before did then we will see the other different time extraction methods so we'll go to settings add data so I'm using Splunk 7.2 dot 1 now so it may be this interface looks a little bit different but functionality-wise it is same so upload is now coming below side so I'm clicking on upload so we will first talk about data 1 ok so quickly I will be doing that setting so break only before did as I mentioned is true right so whenever you are mentioning this one you need the time prefix this is a time stamp extraction configuration that has to be true so triumphal fix is only required when your time stamp has some kind of prefix so for us we have GMT right so it will be GMT now time format you need to give the time format as that Splunk will it be able to determine what kind of date format it is so that it can correspondingly parse the data and then you need to go to max timestamp look-ahead so I will be adding this settings as well so max timestamp look at is basically telling us after the time prefix okay so what is the maximum number of character positions Splunk needs to find the time format so basically if I remove that GMT part and if you see this coming nestled in just 24 so that's why I have given 24 here so if I apply that so Splunk is automatically creating those breaking those whole chunk of raw data into different different events right now if you see here Splunk is automatically complaining about this particular event what it is saying this part the time format specified here it's outside the acceptable time window okay now what is acceptable time window here you know if I closely see these events that times so our current time is December 10 right 10:59 p.m. so basically if you see this particular event it's for today only right now this particular event it's after two days if you see it's a 1212 and this particularly friend is for 13 so basically we are talking about future events here right now Splunk is accepting this future events but it is not accepting this future event because there is a settings called max days hence okay now what it does mean so it means in future how many days you can have the future events in future so by default it is two days that's why it is accepting this twelfth event but it is not accept in the thirteen right now whenever it is not able to accept any event basically the parts the time stem okay if you see here for tenth the timestamp it is already extracted there stain for twelfth it is times 10 12 extra days s 12 but if you see the 13th event that timestamp extract is a stain on D okay so our purpose history if you if we want to allow this event that means if you want to extract the time from this event accordingly so that should be coming as 13 right so you have to you have to work with this max this hints if I give 3 here okay apply settings now if you see the big the event time is becoming 13 okay so I'll just remove this and apply settings okay it does not point back okay let me go back select the data okay and France new settings so break only before date is true then time prefixes GMT and then time format is our trying format and then max timestamp look-ahead is 24 apply settings now if you see I just wanted to delete that because I need to discuss another stuff here now if you see why it has chosen tin instead of 12 because the logic is of anywhere it is not able to parse the timestamp properly it takes the time stem from the previously successfully extract the time stem from the previous event only see there if you see the previous events is successfully extracted is the 1012 right that's what is giving the same same date format now now if if Splunk do not find any of the previously events successfully extracted time step with successfully extracted time stem it will take the either file modification time if you are talking about file upcoming file imp as the input or it will take this time set by for order input at the input layer okay so we'll see that one as well in another example okay so that is how max days hints work now let's talk about another one another example okay a negative case where none of our events are matching with this following with this with this range okay Lynden let's see how it's working over there so to do that I will work with theta1 underscore negative case here okay i'll go to next similarly for advanced I will just do quickly we float a true time prefix GMT time format or time format next time stamp look ahead is 24 let's apply this okay now if you see this particular event all has day test 13th that means which is basically falling outside of that two days by default window right and if you see all the date is here it has chosen as thin now from where it got thing is the file the file I am using date 1 underscore negative case if you see if I show you here okay this is the file modification timestamp so it has taken that one ok so this is how it works now let's move ahead so there is a another settings this opposite of max day hence is the max days ago so max days hence talk about in the future max days ago talked about in the past so let's let's see that one so for that we have our data to file so I'll just do that one I will quickly do that one for date true our time prefix as GMT our time format as this one and then max timestamp look ahead okay 24 now if I apply this if you see here even though all the events are from the past right if you see all from 8 7 and 10th right so if you see these two at least are from the past but Splunk is not complained because max days ago the by default values I think 2000 days okay so to see whether how max days ago works let's add this particular settings over here and let's give it a very small number let's say - if I apply now if you see for this two events Blanc is complaining because and the max days ago by default value is two then those are falling outside of it okay both both are falling outside of me so that's why this this for to these two events it is basically and now if you see the timestamp as well for this to a base there is no previous event which has successfully extracted timestamp so that's why it is taking the file modification time okay so this is how max Daisy goes is go work okay now there is another settings called add extra time fields add extra time fields so let's talk about that before we talk about other time settings so to do that let us index this one okay so I'll just click on next I'll give you a source type name called demo okay it will be stored in the search and reporting app click on save so default I'll say main leave you submit search so now if you see here when I index this data right apart from those those field value there a lot of date our they named a minute month second this value also getting indexed okay if you see me in real time scenario as well whenever you index the data all these fields should be by default added by Splunk so if you don't want to do that okay so there is a settings called this one add X at extra time fields this is taking either true and false value so you can if I make it false so to just demo it I'll just delete this data index equals to maintain delete okay I'll just okay I just need to add that rule so this is how if you do not able to delete any data from your index because of insufficient privilege this what you need to do to go to the settings plunks okay you need to go to our roles in to click on the admin role currently I'm administrator so click on admin role even admin role cannot have by default that can be literal so you need to apply this candidate role to yourself okay and then click on save okay so now let's go back to our settings add data so we'll go to upload we will select the data to here ok we'll click on next and now we will be doing in the same way I will be going to add once and then time break only before date is true then new settings time prefix equals to G Mt and then time format is equals to this time format and then timestamp look-ahead is equals to 24 and then the new settings called this one I had extra time prints okay so if I add this one to false ok and click on apply settings click on next I will give it a name called demo click on save preview so start searching ok if you see those extra time extra time fields are gone now ok so that is the use case of extra time fields now there are a couple of other settings called max diff second Seco and Mae Dave seconds hints so this settings are important when you have multiple sources in your system and all the sources through forwarder you are providing that are to your in the extreme right so in that case these settings are very much useful the max tip seconds ago are basically telling Splunk that the current event you are indexing and the previous event comes there is a this much gap in the in seconds in the time stamp for me the seconds this much gap so I think the default value is one hour so you can play around with this value and you have multiple sources it is always a possibility that there will be a huge time gap between them so you can play around this max div seconds ago and Max div second hints so seconds ago talked about in past okay and second hints talked about in future so similar concept of max days ago and max day Saints mmm apart from that you can have your own time zone specified in your problem as that all events will go by that time so mmm but if you have unioad in in your event you can you can extract time zone you can always better to use this time format to extract the time zone as well mmm time zone alias also you can you can specify where whenever you have a similar name of the time zone you can you want to differentiate among them or between them so you can have this time zone alias as a key value pairs as well there is another conflict called date/time config okay so that's basically specifies a file in Splunk Enterprise well let me show you that file first it is see in the CTC folder Splunk home it is a folder there is a file called a time dot XML okay so all different kinds of dead time ill dead time related configurations basically how Splunk extracts that this time are stored oddly logical stored in here in this file so if you want your own extraction method you can create a similar file in that directory and give that name into this particular settings data in config file name so you can you can always give that ok so that that is how this different time stem-related configurations works in the next video we'll talk about data extraction like field extraction using props dot convent and rotor come see you in next video

Channel: Splunk & Machine Learning

Views: 7,052

Keywords: splunk, how to, timestamp extraction, props.cnf, transform.conf, DATETIME_CONFIG, TIME_PREFIX, MAX_TIMESTAMP_LOOKAHEAD, TIME_FORMAT, TZ, TZ_ALIAS, MAX_DAYS_AGO, MAX_DAYS_HENCE, MAX_DIFF_SECS_AGO, MAX_DIFF_SECS_HENCE, ADD_EXTRA_TIME_FIELDS

Id: Q5EWCT79nZ4

Length: 18min 22sec (1102 seconds)

Published: Tue Dec 11 2018