Splunk Configuration Files : Event line breaking using props.conf

okay today we'll be discussing about event line breaking in Splunk okay using the props con file so if you see if you follow my previous video where I have discussed the different phases a data goes through before we search the data right so the event line breaking happens at the parsing phase right where the input phase we get the raw data and then in the parsing phase the event line breaking happening using the different different configurations we have based on that what kind of data we are dealing with so whatever configurations will be we'll be seeing today that will be applicable for the parsing phase okay that means based on your server settings you need to place your profit count file to the server component well where it's basically dealing with the parsing of the data so that means if you have this architecture then in the in in the index R you have to place your flops that confer because index array is doing the parsing okay so now today mainly will be discussing about these settings okay now these are the these are the settings which are applicable at the parsing phase now there are a couple of settings which you can do it at the input level as well in the universal for water I will try to briefly explain them and while discussing them we need three of the settings for time stamp extraction and which we'll be using today now for time some extraction we'll be seeing in separate video for other settings okay so so before I start so generally this this happens whenever you have a new kind of data in your system right and you want to build a logic basically which will create the events from this raw data so that means you want to basically create this props dot-com file logic different settings right you want to explore with then generally it is a good idea to create a new text file with those data and then come over here in this settings add data okay and then upload that text file and then try with the friend of an option since you can see live house Blanc is breaking those events okay then then accordingly you have to tweak those those settings and then you will get your final props calm so for today let us talk about before I before I discuss any of this any of these events or any of these settings let us talk about how how Splunk treat the raw data okay so suppose we can have our interview or something like this okay just give you an example something like this so in each and every line we have some information okay so now by default Splunk basically treats each and every line so before creating any event okay by default Splunk breaks that each and every line okay now based on the settings of should line merger or not there is a settings called should line merge okay based on the settings this the specifics basically tyll tyll Splunk whether all my events are single line events or a multiple line events so for multiple line events the should line match will be true for single line events should line module be false okay now should line much basically as you have seen by default it is breaking at the new line level right whenever it finds a new line it's basically creating a new line of event and based on so if I say suit line majah goes to true and suppose I have some kind of settings like this so Splunk will create two events like this okay so if my should line merge equals to false so Splunk will create event for each line okay so this is how it works should line merge now should line much is basically defaulted to this new line to overwrite that you have this line breaker rule so here you can give any kind of regular expression which will tell you which will tell Splunk that instead of new line take this regular expression as a breaker or the new line replacement okay we'll discuss that one as well mmm so before our first file we will discussing today's data three okay so where we will start with line March mmm our line-break are basically line underscore breaker so let us see the data first okay so if I open the data tree file if you see it's a toy XML right it has messages tagged different kinds of different different messages tag inside this messages tag in today's message tag as well so what I want to know one to break is each and every message that combination should be a single event that is what I want to achieve now okay so now if you see I have just have this kind of data as a raw text file right so what I will do I will come over here I will upload that data data tree okay I will come to next now if you see this window comes up right so by default the unbreak event has been kept on over here at the right side window on the left side if you see there are three options called even breaks time stem and advance so will be in advance whatever configurations you can have it right we can put it one by one now even breaks also you can put a regex so that value is basically for the big break only before also so either you can use this break only before in the advance or you can give this regular expression over here it will create a break only before okay we will see that and time stamp is basically for the time stamp extraction from your event which we'll be discussing in a separate video for now we'll just try to see how houseplant breaks each and every event okay how creates now let us so we have already have this data three right so for data three so I'll be saying my line breaker should be what at the message in level right so if we let me I think that close to the data tree let me open that so that means at the each and every messages that this should consider is my line bacon instead of new line right so that's why I have given that so we can come over here we can add a new settings give line breaker and give this regular expression okay so let this this is a very single regular simple regular expression which we'll test it over here okay so for that what will I will go to data three copy these sticks come over here and give this regular expression okay so what will happen if you see it is just identifying all the messages tag opening message this tag not the closing one right that means it should break before any opening message is Ted coming up okay that that's my line breaker concept mmm so generally we don't use should line much equals to true with the line breaker but I will keep this one just to show you how it works so now if you see this is a single event right and with all this raw data so if I apply this setting now if you see the line breaker just I didn't Splunk this what it is it supplies this line break a rule but it's still not catering you event okay it just applies this line break a rule but it it's showing us it is breaking it is considering it'll be line at the message is state-level whenever a new messages messages tag is getting opened okay okay so so now we will break this into a event so for do that as you as you see from this data I know from where my new wins are starting up right it is from the from the messages tag when we're a new assisted is getting opened right so for that whenever you know from where the new new events can come up and it can be generated you can use this particular settings called break only before okay so here also will be using similar kinds of settings break only before new settings so what we'll do we'll give this messages Tech opening messages right so it should break only before that so if I apply that now if you see Splunk already break those messages tacky for universities take discreet a new event so if I just go back to before applying this break only before you will see so if you see Splunk is breaking whenever it is getting that particular regular expression occurrence as a event new event okay so this is how break only before works okay so I just put put it back over here so break only before and message is opening messages right so there is another settings called must break after that means Splunk will automatically break that event whenever it is find that accredits but there also you can give that Ziggler expression so must break after generally use when you are you're ending of that event is so you know where your event is getting int but you may not have the proper information fin but your event may getting start okay we will see one example for that mmm so here I I can apply this one as well to make this particular settings more robust right so here I am getting I'm saying I'm saying whenever it is finding the closing messages tag it should break as well so it should give me similar events idea okay so this is how it is basically doing that so this is how you you basically do this break only before and must break after now if you see this XML tag is also coming over here so I can always use should line much false so that it will it will be this separate line mmm and all my messages tag will be separate separate event so you can you can always ignore this particular event generally you should not be putting this particular XML tag as the event but as I just I just took a raw XML file that's why yes I just show you this one okay so we saw how how this this setting works okay so let's move on and let's talk about this file okay we will try to break through time stem okay so data - we have another file data - so let's see this file is a very simple event file and we have some information here we have our time stamp over here some other information after that okay we will we will tell Splunk break these events based on the time stamp whenever you find the time stamp okay so similar way I will go to go back I will select the data - dot txt file I will click on next and then I will try to break on based on the time stamp right so there is a settings called this settings break only before date okay so if I if I so this this or this can take true or false value so if I say true okay in that case you need to give some time I time format identification so that's why we will be discussing this three time time extraction stuff okay so the the one is called the time prefix that means if you have any kind of prefix before the time stamp start you can give for us it is GMT so we'll be giving GMT okay so then we have our time format what kind of date format it is so it's it's Y M D then T then the time and the time zone mmm we have this format so I already have this format so I will be copying this and there is another format called max timestamp look ahead so what it do what basically if you if you copy this thing this stuff over here okay and now if I just remove this GMT portion the maximum length is 24 if you see here the length is 24 so that Clint we have to we have to give over here okay so if I apply this now if you see Splunk started creating new events whenever it is identifying new timestamp over there so this is how it works so let's talk about board settings so we have we have discussed opposed must break after right so there is an another settings called must not break up em break before and must not break after and must not break before so must not break after is just opposite off must break after right must break after is telling us if you find this particular regular expression break over there create a new event from the next make from next string onwards now must not break after it is saying me like what you you Splunk should not break over there okay it should continue there could be lot of scenarios in which you can you need to use this one so I'll give an example with this data I think I have data for must not break before yes so I'll just show you the data it's a very simple event where suppose I am running some bad job and that's that bad job is printing something over here and I want to analyze that so I have my events could be looking like this right so the job jobs each and every state of the job like whether a particular statement succeeded or not whether it's a it's it is some request has been sent or not it's some some file has been copied or not then the bad job completed then kept try to finish the piece press any key right so I always wanted to break as a to finish a finish press any key right so let me go back over here data for all right so now click on next right so now all all the raw data is coming together I will go to add once again so I will go to prom I will say must break after equals to that string to finish press any key correct so if I apply this it is it is creating that even two events for me to I just give you an example of must not break before okay so in this case what will happen this it will not clear an event before that okay so bad has been completed so okay apply settings so that means you can make it more robust maybe it is not the best example but in lot of scenarios you may need this must not break before okay so it should dismiss plonk will not break over here okay so must not break before you can think of it just opposite of break only before okay so we talk about a lot of configurations here mmm talk about data - yes we talk about it at three yes data for data Jason will give in simple example but before that let's talk about truncate and Max events okay so now I have given you an example right how how Splunk breaks the events right now it could happen and this is basically it's true for your multi-line events trunk it's basically tells us how many characters maximum can be there in each line and I think the default is 10000 okay and max generally tells us how many lines maximum you can have in your ribbons I think this is defaults to 256 okay so now you can always overwrite this but be very very careful on that part because generally in general sense you will never cross this limit but if you cross there is a high chance that you are ingesting bad data new system so we will be very very careful on that front apart from that okay so there are two other configurations called event breaker enabled and even breaker you can do it at the universal for available that means this will be applicable at the data input level right because Universal for order is always deals with your input I'm talking about input phase here this this input phase here right so here you can use a very low level of event breaker over there so to do that you need to enable this event breaker even Abel equals to true in the props conf okay and you can give a regex over here so that that using that rejects you can you can divide so Splunk will do this very low level of event breaking so that this generally useful when and you have multiple indexer and universal for really sending data to to do all this indexers the load balancing will be better in this server okay we also talked about this timestamp extraction now if you see let's let's take our final example then we'll show you that one so we have another file called a turn disco Jason if you see it's adjacent file but I say bleedin a text because Splunk is intelligently enough if it waits find it's a decent format then in the automatically format so I won't be able to apply my settings over there so if you see I have this kind of it I have two events basically here right it's basically a do not data and what kind of donut it is and how how the donors get been getting prepared like what kind of batter I used those kind of data it's just a simple data funny data I wanted to create so I created like this so if you see the ravines are separated by this one right then curly bracket then our this bracket and then calibrated end so we will try to see how we can create events based based on this kind of data so I'll go back I'll select data Jason I'll click on next if you see raw data is there so if you see I know already from higher my event is getting baked right so I can just give back only before equals to that so now break only before I'll show you from here even breaks I say as I told you in event breaks if you give any rejects right that creates not only this this is this particular configuration see if I give this one so Splunk is basically creating those those events right Totti variants if I go to add once if you see break only before only it has added through this one so now you have two options from here either you can copy these configurations okay so that if you already have your prob comp you can copy this configuration to place it over there or you can save it using the Save suppose I give demo right and category I choose custom a by two slits is such anybody so that the props are count will be saved in search and reporting app I'll just click on save okay so now if you see if I go to my Splunk home right I go to ATC apps and then I go to search app right if I go to search app local folder this props con has this demo stand yeah right and all these settings so as as I told you we we are doing it for the parsing level N and as we have different different kinds of file that's why it is a source type level configuration if you see it here so now what what you can do in the next time onwards if I go back I have this data does this one under Scottish undertakes selected right if I click on next instead of applying this file I can select this source type from here Splunk will automatically apply that if you see the result is same so this is how it works the whole cycle works so if I have some raw data you you you get the sample some of the sample of the data come over here do the experiment say very nice prop those comps and use it in use that particular source type everywhere and if you see wherever you go in in Splunk data inputs everywhere you will have an option in in sprung each and every in particular option to give the source time so you can always give that source type you have created then accordingly spunk will break those wood or data entry events hopefully it will be useful to you guys see you in next video

Channel: Splunk & Machine Learning

Views: 12,437

Keywords: splunk, how to, event, line breaking, TRUNCATE, LINE_BREAKER, SHOULD_LINEMERGE, BREAK_ONLY_BEFORE_DATE, BREAK_ONLY_BEFORE, MUST_BREAK_AFTER, MUST_NOT_BREAK_AFTER, MUST_NOT_BREAK_BEFORE, MAX_EVENTS, EVENT_BREAKER_ENABLE, EVENT_BREAKER, TIME_PREFIX, MAX_TIMESTAMP_LOOKAHEAD, TIME_FORMAT

Id: 2utc62Xy3Qc

Length: 21min 40sec (1300 seconds)

Published: Wed Dec 05 2018