Dashboard Dirty Tricks

[Music] thank you all for coming today to splunk b-sides and today we're going to teach you dashboard dirty tricks of the simple xml type that was my son noah and we are the father and son core team of splunkster noah is my senior splunk engineer he's been with us since he was able to splunk right after college straight into splunking for the last six years uh i've been splunking since 2006 since splunk 3.0 and full-time and have had my own company for two-thirds of that time at splunkster we really focus on the tough cases um if you've got something that you've tried and failed you've had other people try and they failed if your situation is a mess we can take you from wherever you are to wherever you hope you should be we specialize in going from good enough to out of this world that's what we do we're all splunk all the time if you need help with your splunk situation give us a call you might have heard me say simplexml and thought isn't that going away i've seen the dashboard examples app the dashboards beta app and i know what's coming you know everything's going to be click a button get a dashboard we're not going to be using simplexml anymore yeah i i've heard that before too but not just this time i've heard it at least four times i heard that with django i've heard that with d3 i've heard that with the new what's the new framework um i can't remember the name of it there's the new one that's coming out it's doesn't matter simple xml was there in the beginning it's here now and going stronger than ever and it's never going to be removed even if they officially deprecated it's not going anywhere and neither is the huge amount of dashboard expertise and dashboard xml code that's out there so you are well served to keep using simplexml and get some new tricks in your bag if you use uh if you've used dashboards for any amount of time you've used tokens we use them on i think probably every dashboard i've built has had at least one token on it and a common issue that i've run into is being confused setting up the tokens you've got all of the different fields and they're not very clearly labeled you know when you're setting those tokens whether through the gui or manually you can't just say your field name in the data so i've had to find ways to get insight into what tokens i'm setting in the conventional approach is to use the developer gadgets app for splunk and if you're your own admin and you can just install whatever apps you'd like it's totally fine to have a specific app just for developers but many admins hesitate to install an app just for one or two people so that much easier sell and the sneaky way to get this is to ask for the dashboard examples app and not only will this be helpful for other people but this will also come in handy for you it's a great reference to be able to get some good well-designed code for everything that the splunk dashboards are capable of doing but you also get with that a free debugger and not just any debugger the exact same debugger from the gadgets app and that has a lot of other cool stuff and there are a bunch of other apps that we demoed along with that one you can check out our conference talk from a couple of years ago 10 best splunk apps and why you're crazy not to use them but right now i'm going to show you how to back door your way into the debugger in case you can't get the full app deployed so i just clicked on dashboard examples i'm here and you'll notice that simple xml examples is that is the name of the app in the app context where we're operating where our code runs and that's important because your dashboard code has to run in this app so once you're here you go to settings user interface which takes you here and add new you're adding a new view and in this view you're going to copy your existing current dashboard code and you're going to give it a name that's easy to find and lets people know that it's okay to debug it if they don't know what it is erase me debug and what you notice at the top is that yours probably looks like this or maybe even thinner like this or maybe even it looks like this it doesn't really matter if it's a dashboard or form i'm not going to get into the differences there but what's important because it all works the same is you put after that first word script equals event toker token browser js in your it's got to be in this app context remember you click save and now when i load this dashboard i'm going to get the same dashboard i'm used to seeing at the top but in the middle there's going to be two extra pieces in the middle is a token debugger and at the bottom is going to be your source code so in one long view you can see everything so here is the token debugger and here's my source code and what you notice is that i can set any of these execution stages to show the debugging and probably mostly what you'll use is done and preview so if i click on these and what you'll see is as code is happening in done it sets a bunch of tokens and here are all these tokens and i have actually discovered several undocumented tokens there are several for trellis that may not even be in the documentation yet and you wouldn't know they exist unless you went into the debugger and started clicking around and see and we discovered tokens that we really needed it saved our bacon the trellis tokens and it the only reason we found it is because we loaded up the debugger and we saw so make sure you delete your dashboard when you're done so you don't pollute the the dashboard examples app another really really helpful um thing in splunk is search macros any amount of reducing duplicate work in anything coding especially the more code you can reuse the better so i'm sure you're already using search macros but a really handy thing is shift control e so when you are using a macro in a search or you open a dashboard panel in a search uh there's that macro name and it's kind of a black box you know you're trying to think what filters did i put in there you know am i am i do i have all the filters i need do i have all the code i need and when you hit shift control e it will expand that macro in the window and merge it with the rest of the code into one search which is great because now you can see all the code together but it's also bad because now you can see all the code together you don't work and you don't know where the macro stops so an easy thing to do is to annotate that macro with um this trace comment like this so you can sandwich it you know it's the bun of your macro and you can say rename trace as beginning of macro and then your macro name and then all of your macro guts and then end of macro and this is going to be inside your macro it's not going to be on the outside in your search this will be in the guts of your macro so you only see this when you use shift ctrl e but this is a great way to not get confused as to which code is where yeah that uh if that code also shift control e might not be the exact thing different languages do use different keyboards and different encodings and it's different on a mac so you might have to look up in the documentation what the exact keystroke is but the point is it's there and you should use it and you can also that works for nested macros so if you have multiple macros uh stacked together yeah it's a good point um that's especially helpful because the problem just becomes more and more compounded the more macros are inside of each other yep he's talking about not the shift control e but the uh the bun that bun is super important all right so uh this project we've been working on recently has a great deal of mapping and i'm sure you've all been touched mapping a little bit it's really cool it does it's a great visualization but what we found is that the documentation is lacking it isn't that the documentation is wrong it just doesn't tell you everything you need to know there's some basic getting started stuff that that some other places like these blogs show that's just not there and it took us forever to really come up to speed on maps we thought it would be really easy it took us weeks to really figure out what we were doing and get it even the very basic things working and what we found was that these links are really what most of what you need but before you jump in and start doing that the too long too long don't read is don't use core maps you can make it work you can you can use some of these tricks that we're going to show you to really stretch its capabilities but what we found is we almost immediately outgrew maps and had to move to maps plus and what was surprising is there's basically nothing that carries over maps plus is a complete redo it's a full uh re-architecture from the ground up there's almost nothing that you can save if you start with maps and you decide to move to maps plus so don't even bother just start with maps plus it's written by splunker it's supported by splunker it's part of his job it's uh he he's very responsive we've gotten bug fixes handled right away we've gotten uh interesting feature requests handled right away i mean i can't guarantee that you'll get that kind of response but you know be nice to the guy it's a great product if you have a good idea he'll probably like it and if he likes it he'll put it in that was our experience apps of sites in the us for a client and we did all the code we had the visualizations showing up and they said we can't zoom in far enough you know we we're not getting good detail we're looking at texas but we have multiple locations in texas we need to be able to see all of them and we dinked around in the settings and we said all right no problem we you can zoom in as far as you like now and they came back to us and said yeah we can zoom in more but it's blank and that was much harder than removing the zoom limitations so you can see here that it says that the default is 7 on the max zoom value and that's for a very good reason because by default when you zoom in past seven the maps are blank so your visualizations your pie charts or whatever will be there but there won't be anything around them they'll just be swimming in the ocean of light or dark mode so the solution is really easy here's the way it looks in the beginning and i'll show you what zooming in looks like if i zoom in well i can't get zoomed all the way in so i'm prevented see it i can't zoom in any farther i'd like to see fort worth but i i can't so okay no problem i'll remove the limitation i'll edit this and i'll go into the tiles tab and okay max zoom seven i'll set it to 20 bumped up great so now we come back out here and now i'm zooming in and what happened but you see the problem all i've got is nothingness where am i where's waldo this is that sea of nothingness that noah was talking about and the reason that zoom of 7 is there is to make sure that this doesn't happen so that there's always detail and if you need more detail at a larger zoom you have to set a tile set and i don't know why they didn't put this in by default it's got to be some copyright problem but they say right here here's an open source tile set just grab it right from here paste it in and use it and once you save it boom now you've got everything look there there's fort worth there are all the streets but if you don't know that trick and it's not documented nobody calls it out it's not rocket science you're going to flounder around for a long time uh wondering why your maps don't look like everybody else's maps and all the demos and all the videos you see but it's really easy to straighten out my pain your gain exactly one thing that is great about splunk is there are many ways to achieve the same thing but there are also best practices and just because something works doesn't mean it's the best thing to do or that we will scale right we ran into this with um logic multiplexing and handling multiple token sets in one search so in this situation in this code here we need to set two tokens each with two possible values based on the search results you'll see in this first condition that it's looking for where count one is greater than zero and count two is greater than zero when that condition is met when that condition is matched you can see that it is setting both show tab 1 and show tab 2. now if both of those are not greater than zero not equal to zero it moves on to the next condition match which is count one not equal to zero and if count one is not equal to zero it will set that show tab one in the third condition match is count two not equal to zero show tab two and if neither of those are true it will not show either of them both of those tokens will be unset and this example is simple so this would work fine but you run into major issues if you're setting multiple multiple token sets like you are quite often exactly it just doesn't scale so here's the same set of conditions but it uses a trick that we discovered by conjecture i we we had some some two or three variables and two or three settings and it was starting to get out of hand already and we knew we were gonna have some that would be have five or six and and so the with the multiplexing logic the number of settings you have and the number of values of each setting get multiplied so if you have um two and two you have four case four case statement you know four different case statements and in each case statement you have two tests but if you have 10 and 10 that means you have 100 different case statements and that's just impossible so we just said you know it sure would be nice if we could handle each variable on its own each token on its own so each so if you had 10 and 10 that means you'd have 10 sections with 10 case statements in each section but you you wouldn't have to multiplex them together they would be completely separate and you could just cut and paste and change one to two throughout the section and two to three and three to four it's super easy to maintain you could even set up some macros and stuff like that work great so but in order to do that you would need multiple sections and it turns out that you can run multiple sections you can have multiple done sections multiple change sections we've had five or six i don't know if they run sequentially or simultaneously but doesn't matter because what you're doing here is separate separating out what should be separate conditions in logic anyway this just works and it's not documented anywhere and probably this is the only place you're ever going to hear about it but this will make your dashboards much much easier to read and much easier to maintain splunk is very set on sorting lexigraphically uh sometimes if you think you've gotten away from it you haven't it'll always come back and resort for you so in this situation you need to sort fields by severity and then lexographically so in within that severity it will be sorted lexigraphically we were having trouble finding a way to do this and we realized that we could pad and cheat the system by adding white spaces um leading up to our field values so as you can see in the comments it says rename comment one of two sort uh worst putting worst access first and left most uh this assumes highest severity equals worst so let's say uh there's a scale of three red yellow green and it says highest severity equals worst so that's going to be three yellow is going to be two green is going to be 1. our eval line line 3 there is going to add three spaces in front of the reds and then within those all of them that have the three spaces those will be sorted lexographically and then two spaces for yellow and one space for green and this was the easiest way for us to get around that limitation and have them sorted by severity which is very important because you want to see especially when things are paginated you want to see all of those critical things first if you take a look at that link we're not going to go there but there's a thread where this topic was beaten to death and there are several different answers including the answer we just showed you it's got a much broader context than just this particular thing so you should check it out there's a bunch of good spl there that's portable for a bunch of different use cases so on our dashboard you can see right now marietta's offline he's sorted first in front of all the green so everything in fort worth is green but i'm going to change that i'm going to make the first asset what would normally be sorted first i'm going to put him in the middle by changing his state to yellow and i'm going to take the one that would normally be last and i'm going to make him red which should make him float to the top now keep in mind that these changes in the code they don't set the state they don't set the color that sets the state of the asset it says my temperature is too high and then that is interpreted as a severity and now you can see what's happened down here asset 25 would normally be alphabetically last but he is floated first to the top in asset one which would normally be first he's second and all of marietta is offline and because offline is less severe than these and then everything else is is okay so if you look at the labels you can see that the spaces at the beginning are not actually there the xml renderer removes the spaces so even though they're behind the scenes having a sorting effect as far as we can see they're not having any centering or display effect of any kind which is exactly what we were shooting for a super fun thing that one of our co-workers happened across is that splunk out of the box supports emojis we used this in the case of an overview dashboard just to get a really simple view of how things were doing so no numbers had to be read or anything you could just at a glance see the little emojis and what they were doing so your os supports emojis in your browser supports emojis and splunk uses that xml and html code so out of the box you're able to just use these emojis and you can see um in this code here at the bottom where it is using the eval and it's just that emoji straight pasted in so you can go to this unicode.org and there's a list of emojis all the emojis from facebook samsung google apple all of them and you can just copy that icon and paste it into that eval statement and it will just show up with that icon so it won't have um you know you can do a table and just have that emoji and you can get a quick at a glance view at what's happening and if you want to see more you can check out the emoji app it shows you how to scale emojis and do a bunch of other things with them i'm going to show you a really basic simple search i copied this right out of this out of the slideshow i pasted it in here and you can see that it runs and one of the artifacts of cutting and pasting in windows is you sometimes get these new line things that that come through so they don't hurt anything you can see my code runs right it runs and it does it but notice what happens when i delete this um i delete that and and it works it deleted it but these um these emoji glyphs they're two character unicode things and they appear to be one character but in actuality they're two characters so if i go to try and delete this this character by backspacing over it doesn't work because i'm actually in the wrong position i need to move one character to the left even then i miss it two characters left and then i can delete it which is even worse the farther you go so it's really weird um so over here i have to be like way like on top of the x to delete it right it's it's it's weird so you're gonna have some trouble editing if you use these so it's best to put them in it at the end and furthermore it's best to use one glyph per line uh that way you can see i just had the same problem you're only going to be off by one set and it's a lot easier to edit if you do it that way otherwise it's going to be really really confusing to try and keep up with but this is a really fun way to play around with quick and dirty graphical presentation don't need to install any apps you just go to that unicode page cut it paste it in it just works and that splunk base app doesn't just have emojis in it it has some different examples of code as to how to integrate it into the different visualizations like the the trend visualization and several others another i'm colorblind so this we actually had to figure out because of me because i butchered a couple of our dashboards what we had was several different dashboards with hex codes and someone said hey noah we need you to make this red that yellow and whatever that you know critical red worn yellow info green and splunk doesn't format the colors and that's it like that that'd be great if when you posted a hex code you know pasted it in be great for me actually that's true it still wouldn't help me it would help other people it would help the gibberish part of it um so you'll run into this where you don't know what that is and critical is kind of self-explanatory that that would be red but oftentimes you're just doing general formatting and you're going to have to go and paste that into a you know google or whatever and see what color that is and the way that we got around this was using tokens so you can see that we have a token color green and then you go in and you find the color green that you'd like to use and you put that inside of the set token and then elsewhere you reference it like you would a normal token so you do dollar color green dollar this is the part that was uh helpful for me because i was able to uh just they said all right green is good green means go so i could just say all right color green and this not only avoids confusion but it also makes sure that you're using the same hex code everywhere you can easily scroll through and say okay yep we're using this color green and if you need to change it you only have to change it in one place and it updates the whole board which similar to the search macros the less code rewriting you have to do in the fewest places the better in this case if they were to say hey make info blue no it doesn't even have to know what what blue looks like he just changes that word green to blue and now info is blue yep but it gets even better so in our case we had a situation where and you'll notice we're using the same code fragment here which is mapping field colors so mapping field colors here was hard coded i went back two slides mapping field colors here is dereferenced at one level but now we're going to de-reference it one more time it's the same mapping but instead of saying a color green we're going to say fort worth color because the fort worth assets are going to be the fort worth color the marietta acids are going to be marietta color and we're going to set fort worth's color based on the results of some search so if in this case the result that color for fort worth is green we're going to set fort worth color to green if it's blue you're going to set the blue and so on and if you accidentally make a mistake we're going to set it to the debug color so if we ever see this funky color show up we're like oh there's a bug in our dashboard that color is never used anywhere else so this is super super flexible and but it gets even better than that with all of these first three solutions i backed up a slide you have the same situation where your single dashboard now is coherent consistent and easy to maintain because red is this you color red no matter where you use it it references that init section but what if you have 50 dashboards and you'd like to have all of them use the same red and they say you know what that red needs is too bright and you need to tone it down now i have to edit 50 dashboards which is a paint but what you can do is you can use macros now this takes a little bit of doing because macros are only leveraged inside of spl and dashboards don't really use spl they use xml so you have to have some transition from spl to xml and so in this case we use a hidden base search which doesn't have any other post process search attached to it its only purpose is to read in any macro inside of this app which starts with xml color and set a token based on that color name to the value that's in the macro and you can even de-reference it one more layer because you'll notice that this color debug is referencing another color macro which is zero zero zero zero so this debug its color can be changed across all dashboards as well uh by setting it to some other color or to its own independent hex code so the key here is that you you have to have this base search and now all of your dashboards across your entire app use the same hex and be aware that splunk in most cases you can use the the pound prefix or the zero x but we did find one case the horseshoe mod is does not like zero x so we suggest that you always use the pound prefix when specifying uh hex codes because you might find that you need that dashboard that horseshoe viz and it won't work if you use the 0x lookups and summary indexes are a great tool to reduce queuing increase search speed um but an issue is that both of those things have to be created ahead of time that's the point is that you don't you're not running an ad hoc you load that data once and you're able to access it very quickly because that date is already curated and this comes into play if you're using a dashboard that has a time picker and some of the options are time periods that are not in your main lookup so say that you have a you know daily like a 365 look up and look up or a summary index and you have a search that's running each night and updating that lookup or creating that summary index and someone selects last 30 minutes on the time picker well if you're just using that lookup it's not going to return any results or it's going to return an error because there is no data from then because your search ran overnight but with this sneaky solution you're actually able to automatically load the search logic from that lookup creator search take that code after loading that job in and run that ad hoc search off of that exact logic and similar like we've been saying a lot of these hints have been saving you time in from pain of having fragmented code so you can see on the line that says set token equals history search you just reference whatever search you're using to create that summary index or update your lookup and it will bring that logic in but apply the time picker from your board to that logic and then display that on your dashboard one thing to beware is if your scheduled search is using all time then this won't work because the scheduled search has to be using something other than all time so that your pipe saved search would use your time picker from your dashboard so ours is running for last 30 days which is not all time which is why it works so the time pick right now is last 30 days and we're going to go down here and we're going to open this up and last 30 days is in the window of our lookup because our lookup operates for the last 30 days so when we go look at what search populated that table we see that all it did was an input lookup last 365 days because that 30 days is within that and that's all it did and it was very fast so i'm going to go back here and change this to today and that's going to load the very same everything on the dashboard is going to reload including this table at the end but today is not covered in last 365 days because today is today so when i open this what you're going to see is it calls the save search called daily populating last365 lookup it calls this search and pulls in its code so it does everything that this does but we don't have have the only code the only spl the only place that exists is inside this search so i don't clone the code and put it in my dashboard now i only have to maintain that code one place no matter what time picker value i use so this time picker example isn't either or but if you did this for all time you could create this to do the oldest time as pipe save search and then the middle year is last 365 days and then the section at the end for now and today uses pipe safe search and we actually did that as well and it worked very well but we kept it simple for this use case so you could see the the core difference is that you do different pieces now there is an additional nuance here inside this search where it says you must list all tokens used in history search in the depend section this is because once i set the token this history search token it's only going to get set one time so if there's a token in here and you change the token if you don't cause this search to rerun then this spl will be stale and it won't be correct so you can either use your token inside this search or you could set it as it depends for this dirty trick we have a dashboard with the default of all for this example we have a site select site selector and uh the default is all and then we have two other options fort worth and marietta and the issue that or the situation that brought this to light was we had people complaining that our pickers weren't working and we realized what was happening was they were selecting fort worth but the all wasn't being dropped from the multi-select so it runs without an error because fort worth is part of all you know that that works but they wanted to only be searching for fort worth so you can see um in this change stanza it says mv count form.site equals two so it's counting how many things are selected in the form and then it's indexing and it's looking for if that all asterisk is the first thing and if there are more than one it's you know it's looking for two in the first one of those is all it's going to drop the all drop that first one and only load the secondary option that you picked because the hall is there and after you select something it shows up underneath that on the multi-select in the envy find that next row down um is looking for if the all is in there the asterisk the wild card is there at all and in this case if all is at the bottom it's going to drop everything else above it because it says all includes everything so if i'm at the bottom of it whatever selected above it needs to be cleared and will drop down to all and this will do this dynamically as long as you have search when changed set to true if you don't have it set to true it causes all kinds of odd fragmentation in errors so when you do this make sure that you have search when change set to true for that input and also this is hard coded for the case where your all set selection is first if you have a different default you can use a similar strategy but you're going to have to completely rewrite this code for your use case but the vast majority of multi selects have an all case in them and they look just like this so we've only ever used this code because all of ours do the same thing so we start with all so that's the default so if i delete it it just puts it back all is always going to be there by default but if i add fort worth notice what it did it removed the all it was there for a second but it removed it by default it would just leave all and fort worth it would rerun my search which is a shame because it's not going to do anything because no data is going to change similarly if i have other things in here and i add all it's going to say oh all is all let me add just all and remove the other stuff now this might seem stupid but picture how bad it is with 50 settings if you have to train your users how to use your dashboard or even worse how to use a splunk feature that doesn't work that great in your dashboard they're going to lose confidence in you they're going to lose confidence in splunk they're going to be frustrated with your dashboard this is a great way to make it work the way that it should work anyways well and this reduces queuing too you know it's processing that before the searches are run so even if the users did realize oh i know why it didn't work they had to reload eight panels again and if you're in tight performance constraints the boards might queue after that because you're loading 16 searches you know back to back so this also is just a lighter workload on the system as well so it increases efficiency in that way as well and that's all we have we hope you guys enjoy splunk b-sides and we hope to see you back at uh conference we're going it's hybrid this year it's going to be in person i'm going to be there noah's going to be there we hope you all will be there and come by and say hi to us thank y'all

Channel: Splunk User Community Experience

Views: 309

Keywords: BSidesSplunk2021, BSides, Splunk, 2021, conference, Dashboard, Tricks

Id: d0XtQcPa5zw

Length: 42min 19sec (2539 seconds)

Published: Sun May 16 2021