6 Easy Ways to Improve your Log Dashboards with Grafana and Loki

welcome my name is Stefan dunkler and I'm a Solutions engineer at grafana labs and today I'm going to show you how to create your custom log dashboard if you want to query logs you typically start at explore mode explore mode choose the logs data source and then you have everything you need to get started with Loki and log ul and get all the data out of your logs that you need you can see what kind of labels there are you can see the label values available you can run the query add some filters do aggregations everything you can do with log ql that is super powerful and in explore mode you can even watch your Vlogs live meaning if you filter them down you can see maybe only the errors or whatever you feel the expression you need it to be now if you want to do this often for a specific use case then this can become quite tedious because you always have to enter the log the log ql statement even though the query Builder helps you a lot with that but if you want to have something reproducible and even interesting for people that do not know log UL then it is a good practice or my recommendation to build a log dashboard and that's what we're going to do now so first of all we are going to add that simple query to a new dashboard now with this panel we didn't gain anything as opposed to the explore mode so what we have here is just the logs using that query and no immediate possibility to filter or aggregate them down instead of just adding new panels and that's what we're going to do first of all I want to filter based on the cluster so what I'm going to do is to add a variable called cluster node I'm choosing the query a variable type because I want to do a log query to my Loki instance to query for label values of the cluster label and I also want to be able to choose multiple clusters or logs from multiple or all clusters currently I only have one cluster called microbs apply go back to my dashboard and I see this drop down list appears to be able to use that in my logs I need to add that label selector the cluster label using the cluster variable the dollar sign is assigned for the variable running this query will not change anything on this log because I'm still only selecting that one cluster as a next step I want to be able to filter for anything in the log and for that I'm adding another variable variables new variable this time I'm choosing the text box calling it filter because that's what I want to do with it it's just a very simple filter variable and I have a text box here now if I enter something like post here nothing will happen because my panel is not using the filter I'm going to line contains it's a filter in logul add the filter variable here run the queries and I get no data because there's an e here if I remove that I should be able to get all of the logs that contain the word post great so I can do custom filters I can add as many custom filters and text boxes as I want so I can combine multiple searches and also do exclusions like there's a negative filter possible that you can also use by just configuring your log URL in this log statements accordingly now the challenge with text box filters is that you always have to enter the search string manually you can literally search anything but sometimes you just want to choose things from a menu like you can do with template variables and labels but best practice is not to label the whole log message so the next thing that we are looking at are ad hoc filters to solve this problem ad hoc filters allow us to use labels that we pass from the logs and apply them on all log panels automatically across the dashboard to implement this I'm going to first parse the message we already mentioned this is in log fmt so I'm using the log fmt parser log format parser what this does is it will add this set of fields parsed out to my Loki log message now I'm going to duplicate that to not rewrite the query edit it and do some aggregations I'm switching to the query Builder mode click on Range functions count over time choose the range interval go to aggregations and sum by label now I want to be able to choose the method post get delete whatever the method is from a menu so I'm summing up my method I also switch from query type range to instant because I'm not interested in their full history of that I only want to know what is the current state in our case in the last six hours I already get like a few of the options that that it provides and now I'm choosing table as the visualization method see the time of the query the method and how often these methods appear in the last six hours in the log messages I'm using transform to organize the fields because I'm not interested in the time might not be interested in the in the value for now but I'm interested in the method and I'm giving it a name changing the panel title and press apply let's make it a little bit smaller and now I have a nice filter now if I choose post here in the magnifying glass it will automatically filter for the method Post in all of the messages I can remove the filter try another one to get method and I can easily filter for the method now let's make everything a little bit more visually appealing to do this first I'm going to duplicate this once more to keep the aggregation edit go to transform get this value because this shows the occurrences of the different methods within my log messages choose pie chart as visualization type and now I have a really nice visual representation of how often which of these methods values are occurred you will also notice this value a year which is when no value for the label method was set if you're not interested in this number we can easily filter that out by applying a label filter like this one method regular expression and then a simple regular expression for it needs to exist and when we run that query we see only distribution of those labels without the ones that don't have a method set now let's apply this put it together on the left hand side copy or let's add a title that's the method distribution apply and let's duplicate it one more time now this is pretty easy because now we can choose the log level for example same principles run queries apply and rinse and repeat once again forgot the title of the panel rinse and repeat duplicate and the path is something that could be interesting in that's an interesting one indeed because now we see there is a lot of different paths and there might potentially be more than you're interested in maybe you're just interested in the top K so what you can do is you can wrap the whole query in the top K function and maybe you're only interested in top 10 path of your application that are being accessed it's as easy as adding this statement to your path so this and yeah we can just make this a little bit smaller put it on the right hand side of this to have this kind of filter here and we just made everything a little bit more appealing next thing I'm going to use is an advanced concept of grafana that you can use with all of your queries not only log queries data links data links can take the query result and put it into an external or internal link or even link to the same dashboard if you put it to another dashboard in grafana then you can do a drill down to specific component if you're linking the same dashboard you can use it as a filter that's what I'm going to do for this I'm copying the URL note that the URL contains the cluster variable and the filter variable and this is the one that I'm using for my manipulation I'm going to edit add a data link use the URL I don't need the first part because I'm linking to the same grafana instance and I'm going to filter for the query result the part of the diagram that I want to click on contains the path now I apply that change and I can immediately when I hover over it it's a hyperlink filter for that specific part across all of my logs now let's make some final adjustments to the dashboard to set it in production to do this I'm moving that panel over there make it a little bigger give that one a bit more space can just the size here if that one a bit more space and give that one a bit more space now I'm going to add it to table and add a few more labels to it like the level and the path which then all can be used as ad hoc filters from within this table when I run the query I see level method and path I will also add the value here give it a name the amount and since this amount the number of occurrences of all of those combinations of log level method and path is not visually appealing I will add an override here to make sure that my base field name with value a has a different cell type called gauge let's choose a nicer more neutral color make sure that we sort by amount and because when I remove that filter here there might be a lot of different combinations I'm going to add another top case statement here so for example the top K20 of all of the combinations that could occur and that should be working fine if I now run this query apply and remove that filter well that looks awesome this dashboard is pretty intuitive but it might not be for some especially if you build a more staged dashboard because if you build a stage dashboard with a specific use case in mind you can make it look much more complex so what I like to do is to add a panel that only contains text with the instructions so in this case I'm calling it instructions you can use markdown or HTML or you can even add code and you can basically add instructions to use this dashboard if you click here then this filter will be activated if you do this then that that are enough instructions for now and because you know not always is it the case that you want to see that block of instructions here that could get quite lengthy you can introduce rows one row for the instructions you would also call it the instructions row and then another row maybe just below that one for log details and now if you're an advanced user of this dashboard you can just skip the instructions and go directly to the log details thanks for watching the video and I hope that you could learn a lot of new stuff and how to build log dashboards with grafana we actually did a lot in the past few minutes we did apply a label filter we added a text box filter we added adhoc filters we added data links we added some instructions and rows and we made everything look super beautiful thanks again and hope to see you next time bye

Channel: Stefan List

Views: 19,183

Keywords: demo

Id: EPLvB1eVJJk

Length: 16min 34sec (994 seconds)

Published: Fri Apr 07 2023