GrafanaCONline: Grafana 7.0

hi and welcome to another guffaw nakhon and another major release so every year I talk about the next kind of major release at every go phonic on and I don't want to make this and it different but it is quite different I'm not on microphone icon stage I am doing this from the comfort of my home home and that's let's get get into it I have a ton to talk about and and show you so let's jump to there the agenda for today um I'm going to talk a little bit through I quickly go through the past year and see what we've done in all the minor releases and then jump into the big kind of things that we have done in 7o with a new panel editor then you can't date a model and pipeline that powers a lot of the new things in seven oh like the transformation feature and the panel inspector and I'm gonna end with with a look at our new tracing story and feature so a lot to talk about it's not gonna be not gonna cover everything that's new in seven oh so there's still lots to kind of dig into in the range log and what's new articles and we'll cover a lot more in in the coming weeks as well through blog posts and videos but let's jump in and take a look at what what we've been up to since last time so last time I did a presentation like this was at Corona con LA and we just launched in front of 600 there yeah and I'm see me looking pretty stupid in those glasses there Matt tricked me into coming out dressed like that with the graph on a cone beach towel as a cape anyway it was a fun conference and we launched a graph on a six oh but since then we have actually done seven minor releases kind of record for graph owner to do that many releases during a year largely due to the fact that we're a lot bigger team now and we did some cool is gonna highlight some of the some of the things we did in those minor releases not all but we added a new panel come building on a new panel architecture and the new new new react based panel add architecture we launched a new mortgage panel which looks pretty cool as well as the stat panel that that's going to be a replacement for the single step panel and this much more capable panel can actually be repeating repeat itself in different layouts supporting multiple series and fields and has some other cool things like automatically adjusting the layout based on dimension we launched this feature called data links that allows you to add links to your data that you can actually interact with the graph for example click a data point or click a series and then use the series name or labels or data point or data value in the link that you can then click on to go to another dashboard go to another system and yet the powerful thing here when you define these you get a help as well for all the all the different variables you can use in we invested a lot in the new Explorer mode as well kind of going deeper into the log feature adding support for streaming logs so if I now go to switch to the Explorer view I have a have actually a live streaming low key query here and you can sort of see results coming in live here that was kind of a really cool feature we we added as well as let me jump back I forgot to show you also that we we added this kind of detailed view for the logs where you can see all the labels and quickly filter on the labels and also look at fields that you have parsed out from the log line and we we didn't want to explore and this logs story to be only focused on loci so we be added support for influx DB as well and elasticsearch and will can expand on this over time as well so let's look at another thing logs context this is also a really cool feature now I'm switch to the light theme here which if you have it's done search for example here I've done a search on through my logs I can now open the context and see the surrounding log lines for this particular line I can see there the log line is proceeding and coming after and I can expand this this log context anyway I just wanted to show those cool features that we've added in Explorer last year in these minor releases but we did a ton more we improved all the cloud that cloud data sources cloud watch in particular but the others as well and its support for custom units and a news panel and a much more but throughout last year we also did a lot of work now foundational work for seven no working on this new data model and pipeline working on streaming capability that we haven't yet taken advantage of and and working on our new plugins platform through kind of focusing on reusable and easy-to-use components that will can speed up the final development and plug in the web so that leads us to seven oh and some of the big things that we have added in 7o which I'm going to demo and talk about in detail here it's the new panel editor which I'm gonna start with so why but why are we updating the panel at a dua in 700 we just kind of if some of you at least might remember that we did update the panel at UI in sick so we added through made this big change add these icon tabs on the side the big thing here was to be able to switch between different visualizations from within the edit mode but we struggled a lot with coming up with this UI design and we were never really happy with where we ended up and so the icons tabs on the side felt a bit can awkward their panel settings like title were a bit hidden and we also struggled with the panel settings there all the visualization settings and the layout we struggled finding a layout that felt easy to scan and didn't have a lot of dead space so we had an idea to put the options on a sidebar pane and we did a kind of quick proof of concept of this assign and that looked like this this was just an early proof of concept and we tested this with user to get some feedback and the initial impair their was really positive that this help kind of put all the options in a single column make them easy to scan make them easier to be consistent between different panels and also made it possible to have a lot more declarative options without kind of worrying about the complex layout and on a wide screen well there was a ton of cool things that that gave us confidence that this was this was a right thing to pursue and we also wanted to do some outs and I have some other big goals for this new ad panel edit UI to make yeah the options more consistent across panel types and we wanted to introduce new form components that also or easy-to-use have descriptions and so forth and we wanted to expose the new data model features that we've been working on specifically around data options and override rules that I'm gonna talk a lot about and so let's let's take a look at this new um let's take a look at this new ad modem is sign-in I remember my password and so I've had created this custom custom Home Dashboard here that it's a good kind of jumping-off point for all the things that I'm gonna demo so let's start with the new panel editor when I'm egg see here we have these this spy bar pane here that has things like panel title description I can collapse these sections in the sidebar pane here here's the panel type selector or visualization selector and if I open the display tab you can see all the display options here and these both these tabs are this side pane here and the queries area here can be resized at will you can also collapse this side and this options pane if you only kind of want to focus on the queries you can have the full full area to use for that let's let's jump to a real visualization here and take a look at this in more more detail here to highlight some of the new options here on new form Styles we have a new radio button designing a double make it a little bit easier to discover some of the options that are available like the display mode here in the in the gauge panel in the bar gauge panel you can see through easily select these these different options and it's option for show unfilled area for example and let's see here yeah and the other thing that I want to cover here is the field and overrides tab but so let's let's dig into that and I'm gonna jump to slides in a sick as well you to talk a little bit more about so these this feeling all right stab and how they fit into what we're trying to do I'm just gonna back like that so the field settings or the field tab is about the data options and our new standard standardized data options that are actually part of the data structure well and these the field tab our data options that govern the whole that govern all the series that we can think of them as defaults for all the different fields or series that we're showing here so if I change this to 200 for example you can see that all the different series here are impacted and so to give you some more context of what what we're trying to do with the field options here is to make a fauna more consistent across different panels we want to have the same UI for defining these data options and the override rules in so we want them to be the same in every panel so I'll just jump to the slides here a quick second and then I'll jump back to the demo to to show show you what I mean here so in the graph panel before we had this UI to select unit and min and Max and decimals - - for the different y-axis we had a different threshold design and UI in the graph panel and if you look at the single stat panel we had a different option to layout and UI for defining unit decimals thresholds and then in max so if you'd then go to overrides we had the graph panel has has a specific override UI that allows you to specify display options per particular series and the table panel had a very similar concept around column Styles that basically allowed you to do the same thing to have display options per column so what we've been trying to do in seven o is unify all this into a standardized model for data options and data override rules and this data option model is actually part of the data structure so all these options can actually be returned by the data source as as well so this this data structure with options can flow through a pipeline where we apply the panel defaults which is something you define in the field tab and we apply the override rules which you define in the overrides at tab so let's let's go back to graph fauna here and take a look at the overrides so this is the kind of unified concept that we're trying to provide a single signals concept for series override or column Styles and that will work for both time series and table data but let's let's pick a series here for example I can pick the the outhouse series and change the max value and if I change this to 257 it should change to max and if I change this to 10 or thousand it will change each other small value here you can see that this changed only impacts only impacts this only impacts this series and I can change any property here I can change instead unit or it is unit 2% maybe I can see that this unit has changed here the nice thing here if oh and I can specify custom thresholds as well and data links for each of these series so let's say let's specify some custom thresholds here and make this go maybe red to yellow and I can also do percentage based thresholds ingre front of 700 this is quite new not something you could do before so this will make the thresholds you know look at the min and Max instead look at the look at the so be relative to the min and Max I'm gonna pick two absolute thresholds me and let's 200 so let's take a look at what happens when I switch panel now switch visualization to a gauge so you can see that all the overrides are still being applied I still have custom thresholds and unit or for this particular series and let's let's which you a graph and you can see that all these that I'm actually visualizing time series here but what happens if I switch to able you can see that if I switch to table I'm only visualizing one series here at a time because all the time series are actually sort of different different result sets with a different time field but I can use a transform which is something I'm gonna dig into deep dig into in greater detail later but I'll use it here to highlight this fact that I do a join here and using a transform I will combine all the time series into a single table and if you look here at the outloud series you can see that it has this unit applied so that the override is still still taking effect even though I've transformed this data structure into a table that the override is still being applied so the override system is kinda can be transparently applied to time series data and table data and if I were to specify a display option here so so the the field level options and the overrides is not only for these standard data options you can also have table or panel specific options here so if you look here I now have column width of an alignment and cell display mode as kind of data options and I can now provide set these options using the overrides as well I could set at the cell display mode here for example on the outside outside theories here and say I want this to be a colored background oh that's a nasty let's insist and I could then maybe try to do a gradient gauge instead there I can use an override system here to specify different cell display modes and I could then set the column alignment maybe I want this to be user Center color centered column alignment so that gives you a little bit a sense of what we're trying to do here with field options data options and the override system and part of the 700s as well a new table panel with these new cell display modes and using this new data option model so I can actually part of this demo is I want to show a little more of the of this table panel just just quickly go through some of the cool options here you have the the color background you have the bar gauge cells play mode here and you have an LCD cell display mode here as well as well as a colored colored value mode and it supports data links as well and let's go back to their lives and talk a little bit more about this data model so during last year we worked on this new data structure that that combines time series and table data but the important thing is that the data structure that carries the data around is also also has this data option model that includes minimax thresholds units data links and custom display options all of those can then be kind of defined directly by the data source using using this pipeline or set by panel options or override rules and the goal here was really to make a funnel easier and more consistent and be more powerful to enable kinda more data-driven options and more dynamic option and also a big theme here is kind of stop reinventing the wheel in every panel but they can share the data option model share an override rule system and of course share the UI and help graph on be more sort of easier - easier to learn and master so just a sour note here or kind of is that we have some ways to go not all the panels in governor are currently using this new sort panel architecture so we still have some some old panels to to migrate to this new architecture some of you might know that Ken Griffin I started with angularjs an old dead library and and we have been migrating we react the best last couple years and we still we're pretty almost like 90% migrated now but the graph panel is the kind of one of the last remaining remaining pieces here so if you look at the graph panel it doesn't have this field an override tab and it will have these old options that you can remember from previous versions so and this old series override system so this will be on our backlog for for the coming minor releases to get this graph panel migrated to this new panel architecture as well um okay let's let's jump to transformations and talk a little bit about that so before before I start demoing transformations I need to move my camera because otherwise I'm gonna cover the transformation editor UI so yet transformation is really a new step that you can define where you can define rules transformation rules that modify your data before it's sent to the visualization and so to show you some of the use cases here I'm gonna go to this transformation dashboard and take a look at one of the most straightforward use cases is when you want to customize the table for example so here we have a table table panel and table result set as well and I just want to maybe change the order of change the order of these columns here maybe rename a column and I can use the organized fields here which is kind of a combo transformation that allows you to maybe I want info to go first and I want min and Max and values right now have adjusted kind of the ordering here I can also do it renames here so this is just kind of manipulating the data in a tiny way to fit the visualization I want change the order and rename and I can also do hide here maybe I'm not interested in min and Max and I just want the value so you can hide columns here and these are this has been a top-rated feature request for the table panel in Ramona be able to do this to be able to change the order of columns hide and rename and now now this is something you can easily do with a transformation but it's also available to all panels and all visualization types so this is the crucial thing with transformations is that allows us to reuse data manipulation between different data sources and different panels so to highlight another thing that that that fauna had in the old table panel we had these transformations to transform time series into transform time series into table structures so before this was a feature that was kind of built into the table panel and hence couldn't be reused by other visualizations so if I look at these time series here and switch to the table panel we had different transformations to to kind of transform this data into a table structure but want to sort of so the goal here with transformations is to bring that out of the panel's or and we have transformations features that was part of the data sources as well and we want to bring those out of the data sources and make them into a reusable concept that we can share across now I can now we have this outer join transformation that will join all these time series together on the time field and we can get a table showing the showing you all the time series and let's look at some other use cases here another use case similar use case for various kind of this concept of transforming time series is in Prometheus they we had this format option and some some other data sources also have this format option to specify how the data should be returned how the time series to be returned so this was basically a transformation feature that was built into the Prometheus data source the transform time series the table using the labels as fields but this was not available to any other data source it was just a custom feature in in the communities data source now this is available for every data source that is returning tags or labels through a transformation we call labels two fields the labels will two fields will then extract the labels and add them as as fields which will show up as columns in the table and and so this feature yeah it's available to every data source now and it's not something built in custom for a specific data other really popular kind of eg requests Agra fauna has been able to do simple math across queries and not every data source supports the doing simple arithmetic adding or subtracting the values from different time series and it's kind of creating a new time series from that result and that's something you can you now do using the calculate from field here so you can create a total time series based on on some other I'm serious here and you can specify which one you want here or you can create a some other kind of calculation maybe create a difference between two different time series you can use the binary operation to really do some kind of scaling if you want to do a scaling or subtract or add a different time series with each other let's yeah solves a really really popular feed request many data sources that lacked it the ability to do math on multiple query results um beat the other one that I didn't demo before was this reducer that allows you to show the minimax maybe last first and last of many time series on the table so this is also some kind of transformation that existed as a built-in feature in the table panel in in the old table panel but now it's available as a transformation so the this is a way of kind of transforming time series can be used by every utilization I can now sort of pick pick another visualization here and show this data lastly the last use case that I wanted to go over for or transformations is around filtering oh and and reusing the same query results but a kind of hidden feature ingre fauna is the fact that you can reuse the same query result in many different panels so in this case I have panel here that returns three time series and what I can do ingre fauna is used it at the dashboard data source and pick this source data panel as my source data and so this will reuse the same query result no new query is issued and the cool thing is using the filter by name transformation I can now filter out a part of that data set that I want to visualize in a different way in this panel so here I kind of isolate to two different time series in their own graph I am here picking the same data but just joining it by time and showing it in a table and in this panel below I'm also using the exact same query result to visualize it visualize the current state the last state using a gauge so if I hover over the last values here they should match so nine in the first server 88 on the on the row back in Oaks and then 17 on the third one so that matches the last values here's and no and all these panels use the size result from the same query so I'm super excited about kinda what Transformations means for graph ona in terms of opening up a whole host of new features that we couldn't really do before or we didn't have a place to put them in that was maintainable and so could be reused across data sources and across panels so that that's why I'm really excited about about that let's let's jump back to the slides and and move on yeah so this shows highlights the fact that I'm excited about moving features that were custom-built for data sources or visualizations and move them into feet into transformations that thing can be reused for every data source and every so a summary here for transformations yeah it's about getting the data in the in a shape that is suitable for the visualization example there is a naming ordering and hiding parts of the data transforming time series into different table structures and reducing the data in different ways and doing simple calculations across query results well that's some of the sort of some of the use cases here that were looking at okay and we down two to go let's take a look at the panel inspector and let's go back to this jumping off home dashboard here and look at the pan so this is also a feature that tries to unify and share more across different and all types ingre fauna specifically around the feature of looking at the raw data looking at the data behind the panel and being able to download that data as CSV works so this is a not something you could do is linger fauna before before you were kind of the only the graph panel and the table panel had a CSV export feature but there was no way to look at the data so directly from from within grow fauna so for example if you look at this baggage here what I can do now in this new with this new panel inspector is look at the raw data so this is just showing the exact same data that that that I was shown in this poor gauge but I can also here turn off the transformations and look at them so the all the values that went into that bar gauge panel and similarly here I have a time series data I can open the inspect Rohrer here and I can look at at although all these different time series and download them as a CSV file there are some other tabs here stats goes on query query times I can so look at the Jason for the the Jason modal for the panel and I can also issue the sort of open the query inspector which is really there for troubleshooting very problems that you can identify here by looking at the raw HTTP request and response that crow fauna is sending and receiving um there is more that we have kind of done in this panel inspector so let's take a look at stats feature for example the stats feature can actually include more information it can include processing the information from the data source so for for metric tank or graph on a cloud in a graphite you will get a lot more detail here that show you how expensive this query was and kind of detail around exactly what went on in the time series back end when this query was being processed so useful for for troubleshooting and resolving performance issues another thing and this is also this is can be implemented by any data source playing can return these stats that can be shown in the inspector or solo keys also returning these stats lots of detail here around kind of exactly how much data that was inquiry to process this logs query there is one more cool thing in the inspector order and that is the data source can implement a custom view as custom metadata a visualization to tr2 to help a user identify so exactly how the query was being sort of processed and rolled up so for a host African craft graph on a cloud graphite and they will get this kind of info icon to show you that data is runtime consolidated and show you exactly what consolidation function was used a very important or when using graph on over time series is really understanding and interpreting the data correctly and for that you might sort of it's important to understand how the data was being aggregated what function was used in in aggregating data either during the query time or if it's real rolled-up data that's also super relevant to know exactly how that process was was done and this is something that you can now do in in graph on a cloud graphite so I can see here that data is rolled up using a 30-minute average function so if I click there I open the panel inspector into this custom metadata tab that highlights exactly kind of the the roll-up configuration and also shows you all the steps that the query went through and all the different aggregation functions that was involved in getting the result you're seeing so super important to be able to really interpret the data correctly you that was pretty much it as it comes to the inspect panel this is a panel now that works on every panel both new and old will get the ability to look at the underlying data and download it as a CSV as well as look at these stats so let's jump back to the slides and I'm going to move back my camera and continue to the final topic and that's tracing so many I think are familiar with the fact that kind of graph ona started with a very heavy focus on time series and time series data we were a lot about metrics and exposing the power of Time series databases over time we've kind of expanded beyond that with additions of elastic elastic search and and other data sources like sequel data sources and last year with kevanna six so we really went deep into logs as well with this custom logs viewer in Explorer and as well as the workflow of navigating between your metrics and your logs but there was a big missing kind of story there of missing work flow in being able to view trace data from right phone with ingre fana that's why it's super exciting to announce that we have completed this this story in this workflow in seven Oh with full support for trace data and the workflow of kind of discovering traces and viewing them right from within Ravana so let's take a look at that wrong link let's go to to explore and look at a log square I'm gonna use Loki for this demo and I want to search for a trace ID because I want to look at trace data but I forgot how to write this query or I remember I have done this query before so I'm gonna sneak in another demo here alright it's a super cool Explorer feature that leg is also available now in 700 that's the query history and Explorer which allows you to kind of look at a test queries you've done and you can sort of filter it I remember doing this query a couple of days ago maybe four days ago and using this filter you can you can really discover exactly the query that you have issued before and explore and you can also look at your favorite queries and and just kind of run them and here I see I have actually stored this this trace ID query and let's let me run that and collapse this or a history tab so now I have my my Loki query searching for a trace ID and I've found a bunch of log lines here that contains this trace ID and let's expand this and I'll see that we have a poorest feel here with a trace ID and some buttons here that I can actually click on and when I click on that it will open big Ray's UI in this separate pane and here I have the full trace through UI where I can sort of click and expand all these kind of spans here and look at the detail for all for this complete kind of trace and we using if using the jQuery UI components here so a big thanks a shout out to the Jaykar project for allowing us to do this it's a really nice to be able to sort of collaborate with another open-source project like this and yeah excited to be able to have this workflow right from within the core phone UI so we we support more than J gear we support Zipkin as well so you can set up this workflow if you're using Zipkin as well and the way you this works at least this kind of work for being able to navigate from your local logs to to a trace as if you go to your data source settings or in loci in this case you can define these kind of parsed out derived fields and this allows you to you identify a trace ID in your log and then map that to a JK data source in Ravana that's all you have to do to get this this workflow working so that is pretty much it for the big demos that was planning to do I think I have time to sneak in one more demo but this is for the yeah all the grid mode if it's just a small feature that started as a hack day project from one one person on the team and it's just a more efficient way to layout stat panels and gauges that the reformer car tries to optimize the space used and this is completely automatic if you use the auto layout in these panels and which allows you to a girlfriend will try to figure out the best possible layout for all the panels and it's pretty cool yeah that was a sneak bonus demo I didn't try to didn't think I would have time for it and I did sneak in a demo on Korra history when I did that tracing demo so let's let's say jump back to some finishing slides let's let's take a look at sort of some more things in the front of seven oh I have this this demo and this talk hasn't been sort of problem everything that we've done there's a lot more to sort of show and demo and talk about in the in the coming weeks um so one big thing that I didn't demo here is full support for Clara birch logs in the Explorer view if you're a cloud chaser there's a that's pretty nice to be able to search and look at your logs right from Agra fauna there's also a really exciting couple featuring in front of 700 enterprise and really excited about what the team there has been able to accomplish in this release I don't I didn't want to focus on on these in this talk as I wanted it to be mainly focused on the open source side of things but Griffin Enterprise M&O has some cool features in enhanced search allows you to identify popular and unused dashboard as well as a dashboard usage stats to understand how a dashboard is being used as well as being able to sort of see the presence of another user looking at the same dashboard as you are and ton of ton of improvements to data sources and reporting and and yeah we'll cover this in separate blog posts and videos in the in the coming weeks I also want to just kind of highlight the fact that we're investing a lot in making a phone a really amazing platform to build solutions on top of I'm excited about through what we could funnel Labs will do and build on and all the tools were also giving customers and the community to build cool solutions on top of grifone are using the component libraries and the documentation that we have published so yesterday due to a demo that a quick we have this that this component site that where you can look at all the different components that are in graph owner even the visualizations are here so you can look at the the bar gauge and get information on how to use this in your own kind of graph on our panel or application so we're investing a lot in that we're also doing the same for backend plugins we're coming up with a completely new kind of way to develop back in plugins or a new SDK that will help o authors get get going there and share much more between different between different plugins as well as investing all in the documentation and tutorials so on that so we're revamping the documentation as well trying to simplify and the structure of the graphical documentation and update them as well as Laurie have launched this new solution space tutorial site where we will cover problems also get get users familiar with solving real problems in Reforma as well as guides and tutorials for how to develop plugins yourself so take a look at that if you're interested let's let's move on to some announcements so I still have two announcements starting with the graph on governance this is something that we're releasing today and this is basically making our internal process more open and transparent we're going to have a public mailing list for development and we just want to be more open and transparent with how decisions are being made and and let the community be more more part of what what goes on when we develop core so another thing I want to cover here is fact that grifone 7o is released today and the states so this table is available I think published just a few minutes or maybe an hour or so before this talk before this talk so it's available you can download it right now and I just want to end with a big cheers and thanks to the graph on a team for working so hard on this release and to the phonic community for being so awesome all these years giving us incredible kind of love and appreciation for what for for the project and as well as kind of helping with a project in in different port form so here are some things

Channel: Grafana

Views: 16,891

Rating: 4.9818182 out of 5

Keywords: Grafana, Monitoring

Id: EAQLgz7AckI

Length: 47min 42sec (2862 seconds)

Published: Tue May 19 2020