GrafanaCONline: Prometheus rate queries in Grafana

hello my name is Bjorn I'm an engineer here at katana I got hired mostly because I'm that Prometheus guy so I talk about prometheus all the time and co-founder Kahn is no exception I want to talk about prometheus read curious Engram I mean I had to put a bit of qivana in there to make it make it viable for co-founder con it sounds like really basic you have probably dealt with katana before that's why you're here at co-founder Kahn and you're probably dealt with prometheus before this is why you pick this talk and watch it in K instead of doing something else because you are not really at the conference you are at home you could just do whatever you want so I'm assuming you a bit familiar and I will not just talk about basic or it curious I will actually get to the bottom of things we will learn everything about it we will finally understand what's the subtle or not so subtle difference between range vectors and incident vectors and range queries and instant vectors instant queries I got it wrong right I mean it's it's kind of confusing terminology but it's important to understand we see how qivana actually grasp something that comes from Prometheus so yeah we all do that I'm a bit sad because I usually love to meet all of you in person at a conference but I also have other things like whiteboards this is a whiteboard behind me I'm nerdy enough to have a whiteboard in my living room and now I have this this unique opportunity to let you all watch me whiteboarding which is usually not possible with a lots of people but yeah I mean modern times we have we have something interesting here and then an advantage of the current situation action so let's wipe on something let's talk about rate and rate is probably the most used function in prompt QL but it's only ever to be applied to counters sadly Prometheus won't yell at you if you apply to something else purpose that will change with Prometheus three or four five at the moment you have to pay attention yourself since counters all the one and anything you apply a write function to let's talk about counters first let's talk about the specific counters counter counters count things but they can also count quite esoteric things like time so let's pick a counter that counts time so that would be a counter that counts the seconds that a process has been using the CPU it could be fractions of a second and this is a really powerful counter to measure CPU usage of a process and because it's so powerful every decent prometheus instrumentation library has this counter built-in if you link in the interpretation library and you don't do anything else you don't actually instrument your own code you will still get a bunch of standard metrics like this one and it's already quite powerful this alone is a reason to instrument all your code with the Prometheus rotation library and use Prometheus to collect this counter now I have forgotten something here I have forgotten the suffix that you should always have at every counter which is the total suffix it's called total because it's totally just a convention it's kind of sad that it's just a convention for me it should yell at you if you don't do it open metrics might actually yell at you if you don't do it or future prometheus version might yell at you but for now just be super careful to always name your counters in a way that they had on total that helps you also to never apply the right function to something that doesn't end on total okay so that's the counter now let's talk about reality and measurement which is kind of a philosophical topic so this is the value of the counter and since we are talking about Prometheus we talked about time series data so this is overtime zero seconds one minute two minutes which means one little tick here is 15 seconds a counter it's doing something in reality this counts seconds that you spend on the CPU so if your process doesn't spend any time on the CPU the counter will be flat it starts to spend it will all go up if it goes up steeply I spend a lot of time on the CPU and if it flattens out again you don't spend that much time so this is how such a counter could look like in reality it never goes down unless there's a counter is that but it could stay the same um this is reality that's not what we can perceive because we as human beings we kind of perceive reality directly even the kernel cannot do that so the kernel measures this value and turns it into numbers at a certain frequency Prometheus has to do the same thing and is even like farther away from the kernel so I cannot do it like every microsecond or something it will do it at fairly long intervals called the scrap interval it's a minute by default but most Prometheus users set it to something shorter we use 15 seconds at qivana that's why I use the tick marks here you should also try to be uniform with your scrap and I will throughout your organization that helps a lot it helps to avoid a lot of trouble sometimes you have to pick something else for specific purposes but you should have something that you use almost all of the cases so 15 seconds is it here Kofun elapsed and then Tobias comes along and scrapes this value it measures takes measurements samples whatever you call it and that's how Prometheus perceives this counter saves it in its psdb then we have to reconstruct reality from those samples okay let's look how this let's check out how this looks like in prometheus this is the Prometheus expression browser it's not yet qivana I've also chosen to use the modern react based UI you can pick that as well if you are on a Prometheus server on its web interface there will be a link try out the new UI and you can see there's a link classic UI which you could click to come back to the classic art but let's say modern use this red base UI we can just type in the query name here process [Music] seconds if I just hit enter now we would see a lot because this Prometheus server monitors a lot of processes I'll just do I pick a name space DNS demo that's that's one of our stock demos that I can also use for this talk I hit enter I get an empty results because totally intentional I forgot the total and here we are so this is the current value of the counter for three different processes our demo app here just has three ports with one container each with one process each it's pretty straightforward in reality you might have thousands of that cluster instance it's all their job labor namespace level from kubernetes whatever it's called an instant vector instant because it's like at an instant moment in time and it's a vector because it has multiple elements it could even have only one element we would still call it a vector it could have zero elements that would be a zero element vector there are also scalars in Prometheus but they are really beef so even if it's of one element vector it's not automatically a scaler keep that in mind the interesting thing is how is this value are actually getting into this UI and UI is running on the Prometheus server but it still gives us the normal Prometheus query API to populate those values your browser has tools like the developer tools that you can switch on and then if you run this you will actually see what this URI is doing in the background it's actually running HTTP queries against the API v1 query endpoint this clearly endpoint is also called the instant query endpoint it's not called Kure instant but believe me this is the technical term so we use the instant we recall where the query contains an instant vector there are two times the word instant but it's different instants so it's instant Pingree and it has a query parameter and you see process CPU seconds total that's our query and we create an instant vector there's an additional parameter here that's the time that's the time the point in time or we want this instant value to be from we could specify any value we could also leave it out then Prometheus would use its notion of now in this case our browser's notion of now is this time step so if you just do this again it will will change the tiny bit actually this is a new query so we have to select it and now you see this has changed for a few seconds you could use a time some in the past of course and you could you can use the time serve in the future if you dare to ask Prometheus for the future if you can try that and help me we won't do it now okay now what we usually want we want pretty graphs in Prometheus and this is where we go from the table view to the graph view and here's a pretty graph one minute let's take like an hour okay I've probably played with that before usually it's one hour by default so this is three countess it doesn't go down as promised but it didn't appear to go up either that's because that's actually they are actually going up we could just can't see it because you see those values are vastly different because those three processes spend very different amount of times on the cpu so I can go to one of them let's use the busiest one the orange one here this is job DNS demo app okay yes and now we see we look only at this one now we have a vector that has only one element and you see it's going up most of the time now the interesting thing is what is Prometheus actually clearing because before we just fear it a single point in time but quite obviously we are looking at a whole range of times here so we check out our developer tools again inspect that query and we see it's now your query range so that's now a range query endpoint and it's thankfully explicitly called query range the query itself is still the same here it contains an incident vector but now we have a start time here an end time and we have a step so this endpoint is essentially just syntactic sugar for running a series of instant queries run one at this timestamp then add step run another one at step again run another one until we hit the end times them that's what Prometheus gives us an API endpoint for so if you want to graph something you don't have to do it individually it's the range query we are using the range query call where the query contains an instant vector and we get we use the result to graph you could do something else pipe it to your favorite stats to all or whatever now let's zoom in let's reduce this time span here 30 minutes 50 minutes and now you see at this resolution that we are actually not getting a continuous representation of the counter we get some weird step function here let's inspect this again run it again so what you're see now the step here is now one was productive he is our step equals one the other parameter I mean start and end is also closer together but the career is the same we just asked Prometheus between those two times stems please give us a value for every second now Prometheus doesn't have that because Prometheus only scrapes every 15 seconds but it prometheus still tries so we get a point in time like this timestamp one second one second one second one second after 15 seconds suddenly the value goes up because that's when a new measurement came in and you scrape so we are asking Prometheus for why higher resolution that it can actually deliver it still tries it's best so it gives us points in time but it's kind of yeah you add all of these times you get get the measurement from here and then you'll get the updated measurement this is how it works okay let's zoom out again because this kind of doesn't really make sense compactor or one-hour representation now what we wanted is not a straight line that goes up or almost straight we wanted a rate that's the whole topic of this talk so let's take the rate of this of this culture instead of a beautiful graph we get an error message so let's pause this error message it's a parse error so second order pausing parse a parse error expected type range vector in call to function great god instant vector oops so we we have already heard about instant query and range query an instant vector and the last permutation of those two things is now in the error message range vector so we have to understand what a range vector is and to understand that let's go to a white pot again so what is the rate our rate is essentially the slope of this curve here now a slope you could kind of look at it and think about it but if you think back what you learned in school about slope you could like draw those triangles and Delta B here and filter and then you can divide this by this and then you'll get this slope between those two points this is how you usually do it in even in school you can also take you like to also triangle it's no problem but you need at least two measurements to do this calculation and same is true for Prometheus because Prometheus just measures the value of the point here Prometheus can't measure the slope at this point committees only has those values so you have to kill Prometheus use this number of points at least you and then please do this Co calculation on those points in the range so that's what you need to do you have to provide a ranger point to Prometheus and how you do this that's called a range vector in prometheus so let's think about a range vector let's go back to the table view let's remove this right and actually also this job selector coming back to our incident vector how do we turn an instant vector into a range vector we just add a range in square brackets and now we hit enter and we will watch how this orange vector looks like fancy so here we get a bunch of points so I want to write or one minute since you have a 15 seconds for our level we will see kind of four points in that one million range and this is what we see we see four different measurements for every counter and we get a timestamp attached so this is different from our range query before where we essentially told Prometheus please give me a value for this time and this time and this time now Prometheus tells us what time it has done the measurement at those timestamps if you look there are almost precisely 15 seconds apart with like a bit of jitter in the millisecond range like here you see a 5 millisecond deviations this is all perfect and this has a 1 millisecond deviation Prometheus really tries hard to do this scrape really regularly but it doesn't have to so if it's a bit late a bit early we will still have a proper time stone here which tells us that the other interesting thing is those time stems are not around you would perhaps naively expect that Prometheus would scrape like 0 seconds after the minute it would scrape the target and then 15 seconds after the minute it would scrape the target again that would work with one target but usually are thousands of target on a single Prometheus server and if the Prometheus are asked to scrape all of those thousand targets at the same time that wouldn't wouldn't work well so Prometheus intentionally scatters the scrape and since these three metrics are coming from different targets they are a different phase in this within this game that's great but also they're all like half a different offset but within each metrics you have really regular almost precisely 15 second intervals this is how it works know check out how that she looks bianca scenes so this is now again an instant query it's the instant query endpoint but the what we do here is a range vector that has this range of 1 minute built into it so we use the info query call where the query contains a range vector kind of the opposite of what we did before for the graphing or we use the range query call where the query contains an instant vector so simple enough we want to close the circle and graph this right so how beautiful must range what you look like like this complex thing it's kind of a vector of vectors almost a matrix but with those different time steps attached this must be look super fancy graphic coming right so I hit this now we get an error message envelop expression type or H vector for range query must be scalar or instant vector interesting also like 10 minutes ago this would have been complete coverage for you now you kind of have all the understanding already to understand that so apparently a range vector is not good for graphing so it doesn't play well with the range query despite the superficial similarity in name so we we don't go down the rabbit hole of scalars here so you want an innocent vector to to graph okay so graphing wants an instant vector because like graphing graphical representation of those almost matrix like or even more complex than matrix things is hard but on the other hand we never ask really we never really wanted to graph a range vector we wanted to graph a rate and we kind of realize array needs a range vector to work so let's just now put all right around this range vector because what comes out of a rate is probably an instant vector again beautiful isn't it so this is our CPU usage now if you think about it right what does that actually mean if you write if you derived with the first derivative of a counter that counts time the counter counts CPU time and if you do this first derivative you get CPU time versus real time so how much seconds you're spent on the CPU per actual second and if this is one you spend one CPU second per actual second bit means you are hogging exactly one call with multi processing you can actually you can have one process actually multi-threading one process can can hold more than one call so you could have two or three CPU seconds per seconds usually it's kind of a fraction and our demo app is very humble with its CPU usage so it's like 4% CPU usage and 3% of you usage and 1% usage that is what you get here and it actually has a nice structure it goes up and down which is something we couldn't really see we as humans we were not able to see those tiny variations in slope in that seemingly straight line before so that rate function is really great it gives us a lot of information you would not have seen otherwise of course what do you want to see now is how is this curate so it's a range query with a step with start and end but the query inside it has a rate around the range vector but at least contains the range vector a range vector with a 1-minute range while our step size is 14 seconds in this case ok so now we have completed the circle we use a range query call that the query contains a range vector range duration and the step here the step duration are they seem independent for now but talking about that why have I chosen one minute here if you have done a bit of Prometheus before you probably heard that this is kind of an average I mean we've also looked at how the rate is calculated and needs at least two points for what effectively happens here is that you average your rate over one minute so now you might say I don't want to average over one minute I want to see like a one second average I want like much higher resolution sadly you get an empty here result but you already know we talked about it you need at least two data points and there are 15 seconds apart so let's at least use the range of 15 seconds which sadly doesn't give us any result either so let's double that let's go to 30 seconds twice this cracker novella and now we get something that looks very similar to my 1-minute graphing before we should we should understand that better so here is our counter if I take a range of one second most of the time one second will be a tiny bit here so we will not hit even a singular measurement if we are lucky we hit the single measurement but a single measurement doesn't help us because we need at least two data points using fifteen seconds it's great on the wrong most of the time we'll hit a single measurement again pretty safely we hit a single measurement but that's not enough if we are super lucky we could hit two points sometimes during demoing this that actually works you see a single point somewhere on the graph because somewhere we were lucky enough to hit this but that's of course not good enough we cannot rely on luck it's an interesting insight here that this range in prometheus is a closed interval it will include points that are exactly on the boundary on both sides that's from my perspective a bit weird decision but purpose of future prometheus Prometheus 3 might change this and do like half closed half open it a lot or something but for now this is the case if you are exactly on the boundary on both sides you get both poles but all of that doesn't help us with our range theory so what we actually want to get is reliably at least two points and that's for the 30 seconds that I just graph come in so here's 30 seconds so this should give us even if I move it around I pretty reliably get two points I even could go here and be lucky and have three points which is still good it doesn't really change a lot it's not harmful however Prometheus can sometimes skip this grape some hiccup network hiccup in Prometheus whatever a data loss I don't know it's it's rare but but not rare enough to know to ignore it you might not have this data point and then you're back to only one point this is where you come up and say okay if you want to defend against this weird corner case let's just take three times of a scrap-metal so let's do something like so with this we should usually have three or if we are lucky I kind of try to hit the lucky point here we have four points in our range and then we are super robust against losing a single script so three times seems to be enough right three times scraper level seems a pretty good rule but we have also seen that sometimes Prometheus scrapes but a few milliseconds too late or too early so let's imagine this point was a few milliseconds too early and this one was a few milliseconds too late and this one didn't work at all then we are back to just one point in our range this is pretty rare to be honest so 3x usually really really works but the Y's have said he should be even more robust so let's let's take 4x as the usual usual so you start here and then you take a whole minute so usually 4x kravinoff one minute usually would see four points in that range if you're super lucky five if you miss one and if you are a bit moved around because of jitter Yorick's Greg you're still good so and if you go longer than 4x you can do that if you're actually interested in a moving average for a longer time but you don't have to do it to be safe that you get enough points if you're now got only one point something is really wrong and it's probably legitimate to only have one point I mean to to see that something is wrong and lose data okay so 4x the scrap interval that's what the Y's recommend and then should you should all do this with 15 seconds it's a minute very conveniently if here for one minute scrapping level would be 4 min it you were probably around up to 5 minute because that's convenient again and it doesn't really matter for maybe 5 minutes whatever all right let's look at this in qivana finally I opened a new panel for the purposes of this demo with the shiny new panel editor which was already talked about in in the qivana 7 talk last week I inserted this query I plot over 6 hours and what we see is pretty much what we expected right so it looks small as the same but your father has why fancier features than the pretty minimal UI in Prometheus and of course in the end you want to build beautiful dashboards we will check out a few nice new Khurana 7 features the one thing that I really really love is the query inspector because now you don't have to go to the developer tools of you browser you can just look at the query here and go fauna itself will tell me what it sent to Prometheus and we see everything expected it's a range query the query inside is our range lecturer it has a start and an end and a step size with it which is 20 seconds whysa 20 seconds graph on our same as the Prometheus UI basically tries to give you one data point per pixel more or less because that makes sense requesting more would be something that you couldn't even render on the screen and requesting less would not make full use of your screen resolution so let's zoom in let's zoom in to 50 minutes and let's look at the cure inspector again because here we will see a difference if we had really liked blow time range high resolution grass Prometheus you I would ask Prometheus in this weird way of talking to itself for a step 1 Prometheus doesn't really like that if it has a 15 second scribe level or it'll give you a lot of the same points for some reason kevanna isn't doing that per fauna is only going to step 15 even if he goes super high risk in here it's only one it's it has an interval you can also see it here by the way is a nice you panel here new information 15 seconds I can also showcase the display options here the course I want to switch on the points here you can see the actual data points which is usually kind of one data point per pixel here it's definitely less right so why is that the case somehow profound our nose it doesn't really make a lot of sense to us Prometheus for more than one data point per 15 seconds there is no magic involved sadly in ideal Prometheus server could answer you that question you could ask the Prometheus server what's your scribe interval for this metric in this timeframe which could actually like different metrics kind of different scrappy level these carbonyl could change over time because somebody reconfigured the Prometheus server it's actually a hard question to answer and Prometheus currently cannot answer that question I said the sentence a lot perhaps a future version of prometheus is three or four or five we'll do that right now we have to work around that and the worker rounding of fauna is that when you create a Prometheus data source you see a little input field it's correct in the row and I told you you should have one scrap level for your organization and this one uniform hopefully mostly uniforms credit level that's what you feel in there and this is what I did here I created this Prometheus data source I'm using now with 15 seconds configured scribe interval and that's why kevanna will never ask for less than a data for more than one data point in every 15 seconds so this step is always 15 seconds if you want to override that because you have this one metric here that has a higher scribe interval shiny new query options I mean that was there before was just not as nicely explained here you get tool tips and here you could say okay I know I want five seconds here and then you see the resolution has increased can show points again oh yeah now we have point density quite high and final proof Curie inspector refresh this step is now five seconds okay you can do that this is for the whole panel I'll remove this again because we don't want it here you can even have an override per query which is mean step here so for some reason the main step only exists for the Prometheus data source and that's why it uses the step terminology here which is the same as the query parameter well it uses the term interval above here because these query options also work with other data sources so interval is a coroner concept that spends all data sources and it translates into step in the prometheus world because this is for all profile data sources we call it internal because this is just for Prometheus we've already call it step but it has the same effect here just an override only for this this query you could add more curious to this panel and use a different meaning step here alright so we zoomed in we saw that crow fauna tries to be a reasonable about the amount of data at request now let's do the opposite let's zoom out six hours was what we had before interval is 20 seconds let's do a whole day in dollars a minute let's you go B let's do a whole week in there our last 10 minutes expect a query because now we get got really used to that so the step is 600 seconds which is 10 minute but of course our range in the range vector is still 1 minute here is that a problem like I told you those two durations in the range theory what's the step and the iteration the range vector they are kind of independent what's actually the impact of having them vastly different that's a question we have to answer on the whiteboard I actually have to delete that let me check if I've forgotten anything don't feel good so I'll do this so what I now graph here will be already the right other plot the I'll actually also change the time scale here that's two minutes now one minute two three four five minutes six seven eight nine ten minutes so now the actual let's say this is how would like look like if he not it's kind of its kind of the real the real CPU usage but if he actually plotted this with some on minion rate we would see like one minute moving average so it could actually look like this now if we plot this expression with ten minute step size it depends where our step ends on what most likely I think it will let's say there's one data point here then the next one will be at 11 minutes this will always be a one-minute moving average right so it looks at this here so in our graph our long seven day graph we will see this data point we will see this data point but we will completely miss this huge peak of CPU usage you could say that's fine it's a lower resolution graph after all but somehow if it's such a huge peak here we want to see it in our measurement even if it's like averaged over let's say 10 minutes it will not be such an impressive feat we could still see there so people could zoom in to see how the peak looks at higher resolution so what we actually want we want something look back from here for 10 minutes and here we want to do 10 minutes again so what do you want in this we kind of want the range here to be the same at the step size that often makes a lot of sense now let's try this out in Kahana it's easier right we just put 10 here now I'll be careful how this looks like our graph has changed a bit like it's now less noisy which is before we kind of randomly picked one minute rate expressions now we take a 10 minute one so it should be a bit smoother and now we could I mean there's nothing interesting happening but we it could be that we see a peak here and then we could zoom in to this peak to find out what's going on there okay so we understand how this works it's kind of the problem solved we just set this to 10 minute if our step size is 10 minutes we're all good and we're done right talk is done 40 minutes great 10 minutes for Q&A but no we have to we have to think about this a bit more because usually a dashboard is not static you might have static wall screens where people look at from the distance but what you actually want is a dashboard that people interact with and one of the most typical interactionist you zoom in so now i zoom to let's say a day that's 24 hours now our interval is 1 minute but the range here is still 10 minutes which is really unfortunate because now I am kind of averaging over such a long time and I see a artificially smoother graph here I actually want to zoom in to see all the little high-resolution Peaks so I'm now have to manually go here and put this up to one minute which you don't really want a dashboard user to go through this is what you want to see right so and here we get the matching of CRO fauna her father has a feature for that you can it's already here in the tab completion you can put the interval into it then this is something Prometheus wouldn't understand but her father does it for you carvanha inserts the step size in Prometheus in in a syntax that promises can understand here into the query before it sends out to do Prometheus and now we can do this like we can go to seven days I will run the query inspector to see what's happening step for 600 seconds as before but now you see 10 minutes here magically 10 minutes appear here and the query now we go to one name and inspector Curie again step size is 60 seconds and magically it's 1 minute here perfect right we are done now the truck is done and we still have 8 minutes for Q&A no not quite because let's go to 6 hours that doesn't look good something has happened to our graph it's broken like broken literally broken if I go to 3 hours it's gone so what's going on now are there's actually one data point it's left but otherwise everything is gone let's refresh the query inspector step is now 15 seconds yeah we know it will go down to 15 seconds because that's the configured scrap normal but now you see the range here's also 15 seconds of course and that's not the 4x that the Y is told us to use so for this query you actually I mean you already know the remedy here right you want to set this to a higher value want to set the interval to 1 minute because if you want to use a 4x right level in your you're right your range vector that you feed to the rate function actually a resolution of less than a minute doesn't really make a lot of sense and if you put this in here we are we are good again so we get we kind of have reduced our resolution here but now the interval is at least one minute so this magically filled in individual also be one minute so finally we are good this is what you want your father has a feature for that great right so now we have seven minutes for Q&A but this is not not everything I have to have to cover something else which is too bad the other thing here is a problem here this is my editor I I should have had this - the Prometheus server but I I fail to do so if you have a lot of processes in your organization then running those graphic Furies if you want to sum up a lot of processes might be super slow and if you have our own use Prometheus a bit in that situation you know the solution is a recording a wall so you can put this rule into a rules well so you can sum up let's say this out the instance level so every job whatever gets all the instances summed up so this would be a perfectly valid query you take one minute here because that's the smallest interval you can go with and then you call it job process of view seconds right one minute that's a convention how you name it you get rid of the total and you put some hints into the the recording your real name - to understand what the rule is actually about so then you can use this pre-calculated rule and we were like in your graph we know I can't demo this now because I didn't put the rule on to my Prometheus server in time but you could just graph this right then you could even do it this election as before like namespace equals T and T and as demo I will totally work you can graph it it's alright you get the aggregated value it's super fast but the one minute is now sadly hard-coded here so there is no magic that profounder can do to change this and and create the rule behind the scenes and historically refill it so you are back at this problem that you can't do and dynamic range for your record but there's workaround it's mathematically a bit and clean but you can do something like this average over time then you can put an injector here with your interval and then this will work again it's it's it has I mean I could talk half no about this what this is mathematically a bit impure but it works good enough I would say so this way even if you're recording your you can still do dynamic intervals because you have just this one minute right but you can add it together over in a dynamic interval um you get almost the same effect for that so that's cool right so we have four minutes left for Q&A but but I have one more thing to tell you go to bed go back to my whiteboard there's one more problem and sadly cofounded doesn't have a feature for that hand sanitizer report these dates but also perfect white cleaners plotting our raw counter and now we do a very specific thing count that looks like this it has a spike of CPU usage and then it flattens again and here we are back to our 15 second resolution so what Prometheus's space is this this this and now if we are unlucky with our one minute perfect reiteration we might see this one with four points and we might see this one with four points if you have this unlock I mean it's rare but it happens then we see 0 almost 0 CPU usage here almost zero CPU usage here and we have missed this jump if you just move this all by one it's all good again but if we have this especially at your regular pattern every minute you have a little bump interview usage and your alignment is unlucky enough you might completely miss that that's kind of a problem and the easiest solution is to actually make them overlap by one sprite interval move them together so you kind of want Kafala to insert dynamically the interval plus ones pretinho and your honor doesn't have a feature for that and that's the reason why I filed a feature request I call it X interval here but that's just a working title that's essentially give me an interval that is slightly larger than the one that I've used before but once grab interval that's an interesting thing so I we are now out of time so I will not go into detail here but you can totally open that issue in github and follow along the discussion so this is David and talkable they are kind of already half convinced that we need this so they will something happening that co-founder will even have a feature for that but with that I'll finish this if you are totally confused now I understand the I'll be all it will be in the slack channel now to answer all your questions thanks for listening but

Channel: Grafana

Views: 30,913

Keywords: Grafana, Monitoring

Id: 09bR9kJczKM

Length: 49min 54sec (2994 seconds)

Published: Wed May 27 2020