InfluxDB + Telegraf Operator Easy Kubernetes Monitoring

hello everyone as caitlyn mentioned today we're going to be explaining in influx db together with telegraph operator and how to use them to monitor kubernetes workloads we're going to be showing some examples a lot of these are based on what's in telegraph operator repositories maybe we'll start by introducing myself so my name is fuji kothian i'm a software engineer at influx data i am one of the people that contribute to telegraph operator from from our company and i'm going to be the one showing it showing telegraph operator today cool so i'll go ahead and introduce myself so i'm pat gone i'm an engineering manager here at influx data um i manage the deployments team um and we are responsible for all the plumbing that is um in place this whole cid ci cd pipeline um for getting um for our cloud 2 sas offering so um that's we'll kind of focus our talk in the space that we know which is kubernetes and um influx data so first i just really want to say so influx data is the remote first company behind influx db so i think you most of you are all know probably more than i do about how to use our product but um i'll give a little bit of an overview and don't hesitate to ask questions um along the way and we'll get to them at the end so it's a so influx db is the platform for building time series applications um and i wrote all these really good words and now i'm having to read them and really at the heart of it it's an open source um time series data so it's purpose optimized for time series data whether that is sensors or like um i have one of those like doorbells where you can see see the person so it's like um there's time data there so wherever there's um time based data influx db is is a perfect uh platform for you to develop application applications around that data so you can start from the ui or you can skip right to it and you can use the raw code and the apis and we've got um we've got apis and client libraries in in several um of the most popular programming languages so telegraph if you're not already familiar with telegraph if you have say that ring device over there and you want to get your data somewhere telegraph has the input and output plugins to allow you to you know get your data from your device into a database of course my preference is if you put that into influx db but we've got um plugins for other other types of things it's it um it's an open source agent um and it's it i think has a really healthy open source community and it's maintained by influx db um and they have like they have a lot of different plugins there's over 300 different plugins that allow you to basically like i said like manipulate your data on the way in um get your data in and help you manipulate your data on the way out it's really a really powerful tool so today we're going to focus on talking about it in this space of kubernetes but um i know that there were several talks i think at influx days north america where they actually talked about telegraph i think there was a beginner session and i think some other things so check it out because it's a really powerful tool so now i'm going to tell you a little bit about the telegraph operator and i wrote some notes ahead of time to prepare for this so the telegraph operator packages the operational aspects for deploying a telegraph agent on kubernetes so this is about having a kubernetes sidecar with a telegraph operator there it's it's um it's a sidecar container based on annotations and it provides the telegraph configuration to to scrape the exposed metrics all defined declaratively it allows you to do define common output destinations for all your metrics so you can send it to influx db or you can also um send it elsewhere and i'm going to pause there because i want to let voice finish setting the stage for his demo and i don't want to take it all so actually voice check's gonna take it from here and and uh and uh do a demo so but i'll let you also finish right so uh thank you think about so as you've mentioned telegraph operator is meant to be running alongside alongside kubernetes workloads and this is what we're going to focus on today let me could you stop sharing so i could start chilling on my end yes but i have to voice first i had to show this slide okay done that now i can stop sharing okay yes so i noticed that there is a question about apis for influx db so i'll just share this real belief brief and keep it open so we have a documentation about all of the apis there are also clients and i'll show it in a bit so it's well documented in the rest apis there is a querying language called flux and influx ql that could be used to get the data and writing the data is relatively simple but going back to telegraph operator what i'm going to do is i have a checked out copy of the source code of telegraph operator and it includes a lot of ways to run it using kind kind is kubernetes in docker which is a way to run a whole kubernetes cluster locally just using docker and that's what we use for a lot of testing of telegraph operator because it is a simple way to run things and also be able to do fancy things like building a custom build a custom build of the container image with telegraph operator and say loading it which is not something we will be doing today but it's also really useful in development and we'll just use the exact same setup that we use when we develop it there is a so what i did in advance because it takes around one two minutes i run a kind start make a kind start command which basically just creates a kind cluster on my on my computer and it's it deploys a few things but we're going to deploy if we're going to deploy in flight db version 2 because that's what we want to demo so i just deployed it this is the open source version of it and as soon as let me just put forward to it okay so let me see what's happening in my kind cluster okay so my influx db version 2 is running right now so i can do this so right now i just i have i created a fresh kubernetes cluster on my machine i deployed in flex db version 2 to it the open source version and i'm just going to bootstrap our cluster meaning that this is the same if i would deploy locally but i just want to have everything in my cluster and i'll just imagine so when i first deploy it i'm going to set up an organization and everything for the influx db itself because this is where we want to get the metrics in and also touching on the question of how to get the data in we also have a ui that shows a way how to write data from a lot of places so say if you're a golang developer it'll give you a ready to use snippets obviously you would want to replace the token and some other things with flux over time and parameterize this but this is a really good way to get started with just putting data in in fact but anyway right now what i really want to do is in order to be able to write my organization i need this token so right now we're just going to grab this and then we can get back to explaining and configuring telegraph operator so i haven't deployed the operator yet because there's one additional thing they want to do so telegraph operator has a concept of classes which are classes of applications or classes of metrics that we're gathering and this basically maps to specific sets of telegraph configurations and one of the things that we should be setting in here is how would my application write the data to wherever i want this because trigger of operator is meant to be generic so we should be pointing it at specific outputs which is part of standard telegraph configuration so what i'm going to do right now is i'm going to tell it okay let's just let's just also write it to my influx db v2 and now [Music] what i'm telling it is in my cluster there is an influx db2 service which is what we were just talking to in the browser in the influx db2 namespace and the part that listens on is 8086 which is the default port i'm just going to tell it my organization is demos i just entered it packet is demo and now i'm just going to copy my token and because it's my local machine i'm fine sharing it because i'll just keep the class later on i will also copy it to believe we will be using the default one as well so i'll just click this so right now i'm configuring the classes meaning that when when we want to monitor some workloads we will need to specify what the class of that workflow is or it will be using the default class if it's not specified so we have created app default and i believe infra i think we will not be using all of them but but all of them also specified that the data should be going to the new influx dbv tool that i've just created this is a standard you're just missing your equal sign on token yes okay yes that would be and it bothered me it didn't just bother me it bothered someone else too i was watching it thank you okay thank you so much for noticing it so right now i'm going to go back to my terminal and i'm just going to deploy it examples classes okay so the example is already committed and the example shows how to use it with things like dbd1 because from development perspective we keep on using the version one for that which is something we should improve but it's just it's just telegraph operator has been created when everyone was there was that so right now i deployed my classes my configuration i can update it in the future and there is live reload so i can change it but right now we deploy that so what i'm going to do next is i'm going to deploy uh telegraph operator and it can be deployed in multiple ways we have the devil file which basically is meant to be used for local development but because i'm doing this in kind i'm just going to use reuse it because it also has hard coded certificates so it's not really production ready but it's enough to work in kind we also have a hand charge [Music] okay so uh right now it's it's on github okay telegraph operator [Music] yes this is what i was looking for for that so we have a telegraph operator hand chart as well that's that's available if you just install our influx data headshots source and then and then you can install it or just use upgrade install which will either install it or upgrade it depending on whether it's already installed or not and this is also this is a this is a preferred way to get to getting production environments running but because we're using kind and because all of the examples are based on on this i'm just going to follow this and not not to the handshake based installation but i can right now i can just go and see what's running in my cluster so you can see that telegraph operator is spanning it's ready to handle to handle the new deployments coming in and adding the telegraph sidecars so now the way telegraph operator works is maybe i'll just open one of the deployments to explain it this is just a definition a very simple definition of how to run redis it is a stateful set but it doesn't really even include volumes in real life this would be a more complex stateful set but this is an example of how to use telegraph operator uh to to monitor things the way telegraph operator works is it for each part that gets created it checks the annotations and if there is a telegraph operator annotation in it it will inject the side current so right now we can see there is just a one container called redis that's just using the default redis image but we can also see that we have the annotation telling telegraph operator that it should be conducting localhost and the standard redisport and using the redis plugin this is this is one of the plugins that pat mentioned and maybe i'll explain this a bit more so actually so voice check before you go into that i was hoping because i don't think we actually showed people the repo oh that's a really good point of operator this is all this is all code you guys can get to kinda yeah is open source and and it also includes a an extensive freebie on how to get started with development with deploying points to the helm chart and this is so if you want to rerun what i'm showing today i think the easiest way is to clone it and i'm basically using a lot of the make targets and just applying some of the things that we also mentioned in the documentation because you can see that we're deploying this we're just playing this through github urls rather than locally but yes the the repository is is on github it is open source you can you can clone it you can run the sync yeah and you're working within it right now i just realized we didn't like that thank you so much for this that is a very good point because i am i am because i am so into the repository sometimes forget to explain things that may be like maybe my day-to-day things but for a lot of people they may be new so it's good to mention it so going back to the configurations i may have skipped explaining some of these things so the way telegraph operator works it combines the output the telegraph configuration that the telegraph will be reading from multiple sources one of the sources the classes that i mentioned which is just a vanilla kubernetes secret with the definitions of all the classes and usually this would be including outputs or some of the tags or some or some of the general things that would be applied to all the metrics related to this let's call this this class of applications that want to monitoring so in this case we added the output to it which which means that everything with the app uh class will be writing to our influx db v2 we also make it output to standard out so it's easier to debug and we have it use global tags that will show in a while in the ui basically type is set to up and then host name and node name would be the name of the host and the nodes that the telegraph is running on and now if we take a look at the redis deployment we are adding some other pieces of telegraph configuration so one of this is we're adding inputs dot redis which means use the redis input plugin and previously we were using the influx db v2 out plugin so we're telling telegraph talk to ready's on this on this port get some of its standard metrics and send them out to influx dbv2 on this specific url we could also tell it like send it to my cloud instance send it to some of the on-prem instances of intellect db or maybe send it to one of the very very large set about it like it's just that we support like i know it could be sending it directly to kafka or some or some other output plug into this part or write it to a file but basically we tell it this is the input these are the outputs that they that are in the secret in the classes and then they get concatenated so my redis definition tells this is how you should gather metrics for my red is my classes that i disable you should be writing this and it also tells it by the way this is the app class meaning that whatever i put in my app class in the classes definition is where the data goes it we can also specify settings for memory requests and limits for the telegraph sidecar this one is invalid and it will be ignored this is more of a development test case but the limit the cpu limit will be will be will be set to the telegraph sector so anyway that's that and let's just let's just go ahead and deploy this so this was uh examples ready to believe example thread is okay so now if we go back to watch we can see that we only specified one container one container for the for within the pot spec we can see it's actually running two containers if we do just click port if we go ahead and describe it let me just do it this way we'll see that there is the redis container we defined there's also the telegraph container that was injected by the telegraph operator and we can see that the cpu limit is set to 1 to 750 milli so 0.75 of a single cpu car we can see it's mounting at the telegraph using and we can see it below using a secret that was generated by telegraph operator so basically when the pod was about to be created telegraph operator combined the whole telegraph configuration put it in that secret and started running telegraph operator and it also tells told telegraph operator that it should be monitoring that configuration to allow hot reloading which i'll explain in a bit because that is a an interesting feature of telegraph operator but anyway at that point i believe the pods are already running so what we could also do is right now i'm just telling er i'm asking for actually let's use something more visual we're going to run a tool called k9s which is a nice uh console based ui for a lot of things kubernetes related and it's much better than what i was doing before that so i think that's going to be more visible so this is my board with the sidecar included i can take a look at the logs of this telegraph sidecar and i can see that because we told it also logs lock all the metrics to standard out we can see that we already have the metrics in here and the metrics are in line protocol which is what influx db fields are on top of but this is basically just because we told telegraph to write the standards output and we didn't tell it to use any other protocol so it's just writing it inline protocol but going back to this we can see it's running so now i can go back to my inflex db and i can see that i have a lot of my reddish data so i could see oh actually i didn't even know what to look at but let's say max clients and i see 10k that my max client is configured to 10k uh i could probably also see a lot of other matches but because there's not really nothing happening i can also just have it show all the metrics so we see some metrics changed over time um but not a lot of them so we can see that there's there are a lot of metrics that better i can just show the update and you can see that there's a lot of data that we have um and we can see that the telegraph operator is reporting this so let's say okay let's try to do something more practical like i would want to monitor how much memory is being used so i already have it and for any other workload that's that i will be deploying in my cluster telegraph operator will will automatically be injecting that but we can also see that the tab type is set to up and this right now influx dbey works in this way that i'm right now i can build out a query using just the ui and i can filter by all of the tags that we are setting and the type equals up was set when we were creating the redis deployment so i can with this okay what would only be so say let's go back let's remove this a bit i could say that i will start by just filtering data coming from my applications and then i can go back and say okay and now let's take a look at all the fields they have right so we for example we have another another thing that we could deploy which is which is an example of deploying nginx in this case it's also an interesting example because previously we've read this we had we were specifying the raw telegraph configuration but we could but if the application is already exposing metrics in the in the prometus format so if you have a so if if you if your application already exposes metrics using the prometheus standard you can say go scrape prometheus matrix on these ports or on one product which is saying just scrapeboard 8080 and then this is the path to go to scrape it every five seconds and the protocol is http and the last annotation we have here is also scrape the intro you get the internal telegraph metrics so once i deploy that apply for examples emonset this will deploy this will deploy the nginx steam on you can see it's being deployed we can see it's slowly running now if i go to the logs right it's mentioning that it can't really scrape locks because nginx is not listening on those ports and also our nginx is not running any application that will expose the metrics but we can see because we also enable the internal metrics we can see some basic metrics that telegraph is reporting so right now if we go back if we go back we can see the internal data in here we can see gather errors we can see a lot of other data in here that's slowly being gathered and based on that we should be able to build a lot of a lot of dashboards out so let me just let me just show a quick example of that while this is this is not exactly what this is not exactly regular for the operator specific but let's just show how i could basically just go and say okay i just want to see i used memory for babies right and then i can just save it and i have my dashboard and this that would be an easy way to just move from having my workload in the cluster to basically being able to visualize it in flex db so and we can see that the methods if i go back to the telegraph plugin that does it sorted to the logs of the telegraph we can see the data keeps on coming in so one other thing that i wanted to mention or show is which is really interesting is as i mentioned we also support reloading of configuration so i could just start adding a new tag let's say new type equals application and for the other one we could say default new default okay so once i deploy this uh classes so the only thing i'm deploying right now is i am changing a secret that telegraph operator is using but if we take a look at the logs and this should take around one minute for telegraph operator to notice this and we want logs from all the time in around one minute because this is how much it takes for kubernetes to re to reload the secret mounted inside the container um in around one minute telegraph operator will pick it up and we'll say okay i note i see that the conduct the configuration has changed so it's going to reload it but it's also going to check what are the telegraph ciphers that i've created what are the secrets they've created and go in and update them as well and we should start seeing that the new data in a few minutes this is this is really useful and this is something we use a lot at influx data and we started using the pod reload as well recently and that was one of the things we really really wanted because whenever any configuration changes we don't really want to restart the whole workload or try to manually restart the telegraph operators what we would really want is and we have it right now is the ability that once we change the settings telegraph operators would be smart enough to detect that uh and then decide which are the things that really need to be updated so you can see that it decided that we don't really need to update the secret for the nginx we can see that it decided let's not update the secret from engine x d1 m and hx2 because nothing changed in there because we didn't change the basic class but let's update the secret for redis because the class in there was up so if i go back and this is the mistake i've made if i go back and also add this class and then tags um [Music] basic basic up if i do that then we should see around one minute we should see 39s we should see it reload again and then in a while we should let's take a look at all the logs we should see another log message thing that's updated but right now i can go back and if i try to filter on if you hold it type i'll be called one second um [Music] okay called it back yes so maybe the change wasn't yet it wasn't reloaded yet on the radius level we should take a look at that in a bit but so vlog just to summarize what you're doing you're basically you're now like kind of you've got the telegraph operator and your local kind cluster and now you're you're adding more and more things for it to monitor right correct so i so i think we've got to summarize right so maybe that's a good point i'll try to summarize what what i've been doing and what's happening what i'm trying to show right now so we deployed we had a kubernetes cluster that did not have any workloads in it which would often be the case or when we would have some workloads and then what's what we did is we deployed telegraph operator which would start injecting the telegraph side cards to any pots that were there any new pot that were created so for any new workloads or any workloads that would have the new annotations added the telegraph sidecars would be injected to those and because changing the annotations on the pod would create a would have would mean that the pod gets recreated so whenever we will be adding the the annotations then then the new post will get created and they would start getting the telegraph side cards included one other thing that i tried to show and maybe i should have done a better job explaining these things is so that would be the day one of operations you would deploy telegraph operator you would add annotations to your workloads and you would start seeing the data inside your influx db or any other or any other place where you would be loading the data to but as you move into day two of operations sometime you need to change some of the settings and this is an important aspect of this or sometimes you need to let's say rotate your tokens which i assume would not be manual it would be some automated process but that would be something that should be happening so let's say you generate a new token you have an automated process go and update it in the classes definitions and then after a while let's say after 24 hours your gdp all token and expect everybody to be using the new token if the hot reload would not be in place this means that all of the workloads would have to be restarted or at least the telegraph cycles have to be restarted with the hot reload functionality in place telegraph operator and then the telegraph sidecar would take care of this automatically and the data operations are much easier with the with this what reload functionality being available the hot reload function you now you boycek you added the hot reload functionality what like was that like two or three months ago or maybe it's a little bit longer now it was definitely this year i don't really really remember one exactly it's all a blur but that came that was exactly what happened as part of our internal use cases so this was a this was a pain point for a lot of things we're doing internally which is that in some cases we just want to change some settings we just want to so for example we want to change the frequency at which we get some of the data because we want to increase or decrease the amount of data we're storing or we we or we want to move some of the data to other places like we may be monitoring some data in our internal systems but we also want to be moving some of the data to do like production systems because we want these to be to be in the same place that our customers use it so we can also use that for uh for it it was a huge game because you'd go and you'd like an engineer would change like you said the frequency and then the next question would be like they'd go and look they're like it's not it hasn't changed what's going on so having that hot reload like adding that that functionality which was added earlier this year fantastic because and also i wanted to say as you mentioned we're using this in-house so um yeah it was definitely kind of a frustration point when people would make a change and then they'd look for the change and it would take a little bit to you know basically it would have to wait i'm gonna say voice check until it naturally like got restarted which is kind of a funny use of the word naturally but let's just ignore that right because whenever the actual application codes changes then we will still restart and see the changes but the thing is then it could be between a day and a week depending on how often the code changes and with this this is a matter of minutes and but like you said like this was a big thing for us and and this is a huge huge huge improvement for us so going back to to the dashboard and uh and the data we have in here so right now if i reload this right now i can see the new type so the field i added and i did not go and restart anything so this is like the thing we talked about it's difficult to to show it because it takes a few minutes for for all the kubernetes mechanics to kick in and and change the underlying secrets and then distributing the underlying uh watch mechanism to notice this but in the kubernetes reality waiting a few minutes for these chains to get deployed to like hundreds or thousands of telegraph sidecars this is very acceptable as opposed to the thing we mentioned which being it would be a matter of days or weeks before the state that is visible so right now i can go in here and see my internal metrics as well so this is so this is a huge improvement and this is this is i think a really nice feature of telegraph operator um and like i said we could we could for example one other thing we wanted to show because if we were to see the logs of say redis and the operator here we would not be able to find the the message that the logs were restarted because we keep seeing this this data flowing in but if i were to say remove the outputs file and deploy that and then wait a few minutes while we perhaps do something else i'll also see that now i no longer will see my uh my data being written to standard which is which is also a pretty interesting feature why would someone use that feature voice what would they use that one for i mean so i think the standard double is more like a debugging tool so the reason why we include it is we included when people develop because then you don't have to go to uh okay so the data is still going where it's it's still growing right okay yes so i mean one of the nice things about telegraph is we is we can put it in multiple outputs right so we have telegraph operator has extensive document the telegraph itself has it has pretty good documentation oh i wanted the documentation yes so we have a lot of different types of plugins and we have a good documentation so basically for outputs i could be writing it to a ton of things we were just using a file okay so let's just we were just using the file and influx db so we were just using this plugin and then we can see it's readme file and we were also using the influx db v2 you were using v1 but that's kind of interesting but basically we could configure a lot of things like he said it has outputs we could be filtering things at the output level as you mentioned it's pretty powerful to be able to do that we could be configuring a lot of things i think the nice thing of telegraph itself is that if it can't write to one of the remaining outputs not inputs if for some reason it can try to can treat or something that isn't really working at the time it's going to retry it and it's going to buffer the data and it's going to be to be smart about how much data it can buffer before it throws away all the data and all of that is configurable which is a really nice thing as well because we were toggling inputs and outputs and telegraph would just be automatically disabling the ones it has but technically i would be able to disable one of the outputs and let's say if it wasn't able to write to another output it would be smart enough to realize this is the same output i'm just going to keep on raising the same buffer so we could and i mean we just we just did change the configuration we can see that right now it was just it just reloaded and it stopped there we go yeah it just stopped writing outputs but the nice thing is like we can do all of these and like i said in kubernetes world where sometimes we don't want to restart like i don't know we have deployments where we have our deployment stakeholder sets and other types of workflows but we have workflows for a single type of a micro service we would have hundreds of pods and then restarting all of them just because we want to tweak a single setting isn't that great whereas here we could just apply a small change reload it and the whole system will just pick it up and none of the things we started like telegraph itself the site here was not even restart it was just entirely within telegraph so we spent a lot of time inside the company across teams to get all of this working and i think i think in general telegraph is a really nice tool to monitor kubernetes because like i've shown that we support like it's really simple to support both just using prometheus metrics and scraping them and they end up going into kubernetes which is what we use a lot ourselves internally because a lot of languages just make it natural to expose the metrics in this format so it really needs that we could just specify the port of parts the path and telegraph would be integral of operator will just generate configs out of that but also if you know that you're running something that telegraph knows how to scrape then you can just use one of them many many many plugins and you just inject this small snippet and telegraph operator will blow it together with what it should be output to and you can also have some additional settings in them in the classes so it is really easy to manage and from our experience we have large clusters we have dozens and dozens of those classes we have to manage it is really useful to be able to do that one of the community contributed features that i think will be showing in the next release that's happening really soon and i'm really excited about is ability to also reference other conflict maps or the secrets and be able to be able to reference some of the metadata so if i would want to get some of the kubernetes metadata i can expose it as an environment variable in the in the annotation i believe the annotation is something like and field ref and i can say that this then my variable name is like namespace name and it will just be metadata space and like i said this is so what is the new feature that's coming vojtec ability to reference various things so in this case i'm referencing a kubernetes field meaning that this will that this will tell my pod what the namespace is or the name of the pod is so this would be like pod name i could also expose that this is useful in some cases but we really want to tie this back to some of the fields but i could also get the like the ip address of the part which i could then use to filter things but i could also do something like secret key ref and like token and i could say for my secret which would be my token secret dot this would be the key name dot let's say token right so with that i could i could for example well this would be the wrong example but let's take token equals token with that i could have my token managed by a secret i could reference that and then telegraph and then kubernetes would load this as an environment variable for the telegraph cycle and i could use it in the configuration so i wouldn't have to inject it in other places so this is useful if for example using other tools to manage the secrets or the secrets are just managed better application because then the director of state car would get it this is this is not where cut reload would work because of the way it's it's working because of the kubernetes internals maybe that's something we we could extend in the future but this is still a pretty nice piece of functionality because if for multiple reasons we have some data in some other secrets and we just want to reference it it's much easier than having to hardcode it in the annotation and you said this was a commun this is a community contribution that's like kind of in review and will be part of the next release of the telegraph yes i'm really hoping it will be and i'm really excited about that because every time we get contribution to telegraph telegraph operator and i think that's that's open source i think that's that's really nice sign that the people using the tool and that people are willing to spend their time extending it so we're trying our best to help whenever anybody contributes in any way even if someone just opens an issue like we've had people open issue that if they run it they then we forgot to create the name space and we were fixing those kind of things and that that's also great because this means someone took the time to to give it a try and if something was broken you also let us know so we could fix it for other people that's really cool the voice did you have anything more you wanted to to uh show to share today all right i think we're going to know i think caitlyn i think we're we're finishing up with our part of the show i mean awesome well you guys aren't done yet we're not done yet well thank you uh for that boy check i i know live demos are always fun um so i know boy check you already sort of answered this um but how does a newbie get his or her arms around apis i know you showed the document the docs link is there anything else that a community member can do to get some help or i think going to influx db so first thing is just getting onboarded in influx db i think the easiest option is to go to cloud influxdata.com and play with the sas offering because it's uh there's a free tier that provides most uh okay i guess my typing was um so basically uh now they're gonna see where i got my picture from quickly so basically just just sign up for influx db cloud which is the easiest way to do this or just run the open source version wherever you want like i i just run it in my beneath this cluster there is a container that you can just run there are binary that you can just grab and run it on like there's multiple ways to run inflixibility and then when you go to the ui there's a way to get started with most languages we also provide ways to get telegraph configurations because that is a slightly longer process but basically um this is a way there are multiple ways to get the data you can also write you can also directly use the api but i think we we try to do our best to just get people started with whatever it is that they need to do right so i could just say just on my system data and it's going to basically okay i'm going to basically generate a whole conflict for me and this is just a telegraph configuration i can save i can run telegraph on my machine and it's going to start writing data to finish tv well and i would like to so let me tell you what i do to go figure out anything on influx db i go and find blog posts from the fabulous anna east so like she has one it's like tldr influx db tech tips creating buckets with the influx db api like i am completely biased but i think her blog posts are fantastic for a newbie and then i think they're also really good for someone who is um not a newbie so i would i would go look for some of those influx db tech tips where i think she talks through using the api to do different how to use the api to do some different things and i think one other thing we're mentioning is influx cli it's a great tool to do a lot of things so anything from creating buckets to writing data reading data and it's possible via the cla there's also a way to explore the import of data and objects of things like dashboards so it's a really powerful tool and it's also easy to get it set up that's another way um i think also you already answered this but can you share this data locally or can you share this code for us to test it locally i'm assuming it's all in the repo yep i think that's where you get that read me for that for that that question inspired me to say oh yeah we forgot to point you in and the make file is also a good starting point because it provides an easy to use make targets like the kind start deploys in fact the b1 deploys a lot of things make paint test basically deploys most things and it's even deploying redis and showing you at the end that redis has the site that has the cypriot container that sleep is going away i just need to merge the pr just didn't have the time to do it today there's going to be a proper cube cattle wait so we will wait for the operator and not just assume 20 seconds is enough but there is a lot there are a lot of make targets that just make it super easy to start and you touched on this briefly but uh how are we how is influx data using the telegraph operator internally it sounds like it sort of was developed from a internal pain point as well so what it was it was developed i think for both internal external users but but when when we when we started deploying workloads and we were thinking about being able to handle our set of kubernetes clusters and large workloads we were just discussing how to do this how to get all the data and given that we already had telegraph as as a very successful and long and project with long history we wanted to use telegraph and we were just wondering how to do that and telegraph operator it's just a natural way of doing this so we use it we use it a lot for most of our workloads meaning that with one of the first things we deploy in our clusters telegraph operator which is obviously automated but that's one of the first things we deploy and then from all the workloads we have monitors we just add the same annotations i've shown they may be slightly more complex than the examples we're showing but it's still annotations that we use and for a lot of the code we write internally we just expose them as tremendous metrics or expose them in other ways so it all depends on what we're monitoring but we're trying to use the native input plugins internet the next telegraph negative input plugins that for so for things like redis we would just be having those plugins get the data from writers internally for things that expose metrics we get the most committees matrix promitius metrics it really depends on the use case but most of the things we deploy just has the annotations and and telegraph gets deployed automatically and you know just kind of curious what for both of you this is a question that i have for both of you what are you guys working on in the next six months that you're really excited about that you know the community will like or get excited about as well that is a good question so i i know that we i mean i think this we should go back to the to the the the why we're using this sidekick containers as opposed to demon sets because we do get this question a lot and i i'm actually surprised this question hasn't come up so uh one we deploy telegraph as a sidecar and this means that if we have lots of workloads then there is a lot of telegraph side containers and there are a lot of processes that could be solved by a demon set and so we chose to use the sidecars because we noticed that a telegraph report has is more successful of getting the data and not being able to buffer it if things ever go wrong temporarily so it's much more reliable if we monitor a single part but we're also trying to figure out ways to um to do something between managed between running telegraph cycle for each port and running it as a demonstration monitoring all the all the nodes these are all the posts in a specific node so just to explain briefly a demonstration something where there is a one pod where each kubernetes so breach dedicated vm or parameter machine depending on what the kubernetes is running and we're trying to figure out if there's a way to also handle workloads that don't really get a lot of metrics without injecting the telegraph as each individual as a cycle to each individual pod and i think that is an exciting challenge because maybe there could be some compromise like something's being monitored as the onset and something's being monitored as psychos but we don't really have a good answer to that yet so we're trying to trying to tackle this because that's one of the things that could be helpful for us internally as well and i'm sure a lot of people have this issue that a demon set so one all the pods is too much data together and then a cipher for every single part is too many resources being useful sometimes really small micro services that don't don't get involved a lot and what about you pat what else i mean you're nodding along so you clearly agree but anything else in terms of like from the perspective of the telegraph operator i think voice check i think covered it but just generally in general we're just going to continue to make influx db our cloud 2 sas product screaming fast and make it my team is good art is working to continue to make it so that our developers can deliver sweet sweet software to the users more quickly so that's what i'm always excited about awesome well thank you both um i i feel like there's gonna be lots of people checking out this webinar and they might come bug you in the community slack with follow-up questions

Channel: CNCF [Cloud Native Computing Foundation]

Views: 683

Keywords: influxdb, telegraf, kubernetes, cloud native, cncf

Id: cAOHppnbqMM

Length: 52min 27sec (3147 seconds)

Published: Wed Nov 17 2021