Kubernetes 101 - Episode 10 - Monitoring with Lens, Prometheus, and Grafana

good morning everybody today is the last episode of kubernetes 101 it's been a really fun journey with everybody over the last few months and be sure to watch to the end of today's episode for a special deal that i'll talk about then but today we're going to talk about monitoring kubernetes before we do that go ahead like all these episodes begin i enjoy one of the things i do enjoy a lot about these live streams is the fact that i can interact at least mildly with uh with people in live chat depending on how involved i am in the actual thing that's going on in the demonstration but seeing where everybody's from is always always great to see just because it's amazing the the global uh reach of something like kubernetes and this educational material can do um and it's also nice to see just where everybody else is from uh who's watching this and what's really surprised me comparing this series to the ansible series that i did earlier last year is there's so many more people from outside the us and europe compared to the other series so it's it's just interesting to see the growth of kubernetes around the globe anyways we'll get right into it you can see i was just bringing up my mini cube cluster here today we're going to talk about monitoring things in episode six we actually talked about monitoring logs and drupal a little bit i didn't get too deep into it then and i talked about elasticsearch and kibana and things i even had a video talking about the company elastic and some of the open source dishes and decisions they've made recently but we didn't get too deep into it then and we're not going to go deep into logging today we're going to talk about some other things um but some people would maybe begin their kubernetes journey talking about some of the things that i'll be talking about today especially the lens app i typically don't like learning new tools using an abstraction layer or a ui for it when i learned git i learned the command line first and then i added some some guise on top of it after i understood the basic concepts that's the way that i learn and i think a lot of people would learn the architecture better and learn how a tool actually works better if they use the primitives or the things like the command line utility or the the primary mode of interaction instead of throwing an abstraction there on top but that being said we are human and we have our monkey brains that can't really wrap themselves around something that doesn't have a ui layer or something that doesn't have a visualization to represent something that's more complicated and complex underneath so it is it is nice to have visualizations and it is nice to be able to browse a cluster using a tool instead of just thinking in your head oh i know it's the cubie cuddle whatever um memorizing hundreds of command line tools is something that we all have to do because of what we do in our jobs but it's not necessarily the most fun thing especially you know if there's more obscure things that you don't do all the time having a tool to help you with those is great so today we're first going to talk about the lens ide it's called an interactive development environment and i guess you could call it that i i generally think of it more as a cluster browser with some advanced features that let you interact with the brows interact with clusters and not an ide because i actually do my development work in a code editor or a real ide i don't i don't manage resources individually in kubernetes clusters that's that's not using infrastructure's code in my opinion that's managing infrastructure like you would in amazon's ui that's in the console that's okay to do technically but if you're really focusing on infrastructure's code you should be managing your infrastructure in code pushing that code up to like a git repository and then that's where it deploys from not clicking an edit button in your cluster and changing a resource that way the second thing we're going to talk about is what i call the de facto standard for all metrics monitoring and kubernetes there are so many different options and we've talked about the cncf landscape before but this is the the tool that is most often used to gather metrics about your kubernetes clusters so it'll be good to really know um to really know how it works and and how to integrate it with your cluster we're not going to go super deep remember this is kubernetes 101. i'm not going to talk about building custom metrics and in managing a prometheus cluster outside your kubernetes cluster or anything like that but we will talk about it and finally we're going to talk about grafana which is really the easiest open source way to manage alerting and monitoring dashboards for your cluster and to test these things out i actually have two clusters i set up two different clusters one is my mini cube cluster that you saw that i had up at the beginning of this episode and i also have if i go to cloud.lino.com i reset up the cluster that we had from episodes four five and six so i have this episode 10 cluster it's running drupal it has ingress it has a cert manager running on it and you can i'll show you that site it's still up at ep6.cube101.jeffcaroling.com and that's running i think drupal 9 9.1 or 9.0 or something like that anyway this is a drupal website that's running in this kubernetes cluster and and so i have those two clusters and the first thing that i want to do is take a look at what these clusters look like through lens and lens is a it's a good name i have to give them credit it's it's a good name for a tool that lets you see into a cluster um i'm a photographer also so i'm used to lenses and i know a good lens is worth its weight in gold in terms of giving you a good rendition of what you're trying to image and and lens is it at least it's my favorite way to see my kubernetes clusters um but and the real reason that something like lens is important and helpful is for most of us once you get finished learning a little bit about kubernetes and you start building a cluster you realize you might need two clusters a dev and a prod or you might need three clusters a dev to stage and prod or you might have like a cluster for one client a cluster for another client a cluster for another client you end up having a lot of clusters and qb cuddle is not really efficient for dealing with multiple clusters you can write some command line wrappers and there are some utilities out there for dealing with cluster context and things but it's just not super efficient and most of us don't want to sit there and configure command line utility to try to give us all those contacts because we already are dealing with the same thing in git repositories and things so it can be annoying to use cubie cuttle with multiple clusters and also another thing about cubic cuddle is it is a very complex command line interface there are so many different parts to it i often have to look up references for how to do things like creating a local proxy to a service and things like that you know there is good documentation in the man pages for it uh but it can be sometimes difficult to remember how to do something so lens lens makes both of those things very easy and a little history on lens first i'll go to the website here a little history about it is it was originally made by contenta that's spelled with a k but in early 2020 so just last year marantis who has been consuming a lot of different cool kubernetes tools they acquired contenta they didn't acquire lens yet so they actually did that late in 2020 but the good thing is that this tool is is managed under the mit license so like many other open source tools if they ever try pulling a an elastic and flipping the license on you it could be forked very easily and it would be because it's such a nice tool and and many people use it um so i don't think we're they're gonna pull the rug out from under us and change any licensing after acquiring this because it's such a good tool and uh i think it's kind of like one of those halo tools where it's like oh mirantis manages this oh they must be good or something but anyway so that mirantis owns the owns the the structure around lens but it's an open source tool that's free to use you can download it from this website and install it on uh linda windows windows mac or linux but on on my on my mac i usually manage things with homebrew so when i wanted to install it i just ran brew install cask lens and that popped it onto my computer it can also be installed with snap if you love snap i don't love snap if you're on linux but anyway once you have it installed now you can just open it and it will come up and ask you to add clusters and just like with cubie cuttle on the command line the way that it connects to your clusters is through a cube config file so i'm going to add well it already has the the main cube config file added this is the one for mini cube and if i set the mini cube context here and say add cluster it should connect to the running mini coop cluster locally and start showing me everything about it so showing me some warnings from events that are happening on the network if i go to workloads it's going to let me browse the pods in the cluster i don't have anything running on this local cluster but it will let me browse these pods and things um and uh so so that's kind of cool this particular cluster isn't doing a whole lot so you know we're going to add the linode cluster because it has more stuff going on in it it's running drupal and all that so i'll go ahead and do that and that cluster config i think it's lino dot yaml so i'm going to add that one in and i'm going to add uh the linode context that's in that cubeconfig file add cluster and what it does initially is it just gets from the kubernetes api all the resources that are in the cluster so it can list it here and it keeps updating this as as things go along so it's a live updating dashboard of everything so if we go to pods this is showing pods from everywhere in all namespaces if i want i can limit the namespace to drupal that we saw in episodes four five and six uh which drupal currently has uh a drupal container and a mariadb container it also gives you a lot of nice things at a glance showing how long it's been running uh how it's controlled so it's through a replica set that's created by deployment if we go to deployments you can see drupal here and it gives you a little at a glance indicator of if the pods are healthy if they're not how many containers are running in the pod this was an init container that sets up the files for drupal so all this stuff is great you can click on a container and it shows you all the information just like if you ran qb cuddle describe the resource this is all that information in a nice formatted list but here's where some of the things get really interesting with lens you can get logs live updating logs by just clicking on that little this little icon up here logs and you can even get logs for ended like containers that are completed the init files container things like that so that's kind of cool and you can browse different containers also by switching in this little selector i haven't found a way that you can merge all the logs for one entire deployment like all the pods in one deployment together but you can browse individual container logs this way so that's really cool another thing that's nice is you can bring up a an active qb cuddle session for this cluster so generally on the command line if i'm going to switch to my my linode cluster right now if i say mini cube or if i say qb cuddle get nodes i'm going to be looking at the default uh the default cluster that's in minicube that's in my cubic dot cube config file but if i wanted to switch i'd say export cube config equals dot cube linode and then i could say qb cuttle get nodes and now i'm in the linode context and here you can just switch context by clicking on the different clusters but if you want to use qb color you can just hit plus down here and say terminal session and this drops me right into the right context already so if i say qb cuddle get nodes it gives me that output for the linwood cluster and if i just switch over here and do the same thing and i say qb cuddle get nodes i'm in the mini cube context so that's again switching between clusters is very quick and easy inside of lens now there's some other cool features and one of them is going to probably surprise some of the viewers of this so i'm on the lynnode cluster right now and sometimes it's nice to be able to dive into a pod to debug something and a lot of pods have a command line environment a shell like bash or just plain sh or something so that you can log into the container and run around inside of it look at files look at logs if you don't have the logs coming out to kubernetes correctly change change things in that container obviously any changes you make in a container unless it's in a persistent folder in a pvc those changes won't persist but it's nice to be able to debug sometimes by logging in so if i click on this drupal pod there's a button up here for podshell and that's going to drop me straight into the pod and i can start looking in this container logged in through linode through my computer through the cluster and see files on it and see see what's going on in the pod so that's kind of cool now another thing and this is the part that might really surprise some people is i can go to my nodes so these are the three linux servers you'll notice that this this little this little shell thing is here for the node too and if i click on it it's going to drop me an as root on my lino node and to prove that i'll do ps aux and show you here's all the things running you have container d running as a service this this drops me in as a root user on the node and um i have in my notes here wait what some people are surprised when they learn that if you give somebody admin access to a kubernetes cluster you're effectively giving them root access to the servers where kubernetes is running at least in most kubernetes configurations and the way that kubernetes is installed in most places and how is this possible well it's this is containers this is the way that they're set up kubernetes runs as root on the server and you can run containers using kubernetes using the privileged privileged flag which allows you basically to run the container in the root namespace on the server itself so you can log into the server as root the way that the way that lens actually does this is it starts a pod with privileged and it runs the command n s enter dash t one dash m dash u dash i dash n and that basically says give me all privileges and all rights as root on the system so something that something that you might want to take a look at if if you are involved in security at all and you're like oh my gosh i didn't even realize this i need to take away admin access from intern gym or or the uh the new developer we just onboarded that that is doing php apps and should have no business getting root access to our servers um pod security policies which i'm not going to go into today that's you know security could be its own entire 10-part series on kubernetes but you want to look at pod security policies and preventing people from creating pods that can that can do things also the other thing is a lot of people when they start using kubernetes if someone else needs access to the cluster they just share the cube config that's not a good idea um other people will just create a new account and like in amazon eks add everybody as a system master and that's that's also not a good idea you need to use kubernetes role based access controls to add user accounts that have fewer privileges so that it can't do things like this if i had a lower privileged account i wouldn't be able to create this pod that can get root access on the server so anyway i wanted to mention that because that's one thing that a lot of people run into once they start getting deeper into kubernetes oops we just created a pod that gives everyone running this pod root access to the server or this php app all of a sudden can have root access to the server that's a bad thing so you do need to to keep that in mind especially if you're building automation tools like if you're using jenkins with kubernetes or some other service with kubernetes and you give that access to your cluster you have to understand that you're giving that app full root access and full control over your entire cluster so be really careful i guess at a minimum and ideally limit the controls that that app can have by using role-based access controls all right so another thing that's really cool about lens though besides the fact that you can get root on your servers is if you wanted to drop into any web web app or web service that's running if i go to network and go to services you can do this using using let's see is there any other service you can do this using um using qb cuddle with what is the command like get local proxy or something i forget what the command is again i look these things up all the time because i forget but in lens anything that's running that has an exposed port you can go into and like let's see here i can click on one of these ports and it will create that proxy for me and open it up so nginx is saying 404 not found because it doesn't have an ingress record for localhost but anything that's in here it gives you a link in the ports section in the services and you can just click on it to bring it up i don't know if there's anything inside of here that that might be interesting to see uh no not really but if there were that'd be cool i guess kubernetes has the api so if i click here i don't even know what it's going to bring up but that is nice because you just you don't have to remember the command line arguments to get into the service that's running if it's not exposed through a node port or something else yeah it's it's not going to do that because kubernetes is a special service so that's kind of cool another thing is and this is where i talked about earlier it's called an ide for kubernetes but again i i don't like doing this because this i don't think this is a great way to manage your clusters you should be managing them through code not through an interface like this but if you wanted to if i wanted to let's go to the drupal deployment down here if i wanted to scale up drupal i could just click on this edit button and i could change it in here to replicas three or five or whatever i could do that and it's the same thing as saying qb cuddle edit on the command line but typically unless you're doing some active development on that resource and doing some testing and things on a local cluster typically you want to do all these kind of changes in your code and not not be doing them directly on the cluster like this but it is nice to see you know if you just wanted to see the the configuration or grab grab the spec out and copy it into a file or something you can do that here so i'm going to cancel that but one thing that you might also be noticing is as i'm browsing through there's a lot of messages on the screen about metrics not being available and even if i go to the cluster level it shows metrics are not available due to missing or invalid prometheus configuration now this cluster does have um if i go to the terminal this cluster does have metrics so if i say qb cuddle top nodes it has the metric server running on it and metric server gives metrics for nodes and pods and things and that's also useful for qb cuddle get hpa drupal we have that horizontal pod auto scaling configuration that will scale drupal if needed but lens actually doesn't use metric server to get its metrics it uses prometheus so that's a good segue into getting prometheus running on our cluster so a little bit of background on prometheus uh in grafana first of all with prometheus it was actually developed by soundcloud there's a lot of these tools in the kubernetes ecosystems where somebody actually developed it for a purpose they had and then they donated to the cncf and prometheus is one of the prime examples of this soundcloud was having some trouble growing their stats and metrics monitoring systems they were using statsd and graphite i think but in 2012 they started building this new tool prometheus to make it so that they would have a faster and simpler backend for managing metrics and pushing them out from servers to a central location this tool started to be really popular they put it open source and other companies started using it and soundcloud realized hey we could you know we could share this and make it official so in 2016 they actually transferred the governance of the prometheus project over to the cncf and in 2018 prometheus became the second graduated project in in the entire kubernetes cloud native ecosystem and that's a pretty significant milestone because graduated means this is a tool that's used extraordinarily widely it's very stable it's very robust and it has a very good structure of development and maintenance and things like that so like i said it's pretty much the de facto way of of managing metrics inside of kubernetes um grafana on the other hand uh and i guess i could show the the actual websites for these things uh prometheus dot io so there's prometheus now and prometheus and graphonic both don't have to be run inside a kubernetes cluster that's not the only way you can run them uh but they're they're great inside of kubernetes so that's what that's the context we're going to talk about them in um uh grafana is actually managed by a company called grafana labs and they have a hosted version of grafana and you know they have a software as a service platform but they also publish it open source under a free license and so that's the way that most people that use it inside of kubernetes will use it an interesting part of the history of somebody's here from mars i seriously doubt that and his name is gandalf so i guess i have fictional characters in live chat now um uh but grafana was actually forked from kibana three and the idea was kibana was was already a pretty nice visualization tool but it was a bit complex and it also didn't focus on real-time metrics monitoring it was focused more on search graphs and things like that so the original the original idea was to fork cabana 3 and make it a little simpler to use and make it focused more on individual metrics and live updating monitoring dashboards so over time things like plugins were added and and the the plug-in repository was was extraordinarily huge uh and nowadays you can find an integration with almost any tool that you want whether it's getting uh getting stats and metrics from amazon or google cloud or whatever or bare metal systems or integrating notifications and alerts with slack and teams and irc and pagerduty or pretty much whatever software you can think of but the other nice thing about grafana is it works with a lot of different systems it not just prometheus as a data source so you can build dashboards that pull in data and metrics from all over the place and build really awesome dashboards that update in real time for any kind of data that you want whether it's in your cluster or business metrics or other things like that so a lot of people use grafana for all these different things and and i've been using it for the past couple years now and really enjoy the fact that it's open source simple and just works really well um so in the past originally it was kind of hard to get all these things working together because each of these programs is a little bit complicated and has its own little quirks and ways of configuring them and things like that especially when you go into kubernetes there's a lot of config maps that you'd have to set up to connect the two together and the first time you do it it can be a little daunting luckily for us there's a project the cube prometheus stack and this is a helm chart that's maintained in the community that actually integrates a project from the prometheus project called cube prometheus and this this does a few different things it installs it installs prometheus and grafana on your cluster but it also installs a few other things that are helpful to have like cube state metrics which helps pull metrics out of kubernetes and put them into prometheus the prometheus node exporter which grabs node node metrics from your actual kubernetes nodes and puts them into prometheus and a few default grafana dashboards that are really helpful for kubernetes debugging and monitoring as well as some prometheus rules and things like that so you could if you wanted to go to the cube prometheus project and install using that it actually has a set of manifests that you can install and for some clusters i actually do that when i need to modify things and customize them a little bit more but for most clusters the cube prometheus stack has a great starting point and in some cases it's all i really need especially if i'm not doing any custom application monitoring that i that i need to set up later so to get this running just like any other helm chart first we need to add the repo so i'm going to do that here so i'm gonna and i could do this in lens too it doesn't really matter where i do it but i'm already in the context of the linode cluster on the command line here and the font size is a little bigger and i don't want to look up how to do that in there so the first thing to do is add the prometheus community home charts repo so i'm going to do that and it's already in here and it's always a good idea after adding a repo to run helm repo update so making sure that all my repos are up to date and then i'm going to i'm going to put this into a monitoring namespace that way it's separated out from the rest of my cluster and generally it's a good idea as we've seen with drupal and other things to put things into their own namespace that way you have a little more control over the way that it's set up and managed and finally i'm going to deploy it by installing uh installing prometheus that's what i'll call this i could call it like monitoring stack or something but i'm going to call it prometheus inside the monitoring namespace and this is the the actual home chart that i'm going to install that comes from the prometheus community chart repository so i'll do that this is going to take a little time uh and and once that's done i can i can watch the progress of it i can actually check in lens and this this is cool because lens is all real time i can check in lens in the monitoring name space what's going on right now it's not seeing any pods in here but if i go to deployments i think still nothing there it's still installing uh we'll see when things start populating and we'll see it come in um should only take okay so here comes here comes some pods uh it looks like grafana and prometheus monitoring and uh i guess this this is the node exporter running on each node as a daemon set um so again lens is nice for seeing this kind of up-to-date stuff you can do this all in the command line and i have been through the rest of the series and like i said i think you need to know how to do that stuff so you know where this information is coming from but it is nice to have this in lens so we can see things coming up but it looks like everything's running there and if i check the deployments in the monitoring namespace everything looks like it's available so that's cool and the next thing i want to do is access grafana actually before that i'll show you really quick just what uh what prometheus itself looks like and i can't resize this column so i don't know what these are uh there's probably some way to resize the columns here but i'll just try to make the window a lot larger prometheus prometheus node exporter alert manager here prometheus so let's open this up and this should just show us the prometheus dashboard uh and it opened on my other screen so hold on a second let me bring this over here uh this is the prometheus dashboard and this is not uh this is probably not the most like oh this is intuitive you have to know what you're looking for you have to kind of know what prometheus how it's set up and things so this is why we use grafana because it's a little bit nicer in terms of the interface for interacting with the data in the cluster but you can see here's the rules that are added by this stack automatically if you didn't have the stack setting these up you'd have to set up a lot of the stuff yourself so you can see why it's nice to have this community maintained stack that does all this stuff but it's showing that everything's okay which means that prometheus is getting these metrics and they're getting into prometheus and prometheus is storing them successfully so we could look up all these different things inside of prometheus itself and that's an exercise that i leave up to you the viewer to do on your own time because we're going to look at grafana instead and there's a few different ways that we can get at grafana the first thing that i'm going to do is we can check and see in the command line i'll do it here just because we've been doing that and we should be pretty familiar with it by this point if i get the services that the stack sets up we can see there's one for grafana and it has a cluster ip so it's not accessible outside the cluster right now but the same kind of thing that we did here if i clicked on this it would set up a proxy for me so i could go to it locally but we can do the same thing on the command line with qb cuddle port forward dash n monitoring service slash prometheus grafana and we'll do port 3000 to port 80 so that we can get the access to this grafana running on there and now it sets up that port forward so if i go in here and go to localhost 3000 this is going to load grafana for me and the the actual the admin password for grafana is set up by default and it's not very secure so if you're going to do this on production you probably don't want to use the default password that's a bad idea default passwords are always bad but the default username is admin and the default password is what is it prom prom dash operator and you can actually see what it is uh grafana has that i'm going to cancel this so now my proxy is gone but you can actually get the get the password through a secret that's stored in the monitoring namespace the secret's name is prometheus grafana and i'm going to look at the admin password inside that secret and again you can you can do this stuff through lens too it's in where are the secrets storage no apps there are secrets in here i don't know that's a good question i've never looked up a secret inside of uh inside of lens so i don't know where where those are they have to be somewhere out of the box grafana is admin admin even more secure yeah so the point here is that if you're going to do this in production you should definitely not use the default admin password and that's a value that you can override in the the chart that you install it's under configuration oh the one thing i didn't expand there there all right so if i look at secrets in the monitoring namespace you can see prometheus graffana is in here and this is the admin password and this is base64 encoded so if we do that it decodes it for us and you see prime operator which is the same thing as doing this command on the command line but you can see like you know if if you want to get this quickly trying to remember all this is a lot harder than clicking through here and finding this but it's important to know how this works so you know this little button here is masking the fact that it's decoding in base 64. if you don't know that then this is just going to be magic and mystery to you so that's why i always start on the command line before i go into these utilities all right so but you know going to localhost 3000 is is not wonderful and something like erfan is typically something you'd want to expose to other users of your application so that they can monitor your systems too grifana has a user access control system that you can set up and so i actually have an ingress record that's just like the one that we used for drupal earlier and it's going to use the same thing it's going to use nginx for ingress and it's going to use the cert manager to get a let's encrypt certificate and i can deploy this and it's just saying instead of you know for the other one it was drupal dot there it was ep6 dot cube101.jeffcolor.com um for this one it's going to be grafana.cube101 or we could do monitoring.cub101 whatever we wanted and i have this domain name pointed at the ingress load balancer in linode already so all i have to do is deploy this so i can say qb cuddle let's see kate's manifest cuddle apply dash f grafana and it's going to apply that and after a minute or so this should be accessible and technically you could go to it too if you wanted and i'm sure some of you already are so you can see it already has a certificate the certificate is from let's encrypt so it's all all verified and happy and if i say admin and what was it again uh prom dash operator is the current password so now i'm in the grafana dashboards now it like i said it this this comes with a bunch of out of the box stuff that's really nice already so if i go to this little dashboards page and manage you can see the dashboards that it comes with um and there's a lot of really neat ones uh for example if you just want to get an overview of how your how your nodes are doing if any of them are like overheating or have too much cpu usage or anything you can go here and browse the different nodes and see memory consumption cpu usage i think these are all uh two virtual cpu nodes so you can see there's there's a graph for each of the virtual cpus the load average memory usage disk io network io things like that some of the other nice dashboards that you might want to look at is the general well let me go back to this view excuse me the general cluster overview here that's showing cpu usage across the entire cluster so obviously this cluster is a bit oversized for the one application that i'm running but i'm sure if if somebody wanted to throw some load at drupal the cpu utilization would probably go up a bit but you can see also a nice overview of the quotas and the usage for each namespace and this is another reason why it's good to define for every application that you put out their cpu requests and cpu limits that lets you see at a glance in a dashboard like this is this application using all of its cpu do i need to consider increasing the allocations i get to it or you know is this application of doing something bad and thrashing or something so you can see that drupal is not really using much at all right now probably because nobody's accessing it um let's see uh i was just checking live chat for a second uh so that's another another helpful dashboard that you can use to to get a glance um i don't want to save any changes there the other one that i like to look at is workloads and if you go in here you can see let's see you can change namespaces and see different things that are running what their cpu usage is it's just a little easier than going on the whole cluster level to get that to pop in here but these dashboards are all created inside of grafana and then they were exported into a config map and you can change anything just by going in here and you could change the actual query um you can change the way that this is displayed um you know take out the lines and use bar graph it's a pretty simple interface there's a lot of complexity here for sure um but once you get the hang of it you can create any kind of dashboard that you've ever seen uh you know whether it's a cool hacker's dashboard from a movie or a tv show like mr robot or a dashboard you can i can see somebody must be doing something maria db is going a little bit crazy right now but you can do all this stuff easily once you get the hang of it uh so that is that's grafana's graphs it'll be interesting to see if uh if we go to where's the cluster compute resources cluster cpu usage has been increasing a bit uh drupal's still not doing a whole lot but uh it looks like somebody's probably doing a little bit of work hitting drupal so thanks for that but grafana like i mentioned earlier has a user a built-in user user access system so if i go in here you can add more users you can have users in different teams that have access to different dashboards so it's nice for that but what that what that also says is grafana just like drupal is a stateful application so these things are stored in somewhere on a file system so if you are going to do that if you're going to set up grafana and have different users and different access and things like that you're going to want to also set up persistence which is not enabled by default so that's something that trips some people up they set all this stuff up and then they upgrade their nodes in their cluster or something then all their configuration is wiped out all their custom dashboards that's because you have to turn on persistence there's a setting in the grafana chart uh let's see grafana helm chart um it's called persistence dot enabled i think uh it should be in here persistence yeah so persistence enabled is set to false by default so you want to actually turn that on by setting it to true if you're going to use this in production and then you also want to do things like back up that volume and make sure you have a process for restoring it if there is a failure anyways so uh you know today we talked about lens prometheus grafana uh it's if you have these metrics in place and and you know it looks like uh let's let's check out how drupal's doing over here in cluster it's not a whole lot of a lot a lot of load but if you have these things in place you can start seeing like what applications do need more cpu or memory or if your servers are constantly running out of ram but they have lots of cpu maybe you need to switch the server type that you're running on and with kubernetes these things are a lot easier because you can swap out servers and replace nodes very very quickly and efficiently especially if you have a system that's more redone and has has the ability to to move to different servers in this case mariadb would have a little downtime when it moves to a new server when you do an upgrade or server replacement but when you have these tools in place it's easier to see capacity issues or if you're over provisioned things like that if you don't have these tools it's harder to see that and you have to do a lot of manual work and lens can save you a lot of time just you know instead of remembering the command line arguments to get this secret and to to base64 decode this to see that it's prom dash operator all those little things can save a lot of time but again i like to learn the hard way and then you know throw on a tool like this on top of it but just like with kubernetes itself i would i would caution against diving straight into lens and doing everything via lens because this can be a little daunting too to see all these different options you know when you click on when you click on a pod and you click on this there's so much stuff going on in here that until you know what all the stuff is doing this is going to confuse you too but it isn't it is interesting to see that now uh now that we have prometheus set up we can see all these stats four individual pods as we click on them and let's go over to drupal and see see if drupal's also yeah we're seeing a little bit more activity here and a little bit more network activity so somebody somebody's definitely been um been pulling down drupal a little bit but it's it's running pretty well on this cluster so that's good um anyways so uh before i get uh that that's all i have to talk about for monitoring today but before i get to the the thing that i teased about in the beginning of this episode um i wanted to thank everybody who has been watching this series and and you know as an infrastructure guy i always monitor all the things i do including my youtube stuff that i do uh like this series so it was as i've mentioned at the beginning of every episode i love seeing the global audience this has and this is a map it's not exactly accurate in terms of the depth of the colors the darkness of the blue and things like that that was a little bit hard to get because there are some people who repeatedly say every episode i'm from st louis missouri or i'm from wherever um but it's it's interesting to see like almost all of europe has had people viewing this there's a ton of people in south america um asia in in the ansible 101 series i didn't see as much in terms of asian countries that were represented so that's cool to see um and you know south america there were a lot more people africa there's there's uh it was about the same with ansible 101 so probably not quite as much penetration of infrastructure work there but it's always encouraging to see more people coming from there and russia too this time there were more uh more russian viewers for this series than i ever expected so that was cool and usa usa usa yeah obviously i'm american i speak english and so probably the the most people are coming from more english oriented countries like usa canada great britain and some of the european countries but um i'm so happy to be be with all of you and uh there are some other metrics that are interesting to share this this i think just shows uh something that that youtube's algorithm does whenever i've done a series i've done about four or five different series on this youtube channel and always for some reason it only recommends that first episode and so you get a ton of people watching the first episode you get a number of people who kind of follow along to the second but then they lose interest by the third if they're not really invested in learning about kubernetes they're just kind of interested in the idea um so you know with any of these series it always has a graph that's like this but i also found it interesting that my operators episode was the least viewed um episode nine is is still only a week old so you know i discount that that'll that'll keep going up uh but for some reason people just weren't that interested in operators um i i really like them but they are a little bit more complicated and maybe that's a topic i should have left out and done in the kubernetes 102 series but um but it is what it is and uh um i'm still amazed by the numbers that this uh series has produced um and i actually have a blog post if you go to jeffgeeling.com you can find out more more about all these numbers and what they mean in comparisons to my other series that i don't did on ansible uh but one thing that that i should note is the uh the revenue i get from these streams including the sponsorship and i have to say thank you again uh to to amazey for sponsoring this mesa.io has uh has been very generous and and made it a lot more worth my while than this number uh makes it seem uh but if you take all the hours that i put into making these episodes versus the money that i get out of them it's it's not much uh and and so i guess my advice for anybody who would be thinking about doing this themselves is don't quit your day job and do this uh this youtube channel has been i've had it since 2006 or 7 or something and i really started trying to grow it a year or two ago and it takes a lot of time a lot of investment and you're not going to see a huge return on investment in the blog post i even mentioned a few ideas that i had like maybe doing this course in a system like pluralsight or udemy or something but i decided to do it in the open and free because that's you know that's who i am i like to to make my content open and free anyways so uh early on in the series i think it was episode one or two i mentioned that i was selling ansible for kubernetes which i'm still writing and i'm still gonna finish i'm i'm giving it to you for 4.99 for all the viewers of the series but i decided this was about a week ago i decided to take all the content from this series and put it into its own book that's more focused only on kubernetes not on automation of kubernetes but learning kubernetes itself and i decided to make that book free i love these little animations i'm sure steve jobs spent years working on that little anvil drop animation but i decided to make this book free for everybody watching the series so if you're watching this there will be a link which is right here if you go to that link you can get a copy free and yeah if you want you can share that link with other people on social media and things but i figured i'd do this special for the people that are actually watching this the book is an extraordinarily raw form right now a lot of the book is actually transcripts from the videos that i have not edited much yet so i'm going to be working on this making it a lot more useful but the book already has almost all the content from the series in it it's just in a form where i still need to go through and edit and every chapter i have a little warning at the top that says i haven't finished editing this chapter yet um so so but please go and get this it's only free for a couple days and it's only free for the people watching this series uh kind of as a thank you for watching um go get it because if you get it from lean pub where this link leads you to you'll get every update to the book free forever and for anybody that's that's bought my ansible for devops or ansible for kubernetes books knows i i continue updating them i keep them relevant over time so it's in my opinion free is definitely the best value here and uh hopefully you'll enjoy it um and and like i said please go ahead and share that if you know somebody else who would benefit from this share it with them as well and since i'm making this for you since i'm uh since i'm um keeping uh you know not not really getting back a ton of money for the hours i'm putting into this and stuff if you do feel like you would be able to do it i would greatly appreciate if you'd consider supporting me on patreon or github sponsors and there are links in the description below um but anyway please check out the book know that it's still in rough form and i'm working on it but over the next few weeks i should have a lot more of it completed so that that it's a more useful reference and for people like me i know i like having a book in my hands i'm going to be publishing a paperback version soon uh once i get once i get the the book in a more final form so anyway i will see you on the internet thank you so much for watching this series um it's been a blast and i will see you next time you

Channel: Jeff Geerling

Views: 17,958

Keywords: kubernetes, devops, introduction, intro, beginners, guide, k8s, geerlingguy, kube101, kube, kubectl, docker, applications, automation, computing, operations, monitoring, prometheus, grafana, lens, visibility, insights, ops

Id: zW-E8THfvPY

Length: 52min 30sec (3150 seconds)

Published: Wed Feb 10 2021