Introduction to Prometheus monitoring

what is that people and welcome to another video and in this video i want to go over container and micro service monitoring monitoring of legacy systems basically look at a tool called prometheus which is open-source decentralized monitoring tool and i want to give you guys guys an overview of how Prometheus works Howard can help benefit you the reasons why I like it so the pros and cons and basically give you guys an overview of the Prometheus architecture so you know how it can plug into modern applications as well as legacy systems so without further ado let's go alright folks so what is Prometheus prometheus is an open source monitoring system literally monitors everything the thing that I like about Prometheus the most is that it's open source and it's free a lot of people get quite sort of hesitant when they hear the word open source because I think there's not going to be much documentation and things around but could kubernetes and Prometheus are extremely large communities and this is all backed by the cloud net of computing foundation which is like the the folks who run the kubernetes projects the other thing is it monitors nearly everything this is what I like about it you can plug anything into prometheus if you go over to their website they have instrumentation they have integrations section just scroll down here and have a look all these different databases including Microsoft sequel server hardware defraud where exporters see ICD and issue tracker exporters they have message systems different storages different api's and proxies like nginx for example the list just goes on and on and on of what you can export into Prometheus so the other thing is you can also use Prometheus to monitor micro services so modern micro service architecture there are a ton of libraries and I'm going to show you guys in a bit that all works libraries for different code bases to monitor micro services and multiple languages supports so whether you write and go no js' c-sharp Python Java it doesn't matter also if you have a linux servers whether they're legacy or part of a kubernetes cluster Prometheus plugs in really easily also Windows Server so whether again whether it's a legacy Windows platform that you run or whether it's modern apps it doesn't matter Prometheus plugs right into it and this is the key thing I want to talk about and this is why I love Prometheus so much is the pool approach I recently did a talk about it's titled performance and scale billions of requests per day check it out on my YouTube and one of the bottlenecks when working with high traffic volume is monitoring when you have a ton of micro services and they're pushing metrics out to a matrix server monitoring systems can flood the network so Prometheus uses a pool approach so I'm just gonna show you guys an example of what I mean by that so this is a traditional monitoring design so you have service a B and C and these could be Windows machines it could be Linux machines it could be micro services running on pure binaries it doesn't matter in the traditional world you install a diamond on each of these and you push your matrix out to your monitoring system and what happens is it becomes really hard to monitor and actually just maintain the system because first of all it flood the network is being flooded to this end point it's hard to scale this monitoring system because you have to aggregate all the 95 then 99% tiles of metrics you have to aggregate it that's one point so you're gonna be running hot on CPU trying to maintain this so this is what Prometheus does well as it reverses this and says I'm gonna come and scrape your endpoint so every service or Windows machine Linux machine whatever it is that you're integrating has like a matrix endpoint allows Prometheus to come in and scrape your services and the other cool thing about Prometheus is that it has like a configuration say yeah mph file and it has the service discovery mechanisms built-in for you to easily tell Prometheus where the end points are that it needs to go scraped so your micro-services your machines serve the legacy applications don't need to worry about where prometheus is Prometheus will find you so I want to give you guys a little high overview of a legacy system so let's say you have like a private cloud or public cloud doesn't really matter you have a network these can have multiple subnets it doesn't really matter this is just to show you guys how Prometheus plugs in and let's say we have post grades MySQL or Microsoft sequel running and we have a few Windows machines they could be Linux machines as well and how would you go ahead and monitor this so the first thing I would recommend is you install a little Linux machine inside one of these subnets where you're running and you deploy also a Prometheus agent on each one of these now if it's like a Linux machine you would get the Linux I'm no exporter if it's Windows you deploy the Windows node exporter and this will extract all the performance counters for Windows if you're using Microsoft SQL you'll get the SQL exporter and what this does is it creates a matrix in points on each one of these boxes so you can choose a port you go by a hostname if you open this in the browser you'll see like a text file like it takes basically blob of just matrix and every time you refresh it these matrix change so that little daemon running here is collecting all these performance metrics and basically just exposing it on a page for you to come and scrape and then what you do you install a little Linux machine and the easiest way to run prometheus is just install docker on a machine and then just pull the Prometheus image give it a Hamel file that lists out all the hosts so that it needs to go and scrape and what Prometheus will do is goes out and scrapes them really really simple and let's say if you have a legacy like let's say it's a super secure PCI compliant network and Prometheus cannot always access all the different subnets or have trouble you know securing subnets and having certain subjects to talk to others Prometheus also has a Federation feature so if you turn on federates you can basically get this URL sis prometheus and over the port and it opens up a federate endpoint and the federates endpoint basically is the same as the metrics endpoint it's just it allows Prometheus to expose all its Prometheus metrics to other Prometheus services so by turning on Federation I can then have another Prometheus sitting on another subnet that scrapes that one so this allows you to move Prometheus metrics around subnets using a full model if you do need to push however you can also deploy the push exporter and that allows you to push metrics out to let's say another Prometheus and this one can run in another subnet or in the cloud so that gives you sort of give you an idea of what the Prometheus architecture looks like for legacy systems and how you can just decentralize your monitoring now I want to show you guys how this looks versus a modern architecture so let's say you're running kubernetes you have a kubernetes platform micro services and you're running let's say six machines on a kubernetes cluster so what you would get is your environment where you can go and deploy containers and whatnot so the first thing you want to do is you want to get the Yama file to deploy a Prometheus pod into your environment but we also have to remember that when we have a coop in a nice cluster there's a coup Benes API and what we do is we deploy our prometheus node a Prometheus pod into the cluster and we configure its Yammer file to point to the API that's literally all we do the kubernetes api has mechanisms built-in and it has a component called matrix server you don't have to worry about how it gets the matrix kubernetes will get the matrix of all these machines running in the cluster as well as all containers running on the infrastructure so the only thing you have to do is say hey Prometheus here's the API I'm into stood in the matrix and what you can then do is you can have another Prometheus sitting outside in your monitoring environment and you can turn the fetter at end point on for this kubernetes and go ahead and scrape that one so this is just that an example of how you would bring cluster metrics out of kubernetes and into another environment so if kubernetes is down this is not going to impact your monitoring system of your legacy workload so it's completely like decentralized and centralized at the same time it's super awesome so now I'm going to show you guys when you plug in graph on a-- on top of this let me just move myself over a little bit you have graph on up that you plug into into that Prometheus instance over here you get a rich dashboard now of all the VMS running in kubernetes as well as all the containers that you've deployed so you can see CPU disk usage memory usage and everything it's like super simple so let's say you now have a team they have a namespace that they've deployed in kubernetes and they have a bunch of micro services running different programming languages and they want they want to now instrument their metrics now out the box if you have this deployed that I've showed you here you will automatically get all these containers CPU memory disk i/o and a bunch of other metrics out the box for the container processes so you'll be able to see the capacity planning of the cluster as well as like how much each container is using so that you get out the box but sometimes developers want to have instrumentation for their application so what you're gonna go ahead and do is you deploy and the library in your in your code so you have like python c-sharp it has all these libraries that you can deploy and then what will happen is you will have a matrix endpoint similar to this similar to the legacy one you get a matrix in points on every container that Prometheus now can go and scrape now you must be thinking how does Prometheus know where my hundreds of microservices are running and this is the cool thing remember Prometheus finds you so it has a component called service monitors which is its service discovery mechanism and what we do is when we deploy a micro service we have a camel file remember for our pod and deployment we add another component to this camel file called a service monitor that tells Prometheus that we're interested in Prometheus to come and scrape us so what will happen is Prometheus will monitor these service monitors and then we'll find your micro service and scrape it and bring the metrics in so all of a sudden now you'll start getting all these like extra metrics for all your instrumentation that you do now I want to show you guys sometimes when you're like running large cluster workloads this Prometheus instance here can fill up with memory it becomes and also you don't want this Prometheus instance to become a single point of failure so what a lot of people do is they have a Prometheus per namespace or pert and this is where the Prometheus operator comes in the Prometheus operator helps you to manage prometheus instances across different kubernetes clusters so what what a team can do is they can say hey I want to deploy my own prometheus and that prometheus user service monitoring to find the micro services inside that namespace only and then uses Federation to federate it out that way if your Prometheus is down it doesn't affect the other teams and also you can start managing the memory of prometheus instances a little bit more efficiently because a lot of at times people have one prometheus server bringing all those metrics into one they have to deploy really large memory to hold all the metrics so this is why Prometheus is so powerful because it's decentralized as well as like kind of it gives you that feeling that everything is centralized because you can federated all together at some endpoints and then put graph on ax on top of it and then get really really awesome dashboards so anyway guys that is it for this video hope that clears things up about prometheus architecture and I'm gonna do some more videos where I'm gonna show how to deploy Prometheus how to set up service monitors and how to do all these things in a bit of detail but I was hoping to make just a summary video to give you guys an overview of the architecture and how you can just plug Prometheus in into your current systems so hope you guys enjoyed it like and subscribe and until next time peace [Music] [Music]

Channel: That DevOps Guy

Views: 24,599

Keywords: monitoring, sysadmin, prometheus, kubernetes, linux

Id: 5o37CGlNLr8

Length: 13min 51sec (831 seconds)

Published: Thu Aug 22 2019