Prometheus SNMP_Exporter Tutorial

hello this tutorial is going to focus on Prometheus and the official exporter for SNMP a couple things before we get started I'm going to assume that you're working with Prometheus in Linux and that you're comfortable with Linux and that you also have some generalized knowledge about SNMP and MIBs and oids and I guess a couple of networking concepts but if you don't don't worry about it so before we get going here we have a Ubuntu virtual machine and we have a vagrant provisioned virtual qfx 10k which is a Juniper switch the Ubuntu VM has Prometheus and SNMP exporter as a service and it should be running and then the qfx has a basic SNMP config both machines have a host-only adapter attached so that they are getting a DHCP lease from VirtualBox and we can see one of the management interfaces has a one 92168 56.9 the Ubuntu VM is 56:11 so they're on their own private network within VirtualBox and can talk to one another if we look Prometheus is up and running on 56 dot 11 port nine zero nine zero and it should just be scraping itself yeah so it's just getting some internal Prometheus metrics right now another thing that the Ubuntu VM has is a working go environment which is going to be necessary if you want to build the config generator tool that that the SNMP exporter repo has so I'm going to put a in the description for getting that set up it's pretty easy there's just a couple commands to run it's easy to test so hopefully by the end of this you can have a dev box up and be monitoring with SNMP and you'll know how to change the configuration around and and extract certain information and learn how to control the SNMP exporter so if we jump in here out of the box you do get a configuration file with SNMP exporter it's going to be named SNMP Hamel and it is kind of a large file that's made up of various what are considered modules which are just these yamo keys at the top here at the top of each walk so we have an AP APC UPS and it's it's a large file so it might have some modules in there you don't need or maybe you just want to target certain oh i d--'s certain MIBs maybe you want to chain certain MIBs together so one of the limitations here is that you cannot change specific modules together so if you wanted to scrape your interface mabh and a system nib that has uptime and system name and description and contact you couldn't chain them together it all have to be contained in one module so we'll do a couple use cases in this tutorial that show you how to change some of that stuff and SNMP exporter isn't that different than some of the other exporters this real Abel config here follows the same standard that black box exporter uses if you're familiar with that but this is very important and you don't need to know what this does but just know that this needs to be on a job where you're scraping SNMP so SNMP exporter typically will run on the prometheus instance itself and you will you will want to scrape the local host on the port number that S&P exporter is listening on and this magic real Abel config here will append the it'll do a couple things but it'll take this target IP address and basically pass it as a URL parameter so before we get started let's let's grab a couple SNMP utilities that are gonna help us with some troubleshooting so we're gonna do an app get install SNMP on this Linux VM and it's gonna go and grab some SNMP utilities for us so now we can do an Essen snmpwalk against this juniper switch and we're using version to taxi a community string is going to be public I think that's what I said it as and we need the target parameter and I'm just gonna do it dot one and cancel it real quick okay so you can see we're getting some data back from that walk so that's that's good so we have good connectivity so you can see that these are just mapping to IDs which isn't great for us we would like that to be human readable for troubleshooting purposes in the future so let's go ahead and grab SNMP maebh's downloader this is gonna go out and fetch some mips for us and it'll put them in a place where SNMP libraries can use them and read from them and it's not going to be every mid under the Sun but it'll be a good start and then later on I can show you how to add vet our specific meds so while that's doing its thing SNMP exporter needs to run on the prometheus instance or another VM because a lot of switches and routers are vendor locked to their operating systems so you're not really allowed to install custom software on them unless you're doing working with some white box switches or something along those lines so it looks like MIBs downloader is done which is great so let's run that again and see if it resolves to human readable name so let's do another walk and I see it doesn't so I think we need to tell SNMP to reload those so we're going to open up the SNMP configuration file loading of MIBs is disabled by default if you added MIBs you can re-enable them re-enable loading them by commenting out the following line so let's comment those out save that and let's rerun that again and it looks like we're getting some human readable stuff which is great system contact name location along with the mid name so here's interface mid we have our interface indexes here which is great so that looks good so let's take a look here through our repo that we have open I'll put a link in the description to this if you don't know where this is so yesterday px worker reads from SNMP M will config file by default this file is not intended to be written by hand rather use the generator to generate it for you so let's go take a look at that generator so it's some go so it's going to use net SNMP to parse MIBs and generate configs so I'm using a Debian based distribution so why don't I go and grab these prereqs build a central and an SNMP library and it looks like it's grabbing unzip so let's go grab that one thing to note is SNMP MIBs downloader if you are running this on Debian I believe you have to add the non free repo to your source list and then do an app get update and then you should be able to install SNMP MIBs downloader looks like we grabbed our prerequisites now we're gonna run a go get and that's gonna grab this generator path for us and then we're gonna change directory into our working our go path and that'll put us in the correct directory looks like it's still grabbing it so that's done so let's change path and let's see what we've got here okay a bunch of ghost stuff and let's go build and after the go build is done we should have a generator a binary file alright let's double check there we go we have our generator file here cool so let's see wants me to make MIBs I'm not gonna do that since we have SNP MIBs downloader and we have a special directory for our mips I'm not going to set that variable but I can see that if we want to run it we have to do dot slash generator generate so this the generator reads in from generator amal and writes two SNMP amal so every time we run it and it's successful we need to remember to copy the SNMP amyl it generates in this go path here and copy that to the directory that the SNMP exporter service is using which is Etsy prometheus which is a directory I made so looks like they list the file format which is great list of modules simplest module is just the name and a set of Oh IDs to walk so module specify a name and then you specify an OID or a name you can go as far as specifying an OID you by interface index so it's you get some really good granular control and I think this is important for when you're working with older devices you really don't want to overwhelm them with SNMP poles and I think it's good to be really focused with what you're pulling so if you if you have a device that has hundreds of port channels you know maybe you don't want to pull a full nib maybe you just want traffic maybe you want octets in and out packets in and out and is the interface up down and maybe errors and discards or something like that so you won't you want to be precise with maybe 1000 IDs rather than scraping whole tree but this this is gonna allow you to be very granular more so than the default configuration so I'm going to connected to this Ubuntu VM in vs code so I'm going to open this directory because it's gonna be easier to work with in in vs code rather than opening it in vim or Nano so let's open the generator directory let's take a look we have a generator dot Y Amal so this this file is contains the generator configuration that the default SNMP ml configures generated from so it's got some good stuff in here so this interface mid-1 I am going to keep but the rest of them I'm going to get rid of okay so let's change our module name here to something obvious so I F mid this is where you would set your SNMP version so we're using version 2 with a very simple community name because it's a test environment don't recommend that for production you should use v3 where possible so if you are using v3 you just change that to a3 and fill in all of your auth probe information down here and the generator will use that so we can get rid of all this v3 information and let's tighten this up max repetitions fine timeouts fine because we're not doing too many and I know it's gonna complete in 10 seconds so this is this is where we start to see some of the power of the generator so what I want to do here is I want to go take a look at I have Lib and I like to use OID view so we can get an idea so here's the mid with object identifier z' for the tables down to the single entries so really I just want ifx table which contains interface name the speed high speed reports your port speed in megabit per second let's see in her face so the interface X table has 64 bit counters so it's gonna take longer for them to roll over and that's what the HC means so let's grab this OID here and let's pop this in there and let's get rid of these two I don't really want the uptime in there with it we'll get rid of that comment okay one important part here that I want to highlight would be the lookup section if we don't include this lookup section what's gonna happen is when we pull the switch all of our metrics are gonna come back with just the interface index label on them and that's just a number and when you're working with prom ql' trying to get meaningful information out just based on the interface index it's gonna be really hard to from a heat you know the human perspective to figure out what interface that metrics metric belongs to without having an interface index map in front of you for that specific device so it's really not usable so this lookup section is going to take the interface index and lookup interface alias and interface description and append those labels to our metrics so interface alias is going to be the user-defined description on the interface and then interface description is going to be the name of the interface for instance 10 gig Ethernet 1 0 1 whatever for cisco juniper would be like et - 0 0 0 or whatever so very useful very important to have this you can change this around two different things might be worth experimenting - depending on what your use cases but now that we have this and the syntax looks good let's go ahead and generate our new SNMP config one err I'm not going to worry about that 19 metrics were written and it was written to SNMP ammo in our go path so generated 387 lines of confessed for 18 lines of generator config so that's pretty cool so why don't we go ahead and remember we got need to copy that SNMP amyl out to the directory that SNMP exporter is going to be looking for and doing LS tack lah on Etsy Prometheus and let's make sure this is 103 which is e TC 803 cool East Coast so now let's restart SNMP exporter so it picks up the new configuration and let's go ahead and verify that it was syntactically correct alright it's listening cool so if we reload this we see nothing there we didn't make a Prometheus job so let's go grab this guy here that I already had highlighted and let's make a job so let's say this is job name vq FX - SNMP target is going to be 56 9 and if MIT is not correct remember we renamed it - I F - MIT all uppercase and we're gonna leave in our very cool real Abel config stuff so let's give that a save and I don't have the admin API endpoint on so I can't reload Prometheus with a curl I need to do a signal inter up on prometheus so let's kill one six nine zero zero and okay yep so we reloaded and it was good so if we refresh our Prometheus here we should see we have another target right now it's currently unknown because it hasn't performed a scrape yet and I think I said it 15 seconds is very generous for SNMP I would probably wouldn't do that normally but I guess for testing it's it's okay so after 15 seconds we should see should see it come up okay so last grape 11 seconds ago I completed in 312 milliseconds so let's see if we're getting some so interface so this is cool we can see we have some metrics for an interface outbound octets some of them have interface description interface index as part of the name and then we have our job and our our instance is the target URL and if interface alias isn't blank it populates with the user-defined description this is really important and can't be understated when you're working in Griffin especially it's very helpful to have these these things on here and follow a standard because then you can do stuff like rate of interface where the description description interface alias is equal to a regular expression that's management that star and we'll take a 30-second average if I could put that in the square brackets maybe one day hello there we go and let's close that off so this is just gonna show us outbound octet for any interface that has management and beginning of the description so stuff like that is really powerful when you're building dashboards and gore fauna and if we multiply that by 8 we will get our bit per second rate and if we graph that we see not much data but if we zoom in on that we can start to see it's it's holding steady so very cool and then we have interface speed so again this is in megabit per second if your interface high-speed all right so now that we know we're scraping and we've got some metrics coming in for things we want let's take a look at getting some vendors specific nibs in here and tailoring our SNMP module to do some vendor specific stuff so I've gone ahead and downloaded some juniper MIBs and we are going to put them on the target machine so let me grab down downloads and I think it's called juniper MIBs I think I extracted them yeah okay so we want to take this is actually going to be a recursive SCP because it's a directory and we're gonna say root 1 92168 56 dot this is 11 and I know that SNMP MIBs downloader puts MIBs and user share SNMP MIBs boom okay so let's um let's double check that let's list all in here okay so we have juniper MIBs as a directory so I don't know if it'll look in the directory I haven't tested that normally I put them top level in here but um we can go ahead and give that a shot so let's let's find juniper MIBs let's say Explorer it's like a type SNMP mmm Explorer that was easy let's navigate through this tree I'm craft alarms so we've got a yellow alarm and a rat alarm so major and minor probably so this is a pretty small one so I'm gonna say we're probably safe to just grab the whole thing so we've got that OID copied now we can't chain modules so we have we can't chain modules in that prometheus configuration so we can't do an if' mabe and then Juneau s alarm nib anything like that so we we have to think do we want do we want to make a new module named alarm nib or do do we want to make do we want to add the OID to the walk and change the module name I think we're probably gonna be safe to just since this is a virtual machine and we saw that the polling were turned in 300 milliseconds I'm gonna say we can just add it on here and let's format that properly and this comment isn't relevant for this one went down a line let's cut that in and let's let's call it something more specific like June OS interface alarm how's that and why don't we give that a save and let's regenerate and see if the generator actually finds those that jnx alarm nib which we should have we'll see it might air out yes he's so saying it can't they can't find that OID to walk so that's not good so I want to take everything that's in this Juniper mid directory and I want to pull it out and put it in this top level here with the others so let's do copy cursive user share SNMP MIBs juniper MIBs star to user share you could do a move instead of making a direct copy of all these which is probably the smarter thing to do but that's okay so now I'm gonna do a user LS share let's double check okay yeah so now they're really cluttering up the but we see here's a mid jnx alarm dot txt so hopefully that o IDs in there and it knows the look so let's rerun that generator so we had some parsing errors but it did generate 26 metrics so if MIT had 19 so that makes sense and a new config was written a stand-in PMO if we go and check that out okay here's our Juniper alarm nib let's see did it put it on the end yeah it looks like it puts it on the end and there's some enum values here so that's nice so we got our yellow alarm States relay mode yeol alarm count okay cool so remember we've got a new module name now because it's not just interface map so we're gonna kind of tailor this to be a Juniper specific so let's go ahead and update our Prometheus job and this is no longer going to be that module it's gonna be that one so let's do another Prometheus reload and double check okay loaded now let's copy SNP exporter to its Prometheus and then let's let's do a restart of SNMP exporter and double check that to make sure it didn't around all right so that's running so okay it looks like you got scraped three seconds ago and let's double check here so we've got jnx jnx red alarm count zero that's good I guess it's a virtual machine of cares alarm state is two so I can't remember what two means let's go back to our numerator and see if two has red alarm state two is off so the red alarm indicator is off so this would tell you if the LED is lit on the front of the device or not and it's off cool so we were able to add a vendor specific oh I D pretty easily so I think the only other thing I wanted to show you is that for instance if you wanted to add a new module you would just break it out like so for instance if we wanted to do a system absque rape for vendor-specific you can break that out into individual models modules or add it to your previous and a lot of this configuration is gonna depend on devices that you're polling the types of devices how powerful they are how much configuration they have what their workload is it's all highly dependent on what you're doing so there's a lot going to be a lot of testing involved as far as how often do do the polling sessions complete and there's some trial and error in there so most SNMP monitoring systems default to five-minute polling sessions so if you have modern-day routers and switches they can usually handle sub two-minute polling without much issue if they don't have many interfaces so you should be able to do some two-minute polling with a lot of modern equipment older equipment five minutes might be good enough so you can default to that and you can of course you can always set the the scrape interval independent of the global setting here so this is just the default that every job will get if you don't specify a scrape interval inside the job so you can set the scrape interval and scrape time out you can also manipulate your retries and timeouts from the generator file here and that will be reflected in your new SNMP gamal file that so that's one way to control some of that stuff I would read more about it before doing anything drastic but two minutes to five minutes is usually the sweet spot again there are there are other exporters out there for some of the big vendors that people have written so maybe later on I can cover some of those that Usenet Kampf or they SSH into boxes and they scrape and they're a little bit more efficient than SNMP but for a lot of older devices you're just locked into SNMP and there's really not much you can do about it but SNMP exporter offers a very flexible way to pull these devices and get really good data out of it so I could do a little graph on a tutorial of how to get CRO fana up and going and a way to make some cool repeatable dashboards that are minimal effort so I think I'll probably do that and split it out into another video but thanks for watching I appreciate it and leave a comment if you have any questions or run into any problems so I can try and answer them and have a good one Thanks

Channel: RandomAccessTutorials

Views: 11,784

Rating: 4.8759689 out of 5

Keywords: linux, tutorial, prometheus, snmp_exporter

Id: dZfN3uYtC2I

Length: 34min 27sec (2067 seconds)

Published: Mon Dec 23 2019