How to Create an Agent Node in Jenkins

in today's video we're going to talk about how to create an agent node in jenkins [Music] are you new here if you are welcome and if you are new here you don't know who i am my name is darren pope and i'm a developer advocate for cloudbees you may have watched a previous video where i was talking about what is a jenkins agent and in that video i gave you a couple of examples of what agents are and how they can be used as we get into today's video we're actually going to set up an agent and connect it to our controller what we're starting today is that we have a jenkins lts controller version 2.289.1 when it was installed we used install suggested plugins and i also have an agent but it's not connected just yet that's why we're doing this video today so my controller is running on a centos 7.9 machine also my agent is a centos 7.9 so let's go through and see how this is going to work i'm going to go ahead and log into my controller and what we're going to do is when you go to connect an agent you can get there in one of two different ways i'm going to show you both ways the first way is to go to manage jenkins go to manage nodes and clouds and then you can click on new node but let me show you a faster way to do that let me go back to the dashboard and instead of following managed jenkins all you have to do is click on build executor status and that takes you right back to the same page so let's go ahead and get started creating our agent i'm going to click on new node for the node name i'm going to name it agent one now this is up to you you can give it a name that matches your host name i see that a lot or maybe you just give it some other random name it doesn't matter this is up to you it's just a name in my case i'm naming it the same as the hostname of my machine i'm going to go ahead and click on permanent agent because this is going to be connected permanently to my controller once i click on ok we get to the agent detail screen i've got agent 1 that's what i just entered in on the previous screen the description i'm going to give it a description of this is my great agent okay number of executors by default this is one here's the rule of thumb typically you only have one single executor assigned to an agent in my case my agent instance has one cpu in it so it sort of makes sense to have a single executor map to a single cpu but let's say for a moment that you have real bare metal hardware and you have 40 cpus available on that machine or maybe you have a virtualized machine that has four cpus available to it or four vcpus what should this setting be now well in the bare metal example where i have 40 real cpus would i set my number of executors to 40 maybe would i set it to a thousand no way so here's the rough rule of thumb never have the number of executors exceed the number of cpus or vcpus available to you on the agent so my case of 40 the maximum i would ever put in the number of executors is also 40. in the example of a vm where i had four cpus it's going to depend because maybe in reality the type of workload that i'm going to be putting on that agent is going to be multi-threaded c plus and my build tools will actually use those and i don't necessarily want more workload to be added to that agent so rule of thumb keep it one if you have extra capacity then don't exceed the number of cpus or vcpus available to you on that agent and let that be a starting point if you see that okay even though i have four vcpus and i've set it to four executors i am maxed out at 100 all the time then i might want to back that down from four to maybe two and see how it works or maybe it's boy i've got four vcpus and running all the time i've got jobs queuing up but i'm only seeing about 25 to maybe 30 cpu utilization in that case i might break my rule that i was just talking about keeping the number of executors no greater than the number of cpus i might bump that up from four to six and just watch and see how things work all told if you can keep the executor account the same or less than the number of vcpus or cpus on your agent that's typically where you're going to stay again you're going to measure and make adjustments as necessary in the example today i'm just going to stay on one for my remote root directory i am going to put in home slash vagrant this is a vagrant-based box and i'm going to type agent now i could call this uh jenkins agent because this jenkins agent whoops forgot the slash this jenkins slash or dash agent doesn't exist this folder would be created for me home vagrant already exists but the jenkins agent doesn't exist also what i typically like doing is when i'm going to be logging in as a user i don't want to have all of the tooling end up in the root of that user directory so i always push it down one level so in this case my user home directory is home vagrant and i create an agent directory or in the case that i'm showing here jenkins agent i'm going to go ahead and change it back to agent that way everything that's associated with the agent from a software perspective is installed into the agent directory labels so i know that this is a centos 7.9 box there's a couple of other things that are on this box so let's think about labels here i know it's linux so i may give it just logically a linux label i also know that docker is installed on this agent so i put in a space and typed docker two more tools are actually installed on this i actually have a maven installation on this machine and i also have a gradle installation so all of these labels can be added onto this agent you can have many fewer maybe just one this is up to you but make labels oh one more thing one more thing that might be interesting you might also want to make a label that is just the hostname because you might want to target not all the time but maybe hey i need to test this on the specific agent i could target it by using agent one whereas if i had multiple agents connected to this controller and i gave them each linux docker maven and gradle because they're all basically clones of my agent one let's say i had an agent one agent two agent three and agent four but i'm going to test something specifically on agent one i could use agent one now jobs shouldn't use that as a normal ongoing way of doing jobs but as maybe a release valve or an escape hatch i could target just that single agent okay let's get down to our usage now with usage that means i can use this node as much as possible or only build jobs with label expressions matching this node so when i create a pipeline job and i say agent if i just said agent any only bill jobs with label expressions matching this node that means this agent would never be picked up with agent any so once we get to our example in a few moments we're going to make this use this node as much as possible now let's look at our launch methods there are three here by default there's launch agent by connecting it to the controller there's also launch agent via execution of command on the controller and launch agents via ssh which is what we're going to do today the first option though is the ability and you'll usually see this a lot when you're connecting windows agents that don't have ssh available on them this type of usage launch agent by connecting it to the controller is where the agent process starts and then connects back into the controller so it's an agent to controller flow to make the initial connection whereas what we're going to be doing today we're launching agents via ssh this is where the controller initiates the connection to the agent so in our case the host is agent one now the credentials i'm going to create a new credential here i don't have one set up yet i'm going to use username with password typically with ssh though i would probably use ssh username with private key just because but today just for our example i'm going to use username with password so i've got a username of vagrant i have a password it's also vagrant in this case id and description all vagrants so now taking a look at credentials i'll go ahead and select vagrant the host key verification strategy we're going to come back and revisit this in a few moments but to start with today i'm going to have a non-verifying verification strategy again we'll come back and look at that in a few minutes let's go and click on advanced we're not going to make any changes here but let's say that your agent via ssh does not listen on standard port 22 for ssh maybe all of your standards within your organization the ssh ports are 2222 or some other random number set that you use this is where you would change that setting value to where okay i'm going to make the connection from my jeans controller into my agent i'm doing it via ssh so what is the port for ssh now java path jvm options all of these extra things we're not going to get into today but if you needed these sections this is where you would add them for the connection directly from your controller to the agent that's the only thing that these settings affect they do not affect directly how the controller is running now also on my controller i've installed the adopt open jdk 11.0.11 that's the jdk i have running the jenkins process on the controller i also have that same jdk adopt open jdk 11.0.11 installed on my agent so both of those versions are the exact same versions on both my controller and on my agent i've also set up java home on both of those both the controller and the agent and set up the path to be java home bin right at the front of the path so having the java home setup having the java home bin on the path having the right versions together that helps me make sure that if i have any issues when i'm making these connections more than likely i know my issue isn't going to be the jdk because i have the exact same versions of the jdk running on both my controller and on my agent it's a really important thing to do if you can keep the version of java on your controller the same as the version of java on your agent when that connection is made if there are problems more than likely it's not the jdk it might be something else all right let's get back to setting up the rest of our agent here for our availability we're going to keep this agent online as much as possible since this agent is permanently connected to our controller we just want it online all the time but here's an option that you might want to consider bring this agent online according to a schedule there may be times where you want to take this agent offline maybe on weekends there's backups or other things that happen and you don't want any jobs to be potentially scheduled to run on this agent you could set up a schedule that basically takes this out of service during that window and then puts it back in service when whatever it is that you're doing maybe you're doing backups maybe you're doing upgrades whatever a maintenance window may be for you and that way you know jobs aren't being scheduled to this agent at that time but we're not going to do that today we're going to keep it online as much as possible now there are some other options down here under node properties if you wanted to add set up some separate environment variables that are created as this agent is connected to from the controller you can do that let's go and click on save let's watch what happens here you can see that it's launching down here i'm going to go ahead and click on agent one and as you see right now it appears to be connected and it actually is we can see our here's our description here are our labels docker gradle linux maven let's take a look at the log so we can see here as it started up we made the connection via ssh with a non-verifying strategy and made the connection to agent 1 on port 22. notice this warning ssh host keys are not being verified man in the middle attacks may be possible against this connection this is what we're going to revisit again in a few moments but as we connect you can see the authentication was successful it renders out all of the environment variables and in fact you can see here that here's our java home directory with ben we can also see that maven is on the path and we can also see gradle is on the path double checks what version of java is running and we get down to the bottom and you can see that hey our agent is successfully connected and online so now that we've got it connected online let's go back and make a change to our configuration and what we want to do is take care of this non-verifying we want to get rid of this warning so let's go back to configure i'm going to scroll down to the host key verification strategy and i'm going to change this to known host file verification strategy what does this mean if we take a look at the help this controls how jenkins verifies the ssh key presented by the remote host while connecting so non-verifying means no verifications at all it just connects ignores everything else but now we want to do a known host so i'm going to click on save i just clicked save we have not actually disconnected and reconnected yet to the agent to initiate that known host verification so how do we do this i'm going to click on disconnect i'm going to say yes so now i've disconnected the agent 1 from my controller let's click on launch agent let's watch the log here for a moment we can see here that it tried to verify var lib jenkins which on centos that is the default jenkins home directory it's looking for a known hosts file it wasn't able to find it and then basically it's just trying to reconnect and it's doing a countdown from 10 where we have nine left so how do we do this how do we fix this to make sure that we're actually making this connection well i have a shell open here to my controller and let's see who am i right now i believe i am vagrant so i'm going to become root because in var lib jenkins my vagrant user does not have access to that if i cd over to var lib whoops when we take a look at that we can see that jenkins is owned by a user called jenkins so let's go into jenkins here and here is our jenkins home directory all the files within that directory are owned by jenkins so how do i want to do this let me grab some notes i'm going to do a make dur for my jenkins home dot ssh because the dot ssh directory does not exist here then i'm going to run the command and copy it here here we go i'm going to do an ssh key scan giving it a dash h of agent one so that's the hostname agent one i have my dns set up to be able to resolve agent one and then i'm going to take the output of that and put it into the known hosts file in the jenkinshome.ssh directory and there is the connections so if i was to let's see where am i if i was to ssh slash known hosts then i can see the entry for known hosts so that's cool almost let's check one more thing if i do an ls here you're going to see that ssh is owned by root but since everything else here is owned by jenkins i want to change the ownership of my dot ssh directory so let me go and grab one more command which should have just typed i'm changing the ownership to jenkins jenkins of var lib jenkins.ssh which will also change the oops dot ssh ownership of that directory and that file so now we have a known host file in the dot ssh directory look what happened here let me scroll up just a little bit because we were working during this time so we kept on failing it was retrying every 15 seconds we got down to three it got down to the next check searching for agent one in known hosts and then what we see here is that the ssh host key matches the key in known host file connection is allowed so once we have that set up we see what we saw before with the non-verifying strategy we have our environment variables it goes through it copies over the latest remoting jar and then finally we're showing that the agent is connected and online so let's go back over to our controller and we can see that agent is connected with one executor available now there's one more thing to see this master block that's setting under build executor status this means that we have the ability to actually execute bills directly on the controller and in general that's a really bad thing in fact in this version of jenkins lts 2.289.1 there was a new administrative monitor that was created for that saying building on the controller node can be a security issue you should set the number of executors on the controller to zero so what i'm going to do is go ahead and make that change i'm going to click on manage i'm going to change the number of executors from 2 to 0. i'm going to set a label of asdf asdf asdf and i'm going to change this usage to only build jobs with the label expressions matching this node so i'm sort of doing a couple of things here almost security through obscurity i'm sending a label that nobody should be able to guess i'm setting it to only match so if somebody was to add a label of asdf asdfasdf to their job it would still fail because my number of executors has been bumped down to zero as well so i'm going to go ahead and click on save now we can see if i go back to my dashboard that that section is now gone from the build executor status now another way i could get to that is if i go to manage jenkins i could also look at configure system and at the top of configure system is a small section that has those same values for number of executors labels and usage so i can get there in a couple of different ways i can either come through configure system or i can come through this object right here to where i can then make the change through configure all of those map back to the same place okay now that we have everything set up let's go and create a job and make sure this agent is working so i click on new item i'm going to call this test pipeline pipeline there we go and pipeline and click ok and i already have a small job set up which i'm going to paste in here i'm saying pipeline agent any remember we set up our agent 1 agent to always use as much as possible not matching build node so i can use agent any and then i know that i've got maven installed on that agent so i'm going to run a maven dash dash version and let's see what happens let's click on save and click on build now jobs running let's review the log here we can see here as it starts up running on agent one so this reference to agent one is the name of the agent itself not necessarily the host name it happened to be in this case but this is the name of the agent not the host name of the agent even though like i said it's the same in this case and then when we run maven version we can see that it's a maven 381 that's what we're expecting and it's also using java version 11011 which is the only version of java that i have installed on that agent so at this point we know that agent is set up correctly we're connected to it and we're able to run a job against it to make sure that it works as expected and that's it that was a quick walkthrough of how to connect a linux agent via ssh to a jenkins controller now if you have a windows agent and you have openssh or some other ssh available on that windows machine it may be possible to connect via ssh however most of the time i would recommend using the other method to where you have the agent connect directly back from windows into the controller if you have any questions or comments you can reach out to us on twitter at cloudbeesdevs if this video was helpful to you give us a thumbs up and if you haven't subscribed to cloudbees tv yet why not take a moment click on the subscribe button and then ring that bell and you'll be notified anytime there's new content available on cloudbees tv thanks for watching and we will see you in the next video you

Channel: CloudBeesTV

Views: 8,894

Keywords: darin pope, jenkins, jenkins tutorial, create jenkins agent, create jenkins slave, jenkins master slave architecture, jenkins master slave configuration, jenkins node configuration

Id: 99DddJiH7lM

Length: 23min 57sec (1437 seconds)

Published: Thu Jun 17 2021