30 Jenkins features and plugins you wished you had known about before! by Joep Weijers

[Music] hello everybody the timer has started running so I'm quickly gonna start I compiled a list of 30 Jenkins tips for you on all different levels and I'm gonna present those to you one tip a minute my name is Elias I'm the build master topdesk my job that developers have the infrastructure to do their job and as part of my job I'm messing with Jenkins quite a lot and I want to share some of the steps with you so I also categorized tips in three levels if you see this icon in the top right corner it's meant for well the really the basic users - read-only users they only go to Jenkins if a job fails occasionally start a job don't you think it's a lot then we also have the power users they know their way around Jenkins and they are comfortably with creating their own jobs and then we'll have to Jenkins masters typically the Jenkins administrators or the go-to guys who always know a plug in for your specific use case when you ask them so here we go tip number one if you're gonna set up Jenkins use docker there are some repositories for Ubuntu that you if you add the package list and make sure that Java is on your path then you can install Jenkins using apt let's not do that just do a one-liner docker command and you can see I use a volume mount here that is to persist all the job configurations in between the different docker container runs keep an eye in the log when you execute this command because there you will find one-time password to login to your tangents tip number two use the Jenkins slash Jenkins docker images there's also a library Jenkins in docker hub and that's deprecated the changes let Jenkins repository provides weekly releases and long-term service releases number three there is a first-time installer and it looks like this first you need to unlock it with the password that we got in tip number one and the next screen you are presented is which plugins do you want to install people have a lot of plugins in Jenkins and in this case I would suggest to always choose the right option because it will pre select all the preferred plugins anyway so even then you have chance to review them I don't know how many of you are still running Apache and/or are using subversion those are some of the plugins that are suggested maybe you want to uncheck those now this is a manual work we don't like manual work so can we automate this yes you can provision them in the docker image there is an install plugins shell script so you can run that written disk amount on against a running container and it will install the plugins for you but you have to do a restart of Jenkins and you do that by browsing to the junk this URL safe restart some other options are here available as well that are less safe another alternative is to create your own docker image based on the Jenkins Jenkins plugin and then you can provide a file containing all the plugins so now you have a Jenkins master running and by default all the jobs are scheduled on that master and this is fine for experimentation you can see we have to build executors here but as soon as you're gonna have multiple jobs or yeah a lot of jobs then this will not no longer suffice the Jenkins master is pretty busy serving the web interface to you so if you're gonna do a lot of big maven builds next to it and stuff will get slow so pretty soon you'll find that you need to go to the notes settings screen and add new notes using SSH or there is a web start using the jnlp and that requires port 50,000 to be open for those who were wondering in the docker command why I also opened up board 50,000 and new notes they will well add come become available in this list and you notice that there's also an architecture column here you can use different operating systems for different notes so if you want to run your end to end test against Microsoft edge fine take a Windows node number 6 adding these nodes in the GUI it's a manual step we don't want to do that there must be some automation for that alright and there is it's called the swarm plugin it's little surface that you run on your agents and that will search the network for Jenkins masters and will automatically register itself to their Jenkins master I'm not going to go really deep into this great plugin documentation now there's still a manual step involved here namely starting up this swarm plugin service on your agent can we automate depth of course we can and i really suggest that you really fully automate your agent provisioning and you should also make them ephemeral so the automating and the agent provisioning that's gonna save you a lot of time in the long run when you're gonna add more and more and more and more agents you don't want to do that manually and these agents can be donker slaves that's been up on demand that automatically or instable provisioned virtual machines fine automated make sure that if an agent goes down for whatever reason that's the ephemeral part make sure we can just throw it away and spin up a new one that's also helps you help you with scaling now we are done with setup we have a Jenkins running with several agents and I'm going to look at the jobs and these are really the stuff that really works does the heavy lifting for you tip number one don't use the Maven job by default you have a freestyle project available in Jenkins and if you installed maven plug-in you will get a maven project as an option when you create a new job as well and and again if you can read it they promise to drastically reduce the configuration that's true but they also drastically reduce flexibility and they well I guess they had some kind of magic they do things with hooks whenever a maven project fails it's pretty hard to reproduce locally because it's really different what happens on Jenkins and what happens when you do a maven run locally so don't use this even better to use our pipelines pipelines are introduced last year with the coming of Jenkins 2.0 and it's Jenkins take on job configuration as code so a pipeline it's another type of job it's orchestrates long-running activities that means that you don't longer only do the build steps and the test steps but you can also model your release and deployment steps in Jenkins our standard way to set this up is to have a Jenkins file in your repository and that looks a bit like this this is a HelloWorld jenkees file and may have a lot of steps in there but I'll get to those in the next tips so for the pipeline's we also say do all the work on agents and how do we do that that's the second line we have the agent directive you can specify any or labels or well the exact choices you should look them up in the documentation but make sure that all the heavy work is done on agents not on the master except when you're waiting for user input it's a feature of the pipeline you don't want to do that on an agent this is how it looks like and this will block the build that's in progress which is what you want because you want to wait for the user input but it also blocks the executor that the job is running on so if you have five executors and you have five pipelines waiting for user input you can't do any new stuff so for this is the exception where we say run this part on the master and you do that by specifying in your pipeline that you want to run it on the non agent it's means run it on the master on a little flyweight executor that's hidden from the GUI but it's there now of course in any stage that do it and that actually does work you have to say pick a suitable agent please not the master except when you're in a stage set it's user input and you really need to specify do this on the master now let's talk about stages status are logical groups of steps so use them as such there are some example stages from a build and these stages you can visualize them nicely in Jenkins in the pipeline so the tip here is to use them but also to keep the amount of stages small and also their names keep them very long names the else it won't fit on your screen and you really want a nice overview in one screen tip number 13 don't mess with environment variables what's don't use fire environment variables no don't mess with them in a pipeline the dollar and environment variable is exposed and this is fine to read from but don't go messing around trying to override the path in this variable because it's used globally in your your pipeline so stuff will break if you mess around with this what are the alternatives there is a with NF directive and there you can specify hey for this next bit of code provide these environment variables so there you can use this to scope your environment variables as well and there is a environment directive in the top level of the pipeline and that will make the environment variable available in the entire pipeline parameters you can use them in a pipeline as well they're not really easy to use but I'll give a quick bit of code to show how you use them you define them and in this case I read the value from the bar parameter and put it in an environment variable so I can use it in my in my scripts but the big downside of these parameters is it's an ordering problem if I start a new build buddy fault it will have no parameters so it will just start it will read the changed Jenkins file it will know to notice hey I need some parameters I should have asked for them and it will just continue to build so you need to specify sensible defaults for your parameters or the first built that you introduce them will fill and when that as filled or has run successfully once then the next time you hit build this pipeline you'll get the interface for hey what are your parameters for the you want to fill in there so make sure to use sense for the sensible defaults when you introduce parameters parallelism is an important part in the pipelines as well they really thought about that and there are two ways of parallelism that I'm going to show you the first one is to paralyze long running tasks over different notes so for example if you have your end to end typically they are slow take a lot of time starting up the entire application selenium all that kind of stuff and your performance tests well you also when I was running for so quite some time well these are perfect candidates to execute in parallel so you define it like this and then as the extra tip make sure that all the work with in this parallel step is done in separate notes as well so you can really utilize the parallelism to also distribute the work amongst your notes next up is it's gonna be a bit harder readable but there's also a way if you have a big maven build to parallelize maven tests there is a split tests function and what it does it will look at the previous run that has actually executed in your maven build and will look at the run times of the tests and then it will try to put those in two chunks in this case two to make sure that they are the run times of those chunks are roughly equal and then you have to do some fancy stuff to make sure that maven gets the exclusions so that it will only execute the test set today and that belong to the chunk that it runs and this then done in parallel and this was even the full example just fits on a slide that's it's not really easy to use when you look at this but it's a great functionality if you have a big large monolith and still put that in a pipeline tip number 16 there are two different syntaxes of pipelines one is the called declarative and one is called scripted script it was the first well basically the first attempt at pipelines it was a bit more groovy ish it's both are a groovy DSL but the script that requires you to program more basically and it was a bit more boilerplate so I wrote down some pros and cons here and all in all in my opinion the declarative pipeline is perfect for the regular users that's what you should first try to use any scripted pipeline gives you more flexibility for the power users so I put the power uses on both these sides typically they'll start with a declarative pipeline then run into something that's yeah we can't do that in declarative I need scripted and thankfully you can put little chunks of scripted pipelines into your declarative pipeline so that will help you here as well you should also keep your pipelines small and why is that the pipelines are meant to be a orchestration tool they are not meant to do heavy lifting as soon they are groove it is else so you can really program a really big pipeline we have one of five hundred lines of groovy and it's not that nice but you quickly do some difficult tasks in groovy that will hit the limits of the groovy sandbox that Jenkins utilizes by default to prevent malicious code from executing on your server so what is the alternative well if you do all the heavy lifting steps you should put them in shell scripts and that has another advantage as well namely you can test them out locally a pipeline is really hard to test locally now we have another Jenkins master topic this is a bit harder topic the non cps annotation inside a pipeline to ensure robustness everything is serializable and this is done to store the intermediate states of every step in your pipeline so that when a master or a agent goes down you can resume your pipeline and the developers of jenkees use the continuation passing style programming paradigm here continuation passing means that instead of when you call a method instead that it returns its value you pass a continuation to the method and at the end of the method it will call the call the continuation with the result as a parameter sort of like callbacks now and there is a possibility that you have to use non serializable classes in your pipeline so here's an example the groovy streaming template engine that's not serializable but we do want to use it so the option that you then have is to wrap it in a method and annotate that method with at non cps this tells to the pipeline hey you don't have to persist this intermediate result and as a result it will not do that so when the master feels when it's busy with this calculation it was just redo it again but typically that's not a problem for our groovy pros I did not use a simple template engine used streaming template engine simple template engine somehow somewhere leaks and non serializable variable that I was able to capture so it breaks the pipeline well if a master goes down sometimes the pipeline gets stuck and it's a bit harder to stop the pipeline than stopping a regular job a regular job would be stopped by hitting this little red cross but if you do that on a stock pipeline you will see this in the console one of my colleagues tried to stop it four times it not works we have to intervene you have to when you press the button you have to go to the console and forcibly terminate the remaining running steps and even kill the entire build if that still doesn't stop them then well we're getting there two thirds of the way the blue ocean user interface I guess most of you who've seen Jenkins think well that looks a bit dated and you're right I agree and the folks of Jenkins also figured that so they introduced a new user interface it's still in active development but I first release is done so this is how your job overview looks now they're the pro or the power users will say hey that's not a fine list to look through and the overview that's not for the pros but more for the regular users now if I click a pipeline you can also get a nice overview of the previous jobs that ran and it looks pretty spiffy but now it's the point where you should really get excited this is the pipeline view that's really a major improvement over the little square blocks that you had everywhere so in this example pipeline you can also see that the parallel steps are shown in parallel and it really looks more modern the blue ocean GUI also includes a pipeline editor for declarative pipelines it has some requirements so your the credentials that you use to check out your Jenkins file they should be right credentials so you can also write any changes to the pipeline back to your versioning system and well when it writes it back it will do some reordering and you'll lose your comments so those are the trade-offs in a pipeline should you be doing more with it and I suggested keeping your pipelines small then you may want to extract that code if it's used in several pipelines for example the annotated method and in non CPS annotated method that I showed earlier that was a piece of code that would show up in or 20 pipelines in our company and Jenkins also has a feature called shared libraries to pull that code out put it in a shared library and then you have one central place to manage it so the Adagio here is don't repeat yourself these shared libraries can also contain files the example was about preparing a deployment Yemma file for kubernetes our pipeline our shared library now contains a standard template that most of the jobs use so also don't repeat yourself with regard to files views is a nice topic on its own but you can build your own view with only the jobs that you are interested in so you go to the my views in the menu and there you can add new views and when you edit that view you get option to select which jobs you want to include and there's some kind they also have regular expressions so in this case I want all jobs that are start with project a and if you will look something like this indi old Kiwi now these views are really useful because I think that they can reduce the amount of emails that Jenkins sent emails are pretty easy to ignore and we noticed that well people ignore their email so bills don't get fixed so we need another way to notify people that their build is failing so we installed the build monitor plug-in the built monitor plug-in will give you a new entry when you create a new view and it looks like this you select your jobs and it will give you a nice information radiator with all your jobs on it and you can fit quite a lot of jobs in there so I also took in live screenshots did you know it's pretty easy to get an overview all the teams in our company have a big screen in their rooms with the builds monitor on it Jenkins also has an API you browse to whichever page you like and you just add a slash API on the end and then you'll get a bit of documentation about the page and how to use the API so an example is this call the API is available in JSON or XML you can with jason api you can filter which elements you want to get returned so in this case i query all the jobs and I want to know their colors and the colors in Jenkins represent yellow is it's a stable build blue is that the build is green to build is fine it's finished and the blue and score anime its animated so it's the blinking ball in Jenkins which means the job is in progress why did I choose this example because you can also do some extreme feedback with Jenkins one issue we noted with the build monitors was that people only look at them when they get up go grab a coffee which is regularly but you don't get the instant notification that hate this build is failing so I built something for that so this a little warning light it makes a lot of sound because it's cheap and it will notify people that hey something is going on on the Billboard you should have a look so I also want to talk about cloud beasts the company that's basically they adopted Jenkins they they do a lot of they contribute a lot to the open-source version of Jenkins but they also provide enterprise features that's how to get their money they also created a certificate so you can if you really think that you have what it takes to become a Jenkins master you can become a certified Jenkins engineer so what are some of those they're features they added some easily easy scalability for Jenkins masters they have a Jenkins Operations Center where you can easily spin up new masters new agents there's also high availability and you get more fine-grained security options for example role-based access control and they also added some extra monitoring making it nice overviews of how yeah the load is on your masters so you know if you need to spin up new masters you can probably do this all in the open source Jenkins as well with loads of plugins but cloudBees has made it easy for you now to wrap up in the last couple of minutes I compiled a small list of my favorite Jenkins plugins that I think everybody should have in their Jenkins environment we already talked about the build monitor the job config history plug-in brings you versioning to your job configuration so typically your pipeline is in version control so that's version but your freestyle job isn't so with the job config history it becomes versions that you can roll back if people screw up then there is a job DSL in domain-specific language to automatically generate jobs it's something that we use a lot with our end to end test jobs that are separate jobs for certain kinds of functionality so we manage around 100 jobs with this the throttle coupe current builds plug-in will give you relief and you can really fine-tune how you load your agents typically it will just if an agent has 5 executors will run 5 builds there don't really regard if there's another agent that is doing nothing at the moment so with the throttle current bills you can specify categories and make sure that the load is more nicely distributed the time stamp a plug in make sure that all the console.log lines are prefixed with the time stamp and the version number plug-in allows you to really tweak the version number in several ways for example our main product the versioning scheme is the year dot the month thought the build in this month and the first number plug-in provides us with that and we team it up with the build name setter to make sure that you don't see just the build number but you actually see which version was built in all the overviews so these were my favorite plugins and actually these were all 30 jenkies features and plugins I would like to share with you if you have any questions regarding Jenkins the stack overflow Jenkins tag it's pretty active there quality a good quality answers there you can contact me directly via Twitter and we also well compiled a lot of technical talk on our blog tech blog topdesk dot-com so that's it thank you very much [Applause] [Music]

Channel: Devoxx

Views: 56,554

Rating: 4.9465022 out of 5

Keywords: DV17, Devoxx

Id: 6BIry0cepz4

Length: 30min 13sec (1813 seconds)

Published: Wed Nov 08 2017