Jenkins Pipeline Tutorial: 5 Common Jenkins Pipeline Mistakes 🔥

hello everyone and welcome back to my channel where it's all about helping you to become a better software developer today i want to share with you five common jenkins pipeline mistakes and how to avoid them these are the mistakes many people made including myself this is not the ultimate list of all possible mistakes and let me know in the comments section what are the other mistakes i didn't mention in this video without further ado let's jump to the first mistake which is not storing the pipeline code in the jenkins file there are just a few exceptions that justify building jenkins pipeline directly in the jenkins job ui and these are like starting a poc or running some experiments in these cases you may want to just put some pipeline code in jenkins job page run it and see how it works however at some point you want to extract it and put it into jenkins file so you can store this file in the git repository and version it correctly using jenkinsfile has also other benefits for instance you can use the ide support to build your jenkins pipeline and if you don't know which ide to use today i want to show you one free ide that you can install on your operating system and this is visual studio code this is free to use it works on every operating system you can go to codevisualstudio.com and download it on your operating system when you do it you can open jenkins file in this ide and there is one extension that i would like to show you that will help you a lot in building jenkins pipelines let's go to extensions tab and let's search for jenkins lint and here you will find jenkins pipeline linter connector this is a plugin that uses built in jenkins pipeline syntax validation that jenkins server offers let's install it i will restart this ide so i'm sure that it shows up in the settings section so now i can click here on settings we can go to extensions and here we have jenkins pipeline linter connector settings there are a few things we need to set up here i will use my local jenkins so let me set it up with the correct uh urls so i will use http localhost 8080 here here i will use my url now i will use my username and here i will use my password okay i can save it now i can close it and now what happens is if i do ctrl shift p here we have validate jenkins file and you can see there is also the keyboard shortcut shift alt v let's validate this jenkins file and now you can see this is the response from my jenkins server that says missing required section agent right yes this pipeline has missing agent so let's say agent fu something like this i can validate the pipeline again and of course it says only agent known agent any or agent block are allowed right so let's set agent any and now i can validate it again and now i have the response jenkins file successfully validated this way i can save a lot of time and i can always validate my jenkins file before i will commit those changes to the repository of course this is only the syntax validation and even though it's very useful it cannot protect you from introducing other mistakes to your jenkins pipeline luckily there are some other tools that can help us and this is the moment when we can jump to the second mistake which is no unit test for jenkins pipeline this mistake is still common because most people probably don't know that there are some libraries that allow you to test your jenkins pipeline automatically and let me show you three very useful tools the first one is jenkins pipeline unit this is the library for junit and you can use it with spock framework as well that allows you to unit test your jenkins pipeline and what this library does is basically it mocks the execution of the jenkins pipeline and jenkins server so here there is no jenkins involved this library simulates the execution of the jenkins pipeline on the jenkins server okay so you can use this library to various things for instance you can mock execution of specific steps and then you can verify that specific condition happened so this is the next step after syntax validation which could only check if you didn't make any obvious mistakes at the syntax level this library allows you to execute your jenkins file in a groovy shell that tries to simulate the jenkins pipeline execution okay and for us as here you have a few examples in this readme file here for instance we mock what specific sh comments return and you can see that this test sh shell script returns exit code 1 which means that something failed and you can see that this test verifies that in this circumstances the job status is unstable and it's not limited to those very simple examples so you can mock what the comment messages what the branch name is what the environment variable value is and so forth so you can actually simulate and see what happens for instance if you run from the pull request you can verify that this specific stage should not be triggered for pull requests or if you run a pull request maybe you run some additional steps etc this is what you can test with this library without running real jenkins which is also very beneficial because you run those tests in your ide and you get the result in just a matter of seconds so you don't have to wait for the pipeline to execute all stages just to see that your assumption for the very very last stage is is broken so this is the library for unit testing there is also the jenkins test harness and this is the library that allows you to run more or less like integration tests here you don't mock any jenkins execution this library runs jenkins in a headless mode and it executes your jenkins file directly on that jenkins so if you prefer more integration like testing maybe jenkins test harness is library for you and there is also a third one which is worth mentioning and this is jenkins file runner this is the incubating project and this is a command line that allows you to execute your jenkins file in a jenkins as a function kind of context right so you can use it and you can see in your command line if jenkins file executes correctly this cli can be installed on your system but there is also an official docker image that you can use and you can just run it for your specific jenkins file and see how it works so these are the free tools you can use to test your jenkins pipeline if you are interested in this subject let me know in the comments section if you would like to see testing jenkins pipeline video in the future and right now let's move to the mistake number three which is ignoring jenkins pipeline best practices i have been watching jenkins pipeline tag on the suck overflow for a pretty long time and i can tell you that almost every day there is at least one question that could be avoided only if someone read the jenkins pipeline best practices page let me show you what does this page look like here in the pipeline documentation we have pipeline best practices and this page lists a few best practices things like avoid complex groovy code in pipelines and it explains a few alternatives for json slurper or http request there is one thing worth mentioning any code you put into the script block either if this is a groovy code or maybe you use some java libraries every code is executed on jenkins master server no matter what your jenkins topology is and probably you use some jenkins nodes and you execute stages on dedicated nodes and then if you put a script blog and you put some groovy code there this code will be always executed on jenkins master and this may cause some you know side effects you cannot predict and that is why one of the best practices is to use groovy code only as glue and only in a few places and the less groovy code you use actually the better this page also explains that you should avoid using jenkins get instance this is a huge uh problem that i see also on stack overflow there are also a few tips for building shared libraries and there is also a dedicated page for pipeline cps method mismatches and this page explains some popular issues with groovy cps this continuation passing style jenkins executes your pipeline code in the groovy cps mode and it uses dedicated groovy shell which is pre-configured to transform your code differently than the regular groovy interpreter and many people think that this groovy code in jenkins pipeline is almost the same as the groovy code its execution is the same as the groovy code in groov interpreter and this is not true and this page explains a few use cases where you can use this non-cps annotation to turn off cps mode in some specific circumstances and what are the potential problems you may face if you start mixing for instance if you start calling pipeline steps from the non-cps methods this page explains it pretty clearly so if you haven't read those two pages i strongly encourage you go to the description there are links to those two pages and you can read just to avoid some common mistakes in the future and now let's jump to the mistake number four which is not using timeouts at the stage level this mistake is common but it is also pretty easy to fix when you start building your pipeline you usually don't think about uh how much time specific stage may take and what happens if specific stage hangs for whatever reason right and we usually don't put any timeouts at the very beginning however if you want to eliminate the problem of hanging jenkins pipelines and blocking executors forever we should consider adding timeouts to every stage and let me show you how you can do it here at every stage level we can go and we can add options and inside options we can define timeout then i can specify let's say 5 unit minutes right so the timeout for this specific stage is five minutes i can go here and let's say this stage i can define two minutes right and now if something happens let's say i don't know there will be some connection issue and i i've seen this many times uh when the dependency couldn't be downloaded and the client that was used by some build tool didn't have any timeouts so the process was hanging forever now if the situation happens for instance if here execution of that stage takes more than two minutes the pipeline will stop after two minutes right it won't hang forever the same happens here if this maven install comment takes more than five minutes it will just break when it hits five minutes time right there is also one alternative solution you can always define options here and you can add timeout here something like 30 this is better than nothing of course because with this 30 minutes time of global timeout we will eliminate the problem of hanging forever and after 30 minutes this pipeline will be terminated however the problem with this setting is that this global timeout is quite large and imagine now you have a stage where it usually takes 30 seconds to complete right so you set the timeout to one minute and now if you if you use only the global timeout this stage that usually takes 30 seconds if there is some i don't know a connection issue and it hangs for forever then pipeline will wait 30 minutes until it gets down right and if you use timeout at the stage level and then when you know okay it takes 30 seconds on average then you can set this timeout for one minute and if it starts hanging after one minute the pipeline will will stop and the executor that was executing this pipeline will be ready to handle another pipeline job right so always remember to use stage timeouts you can add it let's say after designing your pipeline you just have to think about how much time specific stage needs and you can just set up those timeouts here this is a very good practice it will it will save you a lot of time and a lot of headaches believe me and now let's move to the mistake number five which is blocking executor and waiting for the input the final mistake i want to share with you today has also something to do with blocking here is a simple jenkins pipeline and let me quickly explain what we have here so we have a global agent we use agent with label docker we have some build stage right here we just sleep for 10 seconds let's imagine that there is some building in this stage happening then we have a stage that waits and asks user if he accepts uh the deployment okay it will wait two minutes and if there is no reaction from the user site the pipeline will be terminated and if user approves this deployment then it runs to the deploy stage in our case we also sleep for 10 seconds so we can see what happens on the jenkins server side okay and now the problem with this pipeline is the following every stage blocks the executor right uh depending on your jenkins topology you may have let's say five jenkins nodes and every jenkins node may have like three maybe four executors so you have between 15 to 20 executors available on your jenkins and now if you start blocking your executor here here's only here we only use two minutes timeout but you can imagine that there are two hours timeout and no one reacted to this approval so the executor is blocked for two hours and during that time this executor cannot handle any other jenkins job and you need to know that executors are pretty heavyweight so just blocking blocking this executor here makes absolutely no sense so ideally what we would like to have is i would like to use an executor here and whenever i ask for approval i want to do it outside the executor so i want to free this executor so i can wait for approval and once approved i can get back to the deployment and here i can use the executor again okay so now let me show you how we can modify this pipeline to achieve exactly that so first of all we need to remove this thing here and at the pipeline level we need to specify agent known this is very critical when you specify agent known right now we just say that there is no executor involved in this pipeline at all so we need to specify those uh docker agents at every stage level right so i will specify agent docker here and this agent uh docker here as well okay so now we can save this change we can comment uh just change to the repository and see what does it look like on the jenkins side here you can see that the pipeline is running and what i want to show you is this take a look right now there is this build running it just slips for 10 seconds however it blocked executor for those 10 seconds right now we can see that the executor is free and if we look what happens in our pipeline we can see that we are asked if we approve the deployment right i will abort it now and let's run this pipeline again so we can see that now executor is being used by the build stage so we will have to wait those 10 seconds now we can get back here we can see that the approval does not block executor and we can approve it to see that the executor now is used by the deploy stage and after those 10 seconds we can see that the job uh completed successfully so this is critical to remember that whenever you use an input stab and you open this window for user reaction just do it outside the executor let him handle a real job let him handle some other jenkins pipelines and you can just wait outside this executor scope and the final question goes to you what are the other common mistakes i didn't mention in this video let me know in the comment section below also check the description for all links and resources i mentioned today and if you like this video give me thumbs up subscribe if you haven't already and watch the next video from my channel thank you so much for watching and i will see you next time

Channel: Szymon Stepniak

Views: 10,162

Rating: 4.9832287 out of 5

Keywords: jenkins pipeline tutorial, jenkinsfile, jenkinsfile tutorial, jenkinsfile tutorial for beginners, jenkins pipeline, jenkins pipeline as a code, jenkinsfile syntax, jenkins groovy pipeline tutorial, jenkins, jenkins tutorial, jenkins pipeline common mistakes, jenkins pipeline best practices, jenkins pipeline vscode, jenkins pipeline timeout, jenkins pipeline demo, jenkins pipeline groovy tutorial, jenkins pipeline agent, jenkins pipeline agent any, jenkins pipeline code tutorial

Id: aFRjn_4nb-Q

Length: 17min 46sec (1066 seconds)

Published: Tue Aug 18 2020