Declarative Pipelines with Jenkins

alright hello everyone thanks for joining us for today's webinar about declarative pipelines with Jenkins to get started I just want to let you know a few things up top one we are recording so if you need to leave early or maybe have a colleague was unable to join they will be able to view this later on - we are gonna be able to take questions at the end we should have a fair amount of time for questions so please as soon as you think of something go right ahead and ask it even if you think maybe we'll get to it during the presentation we'd rather you ask so we know what people want to learn more about then not asked at all there's also going to be a very short survey at the very end just asking what you saw and what you like to learn more about beyond that I'm gonna introduce our speakers in just a moment and for anyone who is attending DevOps world and Lisbon coming up in about two weeks you'll be able to meet myself mark and KK all at the event and asked whatever kind of questions you want about this in person so once we get started I'll be posting a registration link as well as a discount code for the one who maybe hasn't registered yet at this time that further ado though I like to get us started so I like to introduce someone to get us to to really kick it off and I think everyone here is gonna know kohsuke kawaguchi the Jenkins creator KK when you get started Hey all right so I guess the good morning good afternoon everyone so today eat so we have two excellent speakers the mark and Tyler you know this is one of the things I'm very proud of about cubbies is we have such a deep bench of people who've been perhaps the drinking too much Jenkins I know the water hose for they in there today they're going to talk about the next slide please so today they're going to talk about the thinking solidarity pipeline so let's move on to the next slide as well and then just a little bit about myself so I created Jenkins at this point it's like a 15 years ago I mean forever Ascari's and I have a lot of and responsible for all the things in thinking store the problems and thinking's including the originally what became the Jenkins pipeline so I wanted to tell you a little bit the story from there at the kanaky castle the next file so I'm I still remember the week I got into the room we saw Jesse click in Los Altos glad these obviously start writing what eventually became the king spike line and in a Pathan dish not a lot of timing the law I think his pipeline has a beautiful note the DSL the domain stretch language on top about general-purpose programming language so what's good about it is it gave the people quite a bit of power but at the same time over the time we came to realize that we didn't guide people well enough so too many people are failing to tame it properly so it's just reviled in the eyes of too many people so next slide so now what we came to realize we wanted to give our users something that's more pain more delightful like easy to handle and adapt and so on and then the key I think he's in layering in other words and if you look at the things that we do is like a CPU well at certain level it is running assembly language but that doesn't mean like you programming those rights it's the same idea so we wanted to create more and sometimes we call it the deep pass but the guided way of helping people using his pipeline and that became the decorative pipeline and now it's being what I think it's been so many years since we've been doing it but we constantly keep getting surprised about this you know how people don't know that they have this wonderful things in there or a toolbox that they use every day in and out and then that's gonna make you so much more productive and faster so that's I believe is the story of today so on that note let me hand it back to mark I think thanks Kay Kay thanks so much I agree with Kay Kay pipelines are easy to create and easy to extend and sometimes we've discovered that our users of Jenkins just aren't aware of how simple it really is to use Jenkins pipelines in particularly Jenkins declarative pipelines we're going to show you today some of the simplicity some of the elegance some of the ease that there is in you working with Jenkins pipelines declarative pipelines with Jenkins so it's all about how do we make it easy so Jenkins has been used for a very very long time to create and deliver great software and it started with the concept of freestyle jobs and grew to the to add multiple freestyle jobs and even matrix jobs so that we could evaluate many different combinations and we would string things together we would combine one thing to follow on to the next thing to the next thing and it ended up that we were creating something that was quite brittle by having these chains of jobs strung together well pipeline is a better way there is a better way to do software delivery with Jenkins and so I'm going to do a first preliminary part and show how we'll create a skeleton declarative pipeline using the Jenkins blue ocean interface it will let us define the process simply and clearly store the definition right with the software development and see it all from the UI after creating the skeleton I'm going to show you how to build and test with it so do real build real test and then we're gonna add tests reporting to it and that will help us keep it simple so that we built like developers do and test like developers do so let's switch and we're going to go to a real real real Jenkins system so here is my Jenkins environment I'm it looks like the Jenkins user interface because this is Jenkins and I'm going to open up the Jenkins blue ocean interface blue ocean is provided by a Jenkins plugin if you don't have it you can install it easily from the update center you'll notice here that the first thing that blue ocean says is ah don't have any pipelines would you like to create a new pipeline so I'm going to do that I'm going to create a new pipeline now blue ocean already presents these five different choices of ways that I could use to store my code in my particular case I'm gonna use git I've actually got readily accessible to me a copy of giddy a free git server that provides all sorts of capabilities and so I'm going to copy this URL to my git server here you'll see what you see in this is a simple java application notice that it's got a pom file so I must be maven it's got some source it's got some tests and it's got a small directory of build scripts that we use for it so I'm gonna paste that repository URL there and I'm gonna click create pipeline notice that I haven't opened up my text editor I haven't done anything beyond simple clicks on the UI so the pipeline will begin creating now let's take a look inside this repository what's happening so this is a repository with source code inside the source there is the source of the application itself and their tests as well here in the source slice test directory and I've got some build scripts in it now if we check this out we should see that Jenkins has huh it tells me right here notice there no Jenkins files were found in this repository and it offers to let me create one here I have a nice interactive UI from the blue ocean interface I'm going to create a new stage this stage we're going to call it build and in that I'm going to create a skeleton entry which is just a print the message building now why do a skeleton why not do the real thing because a skeleton lets me visualize very rapidly if my pipeline is doing what I want so we're gonna say building we're gonna create a stage for test and we're gonna do the same thing print a message Tesla I again the same thing for deployed and we're gonna print a message deploying well that was a pretty simple thing to do I've now got a three three stage pipeline build test and deploy pretty common things that happen in a software development process I'm gonna click the Save button here to store that directly back to my git repository and a skeleton pipeline save it and run blue ocean writes the file transmits it back to the git server and now we're going to go back to our git server refresh this page and there's the Jenkins file now I didn't enter any curly braces I didn't do any logic I didn't do anything more than something that is every bit as easy as what originally attracted me to Jenkins the ability to configure entirely from my web browser notice that here's a stage name build here's a stage name test and here's a stage name deploy and there are my little simple skeleton statements inside now let's go back and see how Jenkins is doing and what do you know we've got a build that's run it ran through the three steps build test and deploy it was that simple and that easy now we can we can expand these and look and see oh wow there there's my message that was output okay that's great but that's really just a skeleton okay let's go to the next level let's really build some code so I'm gonna go back to my editor in the build stage I'm gonna leave in that step and I'm gonna add one more step now it just so happens that I have a script in this directory called Jenkins slash build Sh this is whatever the build script developers would usually use in their normal environment it's not something special or specific it could be maven it could be ant it could be a call to make whatever your developers typically do that's what should be put here we intentionally do that thing so that we know we're testing the we're test building and testing the way developers build in tests so there's that one now we're going to I happen to also have a script that calls Jenkins slash build - or test - all tests - oh Sh and we've got deploy where we say to run a script stinkin slash deploy dot sh so I've now switched from using a pure skeleton to using something that will really perform operations so when I save this it's going to run this job it's going to invoke maven compile the code test the code and take now I admit the deploy step I've made a dummy but the others are real so let's do a save and add build test and deploy stripped calls again this will go right to my repository we'll be able to go look in the git repository and see just how how it went so notice it's already started the build and if i refresh this page let's go back to the top level look at Jenkins file and there's the build script call there's the test call there's the deploy call it was that easy now if we want to watch what's happening let's look at what happened in the build process so I invoked the build script and here is a whole bunch of now those of you who are maven users will recognize this it looks like this downloaded roughly half the internet in order to build this trivial little maven project and that's very much the way it feels and it shows me the reality of what it did it downloaded all those things and is running them now we've got tests that ran and here the tests show us it ran a test no failures no errors and none skipped and it read my little deploy step now the deploy I admitted is just a dummy it says deploying and does nothing it was that easy to go from a skeleton to actually building testing and deploy now one of the the niceties that blue ocean offers is across the top here it will tell us the things that some other things that are going on like for instance the changes that occurred so here we see the last change that it recorded was that I added build test and deploy scripts ah but what about my tests oh no tests archive because I need to add one more step to record and display the test results so I'm going to do that in my test step I need to add another step and sometimes I have difficulty remembering these things so I'm going to just use the word test and there it found the correct step I'm going to click this button to add an j-unit formatted test result display and I happen to know that my test results are in the target directory and I never remember how to how to spell the whole sure our things so we're just gonna say start on XML and get everything that our XML files there so now I've got print my message run all my tests and j-unit formatted test results let's save it and now so now record test results when I click the Save button and run it will launch off and run it again so let's go back and look in our get process our get server and let's see what we've got there it is it's recording those ready to record those results ready to show them to others now this will take a little bit because it's still got to run things so the running will happen it'll continue building and we'll be able to see test results shortly the idea here is that we want to keep it simple we really like the notion that simplicity is all about building like the developer builds use the same commands thus get the same results test like the developer tests same command same results and do it all from the user interface it's intended to be that simple make it that easy to understand let's look at that result and see how it's doing so we ran our tests they all passed we look at the tests and it says yep all the tests are passing so I've completed my piece timer you want to take it from here yeah I think that'd be a great idea thank you mark and I got a quick question I think for folks that are currently using freestyle jobs mark I've heard a lot about pipelines is it really this easy just to download the blue ocean plugin and bring my repo is there anything else or a gotcha that I'm missing there there really aren't more gotchas thanks to Jenkins own dependency management when you choose to download the blue ocean plug-in it will download the plugins that it needs it will make itself available to you and you can just run so it works great now there are certainly plenty of optional additions that may they may find helpful useful but exploring blue ocean is just as simple as as we've shown so far awesome and then is there a reason that I hear so much about pipelines over freestyle jobs because I've been using Jenkins for a while and freestyle jobs seem to meet my needs today but I hear a lot about pipelines why should I have migrated towards pipelines well for me at least pipelines have been a real positive because now instead of storing the representation of my build process in jenkins own configuration files i'm storing the representation of my build process right in my source control system so we talk about doing things as code and in this case pipeline as code means that my git repository has right inside of it the definition of exactly how I build my software including the Jenkins job that does the build so for me that is a very attractive piece of what pipeline can do for us makes a lot of sense thanks mark so I think what Mark has shown us to this point is is a really skeletal view of a pipeline and then we've added some mud to the frame so we're actually doing some real-world tasks but oftentimes that's not going to be the complexity of my pipeline it's not that simple I'm gonna need more complex operations to fit my actual development needs so what I'm going to cover is going to be adding parallel tasks so that you're not running tests only in sequence that way you can optimize your builds there's also a way to follow your branching strategies and add conditional statements that way there's only execution of some stages based on the branch on which it's being run from that way you can collaborate with other developers and then lastly there will also add a timeout to some of our test stages as well that way we can avoid you know hung test results or errors that might lead to misleading failures rather than just sitting on endlessly waiting for an execution to to conclude we can now add a timeout that way we know that our tests should only live for a certain amount of time in our pipeline so the first thing that I'm gonna do when we go back into blue ocean is we're going to add a multitude of tests I'll add a smoke test for integration and performance testing as well we'll also add a conditional to the deployment step that way it will only execute when we run that specific pull request or that new execution of our pipeline on the master branch this will also cause us to addy a development branch directly from the blue ocean UI as well and then lastly I'm going to add a time limitation for one of our testing stages that way I can determine whether or not it's going to fall within the expected amount of time when it comes to execution so what this allows us to do is keep it simple when it's easy to maintain all of my different tasks it's easy to incorporate multiple different development teams because we don't develop as an individual we really do have an organizational need to cooperate within our than our build tests and additionally it allows me to segregate the duties I may not want a developer to run an end-to-end task and deploy to production every time instead I might want them to be able to build and test their micro service or artifact before someone like you a manager or a release engineer which will be played by mark in this example then they can run the deployment to any environment that we deem necessary so I'll dive back into the blue ocean editor you'll notice that I can already see the pipeline that Mark has created I'm actually going to create another pipeline and I know we're using gitche for this example but I would also like to highlight what it looks like to run tests and sequence rather than just parallel so we can get a comparison between execution time and we can see how we can optimize our actual pipeline execution so for this I'll just add my github token you'll notice that we're also highlighting the github as a free solution as well you can use git github bitbucket github enterprise and bitbucket paid versions so it's really an extensible tool as well you can see it is now logged into which organization I want to use for my specific Jenkins file for the case of this webinar I've actually already created this Jenkins file and it is an exact replica of what we're going to be creating with our lost ages and from this terminal it will create is pipeline for me so just for an idea of exactly what it's pulling from this is the github page this is the Jenkins file and this is the exact same pipeline that Mark created previously you'll notice that I've actually added a couple of test stages which will add in our parallel steps within blue ocean so you'll notice that this will be exactly the same execution that we're going to run from the blue ocean editor as well and it is already executing so it's loading in the test scripts that mark our D outline so we can see our J unit tests as well from this tab and then it'll also add our performance and integration testing stages as well so as this slows it's it's important to note that this allows me to use multiple different editors because not every organization is going to be standardized on a single editing platform this allows your developers to have the freedom to use a tool that they're most comfortable with as well you'll see now we have executed our test stage our integration test and then in the performance I have a sweep command and it's followed by a timeout as well so that we know it'll execute within that allotted amount of time so as this runs I will go back to my pipeline editor and you will see now I have two different pipelines so I will go into the exact example that Mark had created earlier and from this page I'm now going to add the specific test stages in parallel rather than in sequence like I just showed you from our github example so I'm going to add this and I'm going to call it the integration testing integration test and we'll add a step we'll do a simple print message and we'll say integration testing and then I will add a additional step as well and this one is going to allow us to add a time out for our performance testing so in order to add a time out it's actually going to be a wrapper around a specific step we're going to enable so I'm gonna set this for 90 seconds remember you want to give yourself plenty of buffer so you don't get any misleading errors should the timeout come across before your execution is actually completed for that test so I will add a child step in this case I will just add integration sorry performance testing complete and now I have a fully functioning test stage as well all right so from here now I've taken the sequential test stages and I have now put them in parallel so that we can optimize our build processes so I will save this and I will save it directly to the master and I will say added parallel so you're gonna put it up wait a second you're gonna put it on on the master branch Tyner I am and then I'm going to add a a specific branch for our deployment as well so first of all we're gonna test this verse it is our example in github and then we're gonna add a conditional to the deploy stage that way we can get a comparison between the two and we can add branch conditionals at the same time so we'll run this specific example and you'll notice that now if I navigate back to get T it will update with our new Jenkins file and you will see has actually added the parallel syntax and it has multiple different test stages that are now within that specific phase of our CICE pipeline so that I seconds file is still editable by human beings if they want to bring up their in my case I love my Emacs so I could edit at other places I don't just have to edit in Jenkins nice absolutely this gives you that principle of running it as code right mark that's what you mentioned before not only can you create this simply in the UI but now we can also store our pipeline syntax its code so we get all the benefits of running it and this solution like get to your github as well so as execute I would like to navigate back to the original pipeline that we created and that was in github so you'll notice the execution time it ran for a minute and 24 seconds and this is our sequential testing with the deploy step that runs as well if you're interesting to see won't you agree mark that we can optimize this by running these specific tests in parallel well and okay I'm amazed because that let's see what the parallel results do I am curious because this is a single hardware execution unit I would be surprised if parallel benefited it very much it did it actually helped it still will affect it actually since it is still running on the same execution code it will still result in some benefits as well so you can imagine my running state steps in parallel even for bills that might run for a longer execution time let's say 30 minutes or an hour you'll see even greater results as well and once again this is the exact same Jenkins file that lives in git T and lives also in github so these are exact same replicas so that we can get an accurate comparison so as this executes s-- we'll see that has now completed I will navigate back to the window and this is run in 50 seconds so we can see that is roughly 35 seconds or so of save time by running these steps in parallel and if you were to leverage that across let's say a 30 minute build cycle or a pipeline you're now saving almost you know what is that 12 or 13 minutes of execution time so you can really optimize your CICE bills well end ok so I assume that those those nodes that are in the test that are running in parallel if we had multiple agents it could have actually run multiple agents and multiple executors whereas with sequential execution it doesn't matter we're stuck waiting for things to finish absolutely imagine if you ran this on specific build nodes and build the agents imagine how fast this would actually execute at that level as well which even greater returns when it comes to optimizing our build patterns so from here what we've done is we've added some test stages in parallel but we still have deployment stage that is running on any commit to both master and any other branch at this point and we don't have another branch and get to it at this level so what we'd like to do is add a conditional statement and you'll notice that I'm going to navigate directly into the pipeline editor there is not currently a way to add this using the GUI editor directly in blue ocean but what I can do from the deploy stage I want to define this as running only on the master branch what does this allow me to do it allows me to separate two uniquely different kinds of users that way my developer when they want to execute this pipeline they are only running builds and tests they're not having to deploy this to a staging or pre-production environment every time they want to iterate a certain change and also it also locks down my ability to not merge it to the master branch instead only folks like mark who may be my release manager my QA manager have those privileges so the developers are running and and deployments with each execution of their CI pipeline so from here I will add a when conditional so it's just a when curly bracket and then I will space over and I will add branch his master so this means that this executional statement in the stage of the poi will only run when I commit this directly to the master branch so I'll update that and I will save and this time what I'm going to do is I'm actually going to create a new branch I'm going to add annotation in here for a comment and say add conditional statement or execution on master branch deploy so now what I'm able to do is I may want to save and execute this pipeline directly from the blue ocean editor and what it will do in git T it will actually go in and create this development branch within my source code repository and you'll see that it will actually skip our deployment stage because it is not being executed directly on the master branch so you've introduced this concept that we deploy only for master so who who or what are some of the cases where you might find that can you describe those a little more absolutely let's think for an example for a lot of feature based development companies I want to deploy a new feature but I don't want to test it directly on my production and since or even wanted to pull it maybe to staging or the QA environment instead I can spin off a feature bring it off of my main master branch I can employ that new code that new feature that I want to test and I can run it as a separate Jenkins job from the same repository and I can test it as a developer before it gets promoted to I'll say a QA environment that way I have the freedom as a developer to have my own pipeline in the same repo I can follow that feature branch strategy nice so you've got I get the benefit then as a developer that I can merge from master to my feature branch I can use it and those merges from feature branch keep me current and then I can propose things back to others that merge into master to go to production I like that very nice absolutely now I can collaborate with my other developers I can spin off my own feature branch I can create these while other developers can also leverage branches that they can use to build their own features as well so now we're really developing as a team and we can just merge this back to our master or our single source of truth for our application code so as we look I just executed it on the development branch and you'll notice that Wow it didn't meet our conditional when statement the deploy will not occur but at the same time all of our tests have run so you'll notice in git T it has actually created that development branch directly for me and the Jenkins file is updated with the conditional to match only on the master statement so from here I have completed all of my work I introduced in theory a new feature or a new branch and now the project would he handed back over to someone like Marc so what they can do is now merge this into the master branch and have the executioner deploy this to whichever environment that we liked you thanks Tyler that's cool so mark you can take it away from here yeah let's see now I've got to take shearing max do you give me sharing their attention ah there we go and I click to show my screen button and now if we're lucky do you see the giddy page Tyler am i showing the correct thing I don't know that's these shoes yeah okay good so this is what Tyler just committed right here's the here's his Jenkins file that he committed all those things that he just did I can see those same things here inside the the Jenkins server it's all visible there it is happening if I look at all the pipeline's I see his blue ocean pipeline all visible to me we're working on the same Jenkins instance it's a good thing so what's the next is you may have a comp eye plane that easin is more complex than what we've described and that's perfectly fair and reasonable to say hey my pipeline is more complex than what you've shown I've got teams but for instance I have a process that may require that we release only with official permission from someone who must grant that permission the blue ocean system has assists the ability to wait for input you may say oh there are things I need to do I need to publish results of a build to an artifact repository yep post-processing steps you may even have places where you say things sometimes can be disposed of because I don't need to run them and we've got a concept of milestones let's talk about all of those so I'll show you how to define wait for input to prompt for input before executing a step this allows interactive approvals so if your organization needs official approval from a person before they're willing to deploy to production wait for input is the solution we'll show how to do that post-processing steps let us choose to do things after all the other things have occurred in a step this one you'll just need to look online to see how to use it and milestones like wait milestones let us skip steps if a later job had already reached that step so for instance imagine that you're running five or six builds in parallel and one of them completes tests for an a later version of this the repository milestones let you say earlier jobs don't need to waste the time running those tests again so milestones and let's take a look at see how that input works so I'm going to go back to pipeline demo and I think we should switch development model so I'm gonna do my work on the development branch and here I'm going to edit this pipeline and what it's going to do is bring up the editor for me as it brings up the editor chrome is not happy come on Chrome there we go as it brings up the editor in the deploy step I'm going to add or the deploy stage I'm good to add one more step and again because I'm lazy I'm going to look for the wait for interactive input step the message will be shall we deploy and on the okay button we want to say yes okay so some are happy this YesStyle this makes sense at this point but I thought DevOps is all about automating this process why would I want to have an interactive input this seems to slow down the process that that's that's a valid question and the reality at least for me was that there are plenty of organizations where they're on a journey to implement DevOps they really are they're on a continuous improvement journey but the police they're at right now is they need official approval before they release something and this lets them implement what they have right now not just the eventual destination where they will get to a point somewhere in the future where they do continuous delivery where every commit to the master branch delivers to production but for right now they may not be there and this wait for interactive input allows them to insert real steps that will help them get their work done while they're on their journey towards continuous delivery now I've got to move this one up because these steps execute in sequence I need to move it up to the top so it's going to wait for interactive input steps been added let's save that and off we go so wait for confirmation before deploy and we're gonna save that and it will get started and run so now we can go back into my repository and see that yes here is the wait for input it will appear down at the bottom this is you'll notice this is getting a little bit of loops yeah there it is right there the input message this is this is getting interesting there are a bunch of steps hiding in here that are doing build tests parallel testing in various forms conditional deployment and deploy only after someone has confirmed yes they want to deploy that input step can actually be further constrained to say there are only specific individuals who are allowed to answer the question so there are all sorts of marvelous ways that you can use input to get things done in your organization so let's look at this one so here's my deployment and it's setting here did I miss not getting my weight for confer before to pono haha I am sorry I guess what I did a marvelous thing I deployed to exactly the deploy branch which completely turned to the development branch which skipped my step I gotta merge this hang on just a minute let's do the merge max I assume we've got a few more minutes so I'm gonna go ahead and do this if you're if this is not okay you let me know and I will I will suspend and we can go to questions it's just I'd like to show this piece before we get into questions absolutely let's let's uh push this up that's because they're not gonna add this directly to the master branch anyways when they want to try some executional status so we're gonna add this to the development branch before we ever merge it to master we're not gonna make any edits ideally directly on master it's all gonna come through pull request so this really showcases best development practices any ones it does thanks very much you you make you make it sound like that was an intentional thing I like that that's that's very kind of you so giddy will let me merge this to the master branch so I'm gonna merge it and off we go okay the pour request has been merged now when I go over here to my pipeline demo you'll actually see that Jenkins was informed by giddy and can also be informed by github and by bitbucket that it should launch the build automatically so my act of committing that pull request caused the master branch to automatically build and here it is it's going to run and we'll we'll have a little bit of time for it to run as it's running we can take a look at how it looks on the master branch just to see that yes this jenkins file really now includes building testing integration and performance test a deploy and the conditional input on the master branch and now if we're lucky we should here ah yes there it is it's asking me the question do I want to deploy so I can click the yes button and it will go on and do the deploy that gives you some hints about the real capabilities that are already built in to your Jenkins server ready for you to get good things done what we've shown you are components that are available for your use no charge they I love free it's amazing what we can do with Jenkins what we can do with giddy what we can do with public accounts on github and bitbucket all sorts of things to make our jobs easier as we're developing software now we're guys a question I have a question for me I don't mind sure I noticed that you have to be on that terminal for me to approve that let's say you are the release engineer and you need to approve it are you expected to sit there and wait all day until that input step comes up for you to promote this to deployment that's it that's a good question so input steps can certainly be made to timeout so I can I can hold for that but ultimately this job will in fact sit and wait until someone approves it it will not deploy without approval unless unless I put a timeout on my input step and say I want to pass even if it timed out was that what you were asking tanner yeah I think that's a good point and also what if I want to email this to an approver is that possible within this as well uh I think it is now that's where I'd have to look at it I don't maybe you'd need to answer that one because I don't recall yeah I think you can you can say a bill that's waiting at this specific URL to a QA approver based on an email message I slack send any of your instant messaging services and that's a good point to add here that way you're not constantly waiting for this to you kind of come up and you're not monitoring the blue ocean portal this allows you to have instant messaging I'm not I'm not restricted to notification of the pending requests requests to deploy only from the web UI I can do it through slack email the other notifications Thanks good clarity excellent so any max I think we're at a point where we should entertain questions if there are any yeah we we got plenty here to go over okay don't know um you already touched on this at the end but one that came up a few times just if we could reiterate what you were showing everyone today was this only for cloudBees customers or was this open source material so everything we showed today was strictly Jenkins now we use the cloud-based Jenkins distribution because it's available to everyone for free but it is the same in Jenkins so blue ocean all of the parts and pieces we did are absolutely available anyone using Jenkins anytime likewise actually I'm kind of proud of this a giddy server that we used is also free and github and bitbucket provide free public counts of accounts as well so we truly showed every component of today's demonstration was free to github pipeline that I created with the tests and sequence that is also a public and free github repository as well nothing you saw today was a paid for solution all right ma just going through all these seeing what we can really really get you so how about this well blue ocean automatically decide the OS for you so like if they want to write the pipeline for windows batch instead of shell or they able to configure that on their own it's so it will not auto decide in that sense what you do is you open up the the jobs definition and in it instead of so if my build needed to run on Windows I could declare that I want this to be on a Windows agent and then instead of an a shell step I would add a batch step so bat I would add a Windows batch script instead of a shell script so they have to intentionally choose which one they want to do but it's easy to do I do it all the time and in fact we build we build plugins on CI Jenkins dot io using exactly that technique where we build on both windows and on Linux did that answer your question max do you think well I believe so but I should say if we get to your question you don't feel it's answered please feel free to follow up again also as we're answering these you can continue to ask what you think of we're probably not gonna have time for everything but we will do you will be able to get to people offline there's also some similar questions I'm gonna try to condense into a you know one size fits all kind of question and answer but if I don't manage like I said just react skit or let me know what what didn't come up and a few things we've been getting questions the whole time so some things have come up with a presentation but I'm gonna ask them anyway to retouch on them so let's do how about this one next what about declarative forcing the jobs to run on the master and not run on the build agents so now I've got to be sure I understand the question so I'm gonna make some guesses on the question so the steps the the groovy code which is at the back end that donut domain-specific language isn't which is at the back end in general execute on the master however the build steps themselves for instance the build on SH that I ran and the test all dot SH those run on agents so so knowing where things run the clarity of is really a good choice specifically because it is strongly biased towards putting the vast majority of the work onto agents not onto the master now there are there are steps that do have to execute on master but in general we like what declarative does and it's actually far preferred over some other pipeline solutions which tend to be heavier weight on the master than declarative is now Tyler do you think I described that okay or is there something that we need to clarify there to give a better answer on that question no absolutely and I think a good example of something like that is sometimes you might be building legacy app occasions that might not be supported by the Jenkins master itself let's say a legacy version of JDK for your running builds it's not supported it's important that you bring a build agent to the table that might have that install on it that way you can run those builds on the agent rather than the master that's just the best practice moving forward and honestly another topic in general but we want to leverage agents to its most capacity for other than running builds directly on the master itself right very good advice in fact it's it's distinctly I consider an anti-pattern we should not run builds on the master if at all possible it should be rare now many people say hey the Masters all I have reconsider consider adding agents and in particular agents can give you multi platform and faster execution by giving you the ability to parallel eyes onto multiple pieces of hardware all right thanks for that let's go on to this next one here so this is a question referring to working within the jenkins file directly is there a way to define Jenkins plugin dependencies such that my builder either fails or installs the plug-in at one time there isn't a way to do runtime plug-in installation and I think that's actually intentional because plugins are part of the system-wide configuration and so if you require a plug-in and it's if your jenkins file requires a plug-in for instance you use some some exotic thing and you're referencing it in your jenkins file that jenkins file will fail at execution runtime because it won't be able to resolve that plug-in component there isn't really a way to do cause it to install the plug-in and given that plug-in installation is a system-level permission that's a good thing we wouldn't want pipeline scripts to be able to make system level changes they should just execute builds and tests is that said I think the idea of allowing Jenkins file to declare what it needs and then making it fail in more easy to understandable way that's something we've got forever I don't know exact if everything is Eastern but I'd love to see that if that doesn't exist mmm good point so so some sort of a scan the file in advance look at it and decide oh we need these things fail early rather than waiting until execution or late stage execution to fail good insight I think that is the heart for the question if I understand correctly they are not very dense more clearly like what this thing is Rio needs or say louder all right let's let's do this one next so if I have multiple git repositories that need pipelines can I put all of my jenkins files in a separate git repository and what are the pros and cons of this approach ah good question so yeah I had a yes and the answer the simple answer is yes you can but then the pros and cons question is a very fun and interesting question because on the one hand it's very healthy for developers to be able to see and modify the way their software is built so that lobbies put the jenkins file inside the repository of the thing that's being built however if you are a Multi repository solution that may not be possible or if you haven't yet persuaded your organization that they should adopt the jenkins file approach you may need to create a separate repository and put the jenkins file in it that jenkins file can in fact then clone multiple other repositories into its single working space so it's actually one very attractive capability of pipeline is that it lets you choose how many repositories you'd like to use and how you'd like to use them now admittedly I think that's a relatively advanced use model a multi repository working environment tends to become flex and tends to cause a class of mistakes that developers sometimes dislike but it is a perfectly valid use model and it can work quite well yeah I think I'll add when Jenkins detects a pipeline itself it's just matching on a marker file for this Intex Jenkins file in that repository so we showcase today by leveraging multiple different branches within a repo allows you to have different Jenkins files still within that repository so at the Jenkins level it really thinks if there's completely different jobs even though it's in the same repo which wouldn't be possible if you're just using one branch right and I like Tyler's point multiple independent branches inside a single repo is not only accepted but it's quite comfortable and can be done very well so so one example Tyler - max - your question would be that that kind of model could be represented by independent branches which don't merge with one another but are still in a single repository or they could use separate repositories all right next one here so for we're using Jenkins should I install it locally or their cloud solutions available there are certainly cloud solutions available and which where you install it locally or in the cloud is is your choice right it's your call you've got the flexibility to do it let's see Jenkins runs either locally on your hardware Windows Linux FreeBSD Mac OS it runs on my Raspberry Pi Linna it also runs in the cloud at AWS Google Cloud as your all sorts of other cloud hosting providers so you choose it can run either any of those places all right great is it possible to modify existing jenkins file with the blue ocean editor it is you so long as you're using declarative pipeline and the blue ocean editor styler showed it to us briefly you get to it by pressing control s and there it is now I confess I'm an Emacs user I love my editor so this is not this is not a world class editor that's here but it does the job dad could you navigate to the blue ocean pipeline that we imported that's a good example that is actually a pipeline that we imported directly from my github organization and right now mark could edit that directly in the blue ocean editor yeah very good I like that let's go find blue ocean here and if I hit the edit key there now if I do ctrl s and that's all I did was ctrl s on my keyboard and here it is ready to edit hope that answered the question max I believe so uh it once again you know I'm just happy ask us if agent or let us know if it didn't excuse me so this next one here so when you have tests running in parallel will they run in the same slave or different slaves and if it's an option how do you configure it so the agent that's to be used to execute is selected by can be selected by label and Jenkins has a scheduling system that decides which agents should be used for what the simplest way to spread things around is to use agent labels to allow it to spread if you've got multiple agents it will tend to distribute the work onto those multiple agents so I'm not sure I answered that question because that one I think maybe need more sophisticated discussion than I'm I'm especially ready to give here that yes it absolutely will execute on any one of several agents that are attached it chooses the agent based on labels that I can assign in the job definition in the in the Jenkins file all right I see a perfect follow-up question to that before I get to it I just want to let everyone know we're not gonna have time to get to everything we only have about three minutes left I'm gonna put my email into the chat here so you can reach out to me directly if you have more questions I can get you in touch with who you need to be put in touch with and also we will be following up on these anyway so by all means you can send me an email or we will be contacting you through but after this next one can you give a quick overview of agents and notes sure sure so agents agents are used by Jenkins to perform its work right what they are thinking of these capabilities or executors and agents can be labeled a label assigned to an agent is usually used to express that it has a particular capability in my environment for instance I label things by operating system so some of my agents have the label debian others have the label Google to others CentOS there's windows I further in my environment label them with the version number of the operating system just in case I have some tests which need to know about the specific operating system version and there are actually plugins that will automate the management of those labels for you so the other the other use of labels for me is to express a particular capability on an agent for instance if I need maven the label maven is usually assigned to that agent so that I can just say I need to execute on an agent which has the name and let's put it here maven and this would express saying I want to do a maven based build on a Linux computer which has at least get to 20 on it did that answer the question max I believe so and will get up a whole up if it didn't oh that's so for for this next one we are currently sharing groovy files of our stages using the Jenkins shared library concept what is the best way to share pipeline code among various pipelines I like pipelines shared libraries a lot the story for me that is was most most delightful on that was I maintain the Jenkins get plugin during my nights and weekends and somebody some very smart group of people created a pipeline shared library that made it so that I call one statement build plugin and a few parameters and it builds and it builds on windows it builds on Linux it uses the current job or the current platform version and it uses newer versions of Jenkins all automatically and they knew all sorts of things to do that I didn't have to worry about I love Jenkins pipeline shared libraries absolutely they're great choice exactly and I think not everyone's as passionate about Jenkins as cake a Marc and I are so they're not always going to want to learn Jenkins syntax themselves what it allows you to do is with shared life shared libraries almost share Jenkins as a service for some of those end users they no longer have to go in and create these in all steps instead you can provide them to them out of the box Thank You max alright so we're at the top of the hour I want to just get through one more question real quick be respectful everyone's time just to reiterate one final time everyone we were recording so we'll be setting the recording out to everyone we can include the slides with that at the very end of this a very short surveys gonna pop up just asking what you thought what you like to learn more about in the future I have put my email into the chat for everyone if we did not get to your question you want to ask me directly and beyond that let's do one more question and maybe a couple of quick closing statements so which do you think is the better practice having Jenkins file within the codebase or have one Jenkins file for all the apps for me I like Jenkins file in the codebase I think developers should have as much control as possible of their own build systems however I know there are cases in there are organizations where they intentionally choose something different I prefer Jenkins file in the in the codebase because that's been my experience it just works better for me to have direct control I also understand that that's not always the right solution for every business in every way they do software delivery I'll second that as well as we move towards microservices it's really important that developers have that ability to kick off their own builds from their code bases specifically lets them iterate more quickly I think actually the answer is both we you know you you do want well in other especially in say the micro services or during any bigger Enterprise like you know you gain the benefit of scale by lots of people lots of things like a doing the same build and pipeline like unity or rather your conversation like we don't want people to be doing uniques not like if you have like hundred micro services so the shared library we have tools and like an abstraction the cater to those things actual libraries and then that way we can keep the presence of two seconds I see a CCD system in the depository minimum example I mean if we didn't talk about this table in your products I previously have this marker file customer I guess that's a feature so you can actually look for patterns presents of the files and so on in the depository to automatically apply certain pre-baked you know did I find so that to me is like a best of both worlds thanks Kay Kay yep all right we're a little bit after the hour so to be respectful everyone's time I think we're gonna sign off for today before we do mark Tyler Kay Kay do you interview any any final comments you'd like to leave us with I thanks for doing this max thanks very much for letting us present privilege yes thank you very much everyone there thank you all right thanks everyone that survey is about to pop up and we look forward to seeing you again real soon

Channel: CloudBeesTV

Views: 21,249

Keywords: Cloudbees, Webinar, Jenkins, Pipeline, Jenkins Declarative Pipelines, Continuous Integration, Continuous Delivery, CI, CD, Blue Ocean, Kohsuke Kawaguchi, Mark Waite

Id: 79HfmjeOTEI

Length: 62min 12sec (3732 seconds)

Published: Thu Nov 21 2019