How to Create a GitLab Group in Jenkins

in today's video we're going to walk through how to create a get lab group job type in jenkins [Music] are you new here if you are welcome and if you are new here you don't know who i am my name is daryn pope and i'm a developer advocate for cloudbees when you've worked with gitlab in the past with jenkins and you were creating pipeline jobs you were probably creating pipeline jobs on a branch by branch by project by project basis with a get lab group we can create one single job point it at that group and any time a new project comes online that follows the rules that we have set up all the pipeline jobs will be automatically created and managed for you this is all made possible by the get lab branch source plugin so here's what we're running today i have a jenkins controller that's running version 2.277.1 when it was created it was created using install suggested plugins once the controller came up i also installed one extra plugin and that is the get lab branch source plugin at the time of the recording the version of that plugin is 1.5.4 connected to the controller i have an agent with a label of linux and i also have a tunnel set up via ngrok so i can receive webhooks from gitlab.com if you are using a self-managed git lab the setup instructions should be the same but if you run into any problems be sure to refer to the documentation for the get lab branch source plugin down below this video here you can see that i have two projects within lab i have a micro one project and a micro ii project inside of both of these projects are just a readme file and that's all nothing else just a readme if you want to see that as well for the micro2 we can see here that micro2 is also just a readme before we can get started working inside of our controller there are a couple of things that we need to do over on the gitlab side specifically we need to create a personal access token and we need to create an ssh key so that we will be able to check out our code even though our code right now is just a readme so we'll go over to the user settings and click on edit profile and we will click on access tokens i'm going to give it a name of jenkins dash darren pope i'm also going to set it to expire and according to the get lab branch source plugin documentation the only scope that we need to add is api so we'll select api and then click on create personal access token right here at the top is the value of that personal access token so i'm copying that and putting it off to the side now what we need to do is create an ssh key so we'll come back over here and click on ssh key and what we need to do is we need to paste in the public ssh key now i have already gone ahead and created an ssh key pair and in my buffer i have my public key i'm going to set it to expire at the same time as my personal access token let's click on add key if we click on ssh keys here in the breadcrumb we can review that yep sure enough it was created just now never been used and expires in a couple of days okay so that's all that we need to do on the gitlab side for right now we needed to create a personal access token and we needed to create an ssh key let's go over to our controller and create two credentials that map back to both of these credentials that we just created here so if we go over to our controller what we'll do is we'll click on manage jenkins manage credentials global and then add credentials the first credential we're going to create is the git lab personal access token so i'm going to go and grab the token value and i'm going to give it an id of jenkins-dara pope and the same for the description click ok that's the first credential the second one that we need to create is our ssh username with private key so in this case i'm going to say the id is darren pope dash ssh-key the description will be the same and the username will be darren pope that is the username associated with my gitlab.com account now what i'm going to do is enter in the private key so we'll click on the radio button we'll click add and then i'm going to paste in my private key i have no passphrase associated with this private key so i'm just going to click on ok so now we have our personal access token and we also have our ssh username with private key before we create the job we need to do one more thing we want to allow our controller to manage web hooks for us within gitlab so we're going to go to manage jenkins configure system and we're going to scroll down to the get lab section and we're going to configure this default gitlab server we'll leave the name of default the server url is correct now if you're running a self-hosted git lab your server url will be different again follow the instructions in case you have any questions the credentials are going to be the git lab personal access token that we created i'm going to select manage webhooks i'm not going to check manage system hooks i'm not concerned with that today all i care about are web hooks today we're not going to do anything with a secret token and the root url is going to be our ngrok url so i'm going to paste that down here and just take off configure i'm just restating it for clarity so i know that any time the hook is coming in it's going to go to that endpoint so now we've created our credentials we've configured our gitlab setup now we are ready to create our job so let's click on new item for the item name i'm going to name it the same name as my account which is darren pope we're going to scroll down and we're going to select the get lab group job type now in my case darin pope is actually a gitlab user but it would work with group as well so if we click on ok we're going to scroll down and look at get lab group so let's look at what we have here we have servers which is the one that we just configured where we turned on manage webhooks our checkout credentials we're going to use darren pope ssh key and for the owner i'm going to type in my owner id which is darren pope after i tabbed out of that field you may have noticed that darren pope is a valid user so what happened i typed darren pope press tab and it went off and authenticated using the personal access token we can see here that we have discover branches and discover merge requests but remember at this point we still have not added a jenkins file to either micro 1 or micro 2. so let's go ahead and click on save oh wait before we do that before we click on save let's go back over to micro1 and micro2 real quick this will help clarify a few things probably in a few moments micro1 settings web hooks right now there are no web hooks on micro one nor if i get there fast enough are there any web hooks on micro 2 so no web hooks exist at the moment on either micro 1 or micro 2. well now let's click on save and see what happens so when we click on save a scan of the get lab group or in this case the get lab user is done it found micro2 and this message is a little bit misleading when it says jenkins file found but it's a directory and not a file what that really means is jenkinsfile not found they found micro one but again same thing jenkins file was not found so let's take a look if i click on darren pope now you're going to see that i have darren pope some sort of icon here and then the folder is empty well before we move on let's go ahead and take a look at our web hooks so if we look at the web hooks for micro 2 we can see now that we have a web hook coming from b95 which is the address for our controller b95 if we take a look over at micro 1 and i'm just going to type it in we can see here that we have the b95 as well so when we created this job even though we did not have a jenkins file in either micro 1 or micro 2 the web hook was still created for us in both of those repositories so once we actually create a jenkins file in either those repositories they should show up here but why is it a jenkins file let's go back and look at the configuration of our job if we scroll down to this section called project recognizers we can see here that there is an item that says pipeline jenkins file the script path is jenkinsfile so what this means is if i find a file named jenkinsfile in the root of the gitlab project then that project is eligible to be part of the gitlab group so let's go ahead and create a jenkins file in micro1 to see how this works so let's open up a shell and let's create the jenkins file and what you can see from here this is a very simple jenkins file all it's going to do is echo out hello from jenkinsfile and it's going to run on the agent that has a label of linux so let's go ahead and save that we'll add it and let's go ahead and push it up now let's go back over to our job and see what's going on so if i click on darren pope up in the breadcrumb we can now see that the project micro1 was found so if i click on that and then click on main we can see that build number one for main is currently running based on the branch index that occurred right here and we can also see that we obtained a jenkins file from and this is the commit hash and if we scroll down what we'll see is echo hello from jenkinsfile so the micro1 project was picked up by this gitlab group due to the web hook coming in it created the micro one project which is actually a multi-branch job and it created the main branch within that multi-branch job a way to think about it is a git lab group is a folder of multi-branch pipelines and a multi-branch pipeline is a folder of pipeline jobs now let's go ahead and create a new branch off of main and we're going to use that as the basis for creating a merge request later on so let's go back over to our shell what we're going to say is get checkout dash b fix dash 123 and what we're going to do is we're going to push that branch up no changes to the project now if we take a look at micro one we can see now that fix 123 has been found if we take a look at the output of fix 123 we can see that it's just echo hello from jenkinsfile because we didn't make any changes it's just the same jenkins file that was on main is now inside of fix 123. now that we have our fixed-123 branch let's make a change to our jenkins file that's on that branch so let's do a vi jenkins file and what we're going to do is we're going to add two stages to this jenkins file so we still have our echo hello that's still the same we added in a stage that will only run when the branch starts with fixed dash so we would expect this stage to only run on our fixed-123 branch we also have another stage that has a conditional when the branch is mr dash so this will only apply to merge requests so now let's go ahead and save this and we'll add it and commit it whoops let's keep that right and let's push it and let's watch what happens on fix 123. so once we get our handshake back now let's click on fix 123 in the breadcrumb and we can see build number two is pending right now and now it's running we see our echo from jenkinsfile that's good we see the cat of the readme which is what the fix dash stage does for us and then we can see the for the mr stage was skipped due to the win conditional because fixed dash does not equal mr dash so right now we have two branches main and fix dash 123. now what we want to do is we want to create a merge request from fix 123 into main so let's go back over to our project here we're going to click on create merge request we can see here it's going to go from fix 123 into main i also want to make sure that we have checked delete source branch when merge request is accepted so let's go ahead and click on submit merge request when we take a look at this what we're going to see is a merge button with a delete branch sort or delete source branch let's go back over to our job for micro 1 and see what we have now we still have main we still have fixed 123 but we also have now a merge request 6. so if we come back over into gitlab what we're going to see is now we have a merge request 6. so that's the mapping there if we take a look at the output for mr6 we're going to see that we have also outputted hello we're skipping the fixed branch because mr dash does not equal fixed dash and then we also see the output for our mr stage so this jenkins file that we made the change on in fix 123 it did one thing differently on fix 123 and it did something different on mr dash in this case mr6 and we still have the hello working just like normal so at this point what we want to do go back over here what we want to do is we want to go ahead and click on merge now before we click on it i haven't clicked on it yet let's double check what we have here on micro one we have main fix 123 and mr6 let's click on merge once the merge completes we can see the changes were merged into main let's go back over to our controller and we can see here that fix 123 is now obsolete as indicated by the strikethrough same thing for the merge request because it's been merged mr6 is now obsolete if we want to clean up the obsoletes we can do that just by clicking on scan gitlab project now that will automatically clean up after a certain amount of time based on how it's configured but if you want to force it that's a way you can force it let's look back at micro one again so the only branches that are left are main merge request there are no merge requests around so if we click on main and take a look at the output of build 2 then what you'll see is we have our echo hello and we see the skips for both the fixed branch and the mr branch up to this point we spent all of our time in micro 1. let's go ahead and start doing some work inside of micro 2. what we're going to be doing inside of micro 2 is something a little bit different than what we did inside of micro 1 just to make you think a little bit let's say that we want to create a jenkins file inside of micro2 that part we know so that we have to do but let's not name it jenkinsfile let's name it something different what we're going to do is we're going to create a jenkins file dash m2 so let's open up our shell and create jenkins file dash m2 and this will look very similar to what we did over in micro one the only major difference is that we have instead of hello from jenkin's file is chicken's file dash m2 everything else is the same so let's go ahead and save this we are on micro2 let's go ahead and add this commit it and let's push it now what do you think is going to happen if you go back to the project recognizers that we saw earlier the only project recognizer that we have is jenkinsfile although the contents of this jenkinsfile m2 is a valid jenkins file the name of the file isn't jenkins file it's something else in this case it's chicken's file dash m2 so what we're going to see when we come back over to our job is that micro2 was not found so in order for this job to work we need to reconfigure let's go down to project recognizers we're going to add pipeline jenkins file and we're going to say jenkinsfile dash m2 and let's click on save now when we click on save it's going to do a scan of the group or in my case the user looking for both jenkins file and jenkins file m2 so we can see here that jenkinsfilem2 was found so when we come back over here micro2 was found on the main branch if we take a look at the output we can see that the hello is hello from jenkinsfile m2 now let's do something a little bit tricky right now when we take a look at the configuration of this job we can see that our project recognizers are both are in order jenkinsfile and jenkinsfile m2 at this point micro1 looking at the log micro one is using jenkin's file that's this section right here so all of this is micro one so jenkins file found and then here for micro 2 which is longer it didn't find jenkins file but it found chicken's file m2 so if we were to add a jenkins file to our micro2 repository what's going to happen is the jenkins file going to run when we commit that jenkins file or is the jenkinsfile m2 going to run as it continues on let's see what happens so we're not we haven't made any changes yet let's go over to micro2 and let's make a copy of our jenkins file m2 and we're going to copy it to jenkinsfile and let's edit jenkinsfile and let's get rid of the m2 so now our jenkinsfile and jksfile m2 are effectively the same file except the echo statement is a little different let's go ahead and add jenkins file whoops i knew something looked weird there m and then get push okay so now if we look at the build that just started for main we can see it's pending we can see here that it is still using m2 now that doesn't make a lot of sense because i have a jenkins file and a jeans file m2 in micro2 when the project was last scanned the value that was scanned and written was jenkinsfile dash m2 although we added a jenkins file since the gitlab group has not been rescanned yet the micro2 multi-branch job that's within the get lab group is unaware of the existence of the jenkins file so in order for micro2 to pick up and use the jenkins file what we're going to need to do is we are going to need to force a scan of the gitlab group now so we'll do that and then let's take a look at the log output we're scanning micro2 and now we can see that the jenkins file is found as the project recognizers are processed they are processed in order so if you think back to how we have it configured it's jenkins file first and then jenkinsfile m2 second so once we find a match in our project recognizers we're done you'll notice here that micro2 is really short this time whereas before with m2 when we checked for m2 it first checked for jenkins file it didn't find anything and then it went ahead and said okay is there a jenkins file m2 yes there was then it matched out at that point now since there's a jenkins file inside of our project it was able to match on jenkins file first and now it's going to completely ignore the existence of the jenkinsfile m2 nothing has changed inside of micro1 now if we go back over to micro2 the job did not automatically run that time so let's go ahead and click on build now if we take a look at the output of three what you're going to see here is it was obtained from jenkins file and not from jenkinsfile m2 and we can verify that from the output of our job now let's say we wanted to reorder and have m2 be first in our project recognizers let's see what that would look like so we'll go back to configure we'll go down to our project recognizers and i'm going to grab jenkinsfile and put it below m2 so now it's chicken's file 2 and then jenkins file let's click save now this time when i click save it's going to scan the repositories because that's what happens when you make a change so we can see here that micro2 was found and it found m2 first so right now this is all of micro2 if we take a look at micro 1 now micro 1 is the one that is longer it first checks for jenkins file m2 it didn't find it but then it found jenkins file so if we go back over now again to micro 2 what we're going to see is once we do a build now we're going to see that build number 4 is now using jenkinsfile dash m2 we were able to create a single job pointed at a get lab group in our case a gitlab user it was able to bring in all of the projects that we wanted to build because we put in a jenkins file or jenkins file m2 in our micro2 case and as we added the new branches and got the jenkins files in the right place the branches automatically showed up within the job as branches were obsoleted deleted they disappeared from within the job so the get lab group job type that is offered through the get lab branch source plugin is a time saver it gives you the ability as either a jenkins administrator or as somebody that's just managing jobs it frees up your time that you're able to focus on other valuable things within your organization if you have any questions or comments you can reach out to us on twitter at cloudbeesdevs if this video was helpful to you please consider leaving us a thumbs up and if you haven't subscribed to klabby's tv yet why not take a moment click on that subscribe button and then ring that bell and you'll be notified automatically anytime there is new content available on cloudbees tv thanks for watching and we will see you in the next video

Channel: CloudBeesTV

Views: 594

Keywords: darin pope, jenkins, jenkins webhooks, gitlab webhooks, jenkins gitlab group tutorial, jenkins gitlab merge requests, gitlab branch source plugin jenkins, jenkins tutorial, continuous integration

Id: it6TOeQ6EHg

Length: 30min 22sec (1822 seconds)

Published: Thu Apr 01 2021