Where Does Jenkins Store Archived Artifacts?

in this video we're going to talk about where jenkins stores archived artifacts [Music] you may have wondered where jenkins stores the artifacts when you use the archive the artifacts post build action in a freestyle job or you use the archive artifacts step in a pipeline job well that's the great thing about jenkins everything is file based it's just a matter of knowing where those files live in this video we're going to take a look under the hood to see where the artifacts live and we'll also take a look at an option for storing artifacts somewhere else other than on your jeekins controller here's where we're starting today i have a jenkins lts controller version 2.303.1 and i also have an agent attached to this controller down in the description of this video there is a link to a gist that will have links to all of the documentation and any sample code that we use so let's go ahead and get started i have an example pipeline and it's on the gist and that gist is right here i'm going to make a copy of this pipeline and before we go and create the job what i want you to see is it's just pipeline agent any so we'll be using that agent that's attached to the controller i've set a handful of options you can choose to do that or not do that and it really boils down to just two stages i have a stage to where i am creating a text file and then i am archiving that file just by passing in an ant base pattern match for txt so let's go create our pipeline job and see what happens so we're going to call this archive whoops archive artifacts pipeline click ok and we're just going to paste it in for this let's go ahead and click on save and then click on build now what we'll see here it will run pretty quick we go in we create our hello world and then we archive our artifacts pretty simple right we created a file and we archived it let's click on the number one in the breadcrumb and we can see here that build artifacts are listed here and if we click on hello text we can see here that it's hello world that's what we put into that hello text file take a look at the url we can see here that it's referencing job archive artifacts that's the name of the job that we created there's one that's our build number artifact and then hello.txt so that's where the file actually lives we'll take a closer look at that in just a moment if we go back up to the top of our job so we're out of the build number we can also see here that the last successful artifacts are listed here at the top and then again it's hello text and if we click on hello text we can see here that it says last successful build and what's happening is it's just pointing down to the last build number which in our case was build one now just to prove this out let's go ahead and flip over to our shell and i'm inside of my controller and if i take a look at where i'm at i'm inside of my jenkins home jobs directory so if i type tree which i've installed on this machine then what i can see here is i have my archive artifacts job builds here's number one there's an archive directory and then here is our hello.txt file and if i was to cat out archive artifacts builds 1 and archive and then hello text we're going to see that it's hello world so this is where the archived artifact lives it lives under the build number that it was created inside of now the file i just created is only 12 bytes so i'm not really concerned about the storage of 12 bytes extra on my jeans controller but what happens when more and more jobs start archiving their artifacts that means your jenkins controller is potentially being turned into a binary repository if your jenkins users are using archive artifacts on purpose in their jobs in order to store the files on the jenkins controller to act as a binary repository they really instead should be using a real binary repository such as jfrog's artifactory or sonatype's nexus repository but what if you don't have a binary repository available to you right now but you do have access to either aws s3 or an s3 compatible storage provider there is a plug-in named artifact manager for s3 that allows you to store your artifacts in an s3 bucket the great news about this it's completely transparent to both your jenkins controller and to your jobs so let's go ahead and go back over to our controller and install that plugin so let's go back to manage deacons back here to the top manage jenkins and then manage plugins available i'm going to look for artifact manager on s3 so i'm going to select that click on download now and install after restart and now that we've logged back in let's go ahead and verify that everything installed fine which it appears to have done so manage plugins installed artifact manager on s3 yep all there ready to go okay so let's go ahead and go back to our job actually before we go to the job we have to do some configurations for that plugin let's take a quick look at the documentation for the plugin and it can be found on plugins.jenkins.io and the prerequisites that you're going to need are you're going to need an s3 bucket and you're going to need an account that has access to that s3 bucket and it sort of makes sense right if we're going to be storing files inside of s3 well we need an s3 bucket we need credentials to be able to connect to that s3 bucket before i started recording i set up a user inside of aws and then i applied a policy to that user i also have an s3 bucket set up let's take a look at the policy because usually that's the biggest confusion point and this is also in the gist so what i have is a very basic policy my name of the bucket is deepo jenkins-archive artifacts by the time you're seeing this that bucket's already been deleted so nothing to see there that bucket is completely private though it is not a public bucket i have three different sections here i have one so i can actually find the bucket that's that part and then once i get to the bucket then i have a list just on the bucket level and then once i'm actually inside the bucket the slash star then i can put get and delete so again i've already set up a user inside of aws i have credentials from that user a key and a secret i've applied this policy to that user and then i have the bucket already existing because i'm referencing the bucket already here in the policy and before we go ahead and configure the plugin let's take a quick look at the bucket just to prove that there is nothing in the bucket as i'm recording this so i will close this up to make it a little bit bigger no objects inside of the depop jenkins archive artifacts bucket okay so now let's go ahead and get that plug-in configured so all the magic can happen let's go to our controller so let's go to manage jenkins configure system let's go down to artifact management for builds click on add select cloud artifact storage you're going to see here the only option we have is amazon s3 so either you're using actually amazon s3 or you're using an s3 compatible provider but first we have to set up our settings so let's go ahead and click on this link that will open up a new tab our bucket name is going to be whatever my bucket is let me go grab this whoops wrong tab we also have this option for base prefix by default it comes in blank and artifacts will be stored in the root folder of the s3 bucket if it's blank if i was to put in a new base name let's say top folder slash then everything that goes into the bucket would go under that top folder slash directory if you will now i'm going to leave this blank so that way everything goes into root these next two options are deleting artifacts and deleting stashes and it doesn't really make sense until you take a look at the help these work similarly that it requires a dash d parameter for when the controller starts up so i can't just check the box and then click save i actually have to set this during in my case the startup of my jeekins controller i prefer to have these checked because i want my artifacts deleted from s3 when the build is deleted from my controller that's my preference same thing with stashes but i don't have that currently set up on my controller so we're just going to ignore these for now close that one and close that one we don't need to set a custom endpoint or custom signing region these are if you're using a compatible s3 provider so again since we're using aws s3 for real we don't need to set those and we also do not need to set these other settings here now i do need to set credentials so i'm going to go ahead and set my credentials to us east you could leave it auto but i tend to set it to where i know i'm going and i need to add a credential so i'm going to click on add i'm going to change my kind of credential to aws credentials i'm going to set this to archive artifacts just for the id it's sort of weird but hey that's okay my access key is over here and my secret is here and i'm going to go ahead and click on oh before i click on it we can see here the credentials are valid but do not have access to the amazon ec2 service when i set up my policy if you think back to the policy it had nothing to do with ec2 only s3 so i know the credentials are valid but it can't get to ec2 i'm okay with that because i know i didn't set it up let's click on add so i know my credential is good but before i leave this i need to select that credential now you'll notice there's a validate s3 bucket configuration here before we click on that i'm going to go ahead and save this and then the way you get back to that aws configuration is right here it'll take you back to this panel i'm going to go down to my validate s3 configuration and if it's successful then it says success if it is not successful at this point your policy may be incorrect your credentials may be incorrect or your bucket may be incorrect those are the three things you would need to look for so you either created an incorrect credential maybe you put a space on the front or back or something else so you have a bad credential you may have accidentally typed an incorrect bucket name that's a possibility or the policy that you created which is right here in my case was not correct either so those would be the three places you would go and look until you get to the point to where you have success do not move on from this point okay so now that you're at success let's go ahead and go back over to our job and you know what i'm gonna stay on the page for a second i'm just gonna click save good enough let's go back to our job archive artifacts now i have not made any changes to this job just to prove it let's take a look at the configuration i just still have my sh and i have my archive artifacts that's it nothing more nothing less exact same job that we ran in the first run so i'm gonna go here but i forgot one thing let's go back and do one more change go back to dashboard manage jenkins configure system so we had everything configured for aws but what i need to do now is go back here to the artifacts section and actually add it because i could not add it until it was configured again amazon s3 that's the only option and now we can click on save so at this point we can go ahead and go back over to our job and click on build now let's watch what happens here build two again echo hello world look what happened here when we archived the artifact it said uploaded one artifact to and it gives me an s3 url depope jenkins archive artifacts right that's my bucket name and then i see archive artifacts and then two and then artifacts well this was build number two so if i click on build two and we can see here that our hello text is being hosted at s3 which is what we wanted so now let's go back here let's go back to the very top again last successful artifact if i click on hello text we're going to see that it goes ahead and redirects back over to s3 let's go back over to our shell on the controller and we are in our jobs directory let's just type tree again let's see what happens this time in build 1 we had an archive directory hello text but with build 2 there is no archive directory so now let's go back over to s3 and see what happens so we can see deep jenkins archive artifacts we have archive artifacts that's our job name we see two that's the build number we have artifacts and then inside of that we have hello.text so should you really use archive artifacts the short answer is no as long as you have access to a binary repository set a different way your jenkins controller should never be used as a binary repository why is that as a jenkins administrator the last thing you want to do is bog down your jenkins controller with excessive network traffic with people pulling files from your jenkins controller especially if those files are very large files as a jenkins user you want to be able to provide metadata around your artifacts and binary repositories such as artifactory and nexus repository are purpose-built to provide that to you however if you don't have a binary repository available to you right now and you need to use archive artifacts then go ahead and get your artifacts moving to s3 or an s3 compatible storage provider if you have any questions or comments you can reach out to us on twitter at cloudbeesdevs if this video was helpful to you give us a thumbs up and if you haven't subscribed to cloudbees tv yet why not take a moment click on that subscribe button and then ring that bell and you'll be notified anytime there's new content available from cloudbees tv thanks for watching and we will see you in the next video

Channel: CloudBeesTV

Views: 714

Keywords: darin pope, jenkins, jenkins tutorial, archiveartifacts, jenkins artifacts, jenkins pipeline, continuous integration, jenkins archiveartifacts, archiveartifacts artifacts jenkins, jenkins pipeline archiveartifacts example, jenkins pipeline archiveartifacts

Id: u6LF-T-daS4

Length: 17min 27sec (1047 seconds)

Published: Thu Oct 07 2021