HashiCorp Live Codes Vault & CircleCI, Part 1: Injecting Static Secrets into a Pipeline

hey we should be live can everybody hear us chat twice if you can hear us I don't know if you can I don't know I don't have switch open yeah we'll see I can't tell if this is because the new mod view doesn't show anything or all right oh there we go hi okay are we is sound good test we didn't do a sound check I can hear you yay thank you cool thank you yes awesome okay all right well welcome to hostile Corp live we're back again once more live coding this time we've got angel from circle CI who's going to help us configure vault and a circle CI pipeline which is maybe pretty fun yeah I think you're gonna help configure well then I'm gonna do the lines yeah would you care to introduce yourself yeah so I am angel Rivera I am currently a developer advocate at circle CI and I've been in the ite slash technology industry game for quite some time so probably a couple you probably know me which is great and today I was actually honored that the rosemary pinged me to see if we could work together in this stream and I do a lot of public speaking at conferences and today I wanted to play around with vault and trying to secure pipelines so that's I guess what we're gonna be doing today right yeah this is gonna be exciting let the joke the last time we did this was that we were pushing on a Friday and I guess we're gonna do this again after yesterday's shenanigans we may or may not be trying our luck here but like I said if github goes down and we can't play with circle then we'll just play with vault right yeah unless our well instance goes down in which case I don't know what to do anymore let's call it a day all right so let's get started and what's the premise of what we're working on right now yes so the premise is basically so with CID pipelines at circle CI a-- for those you don't know where a CI CD as a service company so what we do is we enable you to build test your build test deploy your code using automation and what I what I've interested in and and I've always had a passion for security alongside coding and what I wanted to do was try to - we use this time to try to use learn a little bit more about volt the last time I used it was using like the key value stores not really using it wavelets tended and you've added a ton of features to it so it's been a long time since I had hands-on so I'm interested in basically you know being able to secure the secrets that we use so meaning a P I tokens username passwords things like that that you know our pipelines need to actually be productive for us and I like that hash Ecore crashy core vault gives us that capability so what I was hoping we could do is take a project that I already have on github and we can start like you know locking down the docker hub credentials because when the pipeline you know we'll test the code then hopefully we can push that out to to docker hub and but we can leverage the credentials being brokered by by hacci core vault if that makes sense and then maybe we can build out the pipeline so kind of show people how to build out pipeline and create docker container the docker image inside of the pipeline and then use may be terraformed to push out a deployment to an existing kubernetes cluster so fancy yes well then well let's start right like pushing creds from from or pushing the image to docker hub and we like slowly built because it isn't an aggressive kind of a task that I'm undertaking but at the end of the day I'd like to see how we can you know just build on that build a pipeline together and then hopefully the hard parts I think for me is going to be how do we broker the credentials from GCP over to a pipeline that that's the one but but we'll walk through it to show people you know because there's gonna be situations where you're gonna have tokens or you're gonna have like we were talking about the app roll rate situation or you're gonna have situations where just the services just use username and password cuz that's a reality right especially and I can tell you especially in enterprise situations the username and password it's like all over the place so this will give you a little nice variety right and I'm curious how I am making some anti patterns in my yeah sure yeah if anybody on the chat had you know if you're Pro if we're doing something and you're like that doesn't seem right just drop us a line and feel free to interact and let us know and we can chat about what's an anti-pattern what's a better pattern so we'll keep it we'll keep it simple all right yeah so I have we have a repo and we've we've pre-staged a couple of things like this kubernetes folder which rosemary has populated with some cool scripts that we'll probably dig into a little bit and then we have this vault folder which I'm really interested in learning what's going on in there so what I created what I created was a new docker file so this project I used to basically so the application I start with the application application just a very simple nodejs like it's a static page right and it has an image and a message here and I do have some tests right I use this so I do a lot of speaking like I said and I do a lot of 101 type work where you know people who are new to see ICD and I've tried to use like like a multi micro-services type of you know example the problem is a lot of folks on that developers especially are not used to see ICD so it the the architecture of a micro-service convolute things when they're first starting out so I realized real quickly that it let's start simple because the application isn't really the focal point we don't when you're demonstrating sed you're trying to teach folks how to use the platform so what we're doing is going to use a very simple application nothing nothing fancy here and maybe in the future we can build on this and build out some micro services or whatever but let's just start very simple I do have some tests using the the mocha testing framework for nodejs and this two simple tests right make sure there's a 200 happening in the application there are ok 200 responding when you run the server and then there's a a static check of some text that's in the application so it's very simple testing to show off you know kind of you're supposed to be testing your app so you spill some tests and then quick thing can you magnify the code on the screen of it oh yeah sure I guess yeah let me do it plus is that better is that good how's everybody thinking do you want one more yeah anyway if you can't see it just let us know and we'll increase it some more yeah okay and I could do that I mean oh wow okay all right awesome thank you yay okay and then I'm gonna also make this non trailer and take the transparency away I think it so go solid black there you go and then I'll plus that up that good yes oh and there was a funny question that rolled in not terribly off-topic but they were curious as to your terminal if your github page was the background to your terminal or was your terminal slightly transparent it was transparent for sure that's what I just did see so now and you can do that by just editing sessions and then the my default is is this and then I see but I'll change it so it doesn't yeah yeah but yeah you can do all kinds of cool stuff I just keep it transparent just sometimes I need to read things behind the scenes so yeah I like that okay cool so let me close these because we don't really need these this is important so we'll leave that open wow this is really weird to work with a big load up screen but that's all good all right so I think we're we're we should start with looking at the circle CI configuration so if you're not using circle CI let me just close all this stuff up here this eliminate the distractions so okay so I have a simple app and this folder right here dot circle CI is where you would basically place all of your circle CI assets so the way circle CI works is if you wire it up in you know to your back-end to github so it's a service that basically web hooks or OAuth into a github or whatever a version control system you're using right now in the cloud we support github and bitbucket but there are other version control systems on the way hopefully before the end of the year but at the end of the day this is the folder where you need to store all your circle CI assets and then from here you need to put your directives or your circle CI actual the syntax right it's a yamo file and it's you just have to call it a config dot yml and once you have that file listed in there you can start doing the cool things I like building out your pipeline so let me close this up a little bit oh wow you really ripped up that I know they took out a lot ah to do some fill in the blanks I was like oh cool maybe we start with like the simplest vault configuration it's not the most secure but it would be the most simplistic right okay cool cool so do you do you want to yeah do you want to start should I add I should probably add the other the old like the testing bits or no do you want to just start working on this first yeah let's add the testing best cuz let's look let's look for a complete pipeline that can actually push them right okay no problem I have that let's just jump into another branch on this thing and then grab it well let me do this then if I jump back to don't you go back to my fault alright no I'm sorry let's go to master and here you got see so I had some some of this stuff here let me grab that and jump back over to this one anymore scroll down no no it's different so what I'm gonna do is put jobs and I'll just grab actually just get rid of this make yeah and then we'll yes I think you're all aligns because it's yellow you have to have and then I have your jobs in there run tests these are ah you may have added yeah okay so oh my bad I didn't see this this is why yeah okay so I see we okay yeah no no worries no worries this is this is my bits okay cool cool cool cool oh so you already had him all right I just thought they were missing but we have everything we need awesome this is great so alright so real quick before we get started I'm just gonna do this because for for my own sanity I'm just gonna put these yeah because there yeah there you go Circle CI has this concept of jobs so they're effectively one exactly yeah basically jobs are a list of commands right are a list of things to do so let's start real quickly jobs is is a list so if you look if you're programmer and you're thinking about an array or a yeah or a list that's what that's what this in the Amal that's what this represents a jobs there are jobs to key and then within that you would say you would start labeling and categorizing the different things you want so if you want like what I'm saying here is I want to run tests so you would create a label for a job right and that's where this is a designator here so if you were to close that up you would see you have a queue config job and a run test job right so we just have two jobs right now what so these are the two jobs right so if I expand this then within my run test job I'm defining something here then we need to run our code in some runtime so what we call a run time in CI CD world or a circle CIS world is we call it an executor which is execute code the run time now we have four different flavors the default is a docker container so your codes gonna be run inside of a docter container and in this case right we have an image key so you have to provide it with a valid image this is an image that circle CI produces nightly automated Li and we're gonna use a no 10 image for this once you have your executor defined then you just define the steps so these are this is another list but this time the list is of commands that you're gonna run in your job right so for run test these are gonna be the steps are gonna be all of the things that you need to run all the commands you need to run to basically run your tests and you know take care of any other actions you need to perform within that flow so what the first thing you need to do we used to get the code right so this check out is it basically an alias for get the gate clone so what we do is give you the backend system knows all right this is a checkout and since it's running on the commits that you know you pushed up it already knows that it's gonna grab the latest does that make sense right so you don't really need it anything else once you have that then you start defining run commands that's what this key is here and then when you have that you just give it a name this right here I'll show you in a minute at what what this corresponds to but this is another label when you go into the circle CI dashboard you know once you if you have many jobs and many steps in your job things can kind of get convoluted if you don't really label it properly so that's where this name comes in you can kind of say hey this is the install and PM dependency step right so if you have commands here right and yeah mo if you want to do multi-line you have to add that pipe and what I'm just doing right here is basically right installing the dependencies for my packet for my program through NPM and then once that's installed I can do another run command and you got to think of these run commands as a new shell opening right so if you were to open a new terminal every time that's what this Run key would represent but once you do that again I'm gonna run my unit test and this is where all the fun happens right so as you can see here I'm actually calling out for the Madoka bin binary for the testing or at least feeding it the the folder where where it lives and then I'm passing I'm using this this tool called omocha and and I'm also using mocha awesome so mocha has this concept of reporters so you can actually you know create testing kind of reporting systems based on whatever it is that you know you how you want to represent your tests I'm just using the default one called mocha awesome which is a different framework but mocha awesome we'll then you know take my test that I run and basically give them to me create human readable and we'll run rule run a quick one local quick tests the show-off would or look at an older one but basically this helps create those test artifacts right so if you run your test locally you usually spit out an HTML file or some sort of you know test artifact and that's all we're doing in this step here and once we have that circle CI these are these are Native keys to the platform basically it will store your test results and and all you have to do is feed it a path so the test results folder I believe is what we're uploading right and then that'll that'll basically tag it or pin it to your to your bills so for every build you have that's you know basically the files the reported files reporting files which show all of the all of the test results basically and once you have that you want to also store it as it as an artifact these two are a little bit different so the test results if you're storing it as they if you're using like an XML type you know with it's a Java kind of format then you know if your if your test results are in XML you can use this test results and it'll show display in a different tab in circle CI but generally people most people use the store artifacts and by the way you can also store any kind of file with this right so if you have maybe some releases right so you built a binary for release you could also store it as an artifact well now out of curiosity if I wanted to let's say store my token files right because vault has tokens to access vault right so would store artifacts be the appropriate place to put that token no the way I would do that and I mean let me just show you real quick let me bring up I'll bring up the dashboard let's take a look at that real quick so now I know that you know we the the stage that we the job that we went through is unit testing so it doesn't really need secrets it shouldn't really depend on any secrets it's not generating anything sensitive in theory because it's unit test but if I were to want to store it then maybe I you know I would be blindly using store artifacts for example to store a token yeah you could but the problem with that is then anybody who has access to the the dashboard like I'm showing you here so like let's take a look at this is the prot that same project and let's take a look at so I should only have because I have not uploaded so yeah it's successfully ran when I just a few moments ago or whatever time I ran this 20 was 49 minutes ago I ran this so basically what's happening here this is what it looks like right so remember when I said install dependencies that label which is right here yep right that's what this little drop-down is here right and it shows you what happened then I ran my unit test which is this one right so in this step if you can see here right it passed it ran my tests everything passed and then my report right which I use JSON right was was saved because I have a step that well let's see uploading yeah so this is it uploading no tests found because it's not it didn't do that I didn't do it next ml so it didn't upload it but this one took all of that right that whole folder and uploaded it so the reason why I say you shouldn't be loading that into artifacts because someone can click on here and get your token okay so note to self don't use artifacts right okay the the way you should do that is let's go to well you can go to the the project settings right over here and this is basically the project and what I would recommend you do is stick it in here okay environment variables and the reason I say that is you can say like I don't know let's say I don't know we'll call it the vault token sorry and that'll be an environment variable that when you create that it'll be listed inside of your your runtime container if that makes sense yeah all right so what I'm going to do here and the way you should not store like let's say it could be a file like GCP credits or credentials what I would do is if I were if I were the person or this is what I would recommend base64 is not a so let's say our token is I don't know want one two three four five six seven eight right let's say that yeah and then you want to do something like pipe and then base64 and then I do another pipe I don't know if this is X clip on Mac I'm a linux guy for this you're not privy to what happened the Mac of the dreaded Mac update happens for both of us I think it crashed my computer like my CPU spiked to like 75 what happened I don't know if this will work but basically what I'm trying to do here is well you know what let's just do this this is even easier so now I have a base64 I don't know why I was trying to go all pan let me so I got this this value right and then I pop it in there and boom hopefully boom is thinking yeah wow this is not good anyway boom it'll be there I mean it could just be thinking or it doesn't like a vault - token or oh no oh wait this is a real vault token right you would do exactly so fault underscore cap token right we got you I got you so let's change that maybe well let's stop that let's do it let's try it again I'm not sure what's going on here but I want to refresh something is not right okay let's try this again what was it vault all caps all right yep okay and then underscore token all caps all right yeah yep okay and then yeah so boom so we got right so now and right we show a little bit so that you understand which token you're using but this is base 64 encoded right so there's layers you can do and then in your pipeline you would do a let's say you would do something like this to to get it out right so let's say you know well yeah I won't work here but if I had a vault thing you would say you know echo vaults whatever in this case we're just gonna say that and then we'll pipe it out to base64 and I think I'm sure the Mac one has a decode function oh yeah there you go so so you would run this line this this would be your environment variable right mm-hmm and then you would just base64 encode it and pump it into another maybe environment you want to use it if you needed to you know access it in any way but to be fair this is the this is the way I would recommend you store anything of value and but and and also in environment variables when you're looking at things like let's look at the dashboard real quick let's get out of this when you're looking at environment variables in in let's see when you're looking at like you know log outputs standard out if you had any kind of variables that were listed in there they would be kind of a masked so they wouldn't be exposed to anyone who you know who's looking at this thing the idea here is yeah don't save any sensitive data to artifacts or tests or you know any kind of up looting yeah or persist them in your workspaces either oh cool yeah all right so with that premise we actually have a vault instance up and running I'm not going to go into how we configured said vault instance but we also have some vault stuff pre-populated one of which is this docker hub a docker hub username and password correct yeah I can well I haven't done it yet but we can do or live I guess yeah why don't we do it live alright well first I'm gonna take you can't see that one time I just moved right know that I want to expose anything so what is that so what is happening is that he's logging interval I'm logging in through the UI of alt and yeah we could do this we could do this with CLI as well but we're just gonna show you kind of yeah easier way to consume this I'm not a big gooey person either but this is more fun yeah all right so so i i've basically gone to this thing and then yeah okay cool so yeah you've already got some some awesome paths for me so what we're gonna do is create one here alright and then so you see this is more of a key-value model right yes it is a key value yeah so the cool thing with with vault is which I'm also learning by the way everyone so like I used it in production but I only use this and this there's like yeah all these awesome things I want to just like dig into but I don't have the time but we're gonna we're gonna create a key value secret which like again rosemary already has a namespace for a setup so I'm gonna dig in and it's our pipeline and so I'm going to create a secret in this here which is called would call a docker hub should I use capital or yeah lowercase whatever I just remember the path right so long you know what let me go lower keys that yeah cuz I'll just forget and then I'll be like what's wrong buh-buh-buh all right so docker hub and then the secret is gonna be I'm just gonna say us are as the username and make sure that is it you could just type and see okay well all right tell you what cheering pause sharing for a second you can't see it right yeah and so a general explanation for those who are not aware you don't get by default this Circle C I know Jay s application path right you have to turn it on so we the the reason why and you know I don't know there's a brief gloss over that we talked about previously which was there's this vault terraform folder and if you're someone who's in operations some of these things maybe you pre-staged so for example i ran a pipeline to create the kv the key value of secrets engine which is why it's here and pre-populate some other static secrets right there's also GCP secrets engine that's also been turned on and configure it in here and and we'll go through that later it's a little bit more complicated but let's just go through the static secret example yeah so basically I created the USR which is user the value and then the password for my docker hub so now that's living in the the Hashi core vault right so now anytime I need I would like to call this I don't I don't need to you know provide my credentials normally the way I would do it is like I showed you earlier right so I would do it through the the environment variable system which I showed you right I would create a new variable and I would call it oh I probably I usually use this more in more descriptive all right and then put the value here and then save it and then you would do one for the password but since we're using vault to store all of this this is really cool because now I just you know make a call from my pipeline to vault and say hey I need those creds yeah and it can be set with policies which rosemary probably yeah so actually let's let's take you through like why would you do this involve versus other you know you would put an R yes that's right and part of this is that you can control who has access to it and you also control whether or not they're revoked a time to live right so there's no secrets engine for docker hub on fortunately but there aren't other secrets engines that what happens is like it will actually revoke and generate secrets on demand so the lifetime of the secret is very much controlled through vault so that's what kind of makes it really neat dynamic secrets are really useful especially in pipelines right because you may or may not want people having all access to pushing to production all the time the lifetime of a pipeline is like order of minutes maybe yeah generally yes but there are organizations that like do some heavy compute right like especially some of our MLR machine learning AI type projects they can take you know quite some time out some of them you know 45 minutes but at the other day you can still set that if you know what your average pipeline is right time you can set one that yeah that can you know cover that the idea is that you know you don't want to have credentials laying around longer than you need them to right exactly so if you click on policies actually so right now we actually are using a route token to login big no no there are other ways and we never do that yeah we're just using this for the sake of example right now but what we actually have pre-configured is as you can tell in a CL policy so if you click on that a CL policies for nodejs - circle CI oh this guy all right that has information to control who gets access to what so anybody who has this policy applied to their identity has access to read anything under the GCP service account key that's generated as per the GCP secrets engine but they also only have access to anything under nodejs Circle CI slash pipeline so there are two paths that they're allowed to have and they're allowed to read from those if you have like application secrets you would want to keep them separately you wouldn't want to keep them under a pipe so if if I were to edit this I just edit it here right and then nice you can edit it there and the way if I actually done it if you go back to we've talked about it a lot so - let's show it and if you go back to the code and oh yeah okay yeah sure - go back to the code we'll hop around a little bit but if you go to vault that folder and then you go to terraform I told you she had cool stuff in here alright so this is using the terraform provider it actually uses tier it uses terraform cloud to do remote execution and basically configure vault for me because I got really tired of configuring it multiple times so you know why not but if you click on kV this is the configuration for the key value store that we identified before so some of these things are static secrets you notice them in the UI they're kubernetes they're terraform cloud for example we could add docker hub to this but then there's so this is all declaring this KB secret store but if you go back to the a Perl file okay sorry go back after all here you go yeah and if you look at that resource policy on line 5 to about 17 that's actually the policy that we saw in the UI is just declared by terraform so there are a couple ways you could do it you could through do it through CLI you could do their API I just do it there terraform it's a little bit easier right well that's cool because you're part your app your products are all kind of integrated anyway right so that's that's good that you used like the same HCl instead of Y Amal like I've seen companies do that or project through that way you have three different DSL or data structures I know I know well and part of this is like you know it's it's maybe familiar if you're someone who's in operations this is something that you can say like okay if someone wants a new policy and it's a new pipeline for a new product or something you want to make sure you're applying the correct ACLs and stuff so it's kind of an easy way to do it that's good and then oh wow so you have some properties here that you can adjust so you know we showed that we were using the route token to login to the UI and that's a bit of a no-no I would say yeah yeah and I don't think it would be advisable for us to use that token as part of the pipeline I think it's fair to say that we don't just want to use a route token okay yeah yeah so vault has this concept called an a Perl and and what an apt role will do is basically help you assign a role identity to something whether it be an instance whether it be you know some arbitrary hash of some kind that will allow you to basically say this is the role and identity associated with it here is a secret that they would use to effectively quote unquote log in and authenticate to vault and then securely introduce the vault token into the pipeline right so the role in itself handles the revocation and the lifetime of the vault token rather than you going in creating the vault token for someone for a specific policy so basically this actually will allow you to handle an issue the vault token without ever actually having to go back to vault and asking for token yourself so what we're going to do is actually configure the app role for the pipeline because the a Perl is attached to the correct policy and will handle the attachment of the vault token for us so makes it a little bit easier I'm gonna pause because I know that was a lot to consume ya know I just opened up the twitch thing I wanted to see anything nobody thing too too crazy but I'm gonna attach the vault a Perl documentation it's actually pretty cool I'm really enthusiastic about hey do you want me to set up this collaboration with you and then you can just do it yeah just do it yeah okay I don't want to do all the work here plus you're teaching me so you know this isn't for me it's it's really stuff yeah I mean like I said we could we could pretty much just get to get the docker hub credentials right into the pipeline by using our existing route token but that just doesn't seem very secure it would be kind of nice to make sure that you know you're kind of using the correct you're using the constructs to allow sort of an ephemeral sorry everybody there's a bunch of keys that you're gonna see there I'm actually running to screen shares that's why it's confusing okay yes I also need more screens this isn't a conclusion that I'm coming to okay all right so how do I I need to add you right some time copy go do the live share button at the three dot on the side oh okay here all right that's not working okay here we go live share link yeah you can actually there should be a link somewhere that generated recent contacts do I invite by email or now oh oh I see I got you yeah there should be a you're endearing all right hold on all right everyone I'm gonna take this to another level here and zoom out for a minute yeah we okay I started alive shares it on this side maybe I've never used this before you know I'm gonna share so you're saying live share so this thing down here somewhere you are live sharing we just need no we just need the look the little bell icon on the right hand side right this one no I think wait hold on let's see share I'm gonna link there you go all right all right I'll send this to you in slack now jump back to code and then I'll fluff it up a little bit for everyone okay you should see me frolicking around now in the light yeah go for it it's all you hold on is signing in no worries how's everybody's Friday going Friday brain last week I had a couple weeks ago I was like what is going on okay anyway besides that and if you are in your terminal why don't we actually retrieve show kind of show what this app will stuff looks like so you notice in this code right you have lines 19 to 30 3 so it issues a role ID and it issues a secret and the secret in itself is unlimited you can actually set the number of uses in the case of a pipeline you might want to use this terraform code for example to be able to facilitate the passing or the construction or the sort of the the orchestration of issuance of role and secret ID and pushing it to circle CI so that could be like you write it to the circle CI environment variables you know because the secret my change after 10 uses or something but in this case we're just gonna leave it you know not updated but anytime that it's not used right then it will be revoked and then you have to reissue it again so that's the idea behind the secret yeah there was a quick real quick we were discussing this the other day that rosemary nice so one of the cool things I think you could do is at least with circle you can do API calls it's back to circle CI and get like historical build reference so you could actually like you know get how long a specific job took or or a piece of a workflow to so if you have you know need to issue something you could call back with an API get the value and then probably send that back as a hey give me a token of this long right so you could do some really cool things to lock it down yet like I said the idea is to shorten the life of that credential to the point where it's you know you absolutely need it and then when when you don't need it anymore destroy it all right let's get rid of it so you know you're trying to shorten the the time to live because you don't want an bad actor to get access yeah exactly and I think that's that's the key part of this right you just want to make sure you're handling the revocation and everything in one place if you're not doing it in one place it becomes really hard to go through and unravel where these things are so those na will take you a whole day to basically sit here and figure out where all of the secrets were in place and what they weren't in where they went and that's happened before I actually did print out keys in a circle CI pipeline once and I took like a whole day to go back and change the keys fun fact yeah that was not fun that's good I know well not that was not good that was not good so basically this terraform outputs a roll ID and the secret ID just for the sake of not comp not compromising them because i would not want to do that what I'm going to do is output them and put them in circle CIS UI because I don't really want to be showing everybody the keys actually no I can revoke them afterwards so why not alright your call okay and so what actually let me see if I could get back to the live share look did it connect me no it's still connecting it's still connecting no okay apparently well let me connect so nothing is working today all right well we have something yeah is it is it just the link or should we try together let me just put them into environment variables and then we'll grab okay but basically what I'm gonna do is I'm actually going to put them into the project variables the specifically the am I allowed to do that actually or do you have to do that that's what I'm wondering know for for circle yeah you your contributor you should be able to so let me send you the link to the project and then you can yeah you can do this so I can update it myself so what I've done is I've just done a terraform output and it I grab the role ID and the secret ID involved what you would do CLI wise is you would do a vault secret vaults get oh sorry you've all read and you would get the secret right from the a Perl so you would do this off a Perl thing a little thing and it retrieves the secret for you sorry it would be a right vault right and then it would be - eff get the a Perl but I think it's just a lot easier to use the terraform to output that I'm going to send you the link to that - where you should be able to that thank you bring that up this is how it Howard we're lucky we have this like tools when I was coming up it was just literally a master sergeant standing behind me smell in my head calling me stupid all right so if you could go into the dashboard refresh sorry refresh this page the project settings oh oh sorry yeah let's refresh that okay all right so now we no longer have a vault token we just have to vault role ID and a vault secret ID um you need both you can configure an app role without a secret although it kind of just you know then it loses the whole point of having the role in the first place the other thing that we need in here is the vault address because well circle CI needs to know where to go to get to vault right yep yeah okay so why don't we add the vault address so it's vault underscore address all caps and then you can put in the vault oh okay do you want me to do we're pairing yeah yeah no worries I can't get BS code work I mean you know let's see so it's the same address right that we were yeah same address so again I might have missed it I was messing around with logging into to some other thing you created these role IDs and secrets in vault right so where did I see that like like if I want to change and click access access now that's our identity management of course so click app role yep yeah so or it may not let you do that because seek review configuration yeah so all of that has all that information in there where it says like yes authentication for after all right has been enabled it does not tell you exactly which out role it is in this case I actually have a specific app role made for this pipeline yes case ok ojs circles so if we go back one yep all right and you have so you always say like enable a new method and then a parole and go through the whole shenanigans there right okay but I don't believe it watch you can click entities and see if maybe it shows up under entity no I don't think so if you click around I don't believe that any of it it shows that yeah it looks like members know Lisa's right now tools maybe it's under oh yeah so unfortunately click on secrets it might be under secrets if anybody is familiar with the vault uija knows where to find it no all right so the acryl basically is configured it's it's configured it's just not in this UI you can I think you can also click the top right with the little drop down thing ya know next to it right next to it click that that I think you can actually do a CLI browse oh yeah what's the command is it the vault yeah so it'd be vault read and then off / April yeah April / rule / sorry no js' - circle see I see - circles yes the name of the app now yes sorry that's my app to me and then circle CI / / oh yeah yep / and then roll - ID let's see if this works yay so there you go that's the same role that we were actually talking about before I'm not gonna reissue the secret but uhm because then it prints it out and then we just show the secret so we're not going to do that but yeah that's the little ID there so oh I see so basically I just all right so I'm making the connection now so right you just have the different I guess categories right so we're authenticating so this is so I guess the way that you would translate this is this is an identity kind of browse right so for those of you that are not familiar yeah so there's secrets which are basically credentials the things you're protecting and you're storing in in Hoshi Corp vault and then there's access which is basically an identity so you can consider this like a username and password for the most part right or our API token right that the system manages itself how do I get rid of that is that right is that my mic and then you set a policy to kind of control all of it right okay nice it's a nice it's basically based on the three right secrets access and policies that's right okay right and I was gonna so it's gonna actually say that it's right so these are access controls right these are kind of the rules nice nice but that makes sense because this is really complex by the way so all of you who are listening Security's not easy at all which you know rosemary I believe you generally work on terraform the fact that you understand this and make all those connections it's pretty impressive because even like you know if you're not doing this day and day out this is pretty bad I mean it's hard to do exactly you know even sitting here trying to explain the association of identity raper pipeline I mean it's a non-trivial per pipeline per executor for whatever you want to talk about right the idea is like identity is complicated and identity can be many in many places so yeah yeah but just to recap these seek the secret again is just the things you your your sensitive data alright that that patchy corpus protect or the vault is protecting and then obviously right again access is authentication so that you created this role which is and you can create a new one if you like right here so you look username and password so you're giving someone a username and password authentication right which is not ideal great and then of course policies you want to make sure that you set right the the it's almost like readwrite commands right so you that nice ok cool cool minimize the blast radius right so exactly a pipeline doesn't need access to everything right what it doesn't need access to the cubbyhole it doesn't need access to GCP and it may not even need all access to no js' circle CI it just needs access to the static pipeline secrets right so that's why it's a neat way to just associate all of these pieces together and even even if you do it with the vault CLI you could do it with the vault CLI I just prefer to do it through terraform because I can create it at will and then remove it when I don't want it anymore yeah absolutely I mean this like I said this is pretty pretty pretty complex things I worked in the federal government for many years and I spent a lot of oh I guess the brain power you know and then spinning wheels around security because of the different levels and layers that you know the data had to be secured and we didn't have tools like this this was like straight up occasionally you know you had a bash script that would just go and run through all the wrong permissions and access controls and then you know compare that with some other database that we had from previous run it was a mess eventually right tooling came out but this is pretty this is very cool and something that I definitely want to wire up so for my pipelines okay so let's so we've got the roll ID we've got the secret ID for that pipeline we've got vault address so we've told circle where to go where we need it right but the one thing we didn't really do was so we've kind of told it here is where to go but what we didn't do is say here's how you authenticate to vault so let's go back to the pipeline and actually talk about how we authenticate developed so I'll just can I close this April April awesome ok so now we're back in the pipeline yes which is you know so far we've just got the unit tests so let's actually talk about how we can get secrets into ya pipeline that's perfect yeah so I'll just I'll just get rid of that one for a sec all right let's talk about it okay so I guess if we kind of preface this right we have the information to authenticate to vault we have a role we have a secret and we have the endpoint of vault but what we need to do is get a token in return that vault token will allow us for a temporary period of time to access the secrets under nodejs circle CI slash pipeline so just because we have a role ID and just because we have a secret doesn't mean that we can just get to vault or get these secrets out of all we still need a vault token so let's create a job is that good and then we're gonna do a docker yeah all right sorry got a didn't image because we got to define well sorry I need to define the executor properly so that's a talker and then we're gonna mm-hm then we're gonna do image yep and then we're gonna what do you have an image that I should use let's just use a Hoshi Corp slash this one well wait no that's mine that's a specific one okay yeah we could do vault and then : 1.3 dot 3 1 . 3.3 all right that like that yeah yep awesome and then from there I guess we're gonna do is steps so with this do you start doing our steps and then from there we do we do we need the code though I don't think so right we do because there's actually a trick to this that makes it really easy okay so so what what we would do right if you were to use the role and the secret is that you would have to tell you would have to initiate this like vault command that would authenticate to the roll and out then using that secret right so just as you would say like I didn't let me pull up the exact command but you would basically have to use that secret and you'd use that roll ID by API to get that secret back but there's actually yeah which is kind of do we really want to have more of that workflow encapsulated in here well the answer is probably not to a certain degree it's actually handled by a neat sub command called vault agent okay yeah so what vault agent will do is it'll actually retrieve the role ID retrieve the secret authenticate to vault for you grab that token and you can choose to store the token locally somewhere and on top of that you can also use vault agent all in all in one go to just use that vault token and retrieve your secrets and populate them into the files of your choosing oh cool yeah so it kind of does all those steps for you so you don't have to do it yourself because previously what you would have to do is you'd have to take all this information authenticate of all right a workflow to retrieve the tokens store the token and issue a second query to vault to retrieve those secrets right yeah so so let me ask you a question then so if like I wanted to continue using this I could install the agent right yep and then because we we defined the environment variables on the project they will be available to that agent and you don't even have to really configure anything as far as write it just pull read those ok so that makes sense to me so I can I can get the Hashi Corp fault or I can just install the agent in here so I would have a command and be like install age and they've do a curl request or W get to wherever pull it in okay cool cool it's all bundled with vault so as long as you install vault you'll get this oh just that binary nice that's good just do it W get on vault that's actually like custom my custom image does that it builds terraform involved altogether and it just moves the binary that image though so I don't so we've got so the way that this is configured and this is all pre staged because this is a little complicated at times to figure out so there's a lovely example if you go to vault slash agent HCl in your file directory oh yeah okay so me agent okay so basically the ball agent needs config it needs to know what to do right so what this is basically going to do is authenticate to well for you and get the token so as you can see there's an auto off method so it automatically does it for you you just specify where are the role ideas and where the secret ideas so remember we configured role ID and secret ideas environment variables it only accepts it as file paths it will not retrieve it from the environment variable right so we have to echo out or approval put it yeah put it somewhere and it's only I guess this is in some ways it should be okay because in theory this will only last the lifetime of the executor correct yeah the container once it's trashed yeah once it's done it's done yeah but but you know you can also write steps in it in here to where like a step could be write it out to file do something then crush those write those files that's what's really cool about ball agent is that it will delete the secret ID file after it after it reads from it oh nice yeah which is really cool it doesn't delete the role ID but it does delete the secret ID no well yeah kind of makes sense so I guess can we maybe write those steps first I guess we should sure yeah yeah so we're checking out the code once we have that then we can do a run command and then from here we can do a sorry and then we can do a command and then I think this is gonna be multi lines so we're gonna pipe that and then we start here so what would be I guess it would be an echo yeah to echo and then it would be a dollar what is it vault roll ID all underscore delimited Oh the well your we're grabbing these right yep I just need rope ID and secret ID we don't I'm sorry you said underscore now under alright no no no I took it as like a lower key forget it that's my Friday brain okay yeah you're right my Friday brain I don't know everybody know you say you said it right I interpreted it as lower case for some reason why am i typing I can do this Friday rain someone is like they have usually they say Friday usually bring pizza and I'm like oh now I want pizza this is Friday rain Friday ringing is thinking pizza so you have let me check what you have the file pass these guys here yeah roll ID and security so we could change those but you know now we'll keep it that's all good and then I'll do it again yeah if I spell correctly of course and then dollar that this bill grabs secret we need is what that's the most important piece yeah and then as it evolves a secret yeah secret ID - so then we have that oh wait but will it will there be a there is a vault folder in this image there's a well there's no vault folder in this image but you did run a check out because we did need the agent HCl so okay oh of course that's a good point so the question is if let's say I did run this command and the director he didn't exist it would airdrop yeah this director she made it in my in the repo see that's a good lesson in CI CD right so I started with the very blank repo and rosemary added to it which is fine but I didn't really recognize or at least review what was James so I didn't know that this was here or I didn't remember I knew it was there but I didn't remember so yeah that was why like this is a great lesson yay and watch it air out cuz you know when we pushed on the Friday is clear no no that's gonna work we're good okay so then the viniq command that's you can issue is basically now you can run vault agent and vault agent will populate all these things and do all of the things for you that you wanted to do so the commander's vault space agent headspace - config okay space yeah okay so it's a parameter R equals yeah and then slash agent HDL so basically you want to read in the agent HCl okay oh it's not HDL yeah gotcha gotcha you can do I think you can write the config and other and other file formats I think JSON as well but yeah yeah I was gonna ask you about that so could you to terraform or no yeah so HCl terraform is based on HCl so you can't technically I mean you can't really use terraform because terraform has like the resources etc but the config file in itself the HCl you can render using terraform local file or whatever you like you could you could excuse the extension tia as does the extension doesn't matter it's just yeah as long as it's the correct HCl formulation then yeah okay but I guess that this is this makes sense to me because if you do TF then you're thinking it's part of your terraform builds which it isn't exactly it's not yeah okay I think it won't work with spaces between the config equals I think it's got to close up yeah no worse yeah no no it's a CLI no worse yeah so cool so the fun fact about this though is that if you run this command like that um it actually opens a watch on values so if you just run it like this without setting a certain config it will actually just sort of watch the values and wait for them to change and it will actually dynamically populate the thing you know all of the values right so in the case of a pipeline and if you go back to the agent HCl yep I just I disabled the the watcher by doing the online to exit after off which is true ah yeah oh very important exactly so after it authenticates then it exits otherwise it will just sit there and keep rendering out to a template which is the template that we have there now one question I noticed you have a big file so is that mandatory as well or um it's more important it's not it's not mandatory but it is more important if you have that watch or open for example yeah I don't know if this is possible in circle CIN that's a good question that I want to ask you is it possible to have like a container that continuously runs as part of a workflow or it's like a sidecar or no well not good question I don't think no you know right so I don't think it would like stay alive the whole time like currently I'm not sure if we're even that may be on a roadmap I don't I don't want to say that because I don't know but I believe we are working on some capabilities that would fit that right so yeah with currently I would say no the way that the watcher is that good for debugging right for the most part well actually it's how kubernetes works the kubernetes vault injector so for those who are more familiar with the kubernetes side we're not going to cover like running application secrets and kubernetes and injecting them in this in this one but basically it will run you use vault agent and you as a sidecar to continuously populate you know updates to keys and and different database credentials and stuff so that's why this watcher is available it's basically allowing you to run it as a side car so you can continuously be polling for updates to secrets and stuff okay yeah cool it could be a good use case though if I could just like run this and then just this updates the secrets but in this case it's just gonna run as part of the stage and then exit so okay yeah oh sorry yes so great yeah that's that's awesome yeah all right so this this makes sense to me because right you're creating all right no noise I just was I saw only process ya know it makes a lot of sense yeah and the other important piece of this is the sync there so as part of auto off on line 12 there's usually there's a sink that's defined so what this is is the location of where you're putting the vault token so what happens is that the agent will go authenticate for you to vault and retrieve a token back and that token has to go somewhere we're also going to use that token for docker hub or we can also use that token to render a template out which as you can see on line 19 that's kind of what it's doing it borrows that template to render out the like a GC P key file for example okay yeah so should we mount I guess we don't need to change that or we could and enter new and for the docker hub one right why don't we I mean it will write it out to a file so you get to choose which file you want to write it - yeah why not and then okay so I'll just cut and paste as we cause developers do and then we change the name right so oh wait do we need to follow that or does it matter or we can it doesn't matter right it does matter the path right so right now it's pointing to GC p key no js' circle CI and I think we put it under no js' circle CIA pipeline something right yeah let's find out from the actual source right soon and yeah here we go so it's basically well okay yeah then I just so then it would be to go back all the way to the front and be secrets okay got you so I just start from here I would start with no Jess circles Yeah right yep and then from there I figured I copied the rest of it I think well we'll give it a shot does that look right to specify the data so here you've got password and user so yeah yeah you wouldn't put it as a path that's where the data dot data dot private key comes in place oh okay so you with that two of those so you could you it kind of depends right so it's a template it's kind of like go template so you write it out to whatever our template you want and how you consume it so you know I do secret profile which can get a little hefty but some you know if you want to maybe do like a source for the environment file for example you could say like export something like export docker use a docker underscore Hollanders or password equals something I don't know yeah yeah okay so like here I would rip replace this private key yeah with a key and then call it okay let's say we want to pump it out to an environment variable may be so completely yeah so destination so the first thing is that we need to tell it where so I guess we would put user US are right because that was how the key wasn't going yeah you want to do like that right right so that just pumps out the secret just the secret into the temp whatever file not sure that we can actually set a destination as an environment variable oh no that's fine it's but but what we would do is say so the data is the actual the key that you're looking for right and and what I so it's that what I have here would that just get me the user value yep that's correct and then the other one would be I would do another one for the password right yes that would be the second line got you and then that data that PWD alright no commas or anything right it's it's a file so if you if we want to read first line second line or like let's say you wanted to source from this file so maybe on line 26 you call this slash temp slash Dockery and V so let's say you maybe want to source from that I don't know and then you would write like then I'll kind of explain well just call it that yeah it'll just say docker just put all together and then perfect all right so if do we want to source this as environment variables or do you think that we could write some steps to just grab the like parse of the information out of the file I think the environment barriers would be better what there's a couple reasons well let me ask you this so can I use the agent to access this file or no so you don't actually after the agent has done all this you don't need the agent afterward basically we just need a way to get these files to other executors oh I see I see so then you would have to put it in like a workspace or something like that yeah exactly I see yeah otherwise you'd have to use the token you could use the token for example which is something you'd still have to persist across the spaces so you could use the vault token and reissue every time and often three and authenticate and grab that and set it as dynamically this is just one way that you maybe grab the files specifically for it and then you just get the credentials yeah there's um the way we that I normally do that is so because every let's go back to the config because every run commands is a new shell so let's think about that so one way we can we can do it is no no because we're going in and out of containers yeah we're gonna have to store these in workspace that's right you're right yep okay cool so let's go back to your thoughts like would I be able to my Google I don't understand vault agent alright and we just turned it okay alright so yeah alright so the question was I'm sorry yeah so my question is if I store these secrets and files right and I persist these files across workspaces is that a problem is that secure is that the same as storing artifacts so no youyou yeah it is it is because a workspace could be retrieved you know anytime you're using like cache or anything like that it's definitely something but but if we're that's why I like the best way to do this would be to install the vault agent and then you know run these files from the container so this right here just to get all that is probably not necessary if you just if it's just because you have that vault agent I would install it into this one and then you don't have that problem you know any of me yeah so is it fair for me to say like let me just get the tote let me just pass the token as a file great yeah - so the token as a file it's you know is I guess persisted across workspaces but none of the other secrets are so basically I would just use every stage the token to basically retrieve the correct secret yeah you you you could actually right you can actually create this as a separate job right and then because when the way you orchestrate those are you through here remember so like if I were to create a job and save altough and I let's just comment this out for now hmm right then I can actually call this in a a commands up here you know so we could even had a we can get rid of this then you see I'm saying so you can just create these commands that really don't they'll run on the executor that you want them so like you would create this off vault off job then it's reusable throughout right and then up here we could write a step that says let's give this a shot and see if it works I mean yeah I think maybe the simple thing to do rather than compile a new a new image might just be we we just write the token out to a file and persist the token file across the workspaces and then this that way if you want to use vault agent to template it out you can look at this example here and get a better sense of it okay but for now we'll just have the token and then we'll just allow other jobs so you can still see the workflow right or we can just run ball agent every time oh no yes so I see what you're saying right so it would be the same I got you but if we if we mmm good good point well alright well we'll go with the way you you're processing it and then maybe have another session or do it more fun yeah where we lock it down even further I got ya I'm just kidding all yeah excited in no I'm just curious to know this is like if I say like persists to work space that it is insecure right I mean maybe the lifetime of these like I could revoke it that it would no longer have access but yeah that well you said they after their run the secrets gone right so yeah you can configure it such that after it's run this after it's used the secret is no longer valid yeah and yeah yeah I got you yeah all right but for now let's just persist it and then set a short time to live right that's the beauty of it you can you can do whatever you need to do to fit your use case right I mean security it's not a one size fits all that's one thing I've learned you know about my career there are you know even like I've worked in the federal government you know they they would come down with these like mad crazy like restrictions on what to do and they would try to apply that federally like across all DoD NASA everything and the problem with it is like NASA they have still systems running from like the 60s right and they want to like these things down and they don't have these security mechanisms to - you know enable that so literally they had to make a choice all right and this was during the like the shuttle program right so do we lock everything down and and just discard all the software that it's critical to launching rockets or or do you get a waiver to relax right so my point is you know not all security scenarios fit every organization in every use case so there are times where security teams are going to do an assessment and say look there's a certain amount of risk that we're willing to take and they will waive yeah so let's go ahead and do the workspace deal let's say we have a waiver to say that we've sufficiently configured a time to live you just gave me the waiver so we're gonna do it all right so I guess here so now that we have wait the files will be written yeah and then ooh you know I haven't used the workspace in a while so dude let me see if I can find some workspace documentation yeah yeah so anyway this is how I would normally like go do a thing let's see how good our Docs are yeah pretty good I got to say the documents our Docs are hard for any any team in the organization so but let's see they have a search queue cool so we really want to use workspaces well I just completely did something wrong there all right let's see all right yeah there's so much let's see concepts workspace is here we go work load work there you go all right so here we go yeah so this is basically right where you you could persist things and then call them back in so the first thing you want to do is attach sounds like right first oh yeah you're right I'm sorry there you go it's been a minute since I use this so let me just grab a lot of things it may or may not be something we should use what's that no the persistent workspace no it's definitely but this also adds time if you have a lot of things right so one of the things you got to keep in mind is I mean this for this is it's it's definitely like you know no problem but I've had folks that try to save like 10 gigs and then they are like wait my pipelines taking forever and it's like well yeah you're pulling 10 gigs across the wire every time you you know implement this so anyway let's persist that's a lot yeah well that's the trial and error pieces right to this so uh I'm not good so let's bring that back over no it means yeah back to another one I think uh wait I'm sorry you're right I'm gonna do this like the right way hold on so I'm gonna grab everything and then yeah do a there we go that should work let's get rid of this stuff and then and then so yeah so we we're gonna call this but let's call it halt all right yeah must be relative path it is well know this would be pat pad this would be like where is it checking out to it would be like no js' Circle CI or something no no it would be a project wait hold on check out would be project yeah no Jess right so okay so I need to do its project and then the name no I think it's just project to be fair yeah it doesn't really give the name by default because I didn't program it to so well let's leave that for now interesting yeah and then well we could do just the token and then authenticate to vault using the token to retrieve it or we can just we don't even need the token anywhere we can just persist the credentials so out with HCL file right yes so we can just do these see right no we don't even need those too because after that's done it doesn't even need it so if you go to agent HCl okay yep we just need to propagate the template file with the docker hub credentials right yeah oh I see I see so just grab that TM yeah we just need temp docker hub which that's my guess we should just write it locally cuz we just met project bolt yeah so we could just do this boom is that allowed like can I just say like temp docker hub but then it persists the root of project vault is this right look better I think that's right let me let me yeah okay so I'm not sure um so we so we have project vault oh I see and then I got you and then me projects right it to vault docker hub instead of temp docker hub probably better is that path created though I can't remember well just delete if you go back okay if you delete um a vault and the two slashes yeah where's on line 35 yeah so just delete the path a vault is and just leave docker hub yeah I guess so then we just do that and if you go back to the agent HCl and then we can just change the temp docker hub to not backslash temp but instead just vault slash docker hub yeah yeah and then it's not an absolute path so we can just put yeah oh you're just saying like yeah yeah just make it easier for ourselves yeah yeah for sure okay so we're grabbing the secret we're gonna grab the two values and then this will pump it out I got you so this is your the contents and destination alright so what do you do with this variable well actually all it does is just write it to a file so that file could be like you know how like your bash RC has like export this path we would write a template that just says like export equal export docker table user equals and then we just source from vault docker hub Zoe this is kind of like the the cat command right yeah gotcha alright before eating files alright cool so we're rating this alright it's all save that and then go back to the config and then yeah okay doesn't smell right to me no I'm sorry something's wrong here yeah so no you're right you're right it's under steps okay I just wanted to be 27 and 27 to 31 though it needs to be bumped to spaces 27 it said this right yeah to 30 30 to 30 and then yeah you're right you're right and then these know I think 31 I I messed it up 31 is bumped back one space okay yeah alright but these are the levy right here I use everybody listening on this live stream you can tell how much I love you you've got a lot of kubernetes work doesn't I know now this is all good every new you know it is what it is it's a data structure yeah all right so let's save and then yeah so I guess we could add it to the workflow and see if it mm-hmm if it works yeah so should we should we create the no blue then yeah let's let's do that right let's do that alright so let me go normally I'll do this in the command line but I'll just do it in code listens we're all really doing things so I'm gonna just put this and then we'll just say I don't know added docker hub I give it a hard time alright well just let's just put a simple thing and then commit it and I'll push it and then we'll check on circle CI to see what happened where's my I had it somewhere there you go let's go back projects and we should see some builds action happening no wait I didn't add it to the damn ah well I didn't even kick off the words you saw they pass because my test pants okay cool so let me go back in and add this job and we're gonna say should we run it in parallel now I will flavor will say required so that way you're you're running it sequentially okay I mean I could run it in parallel because both of these are independent of each other tests the unit tests don't need secrets yes let's do that let's just run it in parallel that way it'll be faster too okay cool yeah so I'll just do that save and then alright it's very upset about something yeah we have a weird relationship alright so what am i did saving alright cool and then I'm gonna put added vault was it edit vol job whatever doesn't matter wait oops let me cancel that because I did not know I can do that you could do it by command-line I'm pretty sure people will be fine it's probably the best to be fair all right come on show me something here yeah this new UI it's there goes so we get to see the tests are running and then we'll this is this which is pulling down your image it's talking dirty to server look at that not even picking up the server yeah they come in very few yes wait a second I did I specify the wrong environment variable though is it fault a TDR or fall address I think it's address I think it's a DDR wait let's go back to this so basically this will keep running so you can cancel this okay because it's gonna keep trying to retry its involved so we do have a timeout parameter to set ah I don't believe so but I can't go to the dogs for it I should have a timeout so we should check the environment variable right variable let's see we can look here and see what so when we spin up let's see if your parent here we go yeah we did it wrong it's vault underscore addr okay yeah well that make that makes sense see we're debugging all right so let's do some project settings it was like I'm reading from a local endpoint so I have it's the address I had right yeah nothing has changed all right so let me just grab it today yeah let me see if I can okay I know all right I thought I could just be slick and get it from your book obviously a TBR yeah okay and then I need to grab the you know care of your server gets shown right that's fine I think we've been pretty good about no tokens plus the server is being destroyed after this live stream oh yeah I'm just kidding him all right let me do this then yeah let me just grab it off the screen up with it discuss you don't have to destroy your work no the whole point is I want to destroy my work okay so that the next time I need to do it I can immutably like I'm not immune just apply it by terraform you know just recreate all of the rules again you know we can just reissue a secret that's the beauty of this entire approach right I mean you're running delete are you are you running this on kubernetes er yes so this is a vault master this is a vault cluster on gke it uses the auto on sale mechanisms for GCP KMS and it also is stored in GCS so it has the H a configuration and the helm charts and stuff so okay curious I like sheep for the for those in the stream I'll just paste the vault on G Katie should we let me just kick off that build again and see if it'll it'll do it so since we changed the environment variable we rerun that build that I cancelled we should be okay so let's rerun let's go back to bolts well we can rerun this one I just run the whole thing right now so there's a new one and bad secret Oh a timed out a secret nice all right well our secret timed out which is good let me reissue it and how long do you have it to live well this is good this is see this is what happens when your secret is expired and removed now is like if you have a pipeline you know and you're pushing continuously maybe you just really want the token constantly but I feel like you know yeah well one thing about this though it concerns me is are our users use well the way circle CI you know people pay for it is by usage so maybe that's something that you know we should look at like a timeout to say like hey or or exit it you know like that's it that's just tweaking of course you know how how is this lifetime you know like I think that you would the one it would be really interesting in a future livestream I by the way I updated it so you can rebuild it but but would be interesting in a future livestream is if we actually tied the executor circle CI executor metadata to the specific like container metadata for example to role ID so you would it would be a little weird though I feel like that would probably get a little crazy but we can work on something right that's always an option hair brained idea no yeah like I would yeah I would be definitely I mean I'm gonna be doing some more things with this after this is really good for me I'm learning a ton so and then I'll be able to collaborate with you right like we've been doing in the past like what was it oh yeah the the remote terraform so that was actually so we were at reinvent and we were at that I was at the booth and he comes over with this like error that sounds like he can't find anything and I was like what I said to Queen oh okay oh lease expired yeah it couldn't get to the vault read error making API request permission denied some of these it's just not able to get access to so could it be in the UI we need to do something no hold on let's see retry attempt so did it get the docker hub you know it was not able to get the docker hub one because I think it needs access to pipeline maybe so if you go back to the vault policies again I mean to stick this you also want to cancel this thing because I actually yeah burned up my credit so I don't even give you crabs it's kidding I get free credits well I everyone does up to a point I actually I actually use the regular the regular liqui everyone standards because I want to understand how our developers are being anyway so yeah I have no superpowers policies I'm sorry yep yeah then click nodejs circle see I it should have no GI Circle CI pipeline read that's interesting but there's docker hub well you don't need technically as it flows down cascades wow that's a big question policy allow access under path you might have to put a star because we might have written this wrong let's see secret yeah slash star yeah cuz I wrote something oh wait I need to edit and then do it yeah perfect so I can update my terraform to figure out that to fix that alright save yeah before we rerun it can we go back to the agent HCl and we'll take out some of the other template files that is generating so let's just do docker hub so that's not what you're looking for all the other ones so everything from 29 down we'll just I'll just comment them yet yep all right I'm safe all right and then is that it all right yeah commented stuff and then BOOM and we'll do a push jump back over to our yeah otherwise if all else fails we use the verb token we are doing we're trying to do this as as with the best patterns that we can but it is a ride a and our Constitution micronized build was cancelled so something different happen no I think you cancelled this build is it an old build oh maybe it is the old No let me cancel already been cancelled I think that's the old built yeah this you I think that's gonna oh there oh it failed commented stuff good ah so my let's see so this doesn't exist projects let's see let's see where so check our code it's on the image oh do you have a get yeah it has get right yeah it looks like it it could just be the project folder like it could just be let's see this is where it saves it so we know that it's not slash project yeah okay yeah I messed it up yeah yeah so I need to add the tilt yeah we might need to just make an explicit direct or some things what has a point is it so we for the this year yeah I think I think and get away with this pretty sure hope I'm pretty sure so let's save it and it will push it and then added route path or something whatever I love the point-and-click being facetious all right so let's go to pipelines the fresh look at this I cancelled this one right I hope so I think it failed on its own because yeah that's right yeah but it did authenticate to vault we know right that was yeah that was the important part the other one was my fail that's why she's no it's well that's what this systems for right well once you get it tight and borrow these patterns for yourself in whatever way shape and form alright so while we're waiting for that yeah so should I talk about the headphones thing oh yeah so my marketing team is basically has put oh wow look it went it went nice we passed at least of the file is saved nice okay so real quick so I work in developer relations normally we wouldn't do this but my marketing team put together a little raffle for this particular session and do you have the link to it okay great so what the raffle is for is some really cool headphones again I'm with developer relations no I wouldn't do this but they asked me to and they want to get way these headphones it's an opt-in situation so that means you know if you sign up for it you're entered into the raffle and they will ping you if you win and right so it's opt-in right so that means you're gonna get communications from circle if you're not a circle person already that's okay but if you are I just want to make sure and be very transparent and clear that you will be you know basically marketed to if you decide to join the raffle normally wouldn't do that but I wouldn't promote that but I'm going to because they did do some work behind it and last minute kind of things so and it's not wrong or bad I just normally don't do that as devrel but if you want to please go ahead and do it again just want to be clear about it did my yeah that's how you know we are it's all kind of muddied waters but I try to keep that at least transparent you know so if you're down with that and obviously right with the new GDP art rules you can opt out right once you win hey I'm subscribe whatever happened here it's up to you I'm all about the developer alright so we did we got a persistent in workspace cool so our next step would be to yeah so we have that file so the next step would be let's build a docker container yeah yeah let's put the docker container by any chance do you have a doctor on this image plenty no but the cube config image at the bottom there actually you could you know what we could actually probably turn let me try some I just out of curiosity um it's called we have a I believe it's called I do this all the time and I can't remember it's set up remote docker yeah let me um let me just look at something yeah I have an old one that I use we use that one just grab it from one of my old ones I normally don't write this config all the time this is usually cut and phase II forget right away if you're not careful here we go so we got to get this and then right and this is perfect cool so we'll just take that out and then right of course it all trusted perfect wait oh I didn't want that back yeah that's good all right so let me get this into a good place okay and then we're gonna say false because I'm not gonna use the layer caching so I think we could we can use this what's going on here with this uh it might be an actual step it doesn't like the fact that it's under docker image yeah yeah yeah yes that's right that's right you're right one more back right yeah those under steps yes he right I'm like placing this stuff all over geez thanks sorry hey Mac keyboard alright here we go here we go you are right alright so we're gonna do this I think here we'll put it know after he checks it out is there a good order of operations for why or it doesn't really matter I don't think it matters it's just for my own head protector and not protecting but my own on the work working process whatever alright so we got that and now if we have that then I can do another I mean you don't even you could probably consolidate it into this one command right like you could yeah exactly that's how I read it to the workspace yeah I would say something like you know do something like run and then actually you might be right let's just do the darker commands from here so you upload yeah so let me grab I didn't I have these we need to yeah I usually use these environment variables that I like to build well we can for this test we can just so we built T and then so these are the the passwords that we need this I don't care we can just use the circle CI sha I think I use up here let me grab my environment variables yeah yeah we can just do something like this for for that I'll put them up here do you think that the export like that what you have with export image name all that stuff actually we should probably do that as part of the template for agent so remind us remind me to go back to go and we'll configure it that way but it will make it easier so you can just say source from that file and then yeah don't worry about like parsing it or reading it I see so yeah right I could that's right you can just like here just this would be the variable and then you just out here say export right exactly you do like export docker login or something and then you can just source from vault docker hub yeah oh I'll be playing with this soon enough so I think honestly well this would be I would say this is the the project name let me grab that from our pin so if I look at the project I like to grab the variables so the repo name there you go this is what nice yeah and then we'll just say dollar and then paste yeah and then that'll save it perfect and then a build number it gives us like the yeah the builds whatever number I'm just looking for uniqueness here alright so so how would I what would be the recommendation to grab these so let's see let's just keep docker login the way you have right so we can just go back to the agent HCl yep Oh actually all right then why don't we just say export docker login okay cools that username yeah exports and if we can get this working I'll be that'll be awesome and then we can come back and do another session if you do that yeah that sound good it's I think it's the cool part of it yeah sure okay so see and then one more yeah and then what did you call like the docker password PW PWD I don't I don't go crazy I don't know the UNIX guys alright so that should do it yeah back to config yeah mom yeah you just say source before dr. dog yep yep so I would say source and then it would be the name of the pumping out so yeah this one right no no yeah sorry let me get back here vaults and then what was it dr. hub hmm you got good memory yes all right the moment of truth here we can sitting there yeah well eventually right so eventually you would you would like you said want to get those those secrets so will well we'll put a pause on it I'll leave this branch up just so and then I'll work off of some other things so like we could jump back into this branch later you know it'd be cooling alright cool so I'm gonna say this theta updated whatever n bars all right and then wait what oh of course this would be nice all right yeah all right now should be able to go point and click the Microsoft way all right let's do this all right vaults boom and go in there hey but oh you didn't like that it's probably my my my image let's see doctor not found all right here it's a big image but it will work go back to the thingy whatever you have and go to cube config okay and then pull the image from there it's a big unit but my personal image that just has a bunch of stuff yeah no I was yeah I was hoping that this would but it doesn't do it you need to have it I just remembered you do need to have it so we'll just yeah well replace rip and replace rightly yours should be good and then I think that works we'll find out I should actually see if I it tests it in there see I actually had to inspect s that's another one that I could actually talk about more rosemary image oh come on hey you know these are things like that's always asking about these like using the circle see I won but like I'm gonna don't worry I'm gonna do some work on this and then I'm gonna share it with you anything and then I'm gonna run it past my my compadres over there it's a circle to make sure I'm not demising correctly - yeah I guess that this is a big image it's because it has docker g-cloud coop CTL terra-man vault on it so it's a big image I want to know how to make it smaller but like I cannot compress it further no I mean that that's just the way it is I think the way you would the way I would do it is a lot of folks like when you're doing CI CD they want to build things like layered so like they do you know they'll do the CRO requests they'll install it every time because you want like to see you know like a like a fresh from a fresh start you know me so you kind of get that uh the view from from again a oh it's the same thing I think my build Oh invalid reference format yes image name it did not like image name okay so let's fix that and hopefully we'll get this and call it a day alright so let me look at what I'm doing wrong so I got a tag I got the circle CI project repo name out of there image name tag and then the docker login is it getting maybe this is not coming across it could be that no it says it's got the docker login I wonder if the source Oh bash env is the boshy and ve ever set like maybe if you just do the export without the echo like don't put it to the bash E&V oh I see I see export yeah we could do that oh you know that's how you persist between runs by the way if you ever need to like if you do a run command within the same image right like I said it opens it like it's almost like building a new shell that's how you do the batch DMV is the way you oh that's right I should have sourced okay let's try that this charges good good ice all right so yeah let's see just and bars are good we're fixing and we're gonna push back excuse me there by the way this is how you would be bug a pipeline and Anila this is cute it's a see they're seeing me using all my like crazy oh did it yeah no okay well come on alright you gotta yada yada come on yeah that is a big image yeah I wanted to like be smaller okay yeah well eventually you know I'm hoping I don't know if you have heard of a unit Colonels but I'm hoping those will make a research a resurfacing people don't want to write things in Oh camel up there we go please ah let's see still doesn't like you huh it apparently either a the echo I think it's still echoing why is it still echoing it could be no don't echo it yeah yeah that's right I just saw that I just yeah you did tell me to take him out and I do no no no it's fine I was like again I keep blaming it on Friday brain and then the single closed it's angel grain no it's Friday brain I definitely a couple weeks ago I was like why does this not work and I realized it was not I didn't actually like commit anything all right that's it I don't care I'm just putting whatever now all right that's that big image is nothing we're good so let's go back to the pipeline I'm gonna let's see we'll just watch it from here like watching paint does anyone in the chav any other questions and we are we how are we doing on time are we have a minute so if we get this fix the clock is awesome okay pretty quiet today but you know we had some we had some discussions and such nice okay well no I'm a I've been I'm logged in as Punk data so anybody wanted like yeah I don't use twitch some new to it oh wow my pictures there that's cool yeah nice now this was fun I wanted I want to like continue this project and finishing this oh wait looks like something's happening yeah there you go the rosemary image comes through that look great yeah by the way this image you know has been it's like a year old or something or 2 years old or something nah no no I just continuously update it I have like oh I just run in spec against it just to check versions and stuff and then I add stuff to it and then take away stuff from it like I'm so using chef inspect yep oh you ought to me all the things that's go oh hey it's in it did it go to docker hub let's find out I think it did alright so occur hub I hope so that'd be awesome oh you don't even know how oh yeah look a few seconds ago they look at that that's awesome all from awesome off from volts it's an area a secret in plaintext insight so one of the things that I'm going to do is put a pin in this right since we only have time yeah and then I think we'll we'll cordon eight and if anybody wants to come for part two where we deploy this sucker to a kubernetes instance and using terraform rate we'll just we'll use that GCP secret engine basically to have rotating surface case yeah that way we how often we expose ourselves to deploying to kubernetes this was very cool I'm so happy because now I have a better understanding of all the new kind of features and and and all the like because I saw you uploading all that code I was like oh and I'm trying to read through it but if you don't get the context I mean it's kind of yeah well next time maybe we can actually go tight we can dive deeper into the the policies and stuff that are configured and because GCP Secrets engines requires a lot of like little pieces to be turned on so yeah let's do that awesome awesome awesome hey great thank you for joining us I appreciate it you can find angel at punk data on Twitter ci well we'll figure out if you're available for another one we'll just you know pop hop on here our next stream is April 17th the same time the same usual format we're gonna figure out I'll let you know what the topic is if you know I can I can come back April 17th if you want but you let me know all right we're gonna make sure the schedules line up and then we'll do part two yeah awesome I'm looking forward to like getting this done against really some things from this and have a good weekend yeah thank you all and stay safe and healthy we'll see you soon

Channel: HashiCorp

Views: 1,393

Keywords: HashiCorp

Id: l6lG7FR5_Ow

Length: 125min 58sec (7558 seconds)

Published: Mon Apr 06 2020