KubeCon NA 2021 Technical Demo: Leveraging Vault's encryption as a service from your Kubernetes apps

all right hitting the button [Music] and you are live hang on we just said we just like pretend that we're not live is that what we do is that is that how this works i think so we're amateurs yeah oh my word i can see myself we better start doing some work rob hello welcome everybody to this uh technical demo session so today what we we want to look at is a problem that i personally have and i need to be able to selectively encrypt some data in my database i don't want to use full database encryption and i really kind of need to do this through application code but i don't really want to use any application libraries because i don't know anything about encryption i'm really not an expert so my my good buddy here rob barnes allegedly has a solution for me um why don't you introduce yourself rob hey so yeah uh like nick said i am rob barnes um and i'm a security magician in this sense right we're gonna do some security magic we're gonna help nick out with his problem um so like he said he's got an application which he needs to selectively encrypt some data um and he doesn't want to do all of the cryptography himself within the application um just out of interest why not why don't you want to do that well it's simple if you ever see my code i think the key thing is that i've got two problems one is i'm not a i'm not an expert i don't know which library to trust so my my kind of natural reaction when i'm thinking about go is that well i'm just gonna fall straight on goes crypto libraries um for a simple fact that i know they're really good but the second problem is what if i make a mistake like i mean i might make a mistake on the implementation so that's one of the problems the the other problem is i've got a kind of well i you know i want to use kind of private key cryptography that that i think is the right thing to do in this instance but then how do i store the private key so like you know if i have the private key on my application if my application gets hacked then the private key gets hacked then basically they've got my data anyway so these are kind of like the cruxes of my problems so then i'm thinking like well okay so do i then do something like asymmetric encryption and then okay you know i can encrypt with the public key and then i can decrypt with the private key and then it just sounds really hard rob i'll be 100 honest with you and and actually the well i know for a fact that there is a better way um and the better way is that vault has an encryption in transit or transit secrets engine um which i can leverage as an api to be able to do to do this so let me let me just quickly show you my my problem my application okay so quickly just share my screen so i have uh i have an application and it's a it's an expense report application if anybody's been sort of following along the last sort of couple of days you would have seen both kareem and rosemary using this application now i've been doing some extensions to this application so i can let me just send an expense to this thing so i'm just curling an expense to to the application and then if i query the database well what's happening is i found that people are actually doing smart things like putting their credit card numbers into the description field because the application doesn't support you know all of those kind of fields so people are hacking the system and that they're doing things like well putting credit card numbers pir all sorts of stuff like that into the description and well that's not great so what i really want to do is i just want to encrypt that description field people can whatever they want in there but at least the data is going to be stored securely at rest and uh and i believe vaults uh encryption and transit is just the job for that exactly um so i think you've said a lot so i'll kind of just fill in any any gaps that you may have left so fault can help you with that problem um it'll do a pretty good job of doing that so like you mentioned we have a secrets engine uh involved called the transit secret engine now secrets engine is the component of fault which either manages the secrets or in this case manages the cryptography right so let's not worry so much about secrets management's uh at the moment let's just think about cryptography so transit secrets engine is the thing that we're going to leverage right and it does a few things for you it manages your keys uh which is one of the problems that you mentioned you don't want to have to manage that within the application it's not even secure to manage that within the application uh and when i say it manages the keys it's not just a place to store cryptographic keys it will also manage the life cycle of that so you know if you need to rotate keys and all of these things here your application doesn't really need to worry about although that vault can take care of that side for you and so rob just just interrupting you i never even thought about key rotation why is that important uh for a number of reasons you know um it it depends on your organization so sometimes there are policies where you have to uh rotate keys every so often um and that's just part of your security posture if there's any suspicion of any type of data leak or anything like that you may need to rotate the keys you know you just have to be ready to be doing these things and you know you just want to do it centrally if you can just go to one place and then your entire application stack who's using all these different keys can just be rotated in that one place then your applications don't need to worry about that rather than trying to roll that over 100 instances of my application it's just in one central store i mean do you want to do that i definitely don't want to do that [Laughter] yeah so so it will manage that for us and i think you you also mentioned that like well obviously is is api first we present you with an api um so the nice thing about that is you can leverage this api to pass it some plain text data using a cryptographic key that's involved and it will give you back the encrypted ciphertext and you can store that wherever you need to store it so i have a i do have a specific requirement so my specific requirement is that like i'm not i'm not a complete security noob i have basic awareness of the principles i want to create separation so my app actually is is two apps i have an expense submission service which is used by the the kind of the web endpoint by the the mobile apps and things like that and i have a report generation service which is used by the back office now all the the sort of the expense application has to do is just store the data we don't we don't provide any way of sort of querying it you know particularly but it's only the report application that needs to be able to query the data so that you know it can sort of finalize all of the expenses and then hopefully pay them pay them back to me so in an ideal world what i want is that separation i want the ability for my expense app to be able to encrypt data i don't want it to be able to decrypt it so if somebody hacks my my expense app which is the public facing one you know um i don't want them to be able to grab my my my key and to be able to to obviously just read the databases if it was plain text so like vault can provide this capability for me yeah absolutely it's um that's not actually a specific functionality of the transit sequence engine this is just built into the way that vault operates it's quite flexible in its um authorization uh policy framework so how we go about doing that is uh to design permissions within vault we have a concept of policies right so it's just a policy document written in hcr generally speaking as developers you normally wouldn't need to concern yourself with that um you would just need to know what the endpoint is that you need to do so what we'll do in your use case is we're gonna we'll set up two policies so we'll set up an encrypter policy which will allow any uh identity that has that policy to use the key to encrypt data and that's all it will be able to do it won't be able to decrypt any data and then we'll set up a second policy and that will do the opposite it will be able to decrypt the data using that key but it won't be able to encrypt or do anything else and then we'll just assign these policies to the relevant services so that way you've got a nice degree of uh separation between the two um which i think is what you're looking for really in your application now next problem i'm using kubernetes how are you what are you what are you how are you going to ensure that my application has the permission to be able to be able to do the the operation um yeah awesome so actually before i move on to that i just want to just make sure something's clear just so that it's not uh misunderstood by you the transit secrets engine is only going to encrypt and decrypt the data it's not going to store it for you right so storage is going to be in your database or wherever that you're storing that information so i just want to just make sure that bit's understood but in terms of how your kubernetes application can gain access to vault and do these things here the nice thing about kubernetes is we have a a vault cycle injector and that takes care of a number of things for us the first thing the first problem we have is how does your your application authenticate to vote right so if we just go back to vault for a second and we look at um how it handles authentication it does that using a construct we call auth methods right so it's the modular plugable system that allows you to bring in your own identity providers so since you're using kubernetes kubernetes is probably the most sensible identity provider to bring this time so what is a kubernetes identity it's going to be your kubernetes service account so what we'll do is we'll enable the kubernetes off method in vault and we will create a role in vault as a role is just kind of a constructive involved which is going to map uh to uh policies so we talked about some of the policies that we have um so what we'll do is anytime a specific service account uh authenticates the vault that it will be given a particular vault role and that vault role is going to have that policy attached to it right so so so my i'm going to put a service account on my application my app correct a service account has a token the token's a jot it's crypto cryptographically verifiable yep with the kubernetes api so i take that jot and i use that to authenticate to vault and that gives vault validates that that's correct and then i can use the encryption or decryption depending on which application i am so all i need to ensure is that my application has the correct service account and basically vault sorts out all the rest well this configuration exactly yeah so essentially you're going to give it the the the the order jwt i don't know how you pronounce it out there and vault is once it verifies that with kubernetes it's going to exchange that 40 volt token which is the core authentication construct involved everything always ends up being a default token all right here's another one for you somebody gets onto my application and they they steal my my vault token but we came over them right game over in terms of what what what are you worried about well are they going to be able to just do apple you know operations on vault forever oh no so the the good thing about about vault is uh tokens are generally speaking short-lived um so they expire you can renew tokens you can get new tokens you know the idea is that the tokens have a life cycle um and that should uh limit the loss radius a little bit right so you can almost think about it as almost ephemeral access to vault right so that's the way to kind of look at it i like that all right now i've been digging through the docks i think i can wrangle the go sdk into action one thing i'm not touching rob i hate to say this but like you don't want me messing with the servers i'm not i'm not your ops guy i can i can just about manage a bit of code but but seriously so you're gonna need to set up all of this uh all of this stuff would you would you mind you show us absolutely absolutely absolutely i'll share my screen and i don't know if anyone here is from the uk but we have a phrase which comes from i think the 90s where we say blue peter here's one i made earlier right does that ring any bells for anyone sure that's for me yeah yeah i mean you're trying to hide your age by not um admitting that you know that comes from the 70s okay cool so uh i have some code here uh which we'll walk through just make this a bit bigger um so like i said the first thing we want to do is we want to enable the transit secrets engine and we want to create a cryptographic key that this application can use um so just just just for clarification rob this is this is like a something that you you as an operator are going to configure for my for my application i don't it's not something that we need to set up every time no no no so basically what's going to happen here is that you won't have to run this code right you don't even need to concern yourself of it the only thing you're interested in here is what the key is going to be called because that's what you're going to have to run your api call against right there's a number of ways you can configure vault to do these things here i've decided to use terraform just because it's the easiest way to manage state um and you know terraform has uh modules so you know you can you can declare all these things in terraform you can write resources for the the sequence engine the keys and so on and so forth what i've done is i've created a terraform module which enables the transit sequence engine and gives us a cryptographic key so if you look at this first block code from line 3 to line 20 which i'll highlight there that's just me calling a module which is available for anyone to use it's open source so definitely check that out we can put links in the chat if you want to have a look at that there and all we're doing is we're telling it the details of the key that we want to create right so we want to create a key called expense report service yeah and we're just configuring a few things we don't want to allow plain text back up we don't want convergent encryption we don't want it to be exportable we do want to allow deletion and that's only because we're running in terraform we want terraform to be able to destroy this if required but you know you can figure out how you need um and you know there's so many different rsa sorry so many different cryptographic keys out there that transit sequence engine supports we've gone with rsa 2048 because because when we did our testing we found that rs 2048 gave us the best balance between security and performance but i mean we could have used a um for 40 96 or we could have used an elliptical curve key right yeah exactly exactly um so yeah that's that's the key that we've opted to use here and we also got sort of minimum encryption and decryption versions and what that is is to do with the key rotation right so if you encrypt some data with a key and then you rotate the key it stores versions of that key so you know it's not going to break your application you know so in terms of the minimum uh uh decryption and and encryption version this will kind of allow you to go back a level to be able to still access your data and not worry about losing that so the module's going to take care of that for us so we can run that and then you know we will go on to your other problem about you want some separation of concerns right you want one service which handles the actual login of the expense to be able to encrypt uh that field that people are putting credit card details into which ideally you don't want them to but you just want to protect them and yourselves and then you've got the other app that you mentioned which is generating a report so that is going to have to read that data there right so we create in a couple of policies here right so this is a vault policy again using terraform resource and we create a policy called report service uh decrypter so this is the decryption and again written in hcl just like terraform is anyone that's familiar with terraform we're just basically saying that against this path here which i'll explain that path in a second we're giving it this capability which is the update permission now why that path is important is you'll come across this when you're working with the sdk is in fault everything is path based so when you run a call against the api it will be your vault address port number forward slash the version of the api which is v1 at the moment it's the only version of the api we have forward slash the path right and this is the path so it's the transit sequence engine um that we're using the action that we want to perform is decryption and the key that we'd like to use to perform the action is the expense report service and you know we're just giving permissions against this path to do this thing here so that's the decrypter service right and then we want to create a second policy which is for the encryption sites this is to actually log the expense itself so same kind of workflow um where this time instead of decrypting we're encrypting but the path is otherwise identical to that and again we want to give it the update capabilities right so this is really nice you've now got your two policies info um but it doesn't really mean much you know you you've you've you've got the transit secrets engine you've got your cryptographic key you've got policies but how do we use the things that we've created so far to ensure that the applications can do the things that you need it to do and that no one else can do these things here right so this is where we move on to the authentication piece right so we did talk about kubernetes and the auth method and the fact that we're going to use kubernetes service accounts as the identity and we're going to map that to a vault role so we want to enable an off back end and that's going to be the kubernetes off back end so it's nothing fancy here we're just switching it on right but then vault needs to be able to communicate with kubernetes um so it needs to know a couple of things otherwise um you know it's just not going to be able to to talk to kubernetes and manage these things for us right so what we're doing is uh we have another resource which is just an off back-end configuration and we just need to give it a few things right so we're linking this this configuration to the auth method that we've just enabled and we're giving it the address of our kubernetes cluster and then we just need to give it a few details about kubernetes so it needs its casa it needs the uh the jwt that's going to be used to kind of give it permissions to check these things and also we need to know what the issuer is for that jwt and in this case we are actually disabling the uh issue of validation uh we can not disable that if we want um but for the purposes of today we're disabling that we can change that in future if we want to so the way so what's going to happen with this so what happens is i give you well i don't give you but i i send vault my my service account token bolt looks at that it then makes a call to my kubernetes api yep it validates that the token is is correct the service account exists and everything is all good before it issues uh a vault token which is why so the vault is going to communicate with the the kube api which is why it needs obviously the ca because we're assuming it's going to be like a self-signed cert for for the api and it needs uh a jwt to be able to access the abi api sorry that that's right yes exactly yeah and i think um normally you wouldn't just be giving out the jwt just the systems like that but i think when you look at the role that vault is going to play in the entire architecture it's uh what you can consider trusted compute it's uh it's essentially going to be um a part of of that that security uh control plan so you you are trusting vault by doing these things here right so just just want to be clear about that but that's exactly how it works and just one last thing as well we we operate a multi-cloud strategy surprisingly we're using hashicorp tools but hey we do right so so i can actually have a centralized uh vault cluster if i've got multiple kubernetes clusters it doesn't have to be a multi-cloud solution i might just have multiple cube clusters because different departments decide to to have their own mini coop cluster rather than sort of name spacing things then you can set up multiple auth methods for the different cube clusters so they can all use this um this encryption and decryption process absolutely for example example say i'm running my my expense report on two different clusters all you have to do is just set up two different auth methods the key doesn't need to be distributed across multiple different things it it just kind of all stays centralized inside of vault well yeah so that that's the great thing about vault but this is also the great thing about terraform as well um so we can literally take this configuration and we could just change a few parameters right so let's just imagine you have a production cluster and a dev cluster for example um you they the code that we're walking through at the moment you can point this at a different cluster and it will deploy all of these strings onto there as well so you can do the same things on there that you can do in prod um and yeah in terms of of the off method itself like you said you can enable multiple off methods i mentioned earlier on everything is path based you would just enable it at a different path so anyone that wants to you know interact with the dev kubernetes will go to whatever path you set for for the dev one if it's project go for whatever one that is um we didn't actually set any path from this in this code because it will do so by default but if you're doing multiple you you can set the path um that's just pretty simple and we're using the internal service reference here because we're actually running uh vault on sorry bolt on top of kubernetes but um of course you could use hcp or you could have vault running on vms or on another coop cluster and obviously all you would need to change there is it needs just needs to be a publicly accessible or not publicly it needs to be a a vault accessible address in which case this is because it's all within cubes networking exactly exactly that so i mean um you know i don't want to go into any specific cloud provider but if we were talking about hcp and you were i know you had your kubernetes cluster on aws and you had your vpc peering that would obviously give your services in your vpc access to your vault cluster and hvnc it's the same kind of thing it's just basic network rules just is it contactable um so yeah so that's that's that's just us configuring the the way that vault can talk to kubernetes to be able to say to kubernetes is this a valid jwt right kubernetes has the final chord it will turn around and say yes it is a valid jwt so fault will say okay it's vault token right and then the app can go forth and do the things it needs to do so we have a vault token now based on this kind of theory but it doesn't have the policies that that we we created earlier on so the vault token isn't really doing much about applications which is where uh the roles come in right so we're going to create a couple of roles so the first role that we're creating is for your first service which is the expense service so what's nice about this is what it's saying is if we're binding the the service account name that comes from kubernetes to this role so if a service account called expense is the one that's submitting the authentication request to volt and it's all approved by kubernetes then it will assign this role uh so we've got a role called expense which is the same as the service account just to make life easier we're in the default name space in this example you would obviously set whatever name space your outpatients generally rob i just use default because um i don't have to i don't like using the dash n command when i'm when i'm typing kubectl yeah exactly exactly um so yeah we're just in default customers that's fine [Laughter] um and the next thing is so you remember you mentioned earlier on about what if someone were to get the token what does that mean for your security right and i talked about it being short-lived uh well essentially what we're doing here is we're attaching a time to live or a ttl to that token so it's going to expire in 3600 seconds right okay so when it expires you'll get a new token and so on and so forth so if that is ever leaked uh the the impact of of an exposure is uh significantly reduced you know when we think about things like threat modeling it's uh it's two things that we're looking at it's the probability of an event occurring and then if that event is to occur what's the impact of that so those are the things that we're trying to mitigate against right so here we're reducing that impact and then the final thing is we're giving it uh we're given any token that this role issues this expense service encrypter policy so this is going to be able to uh any any application that has this token is going to be able to use that that cryptographic key to encrypt data so the role is a link between my kubernetes service account and vaults policy which gives me the permission to perform certain actions within vault or in my instance being able to to encrypt or decrypt data 100 that's exactly how it works yeah so you just think of it as a mapping um so we're doing that that's the expense service and we're essentially doing the same thing for the report service as well uh you can see the only things that have changed here is that the service account names that we bind into this role is now report we obviously got a different name for this role and the vault policy is different so this one is the decrypter policy that we created so you know the same rules apply it's just a different policy okay and with that everything in vault is pretty much set up and ready to go from your applications perspective so vault's configured now maybe we should walk through your app and see how we can make your app uh utilize some of the things that we've done unless you have any questions before we move on no you i know you're dying to just watch me write a bunch of bugs and see uh see all my code fail so let's go that's what always happens when it comes to your name okay so um my application is pretty straightforward so as a kind of what i was like saying earlier on it's broken into two parts i have an expense application which writes and i have a report application which reads from from the database and it's it's just a simple restful full application so if i kind of like look here you know it's a typical kind of go app i'm not using any any fancy frameworks i'm well just using sort of gorilla but um i'm creating a database setting up a logger um and then just my router and my i only have two you know two two end points in in this application api expense which is going to handle the the receipt of uh my events and store it into the database and a health checker for for coop now one thing i do want to mention is that um people are probably like going to be horrified in a little bit to see that how i'm managing database credentials and passing them to my application i see you shaking your head maybe we'll talk about that another time but vaults are actually got some handy tricks on that now i'm going to pick your brains on that at a later date let's get this and i'll show you what i want to do so my code is really straightforward i take my my expense and this is what my my expense structure looks like it just has a name a description trip id cost currency etc etc and i i decode that json anybody who's familiar with with go will kind of see you know they'll know this pattern here using the json decoder decoding the json from the request body if i have an error i'll report that and then all i'm doing is inserting that into the database so just quickly kind of show you there there's nothing fancy going on in in here i'm using uh the sqlx package which i really like for for go and i'm using the the my sql driver and all i'm doing is insert into expands blah blah parameterize query and values for my object so it's you know it's it's it's pretty straightforward it's almost as simple as you can get but what i want to do before i do this insert is i'm going to hopefully be able to kind of get that description there and encrypt that data so let's let's have a look so to do that what i'm going to do is i'm going to create myself a vault client now i'm going to use vaults api and the vault vault has a go api which uh we'll we'll kind of share the links for but uh you can kind of see this just there but the reason that i'm going to wrap this inside of my own client is is purely for from a testability perspective because you know in my my unit tests and things like that i can define a as kind of a simpler more abstracted interface around the the the operations and the the kind of the first thing that i want to do is i want to create a new client so what my my vault client needs is a configuration so the configuration needs the address to the vault server we're going to pass that in when we create the client then i'm going to actually create volts restful client using the sdk here with new client passing it that config and i'm going to set it the token so you're going to need to to be able to provide me the the vault token so i can pick that up from my app i could have that as an environment variable or a file or maybe you've got a cleverer way to to kind of do that we can look at that in a little bit so then what i'm going to do is i'm going to to encrypt some data now if i want to encrypt some data let me let me zoom this up what i'm actually doing under the hood is i'm making a call to volts api to the transit secrets engine and the transit secrets engine method that i'm going to call is we'll get that we will get there we're going to get there in a moment we're nearly there here we are is encrypt data and the method looks like this so um this is the the path that i'm going to call and these are the parameters that my payload that i've got a pass vault and you can see here that i can you know i can do batch input as well so i can encrypt multiple pieces of data at um a single request if i want that's obviously going to encrypt the data and it's going to give me this response so the sdk is is kind of going to handle things like you know handling the the transport handling the request making sure the token is is set on to the request in the right location and things like that but i i need to kind of just implement those those data struts now with go they you know they're super simple so i i have an encrypt request my encrypt request has a property plain text and that's all i need to send then that's going to get me back a response the response is going to be a request id from vault it's going to give me a data block and the data block is going to contain ciphertext so my function to to kind of do that do that encryption looks like this so i'm creating my encrypt data function i'm going to specify the data that i want to encrypt and the key now this isn't the key that i'm going to use for the encryption it is the name of the key that you've set up for me earlier on and that's going to just return the the encrypted data is a string or an error if anything goes wrong so the the first thing you've got to do is that the the kind of the plain text that you're going to pass to vaults api has to be base 64 encoded because it doesn't actually have to be text you know i could i could actually encrypt a file using transit encryption it could be a binary blob so as long as i just kind of base64 encode those bytes i can use that in in this endpoint i'm just using strings because that's a simple example and then i'm going to make it a client request using vaults sdk it's a post because that's what the the encrypt api requires and i have to pass the path so what's the path going to be well the path is going to be v1 which is the version 1 of the api transit slash encrypt and then the name of my key so that is exactly as we have yeah so just point out that's exactly how we set up the policy as well remember earlier on i was mentioning everything well the path thing so everything involved is a path that's the path exactly i get you so then what i do is um i'm creating my strut setting my data and i'm using the sdks um method set jsonbody which is a convenience method which is going to convert my strut into adjacent payload set it onto my request and then i can make my request now i do want to kind of like just mention that if you're using something simple like the logical secrets engine if you just want to access sort of plain secrets in vault then the sdk has a more rich sort of a richer api around those um i'm using the kind of the the really sort of raw stuff but you can see it's pretty straightforward so i'm going to make that raw request the raw request on the client is going to make the call to vault it's going to set the token and do all of that gubbins and it's going to get me a request a response back so vaults api json api it's restful you could literally just be curling this but or you could even just be using goes um built-in net http there is no reason sort of you know at all that you have to use anything fancy but the there's a lot of convenience around using the the sdk it just makes life a little bit simpler and i'm all about that so then i'm going to get my response so my response is going to be of type encrypt response so remember it's going to have that request id that data the encrypt data and all i'm going to do is again i've got those helper methods on in the sdk so i can just do decode json to my type then i can get that cipher text out of there and return it from my function so that's you know that was like pretty uh pretty easy so when i want to use that like how how can i i kind of go about sort of using that well i can just use it like this so what i can do is i can just call my my vault client i've just got this as a as a property here on my my strut i'm injecting my vault client i'll go and paul pick that up in a second but i'm injecting that i'm going to call that encrypt data i'm sorry that's the wrong wrong type it's not the official one i told you i'd make mistakes there we go reference to my vault client obviously this is going to be an interface because then we're going to replace it for our unit tests i'm cutting corners i know this but um you know it's still it's still pretty easy all of this code for for reference um i literally wrote the the entire application i mean rosemary created the the application in um uh in.net so we had things like the database but but i basically replicated all of her code and rewrote everything in this in go uh in about three hours or so so it wasn't um it wasn't particularly difficult that was that was everything it wasn't just adding the vault stuff so i encrypt the data i'm encrypting the string from my my expense i'm using your key that you're providing me and then what i'm gonna do is i'm going to set that back onto my expense object before i insert it into the database and everything should just work fingers crust yeah so the decryption so i need to handle sort of decryption as well now decryption is going to be in my my other service but what i'm going to do is i'm creating a shared package this this vault client is going to be the same as my database client is shared across my two services if you want to kind of go and keep those independent like you can do but but in this instance they're pretty tightly coupled anyway so you know it's it's absolutely fine you're not really breaking any any sort of microservice rules or anything like that if there was such a thing so so decrypting data so what that looks like is again we we need to kind of create a response uh sorry a request i need to be able to create a request because i'm going to call a slightly different api i'm going to call the decrypt api now my decrypt request is very similar to my to my encrypt request in the at a top level it's a json block which has a single parameter called ciphertext and that's going to give me a response which has a parametric data which has a parameter plain text so it's you know it's fairly fairly similar to to what we've we've just seen and the function for doing my decryption well it looks like this so again decrypt data the string that i want to decrypt and the key not the actual key but the name of the key i should just rename this name key shouldn't i that i'm going to use for decryption so again walking through those steps i'm making a call against vaults api now this time the api that i'm going to use is decrypt data so again the documentation i'm just using to be able to find all of the parameters and various things it's actually pretty good i'm wishing that i'd wrote it because i really like it but um everything is is there that you you kind of need in terms of the the various parameters etc so that request there you can see here that we're going to kind of just do that ciphertext and it's going to give us back a response containing the plain text which is again base64 encoded because my instance is a string might not be you might be encrypting a file so we get that back again the all of the kind of the requests i'm creating this um this vault client request which has all of these nice helper methods so i can just use set json body if you were doing this just using plainnet http you're going to use json json.marshall things all the the encoder i'm going to return an error if anything goes wrong i make the raw request i do the decryption decode the this json that i get from vault back into my my entity and then what i can do before i return it because it's going to return me base64 i'm just going to base64 decode it and return the the plain text from from my function so it'll be you know it's it's pretty pretty straightforward there like just to kind of show you the difference there again i'm using like transit decrypt now because of the way you've set up the vault keys rob even though i've got my encrypt and my decrypt methods in this same package if i use a token if i have a token for encryption say in my expose service i'm not going to decrypt will just fail it'll give me like a 403 right so i don't have permissions to call that api is that is that correct you get a permission delay correct yeah it's very uh brand new least privileged approach yeah yeah and then vice versa obviously my my report service which can decrypt is only can will not be able to encrypt encrypt because it doesn't have permissions to call the uh the encryption path exactly that's exactly how it will work perfect so let's look at the the the implementation of that inside of our report app so if we go over just closing the gosh i have like so many windows so this is a lot of bugs you can see i write bugs i i think you've got to have a lot of files open if you want to write a lot of bugs so my my report application is is exactly the same well it's not exactly the same but it's you know it's very similar to the the expense application it's just a pretty bare-bones go micro service so i have a logger that i'm creating i'm just sort of reading in some environment variables which allow me to do some some configuration um creating my database connection and again i'm using that shared package i then [Music] create my expense handler which has two endpoints a health and a get method for expense to to return the the expenses so let's look at that so my my health handler just looks like that because everybody implements health handlers like that that's why your code doesn't work rob that's why that's why you can't trust your kubernetes liveliness and readiness probes because you're just doing that no comment it's what i do anyway they're fine um but the handler itself so what are we doing we're not doing a lot you know it's again it's a really straightforward service i'm getting my expense items from the database so again using that sql x all i'm doing is select from star from expense items and i'm just returning that object the really nice thing about sql x as opposed to just kind of like fewer go sql libraries is that it kind of wraps a little bit of functionality which i'm kind of a tangent here but i like it a lot but so i can make a select statement against my database i give it the query and i give it an object or sorry a strut that i want well in this case a list that i want the the query to be decoded into and sql x will will automatically decode the data from the table rows into this um this list of expense and all i have to do is just simply augment things with with the id field i brilliant package if you're not aware of it check out big big props back to my back to the point so so that's all i'm doing now the problem is that get expense items the expense items are obviously going to have the the description field encrypted now i don't want that for the report i need to decrypt it so how am i going to do that again you know actually really really simply so what i can do is i'm going to to take those encrypted expenses and i'm going to create myself a new list i'm going to loop over my expense expenses and for every expense i'm going to call the decrypt data method on my my vault client so that's the one that i've just written here which is in my or shared package that's going to make the cold vault it'll decrypt the data i then set that back onto the object and then i can basically return that to the um to the end user and and it's really you know it's really that that straightforward so the only kind of last little tweaks that i really need to to make is while i i need to um i need a couple of environment variables because i'm gonna need to be able to know where the vault server is and i'm going to need the vault token i guess um and then i can just you know create a create an instance of my a vault client and pass it into my um my handler which is pretty epic so where am i gonna get the vault token from rob how do i get the the vault token because what i have is is a kind of a pretty bog standard kubernetes deployment here so so we mentioned everyone that there's there's the cycle injector okay um so that's what you're going to want to interact with and you can just do this in native kubernetes right so you're on annotations at the moment you just need to stipulate a few annotations and it will transparently take care of all of this uh for us right so i think there's an interesting one at the bottom there uh the cash and agent cash enabled which pertains to a problem that you mentioned earlier on about the token expiring and how it's going to keep it updated this is the thing that's going to allow it as long as you set those annotations yeah your application should be able to grab that token and we'll default okay so i own this config file so as a developer i can you know i can manage these annotations i don't need to do anything else other than make sure i've got the right service account the right annotations so the the agent inject annotation like correct me if i'm wrong here so what this does is when i when i when i submit my pod to the kube um cluster bolt has a kubernetes controller running the kubernetes controller picks up that annotation and it will automatically add a sidecar to my to my application which is responsible for communicating to vault and getting secrets and all sorts of things like that exactly yeah so that's that's that's the whole point is that that cycle injector can take care of that whole vault interaction for off your application need not worry about that because can you imagine if you had to i mean there are cases where some people have to do this but if you have to implement the whole authentication or flow inside of your application there are so many different use cases that you need to kind of uh build your application uh to have resilience against so um robert's having what agent do it trust me i've done it i've been here i've been there there you go i've i really i i exist before this this magic was around i've had to do this the hard way and uh i'm taking a break you can retire from the hard way and live the easy life from now on your fault psycho injector will do it for you and then that's going to create so that's going to create an init container which is going to do the vault authentication it's going to take that kubernetes service account pass it to vault exchange it for a vault token and then that's going to set up the agent running as my sidecar so i have to specify the role that my i'm going to use and exactly the one which is linked to my service account we're just keeping things like a one-to-one mapping because it's nice and nice and simple but we don't have to um and now this this agent cache enable so this is the this is a little bit of secret source right so when we were kind of saying you need a vault token right and i was asking you how do i get a vault token to my application the thing that you didn't tell me was i don't need one because what you didn't tell me was that the sidecar can automatically proxy the requests for the app and automatically add the correct vault token to the request so basically what's the address of the vault server i don't care and i don't need to know because i'm just going to use localhost just talk to the proxy talk to the property we'll take care of it that's what's all you have to do don't need to care because the proxy's got it and the proxy's managing it for me so i don't need to worry about rotations renewals or anything like that that's all 100 transparent for me i like this that's that's exactly what i didn't tell you so so that's all there is to it really um you know we've got the service account there we um we're we're doing some bad stuff and i did say earlier on that um look never do this never ever do this don't don't put well for a start don't don't have your my sequel database credentials as root and password but you know don't pass credentials like this uh in in a way that can be discoverable this can leak into your see icd it can leak into your git like it it's just bad practice now bolt actually has some capabilities and if you watch barry's rosemary stream yesterday sorry barry if you watched rosemary's stream yesterday she showed how vaults uh actually has dynamic database encryption capabilities so you can actually inject uh a dynamic database credentials so um we're just keeping things simple and and that's all that's all there is to it here so let me let me just um i don't need these these things here because i've already got those configured so let me run my expense report okay apply uh [Music] oh wow you know that isn't my muscle memory now [Laughter] right so the pods initializing i've just i've just restarted my application there now there's there's only one container specified in the app but i've got two declared so let's let's take a look at that and if we just back scroll a little what you can see is where are we there we go that vault agent which is automatically added to the thing and um there's somebody at my door which my dog is ecstatic about as you can hear so what we're we're gonna do now is let's let's kind of just um run that curl again so i'm going to to just kill that api expense at the server and then just to kind of show you the you know the payload there because i don't kind of want to hide anything um this is the payload that i'm sending to to my api now if i select from my database again you can see my my previous row which was using the old version of the application which is not encrypted and my new version which is encrypted and this is false ciphertext the v1 there is the the name of the key because if we're using sort of multiple keys that's the version of the good thing right it's the version of it of the game yes so yeah i mean the key version yeah sorry yeah um so that's that's awesome so let's well let's clear that database down and [Music] let's just apply the same thing to our report so let me just go and clear this up because i don't need this got my service account again those same annotations the role is obviously different because this is this is the report app so i'm going to get a different different key with different permissions everything else is the same please don't do this my sequel thing like i'm a bad human i know it but you know please please don't do this do this oh you're a good human bad developer there's a difference yeah yeah good point good human bad developer um so so let's uh let's apply apply this one too and so we can see see there that that is initializing and [Music] have i done rob broken it rob what have you done let's have a look crash loopback let's just clear this down come on we were actually creating this stuff with terraform so with hence that error message that we got before okay so now there we go i don't know why i thought it had three containers before it must have been a uh that was kubernetes fault that was not my code no comment shout out to digitalocean we're using digitalocean docs today um as a dev i think this is a probably the simplest way to to run sort of kubernetes in production dev or test or whatever you want i love this system i think it's really good so let's again we're going to just insert a record into our database we're just going to check that that's there you can see it's there we can see the the ciphertext now if i retrieve that data what you can see is i've just hit the uh this this public am api is my report and you can see that i've actually got the decrypted text there so so everything is everything is working i wasn't lying there is no smoke and mirrors there was a little bit of a bump but um that is literally how you can use vault's encryption uh encryption service or transit secrets engine to do um rsa key based encryption of your data so we're encrypting that storing it at rest on kubernetes and it's easy enough for me to understand i hope that uh you all too can understand and if you can't and you have any problems you know then please uh please look out for us either we're often on streams you'll find the [Music] live streams going on on twitch you can ask questions on hashicorp discuss the forum we're happy to answer anything on there and uh if anybody does have any questions we're happy to hang around on stream for a couple of minutes i know we're we're kind of running three minutes over um caitlyn has a oh sorry catelyn has a question um oh rosemary has has answered it but yes um it would work with stored procs um catalan because all you're doing is you're encrypting the text using vaults transit secret before you add it as a parameter in your stored procedure um like theoretically and i know you can do some really fancy stuff with database stored procedures which you should probably never do but you can do it there's a neat experiment in something like my ms sequel you could actually make a direct call to vault you could even have your my sequel stored procedure um pulling vault to do the encryption you should probably never do that but you probably could but but in a direct answer to your question yeah absolutely because all you're doing is exactly the same as what i'm doing here i'm just encrypting the parameter before i pass it to the stored procedure or the sql query to then be inserted into the database and [Music] okay thank you very much the live stream that's been i appreciate everything you've done to help me and i'm i'm really excited to be able to to use this and to keep keep things safe and secure like like and subscribe you know this is really important people but um [Music] that's what youtubers say anyway um we better go we're both youtubers anyway aren't we yeah oh yeah check out devops rob's channel and um as well i haven't really seen that next channel's magic but yeah i appreciate the love and and definitely also check out uh rosemary's live stream from yesterday which is live on youtube some of the problems that we talked about here and also kareem's where he goes into depth on how all of this functions with terraform and um which is kind of actually a really excellent way of managing kubernetes so those three parts hopefully will keep you going for a little while enjoy kubecon thank you so much people i'll see you next time peace

Channel: HashiCorp

Views: 819

Keywords: HashiCorp, Terraform, Mozilla, Mozilla Pocket, GetPocket, CDK for Terraform, CDKTF, Infrastructure as Code, AWS, TypeScript, JavaScript, NodeJS, NPM, ContinuousIntegration, ContinuousDeployment, PagerDuty, AutoScaling

Id: bLA6raoOzT8

Length: 65min 29sec (3929 seconds)

Published: Thu Oct 14 2021