Securing Kubernetes Secrets with Azure Key Vault

hello everyone welcome to this video so today we're going to discuss the azure keyword provider for kubernetes it's a basically um a provider that allows you to pass azure keyword secrets into kubernetes pods so there are many other solutions that you can you know use um that more or less do the same thing for example the hashicorp fault as one example but if you're using azure i find that using azure keyword provider is basically you know very convenient since you have everything in inside your azure cloud and just easier to manage there are also other ways that you can more or less do the same thing but they are very convoluted in order to take a look at them let's first discuss what the basic premises and what the basic problem is and what is the problem we're trying to solve so typically you'd have a you know um in your repository you'd have a deployment um yeah and here this is basically a deployment yaml for a website the website depends on the backend database in order to function and um the backend database of course needs a host name username password and so on and so we have a very clear problem here is that these are in clear text we don't want to pass these parameters or to pass these environment variables into the pod in clear text for an obvious reason even if you could some way you know construct the cml file using helm or using your pipelines if you can construct this in a secure manner and then pass it into the pod you probably wouldn't want to do this in the first place and the reason is um you can you know let's deploy actually this pod so that you can see apply dash f and then the deployment yemen that we have and so let's get the pod and this is the pod let's describe the part to see the contents of it and you can see here very clearly you know db host username password and everything is included text so apparently this is of course not what we want to do so um what we can do is use secrets and this is for example a secret file and the secret yaml basically creates secret data and it's coming from string but once this is saved inside kubernetes it's going to be saved in base64 string so let's deploy the secret as well so again let's just clear this and do cube cutter apply sacred dotm and now we're deploying our secret so now you gotta get secret and this is our secret so we can do better describe secret very good and you can see here we have you know 12 bytes of host password and username we don't exactly see the real values i mean there are ways to get the real values behind it but that's not our concern like it's somewhat more obscure right now and you know you can secure this using different types of permissions and so on so then okay if you can do this what's the problem well the problem is if you move this pod to a new cluster you're going to have to move the secret with it or basically redeploy the secret with it so you're going to have to deploy the secret using your pipeline you know and again the pipeline is facing the same problem um it has the you know it now in a different file so here to make this um use let's let me do this very quickly now we need this to to come from the secret so instead of value here we said value from the value is going to come from a secret key ref and the key is here is the host and then you also need like the the name of secret itself and it's this nginx secret so let me copy that and do it like this so basically here we're telling you know get the host from this secret and the key to look for is host which is this one so it's gonna get this value right so things are good now we can you know duplicate the same thing now everything should be fine i can actually redeploy my pod so cube cutter apply dash f deployment.tml this is the part so i get the new one and describe the pod and now if we look at the secrets you know it is set to the key host in secret nginx secret and again it's it's a little bit more secure um and the the only problem is i still need to to do this so there are ways that you can you know like instead of putting the real secret here we can do something like this so let's show you so instead of this we're gonna do you know underscore db posts and i'm using double underscores here as placeholders and this is basically going to be a placeholder so now i just want to commit and now in my pipeline so i'm just creating a brand new pipeline starting with an empty job so it basically does nothing and all i need to do is to come here and do a replace token i can add this job and this is a free job of the market so target files is you know ml files you can even choose the like the root directory so anything in deploy right here so you can target the file specifically and here is my token prefix and suffix is a basically double double underscore we talked about so it's going to look for a variable named like db host for example or db username and so on we add a publish build artifact job and now we need to add variables to our pipeline so these can come from variable groups as well like if you do if you go to environments and you can have variable groups i'm just going to go like the easy way and just add pipeline variables right here so i can add you know db host is secret host and you can actually lock this so it's like a password and we just also need to copy those yml files so let's search for a copy job and this is copy files and so our destination is the build artifact staging directory so this is the target folder and the source folder is going to be you know whatever is in this deploy so let's cue this and i want to show you what it does and so we have one artifact produced we can now click on this artifact and you know see our drop folder it contains the secret but now if we take a look at the secret i've downloaded it we can see the proper values has been you know replaced and inserted so this is the original one with the tokens and then the pipeline has replaced the tokens for so again this is one solution that you can do but as you can see it involves you know a lot of you know putting placeholders in the right places then making sure the variables are all with the proper name and um it's like a big mess to be honest you know sometimes it's it could be a good solution but what if you want to secure this even further you know use key vault now this is also going to be for other stuff like also if you if you if you're ever publishing your website using you know https or something you you probably want to assign a tls certificate to your ingress and again where is the certificate going to come from now it's going to be very difficult to pass a certificate using your pipeline but if you're using azure key vault then you can just save the certificate there and it's going to take care of itself so we're going to discuss that um part now we go to the deployment now there is i'll share the link for this in the description and you can see there is a walkthrough here that is very easy to use so i've already have my easy login and everything like that so if you're doing that you know make sure you follow the steps exactly so the key vault resource group that we want to to create i'm just going to go to the resource group where i have this aks my keyboard location is west europe you know choose the location that is closer to you i'll i'll use my own name so the next step is to add the helm repose and so this is basically the helm repository where the csi driver is installed and then i'll just deploy the package to my kubernetes now before running this make sure that your cube ctl config is pointed to the cluster where you wanted this to be deployed so i'll just run these right here and then run these two so now we are deploying the csi um secret store provider for azure keyword onto our cluster and now we have to install the keyboard so i'm not gonna install the group because the group already exists but i'm going to immediately go and install the keyboard good so now my keyboard is installed and i can actually verify by going to my azure resource group and here is my keyboard that i want to use so um we can actually go ahead and start creating the secrets i mean in the walkthrough um there is a very you know good way to create the secret here but i'll just do it using the ui instead and i'll go to secrets generate a secret so this is going to be my you know db host good now we have all our secrets saved into this um keyboard so the next part is okay so how is the kubernetes going to gain access to this keyword so we need to do access policies and we need to add an access policy to allow our key our kubernetes to access this keyboard but it's going to need to do so you know to have some kind of service account to to do that so the easiest way is to just use a service principle and what service principle is like a service account um so this is basically the process of creating a service principle is we run this azad sp which stands for azure active directory service principle create for rbac means create for role based access control so i want to create a service account that can be assigned permissions onto azure and skip assignment means don't assign it to anything i will assign it manually and then you know you use the the name of the of the account that you create and you want to query the password because you want to save the password in this variable right here and then you do the client id so and there's a small typo here you should add a dollar sign before this part so and now we got the id in this value as well and now just to show you what we created you can come to your azure active directory tenant and you can go to app registrations and here is in app registrations and you can find the the service account that you just created and you can see here is the client id tenant id and all the details and the secret we created is just saved here so you will find here are back and there is a hidden secret here so this is basically what what we've done we could have done it using the ui but you know using command line is much more convenient and now we need to set the policy so you can do it from here i'll also do it from the interface just to show you exactly what it entails and so we add an access policy and i don't want to use a template so the access policy can only get secrets that's the only thing it needs to do if you're using you know an ingress for certificates and so on you're going to have to add get keys and also get and list certificates i think but i think you don't need to do that right now in unless you actually need to go and grab certificates from the keyboard for now secret permissions is get and that's all we need now we select the principal who are we assigning this policy to and the policy is assigned to our account that we just created right and then we add so now we have added permissions for this service account to our service principal to do get on the secret and we save don't forget to save and now we need to store um the secret um the the credentials of the service account which is the client secret and client id that we got we need to store them as a secret in communities good and the secret is created so actually if i come back here i can do cube cutter get secret so now this secret is going to be used by the csi driver in order to contact the key vault so the next step we need is to actually deploy what is called the secret provider class and the secret provider class is basically a definition for the key vault that we're going to use and how to reach it and what to grab from it and so on because maybe my keyboard contains other secrets that i don't need to bring and we're also going to specify how to how to reach them so let me just copy that to here and let's edit this very quickly so the first thing we're going to edit is the keyboard name right here i know my keyboard name so like so is it using the pod identity so we we use this service principle to to to contact the the key vault or we're using a service principle but we could have used any other methodology as well so we could have used the identity of the pod so the pod itself is going to be responsible for providing those permissions or we could have used the vm managed identity if this was running in the vm and the vm had a managed identity we could have used the managed identity we could also have created a user assigned identity which in my opinion is a cleaner way so consider using a user assigned identity later on you can find more details about in documentation i think a service principle is just you know easier a little bit to to handle so i don't need to create anything else and here i put the name of the keyboard that i want to access and so this is the name of the object we're creating don't confuse those two this is just the name of the class so you can add like class to it but i'm not gonna it's just easier for me to to see things like that keyboard this is the actual name of the keyboard and then what do we need to grab from it so well we need to grab um and a secret object from from it we can also grab other stuff like certificate not just secret but here we're grabbing a secret and we know the name of the secrets we want to grab it's db host and this is a list so you can actually add to it like this and db user and db password so these are the three things we want to extract from the keyboard and just to understand what we're doing we are grabbing these three so um just set the object name that we want to bring right and then the tenant id and the reason the tenant id is needed is that it's going to need to authenticate to this key vault using the client id and client secret so we created a client id and client secret but we didn't tell it which tenant it is so i have my tenant id right here i'm just gonna copy it and paste it so this is the tenant id onto which the um app registration was done and i'm gonna run this good so now i can do cube cuddle get secret provider class and i have my secret provider class right here so you're gonna wanna do an individual secret provider class for every key volt you have so if your applications uh need multiple key volts so we need to grab secrets from multiple keyboards then you need to do multiple secret provider classes for it and now we just deploy our um secret so the way we deploy it here is using a volume mount we're going to talk about that very quickly and just copy this part right here and i will go to my deployment yammer and in the end right here i'll paste this so the volume mount here i can call it you know secrets mount and have to use the same name here so first let's let's see the volume definition it's the secrets mount and the driver i'm going to use is the secret store to see so i do commentaries with io this is just the the name the the namespace of the driver and um is it read only yes i don't need to change my secrets and what secret provider class am i going to use so you use the secret provider class with the same name as the keyboard that we want to access and what are the credentials are you i'm going to use in order to connect to this key vault so it's the secret name here secret store creds and that is the same name as the secret we've created it contains the client id and client secretary of the service principle so now that's all done we can actually save this and deploy this part so when i get the pod and you can see here this spot so first let's describe it to show you something and we can see here the volume that we're mounting is the secret volume right here and it's actually mounted on this location secret store so um if i hop into the container and i can do that by cube cutting exec t and then the name of the container sorry of the pod and then bash so now i'm inside the container if i do an ls i'll see a mount folder right here and if i do slash mount we see a secret store and inside i will find the values for db host db password and db users so i can you know here is the you know secret host super secret password and so on so this is fine however my application is not configured to grab the secrets from this location it is actually configured to grab those secrets from the environment variables so how do we mount these values as environment variables and um there actually is a very easy way to do it so i'll take this whole part this is the secret provider class and i'll just create a new file right here i will call it a and i paste the secret provider class now the the objective here is that we want to create secrets in in kubernetes so we already have our deployment looking at environment variables that come from the nginx secret so we want to change that we want to we want to come from a different secret and we want that secret to be created by our provider so we come back to our provider class and we're going to add secret objects and the secret objects basically tell it what the secret name do i want to create so secret name is going to be nginx secret which is the same name i'm actually using in my deployment right here and the type of the secret is opaque so this can be any type you want you're basically creating a secret that that is coming from um from key vault but but the secret parameters you're defining right here it is like you're defining this file um so if you're creating a tls you know so your type is tls and then you supply the tls details as follows so here we're creating an opaque secret and we have three keys and three param three values that we want to pass so let's do that as well in the provider class there is a data part and the data is coming from where the data is coming actually from the object below so we just you know take the same name as here so the object name is db host and the key we want to create is the same name as the key we're expecting in the deployment file so the key is host so let's repeat this and this one is username and this one is password i think and here the names are dbuser and db password so now let's just revise what we have done we want to create secret objects as a part of you know we want this provider to create secrets for us the secret we're going to create is nginx secret and it contains its type opaque and it contains three keys which are host username and password same as these one host username and password and the values for them are going to come from this object in key volt and this object on key vault is actually defined below right here right so that's basically what we're creating so before we proceed let me delete the existing secret cube cutter get secret just to show you and this is the nginx secret that we that we depend on in our file so we want to delete that because we're going to recreate it using our provider so now the secret is deleted i'll also delete the deployment so now let's just update the provider class that we've created so we have our provider class right here and we we have it in a file right here so we want to just update that so cube cutter apply secret provider class so let's get the secret provider class and let's describe it so that we see the details good so we can see here the parameters we are grabbing from keyword three secrets db host db user and db password and we are using those to create a secret with the name host sorry with the name um with the type opaque and then secret name is nginx secret and these are the key and value for objects inside the secret so that looks very good very promising and now we need to deploy our application again so just for you to know if i get all the secrets now we're not going to find the um the nginx secret right now and the reason is it is only loaded when it's needed so it's not there unless actually the application is deployed so now i do cube cuddle apply the deployment and now if i get the secret again i should find my nginx secret loaded right so the secret is automatically loaded by the secret store provider and let's describe it to see the contents and again host password and username and again that that means that our application could use it so if i do a get pod and now this is my pod let's actually describe it so you gotta describe what we can see of course the volume is still mounted um but here are the environment variables they are coming from engine secret as as promised so let's also hop into the container again and let's actually echo the value of db underscore host and you can see the environment variable is loaded properly but now this is coming from key vault instead of everything being you know written statically or whatever um so one just peculiar thing to tell you about this is that well theoretically speaking now you do not need this volume mount um because now you're passing secrets so let's remove this volume mount right here and i want to show you something peculiar so we've removed the volume and the volume mount because we just want to depend on our secret right so now delete the deployment and deploy it again so again deleting the deployment should remove the secret as well because the provider is going to see that this secret is no longer needed so if i get secret it's still there but it should be removed at some point and now it is removed so let's deploy again the deployment but after we have removed the mount from the volume mount and the deployment is created so now let's get the port and create container configure so why is that let's describe so the secret is not found and it's never going to be found the in reality you need this volume mount to happen and the reason is this is the only way the provider knows to actually load the secret um even though you might not need the volume and set so now we can bring this back and deploy our container again and now our container is running because it could find the secret eventually so there you go so um what how does that affect our workflow well now you can actually save all that um into the now you don't need the secret file anymore you can just delete it so because the secret is going to be created automatically as a part of provider working for us so now i i can move this file now this has no secrets whatsoever it's nothing special in it um the only part is the environment preparation and now so for every new cluster i create i have to include the the process of adding the provider to it but if that is already done then i can actually take this deployment and this secret provider class and deploy them together to any new cluster that is just configured to begin with to to deal with key vault and it will automatically do that um it's it just greatly simplified moving the pods from one location to another and the added benefit of it well all my secrets and certificates and everything are saved here securely in key vault so it's not you know exposed in in pipelines or you know like there is only one place where the secret exists it's in key vault and i don't need to worry about you know copying the secret to multiple locations and the secret doesn't need to exist in my pipeline anymore so i can actually come here into the pipeline and you know simplify my pipeline greatly now i don't i want to deploy those yml files but i actually don't need to do anything in it i don't need to modify the ml files just deploy all the ml files to the to the artifact drop and i can remove my variables from here so no need to save variables anywhere and everything just works fine now there is a lot more to this provider it can do like there is stuff related to certificates that are really fancy that you can do and so on i'll let you explore that that just a very quick intro on how to use it and as you saw there are many steps to do but they're all like steps that make sense really simple steps nothing too convoluted i hope that helps and you know i hope you learned something

Channel: Zoom Speaks Tech

Views: 3,501

Keywords: azure, kubernetes, key vault

Id: KGjxZmlb4Bo

Length: 31min 59sec (1919 seconds)

Published: Sat Feb 27 2021