Kubernetes Secrets Management with ESO | Kubernetes Tutorial | KodeKloud

hey everyone it's Sanji from code cloud and today we're going to take a look at a tool that'll help us manage Secrets within kubernetes the tool is called external Secrets operator and so in this video what we're going to do is we're going to start off by going over some of the challenges that we face when we work with the default Secrets within kubernetes some of the limitations and we'll take a look at why we would want to use a tool like external Secrets operator to actually manage our secrets so let's go ahead and get started and we're going to start off by going over some of the limitations of Secrets within kubernetes all right so why do we need need external Secrets operator so the external Secrets operator helps us address a couple of different issues that arise with working with Secrets by default within kubernetes so let's say that we create a secret in kubernetes just a plain old generic secret what happens is kubernetes Will encode the secret for you by default and a lot of us have a tendency to get this mixed up because we assume that encoded means that it's secure and nobody can read it not exactly right usually when you wanted something to be secret you would need to encrypt it that that means nobody can read it encoding does not provide us that functionality anybody can decode a secret just as easily as they can encode it so if you take a secret and then upload it to GitHub then anybody that has access to that repo will essentially know your database password because once again encoding is not the same thing as encryption so this is the first issue that arises when it comes to working with Secrets right we need a way to be able to check our configuration into GitHub without exposing our secrets and our passwords to anybody that has access to our repo now the next issue that we might come into is let's say that we have more than one kubernetes cluster and we might be using the same Secrets across both of our clusters and managing your secrets across all of your clusters right imagine instead of just two clusters you have hundreds of different clusters and they're all using potentially similar or the same Secrets then we would have to individually go to each one and configure it and manage it and update them and rotate them accordingly and obviously that's going to be cumbersome so we would like some ability to be able to sync our secrets across all of our clusters so that we don't have to manually go in and update them one by one in addition to that it's very common for organizations to already be using external Secrets stores to manage their secrets so things like Hashi Corp Vault aw Secrets manager Google Secrets manager and Azure key Vault and so since all of our secrets are already being stored and managed by those external software we need some way to be able to have them sync up with the secrets that exist within our cluster so that our pods and our containers can actually use the secret secr that are defined within those external Secrets stores and so this is where the external Secrets operator comes into play so the whole idea behind it is to help us address some of those issues that I just talked about so what is the external Secrets operator the whole idea behind it is that it's supposed to sychronized secrets from external apis into your kubernetes cluster and so ultimately it's going to allow you to manage your secrets in a central place like hashy Corp Vault or AWS Secrets manager and it's going to allow you to take the secrets that are stored Within in one of those tools or software and have it sync up with a secret within your kubernetes cluster so you're no longer managing your secrets directly within your cluster instead you're defining it within Secrets manager and your cluster is going to automatically sync it for you by utilizing the external Secrets operator and so since it's going to actually pull in these secrets from the external Secrets manager you no longer need to worry about checking your secrets into G because your secrets aren't actually stored within your kubernetes manifest they're stored within AWS or Google or Azure or whichever cloud provider you're ultimately using and it's going to dynamically pull those in so you don't have to worry about accidentally checking it into get so how exactly does the external Secrets operator work to get it up and running in your cluster the first thing that you have to do is actually install the external Secrets operator and we can utilize something like Helm to actually deploy all of the necessary resources within our kubernetes cluster so Helm is going to deploy the operator and it's going to deploy a couple of extra things it's going to deploy a couple of custom resources the first one being a Secret store so what is the purpose of a Secret store what is it configuring what is it actually doing and contributing when it comes to the overall solution let's say that we are storing our secrets in aw Secrets manager and it doesn't really matter which service you're ultimately using you could be using Azure key ball it's the same idea our operator and our cluster needs to actually pull in those secrets from AWS and when it comes to working with any one of these tools or any one of our clouds we need to be able to provide credentials so that it can actually talk to the cloud right nobody has access to our secrets by default they have to have explicit permissions and they need to be able to provide you know their access keys and their secret keys so the Secret store is all about configuring and telling your operator how to connect to the secrets manager what user to use what credentials to use so that it can successfully pull in the secrets from aw Secrets manager so when we create a Secret store object it's going to look something like this and if you take a look at the configuration it's going to be pretty much exactly what we would expect to look so you can see here under provider we've got AWS and you can see we've got some extra configuration like what region is our secrets being stored in you can see here it's Us East one and then we've got the authorization so this is going to be a couple of things like the access key information and the secret access key so that's the credentials to be able to actually talk with AWS we need to make sure that our operator can actually successfully authenticate with the AWS service so it can pull in those Secrets now you can see here that there's a secret ref so it's basically saying that we have to store the access key and a secret access key within a regular kuet secret and I know there's a lot of secrets that are being involved in this lecture but now what we have to do is create a plain old kubernetes secret and it's going to store the access key in the secret access key so to do that we can just run cctl create secret generic give the secret some whatever name you'd like and then you can see here we press in the access key and the secret access key so the access key here is going to reference the key that we provide here and the secret access key is going to reference the secret access key here so that's how we pass in the AWS credentials or whatever cloud provider credentials that you're ultimately using you provide those credentials within the Secret store configuration now the next custom resource that gets created by our operator is going to be the external secret so this is now how we're going to be creating Secrets we're not going to be creating Secrets using the regular kubernetes Secrets anymore as users instead we're going to utilize this custom resource called external secrets and let's say that in our AWS Secrets manager we've created a secret within the AWS Secrets manager not kubernetes but within AWS we create something called DB creds and it's got two key value pairs the password and the username so what we want to do is ultimately one of our pods or containers are going to need to have access to those credentials we create a external secret object now if we take a look at the configuration for the external secret object the first property I want to go over is the refresh interval so this is basically going to tell our operator how often to actually query the AWS Secrets manager for our secret looking for any changes or anything like that so every hour it's going to send a request to the AWS API to retrieve those credentials just in case there's any kind of update to those credentials the next thing that we have to specify is which Secret store are we using so this is basic referencing The Secret store that we created just a couple seconds ago so we have to tell this external secret object you know what external API are are we actually connecting to with what credentials so you just pass a reference to the name of the Secret store that we just created a few seconds ago and then we have to specify what is the name of the secret within AWS and so as you can see here we're going to match a DB creds with the name of the secret that was created within aw Secrets manager so these two things are going to have to match up you're just basically telling the external secret object what secret should we look for within AWS and then finally we have the target so the way that the external secret works is once it pulls the secret from AWS it's going to send a request it's going to get the DB creds secret it's going to actually go ahead and create a secret a regular kubernetes secret for you and the name of that secret that gets created for you automatically is what you specify here so it's going to be the target so since I called this Cube secret it's going to create a plain old secret called Cube secret and so this is ultimately what we're trying to get to at the end of the day we want a plain old kubernetes secret that we can then provide as either environment variables into our containers or mount it as a volume and so this secret that we have can be used just like any other secret we can pass it in as environment variables or mount it as a volume on any one of our containers but the great part is this Cube secret it's not going to exist in any of our manifests and so we don't need to worry about it checking it into git by accident we don't have to do that this is the only thing that's going to be stored within git which is the external secret configuration and you can see here there's no password or anything like that in the configuration there's no credentials or anything like that we just provide the name and what is the final name of the secret that should get created so that's the main gist behind the external Secrets operator it's not too complicated but let's go ahead and do a demo and I think once we get to the demo you guys will see just how easy it is to work with the external Secrets operator all right so if we go to the external Secrets operator page there's going to be a introduction section where we can select getting started and this is going to walk us through how to actually install the external Secrets operator using Helm and so this is going to be basically two commands we're going to do Helm repo add external secrets and we're going to add that chart repository and then we're going to install the external Secrets chart from that specific repository and so we can just go ahead and just copy these and run these commands directly in our terminal and it's going to set that up for us so I'm going to run that first command and if I do a Helm repo LS we should see the external Secrets repo and then we can then go ahead and copy this command and I'm just going to copy up to here and this is going to install that chart in a nam space called external Secrets all right and now we can see that external Secrets has been deployed successfully if I do a Helm ls- n external Secrets we'll take we'll just make sure that it was successfully deployed and we can see that the external Secrets chart was in fact deployed successfully now what I want to do is and I don't want to focus too much on this but let's just take a look at what that chart actually deployed for us so if I do a cube CTL get I'll just do a get all and I'll just do a dash namespace external d secrets we'll see everything that was deployed in that specific namespace so you could see we've got a deployment for external Secrets we've got one for the cert controller and we've got one for the web hook and that's going to create the respective a sets and pods and we also have a service for communicating with our external Secrets web hook and if I do a cctl get crd you'll see the custom resources that were created by the external Secrets operator so there's a whole bunch of them I don't want to focus too much on that that's a little bit outside of the scope of this video but I definitely recommend you guys dive into it and take a look at all the different custom resources that it creates for you all right so now that everything has been installed let's go ahead and configure a secret within the AWS Secrets manager so I'm going to go to my AWS console and I'm going to search for the secrets manager service and I'm going to store a new secret and here I'm going to say other type of secret they've got some automatic ones for some of their services like RDS but this one's going to be utilized by kubernetes so here we're going to say other type of secret and then here we're going to provide all of the key value pairs that we want within our secret so I'll say dbor password and I'll say this is my password that's going to be the value of our password and then the username the key will be dbor username and we'll say the username is user one and I'll leave everything else as default the secret name within AWS Secrets manager I'll call this DB uncore or I call it db- credentials and I will go ahead and hit next we'll leave off automatic rotation hit next and then we will then go ahead and hit store and that's going to create our secret if I hit refresh we should then see db- credentials and if I click on that we can see our credentials here and if you actually want to see the values you can select a retrieve secret value and we can see the values here and if you'd like to you can also edit it there so now that we've configured the secret within secret manager we have to remember that for our external Secrets operator to actually be able to retrieve these secrets we need to give it access to AWS so we're going to have to give give it some credentials and we need to make sure it has the necessary permissions to actually read those secrets so I'm going to create a user for the external Secrets operator so I'm going to go to IM am here and I'm going to create a user for the external Secrets operator and I'm just going to call this user say external Secrets operator while hit next then we have to give it permissions so here I can say attach policies directly and let's see what the default Secrets manager permissions are so there's this Secrets manager read write and if I take a look at this will basically give it permission to read and write Secrets but keep in mind our our external Secrets operator should never need to create secrets so you don't want to actually give it permissions to both read and write we just want to give it permissions so I've already created a policy for that and here this allows you to list and get secrets in fact you could probably even trim it down a little bit further it probably only needs list secrets and describe secret and probably get secret value you probably don't need the other ones but this was just like a default one that I just quickly created but remember you always want to make sure that you give only the permissions that it absolutely needs so I'm going to give it this this is going to give it read access to technically all of the secrets which once again you wouldn't technically want to do you want to make sure that you drill down as specific as possible and only give it access to the secrets that it should be able to read and not all the secrets like I did here I'll hit next and I I will then create the user so now that the user's created let's select it and then I'm going to go to security credentials and I'm going to select create access key and here I'm just going to say CLI we just need to just get some basic credentials I'll just confirm that and then I'll hit next and we'll create access key so now I've got the access key and the secret access key so if you're following along just make sure to copy those and just note those down someplace for later reference and once you're done with that go ahead and just select done all right so we've now got our credentials and all of the permissions set up and so now we can go ahead and create our secrets store which is going to tell our external Secrets operator how to actually communicate with AWS and AWS Secrets manager all right so now I've got this secret store. file which is going to have a basic configuration for a Secret store and I'm just going to walk you through all the configurations so here under kind this is going to be kind Secret store so this is the custom resource that comes with our external Secrets operator give it whatever name you like I just called it secret store- sample then under the specifications under the provider since we are using AWS you just specify AWS obviously if you're using one of the other providers it would be Azure or gcp or whatever or vault or whatever but I'm using AWS the service is Secrets manager the region that I created that secret in was in Us East one so I specified that here and then here this is going to be all of the authorization configuration and so here we're going to need to provide the access key and the secret access key and there's a couple things that we have to specify so for the access key we're going to tell it where is the access key how does the external Secrets operator get access to it it's going to be stored in some regular kubernetes secret right this is just a plain secret nothing special about it so we're going to have to create a secret we can call it whatever we'd like in this case I'm going to create um call it awss sm- secret and then we have to provide the key in the secret that's going to have the access key and then we're going to do the same exact thing for the secret access key and keep keep in mind they can be stored in two separate Secrets if you'd like but it makes sense to just keep it in one just to simplify it but if you did want to you can put it in a completely different secret so you going to specify what's the name of the secret and what key should it look for to get the secret access key and so those are the two things it needs to be able to connect to the AWS API authenticate and properly retrieve those Secrets within Secrets manager so let's go ahead and create that secret that's going to store all of our access keys and our secret access keys so I'm going to maximize my terminal and to create a secret this is just remember once again a plain old kubernetes secret so I'll do cctl create secret generic and then we're going to give the name of the secret and the name is actually important because remember we specified it within the Secret store it's going to look for a secret called awss sm- secret and it's going to look for both the axis key and the secret access key within that same secret and keep in mind like I mentioned before they could technically be in two different Secrets but to make it simple I just put in one secret and it looks like there's some pasting issues but awss sm- secret and then we can pass in the key value pair so I'll do D- from- literal equals and then we'll say the first one is going to be the access key so I'll say access Das key equals and then I stored the access key here so I'm just going to copy and then we're going to provide do the-- from- literal again and this time this is going to be the secret Das access D key which equals the second value here I'll run that we created our secret if I do cctl get secret we can see our awss sm- secret which is just a plain opaque secret and so now everything should be good to go in our secret store configuration we can actually go ahead and just apply that so I'll do a cube CTL apply dasf and I accidentally hit enter so I'll say secret store. yaml and we can see that our secret store was successfully created then if you wanted to you could do cctl get Secret store and you can also do a describe if you want to see more detailed information but it looks like everything is good we can see status valid ready true so now we got everything set up now it's just a matter of creating our external secrets so anytime we need a password or secret that we want to use within one of our pods or containers that's technically stored within aw Secrets manager or any other external secret vault of any kind we're going to create external Secrets remember we no longer work with regular Secrets regular kubernetes Secrets we're always working with the external Secrets now and so to create an external secret I've got a example config here we've got here under the kind it's going to be set to external secret because that's the custom resource you give it whatever name you want here I just called it example then there's going to be a few configuration so there's the refresh interval so this is going to be set to 10 minutes and you can customize this however you like I just set it pretty low just for demonstration purposes so it's going to send a request to the AWS API to fetch those Secrets or this specific secret every 10 minutes it's going to look for updates every 10 minutes the next thing is this Secret store ref so what Secret store should it use what external Secret store and what credentials should it use to actually connect to that and all of that information is going to be stored in the Secret store that we just created earlier so if I take a look at this Secret store you can see the name of it is secret store- sample and if I go here we can see secret store- sample then what we want to do is we want to get the data from so this is basically telling it what specific secret within AWS Secrets manager should it look for so we're going to extract a secret called db- credentials so what is that from if you guys have forgotten I'm going to open up my AWS and I'm going to go back to my secrets manager My secret is called db- credentials so that's why it's referencing that specific key so this is going to pull in all of the key value pairs from that secret and then finally we have the target so once it pulls in the secrets from here it's going to store it in a regular kubernetes secret called Cube D secret you can call it whatever you want I just called it Cube Das secret so we have all the configurations let's go ahead and do a cube CTL apply DF external secret and it's going to go ahead and create that external secret all right now I'm going to maximize this and so now I'm going to do a cube CTL get and let's take a look at the external secret let's see what we have here we could see that we have an external secret called example we can see which Secret store that it uses we could see the refresh interval and more importantly we can see the status and we can see that it's secret sync that means it was able to successfully pull the secret from AWS Secrets manager and we can see that it's ready set to true so it looks like it did everything which means that it should have created a plain old kubernetes secret for us to actually use and mount within one of our specific containers so if I do Cube CTL get secret you could see that it actually went ahead and created that Cube secret secret and that name once again comes from the configuration that we specified under Target so we called it Cube secret here then that means that it created this secret here and so if I do a cube CTL get Cube d sec- o yaml and this should be get secret Cube secret we can see that there's two values the DB password and the DB username and so if I I just take the DP password and I just do an echo and I pipe that into base 64- D this is going to decode it and we can see we got my password so I was able to successfully retrieve that password from the AWS Secrets manager and so at this point this secret right here Cube secret it operates just like any other secret that you would have so you can pass it in as um an environment variable into a container you can also mount it as a volume it operates no differently than any other secret now before we wrap up this demo there's one last thing I want to show you guys and that is I'm going to go back to the AWS console and what I'm going to do is I'm going to go ahead and change the value of one of those key value pairs I'll select retrieve secret value and for the password I'm going to edit that and it's going to be my password two version two of the password so I made the changes now what do you think is going to happen within the secret here so if I do a describe again or do it get secret we can see this is the value of the DB password let's take a look at this and I'm going to decode this once again and when we decode it we can see oh wait a minute what's going on here it just shows my password but we updated it it should show my password to not just my password this is the old password so why didn't it get updated it really just comes down to the configuration of our external secret so we specified here the refresh interval is going to be 10 minutes so it's going to pull 8 secret measure every 10 minutes in this case so we're going to have to wait for that 10-minute timer to expire where I can then you know pull the API and get the secret and get the updated database password so if we wait 10 minutes and run this command again we should see that the value changed I'm going to pause this video we're going to wait the 10 minutes and then we're going to double check and just make sure that after the 10 minutes pass that the password got updated to my password too so it's now been 10 minutes and if I go ahead and run the same command to get the secret and if I take a look at the secret now for the password I'm going to once again decode this we can see that after the 10 minutes it's now been updated to the new password which is my password too so that just confirms that once the refresh interval expires after the 10 minutes or whatever value you set it's going to go and retrieve those Secrets From The Secret store and then update them accordingly all right guys so that's going to wrap up the video for today we got a chance to dive into the external Secrets operator we got a chance to see how we can utilize a external Secrets store like either vault or AWS Secrets manager to manage our secrets and we configured kubernetes to automatically pull in those Secrets when we need to and so that way we don't need to worry about checking in our secrets into our G repository accidentally and so hopefully you guys enjoyed this video definitely stay tuned for the next video where we are going to take a look at another tool that we can use to manage Secrets within our kubernetes environment we're going to take a look at a tool called the CSI Secrets store and so that's just another tool that we can use to manage Secrets within kubernetes anyways I'll see you guys in the next video yeah

Channel: KodeKloud

Views: 6,098

Keywords: Kubernetes, Secrets Management, External Secrets Operator, Kubernetes Security, DevOps, Cloud Computing, AWS Secrets Manager, Containerization, Kubernetes Clusters, Configuration Management, Tutorial, KodeKloud, ESO, kubernetes tutorial

Id: EonWeoFPpvM

Length: 25min 13sec (1513 seconds)

Published: Thu Feb 22 2024