Getting to Know AWS Secrets Manager - AWS Online Tech Talks

hi my name is Jeff Levine I'd like to welcome you to this webinar entitled getting to know AWS secrets manager I am a senior Solutions Architect in the security and compliance team at Amazon Web Services first just a little bit more about me for history sake the first computer that I used was an HP 3000 series 2 I have used key punchers and cards sorters you might find those in museums today my first programming language was Fortran I came to Amazon Web Services in 2016 and I help customers with real-world security issues I write blog posts and white papers and I love to scuba dive so now let's move into our presentation today will talk about foundations of secrets manager the lifecycle of Secrets how they're stored rotated audited and deleted challenges with secrets manager and how to address them and then we'll talk about some demo code and a presentation and there will be time for Q&A later let's talk about foundations first first a secret is a piece of information that's meant to be hidden because knowledge of that secret can have undesired consequences so in the world of info SEC a secret is used to gain access to additional information consequently we need to be able to control the access to our secrets and we need to be able to audit the access to our secrets examples of secrets in AWS include things like database usernames and passwords that's specifically what we'll be addressing in the demo that we do later today but they could also be application usernames and passwords as well so what are the main benefits of secrets manager first it supports the complete lifecycle of secrets management that includes storage rotation and auditing and it includes native support that's out of the box so to speak for Amazon RDS and Aurora but you can also extend the secrets managers functionality to include other credential sources such as application credentials with rotation functions built on AWS lambda and for those of you who aren't aware of that AWS lambda is the serverless computing platform for Amazon Web Services and we develops the secrets manager based on the experiences that we at Amazon have in managing millions of secrets globally when you have to keep track of them and rotate them we've learned over the years what capabilities are needed and we incorporated those into secrets manager so now let's talk a little bit about the lifecycle of Secrets first in terms of storage all secrets are encrypted by kms using either the default service key or a customer managed kms key you can attach identity policies to users groups and roles to control access to a secret but furthermore you can also attach a resource policy to a secret to determine what principals can access the secret so think about it as putting policies at both ends on the principal's who want to access the secret but also on the secrets themselves and the identity policies and the resource policies have to be satisfied in order to grant access so the principal needs to be able to access the secret and the secret needs is to allow itself to be accessed as well the two of those policies have to meet in order to grant access a little bit more about storage secrets are versioned so this allows you to go back in time and fetch previous versions of the secret if needed so for example if you have to retrieve an old version of a resource like a database or an application table of some sort and you need to get a previous version you can do that with secrets manager and secrets also have what we call staging labels you can think of these as attributes that can be used to identify versions of the secret and AWS uses special staging labels during the rotation process and when we go through our demo today we'll talk this now let's talk about rotation and that this is one of the main benefits that secrets manager brings secrets manager rotation is handled by lambda functions again lambda is the serverless compute platform on AWS and AWS provides functions for native RDS and Ellora rotation so my sequel press sequel and so forth all of those are provided out of the box you don't need to do any coding whatsoever the native rotation functions are derived from templates in the serverless application repository and our instantiated via cloud formation I'm only mentioning this because when you enable secrets manager rotation and you go to your cloud formation console you'll see an extra cloud formation stack that appears that represents that sir from the storage application repository that has been accessed to build a secrets manager rotation stack you can also build your own lambda functions to meet your unique rotation needs a little bit more about rotation when a rotation is initiated the secrets manager generates a new credential with identical privileges and attaches the AWS pending staging label to it so that's a that's a reference to that staging label concept that I mentioned before and then sequence manager tests that credential if that credential is successful if it can login the way it's supposed to then sequence manager also attaches the AWS current staging label to it and then after that the AWS previous stage and label is then attached to the previous version so this gives us a way to identify the current pending and previous versions of a secret and if a problem arises you will be able to use the stage and labels to help you recover so again this is part of what we've learned at AWS in terms of managing secrets on a large-scale basis these staging labels will help you not only with the actual rotation but then the event a recovery is needed let's talk about rotation functions and how networking fits in so in order for rotation to happen the rotation function has to be able to communicate with the protected resource so an RDS database or Melora database and so forth but it also has to be able to communicate with the secrets manager service itself so again you see them this diagram this two-prong communication that's needed by the lambda function - secrets manager and - in this case Amazon RDS if the protected resource is not publicly accessible what secrets manager will do is bind the rotation function to the B PC and then all communication will happen within the V PC this is the preferred way to do it because this way all of the communication stays private and does not have to traverse the internet if the protected resource is publicly accessible by default a secrets manager does not bind the rotation function to the B PC and communication does not reside solely within the V PC you can customize this behavior a little bit by making modifications to the network parameters of the B PC itself and of AWS lambda so the preferred way of accessing secrets manager is through V PC interface endpoints but there are some caveats that you have to be aware of when you begin to use them so when you use a service like secrets management offers V PC interface endpoints you have to look up the ACS that offer the endpoints and creates subnets in those AZ's and remember that not all azs within the region offer in interface endpoints to a service so again be aware of the fact that interface endpoints are not available available in all AZ's so you do need to look them up and design your architecture around this also remember that AZ is are mapped differently per account so if you're an organization and you have multiple accounts for example us east 1a is not necessarily the same physical data center for each account so you need to look up the AZ's for your accounts to see which a zs offer the appropriate interface endpoints also I suggest you enable a private DNS to make it possible to access the service via its default name and to do this there are some attributes in the V PC you need to set enable DNS host names and enable DNS support and set those to be true alternatively you can access the secrets manager and pull way through the public internet if you want to do this you would need a NAT gateway and the traffic would exit the V PC encrypted but it would exit the V PC to access secrets manager we do not recommend this approach again the preferred way of accessing secrets manager is through V PC interface endpoints in terms of auditing secrets manager is integrated with AWS cloud trail it will capture all the api's for secrets manager events including both from the console and from code and it can work with cloud watch events to trigger alerts when administrators specified operations occur for example if someone goes and deletes a secret you may want to trap on that and alert or also if someone is accessing a secret that is pending deletion you may want to trap on that as well let's just say a few more words about deletion AWS secrets manager does not allow you to immediately delete a secret you schedule the deletion for a minimum of seven days in the future and you can cancel the deletion of the secret at any time prior to its actual deletion now let's talk about challenges that arise and how to address them the most common challenges are related to the topology so again make sure that your rotation function has access to the secrets manager endpoint as well as the resource that is being rotated also the preferred way as we said before to establish connectivity to the secrets manager is through the VPC endpoints and establish subnets in the AZ's that offer the endpoints if you don't have a subnet in those AZ's you won't be able to communicate through the VPC / - secrets manager and remember again to turn on private DNS in the V PC to simplify the resolution of the service point name and then make sure your rotation function can access the private resource itself also check the rotations function access to the V PC - another challenge that arises is the matter of conflicting iam policies you want to examine the identity policies that are applied to the IAM principles the users groups and roles that come into play and examine the resource policies that are applied to the secrets again remember the identity policies and the resource policies need to match they need to give X access both to the principal and to the secret itself and again as I said make sure the intersection of the two allows the access that you need so now we're going to walk through some demo code and then actually do a presentation of the demo this demo is located in github in the AWS samples repository under AWS secrets manager workshop and we're going to do module one of the AWS secrets manager workshop and it's entitled rotating the secret for a private Amazon RDS database so let's talk a little bit about what we're going to build here today we have a B PC that's in US East one it has two subnets and it actually has one son that that's visible here and one that's not and I'll talk a little bit about why we have a bastion host that is running on Amazon Linux - and you see - and we also have a database an Amazon RDS my sequel database that's called book info and we also have a secrets manager endpoint here as well there's actually another seven that I'm not showing here because when you set up an RDS database you do you need a second subnet for the subnet group but we're not actually using it which is why I don't have it depicted here on this diagram so let's talk about some of the considerations that are within the code we need to provision subnets and availability zones that offer AWS secrets manager endpoints I've made this point a couple times I want to emphasize it because it's very important again not always either the endpoints so we need to look them up also I am creating random usernames and passwords for demonstrations like this one because I don't want to have a situation where we have where we have a default username and password in case for any reason the VPC in order to become unprotected perhaps by changing a security group I just want to make sure that we don't have default passwords out there I here's an excerpt of the code that what that is in the cloud formation template and you can see here that there is a resource type here called custom so what we're defining here is a custom resource that is going to look up the endpoints that are available for a given service and the custom resource capability of CloudFormation allows you to extend cloud formation by adding additional functions to it which are powered on the backend by lambda and so while this this webinar is not meant to be a comprehensive course on cloud formation I just want to say here that were passing two parameters number one the name of the the endpoint that we're looking for which in this case is calm Amazon AWS US East one secrets manager and the number of azs that I want to make sure exist that offer the endpoint and I'm saying to here because as I mentioned before a subnet group for RDS needs to contain to a ZZZ and therefore to you subnets and so what I do in my cloud formation Python code that's making up my lambda function is I use the described V PCM and point services API to look up the service that I just mentioned namely secrets manager and you can see right below there that I pick off the returns availability zones from the response array as well as the number of AZ's and I use the length function to do that and then I return the AZ's using the join method to separate them by commas and I also return the number of a ziza as well and then I use that information in the main part of my code by calling the get ATT intrinsic against the return value for my V PC endpoint services function using the AZ's element that is returns so essentially here I am picking off the first and second AZ's to create my subnets and subnet group and then at the bottom when I create my DB instance I'm also pulling off the first subnet as well the first AZ as well that's returned from the function now I also have another function in my called called random string function which as its name implies generates a random string value you simply pass it the length of the string that you want to generate and I use this for creating usernames and passwords I'm not going to go into the details of how that function does what it does and but you can take a look at it when you look at the Python code that's contained inside the cloud formation template again everything is all within the cloud formation template the the main cloud formation code as well as the lambda functions so now what we're going to do is to move into the demo so I'm going to exit my PowerPoint presentation here and switch over to a browser set of browser windows that I have that will that we will be using here today so if I go ahead and create this if I want to go ahead and run the demo I go into CloudFormation and I select create stack now you can see here that I have already selected the file one underscore rotate private are your dsdm oh and if I click Next CloudFormation parses that template and then I can say something like secrets manager demo test I have to have a key pair ready and then I click Next cloud formation prompts me for a few more things I don't need to change anything so I click Next again and if I click I to acknowledge that I am going to be creating I am resources and then if I were to click create cloud formation would then go ahead and build this whole stack for me now because we're having a webinar rather than waiting 10 to 15 minutes for this stack to build we're going to pretend like this is one of those TV cooking shows and we're going to switch over to a stack that I have already built that has this information already populated in cloud formation so I have my SM demo webinar stack here and the first thing that I'm going to do is look at the outputs of that stack and when we look at the outputs of the stack we see a couple of things here we see a DB password and a DB user these are going to make up the values for the credential that I'm going to be storing in secrets manager and then we have a database endpoint which I won't actually use in the demo and then we also have a bastion IP this is the IP address of that host that I had created as part of the cloud formation stack now you'll need a program on your system in this case I'm using Windows so I have a program called puddi or putty depending on how you pronounce it which allows you to open up an SSH session to a host using using a private key so I'm going to bring that up I already have that here so let's see that's all running fine so now what I'm going to do is go to our secrets manager console and I am going to store a secret now it's going to want the username and password that I'm going to associate with the secret so again I have to get that from my cloud formation outputs so I'm going to copy the value that I see here for DB user I'm going to paste it in to my username and what I suggest you do when you're doing this yourself is just make a note of what the first and last characters are so you can make sure that you copied and pasted it correctly now we also have to do the same thing for the password so I'm going to copy over now the database password I'm noticing that it begins with an A and ends with a G so I'll copy that good for the sake of making sure I did this right I'm going to turn on show password so I can visually see what it is and I can make sure I've copied it correctly I'm going to use the default kms encryption key for the sake of this demo and then I have to pick the database that I want to apply this secret to so this is the database that was created it's a my sequel database that the cloud formation template created and I'm going to click Next I need to pick a name for the secret so I am just going to call it SM demo and for my description I'll say s them demo webinar you don't even need to put a description let's do it anyway I'm going to click Next now I am going to leave rotation disabled for now I just want to store the secret will enable rotation later I'll click Next you can see it shows me what parameters I've already entered and then if you want some example code fragments you can see that we provide you with sample code fragments for Java and other languages down here but I'm just going to say store and now you can see that I have stored a password or a secret called SM demo that will contain the user name and password that I had entered now I'm going to flip over to my console session over here and one of the things you'll notice in in this console window is that when I build the instance I decide to create a couple of scripts that can help you to see what secrets manager is actually doing now we'll talk about the to my sequel scripts here and the first one we're going to talk about is my sequel old way dot SH so let me just cap that out so you can see what it's doing and what it is doing notice here that I have hard-coded the username and password in for the database as well as the endpoint one thing to remember with the password there is no space after the P I made that mistake a couple times so I thought I would point it out to you as well and so this is just going to go into my sequel the way would we would typically do it with a hard-coded password and so if I say my sequel the old way dot SH and enter you can see that I've gotten the prompt I'm going to say Jo databases and I have a database here called SM demo I'm going to say use SM demo and then I'll do a show table so you can see what's in here I have a table called book info and I am going to select star from book info and what comes back a tale of two cities from Charles Dickens okay so I've been able to go in with a hard-coded password now I'm going to do the same thing with the my sequel dot new way des age but actually let me go ahead and cut that out for you as well so my sequel got new way dot Sh and here we're doing things a little bit differently what we're doing is we are using aw with secrets manager to fetch the secret associated with this with a secret value that is supplied as a command line parameter and then we're using JQ to pick off various parts of this information to extract the user name and password now typically you would not use environment variables to hold usernames and passwords that's not just a it's not a generally good security practice to do so what we're doing is we're doing it this way just for the purpose of providing a demo so now I'm going to do my sequel got new way dot SH and I'm going to mention the name of the secret that I provided so again I am now in the database or in the database instance I'm going to say use SM demo and since I know what the table name is I'll just say select star from book info and again you can see the contents now in this case I didn't use a hard-coded password I use secrets manager to provide me with the credentials and so you can see that we were in fact able to retrieve them and gain access to our database now it gave you one more script here and I just want to show what this is display it secret for SH and I'm going to type the name of the secret and you can see here that I have one version of the secret with this long version string and here is the username password and that's the version stage here which is AWS current that's what I was talking about before with those with those labels those stage and labels to help you determine what the current version is now what I'm going to do is I'm going to go into my secret and I'm going to edit the rotation I'm going to enable the rotation and I'm going to set it to 30 days I'm going to say use this secret which means I'm going to use the credential in the secret to gain access to the database and then change that same credential so I'm going to click Save here now and notice that there is a message up here saying rotation is being configured stay on this page this can take up to two minutes to complete well it's happening here is the Consul is initiating a connection to the secrets manager API to perform this rotation it's doing this on your behalf behind the scenes but it's important that you not leave this page while this rotation is happening it says that it can take up to two minutes to complete but in practice what I've noticed is that it actually tends to take under one minute to complete for a database like this one so we should be able to see this happen very soon and yes the the secret has been stored and rotated is generated a new value now with some browsers there are things that are cached so I'm going to just refresh my screen to make sure that we have the most current credentials and if I retrieve my secret I can actually see that my password has in fact changed in now starts the B and ends with an M now let's go back to my console window now if I run the old ways with the hard-coded password this should fail and it does it says access denied because my heart kind of credential is no longer working so now if I say my sequel got new way and I specify the parameter or the secret parameter you can see I am back in my database because what secrets manager has done is it has fetched the new value of the credential and I can say again use SM demo select star from book info and again I see that I have access to the table just like I did before let's quit out of this and let's do one last thing run my display the secret versions dot s age file whoops and I have to specify a demo or a secret name notice now that I have two versions my new credential I have the username my new password that begins with a B and ends with an M and this now has the AWS current label as well as the AWS pending label remember what I said about rotations first secrets manager attaches the pending label it tests the new credential and then it moves AWS current to it as well and my previous credential is now stored on with the AWS previous stage and label so now you see how secrets manager works to enable you to fetch the value of a secret we've shown how you can use secrets manager how you can retrieve values from it to access a database and we've also shown how you can look at the different versions of a secret and see the different staging labels such as AWS current AWS pending an AWS previous so you can see both the current the previous versions of a secret I'd like to thank you for watching our webinar today that concludes our presentation of secrets manager and we're now going to close out our demo and we're going to move to a period of questions and answers

Channel: AWS Online Tech Talks

Views: 6,592

Keywords: secrets, credentials, SSH, keys, rotate, rotation, AWS Secrets Manager, AWS, Webinar, Cloud Computing, Amazon Web Services

Id: ydrcyy1nrQM

Length: 31min 8sec (1868 seconds)

Published: Mon Oct 22 2018