Storing secrets CORRECTLY in .NET using AWS Secrets Manager

hello everybody i'm nikhil in this video i'm going to show you how you can properly manage and load your secrets in your dot net applications using aws secrets manager now you can do the same thing with azure key vault or hashicorp's vault feature however in here we're gonna see how we can do it with aws because that's the platform i've been using for the past three years professionally and i've seen this thing scale so i can talk about best practices this video is also sponsored by aws so the code can be found down below for this demo for free if you like a lot of content and you want to see more make sure you subscribe and ring the notification bell and for more training check out nickchopsters.com so i'm gonna start this video with explaining some configuration basics because it's going to be very important for you to understand how things are loaded and the first demo will be all about how you can use this and correctly load secrets in your asp.net core applications because i think that's most likely where you're going to use this later in the video i'm going to touch on some more advanced stuff but that's later i'm going to start with things you can actually just use straight away so let me show you what i have here i have an api here that has a single controller which all it really does is we're injecting an eye options object of the database settings object that i have made over here and this is very standard you have a section name for your settings and then you have the settings themselves so in this case i only have connection string and you can find that here in this app settings.development.json and then you have the default one which in our case is the prod one and if i was to run this this is registered in the program.cs over here this configured by section and if i go ahead and i call that endpoint as you see i'm getting connection string is dev here so this is all this is doing and the reason why i'm doing that is to demonstrate how secrets will be loaded in our application now let's talk about some configuration basics you will see that i have two app settings files here in your applications you might have more you might have created the qa one the sandbox or the pre-prod one you might even have a named production one the way this works is that depending on the asp.net core environment variable that is loaded when the application runs and in our case this is development here it is coming from the launch profile and the environment of the application will be determined and then will be used to load first the generic app settings.json and then the environment specific upsettings.json and the environment specific one will override what's in the default one so for example if i say log level warning in the app settings.json but i load this application as development environment then that value will be overridden by information which is in the development same goes with a connection string now if a setting does not exist at all so if i went ahead and i deleted that and i run it again then obviously there is nothing to override the value that is in the profile so it's gonna just default to prod that's it now it goes without saying that you should not have secrets like api keys connection strings anything sensitive in your app settings.json just don't you should never have them there and you should never check them in it's a bad practice it's a security vulnerability you should not do that if you want to manage secrets locally dotnet has actually a user secrets api which you can use so you can go to the top level of your project and say in the terminal dotnet user secrets init and this will just initialize the secrets for that project and those are stored in your pc or mac or whatever you're using and you will see now that if i go to the cs proj this is creating this user secrets id which is effectively just linking the gui generated here to something stored in my pc to be able to match those values so what i'm going to do now when i just made this bigger so you can see it without my face covering it is i can say dotnet user secrets and i can set by using the set command the value i want to set so for example here i want to set a secret connection string for my database so all i need to say is first the key name and the key name is the top level for my section so in this case it is database and then you can go deeper into the nesting levels by using a column so i can say a connection string and then the next value is the connection string so i'm just gonna grab the localhost connection string and this should look something like this so this is just the local connection string for ms sql so i'm just going to set that value and the moment i do that if i go ahead and i debug this application now irrelevant of the environment loaded if i call that endpoint i'm going to get the value based on the secret so this is very important for you to understand on how the layering of the different secrets work now what i want to do is i want to securely load secrets from a location that is specifically made to store securely those secrets and allow me to load them in my application securely clearly this is not the app settings files and the only reason why i have these values over here is so you can distinguish which value is loaded from dev and which value is loaded from prod in the actual secret we're going to store the full value of well the secret so keep that in mind and i'm going to delete that value that i just said i can just do a remove over here and that's no longer there so if i was to run this again and i call the endpoint as you can see this is all dev so we have that in mind and what i'm going to do is i'm going to go to the aws console now i will not be authenticating against aws using basic authentication instead i'm authenticated by authenticating my whole machine against aws and this is really how you should do this because keep in mind that when you want to use secret manager but you need a secret to load the secrets from the secret manager you have a bit of a chicken and egg problem where your secret will be in secret manager and you cannot be properly authenticated against aws with basic auth if you want to get the secret to use the secret to load the secret i know it's crazy so how we do this is with permissions and i am roles and we authorize the system itself or the pod itself in kubernetes or the ecs task itself to properly do this so keep that in mind this is important now i'm going to go to the aws console so this is what it looks like and all i want to do is go up here and say secrets manager that's the service we need it's specifically made to store secrets and load them just very very securely and will also allow us to do things like secret rotation now i'm not going to show you how you can do secrets rotation here but the idea is that when i actually update a secret usually based on an interval let's say every month or so or every time for example it leaked for some reason and automatically have your services using that update and the database update and secrets manager has support for some databases out of the box for example aurora rds postgres mysql those things can do that automatically for you we're going to be doing the process manually assuming that this is not a specific database key but i will be using a database connection string just to keep it relevant to what you might be doing so you can see the pricing right away per secret per month you're going to pay p and then per 10 000 api calls which an api call can be rotate the key or retrieve the key or list the keys or secrets which is really the terminology i should be using um then you're gonna pay 5p for that so 40p per secret but each secret can have a key value pair as a json object so how you're going to load that is really up to you i'm just going to show you in this video a very nice and easy way to do this so first we're going to go with storing new secret and i'm going to say other type of secret here and i'm just going to do plain text so i'm going to delete that if you want to have a key value pair in the secret you can uh but i don't want to i just want to have a connection string and the connection stream will look something like this it is just a database connection string which chooses the server the database the user id and the password they should be secure they should not be in my app settings file it should be here and then i have to choose the encryption key now i'm using the default secrets manager key i won't go too deep into kms but the idea is that you can actually specify your own key as well if you want which is used to encrypt the secrets behind the scenes we don't need to do that there are cases where you might want to do that but maybe something for a different video so we're going to go with next now and i'm going to go to the top and i'm going to give my secret a name so the way you might want to do this and this heavily depends on your aws account structure but i'm going to show you something assuming you're using the same account for a development prod soundbox whatever which isn't really recommended but just to show you how flexible this thing can be i'm going to say production here we're going to distinguish that this is a production secret and then i'm going to use the application name here so in my case the application name is going to be secretmanagement.api so i'm going to paste that here and then i'm going to say underscore and i'm going to have the same structure in here as i have for my settings so database is my section so this will go here so paste database and then double underscore and this is important you will understand in a bit why it is important and then copy connection string here and paste it here and that's it you can use tags if you want to you can choose permissions and you can also replicate the secret in multiple regions again we don't need to do that this is a bit too advanced maybe something for another video or a course and once we have that you can enable automatic rotation where in an interval let's say every 40 days you are automatically rotating the key changing the key and every service using it is also automatically updated but we don't need that this is a bit too advanced too and once we have that we get the chance to see what's going on it also gives us a bit of a code snippet if we want to just grab that and retrieve that key in our code we will not be using that we'll be using a way better approach so i'm gonna say store here and i'm gonna do the same thing with a development key so i'm just gonna copy that and say store secret again and i'm gonna store a second value but instead of using the production one i'm gonna say just dev so i'm gonna use dev just to differentiate between the development environment and the production environment so my dev user name and then my dev password and that's it i'm gonna just save the key same settings as everything else um the name will be of course development and then the name of the thing and then same thing underscore database double underscore connection string paste that here we go and that's it next next store so we have our two keys now the production one and the development one let me just open them in new tabs because we will be playing around with them so once we've done that what i can go is i can go back to the code and actually implement the code that retrieves the secrets and adds them into my configuration natively so i'm gonna do that by going to nuget and searching for aws secrets manager and i'm gonna use this package called cr extensions.configuration.aws secretmanager and this is a package created by a third party and you can find the code on github it's a very small code footprint but it has some really really nice features that just makes this a breeze to work with so i'm going to add that and we have it now in and all i need to do to integrate my app with that is the following so all i need to do is say builder.configuration.add secrets manager now i have to specify if i want the credentials but i don't need to and the region will be inferred automatically but i will say um west to eu west which is london and i'm also going to say configurator and i'm going to pass the options down so we're going to configure a few things now there's a few problems we have first i do not want to load production secrets if i'm just running this into development and i don't want to load development secrets if i'm just running this in production i say just in production in production so what i can do here is i can say options dot secrets filter and i can specify a delegate where a filter is applied when the secrets are loaded and if they're not relevant to what i want them to be then i can just ditch them so for me what i want is the entry dot name of the secret you actually have a bunch of other things you can actually base this on from the kms key id to stages to parts of different things but all i need to say is that entry dot name dot starts with because i care for my secret to start with development or production so my environment plus the application name so i need two values first i need the environment and i can get that by saying builder dot environment dot environment name and i can also say um app name equals builder dot environment dot and application name now something i should point out it is very common for people to not want to use secret manager when they're locally working with the project so development here is really referring to the development environment not so much with the local host running some people use a different name some people use the local name to differentiate between the two but if you wanna not load those things when you're working locally and just use your user secrets instead you can do builder dot environment dot is whatever is development or you can grab the environment name and match it to local or whatever and say that if it's not local for example then don't load the add secrets manager call so keep that in mind not everyone wants to use secrets manager when running locally and all i need to do here about my start with is say if this starts with end underscore and then app name underscore then load them that's it however i want my secrets to automatically be mapped into my configuration and for that to happen there's actually a configuration provider behind the scenes but i need to adapt the key look and feel to what asp.net core accepts you know the colon approach to do that all i need to do and this is very simple is say options dot key generator and say s dot replace and i need to replace the environment and the app name with nothing so remove the beginning things so i can just copy that and say replace this with string dot empty and let me just newline this so you can see it and then i also want to replace all the double underscores with the colon that's it now once i do that then i can actually just discard the entry because i don't use it once i do that keep in mind my system is authenticated against my aws account if i now run this and keep in mind development has dev here and production has prod here so i'm going to debug this and go back to postman and this is running so i'm going to say run and this is now loaded automatically from aws secrets manager just like that i didn't have to do anything it just automatically maps it and this is awesome because now if let's say this was launched into production which is what your system would do if you were running this properly is just debug this go back remember dev server address this is all dev if i run again production production production this is loading the right secret the appropriate secret for this environment without over fetching four secrets that it should not know about now this is fine however here's the thing secrets can change and they might be rotated so what i might want to do is i might want to have an eye options monitor instead which the i options monitor in case you don't know will actually get you the current value and it will account for edits of the value not magically we'll see that but if i change that now and i go back to post when i call this then as you can see i'm getting this absolutely fine now if i go to secret manager and i select retrieve secret and edit and i say my production server address instead and say save now if i go back to postman here and i click that this isn't updated it just says production server address even though i'm using an i options monitor it doesn't actually update so this is the problem how we can deal with this it's very very simple all you need to do is go to program.cs and add an extra value say options dot polling interval and this will pull for changes on the secrets we care about so i can specify that as anything i want we're using a time span for demonstration purposes i'm going to use 10 seconds keep in mind if you're using this you're making one api call per secret so your allowance might be affected by that and the rate with which you're being charged will also be affected by this choice so for me i don't really care because this is just for a video but you might want to have it in a different polling interval so i'm just going to do 10 seconds so i'm gonna say debug now and run this and with this running i'm going to go back here and you can see now my production server address i'll go back and edit it again and say my second production server address i say save go back here click send and then on a 10 second interval if i keep clicking my second production server address so keys can change as they're being rotated and my application can be updated too without having to store them in an inappropriate place so this is absolutely solving the problem with secrets management and loading securely so if you're already in nws you should be using this if you're not in aws you can still just use this feature from aws and this is not just the only thing this tool can do we're gonna go a bit deeper now in a separate project we're gonna go into the console application project and we're gonna take a more of a deep dive now this console application has nothing but what i'm going to do is i'm going to add the aws sdk dot secrets manager project and i'm going to install that and this gives me access to the i amazon secrets manager so i can say secret manager equals new amazon secrets manager i'm gonna specify the region to be west eusto here we go and once i do that what i can do is say var result equals weight secrets manager dot get secret value async and i need a request for that so i'm going to say var request equals new get secret value request and all i need by default is the secret id now the secret id is this the secret name so i'm going to load that and once i grab that i can just paste it here and i'm going to use this request to get the id so i'm going to say console.writeline and i'm going to write result dot secret string you can get other values too but just for now i'm going to print that value so i'm going to run this project and as you can see we're getting the production value however there's other things i can request for that secret now here's a hypothetical you have the secret you want to roll back what do you do well you can actually specify two things the version id if you know it but also a version of stage so if i use nothing by default it will use the aws current version stage so you don't need to know the version id itself which is just good but if you do aws current it will give you the current version of the secret but you can also do aws previous and if you do that we are going to get the previous value my production server address of the secret but you don't have to do that what you can do is say list version ids request equals new at least version ids requests and i can say secret id and pass down the secret id uh here so i'm just gonna grab the whole thing and once i do that let me just move this request below the secret retrieval and i'm going to not use the version stage instead i'm going to use the version id object and say versions equals a weight secrets manager dot list secret version ids async and pass down the request object and i'm gonna say version dot versions dot first and i'm just going to grab the first thing in that array and i'm gonna use the version id for that and that's it and of course you can customize it in any way you want and once i do that i'm going to run this application and as you can see i'm getting the first thing again let's debug it too so we can see exactly what's going on behind the scenes so i have my versions i'm gonna grab the vision response and as you can see if i list this where is it versions you can see i have two different versions of that secret with the version id and the stage so the current and the previous and you can roll back if you want to very very easily and you can grab the version id and get other metadata about that thing too very very nice the sdk is very flexible so if you want you can do other things like series manager dot create secret delete secret describe the secret list it restore it rotate it you can do so many things now personally in my applications i almost always just use that package that i showed you because it makes the 99.9 of my use cases so so easy if i need to write something very custom i might use this approach but probably you don't have to so keep that in mind you can grab the sdk straight away and do anything flexible you want but that other package is very very nice i'm going to leave a link to it down below so if you want to give it a start please go ahead it's a very very nice package well that's all i had for you for this video thank you very much for watching special thanks to my patreons for making videos possible if you want to support news are you going to find it in the description down below leave a like if you like this video subscribe smoke on the light is hearing the bell as well and i'll see you in the next video keep coding

Channel: Nick Chapsas

Views: 35,917

Keywords: Elfocrash, elfo, coding, .netcore, dot net, core, C#, how to code, tutorial, development, software engineering, microsoft, microsoft mvp, .net core, nick chapsas, chapsas, clean code, dotnet, .net 6, aws, aws lambda, azure, aws secrets manager, secrets manager, aws secrets manager .net, secrets manager .net, user secrets .net, Storing secrets CORRECTLY in .NET using AWS Secrets Manager

Id: BGW4FnEB-CM

Length: 23min 28sec (1408 seconds)

Published: Mon Apr 18 2022