Using Secret Manager to Manage Tokens - Google Cloud Python Tutorial

hi welcome to playground in the clouds i'm you in our last video we talked about how to build a discord board on app engine at some point we mentioned that it's not the best practice to store the token in the code directly so today we are going to talk about how to secure your discord bot token using a service called secret manager so let's get started so as you can see here this is the token i'm talking about so this is effectively the password of your discord board and imagine one day you want to publish your code on github then anyway anyone who have access to your code will be able to access this token obviously we don't want this so we want to store this token somewhere else but our python program can access um some of you may say can we store this token in an as an environment variable yet this is a quite common practice in personal projects to store your secrets as environment variables but in app engine it's a little bit tricky because the way we can set environment variable is through the app.yaml so it is the same app.yaml file we used last week to deploy the app engine service so as you can see here if we put the token as an environment variable effectively we will have to put the token as plain text in the after yamu effective we are putting in our code again so this is probably not the best way to do it either and today i'm going to introduce a service uh also from google cloud called secret manager and hopefully you'll see it is actually very easy to use it and it's also worth mentioning that this method can also be applied to storing other kinds of secrets so not just the discord board token you can use it for your database password you can use it to to store secrets to add other services as well um so it's uh it's very useful and it's a very useful service how do we get started i think the best way is probably go to the secret manager service to take a look the best way to get secret manager is actually um just type in secret secret manager in the search bar at the top of your google cloud platform console page so and then we choose a service called the secret manager another way to find this is to use the left hand side navigation menu and it's in the security tab so we can actually pin the security security category and the secret manager is one of the security services and secret manager is very easy to use as you see here we just have one option which is create a secret so we click this button and create a secret and we can name our circuit as anything you want here we call it this code bought token so the value of secret not surprisingly it's the it's our token so because um since last project i already changed my discord bot token so i will have to go to the discord developer portal so and find the bot and here we can copy this token again so i'm just going to click copy and where when i come back to the secret manager page i just copy i i just paste this value in this box and i can ignore everything else on this page for now and just click create secret so you can see it's a very straightforward process to store your token in secret manager once we store this token and we can notice there that there is a path and this is actually like the id of this token sorry the id of this secret so the beginning of this path is projects and then my project number and secrets and the name of the secret we just created we will need to use this id a little bit later so now we already have the our discord bot token in secret manager now the next problem is that how can i how can my python discord board actually access this token so i will do a little bit whiteboarding here so last week we built the discord board in python and is running as a service in app engine right now and now we put a token in the in secret manager but how does secret manager know who i am and whether i can access the token or not and so here we need to introduce a concept that is called service account so what is service account service account is effectively a non-human user it's like um when you use google cloud you will have to create account for yourself and then when the servers in your app engine want to use google cloud therefore access some of the other services in google cloud we associate a service account to the app engine service basically your person code will be running as if they they are using this service account so with this service account we can give the service account permissions so it will have permissions to do different things and in our case it will have permission to access the token in secret manager just make the box a little bit bigger so it looks better so with the the permissions and the service account our discord python board should be able to actually access the token and one missing piece of the puzzle here is how do we actually program programmatically access the token using python so as you probably have guessed we need to use a sdk and the sdk will be the secret manager python sdk so for those not familiar with the concept of sdk sdk stands for software software development kit so effective it's kind of it's a collection of library um usually for a certain language and to enable you to use certain services and sdks are normally provided by the service provider in this case google cloud but there are exceptions as well actually we already come across the idea of sdk in our first video so remember we when we build this codeboard we use a library called discord.py so this discord.pui library is effectively a sdk um for discord api so we already come of course this concept and it's really just a library so now we have two things to do one is to give the service account attach a service account to choose our app engine and then give permissions to our service account and the second thing is we need to incorporate this secret manager python sdk in our python code and programmatically access the token so we are going to actually look at the service account first and here luckily when you create a service in google cloud it actually automatically attach a service account sorry when you create a service in app engine actually automatically attach a service account to the server to the app engine service in in this case our discord board so we just need to give it permission to access the circuit secrets we just created so to do this we go to the im and admin collection and in this category we find the service called iam so this is another google cloud service and im stands for identity and access management um put it simply it's it manages who can access what so in this case um on this page we can see there is a service account already being created and the name of the service account is app engine default service account service account so this is exactly the service count we need um now it already has the row called editor and this row is actually quite wide gives it a lot of permissions to do different things um we can probably narrow it down a little bit later but for now we need to give it a specific role that enables this service account to access secret manager so we click the edit button and we choose to give this service account another row so this is add to the existing row and then in the select row we click on here the best way to find the fastest way to find the exact row you need is is you need type the service name so in this case we type in secret manager here you can see there are a few rows you can choose from um and of course here it it requires a little bit experience um and the more understanding of google cloud to decide which is the best role um for your purpose and sometimes you need to create custom rows as well but here for the purpose of this demo we are just going to use the existing row and this row would be secret manager secret accessor what it means basically it means it enable the service that uses this service account to access secret so it's quite self-explanatory so we're going to add this row and we save so now this service account should be able to access the secret in the secret manager already so this is one thing done we already attached the service account which is automatic automatic automatically managed by app engine and then we give the service account sufficient permissions to access the token in secret manager but of course because i'm doing this live so i don't know what exactly needs to be done either we may find we have missed something but hopefully not but even if we miss something we can always go back to fix things so this is one thing done we're going to move to the next thing which is incorporate the secret manager python sdk in our discord board and programmatically access the token in secret manager so the best the place to go because i have already used the secret manager before so i know there is a library called i think it's this one called google cloud secret manager so quite self-explanatory name again so this is the library we need um this library so in the quick start guide it it actually asks us to do a couple of things that we have already done for example enable the secret manager api so by the way if it's the first time you use secret manager service in your project there there will be a enable api button you just need to press this button and then in a minute you should be able to use secret manager and we already set up authentication hopefully that is giving the permissions to our service account in order to access the secret manage so personally i found actually the the example code in the in the product page of secret manager is is easier to follow so i'm going to find the secret manager quick start documentation so this is the product page of a secret manager and there's a quick start tutorial and we we are on the python tab so this example code is where we can actually start from by the way we here we we actually need to install the library as well before we can we can go into the code so i'll just this library can be installed through pip so i'll just copy this this part so i'm already in the virtual environment so i can just use peep three and install and now it's already installed so we can go back to the example code we can copy the we we can copy line by nine and decide which part of the code actually is useful for us so we i will just copy the first line to start with so i will put it maybe just the before we define the discord client so after we imported the discord library so we import the secret manager sdk as well um and then the next two lines are defining the project id and secret id we copied here so how to find my project id and there are different ways but one way is to go to your google cloud platform console and click on the project name and then you can see the id on the here on the right hand side so we just put project id here and the secret id now if i know if this is your first time using um secret manager this may be a little bit confusing because on the on the secret manager page it actually does not tell me which is the id so i will just copy the secret name for now replace and then we go back to the the example code again so next line is we need to create a secret manager client so this is very similar to the discord client we created last time so a client is effectively a long human user that we will we will use as a proxy to access a service so this client is for secret manager services service specifically but we have a little issue here that because in the for the discord client we already call this client that we defined client so here if we want to this secret measure client is a different one so we'll probably want to give it a different name so i will just call it secret manager underscore client so the issue is resolved and from here we actually cannot use much of the code in the example anymore because what this example trying to do is to create the secret and store the secret in store the token in the secret um since we already created the secret and stored it um from our secret manager console here we don't need to do this in our python code anymore all we need to do is to somehow access this the secret so here we actually need to go to the the documentation page the reference of the google cloud secret manager sdk or the library now so if we go to the client library documentation and i know if you are fairly new to python or to programming in general you might find a little bit difficult to read a documentation or reference page like this and don't worry about that it's perfectly normal and i think if you're watching this video you are actually doing the right thing because personally i found it's very effective way to actually watch over someone's shoulder to see how other how another person finds the information they need i think it's a very good way to learn how to how to read a reference web web page like this and don't worry about that if you find a little bit lost from time to time if you watch this video it will come within time so we are at this library documentation page the reference on the left hand side of the navigation menu there actually they are reference for references for version 1 api and the version 1 beta 1 api for the purpose of this demo we are just going to use the version 1 api so we are looking at services for now and there's only one service in the library which is secret manager service and here the first class it defined or described in this reference is a secret manager service client does it look familiar to you yes we just used this secret manager service client actually we just called the constructor just now so this is a class we are using so a class comes with methods so we probably want something like access the secret method um to go with our client so we can scroll down and the first method we come across is called access secret version um here i can tell you with 80 certainty that this is the method we need but why is access circuit version instead of access secret um so here we actually need to go back to our whiteboard a little bit to explain the concept a little bit so just now we put the token in secret manager but the secrets in secret manager is managed as well not surprisingly secrets so they are the secret so you if you remember we created a a secret in here so the resource in secret manager is called secret but the token is actually not assigned directly to the secret resource instead it it it is assigned to a specific version so for example here we have version one and the version one has this token and the secret has version 1. so in the future we may have version version 2 version 3 and so on but for now our token is stored in version one of the secret so those are the different levels of resources within secret manager so in order to access this token we actually need to access the circuit version instead of just the access the secret because if we access the secret resource it it will probably tell us i never tried before but it will probably just tell us you have version one version two and version n but if we access version 1 directly we'll be able to get the value of the token which is what we need so i think this method is what we need but um when you come across a new library like this it's quite normal that um it takes some trials and errors to actually figure out exactly which method and which and how you can access certain things so but that's fine we can just give it a go for now so i'll copy this method and what i want to do is to because it's a method of the secret manager client so i will just type in secret manager client and dot actually because i installed this library is my ide which is vs code already give me a hint of all the methods and the properties i can access and very lucky the first one access secret version is is what i think i will need and we need to probably pass some arguments to this method for example you need to tell the your secret manager client which secrets and which version you want to access so therefore we although invest code it gives me some hint but it's not the best way to read it so i probably want to go back to the reference page so it says it takes a few parameters and some of them are probably optional and some of them are mandatory so here the first argument is request and because i used this library before so i can probably just fast forward a little bit so if you if you keep reading on there's a name uh argument as well and it says the name argument corresponds to the name field in the argument in the request in on the request instance if request is provided this should not be set so basically what it says is you can either give a pass a request object or you pass the name but for the simplicity we will just pass the name as a string and this is most straightforward and the the name of secrets should be passed in this format so projects and i guess here could be a project id and the secrets and probably will be the secret name or secret id and here we'll pass a version for us it's probably version one because you can because you can see here this is version one so um we'll just uh we'll just provide this name and i will copy this format here so we are already at this method and in python we can pass a named argument like this so to be named equals to use text stream formatter so in here i will put my project id so i just pass it as a variable and here i'll put my secret id and here version i'll just choose version one just simply keep it like this for now so now we can already test whether our secret manager client can access the secret and the secret version so i will just use the terminal to run our code so we will just use the python3 command and the name of our file is main.py we'll probably have an arrow here but we'll we'll see how we can fix it so the arrow here is um google us exceptions so it's probably an exception to do with authentication or permissions could not automatically determine credentials please set google application credentials and blah blah so the reason we are receiving this math this error is that remember just now we said in order for our secret manager to know who we are we need to give a service account to app engine and we have already done this but now i'm running this test code locally on my computer so actually the service circuit manager does not um know who i am like and who and whether i should have permission to access secret manager or not so what i need to do is actually go back to the service account page and find the service account we for app engine so this is app engine default service account and we click on the email of the service account and we go to we will get to the service account details page and now we can add a key to the service account so this key is like the credential for this service account and we are going to put this key locally and so my local computer can assume this service account when it access it accesses a secret manager of course if you come from a software engineering background and you have worked on large scale applications you probably know that this is not the best way um it's not the best way to use the same service account for your uh production workload and your local testing but for the purpose of the demo and the fall because it's a personal project um we can't just do it as it is for now so we for the simplicity we are going to use the same service account as our app engine and now my browser has already automatically downloaded this json file what i usually do here is to create a folder in my um in my project repository in this case the discord board repository so i could call it i'll call this folder just the credentials um and then i put the credential we just downloaded into this folder and here we can take a look at what is inside this json file as well so the most important thing in this json file is a private key so if private key id is the username of our service account the private key will be the password of this service account as we said before uh a service account is effectively an account for a non-human user so it's very important that to keep this credential file safe as well as keep this private key safe this private key is actually quite long you can only see part of it so now we have got our credential file the next thing we need to do is actually set the google default application credentials so the way to do it is simply to set an environment variable so i use export google application credentials and equal to and in various code i can just copy the path the forecast of our credential file so effectively we are passing the path to this credential file to this environment variable called google application credentials now we can check if the environment variable has been sent correctly by typing echo and the google application credentials and see the path the the value of this environment variable now is a path to this file so this is correct now we can try to run the python code again and this time it should print out the name field in the response let's see so as you can see here it printed out the message which is the name of the um the secret version we have here but of it obviously this is not exactly what we're after what we're after is the value in this secret version so uh if you remember just now we said another field in this response is payload and in the payload we have data so the data is actually secret data so basically the value of our token but one tiny issue here is that this data is returned in bytes so it's not human readable it's not exactly the the token we put into secret manager so we need to decode this data hopefully if we go back to the quick start tutorial here it already tells us to how to decode this data so how to decode this payload data so we can just simply copy these lines into our code and we can remove this this way it should so this way it should decode the payload data in our response and exactly the payload will be the plain test of our token that's what we need now we save this file and run the python code again as you can see here it's it prints out a plain text that we can read and it starts with oda and ends with w and if we want we can go back to the discord developer portal and and it's exactly the same token we have here so this is how we get the token bank programmatically in our python code now what's left for us to do is really easy so effectively we just need to uncomment this code and put our token here so currently this variable name is called payload it's not very descriptive so we'll just call it this code token discord token to be specific and we replace here as well and then we put this variable in between the brackets here now if we run this code um it should be able to call the secret manager api and get bank the value of the token that we stored in the cloud and pass it to the discord library and log on this code successfully so let's try so it gets token back and locked in successfully as this code board demo so um that is really the end of our um our practice today and so just to summarize we have uh stored the token as a as a secret um and we store the token as a version one of the secret in secret manager and we give the permission we give some permissions to the service account attached to our app engine and the permission would allow our app engine service to access the token in the secret manager and then we incorporated the python sdk or the python library to access the token programmatically from our discord board and one last thing we need to do is actually deploy the updated version into app engine and again this is if you watch the video last week you know this is very simple all we need to do is actually just to do gcloud app deploy so as the code is updated and we don't need to make any change in the app.yaml actually there's one thing we need to do before we deploy this new version because in today's demo we actually added a new um dependency that is google cloud secret manager library we probably need to put this as a dependency in our python requirements.txt as well so if you remember we can use pip3 phrase so and this is the library we used and this is a version so we just copy this line along and put in the requirements.txt and now we deploy the so this code is a service and they should google cloud should be automatically figure out which dependencies we need and update our code so we can confirm this is correct and it may take um a minute or two to update this project so maybe using this time we can talk a little bit about comparing the this way of matching secrets so use secret manager with um using this mention the secrets as environment variable we know that in app engine it's a little bit tricky to manage uh your secrets um as inverted variables but if you are using some other services for example you have your virtual server you may be tempted to put your your token as an immune variable so comparing to um to match the secrets in a service like secret manager managing the token as imagine variable is more prone to human errors um as you probably know that most the security incidents result resulted from human errors so although we can put our tokens as environment variables relatively safely for a personal project but as it's too easy to access by developers and if you uh if you have multiple people working on the same project uh pretty much everyone will have access to the environment variables as well so it's not the most secure way and also um we cannot avoid like human error for example if i'm really recording a video i may accidentally leak the the token through my environment variables but if we put in a secret manager once we confirm the service account works we actually can even remove my access so the developer's access to the secret version completely so no human users can actually access the secrets apart from only the service account we we are we have chosen can access the secrets so this way we can mitigate the uh chance of human error at at the maximum level and now if we go back our terminal our app engine service has already been updated so we go back to the we refresh this page and here it already says this version 2. so um since it's deployed i can probably go back to my discord i need to find my where my discord is here we are we are still at last week's demo so our discord board is online and at the moment it's not running locally from my machine either so um we still haven't changed the feature so we basically still type hello and they should return as hello but to see the change we actually made we need to go to the go to the versions we see like this new version is being served and i think we could probably see so yeah we can see the log in google cloud um but obviously um we don't really want you in a production service we probably don't want to print out the um the token here in the log either but just to show that this is working we can see that in app engine printing log actually what i'm i'm going to do immediately is to remove this print print function from my code and perhaps deploy again because the log is again not the most safe place in safest place in the world for you for storing your tokens so it's better not even to print out the token in your log um yes so i will update the service but with this we can wrap up today's video i hope you found it helpful and [Music] this secret manager service is really useful for many other projects as well thank you for watching

Channel: Playground in the Clouds

Views: 1,081

Keywords:

Id: wb4RU5rpooQ

Length: 45min 42sec (2742 seconds)

Published: Sun Feb 14 2021