Golang Microservices: Configuration, Environment Variables and Vault

hello my name is mario welcome to another video in today's episode i will share with you another tip for building microservices in go specifically secure configuration using environment variables using hashicorp's vault what is the agenda of this video we have four items we are going to be talking about some ways to configure services while using environment variables packages that i like to use for configuration and i will show you the code that is using all of this what are some ways to configure services there are four ways in my opinion one of them will be bundle the configuration with your artifact for example a config file an xml a gml whatever the case may be is bundle part of the binary that you're going to be distributing or part of the artifact that happens to be maybe running in a docker container that's one way to do it another way we'll be using the cli passing in arguments to the binary that then the the binary will be using for actually configure configuring the service itself the other one and this is the one that i like doing is a combination of using environment variables that are injected during runtime and actually using the other there are two above which is a combination of a sort of like a configuration file not a real jam or an xml but also something that we can load through the cli and you will see that when i show you the code so why even why even what are using environment variables well there are a few pros and cons cons like in everything in life in software development the trade-offs here is that the pros is that allows it allows us to easily change configuration during deployments you can literally keep the same artifacts and then depending on the environment you can change the database name perhaps or maybe the password or the username or maybe point to a different database itself a different address and still use the same logic that happens to be living in your artifact that lives in your service nothing changes outside of that it's just a way to run a different way to run the service obviously there are cons and if it depends on if you have a lot of configuration values if you are trying to configure something that happens to have i don't know a lot of details it's most likely using the environment variables is not a good idea and i will show you a way to sort of solve this and it is not secure by default what this means is that obviously if you are going to be using the environmental variables and you are going to be trying to pass in a password for example that is definitely not secure other than the key value data type that we have using environment variables there is really no that there is no data store for that no not i'm sorry not data store but not an actual data type and again there are ways to you know work it using a go package that i want to be showing you but still there are some limitations and there are some some some good ideas right there so let's look at the packages that i like to use for configuration again using environment variables and specifically a computation of the three things i just showed you before a file that we're going to be bundled as in the artifact a cli for passing in some configuration values and the actual environment variables are going to be loaded dynamically depending on how we are running the service there are a few of them obviously the the common example will be using get m which is included in the standard library part of the os package it literally just pulls the value as is from the environment and that's it it's an environment variable not nothing overly complicated now go.m is one that allows you to load a file that it also now nowadays supports variable expansion but it's literally use uh sort of like a port of the ruby.amp project if you're familiar with that one but as you literally just take it it takes in a file a dot m it doesn't have to be a temp but the format is sort of like a key value and it loads those values as environment variables in your pro in your process in your binary the other one is more a little bit more complicated is called m config and this one allows you to use a the same configuration using environment variables but but depending on how you name those environment variables it will let you create a data store and it will also allow you to transform the types from strings which is literally the only thing that the environment variables support to more strict goal types and if you look at the example you will notice that there is this my app with a bunch of different a names and these are applicable to fields that are transformed to fields and depending on the type that is defined in the field in the type struct it will try to transform and type cast those values to the corresponding value that is defined in the m-file so how does it work you know if you're combining a bunch of different things together how does it look like uh i will show you next but before that one of the things that i like doing with them is i like wrapping all of those packages around a secure wrapper which is the the whole point of this video showing you how you can still use environment variables and a paradigm that i like to use for wrapping secure um secrets or something that you need to have a store in a vault or some sort of service that is secure and not open to the public literally and then still use environment variables so let's jump into the code to show you how this thing works compared to the previous version where when i was talking about the project layout and repository pattern one of the things that i did was to refactor the way the the postgres database is configured and now instead of using the database url environment variable that long string that was equivalent to the actual url the postgres url that included all the other values that are needed for connecting to the database now we are defining specific variables for each one of the fields and those fields are being pulled from the environment as environment variables so the way it is right now if you notice is using this type called n nmvar configuration and what is what this thing is doing is literally doing the first part trying to get the values from the environment variable and and the important bit about this implementation is that i'm adding this paradigm that i also discuss in a blog that will be linked in the description that i will show you that it shows you a way to somehow still define defaults but if you happen to have a variable that happens to be has a suffix called secure or young you can literally call it whatever you want to call it it will allow you to define some logic around it and perhaps pull the values from from a secure uh data uh the secure repository that happens to be stored in only secure data which in this case would be vault but there are examples for example in aws you can use the parameter restore system managers if i recall correctly that's the name and then you can pull and store secure data and pull the values from there the idea is practically the same the way it works is that as soon as you receive the name of the environment environment variable that you're trying to get and again if we go back to the file you will notice that what i'm trying to do here if i jump jump down i'm trying to get the value for database host okay and then i jump into the configuration and i'm trying to call get and what configuration.get does is get the value from the environment variable and then it tries to get a value that happens to be again using the suffix secure if the value exists it goes and pulls the value from the remote provider which happens to be the secure data store handling all the secure data and only secure all the secrets and whatnot the implementation that i have right here it's a vault and i'm running locally a volt server that is going to allow you to get the values from that secure source the way this class this class this type is implemented is that there is a is receiving the provider type which happens to be implementing this get method then the volt type which is in the vault package is specific to the vault api is also implementing the get method i will not go through the whole logic right here but basically what it's doing is just defining a new key and then a path and depending on the key you will get the values from volt and pass down those values back down stream to the actual server and if i run this one thing i want to show you first before running anything is the example without the uh volt implementation that means or see if i'm running no i'm not running the container so the wrong volt there is an instruction right here that i can use oops oh there's a typo okay so if i run this there is a vault uh docker container running right now so the way volt works is that there is a secret engine right here that is using the key value data what is the name they call it engine the cake value engine and we're going to be doing here is defining a path called database and all of this is going to make sense in a moment i just want to do it right here so you can see it and we're going to define in the username the value will be user the password will be it's going to be password and we're going to be saving it now the idea of this is that when we go back to the example using the secure environment variable we will be able to pull the values from vault let me show you now that everything seems to be configured we're going to be running the same example using m example which is this file and it will be doing what we were describing last time it will be going to the database and inserting a record now if i decide to you know what let's pull the values from vault or from the secure data store i can do exactly the same and it will be inserting the values as before now a little uh recap in case i was going too fast the way this works is one we are defining a data store in charge of connecting to our whatever service that is in charge of handling met and handling tokens or handle handling secure strings and those kind of things that's one part the other part is wrapping that using an interface type for loading the configuration that is coming from the environment variable and then depending on whatever logic we decide to have we can pull the values remotely from that provider the way we are determining when to do that is when we find a variable that happens to be using the suffix on the score secure if that exists we go through the process it doesn't have again doesn't have to be underscore secure it can be anything and that's one part after that is we need to go and update our main file to instead of using the long urls all the long form environment variables to start using short form or concrete field based environment variables and that would allow you to give it will give you more it will give us more more flexibility when trying to configure things the other thing to to call out here is that obviously this configuration which is the important thing in volt and is the token that depends on on how do you how do you pass this token value around that depends on how you're running your your binaries if you happen to be running a token container maybe you can inject that when where you're building the not the image but more when you are configuring whatever is running that uh image so if you are using ecs which is in aws you can configure that via the the i i am or if you are using a different service you can configure that in a different way the whole point again because volt token is literally the key to unlock the default uh you need to consider that as a really important secure token value that you you shouldn't be passing around or make it explicit when the value itself is in the is being run through the docker container for example okay hopefully that makes sense the way it is right now for development purposes and for demo purposes is that i'm doing it here much more explicitly and hopefully all of that made sense and if not just leave me a comment i will do my best to explain it again as usual the code is in the the link to the code to the repo is in the in the description so you know feel free to check that out whenever you have time and you know as usual thank you for watching and then you know i will talk to you next time do don't don't give up you know keep it up see

Channel: Mario Carrion

Views: 7,203

Keywords: golang, microservices, golang microservices, golang web development, building microservices golang, golang rest api, golang ddd, golang domain driven design, build microservices golang, golang ddd layout, golang environment variables, golang vault, golang hashicorp, golang configuration, golang envconfig, golang godotenv, golang dotenv, golang env vars, golang hashicorp vault, golang secure configuration, golang secrets configuration, golang tutorial, golang env var

Id: 7UmJR0dOkjM

Length: 14min 41sec (881 seconds)

Published: Tue Feb 23 2021