DevOps Crash Course (Docker, Terraform, and Github Actions)

[Music] hey team my name is sid i run a youtube channel called devops directive where i create videos to help software engineers level up their devops and cloud infrastructure skills first off i want to say a huge thank you to brad for having me onto his channel and also thank you to you for clicking onto this video i know it's a big investment to spend a chunk of your time watching a video like this so i promise i'll make it worth your while this video is meant to be the ultimate one hour devops crash course if you're not familiar with the term devops it's a set of practices techniques and tools to speed up the software development life cycle by bringing together two historically separate functions of development and operations in the next hour we're going to take an application from development to production including containerizing it with docker setting up infrastructure as code using terraform building out continuous integration and delivery with github actions and then finally deploying it into multiple environments including staging and production by the end of this video you'll see how all of these pieces come together and understand the benefits of devops practices working together in concert when i was thinking about which application to use for this tutorial i scrolled back through the videos on brad's channel for inspiration and saw that in june he released an epic two and a half hour tutorial building out a full stack application using a mongodb database a node.js-based api and oauth authentication using passport here's what the application looks like you log in with a google account and then you have a page where you can create both public or private stories as well as one where you can view all of the public stories this makes the perfect starting point for this video as i'm going to take that project and apply these devops practices to it to get it ready for prime time i actually have an instance of this application running and i'll provide a link to that in the description below at storybooks.devopsdirective.com i'm planning to be live streaming during the launch of this video monitoring system resource usage and traffic my fingers crossed hoping that the site can withstand a traversing media hug of death so you should definitely open up that application in a new tab refresh the page a few times and create a few stories to add some load in terms of application architecture as i described it's a nodejs application and and we're going to deploy that onto a virtual machine running in google cloud if we wanted to scale further we would likely add additional replicas of this and put a load balancer in front but that's outside the scope of this video for the back end we have a mongodb database and while we could run this on another virtual machine ourselves i'm not a database expert because of this i'm going to offload the operations of the database onto atlas which is a database as a service product from mongodb while it costs a bit more to do it this way it's nice to know that the operations of the database are in good hands finally i'll be using cloudflare to setup dns to route traffic to my server as well as provide ssl termination so that i can have https set up i'm using their flexible ssl option which works fine here but because traffic between cloudflare and gcp will be unencrypted this would not be a good choice if the site contained sensitive data if that were the case we would want to set up ssl certificates on our own server so that we could use the full encryption option using a free certificate from let's encrypt and getting it set up is not too difficult but it didn't fit within the scope of this video now there are a few software prerequisites that i'm assuming you have on your system before this video if you don't have them set up they're fairly easy to google and then install the first is node and npm the second is docker the third is terraform fourth is g cloud the command line utility for google cloud platform and fifth is make now if you're a windows user you'll probably have to use the windows subsystem for linux in order to have my code work properly on your system now speaking of code if you want to follow along the original code can be found in a repository on brad's github and the modified updated version can be found as a fork of that repository on my github i'll link to both of those in the description without further ado let's get into it because i'm using brad's project as the base the first thing that i'm going to do is go to his repo fork it into my own repo and then once that finishes i'll clone it onto my system now that we have the source code let's install the dependencies with npm install npm found a few vulnerabilities in the versions of dependencies used i'll see if i can fix them by using npm audit fix this will scan the project for vulnerabilities and automatically install any compatible updates with the nodejs dependencies installed the next step is to get a mongodb database running on my system the easiest way to do this is to do it inside of a docker container i use docker run and then port forward 27017 which is the default port which will be listening on i'll run it in the background with the dash d flag and then use the 360 neil tag okay now we have a database running the final step to getting this app running locally is going to be to set up our config.environment file with all the necessary variables running on port 3000 is fine we need to set the uri to match that of our docker database so mongodb is the prefix to let it know it's the connection string it's running on localhost on port 27017 and storybooks is the name of the database we also need to create a google oauth client so that we'll be able to authenticate users as they log in and i'm just going to create a brand new google cloud project so i'm starting from scratch by doing this i can ensure that no settings from old projects will bleed over into this one and when i establish all of this infrastructure as code later in the video i'll be confident that it can run for me or for anyone else who tries to duplicate my work now i'll go to the oauth consent screen configuration page choose that it's an external oauth client name my app and set the authorized domains to devopsdirective.com because i'm going to host it on a subdomain of that now that we have the consent screen set up i need to create a client under the credentials tab i click create credentials oauth client id it's a web application this is just an internal name for the client and then i need to add the authorized origins where the requests are going to come from and where the oauth client is going to redirect to after a user is logged in first i'm going to set up the local host ones but i'll go ahead now and add both the staging and production versions of the devops directive subdomains this way i don't have to come back and do this later for the redirect uris we're going to use the same domains but we're going to append slash auth google callback which we defined in our passport.js function as where google is going to redirect to with the oauth token with the client configured we just need to copy this client id as well as the secret into our configuration file and hopefully we'll be able to run the app i'll start the app in development mode by using the npm run dev script and then go to my browser and access the application on local host port 3000. if i log in with one of my google accounts it looks like it was a success awesome with the application running we can move on to the first step of applying our devops techniques the first is going to be to dockerize the application now docker is a great tool that allows you to make sure that your development environment is as close as possible to the production environment so that you won't have to worry about slight inconsistencies causing bugs down the road before i actually populate that docker file i'm going to create a docker ignore file in which i add the node modules directory this prevents my locally installed dependencies from getting copied into that docker image and causing issues now i'll create a simple docker file for the application i'm starting from a node version 14 base image using the slim tab helps to make the image slightly smaller i'm setting my work directory to just a standard convention of user source app next i'll copy in the package.json as well as the package lock.json so that we can install the dependencies now the reason that we only copy those files first rather than all of the source code is that it helps docker's caching process so that we don't have to re-install all the dependencies with every modification to the source code if you're interested to learn more about docker and how to improve your docker files feel free to check out these videos on my channel in which i walk through some of the core principles with the dependencies installed we can then copy in the rest of our source code and modify the default root user to use a non-root node user we'll then expose port 3000 where the application's listening for requests and set a command that's going to be run when the docker container is started in our case it's npm start which will start the application in production mode to build the docker image from this docker file we use the command docker build and then the t flag allows us to specify a tag and then the period indicates that it should use the current directory at this point i could run this container directly by issuing a docker run command but because is also running in a container on my system i'm going to use docker compose to coordinate those two and the networking between them to configure docker compose we create a yaml file in this file we need to specify each of the services we're going to have our api server as well as the database then we create a network for these two services to communicate on each of our services requires us to specify either an image tag or a build directory for the api server i'll just use the local directory as the build directory so docker compose can build it on the fly i'll also pass at the location of our environment variables file configure the ports so that port 3000 on my localhost will be forwarded to 3000 inside the container and then add it to that storybooks app network finally i can specify that this first service depends on the service because needs to be running in order for the app to connect for the service we're going to use that same public image as before i'm going to set this init db database environment variable to equal storybooks this ensures that when the container starts up it actually initializes that storybooks database i'm going to port forward 27017 and then connect it to that storybooks app network finally i'll add a data volume where it's going to actually store its underlying data this ensures that the data will persist across restarts which wouldn't be true if we just stored it within the container itself this volume then gets mounted into the container at slash data db which is the default location for to store its data there's one additional configuration setting to update now that the application is running inside of the container localhost no longer has the same meaning and instead we need to connect to using a hostname equal to the service name in our docker compose file in this case it's now i can just run the docker up command and it'll start both containers and the application will be ready to go let me just log in to make sure things are working and now with the application dockerized we can start to provision the cloud infrastructure for us to deploy this before i go off and do that i'm going to create a file i'm going to use this makefile to store different commands so that i don't have to remember them in the future so i'll create this target run local and it will execute docker compose up while it's not exactly this case you can almost think of it as a way to store a bunch of different little shell programs that i'm going to execute independently so for the rest of this video you'll see me adding things to this makefile incrementally as i need new commands and this is really just me doing future me a favor so that when i come back to this project in six months and have no idea what any of this is doing i can check the makefile and see exactly what commands need to be run while we could deploy all of these resources using the cloud provider gui's it's much better practice to use a tool like terraform so that we can define all of our infrastructure as code and be confident that the deployed resources matches our desired configuration when it's run terraform produces a state file can contain sensitive information because of this it's better not to store that locally and instead to configure a remote backend in this case google cloud storage where we'll store this date file in a bucket that way we can protect it with our identity and access management roles this prefix value just tells terraform where within the bucket to store it i haven't actually created this bucket yet so i need to head back over to the make file and create the command to do so i'll do this using the gsutil mb or make bucket command and pass it the project id as a flag rather than hard code the project id i'll store it as a variable so that i can use it repeatedly throughout this makefile after the project id i pass the name of the bucket i can use my project id variable within my bucket name here it's important to note that while most of these resource names are scoped to within your own project the names of google cloud storage buckets must be globally unique i can now execute this make target to create the bucket now i'll copy the name of the bucket into my terraform config and i'll delete the prefix in order to grant terraform the permissions to actually interact with google cloud i need to create a service account this is a user account created explicitly for providing a security context so we'll be able to create a key that will grant terraform exactly the set of necessary permissions to perform its job while additional roles will be necessary i know that to initialize this back end it's going to need storage object admin access and so i'm granting that here then i need to create a key file which will download to my system and i can point terraform to that to use it to authenticate i added this key information to a local json file and then added that to both my git ignore and my docker ignore files so that the key wouldn't get accidentally checked into version control or accidentally built into the container the standard way for applications to use this type of credentials is by setting the google application credentials environment variable so i export that and point it to this file at this point we're ready to run the terraform init command which will initialize the working directory and store the state file in google cloud by navigating to the google cloud storage browser i can confirm that the state file was stored properly however it's using the default workspace we want to separate into a staging and production environments so i'll go ahead and set that up now i'll create a make target named terraform create workspace which will execute the terraform workspace new command taking this environment env variable as an input for now i'll hard code that to staging but eventually i'll set it up to dynamically switch between staging and production also because my makefile is in a parent directory of the terraform directory i'll need to add a change directory command ahead of this the double ampersand combines the two into a single shell invocation because otherwise each command within make will be executed separately invoking that make target will create the workspace and then we'll still need to re-initialize it as we did before this make target will be similar we'll cd into the terraform directory select the workspace and then call terraform init creating all these make targets may seem overkill now but it will pay dividends down the road when it prevents me from accidentally taking an action in one environment that i meant for another with terraform initialized i can now go about defining the configuration for all of the cloud resources terraform is going to look for any file with the dot tf file extension so you can organize this however you'd like i'm going to have one file per cloud provider i'll also create a variables.tf file where i can store the different variables that will be used across the resources terraform uses a declarative language named hashicorp configuration language or hcl now while it's a little annoying to have to learn a new language to use this tool i have found it to be fairly readable that being said there are some competitors such as polumi that allow you to define your infrastructure as code using more general purpose programming languages such as python to define a variable we establish a variable block with the name of that variable and then the type inside of the brackets for example this app name variable will eventually get set to storybooks and be used in many of my resource names the way that terraform knows how to use the apis of all these different cloud platforms is through what's known as a provider so the first step for using any of these is going to be to find the associated provider and add a block to define it within our code one of the best things about terraform is the breadth of coverage of these providers there are a number of officially supported providers and even more provided by the community for the google provider that i've copied here i'm going to make a few updates first i'm going to set the zone and that's where the resources will be deployed into i'm going to pin the version number and the reason that i do that is that if google goes and updates this provider in the background i don't necessarily want to pull that newer version because it could have breaking changes in the api next i'll point it to that credentials file that i created for the service account earlier and set the project id accordingly within this file we're going to need to create five different objects the first is going to be a static ip address for our virtual machine to utilize the second is a reference to the network the third is a firewall rule to allow http traffic the fourth is the operating system image that we're going to use on our vm and then the fifth is the compute engine instance itself i'm just adding these comments here to remind myself later as i go through and fill it out anytime you add a new module or change the providers within your terraform configuration you'll need to rerun that init command this will tell terraform to go off and download the code associated with that provider before actually populating any of these gcp resources i'm going to go ahead and initialize the providers for atlas as well as for cloudflare again i'll pin the version number so i don't get bitten by an update in the future also this quotation mark dollar sign curly brace syntax is no longer necessary in the latest version of terraform for string variables so i'll remove it to stop it from issuing a bunch of errors in order to use this private key and public key authentication variables they need to be initialized within our variables.tf file so i'll copy them there and then define the type as string we're going to need to define three resources within this atlas.tf file the first is the database cluster itself the second is a user who's authenticated to access that database and then the third is an ipwhitelist which will reference our gcp instance so that we are only allowed to connect from that ip the third and final cloud service that i'm using to deploy this application is cloudflare where i'll set up a dns record pointing to my server again i'll copy the sample code for initializing the provider but in this case rather than using this email and api key i'm going to use an api token so i'll remove those and add a variable corresponding to that token within cloudflare we're going to need to set up two resources the first is the dns zone and in this case i've already set up the zone because i'm using cloudflare for the root devops directive domain the second is going to be a dnsa record corresponding to the sub-domains of our choosing in this case it's going to be storybooks staging and storybooks now that i have all of my terraform providers configured and a skeleton for each of my files i'm going to go back and actually populate those resources as with the providers my general approach is to find the corresponding resource within the terraform documentation copy that example and then modify it to meet my needs in this case the only configuration needed for the static ip address is a name i'm going to use terraform.workspace in my name so that i can differentiate between the staging and production static ips and prevent a naming conflict for the vpc network where this vm will live i'm just going to use the default network that compute engine sets up because this is a brand new project i actually have to go in and enable the compute engine api from within google now this is a good time to explain the difference between resource and data blocks within a terraform file a resource block is going to be some resource that we're creating a data block is just a reference to an already existing resource in this case because our network already exists we're going to use the data block where we provided the name and then it will find it within our gcp project and we can use that to reference it later next up we're going to define a basic firewall rule in which we allow http traffic on port 80 to make it to our virtual machine starting from the example usage i'm going to modify both the internal terraform reference name as well as the name of the firewall rule within gcp here we see that it actually uses that data object that we defined above by referencing google compute network which is the data type default which is our internal reference name and then the name field we don't actually need icmp access so we'll just delete that block entirely we do want to allow tcp requests but only on port 80 so we'll modify the set of ports we also want to include a source ranges configuration where 0.0.0 0 represents any ip so any requests coming on port 80 will be allowed we set up target tags and that's how we actually attach our virtual machine to this firewall rule we'll include this target tag within the metadata of that virtual machine with the firewall rule configured i now need to create a data object to reference the operating system that my virtual machine is going to run in this case i'm going to use container optimize os which is a purpose-built operating system from google specifically designed to run containers as a number of benefits such as a smaller attack surface area for improved security that being said because container optimize os does not include a package manager you'd be unable to install additional software packages directly onto the instance so any configuration options will need to be handled at the container level the final gcp resources that we're going to define is the compute engine instance itself this is the virtual machine where we're going to spin up our node.js server and it's going to handle our requests once i copy in the example usage i'll just make the necessary modifications first i'll change the internal reference name as well as the name within gcp by using the app name within this instance name it would be easier to tell which vm was which if i were hosting multiple applications within the same google cloud project rather than hard code the machine type within this template i'm going to use a variable here so that i can use different size machines for staging and production this way i can have a smaller affordable instance type on staging and a bigger machine type on production to handle the production load as with all of my variables i need to also declare it within the variables.tf file this tags field is how we're going to apply that firewall rule that we defined above we use the data type google compute firewall we reference the internal name of that firewall and then we specify the target tags key the boot disk configuration is where we're going to reference that operating system image that i defined above because it's a data type and not a resource object i start with data specify google compute image and then reference it by that internal name here i also decided to change that internal name to be something more specific finally we use the term self-link to reference the entire object rather than a specific field within it we don't have a need for a scratch disk so i'm just going to delete that block it might be useful if we had some sort of caching or a need for a local temporary data store but in this case i'm just going to eliminate it for the network interface while technically leaving it as it was would have worked i want to reference the data object i created earlier pointing to that default network that way if i ever decide to modify this in the future it'll be easier and i only have to change it in one place rather than having the string default hard coded in multiple places in the access config block we can see this comment from the example code saying ephemeral ip we're going to replace that with a reference to our static ip that we defined above we don't need this metadata or metadata startup script blocks and we're going to modify our service account scopes to have only storage read-only access this access is necessary in order to be able to read our docker image from the google container registry with all of the google cloud resources defined i can go back to the makefile and create a target to execute the terraform plan command which will create an execution plan to actually provision these resources before i execute this command i need a way to pass in my terraform variables some of these will be common to both staging and production and some of them will be specific to each i create an environment subdirectory and a staging subdirectory within that i then define config.tfrs within there this is where anything specific to staging will live i also create a common.tfr files that'll have any variable shared across my different environments the common variables include both the app name as well as the mongodb access keys for now i'm just going to use a dummy placeholder one two three four eventually i'll need to replace that with the actual keys the gcp machine type i'm making unique to each of the environments so i'll put that within the staging subdirectory within my terraform plan command i now need to pass these variable files in using the dash far file option for the environment specific file i can substitute that env variable in when specifying the path to the file once i run this command i can look at the output in the terminal to see exactly what resources terraform is going to create when i apply this configuration with the plan looking good i'm going to update my make target to be able to apply it while i could just copy my previous target and modify it from plan to apply instead i'm going to make a generic target that can perform both by using the environment variable tf action using the question mark equal sign syntax indicates that if no environment variable is set it will use plan otherwise it will use the environment variable so in order to use this target for terraform apply i'll set tf action equals apply and then run make terraform action terraform gives us one last chance to bail but if everything looks okay we enter a value yes and it will go off and provision those resources or if we forgot this im role it will just throw an error because terraform is establishing a compute engine instance using this service account we need to give it the service account user role running our terraform apply command again we see the compute engine instance created successfully within the console and with that we can move on to defining our mongodb resources within atlas the first thing to do is to create a public private key pair to provide terraform access to atlas this can be done under the access manager by selecting the specific project clicking the api keys tab and then clicking create api key give it a name and copy this private key between filming this and the time it's going to air i've already cycled this api key so i won't bother to hide it i also need to grab the public key and then set the project permissions such that terraform will be able to create and modify database clusters within this project the first atlas resource that we're going to define is the cluster itself it's important to note that the terraform provider for the mongodb atlas doesn't support the free tier instead we're going to use the m10 tier cluster which will have 10 gigabyte storage by default and 1.7 gigabytes of ram atlas supports all of the major cloud providers so i'm going to select this example gcp cluster and then modify it to meet my needs rather than hard code the atlas project id i'll store it in a variable the project id can be extracted from the url when you visit the atlas website i also need to remember to declare it within the variables.tf file the fields that i need to modify here are the internal reference name the name of the cluster within atlas which version of mongodb i'm using and i'm just going to use 36 to match the version i used earlier and finally the instance settings including the size of the disk the size of the machine and the region it's deployed into next up with the cluster defined we need to create a database user for the application to read and write from the database as per usual we update the internal reference name then i'll use my project id variable create another variable for this user's password making sure to define it within the variables.tf file and then setting it to some random string in our tfrs file for the username i'll include the terraform workspace so that i can have separate users between staging and production i then need to update the database name within this read write role to grant access to the storybooks database i can get rid of the admin read any database role eliminate the labels and remove all these scopes the final resource that we need to define within our mongodb atlas configuration is an ipwhitelist that will allow our virtual machine to connect to the database cluster this is where the power of a tool like terraform really starts to shine where we can have resources living in different cloud systems and be able to cross reference them between the two configurations rather than using a cidr block we'll reference that ip address directly the resource is of type google compute address the internal name is ip address and the field we are referencing is address with this all configured i can rerun my terraform apply make target and it should provision those resources the cluster takes a while to provision so in the meantime i'll go ahead and get started configuring cloudflare as with the other cloud platforms the first step is to generate the access key the page to do this can be accessed by clicking the settings menu in the top right and then my profile on the api tokens tab when you create a token you give it specific permissions in this case i need to have read access at the zone level so that i can see the zone that already exists and then i also need edit access for the dns settings for that zone you can also choose whether these permissions apply across all of cloudflare or only for a specific zone i can now place this token within my tfrs file and use it in terraform to authenticate also because i forgot to declare the variable earlier i'll do that now because i already have a zone set up for the root domain of devops directive within cloudflare i'll create a data source to reference it if instead i was setting it up from scratch i would create a resource for that zone rather than hard code the domain name i'll create a variable for it named domain and populate the value within the common.tfrs file the second cloudflare resource that we're going to create is a dns a record or address record pointing from our domain to the ip address of our virtual machine this record is going to reference the zones object from above which i'll change the internal name from example to cf zones i'll also update the internal name of this a record the zone's data object above returns an array so in order to reference our zone id we need to use data cloudflare zones which is the type c of zones is our internal name we reference the zones field at the zeroth index and then the id field the name field here represents the subdomain that the site's going to be hosted on i can come up with an expression using a ternary such that for staging it will be storybooks dash staging and for production it will just be storybooks i do this by checking the workspace and if it's equal to prod i return an empty string while if it's equal to staging i return a dash and then the workspace this would allow me to deploy other environments as well such as development environment just by adding a development workspace for the value this needs to point to that ip address from the gcp vm so again we're going to reference it with google compute address ip address and then the address field i also want to turn on proxying so i'll set the proxy field to true as you can see from the output in the console that mongodb atlas cluster is still being provisioned six minutes later so we're going to go ahead and move on while that finishes before we can deploy these cloudflare resources now while we're waiting on that i'm going to go take care of something that i think gets overlooked too often on youtube tutorials and that's management of secrets so up into this point i've been storing my secrets as either environment variables and end files within that dfr file to authenticate terraform i'm going to move those into google secret manager and only retrieve them when needed that way i don't need to store any of this private information on my local system the only thing i need locally is to be authenticated to google cloud the process of creating these is pretty straightforward so i'm going to hyperlapse it so you don't have to watch me type them all in proper secret management becomes even more important when you start working with a larger team in a corporate setting there are a number of tools available to help make this easier and some of them can even provision short-lived or one-time use credentials such that the impact of a leak is minimized okay it looks like the mongodb cluster has finished provisioning so let's run terraform apply and try to create these cloudflare resources oh i had used the incorrect field name for the api token rather than just token it needs to be api underscore token running the terraform applied make target once more and it looks like it was added i'll confirm this just by going to the dns tab within the cloudflare gui earlier i created all those secrets within google secret manager but i didn't actually explain how i was going to use them i'm going to create a little helper function within my make file called get secret where i can retrieve those at the time that they're needed this function will invoke a shell that uses the gcloud command line utility to retrieve the secret this dollar sign parens 1 references the first input to this function so we can pass in any secret name i can now remove all of those secrets from my tfr files and instead pass them in directly with the dash var option to use my function i use call the name of the function and then pass the name of the secret key this way i can eliminate all of that sensitive information from my terraform configuration i'll just add the atlas user password and the cloudflare api token and then i can delete them from the tfrs file let me check that things are working by running our terraform plan command and it's asking for the mongodb atlas private key oh it looks like i used the wrong variable name i called it just atlas private key let me fix that if i run it again the command succeeds and we see that there's no changes to the infrastructure this is good with all of our infrastructure defined as code and successfully provisioned we can now move on to deploying the application there are a number of ways that we could handle deployment for this application but here i'm just going to keep it simple i'm going to send some ssh commands that execute the necessary docker actions and that's it the first step to accomplishing this is to create a make target for sshing into our virtual machine this will use the gcloud compute ssh command the command takes an ssh string as an input which is user at and the name of the instance and then i'll also specify the project id as well as the zone to uniquely identify the machine executing this make target takes a little while while the ssh key propagates to the compute engine instance metadata but once it finishes we should have a live shell running on that vm while we could just type out the docker commands that we want to run here within that session i'm going to create another make target to send individual commands over ssh this will be much easier to reuse down the road when we build out a github action to automate this process this make target is going to look nearly identical to the one above except it's going to take in this cmd environment variable so we can specify different ssh commands to execute rather than opening an interactive ssh session this will send that one command and return the result for example if i execute this make target with command equals echo hello that will get executed on the virtual machine and i'll get the string hello back in my local terminal before i can actually deploy the application i need to build the docker image and then push it to google container registry where my virtual machine can pull it in order to do this i'm going to create build and push make targets when i build it i'll apply this local tag but then in order to push to google container registry it needs to follow this format so my push make target will first tag that local image with the remote tag before pushing it to google container registry the container registry api is not enabled in a brand new google cloud project so i'll need to go and enable that before i can push now i'll run make build which will build the latest version of my docker image followed by make push to push to google container registry depending on your internet speeds this might be very fast or very slow okay it's time to write our final make target for the day that's going to be our deploy make target we're going to craft a number of commands that use that ssh cmd make target and execute those commands via ssh on our virtual machine the first command that we want to run is this docker credential gcr helper function which will configure the docker instance on our virtual machine to be able to authenticate to google container registry this is necessary because it's a private registry and we need to be authenticated in order to pull our docker image technically this command only needs to be run once on the virtual machine so it could be moved to a startup script but i'm just placing it here for convenience the next command that i'm going to run is docker pull to fetch the latest version of our container image because i'm only running one instance of our server there's inevitably going to be some downtime from the time that i stop one container to start the next and pre-downloading that image helps me to minimize that time at this point we want to stop any existing container running on the virtual machine we do this using the docker container stop command followed by the name of that container now if we didn't specify a name for the container when we started it it would be assigned a random name by docker but instead we can pass it this container name flag so that it will have a deterministic name we will also want to run the docker container remove command to remove the file system associated with that container depending on how you issued your docker run command this may or may not be necessary you can add a dash dash rm flag and it will do this cleanup automatically when the container is stopped i'm also going to add a dash to the start of each of these two commands that will tell make that even if the command returns a non-zero exit code meaning that it failed we'll proceed with the following commands for example the very first time we run this deploy command there will be no container running so docker container stop and docker container remove will fail because no container is running but we still want to proceed with the rest of the make target at this point the only thing left to do is to issue our docker run command i'm going to run it with the dash d flag to run it in the background as a daemon i'm going to pass it that container name that we defined earlier specify the restart unless stopped command so that if the container does die it will restart automatically then i want to port forward on port 80 of our vm which is where http request will come in to port 3000 inside the container where the application is listening next i'll set a few environment variables with the dash e option we want to set the port to 3000 then we need to set the uri environment variable to tell the application how to connect to that atlas db instance we can go into the atlas gui and it will provide us a template string i added escape double quotes to the beginning and end of this environment variable definition to deal with how the shell is going to expand the string before it gets passed to docker within the template string i then needed to update both the database user as well as the database cluster name so that they can easily switch between staging and production i then retrieve and pass in the database user password from google secret manager and set the database name i then need to pass in the environment variables to configure our oauth setup we'll paste the client id here and for the oauth client secret we're going to store that in google secret manager so we'll use our helper function to retrieve it before creating it within the console the final argument to the docker run command is the container tag in this case it's going to be the remote tag where we pushed our image to google container registry barring any errors or typos i should just be able to run make deploy and it should spin up the container on our virtual machine if i now navigate to the storybooks-staging.devopsdirective.com url i see that the login page loads and i'm able to log in let me just create a test story looks like things are working at this point we want to take that manual deploy process and automate it using github actions i've used a lot of different ci cd systems and github actions is definitely one of my favorite one of the more unique things about it is that they really have an emphasis on sharing and reuse and so it's very easy to find an action that someone else has built that takes care of most of what you need to do and then you can build upon and modify that action to suit your needs because i'm going to use these make targets within the action i want to make just a few improvements first i'm going to add the at sign in front of this final make call as well as in front of the gcloud command in the ssh command make target this will prevent make from echoing the command into the shell before it executes and will prevent all the secrets that i'm setting environment variables for from getting printed to the console in the github action logs i'm also going to add just a few echo statements so that now we'll see the progress as these commands are executed now when we rerun make deploy we won't see that sensitive connection string or the oauth client secret in the console at this point we're ready to create our github action workflow lucky for us google cloud platform has released an official action that they have developed for deploying to their virtual machines i'm going to use this as the base for my workflow and with just a few tweaks we'll be able to deploy from github actions to our instance the way that you specify a workflow within your repository is to create a dot github workflows directory and then any yaml file within that directory will get interpreted as a workflow i'm going to name it build push deploy because those are the three steps that it takes after pasting in the code from the google cloud platform action the first section that i'll modify is this environment variables section because the instance name and the zone are handled within the makefile i can just delete these two i'm also going to set the project id directly here rather than as a secret within the repository if i were using the same project within many different actions i might store it as a secret but here it's simpler just to do it this way we can see that the first step of this workflow is the checkout action this is a standard action provided by github that will be the first step of almost every workflow it grabs the version of the code corresponding to the event that triggered the workflow the second step is gcp specific and is where it configures the gcloud command line utility here we need to modify the project id to use that environment variable we set earlier and we need to pass it the key for a service account that is authorized to take the necessary actions while i could use the service account that i created earlier for terraform it's better practice to create a new service account with exactly the roles that are required in this case this service account won't need permissions to create and destroy virtual machines only to deploy to them we will also need the surface account user role because the ssh commands that are sent to the virtual machines will technically be executed as the surface count identity associated to that machine because we're going to retrieve secrets we'll need the secret manager accessor role and then in order to push to the google container registry we'll need to have access to the bucket where those images get stored with the permissions all configured i can now create a json key for this service account i'm going to copy the contents of this key into a secret within the github repository that can be accessed during that workflow to create a secret you go to the settings tab for the repository click secrets and then new secret i'll then paste in the content of the key to the value section and give it a name matching the one i used in the workflow as described by their helpful comment this third step will enable docker running in the github action to be able to push to google container registry these final three steps are where some of that work that we put in building out that makefile are going to start to pay dividends rather than using these commands that they have specified we're literally just going to put make build make push and make deploy and our makefile is going to handle all of the different configurations and environment variables and pass those to those commands this works because the github action is running on a linux based system with make installed now while this would work as is i'm going to make one change to improve it previously i was just using the tag latest whenever i built and tagged my image it's better to tag with a unique version number for each version so that when we pull a specific version we know what that corresponds to within github actions there are some reserved environment variables including github underscore sha which will represent the commit hash of the event that triggered the action here i'm going to tag the image using that hash the way that i'm doing that is a bit hacky when i run it locally this variable will fall back to latest but when it gets run in github actions it will use that environment variable the github action is configured to run whenever there's a push to the master branch so i just need to commit all these files and then push it to the github repo rather than just stage all of the files at once i prefer to go through and look at each file briefly to make sure that i'm not mistakenly checking in any secrets or if there are any outstanding to-do's to take care of for example here i noticed that many of the files listed are actually within this dot terraform directory which is just a local cache that i don't want to version control so i'll add that directory to my git ignore with all of the files staged i'll just add my commit message commit and then push now if i go to the repository within github under the actions tab i'll see the workflow executing for the first time within this view we'll see the workflow progressed through all of the steps that were defined within it it's checked out the code it's setting up the gcloud command line utility authorizing docker building our image pushing that image to the google container registry and finally deploying onto our virtual machine with the green check mark signaling successful completion let's go try to log into the site again and make sure everything's still working cool looks like we managed to not break anything while everything up to this point individually has been important this next step is sort of where it all comes together and for me is the real aha moment for how devops is so powerful if you're working on a project within a corporate environment you're not going to have just one deployed setup you're going to likely have multiple staging environments and then a production environment where your users are going to interact with your application because of the groundwork that we've laid thus far we'll only have to change about 10 lines of code and we'll be able to spin up a production environment in no time previously we had hard-coded this env environment variable to staging now we need to be able to switch between staging and production because i want to prevent someone from accidentally causing these make targets without this environment variable set i'll add a little helper function to make sure that it is defined and if it is not it will throw an error telling you to set it to staging or prod i can then put this check and make target as a prerequisite for these other make targets so that it will get run before they do because production will live as a separate workspace within terraform the first thing that i need to do is run my make terraform create workspace make target with env set to prod with the workspace created i can then run make terraform init again with the nv set to prod to initialize that workspace before i can actually provision the resources i need to create a production environment within my environments directory and the tfvrs file corresponding to production as a reminder this file contains a variable representing the machine type for my virtual machine within google cloud platform once that's set i can use my terraform actions command with end vehicles prod and tfaction equals apply to provision those resources and that's all we had to do to provision a whole nother set of resources associated with production separate from our staging deployment compare this to what we would have had to do had we provisioned those resources through the gui or even just via command line calls the power of a declarative tool like terraform is you specify the state that you want your infrastructure to be in and then it handles reconciling that with the current state of the infrastructure to make sure that it achieves the desired result while terraform is working on provisioning those i'm going to go modify our github actions slightly to support different deployments for staging versus production the strategy that i'll use here is one that i've seen in many of the companies that i've worked with and that is to deploy to production on version tags associated with releases and so i'll specify that we want to trigger this action on tags that match a regular expression starting with a v followed by three digits for example 0.0.1 etc this way the normal workflow for making a change would be to build that change merge it to master view that everything is working on staging before issuing a release on github which will trigger a production deploy the other thing that i need to add to the action is the ability to set that env environment variable to prod or staging depending on which event triggered the action i'll add a step to the front of this workflow to do just that in order to do this this step is going to take advantage of another predefined environment variable within github actions github underscore ref this will contain a reference to the event that triggered the action if it's a push to the master branch it will be slash refs slash heads slash master or if it's triggered by a tag it'll be slash ref slash tags and then that tag i'm going to use a bash technique called prefix removal with this pound sign to remove everything except for the substring after that final slash then i'll check whether that's equal to master or not if it is i'll set the environment variable to staging if it's not i'll set it to prod it would be better not to have the else condition fall back to prod so one improvement here would be to also check that the second conditional matches that regular expression from before to ensure that we don't accidentally push to production for now i'm just going to leave it like this though now before i commit these changes i'm going to just add a few exclamation points to this header just to prove 100 to myself that these changes are getting properly integrated and deployed through this process now i'll just stage all these changes commit them and push because we were on the master branch we should see a new action triggered and if we go into the logs and look at the environment variables for any steps after our first one we can see that it set env equal to staging perfect once that workflow completes we can go back to our staging url refresh the page and see that our changes are reflected after a few more minutes we see that terraform has finished provisioning our production resources and so by tagging the latest commit and pushing that to github it should trigger a production deploy once again i'll go into the github actions interface to confirm that my environment variable got set properly looks like it has once this workflow completes i'll pull up the staging environment on the left and the production environment on the right as you can see these are two completely different setups with different databases and different underlying data this configuration will allow us to have high up time on the production side while continuing to make modifications and build new features on the staging side and with that we've reached the end of our one hour devops crash course i hope that you've learned a few things along the way that you'll be able to apply to your own projects to make them more robust and easier to work with thank you so much for watching and sticking around until the end i would love to have you come join me in my small but growing community over at devops directive so head on over subscribe and leave me a comment telling me you came from brad's channel that's it for today again i'm sid with devops directive and remember just keep building

Channel: Traversy Media

Views: 106,773

Keywords: devops, dev ops, google cloud platform, terraform, github actions, docker, devops docker, devops tutorial

Id: OXE2a8dqIAI

Length: 58min 4sec (3484 seconds)

Published: Thu Nov 05 2020