Database Migrations with Golang Migrate and Docker

hey what's up guys Daniel here today I decided to shoot a quick video showing you how to use : my great package so first we'll look at what our migrations and then we'll dive into technical details so welcome to migrations 101 so what are my equations database migration is a management of incremental reversible changes to relational database schemas it's actually sounding fancy but it's just a bunch of files describing the alterations of the database so why would you use it well it is to be able to recreate your database from scratch it makes it clear at what stage your database is and at all times and it gives you a way to manage your database version so pretty cool so go back to this golang migrate package if you scroll down you'll see that you can use either the CLI tool or the library version today we'll be using CLI version of that now explain you why also you can choose the database that you want to work with will be working with positive SQL today so let's scroll down a bit here and see ok so first why would you actually use a CLI tool well first if you want to do migrations manually it's very nice to run a CLI command to do that and why would you not want automatic migrations well imagine you have like 100,000 users and your life database and then if you have automatic migrations you can't really control them cannot really manage them and then you know you sometimes want to do a partial migration or something like that based on your code and with automatic migrations if it's done on deploy or something it can mess things up the second reason is as you see there's a darker usage that you can use to just run the migration command just by using the image in the docker hub you don't really have to install anything on your machine also in my case later on I like to just make a separate image where i copy my migrations there just make a volume and then use the migration tool from there not to overload and container like back and put in migrations there and running them I like to have a you know separation of concerns and just running it on a separate container but if you want to use the library you can dive into the documentation here you can see that yeah you can create a new kind of database here with a migration or you can use the existing one but you just refer to documentation and see it's very nice the commentation well-written it's pretty self-explanatory how to do that if you want to use the library so let's dive into how do we actually write migrations so as you see the structure of migrations is very simple you have a version you have title and then you have extension so let's look at the example here so it says one initialize schema down SQL and up Skrill so up migration will actually do the changes to the database and down migration will reverse the changes that were done by the up command and here is the version number of the database so for example if I do a down migration here it will again be on the version first if there will be a version three and I would run a down migration of the version 3 it will reverse the changes that were done with the third up migration and it would be back on version 2 of the database an important thing to notice here is that the version number just has to be incremented it's an integer that has to be incremented it doesn't matter if you do timestamps or version numbers I personally like version numbers but you can do whatever you like it's very important just to increment them so imagining how a project like this you have the database you have migrations you have client and API those two folders are empty just because you know it's an example project for you to understand that that could be a project structure that you are using so you have a docker compose file of course if there would be API and client there will be more services but for the just this tutorial we are using only one service here which is just starting the database so we can go ahead and run the docker compose command so let's say docker compose up d so as you can see it just started d database of course it will do some more stuff if you just initialize this image if you didn't have it before but in my case I already had it so it it was very quick so if you look at this Locker compost file you can see that there is environmental variables there is port mapping happening there is a volume and there is a built file we can just dive into the build file here is just you know copy base images Postgres and then it's exposing the port which we are mapping to here that's the local port for the database so let's try to log into the database here I'm using the post echo client but you can use whatever client you really like so let's make a new favorite let's call it test DB it doesn't really matter how you call it here it's just a nickname so how's this localhost and seven five five seven is a port because we specified here in the locker compose file then we do docker because it's specified here then password the stalker and then the database is test DB as we specified it right here so let's connect okay and we connected to the database here now let's try to do some migrations so let's first create a migration called create tables so the version is one because it's the first migration we just say create user table dot up SQL and then we have to do a reverse migration so create user table dot down SQL so in up migration we just want to say create table users and then we say name varchar' 50 then in the reverse migration we just have to delete this table because it's reversing the changes so drop table users and then let's create another migration which we'll call add email to users so it will just add and calm email to the user so alter table users and add column email varchar 50 and then we have to do a reverse migration which is add email to users down SQL then we just have to alter table and say drop column email so now we can try to run the migration let's go to the golang migration package here and find the docker command that we want to use in order to run it so here it is so let's say okay so docker run now we have to mount the migrations folder that we just created here we just have to figure out what's the path to it so let's figure out real quick okay that's the path has to be full path in your system so let's say darker run the volume and then we map it to migrations folder inside the container can we say network host then we're using migrate migrate image here if we go to docker hub you can see that there is an image already that you can use for migration - you don't really have to install any CLI tools or whatever on your local machine that's the power of darker here so then you just say path migrations and that's just for the CLI tool to know where the migrations are located inside container and then the database is we have to figure out what's the path to our database so in in the posting over here you can just go in and copy the URL of the database we have to put it into parentheses here because we have to disable as a cell mode as well you just say disable and then you just say that you want to do an up migration also you can do up to which will mean that it will just up the version of the database by - so up migration will run all of the migrations so let's try that okay so we see that create user table and add email to users was executed let's see how it looks like in the database okay so we see that the email column was added so we can start playing around with the database here and by the way I just noticed that I miss the users here in altar table column drop column email so just add this and if we go to the database and we see schema migrations it says that the version of the database is - the false means that it's not dirty if there would be a error with your migration it would set the flag to true and you come to true and then you would have to fix the database and force the version number so let's try to down the version of the database so as you can see it executed the down command for add email to users so let's go to the database here and refresh that and you can see that there is no email column because we down the version by one and now it's at version one if we down it one more time then we see that there is no users columns anymore because it was down by version but by one version again and we can do the same with up command just by one version then we'll just create the user stable without the email column but if we do it one more time the email column appears here and you can do the down command without any flags which will just execute all the down migrations and there will be nothing in the database so now we can actually create a separate container that will host our migrations and will not have to mount it separately in the CLI so let's go back to the project here and create a new folder that's called migrator and then move the migrations to Migrator and then create a new docker file here then we can just say that from migrate migrate which is the base image that we are using from the docker hub here we can just copy our migrations to migrations folder inside the docker container so migrations folder here to the docker container so let's try to build this image here let's go to let's just build it here so build T which will give it a tag migrator and then this directory so we have the image built now we can actually use the image in a way with docker command so we just go to remove the volume because we don't need it anymore we already have it in D in the container and then we just say that we want to use migrator image that we already have in the system and let's just say that we want to do an app migration so as you can see it just worked right here so this way you can have this separate container for just migrating so I hope this video helped you and was useful to you if you liked the video just give it a thumbs up and also if you like this format of videos just subscribe to my channel because I will be releasing a bunch of them in the future so see you next time

Channel: DanielGSC

Views: 7,763

Keywords: golang, golang-migrate, migration, database migration, postgres, postgresql, docker, container, cli, schema migration, schema, database, db, go, migrations, docker compose, golang migrate, golang migrate database, golang database migration, database migrations, golang database, docker golang, golang postgres, go migrations, go migrate, db migrations, db migrate, danielgsc, daniel grek sanchez castellanos

Id: 6_CH8Gx414A

Length: 13min 30sec (810 seconds)

Published: Thu Mar 28 2019