Continuous integration or CI is one important
part of the software development process where a shared code repository is continuously
changing due to new work of a team member being integrated
into it. To ensure the high quality of the code and
reduce potential errors, each integration is usually verified by an
automated build and test process. In this video, we will learn how to setup
that process using Github Action to automatically build and run unit tests
for our simple bank project, which is written in golang and using Postgres
as its main database. Github Action is a service offered by Github that has similar functionality as other CI
tools like Jenkins, Travis, or CircleCI. In order to use Github Action, we should define
a workflow. Workflow is basically an automated procedure
that’s made up of one or more jobs. It can be triggered by 3 different ways: By an event that happens on the Github repository, By setting a repetitive schedule, Or manually clicking on the run workflow button
on the repository UI. To create a workflow, we just need to add
a yaml file to the github workflows folder in our repository. For example, this is a workflow file ci.yml The name of this workflow is build-and-test We can define how it will be triggered using
the “on” keyword. This is an event that wil trigger the workflow whenever a change is pushed to the master
branch. And this is a scheduled trigger that will
run the workflow every 15 minute. Then we define the list of jobs to run in
this section of the workflow yaml file. In order to run the jobs, we must specify
a runner for each of them. A runner is simply a server that listens for
available jobs, and it will run only 1 job at a time. We can use Github hosted runner directly,
or specify our own self-hosted runner. The runners will run the jobs, then report the their progress, logs, and
results back to Github, so we can easily check it on the UI of the
repository. We use the “run-on” keyword to specify
the runner we want to use. In this example workflow, we’re using Github’s hosted runner for
Ubuntu’s latest version. Now let’s talk about Job. A job is a set of steps that will be executed
on the same runner. Normally all jobs in the workflow run in parallel, Except when you have some jobs that depend
on each other, then they will be run serially, one after another The jobs are listed inside the workflow under
the “jobs” keyword. In this example, we have 2 jobs. The first one is build, which has 2 steps:
check out code, and build server. The second job is test, which will run the
tests of the application. Here we use the “needs” keyword to say
that this “test” job depends on the “build” job, so that it can only be run after our application
is successfully built. This test job only has 1 step that runs the
test_server.sh script. Steps are individual tasks that run serially,
one after another within a job. A step can contain 1 or multiple actions. Action is basically a standalone command like the one that run the test_server.sh script
that we’ve seen before. If a step contains multiple actions, they
will be run serially. And an interesting thing about action is that
it can be reused. So if someone has already written a github
action that we need, we can actually use it in our workflow. Let’s take a look at this example. Here we use the “steps” keyword to list
out all steps we want to run in our job. The first step is to check out the code from
Github to our runner machine. To do that, we just use the Github actions
checkout@v2, which has already been written by the Github
action team. The second step is to build our application
server. In this case, we provide our own action, which is simply running the build_server.sh
script that we’ve created in the repository. And that’s it! Now before jumping in to coding, let’s do a
quick summary We can trigger a workflow by 3 ways: event,
scheduled, or manually. A workflow consists of one or multiple jobs. A job is composed of multiple steps, And each step can have 1 or more actions. All jobs inside a workflow normally run in
parallel, unless they depend on each other, then in that case, they run serially. Each job will be run separately by a specific
runner. The runners will report progress, logs, and
results of the jobs back to github. And we can check them directly on Github repository’s
UI. Alright, now let’s learn how to setup a
real workflow for our Golang application so that it can connect to Postgres and run all the unit tests that we’ve written
in previous lectures whenever new changes are pushed to Github. OK, this is our simple bank repository on
Github. Let’s select the Actions tab. Github knows that our project is written mainly
in Go, so it suggest us to setup the workflow for
Go. Let’s click this setup button. As you can see, a new file go.yml
is being created under the folder .github/workflows of our
repository. We can edit this file directly here using
this Github editor. However, I prefer to add the file to our local
repository first, Then edit it locally with visual studio code
before pushing to Github. So let’s open our simple bank project folder
in the terminal I’m gonna create a new folder .github/workflows Then create a new yaml file for our workflow
inside this folder. You can name it whatever you want, just make
sure it has yml extension. For me, I’m just gonna use ci.yml to be
simple. Now let’s open this project in visual studio
code. Here we can see the ci.yml file under .github/workflows
folder. Let’s go back to Github and copy this file
content Then paste it to our ci.yml file First we need to set a name for this workflow,
for example: ci-test This name will be displayed in our Github
repository’s action page. Then we define the events that can trigger
this workflow. Normally we would want to run tests whenever
there’s a change being pushed to the master branch, or when there’s a pull request to merge
into the master branch. There are many other events that you can use, please refer to the github action documentation
to know more about them. Next we’re gonna setup the jobs. In this template that Github provides us,
we have only 1 job Its name is build, and it runs on a ubuntu
runner. There are several steps in this job. The first step is to setup or install Go into
the runner. In this step, we just need to use the existing
Github action called setup-go version 2 And we use the “with” keyword to provide
input parameters to this action. In this case, we can ask it to use a specific
version of Go, such as v1.15 This id field is just a unique identifier
of this step. We might need it if we want to refer to this
step in other context. The second step is to check out the code of
this repository into the runner. To do that, we also reuse an existing action:
checkout version 2. The next step is to get all the dependencies,
or external packages that our project is using In fact, we don’t need this step because
go mod will automatically download missing libraries when we build the application or run the tests. This build step is also not necessary because the application will be built automatically
when we run go test. So the last step is to run our unit tests. We already have a make test command defined
in the Makefile for this purpose, Therefore, all we have to do in this step
is to call make test. Alright, now I think we should
rename this job to test because that’s the main purpose of it. So we’re done with the first basic version
of our CI workflow. It might not work yet because we haven’t
setup the Postgres database, But let’s just push it to Github to see
how it run. First, we run git status to check the status
of our local repository. Here we can see the newly added .github folder. Let’s run git add . to add all new changes
to the list of our commit Then run git commit with a message saying
“init CI workflow” OK, now we run git push origin master to push
this change to Github. Then go back to our repository page and select
Actions tab. Now we can see our ci-test workflow here, And there’s a new run of it for our commit. When we open this run, we can see 1 job in
progress: Test. Now all steps are listed on the right. The setup job, setup Go, and checkout code
steps are finished successfully, And the Test step is still running. We can click on this icon to see the logs. Now the Test step has finished, but it failed. We know that because of the red x icon next
to it. This is expected, because as we’re seeing
in the logs, The code cannot connect to port 5432 of Postgres, since we haven’t set it up in our workflow yet. So let’s do that now. Let’s search for github action postgres, And open this official
github action documentation page. Scroll down a bit. Here in this section, we can see that Postgres is declared as an
external service of this job. Let’s copy this block of code and paste
it to our workflow file. So we use the services keyword to specify
a list of external services that we want to run together with our job. In this case, we only need 1 service, which
is Postgres. And since we’re using Postgres version 12
in our project, Let’s set this docker image name to postgres:12, You can check out available versions and tags
of this Postgres image on Docker Hub. Next we need to set some environment variables
for the credentials to access the database. If you still remember, we’re using this
secret password in our local Postgres container, so let’s set the same value here for our
ci workflow. We’re also using root user to connect to
Postgres, So let’s set this environment variable here
as well. One last thing we need to set is the database
name, which is simple_bank. In the documentation of the Postgres image, It says we can set the default database name
with this POSTGRES_DB environment variable. So let’s copy it, and set it to simple_bank
in our workflow. Now this health check option is used to tell
the runner how to check if Postgres has started successfully
or not, so that it can know when to run the next steps
in the workflow. That’s great because we only want our tests
to be run after Postgres is started, Otherwise, the tests will still fail because
it cannot connect to the database, right? OK, now the Postgres service is defined, But in order for our tests to run successfully, we also need to run db migrations to create
the correct database schema for our application. So let’s define a new step here, after the check out code step. Its name will be “Run migrations” And the only action it needs to do is to run
“make migrateup” Alright, now let’s try to push this new
workflow changes to Github to see what will happen. OK, now in our repository’s Actions page, We can see a new run for our new commit. Let’s open it! Here the Test job is still running. The job is set up successfully. And now it’s initializing the containers. From the logs, we know that it’s still waiting
for Postgres service to be ready. As soon as Postgres is up, all following steps are run immediately. Here we can see a log saying postgres service
is healthy. The Setup Go step is also successful. Then it checkout new the code, Now the migrations step is failing. Let’s find out why. OK, it failed because migrate is not found. We forgot to install the golang-migrate CLI
tool to run the migration. So let’s search for it. Open its Github page, Click on this CLI link, And follow its documentation. There are several options depending on the
OS that you use. We’re using ubuntu for our runner, so I’m gonna copy this curl command to download
a pre-built binary of migrate CLI. Now in the workflow, let’s add a new Step
to Install golang migrate. Then in the run action, let’s paste in the
curl command. We have to set the correct URL for the version
of migrate CLI and the platform we want to use. So let’s click on this “Release downloads” link The latest release is version 4.12.2 And because our ubuntu runner is a linux platform, Let’s copy this linux-amd64 link address, then paste it to our curl command. It will download the zip file, and unzip it
to give us the migrate binary. Now in order for the migrate command to work, we have to move that binary to the /usr/bin
folder. So this step will include more than just 1
curl command. We use this vertical pipe character here to
specify a multi-line command. Now let’s try to run this curl command in
our local machine to see the file name it gives us. It’s migrate.linux-amd64 We have to move it to /usr/bin folder in the
runner machine. So let’s go back to our code, and add this
move command. Now note that only a superuser can change
the content of the /usr/bin folder, So we have to run this command with sudo. Now let’s add 1 more command here: which
migrate Just to check if the migrate CLI binary is
successfully installed and ready to be used in the runner or not. OK let’s remove the binary in our local
machine, commit the new change of our workflow and push it to Github. Now let’s check our repository’s Action page. A new run is started. Let’s wait a bit for it to finish. Now the job is still failing, but this time
it fails at the install golang-migrate step. From the logs, we can say that the binary
file was successfully downloaded. So it might fail because of the move command,
or the which migrate command. OK I know why! That’s because we’re just moving the file
migrate.linux-amd64 to /usr/bin, but we don’t rename it to migrate. So when we run which migrate, it cannot find
any binary with that name. All we have to do now is to add “migrate”
to the end of the move command, So that the binary file is moved to /usr/bin
with a new name: migrate. This will ensure that when the make migrateup
command is run, the correct migrate CLI binary will be used. Alright, let’s add this new change Commit it, And push it to Github. Then go back to our Repository’s Action
page to check the job’s status. It’s still running. Let’s wait a bit. OK, it’s finished now, but still failing. However, this time, the install golang-migrate
step is successful. The step that fails is Run migrations. And the reason is: it still cannot connect
to port 5432 of our Postgres container. Why? We’ve already added postgres to the services
list right? Well, yes! But we haven’t exposed its local port to
the external host yet. That’s why our code still cannot connect
to the port. So let’s go back to the Github Action documentation
page and scroll down a little more to see how to
config port mapping. Here it is! We can use this “ports” keyword to specify
the ports that we want to expose to the external host, just like what we normally do in our docker
compose file. OK now let’s add it to our CI workflow. The default port 5432 of postgres is now available
to our job runner to access. Let’s add this new change, Commit it, And push the change to Github. Hopefully this time it will work. OK let’s open the Action page. The job is still running. Let’s be patient and wait for it a bit. It’s done! All green ticks. So finally our CI-test workflow runs successfully. We can see all the logs of each step. Here’s the logs of our unit tests. After all steps in our workflow are completed, Github do some clean up steps and stop the
containers. And that’s it! We have learned about continuous integration
by writing our first Github Action workflow to run Golang unit tests that need to connect
to an external Postgres service. There are a lot of more things that Github
Action can do. I encourage you to check out its official
documentation to learn more about them. I will put the link in the descriptions. And that brings us to the end of this video. Thanks a lot for watching! I will see you guys in the next lecture!
