Continuos Integration for PHP projects using Jenkins, Composer, PHPUnit, BItBucket and Digital Ocean

hey guys welcome back to your source this is me I'm your host and today we are going to follow up on the previous post where we had a quick introduction to DevOps we've discussed what it was and what problems it was trying to solve but now I'd like to go a little bit further and focus specifically on this section here continuous integration if you understand Serbian feel free to quickly go over it again if you wish basically my goal for today's lesson is to show you how to set up continuous integration for your projects using a handful of tools such as Jenkins primarily but also composer phpunit geet bitbucket and maybe something else we'll see down the line okay so before we start let me show you at the very high level the flow we are going to develop here we have a single developer or a team of developers team size doesn't really matter but the idea is that someone is working on the project and pushing the latest changes to the version control server in our case it's at lessons bitbucket because they allow us to create private repos for free so we never developer pushes kiss code to bid back a triple we'll configure it to immediately trigger a build on our Jenkins server if you're not familiar with Jenkins think of it as a software which installed a server to do some heavy lifting for you such as pulling the latest code installing dependencies running tests and finally deploying the code on your environments of course only in case when everything passed with no errors otherwise it would stop the process and let you know exactly what went wrong so you could go back and fix and the last component is the application server or servers depending on how big your project is this is where Jenkins deploys highly tested code for us and that's it let's start now by creating a new project I thought about simulating a very basic password validator class because many times we are more interested in learning just a workflow rather than in building some complex applications that's why i'll focus on the things like how would you go about setting all this stuff as we said Jenkins for continuous integration composure for dependency management and autoloading phpunit for testing kit and bitbucket for source versioning there are a lot of tricky things that might be said wrong of course to follow along make sure you have installed composer it's a very simple process just go to official website and read short instructions for your system I can simply verify that composer is running for me by typing composer in command line you see it outputs me a list of available options which is good alright once you have composer installed you have access to a huge amount of packages created by a community including PHP unit which will use for running our tests so if I go to packages dot org a single place where all composer packages are hosted in search for PHP unit sure enough it's available so to get us started I'll create a new repository in big bucket and I assume you already have your account so we should be on board here let's give it the name of vs CI short of view source continuous integration and everything else is already set as it should be it's private it's good I don't really care about issue checking and wiki for this project so we can move forward all right it's created for us and we got instructions for the next steps since I'm already into my project folder I'll do git init and copy the second line to add remote repo okay that was simple now I'll add the project to my editor and the very first file I'm going to create is composer that Jason this is where we define dependencies for the project I only need PHP unit so within here I can type require the block this is for all dependencies that shouldn't be installed on production and if you're not sure what the latest stable version of PHP unit is you can always go back to packages to check seems like it's for 3.4 so let's copy this and paste into our required air block that's all we have to do now we can go back to command line and run composer install the simple command is going to download PHP unit and all its dependencies into vendor folder for us because by default composer install command assumes you want to include dev dependencies this might be a little bit slow so let's get this down and continue once it's completed it's done you can see that I got vendor directory with PHP unit and some other packages that PHP unit depends on here we see composer log file which is also important because after installing dependencies composer writes the list of the exact versions it installed into this file so that anyone who sets up the project whether it's another developer or jenkees machine will download the exact same version of the dependencies that's why we'll put this whole vendor folder in to get ignore because there's no needs to add all of that code to the repository it's enough to have composer JSON and composer lock only and let composer to do its magic I'll create a new file save it as kit ignore and quickly add vendor folder in it ok we are good and ready for the very first and initial commit if you check its status we see three files are injected there's no vendor folder so we can add all of them and commit get add all get commit I'll put the message app in it and here we go ok good now let's quickly create some simple code to work with as we said it's going to be incredibly simple just a single validate password class so let's create a test folder first following the rules of TDD and the new file in it we'll call it validate password test dot PHP so let's see how might we do this well say class validate password test and it's going to extend PHP unit framework test case so let's begin with a very simple bit of a behavior this class should here for example functionality to validate the length of the password so public function test valid length we're testing validate password class so let's assume it exists so we can create a new instance here so well pass equals new validate password again this still doesn't exist but we'll fix it shortly and then we have access to a large amount of PHP unit functions so this assert false and we expect valid length this is method in our validator class to return false when we provide for charters password because apparently it's too short now we can implement many tests here but it's not what we want to focus on let's rather keep it very simple one thing that I do want to call out though as you see the first line here creates validate password class instance it's okay in our case because we have only one test but if you don't want to instantiate it over and over in every test you write you can leverage PHP units fixtures and specifically a method called setup I encourage you to visit PHP documentation and have a read alright as you might imagine this is not quite ready yet but let's try to run PHP unit I'll go back to my terminal and type dot this means start from the current directory then vendor bin phpunit because this is where phpunit binary lives and of course we need to specify what to test so as a second argument I'll pass dot tests validate password test not PHP all right it's telling us it tried to run the test but it couldn't find validate password class so let's see how might we go about fixing that within our project I'll create source folder first and validate password dot PHP file so class validate password will say that password length should be between six and twenty characters so we need a constant min length to be six another constant max length to be twenty okay now we'll say public function valid length it's going to accept the password and we should only check whether or not it passes required length interval first let's say pass relent in a variable path length equals string length and as argument it's accepts password and then we can easily do a check when we try now it still can't find validate password class and of course it can't as you probably guess we created a class but we hand referenced it inside validate password test there's nothing signalling that we are going to auto load the class so it still can't find it so we can require it within here at the top but you know what that would be a bit of dated practice instead we'll take advantage of Auto loading using composer because it works out of the box you don't need to do absolutely nothing it's already there that's another nice feature of composer so the only thing we need to update is compose JSON file because this is where we can specify what to autoload are we doing PSR autoload or PSR 4 or maybe class Maps personally I got used to PSR Oh although psr-4 might be a better choice these days nonetheless I think everything will do for us in this eel example so we'll need to create autoload block and tell composer to follow PSR Oh standard and the object that comes after represents namespace I'll leave it empty our code lives in the global namespace and this is the physical part where our code lives which is source the very last step is to run composer damp autoload which will regenerate the list of all classes that can be required by our project alright we should be all set by now if you run our tests sure enough it's able to find the class it needs and our test is passing let me commit latest changes I'll do git status I'll add all and finally I'll do git commit let's validate password class and test and finally let's push it to remote repo because from now on we'll mostly work in the cloud for our hosting needs will turn to digitalocean so both jenkins instance and our application will be hosted on digital ocean now when you first go to digital ocean comm you'll see a page similar to this one digital ocean is a cloud hosting much simpler than Amazon Web Services for example you start up your own instance your own droplet as they call it and droplet starts at $5 per month or if you prefer dot o7 per hour if you have any question about pricing head over to their pricing page you should be able to find useful information all right let's go ahead what you're going to do is to click on signup button you'll be asked to fill in your billing information it's a really simple process and once you're fully signed in you won't have any droplets created it will guide you through the process you'll be given a create button and once you go ahead and click on create button you'll be able to enter your host name this is going to be our application server so let me call it simply main app next you can select the size you need typically people start off with one CPU 20 gigabytes SSD drive usually this is enough to run pretty simple website and of course you can scale up at any time so I'm going to select the most basic size then we are asked to select the region which is closest to us you see they offer us five different locations closest to me is Amsterdam so let's choose that one next we have available options to choose from but I'm going straight to image selections they have five different choices here all of them are UNIX based I'll select the latest boon to 64-bit now because we are going to use PHP I'll make sure to go to applications tab and select a lamp on a bun tool that's all there is to it I can go ahead and click on create droplet usually it takes just a minute I'm going to pause this video so you don't need to wait for the droplet to create itself and start the session when the droplet is created all right so the droplet finished up creating itself I was redirected to droplet panel where I have some information about it then some options to manage the instance but let's move forward and login to the server with SSH here's the droplets IP address digital ocean send me an email with the password for the root user I have pasted it in my editor so let's use Patty for Windows for SSH axis so log in as root password copy the password that digital sent now once you get into it will immediately prompt you to change the password for the root user so let me go ahead and first right click to paste the old password from the clipboard and then I will enter my new password I'll retype it and nice I'm logged in into my bun to drop it here I can type clear to clear this view up and then I can begin navigating through my droplet so the first thing I'd like to do is to go directly to bar and then web folder and there's actually HTML folder within here this is where you typically put your web application files ok let's get our app onto the machine thanks to digital oceans application images our machine comes with lamp so we don't need to worry about PHP and Apache installation but we still need to install kit which should be as simple as apt-get install kit once gate is installed we can use it to get our app from bitbucket we only need to copy the clone URL from the project bitbucket page and then run this in our server it's asking me for password alright that was fast enough we see it got cloned I'll do LS - la now before we can dig into composer and installing our dependencies of course we need to install composer I prepare the command we can see here it's just a matter of copying and pasting this to the terminal window now you see that it installed composer dot far file think of this as PHP archive file so if we want to run composer we can type PHP composer dot far and we get the output however it's always a good practice to enable composer globally to do that all we need to do is to move this file to bin directory so I'm going to move composer dot far to my local bin directory which is user local bin and composer and that's it so now I should be able to type only composer and get the exact same output now to demonstrate again how to download all the dependencies we should first CD into our project and then only type composer install but this time with no dev option now you might be asking yourself why do we need this node eruption and the reason for this is well this is the application server we don't need to run test here instead it Jenkins responsibility to run all the tests and deploy highly tested code to the application server so when we run this command we see there was nothing new to install but still it generated auto load file which will bring autoloading over to us very simple and useful all right we can move forward we'll start with the installation process for Jenkins the first step of course would be to set up another digitalocean droplet for Jenkins we'll repeat the process from the previous time so I'll go ahead switch to digital ocean tab and click on create a droplet again bunch of options I'm going to call this one Jenkins choose the cheapest plan select Amsterdam images the bun tool which is fine and let's make sure that we have lamps set up by default that's all I'll give it the moment or so and it will install all right here you can see that I have my Jenkins VPS up and running digitalocean has sent me credentials for ssh so let's go ahead and log into it I'll start another instance of my Pattie pays the host here okay username root I'll copy my password and right click into patty again it's asking us to change default password so this is the old password I'm going to right-click again and I'll type my new password twice and we're in our Jenkins server excellent now before we can use Jenkins we need of course to install it we are going to get everything set up from scratch it should only take a handful of minutes thankfully digitalocean has published this nice guideline so we only need to copy these four commands into our command line usually we install stuff on debian based systems using apt-get but we need to add a new repository first to make it able to pull Jenkins installation so it's done in those two commands the first step is to add Jenkins key then we'll update our source list and include Jenkins repo now before we can actually run Jenkins installer we need to do apt-get update and there's the very last step we can type apt-get install Jenkins the solder is to it as always I'll take advantage of screen casting magic pause the video now and continue once it gets installed okay it's finished by default Jenkins listens on port 8080 so if I switch to my browser and paste the IP port 8080 yes we see Jenkins welcome screen apparently it's completely insecure and everyone on the web can access it but it's not concerning in this case at all I'll kill the server as soon as I'm done with this video for real projects though I highly recommend you to go to many Jenkins and enable user authentication it will ask you to create a user and once you are done it will prompt you with login screen before you can manage Enki's I'll leave it to you let's focus on the things that really matters we know the purpose of Jenkins is to pull our project from big bucket and run its tests so we need to install kit and composer but also we are going to need big buckets plugin for Jenkins so let's add missing pieces quickly and move very fast since we've already covered it while setting up our observer so to install git I'll quickly type apt-get install kit for composer it's two steps and none the second part is to install bitbucket plugin for Jenkins in order to do that I'll go to many Jenkins then manage plugins go to available tab sometimes when you install Jenkins you will notice that the list of available plugins is empty if that is the case from advanced tab on the manage plugins page click on check now to forcely check for the new updates once that is done you should see the list of plugins in my case though everything is fine I see the list let me search for bitbucket now click on download now and install after restart this will take some time so let's fast forward now we are ready to create our first job in Jenkins that connects to our bitbucket project does automate builds and display some results I'll click on this link here to get us started our job will build application server so let's give it a name of vs CI - Build and I'll select freestyle project in most cases you will end up with this option great as you see we have project name description that help us explain what's the project about we see a bunch of other parameters the important one to fill out is this section called source code management this is important for Jenkins to know how to communicate with my source code repository you see we have here some choices CVS kit subversion of course we have none sometimes we don't need to have our job communicate with our source code ripples in this case we do so I'll select git and I'll provide SSH URL to my project now this part here branches to build is interesting the top ssin is provided by big bucket plugin we have installed and because it's so useful and important at the end I'd like to break it down a bit you need to understand that bitbucket will ping Jenkins server on every single push whether it's on master branch or developer or any feature branch you can't really avoid it that's how bit back it's post hook works that means every single time a developer pushes his code request comes to our Jenkins server which will trigger the build if you think about it it's pretty useless and dollar whelming right we certainly don't want to do a build when someone commits his work to a future branch but with this plug-in installed we'll hear it parts big buckets post requests for us and run the built only when it's dealing with master branch technically that happens when you merge any of our future branch into master next section we care about is build triggers I'll go ahead and check build when a change is pushed to big bucket this option is also given by big bucket plugin hopefully it makes sense why it was necessary to install it this won't work unless we set the post who can be back at repository tool we'll do it in a minute and finally this is where you add your build steps personally I use execute shell option in almost every job I create basically that instructs Jenkins to execute shell commands for me which is pretty handy so what we want to execute well at the moment big bucket plugin will pull the latest code for us so next thing you would do in terminal is run composer install and run all tests so I'll go ahead and write this is where our PHP unit executable is and this is a folder where our tests live now this still won't work first we already said that we need to instruct bitbucket to send a post request to Jenkins on every push but you can also see it can't connect to bitbucket and of course that's the case because our repo is said to be private when Jenkins tries to clone the repo via SSH it doesn't know how to identify itself so we need to set SSH key pair and let's do it first I'll go ahead and switch to my terminal when Jenkins installs it creates a new user called Jenkins of course it executes all commands with the user so we need to generate our key with the Jenkins user so let's switch to it I'll do su Jenkins now navigate to home directory so let's look and see what's in the home directory usually a server keeps its key pairs in to dot SSH folder apparently that one doesn't exist at the moment but let's see how might we solve that if you just type SSH - Keegan we are making key pair but it will be DSA by default RSA is actually a little bit faster so let's change the type and set it RSA it's generating key pairing two separate files that link to each other this is where it's going to be saved which is fine I'll hit enter now it's asking for passphrase which will leave empty by hitting Enter since the whole point of this is to avoid passwords especially because we'll have automated jobs which have to be authenticated but they can't really type passwords I'll type enter again and it's done if you do LS - la we see that SSH folder and inside of it both private and public keys now what we need to do is to copy the content of this public key and paste it into bit bucket so let's take a look at the public key I'll type cat ID - RSA and copy everything from here input T you can copy by pressing enter I need to switch back to my repo and then I'll go to settings deployment keys and as you might imagine add key the label field is to help you remember what this key was about the key is important so let's paste everything from clipboard ok we should be all set go ahead and press add key since we are here let me also register a postcode for this repo I'll go to hooks tab and search for post okay the URL needs to be the domain of our Jenkins server I'll grab IP since we don't have domain this is something I found in bitbucket plugin documentation so with this setting in place when a user makes a push against the repository bitbucket makes the post request to the URL we provide here the body of post request contains information about repository branch a list of recent commits the user that made the push etc that's how the plugin is able to decide whether or not to run the build it simply checks the branch and if it matches with master it's going to proceed okay we are almost done there's only one very last thing we need to fix you see that annoying error still shows up well the reason for this lays in the way how SSH protocol works when Jenkins tries to establish SSH connection to bitbucket it needs to know it's connecting to write SSH server that's why we need to run this command which will basically update known hosts with big buckets public key alt IPS I'm sure everything is fine and safe and that's it now if we wait for some time this message will go away but you can simply save the project and go back to it and sure enough nowhere so far if you run the build at this time it won't fail but let's think what's this job going to do it will pull the light this code it's going to run composer install and run unit tests and that's all ice we're testing our changes but what about deploying these changes to the app server well you remember our flow from the beginning if these two commands pass with no airs why don't we simply all application files over to our application server will use our sink a very handy command which offers a large number of options that control every aspect of this behavior will pass verbose recursive this Z stands for compressing the data each for human readable output and finally 'flag to pass options to ssh command if we provide this we're basically instructing our Jenkins server to always add new host keys to the known host file so we're bypassing the exact same situation we had with Jenkins at build bucket and this is a well-known way you might also imagine that we don't want to transfer vendor folder since we use composer so that's why we want to make sure we exclude it plus flow parameters our source folder for the files that are going to be transferred and of course destination folder as a source folder I'll put only a dot which translates to the current job folder and for destination folder output it needs to connect as route at IP address of the server followed by the path to our application which is var web HTML and vs CI at the end at this point it's synchronized source and destination now what we need Jenkins to do next is to SSH to the application server then CD into application directory and run composer install with no their flag which skip installing packages listed in required dev sections we never need to run multiple commands within one SSH session the cleanest way is to surround it with hair script like this this is so called hair script and ok I think we should be all set on Jenkins part now the very very last thing we need to do is to create a file called authorized keys on the main app server and paste the Jenkins server public key that way whenever it tries to SSH into the server the observer will allow it since it has its public key in authorized keys file I hear my Jenkins public key here again it's the same one I put into bitbucket and I'll need to switch back to my main app terminal I'm logged in as root here so I can type CD tilde which will basically open routes home directory within here I can go to dot SSH folder and do LS - la since authorized Keys file is already there I can edit it use it Nano and I can simply paste the key and save the file and that's it we had a couple of things to configure but hopefully everything is working as we would expect I'll switch to Jenkins and for the very first time let me save the configuration first and then I'll manually run the build just to see how it works so to do that I can simply click on build but before that let me enable auto refresh at the upper right corner you see it's done this blue circle tells us the job has finished successfully typically it's red when something's goes wrong now if you're interested to see the entire output from the console you can click on console output you see the first part is related to fetching the code from our repo it's done by bitbucket plugin next we see logs coming from PHP units so everything fine on that side one test passed and finally we see our sync output it's transferring all get files hmm which is interesting I completely forgot to exclude them however I'm not sure whether it's okay to leave them or exclude them what would be the best practice honestly I don't see the reason why would we need get files on the application server on the other hand though we are cloning the whole repo so it makes sense to include them as well for now let's leave with these settings and alidade below if I find by the way and this last bit here is from Jenkins when it SS aged into the observer at the very end we see that Jenkins finished with success excellent to finish up with this lesson let's see what will happen with two aged cases the first scenario we're interested in is to see what will happen if we make a commit in a future branch and not on master Jenkins shouldn't start a build in that case the second interesting scenario would be to commit the test which fails on master branch in that case though Jenkins should run the build but since our test is failing it should not deploy anything to our observer let's start with the first one if we do get branch it says I am on the master branch let's create a new test branch and switch to it okay it created a branch and switch to it what I'm going to do is to add a few empty lines into this validate password test class and now if I run git status it's modified so let's commit and push this branch to a remote git commit dash am adds feature X git push origin test okay nice we see a new branch was pushed to remote so let's quickly go to Jenkins and check if push trigger the build I'll click on our job and we can verify it indeed didn't run the build let's now try the second case I'm still working with the same file so let's add the valid length for password for example 1 2 3 4 5 6 7 8 9 this will obviously pass but we're expecting false in this test now commit this get commit - am adds failing test just for fun git push origin test ok it went to remote which is fine at this point we know it's not triggering the build but let's merge this future work into master I'll do get checkout master and then I'll do git merge test branch with - no ffs I need to pass no pass for reading option and of course how to get push origin master okay nice we'll wait a couple of seconds and you see the build here started let's give it a moment sure enough it's red if you click on build information you see it's very clear validate password test test valid land failed asserting that true is false and at the end finished with failure which is all good ok that will do it for this lesson this project will be up on bitbucket I'll unlock it for the public so have a look through it see if you can extend it try out some different approaches for example try to set up Jenkins running this job periodically and not on every push there's a lot of options the important thing though is that you dig in and experiment with it if you need digitalocean account you can use this referral link and earn ten dollars for free or if you are student there's a nice promo pack at this time coming from github which will give you $100 credit thanks for your time my name is Milan and please reach out to me if you have some comments on this cm

Channel: viewsource rs

Views: 49,057

Rating: 4.9674354 out of 5

Keywords: Jenkins, Continuous Integration (Software Genre), PHPUnit, Digital Ocean, Dependency Management, DevOps

Id: TiVIb0KRIZw

Length: 39min 50sec (2390 seconds)

Published: Sun Oct 26 2014