Creative Branching Models for Multiple Release Streams - Atlassian Summit 2015

hi everybody if you do not mind if you could please scoot in towards the middle of the row it makes it a lot easier for people who are coming and post-lunch little late to find a spot get cozy with your neighbor everybody's great here hopefully you all enjoyed lunch awesome thank you and please also put your phones on silent and so it does not distract the speaker but thank you for showing up this is the build tract for people who have not been here yet today my name is Justine I'm the emcee if you want to hang out with me all day long stay in this room so up next we have Matt Shelton from nuance communications who's actually been with them since 2005 working across technical support for enterprise healthcare platforms and he wants Communications actually just switched from subversion to get so matt is going to talk to you a bit about how they chose a git branching model and what they went through with this process so without further ado Matt thank you hi I was loud oh great thank you all right thanks Justine for the intro so I'm an engineering manager at nuance communications and we're all about reinventing the relationship between people and technology and in our healthcare division we are focused on improving clinical documentation helping doctors create better documents deliver better patient care and helping health care institutions achieve greater financial integrity and compliance my team builds a platform of web services to offer natural language processing technologies to other healthcare software vendors so we're primarily a business-to-business sort of arrangement like I said our primary delivery our web services in a reliable SDK that our consumers some of whom are also lasting customers can use to deliver solutions to hospitals so my team's roughly about a dozen people to different parts of the world and we recently moved from subversion to get so I want to talk about that process why we did it why we picked it how we picked a branching model and some of the problems we ran into so before I get started how many people here are in a software engineering role where you're wait where you have to worry about dependency management in some technology that are you using awesome how many of you love maven as much as I do awesome great you picked the right session I hope so I want to talk about my team's migration to get from subversion and why we picked it so Nikola pallucci he's a developer advocate here at Atlassian conference a few months ago had this quote which I really appreciate it says those of you who have a hard time reading it says if you don't like speed being productive and more reliable coding practices then you shouldn't use get a little tongue-in-cheek but it really hit home for me because my team really values productivity and we really value speed and being able to deliver solutions to our customers and our part as quickly as possible we value being productive we value reliable coding practices in the low defect rate and so we wanted to switch from a situation where we were asking ourselves questions that we shouldn't still be asking ourselves after being around for half of a decade things like I had this code over here but I need to release this thing but it's not ready so I got to pull it out release it push it back in or maybe we should make a branch what's a branch I'm not really comfortable with that idea and how we merge it at the end or we have this external dependency and it's two different versions across three of our vertical products and we need to do feature development in all of them and oh this one needs to get upgraded it was ridiculous we spent a ton of time we wasted a ton of time in these sorts of conversations so we wanted to make a change so in evaluating this migration we wanted three things out of it the first was we wanted to be able to isolate our development so the developers could have free space free reign over what they were building and had the ability to create and not worry about impacting our release plans or what QA was testing except when they wanted to QA testing we wanted automation we didn't want them to have to do anything manually if they didn't have to and we wanted deep integration with the atlast and toolset we use pretty much everything that Atlassian provides we have JIRA server and confluence bitbucket server bamboo HipChat fisheye crucible clover actually I think that is everything that Alessi makes so we wanted deep integration with the whole stack and we wanted to be able to use it as seamlessly as possible also like I said an ammunition I'm an engineering manager so I wanted my team to be happy that's a big metric for me so when we started to look at get we needed to evaluate what branching model we were going to use and if you're not familiar with the term of a branching model here's what it means so we get branching is cheap it's easy and it's really painless and emerging which is never really easy anywhere is a little bit easier and yet than it is in some other source control systems it's very powerful and when looking at how we wanted to make use of that branch branching concepts and capabilities and get we asked ourselves a few questions we asked ourselves how distributed we were or were going to be because when we started this process we had one team on the east coast of the US of about six engineers and today we have a team that same team and a brand new team of the same size in India and we knew they were coming so we needed to make sure that what we were going to do was going to fit that model we also thought about how often we release or how often we wanted to be able to release and we needed to make sure that anything we picked was going to empower that idea we thought about do we like our existing methodology or do we want to do something different if you have a methodology that works is there a way to take the capabilities that get provides in terms of painless branching and apply that to that methodology to empower your team so a branching model is how you make use of branching to be effective with git for source control if this is new to you let me run through a few branching models the first one is very simple there are no branches you have a master everybody works in master one commit after another you're released from master you test for master your team breaks things and master one branch this is good if all you want to do is get used to the commands but maybe you need something a little more complicated maybe you need just some simple feature branching or a developer can create a branch for doing their feature work build test break fix and then have that code either pulled or pushed into master when it's ready to go out this is a great way for teaming to we're getting used to branching concepts to get started and it's really great for teams who have very strict processes around code review around how the code is merged to just introduce this first to get used to the idea let's say you are a continuous delivery shop it's a branching model for you looks kind of like this you have a release stream of what's in production you have a staging branch for what's next to go into production this is where all the testing your performance testing all of your your QA processes are taking place and feature branches off of staging generally you would apply some sort of continuous delivery model at the end to get what's in release to your customers and what's in staging is some sort of testing infrastructure or maybe you have a problem like what we have where we have the need to have long-running versions in production and we need to fix things in old versions and new versions you might need a model like this where you have a master branch and feature branching for your feature work and long-running branches for releases that are in the field bug-fix branches for fixing bugs and you might use some sort of merge process to merge those changes back up all the way through your old releases and into master to ensure that master always has the latest code this is actually the branching model that the bitbucket server team uses if you wanted to take a look at how they use it has a great site there we looked at this we considered this for a long time and in the end we found that though it was going to cover most of our use cases it wasn't quite what the team needed so what do we pick onto that in a minute when you were trying to figure out what branching model to use and even how to use git I really want to encourage you to not make this decision in a vacuum there are a lot of great resources out there Atlassian has a fantastic get microsite at Atlassian comm / git I didn't think I could fit everything into this 30 minute presentation so we wrote a three-part blog a blog series it's a last name comm slash get slash articles so if you want some more information and I'm not able to answer your question later take a look there and be happy to help out later as well there was a presentation two years ago at summer 2013 that goes into some of the workflows as well and one other link that I will speak that's not on screen last seen recently published a continuous delivery microsite at last seen calm slash continuous - delivery and it is a fantastic treasure trove of knowledge about how to make use of branching models and other technologies together that Atlassian provides to empower a continuous delivery workflow so check those out so we picked get flow hey that advance for me how nice all right let's go back one slide so we chose get flow get flow is introduced by Vincent Driessen in a blog post and there's a short link to that if you want to take a look at it at some point he also wrote a set of get extensions that flow and it looks like this we like colors on our team so we picked something colorful so get flow starts off with a master branch starts off with a master branch and all development work happens off of a develop branch that is branched off of master master is always clean there are no releases off of master master doesn't really build it sits there as a record keeping location for tags and all your development work happens off of master that's where all the fun takes place but if you find a a bug with a version that's in production we have a concept of a hotfix branch ha fix is a branch that branches right off of master work is done work is tested and then when it's ready to go out the door two things happen first it's pushed up to master and tagged to record that that release happened and then it is pushed down to develop so that as develop program development progresses it's always up to date in the develop branch features come off of develop for each feature that's implemented you branch off a development alpha develop and you continuously fetch new fetch new updates from develop to keep that feature branch up-to-date this is where developers get to play they get to break things and then when it's ready it can either be pushed or pulled via pull request into the develop branch and once your code and develop is ready to go out the door once you've said we want to have a release now you create a release branch from develop release doesn't exist until this point final testing is done here you validate this is what we want to put out into the field and then again two things happen those changes are pushed up into master and tagged and then any changes that had to take place after release was created are pulled down and developed this solved about 99% of our concerns into of branching again well like I alluded to earlier we need long-running support branches for versions that are in the field so we introduced a sixth branch which is called support support differs from hotfix in a couple of ways first of all it's yellow it differs from hotfix for one primary reason and that is that these are changes that aren't off of the latest version that's in that's in master these are from previous version I even illustrated with a minor version back a major version back excuse me to illustrate that these are these are old things but we needed new feature development in those old versions and these are also changes that are hard to merge we might have to cherry-pick we might have to hand merge we might have to merge we might have to not merge at all and reimplemented support this is a great question why do we need this level of complexity well we have a lot of partners we work with a lot of other a lot of other software vendors and our products don't have a consistent release schedule so let me give you an example we have a partner who is working on an integration with us for one product they have a beta customer in the field it's an on-premise deployment for that customer and we're a hosted solution they're running with one version of our software that customer says we will not take this product live unless you do X so the partner comes back to us and said we really need X and we need it on the version you have out here let me say well we have this other version it's the next one we'll do it there no no no no your next version has this other thing and we don't want that and see this sounds eerily reminiscent of the problem we had before we were trying to solve and we didn't want to repeat that so we needed a way to do feature development on older versions and then choose how to bring those new features into the latest and greatest version that might be going out in the field the other is that we have a couple dependency with an internal an internal library for working with our NLP technologies and as they implement new features we might not want them in each one of the product offerings that we have so we need to be able to pick and choose which of those product verticals make use of new versions so we again need to be able to have old versions receive new features also we use maven for dependency management maven is great name it's also hard if you've never heard of maven and and even if you're not a java sha but this problem applies to non Java development environments as well if you're using NPM for node you're using NuGet for dotnet whatever you have in place the need to resolve dependencies that you might that might be external or might be internal is universal across many software engineering environments but if you've never heard of maven and you'd like to know what it is you can look it up no maven is a build automation tool it is primarily used for Java projects its purpose is both to describe how a software project is built and also to describe its dependencies it has a concept of a build lifecycle with a bunch of stages or goals and we produce snapshot builds for things that are still in development and then various flavors of release builds as we release there are other things out there whose Gradle there's Ivy but we've been using David for a long time we're really happy with what it gives us in terms of its feature set so that's what we're using and dependent dependencies are a huge challenge for us so this is what our project structure sort of looks like we've got a common package for common stuff and we have four major product verticals and each one of them is very similar to one another but has a different workflow a different customer base and we produce artifacts as a result of a build process to go into a tool called artifactory Pro for storing both build artifacts produced by bamboo and also external dependencies it's great for caching things like we're using log4j for logging we don't want to pull that down from the Commons every single time we do a build it gets persisted on our local network and artifactory pro so when we're doing active product development we're usually pointing to the snapshot building of the in progress builds for common as we are doing product work if you're trying to take notes by the way these are gonna get posted within a couple of days so relax because I didn't want you to have to transcribe this slide alright so for each one of the branches that we have that has a slightly different location in artifactory where we toss builds out of bamboo for the branches where we're doing active development there builds generally go into a snapshot repository and then when we released a QA there's a QA repository so QA is one place to look for everything they've ever tested and then from the branches that actually release to the field they go into a staging repository for release candidates and then into a release drop posit ori for the things that we've said can actually go out the door so what's the big deal so all of this though very wonderful for us and it actually really makes our development process work fantastically much of the time it runs headfirst into a wall once we start branching for two huge reasons one is that when you produced artifacts via bamboo and store them in art a factory with a version number and you have two developers working on the same version in parallel feature branches you start to have collisions between the artifacts that are produced and then managing those version numbers during the during the branching process changing them on the way in and out of branching is actually a huge pain let me illustrate that for you developer a we're going to call her Amy Amy does some product work it is for product a consistent theme and it has dependencies in our common project before she started working there was already one artifact in our artifact repository of this version so she branches for a feature branch and does some work in common and then working her in her product build everything's green all of our unit tests paths she pushes up to bitbucket server for her feature work and bamboo builds common and creates a new artifact in artifactory unit tests pass everyone's happy green yay bill is working on a feature in project B I'm very clever with this consistent naming guys you're gonna love it all right so he does the exact same set of steps produces the exact same named artifact and I hope at this point it's clear to you what the problem is because then when Amy realizes she missed one feature in her product project it implements that and all of her unit tests pass and she pushes that up to bitbucket server and all of her tests fail we have a problem because that's not the common build she thought she had we also run into a version pollution problem when we try to fix this because let's say we say all right well start with two window snapshot and then I'll name it based on the branch and we'll change it or maybe we won't change it and we'll push it back up to develop and then it's this and the next developer changes it again and then it's that and it's something else and somehow we're producing builds with different names in different places and it gets really messy but there have to be ways to automate this right we're all software developers here it's got to be some ways to script some stuff and there are there are some great techniques out there we thought about ways to modify this on the branching and merging operations so that again our developers could not have to do a whole lot of manual steps so we'll go ahead and change it when we branch right after we branch because you can't change it when you branch and then we'll change it after we would get it merged back into development we can automate this we can use there's a Java implementation of get flow that last seen actually builds called J get flow that does this for us we can do this with just scripting maven version set version that can do this it has a couple of problems and it's those two commits cuz they're wrong and they produce artifacts that are from the wrong place and there's a tiny window of opportunity in there for us to override someone else's build and to produce code from the wrong place we thought about then just doing this on branch only and you know there's a presentation tomorrow and Nikola Lucchese doing a presentation on merge strategies tomorrow I encourage you to check it out there's a really interesting merge technique called there's versus ours where you pick which version you want to merge and say this is the actual change this this change is the one that's right and we figured out that right after we branch we could merge with develop and say choose our change take ours and then when we pulled the change back up and developed it ignores the fact that the version number was different between the two branches and we have we have happiness developed version never needed to change and the feature was by itself we still have one bad build and it's still a manual operation and developers are not happy we were looking at this the wrong way we needed to think about ways to actually isolate the build artifacts because it wasn't broken and maven though I'd like to blame it for a lot of things wasn't broken and my team wasn't broken but the problem was that we were over writing build so we needed to figure out how to isolate them and I mentioned we were using artifactory Pro we have this lovely list of places where restoring artifacts and the problem was our feature build so we thought let's get clever let's use a maven property to put all of our feature builds in in a custom repository pattern based on the get branch we wrote a maven plug-in that pulls the get branch out of the branch of the the URL for the branch tosses it into our repository URL and we were successful doesn't that sound simple I lied I'm sorry nothing is simple because maven is hard fans alright so the developer who was working on our maven plug-in sent me that gift and HipChat and I was like that's going right into the presentation so it turns out that when you write a plug-in for maven and attach it to the earliest stage in the lifecycle it can't touch the URLs for your repositories they've already been set they're inaccessible so though we were actually putting in a different repository for features it wasn't different across features it was only there's only the one and we were sad but it wasn't the end because maven actually has this other way of extending its functionality called extensions and extensions though horribly horribly documented do actually let you modify that repository URL if you can use a combination of reflection and figuring out the right API to use setting it in memory by writing your own parser to refill - it's crazy but it works so we were able to write an extension that when you are working in a branch all of your artifacts go into a branch name location artifactory you never step on people and we don't touch the version the version stays exactly the same between feature branches and develop I gave this presentation at nuance and I got all these hands up to said we want to do what you're doing we want your extension and I said great you're all friends you're all here I'd love to give it to you and I would love to say to you today that I would be happy to share that with you but our legal department never got back to me so once I find out if I can I will be happy to publish this so that we can make it more available for other maven fans out there who want to use get flow and if that doesn't happen I will write a set of instructions to do it yourself because everyone should be happy together so at the beginning I Illustrated that we had a problem we had a problem with subversion and a problem with the way we wanted to do software development and not being able to do it so I had to ask myself after doing all this work which started nine months ago were we successful is my team happy now because we have this a very colorful set of lines that illustrates how we do software development how we do branching and did this solve all of our problems where were you successful so there were some things that came out of this that we're awesome and we love it and there are some things that came out of it they are a little inefficient and so I'll talk about both of them first of all branches are super super easy remember we want a deep integration with the toolset and we get that we can go into JIRA and click on the link that says create branch and we create branches and we don't have to do anything else except pull and check it out and start working but we don't have to do anything else to get the branch created everybody loves this I was I'm a nuance is a Technical Account Manager customer is a part of the enterprise offerings here and first a quick shout out to our Technical Account Manager he's awesome his name's Xavier he's great very helpful in this process and one of the things that customers have talked about as a metric is how quickly can you get to writing software from the time you know you need to start and the time to get to branch creations start working here is seconds that's huge for us huge for me as a manager because I want my team to be as productive as possible and these tools work incredibly well together you don't have to be using get flow or a really complicated branch strategy which I hope none of you implement a really complicated branch strategy but you don't have to be doing anything fancy to get the benefits out of all these tools working well together you can see all of your feature builds in one plan in bamboo bamboo can detect those branches and automatically spin up a branch plan for them so you immediately have continuous integration every time one of your feature branches is created our team can still do code reviews and crucible which is what we prefer to use for code review and we can get alerts like crazy and HipChat whenever every single time we get a feature branch period or it fails we had to turn some of those off there were too many of them in JIRA ties all this together in one place I can look at the development area in JIRA for each issue and see all of its branches all the commits its code review status has it been pulled back in to develop it's wonderful we love it and pull requests I'm a huge fan of pull requests I mentioned we spun up an offshore team they were super easy super easy eager to get started and that was good we want them to be eager to get started we didn't want them to commit things to develop right away we wanted them to get used to our infrastructure and so having a process in place where we could say no you can't push your code in to develop because you actually haven't finished the design process yet was very helpful it's also an excellent gate for developers who are a little click happy or people who didn't respond to all the comments that were that came out of code review and it means that whoever reviewed the code and said yes this is this is ready to go can then do a sanity check right before it goes back and develop to say yeah that's the code review we'd love that feature there are a couple of things that are slowing us down right now that we're looking at ways of speeding up so I mentioned our dependencies a while ago for major projects in a common we have some features that need to go across all products sometimes that means we have do that branching action between 1 and 7 times if we have some common dependencies in the support branch and some and develop same for the products it's a lot of clicking and the branch name matters because that extension depends on that branch name being the same across each one of the branches in which you're working so we have to make sure that every time we do one of those branches it's named properly that takes a couple extra seconds these aren't huge inefficiencies it's just a matter of seconds and the last is that bamboo because we're using the blanch the branch plan concept when you go to bamboo you see a bunch of plans that look like they've never built which is not at all true they're just underneath that icon and so it's that whole extra second to click on the icon to see our status we've come up with some creative ways of working around this one of our guys built a grease monkey script to hit the REST API and toss all the rows under the screen that works for him and people who use Chrome or Firefox I suppose and I'm working on a wall board project right now to accomplish roughly the same thing so the whole team can see status so we've got ways around this so we think we were successful and I appreciate that everyone has sat there mostly still for the last twenty seven and a half minutes here's what I hope that you you got out of this for product release branching git flow is great we're happy as a team certainly we're striving for continuing to improve our process and and make ourselves more efficient we recently our D been debating whether or not we actually want to use hot fixes the team likes to have one thing to do to fix something that's it's out and prod and we might do support branches for everything it's one of those 5050 sorts of decisions for us so we're thinking about that and we're able to react quickly when priorities shift which is like ten times a day maven tough adversary but it can convince to be an ally there's a great community out there of plugin and extension developers so please don't be discouraged by its complexity or Dobby hitting his head against a dresser and regardless of your branching models the tools work really well together we couldn't be happier with how seen we're able to use JIRA to just start and work on things and then all the other tools to view status do code review Get Alerts etc I want my team to be happy and for the most part this transition left them excited I had one developer who's is a pretty tough nut to crack and we actually moved his stuff after he moved the rest the team he uses a slightly different set of tools and the right before the trip I said oh yeah so how do you like it and he said this was the greatest change that's ever been made since I started at nuance and that was that was incredible for me to hear great feedback from the team they're happy and so we hope that because they're happy we can continue to build good software and then keep them that way so thank you very much for your attention there are 46 seconds left so I can probably take one question if anyone has one and otherwise if you've been putting them into the app I will answer everything that gets posted there or you can buy me a beer later or I can buy you a beer later or I'll be out there so actually since he gave you such awesome content we are unable to do a QA but I will have him respond to questions as well as a scene calm / get is our get microsite and Matt just is awesome and took the liberty of writing a three article series on how they did this in detail so please check it out there if you want to click reference also the slides will be up at Summit Atlassian calm within I'm hearing hopefully by the end of the day but don't quote me on that and I'm saying this over microphone but a lot of people so this recording it's okay but otherwise give it up for Matt it was an awesome talk thanks and guys you have about a 15 minute break here at 2:00 we're actually gonna have

Channel: Atlassian

Views: 38,814

Keywords: Atlassian, Atlassian Summit 2015

Id: bCU_D7EHqLg

Length: 31min 27sec (1887 seconds)

Published: Wed Nov 04 2015