From Self-Managed to Powerful Jamstack Sites - Netlify + FreeAgent

good morning and welcome to everyone joining us today super excited about this this fireside chat we have here with the the free agent team and all about their migration to to jam stack powered by nutlify we are going to go ahead and get started with a quick round of intros we do of course have a few kind of planned questions for the free agent team but this is really intended to be an interactive session so we will wrap up with 15 minutes of audience q a so please do keep keep thinking about those questions as they come up and submit them in the chat to give you an idea of who's here on the panel from the netlify side we do have myself and ramin your uh reminizo senior solutions engineer and i'm an account manager and we really work with businesses that are leveraging netlify uh in their in their web work i'll go ahead and let the free agent team kick it off with the introduction and maybe some background on their business okay thanks nelson so um free agent as a company we we provide accounting software for small businesses and accountants and bookkeepers so really just a piece of software that makes it easy for for businesses of the size to kind of manage their their finances invoicing and accounting expenses the mtd vat and all that kind of stuff so that's effectively what regen provides that's the key product we provide as a company in terms of personal introductions my name is olu i'm an engineer managing free agent i look after the website and marketing platform teams and we kind of look after all the systems that underpin our ability to kind of engage our customers and existing customers and potential respective customers across multiple channels as well as applications so um on that basis i would let jeff introduce himself as well hello thank you earlier uh so yeah my name is jeff as who said and i'm a senior software engineer i'm a free agent and i work on the marketing platform team with ooly and i think he's pretty well described the type of work that we that we do on a daily basis nelson so we're going to go ahead and and kick off the the planned questions today so for the free agent team can you give us an idea of where your gems type journey started from a technology perspective yeah sure so i think um as the title of of this uh presentation kind of suggests it was very much a self-managed solution um but i think we actually we should have some images that might actually kind of help just to talk through the whole process and how things yeah there we go how things were set up so um so i'll start with the original website stack there at the top um so what we had is our website code it was a it's a middleman static site that we use and all the code for that was sitting in a github repo um so anytime that we pushed to branches or merge into code into production then we had a jenkins pipeline that would pick that code up so jenkins was that was managed by our dev platform team um and at this point before we kind of moved to the fight that was all hosted in aws so jenkins would pick up the changes it would run the tests it would then build the middleman site and then take the contents of the middleman site and push that into an s3 bucket and then if it was a production deploy the s3 bucket the contents of that would then get moved across onto our production server which again um our infrastructure that was all self-managed from those two data centers in the in the south of england um other options if we wanted to then push it on to integration or staging servers it was the case for the engineers we could give another couple of commands in the terminal and then again it would lift the code from the s3 bucket and then move it on to on to our servers that was kind of the the main website stack just a bit of an overview and the other part we had as well was the we had a cms so that was more for our content editors if they needed to make make changes uh such as blog articles and that type of thing and we also had a bit of a mini deploy preview uh so for the cms kind of similar to the website if we made changes putting it into production jenkins would run the tests and build that and then push that onto one of our production servers but if we needed to test that on pull requests there's a bit more complexity of things that we we kind of had to set up a build zone kind of prep the server and then kind of push the code onto there and then before we can view the changes and the deploy preview as i say that was for our columns teams so it was really just an index page they could go to they could get a list of all the um branches that were that were on the system they could pick the one that they were working on and choose to view the preview for that and then at that point we had a few different instances that would grab a free one um fetched the pull request and then run the local middleman server so um so they could see their changes before they deployed into into production so they very much self-self-managed solution that we that we had going well thanks for that uh kind of description jeff and and getting a feel for for where things really started and a lot of the the complexities there so um you know maybe i'd like to talk a little bit more about the options you considered when you started off on this journey right and you were thinking about genstag could you tell us more about you know what you were considering in the migration to cloud infrastructure uh yes yes so um just like jeff mentioned earlier we at the point where we were thinking of the migration we already had a cicd pipeline running at aws so that's all the things that we orchestrate by jenkins so it was really a question of okay do we move a bit more of you know the infrastructure that would underpin the website uh the cms the previews do we move that into aws as well and effectively leverage some of the services you have in aws like s3 lambdas out front and all of that stuff or versus you know moving the other way and just saying we'll go for a managed service like netflix so i think those were kind of like the real two competing options we had in front of us in terms of determining how the migration would have gone i think apart from that as well as as a key consideration of migration we were very very interested in keeping our stack the same and also kind of not really moving too much of them or not not necessarily delving into two new technologies as part of the migration so currently all our current setup at the time was um javascript with a combination of statics like generator which is middleman combined with some sort of cdn in front of the in front of in front of the deployed website and using some of the ruby code behind the scenes to kind of tie things together so we're quite intent on keeping that the same as part of the this migration and not really changing that side of things so i think when it really boiled down to it it was really just do we want to do this ourselves and and take advantage of the individual services in aws to kind of piece the whole thing together versus just moving to a service like like netlify that gives you everything as a single package and it's quite easy and a lot more nimble in terms of being able to get things up and running so i think for us those were kind of like the key considerations in terms of what else were we going to look at if it wasn't going to be necessary awesome so i think you did give us a little bit of an overview of what you ended up ended up selecting but uh i have this this diagram here and and i think that does kind of show show some of the the points you mentioned were there any other points that you wanted to highlight or would you rather you know give us more ideas as far as what influenced that that final decision uh yeah in terms of uh the other things going into the final decision so first was you know what are the alternatives out there well the other part was all there's a financial consideration here and we needed to understand as a business the true cost of of doing this ourselves versus using a managed service like netflix and that's not always easy to figure out because at times as a company you have all these different teams that you know work together to to provision infrastructure and manage it but you it's it's at times it's a bit hard to put a finger on what the true cost of of that actually looks like so in our case we kind of undertook a very simple exercise where we kind of looked at some of the services we already had running aws specifically on the ci cd side of things and we also looked at potentially if we're going to do this ourselves in aws what are the other services we're going to look at what were the estimated using and the cost involved and then we also did a small exercise as well trying to to kind of account for the time of our devops team in terms of the time they were actually going to spend with us getting through the migration and also the operational time that's going to be involved in actually keeping it running over time so we factored all of that in to come to some sort of um numbers that we could put side by side against the potential cost of doing doing this in netflix i think i should have i should not do that that still didn't paint the full picture because at the end of the day there was always another opportunity cost that we kind of struggled to put a number on which is around if we did it ourselves we'll be committing the time of some of our more critical teams in the business to implement this and also look after it and there's an opportunity cost of what other projects do we have in the business that high priorities that we would potentially have to um potentially have to kind of push back a bit in order to make time for this migration and we you can't really put a number on that you just need to understand how critical some of these future projects are to the business at that point in time so it was made up of um kind of like the bit we could put a figure on and then the bit we couldn't really put a figure on but we identified that you know there is an opportunity because to spending time ourselves doing this and also the time we're going to use that's the time we're going to you know have to spend maintaining it at the same time so i think from our point of view those those were kind of like the different bits that would go that went into the picture to help us really really get to a point where we said you know what this is the decision we're going to make in terms of how we proceed with migration awesome and and yeah that this is what we hear from folks all the time you know how do we quantify uh some of these um deciding factors and i'm sure the audience as well you know some some folks here are also thinking about how do we bring bring gemstick and network into the business so i think a lot a lot of great great points that you've mentioned there lou so you know once you made that decision and you decided you know the gym stack it is going to be a good good fit for free agent and will decide to to use a service like netlify for uh managing this how did you go about preparing for that migration can you tell tell us and the audience more about the steps that that were there yeah sure um so i think that image that you had up there gave a pretty good overview of where we ended up but i think there's there's quite a few steps before we before we ended up there so i think the first thing we wanted to identify was um kind of what's a small kind of system that we can work on that we can maybe move across and kind of test the water and see how it works so we chose the cms that was the that was the first part we decided to move it across it was actually netlify cms that we were using so we extracted that from our self-managed service we set up a new repo for it connected it to to netlify and uh we got got that got that live and working so that was um that's quite a nice way to test the water to to see how it all works we could see how the build process worked how it was what it was like for um getting the site live all those kinds of things so that was a nice a nice starter and we we enjoyed the process it was really easy to set up the site on on netlify um so at that point we thought right where can we go next what's the how can we start to migrate the the next parts of the of the system across so what we actually did we got our main website repo and we connected that up to uh netify we kept all our jenkins pipeline all that stuff running at the same time so that was still live but we had netflix kind of running side by side with it so it was um doing all the builds and actually deploying deploying the site as well so we spent a lot of time just making sure the build process was was working reliably and we also needed to make sure we could get all our tests running on there we had sort of our spec test with jess test there's a few different things so we need to make sure that that whole process worked and once we were we're happy with that um yeah that was um that was quite nice to get to that point that we could then kind of started to take more measurements at that point to see sort of we looked at the the jenkins bill to see how long that that was taking in comparison to the netlify build we kind of broke it down into the stages of the of the net the fight builds and kind of recorded that actually over the course of roughly about a week and at the end of that we're really happy with the within that the fight it looked really solid it was really reliable it was kind of pretty pretty consistent actually just in terms of all the all the builds so um that was working well so i think at that point we were we kind of recognized that given the size of the instance that we were building things on with jenkins we knew we maybe need to step up to the high performance products on netlify to try and get kind of something reasonably similar or kind of trying to beat those kind of deploy times and the performance that we would that we were seeing so that was the point we really got in contact with with netlify to and we've kind of discussed setting up a kind of a pilot scheme for for a few weeks so i think it was quite nice at that point and probably recommend doing it that we actually had everything working well kind of before we actually went into the pilot phase so the pilot phase then was more about trying to improve what we what we already had to try and speed up the build process there's a few things that the sport teams and so on were able to help us identify where we could we can improve things there so i think that's that was that was definitely a good idea to go into that with with things working already pretty well so yeah i think at that point it was really just doing the comparison again before what we had with the standard products and see how it compared and yeah we were we're starting to get really really close to the build times that we were looking to see and we also tested the the high performance edge product as well on that phi so we used um personally ourselves we were using web page tests to kind of test out the performance of that we were able to do measurements and kind of see how that how that was all working and how it compared to the to the uh the standard edge as well so that was that's quite a nice process we had quite a lot of data to do the comparisons and we could see that the high performance products were were going to work for us so at that point we had a whole nice schedule of 501 to roll things out the idea was to first of all try and roll out the deploy previews to our comms teams but leave the production site as it was and then once that was done we would then migrate the production site but as things go the plans never never quite worked out so it turned out we got hit with a ddos attack on i think it was the 7th of january it was late in the evening which actually ended up taking our taking our website down it was kind of attack coming from all around the all around the world um and we managed to get the site back up again but the attacker kind of went away but came back again the next morning and at that point we were kind of discussing if this keeps happening if they keep trying to attack the site are we in a position to actually just quickly migrate the whole site really really quickly onto onto netify and just move the whole production site uh because we'd actually done quite a lot of prep work at this point and we're confident that things would would hold up well there was some kind of work we wanted to do to tidy things up but that could maybe wait till afterwards so again friday night came another attack came and it was starting to take the website down so we took the decision friday night not the best time to do these things but lift the whole site move it across onto onto net the thigh and it works really well in netflix because they've got a distributed cdn all across across the world they were able to actually soak up that traffic really easily so the site just stayed up never went down the attacker got bored disappeared never never came back again so it was working that well we just we then just kept the whole site on on by the production site so that was the production moved over and then over the course of the next week or the next month we got the the comms teams moved across onto the deploy previews and then just gradually started to remove the other infrastructure that we had with uh with jenkins and uh and all the other stuff so yeah it was a bit of a process to get the get the whole thing get the whole thing moved across and let's say i don't know if you want to bring that image up again just as the that kind of final final bit that we have yeah there we go so it's just very much very easy to manage because what you have got the website github repo there and the cms github repo so we have all the the code for the site in there the code for the network functions lives in there we've got the code for the build configuration that lives in there so that's really the only part that we have to to manage now everything you see after that is managed by by netlify and we just we push the stuff into there the high performance builds it's giving us really good build times and then once that's built it then pushes on to the high performance edge and we've got the the deploy previews are kind of standard with all that it's not something we previously we were having to kind of rule our own solution to that it's all just kind of baked in with netify so yeah i think it works works really well and you can see it's nice nice and neat and just uh much less for us to for us to look after awesome jeff and thanks so much for the context and that that really sounds like quite an eventful friday night uh i'm glad it all worked out wouldn't recommend that it's not the nicest but uh thanks for the uh move across ben very well ultimately a great story yeah so thanks for that kind of anecdote on the migration and and doing it during adidas attack maybe something that the audience will hopefully not have to experience so what you know now that you are live with this shinstax site on netflix we're wondering what has the experience been post go live and and what has the business impact in uh thus far yeah um it's it's been really good since since we've moved over i think for starters um as a team as the the marketing platform team we we're now in a position where we have a lot more autonomy we're a lot more self-sufficient which is quite good because for some of the projects we're looking to do this year in the business um we're able to get on them faster and also it kind of simplifies our it simplifies our um the process of kicking off our projects mainly because we we were able to take out that step of coordinating with other engineering teams for resources that would underpin the project of some of the of the size of some of some of the work we're doing right now so because all we need to know is we need to work with our stakeholders and once we're happy with that we can just you know make a start we don't really need to coordinate resourcing with devops and other teams so it kind of makes us a bit more agile and we can move on with our project a lot quicker i think like jeff mentioned it's nice to kind of have all our all our assets and code base at the same place under the same repo which is quite good nice and tidy and i think more importantly as well for for for for our devops team which kind of really really supports the our main product the software service accounting product we offer and they're able to actually focus on more important requirements within the business uh that actually directly impact our product rather than being slightly you know distracted by having to support the marketing platform at the same time so i think on the whole it's it's been a very good experience and this is just how many months in this is people taking four months in but what was starting to do some very interesting things and and i think we were definitely very 100 sure that this was the right move for us as a business awesome yeah and i think one last question for for the free agent team you know now that this migration is complete what's next for free agent what what will you you folks be working on yeah interesting times ahead but without going into too much too too much of of the specifics we we we had a bunch of product projects lined up from last year and they all kind of really focused on improving the experience for our accountants and direct customers and also prospective customers as well in terms of learning experiences via a website and also tailor the experiences in terms of how we get to kind of surface the features of our accounting software better and tailored that is based on some of the information we have about our current users so we have a whole pile of different projects that's lined up for this year some of these are about learning some obvious about personalizing experiences for our users but those are some of the key things we're looking to kind of to kind of embark on this yet a bunch of them already in motion as we speak so i think uh in terms of what the future holds it's it's it's interesting exciting times and um i think it's quite good that you know we're in a position where we we can actually start to deliver some of them we can start to deliver some of this uh some of these features for the business i think the last note to add to that would be the fact that as a team we're moving away from being uh we're moving into being more of a product or a feature team for the website rather than an infrastructure team if you get what i mean because now all of that is handled by netflix so we can actually focus on experiences we want to deliver on our marketing platforms rather than you know trying to do that at the same time while you still maintain infrastructure that underpins that so i think that's where we are you know and that's what we think will be for the future i think um actually just to add to um probably the previous question actually just in terms of how we're finding things with with netlify as well i thought might be just quite useful just to mention from the engineering point of view quite actually really enjoyed working it's got netlify dev um which is uh it's really nice kind of workflow for working with our uh with the functions for for testing that um again because everything's all together you can kind of test the whole flow it's not like the functions are often a different system and they're quite quite different difficult to test so it makes that whole process really really nice that's um and even other things that kind of find they've got netflix live so i've been able to be really easy for like web hooks if you want to test stuff coming back in you can give it that that site and it just you can see all the console logs and everything coming up in the in the terminal so it's really really good really enjoying that that part of it and and yeah just also a good shout out to your support team as well because i needed it too much but just in terms of there's a few things that we wanted to try and improve on with the bills even afterwards and they've been yeah they've been really really good and in terms of coming back to us quickly giving us lots of help and pointers and how we can improve things and i think the identified community as well um even just the free for everyone to to access and kind of get get different uh hints and tips on there i think that's really really useful resource as well so i've learned to learn quite a lot on there too austin jeff and i really really appreciate can all the shout outs as far as you know the the tooling you're you're finding useful and oh great to hear that you'll be able to focus on features and shipping new new functionality i think that that's where a lot of a lot of the folks here in the audience want not want to get as well so uh now that we have kind of completed our planned questions i did want to remind everyone here in the audience you know now is your opportunity to ask these folks on the free agent team any questions you might have about their migration or kind of how they brought jam tech and and not buy into their business and some of the stuff they took so we do have one question here which is uh jeff i think you you had mentioned bringing down build times and all about maybe optimizing there can you take can you tell us what other steps you took to reduce build times yeah so um i kind of mentioned briefly earlier but the netlify build is almost kind of split into kind of three sections so it's like the pre-build phase the actual build itself of the site and then there's a post build phase so one of the things i could i could see what netlify does um after you've performed a build it pushes all that code into a cache so um so whenever you go to build the site out again you don't need to reinstall all your gems and all your npm packages and uh download the whole github repo again but we noticed one of the things that our cache was massive it was about it was about 2.4 gigabytes so it was taken took a wee bit of extra time to kind of fetch that and extract it so again we're talking about support we've got in touch with this we're just to find out a bit more about what was actually in that cache and kind of where it was coming from so we kind of found out it's to be honest and most of us to do with our github repo because it has been it's been around for a long long time and it's been very very big and there's been a lot of a lot of history behind that and so we ended up doing a bit of work with our dev platform team to essentially do a rewrite of the of the github history to get rid of there's a hope there's a lot of stuff in there that um didn't make sense anymore didn't need to be in the repo and we're able to yeah spend a bit of time and just trim that the size of that down so we ended up bringing that down by i think we've gone down to about 1.1 gigabytes of this now so we shaved off quite a quite a chunk of it and i think that even that itself that was a good time scene that was probably about 30 seconds or so off the of the build times um another part that we did was um nipplify has this bit i don't think it's that well advertised but there's like a cache folder that you can push content into um during a during a build process so we're kind of um on our test runs because uh the high performance build machine is over kind of 10 cores we were splitting all our tests across those but they were just kind of randomly assigned so the tests weren't particularly well balanced so one set of tests on one process would finish really really quickly and another one might take a long time so they weren't balanced particularly well so what we did at the end of a test run was take the times for each one of the tests and then push the results of that into that cache folder so the next time we did another build we could pull that information down from the cache and then we could distribute our tests across evenly across all the processes to get all the tests balanced a lot better and again that was another thing that took probably another i think it was roughly about another 30 seconds or so off the off the build time and then the other part was just more in our code base itself and i don't know if anyone here kind of works with with ruby with with middleman but calling partials and so these parcels are shared bits of code across pretty much appear in all all the different pages on the site like say something in a header they're quite slow to call in uh in ruby so it was just a case we able to cash some of that stuff during the course of the build and um just caught from cash when we were building out the site so that took another chunk of time off of the build so so yeah we're able to actually take take a good chunk of chunk of time off and then say just working with support just to get help with some of the stuff that we couldn't necessarily see but they were able to kind of see from from their build logs and so on so that was uh so yeah that's quite a good way to get get things done very cool thanks chef and we do also have uh for folks here on the line we do also have quite a few resources that we're happy to share with with folks who are kind of going through this process of optimizing builds as well uh going into the next question we had from the audience was there an impact on other stakeholders on the team so i'm i'm imagining that means folks uh who who weren't kind of in your immediate web team uh or you know on the broader free asian team uh what what was the impact of this migration for them i think that um so um in terms of impact there was very little actually some of our key stakeholders will be the content eaters who usually you know they manage all the content on the website so effectively the goal of the the migration from our point of view is do it so smoothly in such a way that they would not even know something has changed it's almost like trying to change the engine of a car while it's in motion is kind of like the other good analogy for this and we're able to achieve that so for all intents and purposes there was maybe a few nuances in terms of slight differences but we were able to do the migration start to finish and not really have any impact no downtime and no no no impact to the users they all they would probably tell you was that no maybe they just noticed things would be faster but apart from that this this all happened kind of like under the hood with no downtime and no impact any of those things which in my view is a good thing and is that was the target of the migration in the first place you know we wanted wanted this to be done in such a way that you know it was seamless you couldn't really tell anything on the user side of things that you know something was changing in terms of the infrastructure that was underpinning the website and also the the systems that allow you to work on the website so it was pretty much smooth a smooth process with no no real downtime or or any problems uh from annex from from a stakeholder point of view yeah i think that's yeah i think it's got a lot of credit to not the fine times as they the build config is i guess some people might have some worries about it's very very fixed you maybe don't have as much kind of flexibility in terms of what you can what you can actually do with it but yeah we've kind of found with this only because we're able to kind of almost replicate everything sort of in terms of how the users were saying things were able to make quite a lot of our own configuration on the builds and how things how things were kind of set up so i think it's there's quite actually quite a lot you can do you never find there's a tunnel file that they have and we were able to use that as part of the build configuration but you can also um we were able to give it on ruby there's these rick mans so we could actually come out of that tunnel file and give it a command then we could start to extract a whole kind of other configuration and make it really really unique to what we actually need needed to do for the for the deploy so it's been i thought it's been been really good kind of learning how to how to take advantage of all that and how to improve the config probably also what mentioned in the login system as well because we we did roll our own solution in terms of trying to make sure we could alert people within the business once that deployed preview was ready because we didn't use the build the built-in alerted system that came with netflix but again that was seamless we rolled down solution in a very short period of time you know exactly what we had when we were jenkins same thing distributing into the same channels and with the same process and we were able to do that i think that was one of the few things we did ourselves but you know it was a very very small piece of work and the plug-in system in itself is the test the the way the plug-in system works is a testament to the fact that we control something really quickly and effortless effortlessly like that they kind of achieved what we wanted so i think that was a good one as well okay cool yeah and then just just looking at this uh there is one last question we did get here open which is what are what were the driving factors in that migration to the cloud from from your business perspective so it's kind of like nothing you know groundbreaking is is the usual factors really around you know we've been we have we've had the website and infrastructure in you know data centers for so many years and we were at the point where we really needed to start to begin you know scalability for starters and we we knew that because we had already been using pockets of aws services while still in the data center we we knew that moving to some sort of cloud service was what's the best way to go it was a function of what cloud service was it going to be so first it was system an infrastructure service that allows the skill better and also to keep an eye on our cost as well because like you know having a data center that's pretty much a fixed cost regardless of how you use it uh meanwhile moving into some sort of cloud service will allow us at least have some some sort of variety in terms of on demand pricing for for resources we use so i think those are some of the key factors and that kind of influence our decision to kind of move to the cloud anyway you know it's it's i think it's the reason most businesses tend to do these kind of things very cool very cool well thanks thanks again uh free agent folks this has been a really insightful insightful discussion and i'm sure that the audience has also really gathered a lot about you know just how to bring jamstack and and the modern way of building for the web into into a business so really appreciate the insights you've given us today um thanks to everyone as well who kind of asked questions here um for folks who are on the line i'll just add the um kind of link here where you can ask any follow-up questions that you might have for today's session um and we can we can follow up on those from the notify side thanks again and uh we'll see you soon thank you thank you very much take care

Channel: Netlify

Views: 146

Rating: 5 out of 5

Keywords: netlify, jamstack

Id: iqNWUOIJUmA

Length: 34min 50sec (2090 seconds)

Published: Mon Aug 23 2021