ElixirConf 2019 - Building an Elixir Team When No One Knew Elixir - Jacob Parry

[Music] my name is Jacob Perry I'm a software engineer at divi divi is a FinTech company it's HQ is in Utah but we're kind of all over the place something special about me is that when I came on to Divi is when I wrote my first line of elixir code like the majority of the people on the Divi team actually so in summary what I'm gonna be talking about in the next little bits is if you can't really find any elixir engineers you might as well just make them because it's easier to make them then find them experts in elixir aren't really needed to grow and have an amazing elixir team and build a great product I think over the course of all the hiring and crazy growth that we've done the vast majority of people have never even heard of elixir before the complexity of elixir at least at big at the beginning is not needed you as you dive into a little ixr and learn more about it I highly recommend that you do you learn more about the advanced things like OTP and gen stage and and these powerful features but to get off the ground running and be running really fast and really well you you don't need them at first so I mean this is the main summary now you know you can leave if you want otherwise I'm gonna keep talking in the beginning for Divi we divvy was founded about three years ago in 2008 16 and we started with elixir from the very beginning I think that's kind of an interesting use case because from everyone that I've talked to you most people are currently porting their system over into into elixir or are just dabbling with it they might have a small section in of their system and elixir but for the most part is running something else and we had a we start with a really small group of engineers why we chose elixir from the beginning is I mean it's a great question the the CTO at the time was familiar with elixir and he he chose for great reasons one of our senior software engineers Clinton wrote a great post about this and it talks about the fault tolerance e of elixir the scalability the concurrency all the things which are really great for us as a company for Divi because we're working with people's money and things need to be running really well all the time and Alixe that elixir really lends its hand really quite well to this use case so fast forward another year and a half couple years later April 2018 that's when I joined the team and that's when I first started writing my first elixir the teams were still pretty small at the time we hadn't started we had just barely started our huge hiring phase and our our growth and expansion so and at that time we had a small group of back-end engineers working in elixir fast forward another year and a half to now August of this year we've grown the team it's grown somewhere between eight and ten x in in about a year in one year is about eighty X since I came on is even more than that the teams are they're bigger they're there's way more of them were working in a great amazing codebase touching a lot of send code but we're able to create a lot of great really great features with minimal bugs even though we're all working together on the same thing so this is kind of the hiring phase over the last year and a half that well two years that we've experienced and for the first while up until about April of 2018 if you can see that the team was about the same size but then we took off like a rocket how and and why are really great questions for this the why is that DB is in a really great niche FinTech marketplace and in finance businesses tend to be really sticky once you have this company that's doing something really well and clients like it it's really hard to to break into that space and find new clients as clients because for the most part they're happy with where they are like it's working I'm just gonna keep things how they are and we the other part of the why is that in order to do this we knew that because things are sticky we needed to get I have a really amazing product we needed to grow really fast and be able to crank out these features that our clients wanted as fast as they could so in a set in a sense we could get there before anyone else so that we could find people who who realize oh this is such a great product and then deliver the things that they wanted and they never want to leave us and the hell of that is I don't know if you've heard Devine the news but we've been raising some really large amounts of venture capital funding in order to to really just fund this huge expansion in our engineering so that we can move as fast as we can as we could as we wanted to do this with Divi as a whole we had to set some reasonable expectations we knew as an engineering team that five years experience for an elixir engineer is just not really realistic I think there are a few that have that but they are really far and few in between we just we knew that we were to end up hiring grunts OH versus pros or experts and I'm not meaning anything derogatory by grunts but we just needed people who'd be really capable of learning any language and cranking out really good code really fast so more like a like a machine or a factory instead of waiting and hiring for the specific person who knew the ins and outs of everything we had to get stakeholder buy-in with the leadership of the company and this was interesting because we had a really great technology that we were wanting to use elixir and oh especially for the leadership of the company they had never heard of most programming languages before but I'll especially not elixir so we had to let them know that we're gonna be using this technology that was more obscure but had really good reasons for why we wanted to use it we we thought that feature development probably slowed down because with hiring new developers who had never most likely never use the licks before it would take a lot of time for them to ramp up and get to a level where they can be proficient and really productive in elixir these last two points proved to be really false we didn't really need to set those expectations because they it turned out not to be true our future development it was like the slowdown was like a blip on the radar for the most part when we brought on new engineers they were pushing code out to production in about two weeks and I think that lends a huge hand to elixir and the thing and the fact that it's such a a friendly language to be able to pick up and and learn and be productive with so when we were trying to find people there's really it's there's some really interesting statistics this is from the 2019 Stack Overflow most popular technology survey developer survey and elixir is way down on the very bottom according to all respondents about 1.4 percent of all people had either used elixir or heard of elixir which is not a lot especially where you're trying to build an entire team as fast you can and move forward with that quickly on the flip side where it's the most loved on the most loved list elixir ranks quite high so we basically figured well if people don't know elixir we can't find them let's just make some elixir developers because they're gonna like working in elixir anyways so when finding people I think we we've hired dozens of people over the last year for our engineering team and we've hired roughly five people who came to divvy because they were looking for a company who was using elixir in the first place most everyone else was meh or agnostic about elixir a lot of them had never even heard it before to be honest I had never heard of elixir before I interviewed with our CTO and at least one of our engineers absolutely hated elixir when he started luckily he worked in the front end for the most part but he he he just he he didn't see the light at first so there was really two sides of our growing the team but first was hiring and the second was training so with hiring we realized really quickly and right off the bat that what people knew coming in about elixir mattered very little because most likely they knew nothing at all so what we really looked for was more culture team fit and capability for these engineers that they had a solid foundation under them in design principles that they were able to learn and wanted to learn new things really quickly and that they would be able to be up to snuff and be productive as fast as they could we gave out a pretty judicious tech assessment and there's a little controversy with this just because what we would do is we'd give out a it's kind of a project we'd say hey we want you to build a budgeting app and here's our tech stack which we used react on the front and then elixir for the back back-end and Postgres for the database and everything go build something for us and show us what you can do and for the most part it worked really well because it allowed us to see where people would most likely fit in if they focus mainly on the front and in the react code then we knew that they'd be a really good fit for our front-end team but if they spent more of their focus on in the backend elixir we knew that they would be really good candidate and but this was rough because most of the time people spent way too much time on the stack assessment I think when I did this I spent probably about 20 hours work for this interview doing this tech assessment which it worked out really well for me I don't regret it at all but we have had a couple of people who said hey this detects the SEC assessment is taking way too long and we've been really mindful of that if they're having family things coming on or not able to have enough time that's fine but if they really is like hey this is too much time I don't want to do this we were like well that's great thank you you don't have to work at divvy either so I'd like you to meet Daniel Danny Daniel is a really good candidate that we hot we hired from this tech assessment and this hiring process we had he gave me permission to do this so as I drag him through the mud he he knew full well what was gonna happen he's and he's actually great he he came on and he spent the most of the time working in the front end react code and that's where we ended up placing him was working in our web app and it worked great but Daniel like culturally for the team for everything he was a knockout he's a great engineer and we really enjoyed having him but at first he's the one who actually hated a lick sir when we hired him on he we'd have conversation he'd say things like you know elixir just it just feels like it's gonna fade out there's this it's just a fad language it's scratches certain niches with like a flamethrower on top of that Alex Lee built on the dinosaur that his Erling the homepage proudly touts the vm's reputation of being low latency distributed and fault tolerant as if these aren't basic features implemented and virtually every other respectful language i know fighting words right and bottom line is elixir is an interesting experiment and enforcing functional programming but if throws the baby out with the bathwater so he uh he stayed on a front then for a while and then on the flip side of the hiring we had training and probably the most important thing that we did at first was that we enforce code conventions we use the tools like credo the these static code analysis tools in our CI pipelines credo to make sure that we are our formatting and things are looking good dialyzer to tell us where things are broken and helping us find dead code or fix bugs before they showed up the elixir format are just so that everything that everyone is looking at is the exact same because when you're bringing on dozens and dozens of new engineers no one has time to try and siphon through and learn everyone else's coding style and documentation we made that required for every code every line of code that people were writing every function and the great thing about all these is that we were able to enforce them with linting in our site CI pipelines to fail if they didn't do anything if they didn't implement them how we wanted and the documentation is great because there's that famous quote that shows it talks about with gen stage where when they were writing gen stage it took a thousand lines of code but they had eleven hundred lines of documentation to go along with it and so we've really adapted that and taken that to heart we've had formal ish trainings with our new employees as they've come on we've had our senior engineers who've been working an elixir for a while teaching like a 100 X or 101 class once a week and people are able to show up for our QA engineers or the the new engineers who've been come on and they're able to go through and just learn as much as they can or they've had class we've had classes where I've taught things on absinthe in graph QL since that's another technology that most people haven't really had a lot of experience with and and how to use that and elixir and then we've had code reviews really strict code reviews and really code reviews are probably the King out of all the things that we've done for really good reasons the first is because we are a fairly remote team we have a lot of people who work in the in the office but we have a lot of people who are working in remote and a code review provides a really nice Avenue to receive feedback and see other people's code as well because as you are learning and learning a new language it's really hard to envision new ways that you could do something and by sitting down having someone go through and be like hey you could actually do this and it would be better or seeing someone else doing something is like oh I've never seen that before I think I'm gonna try that try that out next time there was a unexpected benefit of building a team in elixir because it really produced an unexpectedly humble and no ego team because as we're bringing on all these engineers we were hiring senior engineers who had years and years of experience yet they were receiving code reviews and sitting down with people who had been working in elixir for a year but we're maybe fresh out of a code camp a year earlier and it really just allowed a great culture across the board that code reviews were not scary we're not menacing if they're meant there to be productive and allow just you to ship a better product and by implementing the the code conventions and things above it really just allowed you to focus on what was important which was the logic and the features that were being shipped and then with our trainings there was always a baptism by fire anytime that we brought someone new on they were always 100 percent of the time place on a really high priority project because we have no low priority projects everything is important we're trying to get everything done as we as fast as we can as with those high qualities we can't we could and I remember when I first came on my first baptism by fire we liked to affectionately call it the TX p2 which is a transaction processor I mean we're a we're a credit card company we're processing transactions are coming in and we had to swap that out and modify that it is it's pretty much akin to driving down the freeway and trying to replace your engine at the same time and have nothing absolutely nothing go wrong and it was it was it was a baptism by fire it was a great time we look we think back and we laugh at it hoping it will never happen again we have certainly experienced some growing pains with growing our team as fast and as large as we have probably one of the main ones we've experienced is an elixir is still even though it's maturing a lot is still a growing and evolving ecosystem things are changing about it all the time not just with elixir itself but also with the libraries that we use every day for example I call this day Pi Day which is March 14th I released some code to production which produced probably my biggest bug to date and had to do with the ecto library I had been tasked with upgrading ecto from 2.2 Dokdo 3 and for the most part it it was great it was really nice to his plug-and-play except there was this one place with ecto in the upgrade that they had changed and they they unified across the board how they handled date times and with ecto - if you created a date time and then you gave it to an ecto change set the change set would mirror that day time completely but with ecto 3 if you said that the dates was like a date time USAC or something it would add on the extra zeros to get to that that precision so it in this case you can see it added on 4 zero so to be at the microsecond precision I didn't realize this at the time but we are using a date/time field along with three other fields and hashing them together to create kind of a unique ID and with those added zeros it's created brand new unique IDs across the board and so I resulted in a lot of duplicate things from something that is relatively minor so it's small changes like that that might not exist in more existing mature languages with deployments and this is stepping away from elixir in general but with deployments as a whole it's been really unique a really unique challenge because we've had to go through different phases of trying to figure out how to increase our deployment speed with out everyone feeling like they're stepping on each other each other Stowe's and feeling like he needed to do regression testing on the whole system instead of just their specific part of their app that they're working on so for example this is in in January of this year and we made some organization changes on our engineering team and we were able to just skyrocket the number deploys from the tens to several hundred a month which is great because even if you're shipping out small CRM ARS at the time and going through and doing small deployments that's better than doing huge big ones anyways and perhaps a general lack of likes or experience as well another thing that I worked on was our background jobs and we were using quantum for that for this at the time and I want to stress that quantum is a perfectly fine library it is great and it works really well but and it's the library that you can use in elixir to schedule background cron jobs essentially and we are experiencing a known use case that elixir had document that quantum had documented on their github repo because we were using kubernetes in production as well and scaling with the number of deploys that we were going through and deploying quantum was crashing and causing all of our notes to come crashing down around our ears and having a really unstable shakey last remaining node in production and it was really stressful I was stressing our DevOps team out there stressing a lot of us out and we we couldn't really figure it out at the time I know the the fault tolerance II thing of elixir and the elixir e thing to do is just let it crash but us letting it crash was not a good option because it was crashing the whole system and we weren't sure why it was taking everything down so there were things that we could have done I could have couldn't rid of them all into generic gen servers and had them running but then we run into the same problem every time we deployed and did a new deploy it would kill the gen server anyways and then just started up over again so ultimately what we did is we converted all of our background jobs into mixed tasks and then we use kubernetes and the kubernetes cron job scheduler to just spin up a new container run the mix task that would run our background job at the time we wanted and then it would spin it down and perhaps there were better more luxury ways like I could have been happen but it's actually how we have things running now in production using kubernetes and as that is running really well nothing great so measuring success and growing our team I think is always an ongoing process they should always be an ongoing process because it's things change standards change as you hire new people you're hopefully hiring higher and higher caliber people that can help raise the bar at your team so the standard should changing and raise the bar as well as you the engineers gain better understanding of how things work in elixir and systems in general that that understanding should be bringing adaptation to your system so that you're constantly adapting your system to work better and utilize the tools in a better way rapid development and deployments I think is a huge measure of success our SVP of Engineering Greg Larson he hewed he would challenge us to gamify the system as much as we could because he's like well small deployments all the time is way better than having a massive deployment anyways and ultimately we engineers as a whole if you can spend less time fire fighting bugs and writing new features I think that is probably a huge measure of success which we were able to see this the light blue line is actually not the measure of our bugs over time that we were introduced new bugs over time that we were introducing and even as we were skyrocket skyrocketing our deploys the number of new bugs that we were introducing was actually going down and I think a big part of that was the fact that we were using elixir and getting better at it engineering happiness overtime and even at the very beginning has just been through the roof elixir is really unique in that because it's still a young and maturing language that there is a really great opportunity to contribute to the ecosystem that and we've had several engineers like Nathan who has contributed to absinthe which is the graph qø toolkit for elixir we've had Sean who has contributed to the Mocs library and I got to give this guy a little grief because he couldn't even spell plataforma tech right so I fixed that for him making him look good everyone on our team wants to cross discipline into a lick sir this is something really interesting we found especially with our front-end engineers they experienced some serious FOMO they are always trying to like hey let me dive in there and start learning more elixir and we're happy to let them which is great and just in general that programming and elixir is really fun so let's talk about Daniel again he's completely changed his mind he's still saying that he's trying to decide if he was held captive by by lick sir and now he has just fallen in love with a what Stockholm Syndrome that pattern matching is the the real hidden gem of elixir that he can't get enough of he misses being able to Google things for most languages like Python or Java where you can just have find the question the answer to any question you can think of but he's really appreciates what elixir brings to the table that it's now the tool that he likes to reach for instead of his beloved PHP and overall elixir is really just proving him wrong and he's actually now one of our most prolific elixir engineers so we've converted him yeah and so in summary like I said before if you can't find your elixir engineers you might as well make them because they're gonna be great if you find the right people who are able to learn the new language they'll pick it up and they'll be really quality and just help you grow and deliver an excellent product like you don't need experts you don't need the complexity of everything right off the bat because just in general the the ecosystem as a whole can let you go really far without needing to introduce some of those things and that you shouldn't wait to grow your team if you're trying to hold out for senior engineers those that have five the five plus years experiences and I'll lecture you you're not gonna find it rarely you're not going to find it thank you

Channel: ElixirConf

Views: 2,468

Rating: 4.8730159 out of 5

Keywords:

Id: yXrw82ULUrU

Length: 27min 57sec (1677 seconds)

Published: Fri Dec 13 2019