ElixirConf 2021 - Simon de Haan - Punching above your weight with Elixir

[Music] [Applause] [Music] well thanks for having me here um it's a friday afternoon this side so i'm about to knock off but good morning to everyone joining today i hope you've had your coffee obviously i was hoping for an in-person conference like many of us unfortunately i i couldn't make it and but as a as a team we're working hard to make international travel possible again with a bunch of the covid related work that we're doing um just a heads up i expect that i'll be running i'll be finishing a little bit early there'll be time for questions at the end um just my setup here i'm not seeing comments or or questions directly so jim and richie i'll need to rely on you to moderate those at the end please if that's that's cool cool so um as i said i'm simon hahn i'm the cto and the co-founder of a small startup called turn.io we're a public benefit corporation we've been around for three years and have spun out of a non-profit in south africa called praykel.org and um we are what's called a whatsapp business service provider and we have a specific focus on social impact work and we make the whatsapp business api available to organizations and so we are a bootstrap team and have received philanthropic funding and grant funding to do the work that we do so our our vision is to make um effective human support accessible to vulnerable people everywhere so i'd like to break that down a little bit so by effective we mean impactful practical and ideally easily measurable by human we mean kind and ethical and trusting transparent present and honest and so it's our fundamental belief that humanness is an essential quality for effective change in complex low resource environments so most of our work actually focuses on the majority world or frontier markets or what some people would call developing countries and so humanness without it in our experience support simply isn't as impactful and so when we talk about support we mean one or a combination of help information advice compassion and guidance so to be clear the kind of support that we want to make accessible is the kind that improves lives not for example the kind that helps you with your online purchases so that's our vision and in our mission it means that we want to encourage and enable organizations to amplify their social impact through chat and so we enable this through the tools that we build the products and the services that we develop and so both the tools and the products and the services are equally important to us and we see how effective both can be especially when used together in the social impact space so so far in 2021 we have helped 200 around 200 organizations supporting 15 million people by having 50 million conversations that improve lives so last year at elixir conf in 2020 which unfortunately i also attended remotely we presented the work we did with the world health organization to launch their global whatsapp covered response service and so this year i i'd like to talk a little bit about what we've learned as a team and how specifically elixir and the ecosystem has supported us in in doing the work that we've done in 2021. so i'd like to zoom in on the principles the decisions the habits and perhaps more importantly some of the paradoxes that we've embraced which has enabled which have enabled us to reach millions of people helping hundreds of social impact organizations extend their impact through chat and so a question we often get is how how do we do this especially given our team size and so i'd like to zoom in on that question today and draw parallels to the elixir language and the larger beam ecosystem that i believe are key to us doing what we do as a small team and specifically how those are embedded in the products and systems that we build and so this is why i taught why i named this talk punching above your weight with elixir because as a small team i believe this is really what we do so i'd like to start with focus and so as software developers i think we have a natural appreciation of focus um the sense of a flow in your work the ability of keeping it both a grand vision of in one's mind of the larger goals and objectives that uh we're trying to achieve while also simultaneously at the other hand being able to entertain the minutia the intricacies of the system we design and write and i think as developers we're also painfully aware of how fragile and sometimes fleeting focus can be so an interruption can tear down a carefully constructed mental model like a house of cards losing all the time investments made before one has had the chance of capturing these thoughts in code or in a test or in in some kind of permanent medium and when we're talking about focus what i don't think we spend much time thinking about is what creates focus what extends the lifetime of focus and so rather than thinking of our teams as predictable and rational machines i think it helps to think of them as organic entities organic entities that respond to their environments in different ways so some environments allow for teams to thrive while other environments prove to be far more challenging and when it comes to focus some environments foster focus and others make it more difficult to achieve and maintain focus so one of the key things that i really drew me to elixir or just confirmed a lot of things for elixir for me as i started looking at the stack is um what i screenshotted here so on the left is a response from jose on elixir form and on the right is the release notes from elixir 1.9 and so what i really appreciate about jose and the elixir core team's vision is that their conviction that the language is itself is largely stable and feature complete i really love that i think i think it takes focus and determination to be able to draw and to hold that line to say it's largely done this is this is it it will still improve iteratively but by and large it's finished and elixir 1.9 introduced releases as the last planned feature and that was over two years ago it speaks of a clear vision and focus and determination it lays a solid foundation for future work and i think that we can agree that we've seen the res the fruits of that rather than stagnating which some of the community members expressed to worry about like does this mean it's finished is elixir still like a a future-proof language and ecosystem that we can build our products and our teams around so rather than stagnating the elixir ecosystem has actually improved with leaps and bounds we've got live view there's surface there's nx to name a few things that we are all benefiting from despite the language and the team have the team largely having said the language is now stable it's largely done and yes there will be incremental improvements but releases was the last planned feature so i think this is amazing of of language creators being able to say it is largely finished and so this is gault's law and it says a complex system that works is invariably found to have evolved from a simple system that worked and actually there's more more to this quote but i didn't want to jam it all on the slide the the kind of the warning in this quote is also that's it says a complex system designed from scratch never works and cannot be patched up to make it work you have to start over with a working simple system and what's amazing about these simple focus systems is that they're predictable they're almost binary either it works or it doesn't work and focus really allows for simple working systems to emerge simple working systems allow for more complex systems to evolve from it like this quote says a complex system that works is invariably found to have evolved from a simple system that worked in my experience the elixir language itself is simple however the systems it enables are complex and those are like i said earlier we're all benefiting of of those things so we're a team of around 16 people nine of us are involved in software development and infrastructure um we're fully remote with team members in japan through india in south africa and kenya in europe north africa colombia and north america colombia and peru and it's our intention to remain small um because we believe it's essential to us delivering on our mission i spoke about this last year as all as well and i think i've spoken about it at many conference because i really feel that this is fundamental this is conway's law that states any organization that designs a system will reproduce will reproduce a system whose structure is a copy of the organization's communication structure and so conway's law suggests that teams will always design systems that mirror their own communication structure how they function how they do their work all that and so for better or worse the strengths and weaknesses of the team will be embedded in the system that your team designs and develops simplifying this to almost an extreme some people say that we as software developers we ultimately will be shipping our team's org chart and i think the important thing here is to to really make sure that the way your team organizes the way you work and communicate how your team behaves as an organic entity is compatible with the product architecture and so this is why for us this is why we believe that our team is really our first product and it has some really profound consequences it means that if we want to have a resilient product we need to have a resilient team and a resilient culture around that team if we want the product to be the absolute best at doing one thing well the team cannot deliver without it also having a crystal clear vision and the mission that brings alignment and focus so this quote is by dita rams it says good design is minimal detail rams designed absolutely beautiful and iconic products for over 40 years at the german consumer electronics company brown his his work continues to inspire the design of many consumer products we are all familiar with and dita rams is set as a set of 10 principles of good of good design excuse me so number 10 states that good design has as little design as possible it is minimal and it is focused so it would be a mistake to categorize this under the popular less is more trope and just kind of move on i believe there's a lot more value to this and so when i started in software development when i was presented with a certain challenge i was often just instructed to juice just do something simple and what i actually heard and what was often meant was that that that meant that what i was meant to do was actually build something cheap and easy and what i came to realize over time is that the cheap and easy approach would more often than not end up being quite costly over time so the truth is that designing simple things despite sounding easy is really quite difficult designing a simple solution takes time effort and discipline and to make things as simple as possible but not simpler takes a lot of experience takes iteration and takes refinement and focus according to dtrams good design elevates the essential functions of a product so good design focuses on the essential functions of a product a really great design thrives within the constraints of those essential functions design constraints lead to unconventional and mostly innovative approaches to problems if we believe that this is true about designing great products and we really take to heart conway's law that products reflect their teams we would really be foolish not to apply constraints and design thinking to our teams especially those teams are tasked with building products and so this leads to a number of constraints and paradoxes that we're uh we're struggling with less resources leads to higher and more focused outputs uh and it makes it easier to apply the discipline and the rigor that's required many startups fail period and many promising ones fail because they received a significant investment so with investments come the pressure to spend and the easiest way to go goes about to go about spending money is to go through a significant hiring around large team large teams lead to higher and efficiencies per person and creates an environment where people seem to often almost just be spinning plates or just carrying information between meetings they lose their initial momentum as a result of spreading their energy wider rather than deep and focused and so i came across this quote by john carmack the other day where he says it is hard for less experienced teams for less experienced developers sorry to appreciate how rarely architecting for future requirements turns out net positive and i think the challenge is that with larger teams invariably they want to solve for all the edge cases simply because they can they have the people to do it the problem is that this makes the problem space fractal or near infinite by having a small team one is forced to be ruthless about what we build and what problems we choose to solve in our space where we focus on social impact organizations and doing good it's arguably harder to be ruthless because you want to do good it's easy to feel a moral almost moral obligation to build universal solutions but the paradox is that it leads to bad product decisions and difficult to maintain architectures it leads to complex systems that are designed from scratch for future use cases that may never actually be required but for which the system does carry an ongoing cost every day of development and so there's an argument to be made for small teams because of huge ambitions so there's this there's this false dichotomy that suggests that one has to choose between a small team and a small ambition or a large team and a large ambition and we really don't believe that that is true we want a small team because we have a huge ambition small teams simple solutions allow for complex systems to evolve but it has to start small and simple simple requires a focus that is easier for a small team to deliver and is a required forced constraint um yeah so sometimes again there's an argument to be made that small teams are better just simply for big ideas smaller teams are more agile and better respond to change um now i respond i appreciate that the word agile has eroded in significant value over the last decade so i prefer to take ships as a metaphor a large shipping container i did some research and it takes 1.5 kilometers to turn 180 degrees whereas a small boat likely takes anywhere from a few seconds to a couple of minutes and so there's value and a place for both kinds but each has their own strengths and they each would be foolish not to play to those strengths being more agile small teams are better positioned to take risks despite being less resourced again one of these paradoxes the larger a team and the supporting organization becomes the more risk aversa becomes despite being better resourced and ensured to carry risk larger teams suffer from the inert momentum to maintain the status quo rather than innovate at the same time this is their essential threat small teams are likely constantly facing an existential threat because of the status quo and therefore are forced in to innovate in creative and unconventional ways breaking new ground in ways that large teams cannot and so there's all these paradoxes and i see i see these paradoxes um in well there's these paradoxes in these teams and these ways of working and i see parallels with them in elixir and the larger beam ecosystem and i remember when i first started looking at erlang and beam and started reading up on joe armstrong's work and just some of the stuff he published there was a sense that he wanted to be able to model what was going on in the real world and this is a quote from him that i found online and so i'm not really surprised that the the paradoxes and the things that we see in production or the things that sorry the paradoxes that we see in teams that we're also seeing those in elixir and the beam ecosystem largely because joe's desire was to be able to model what was going on in the real world and so the beam philosophy of letting it crash comes to mind you want up time you got to let it crash again a paradox this philosophy i believe is key to building a resilient system the paradox here is that to build systems that run for years to achieve an uptime of many nines one needs to let things crash for a while at turn somewhere around march and april of this year we just had a lot going on we were battling issues scaling elasticsearch we were hosting a social justice summit in partnership with whatsapp it was all online we were supporting national vaccine registration launches with ministries of health in various countries and there were just many many things going online going on and just up in the air and somewhere along the line we shipped a few bad releases and we lost some confidence in our ability to ship quality and reliable code the immediate consequence of that is that we started shipping less frequently and so somewhere we made the decision that to retain stability it was better to leave the system alone rather than frequently shipping new code and while i can kick myself now and we definitely should have seen this coming the side effect of that is that rather than incremental improvements our releases became big bang releases the net effect was that we packed more changes in a single release increasing our chances of shipping bugs which would be difficult to roll back and not surprisingly our confidence in our ability to do stable releases and ship quality code dropped even further and so the paradox here is that to get better at shipping quality releases one needs to ship releases faster rather than slower and so there's this strange thing that happens with things that are stable we could become hesitant uh to touch them it's it's weird stable components do not need attention and the things that do not need attention do not receive attention and so in the long run precisely these components end up being a liability because they receive no attention so to maintain uptime it is better for things to crash every once in a while than to assume they will never crash at all so for services aiming for high availability and long up times the ability for a system to crash recover and self-heal is far more important than just not crashing and so erlang is face is famous for its nine nines of uptime in the system which joel armstrong described in his thesis so if we're talking about high availability what does a high available team look like what are the paradoxes that we need to embrace to achieve this so paradox is paradoxically i think highly available team is one that thinks about being in a position of working less being able to work asynchronously and independently and being able to be offline more so there's the paradox you want to be highly available as a team allow your team members to be offline if you're designing for failure being asynchronous and offline then you'll be wise to invest in solid monitoring and escalation procedures if you're assuming things will crash you will design your system to be self-healing rather than having team members looking at screens 24 7 you will have design systems to either self-heal or escalate when that is or was not possible so this is where small teams embracing the idea of crashes will embracing the idea that crashes will happen will win small teams choose not to have the luxury of having people on standby and so we'll be incentivized to design systems in such a way that those team members are not necessarily are not necessary as the system itself is resilient to failures so sasha did a great talk about the soul of erlang and elixir at the go-to conference in 2019 he says that high availability is not about achieving an amount of nines of uptime but having a system which is there for its users when they need it the central idea in the beam ecosystem and i'm paraphrasing sasha here is that concurrency is achieved through processes none of this will be new to most people processes are essentially autonomous little programs trusted with a specific task the beam vm runs these processes in parallel and completely autonomously from each other the only way for these processes to communicate with each other is to send messages to send and receive messages back and forth the processes escalate to supervisors and these apply a restart or recovery strategy should an error happen the supervisor managing the process is best positioned and closest to the error to help determine how to best recover into a last known good state and again i see parallels with a small team operating around trust and autonomy one of the challenges of a larger team is that when problems surface those with decision-making power are often furthest removed from the problem lacking the full context required to quickly come to a good decision on a way forward it's like trying to bubble through a stack trace in your through your organization smaller teams have fewer layers individual team members must simply be trusted with decision making otherwise the whole system cannot work how do we achieve this with clear principles that the team aligns aligns around similar to the principles expressed a strategy is embedded in the beam vm supervisor model a team's guiding principles gives individual team members the freedom and safety and safety net to make the best decisions possible when challenges arise they are closest to the problem they have all possible context and are best positioned to decide the next best action and route to recovery so back in 20 2006 2007 somewhere i attended a con a conference called future of web apps in london where a talk was being held by i believe an early employee of twitter um and they discussed whether ruby did or didn't scale and um it was relevant because this was also at the time when the twitter fail whale was a regular occurrence for those that remember that the conclusion that i remember was that one's choice of language had less bearing on one's ability to scale than one would think or expect and i think i mostly agree different languages scale in different ways and have different trade-offs with regards to productivity running costs and developer happiness with that being said i do find myself having a language preference when building a product team some programming languages have significantly less faith in building a solid team around than others as an example years ago i found it easier to assemble an aligner team around the python programming language than the ruby programming language it's hard to say exactly why but i i believe it relates to python's to python having fairly strong opinions about how things should be built principles like there should be only one way to do it and explicit is better than implicit that are embedded into its design and so these embedded opinions and philosophies lead to easier team consensus around ways of working so for me there's a clear correlation between the goals a team is meant to deliver on and whether or not the technological and architectural choices are aligned and supportive of that goal so we are a social impact organization um we're looking to scale human human empathy and support using chat in ways never before done and we are somewhat intentionally constrained in terms of resourcing in people and so we're we're a team wanting to build simpler systems that have impact simple is truest form not in the sense of cheap and expendable but simple in terms of powerful and flexible simple in ways that allows a complex system to evolve out of it and so we believe we're at the tip of an arrow that is defining a new vertical of how to use chat for positive and lasting social impact we are finding that the philosophical underpinnings of the beam ecosystem in the elixir language are exceptionally well aligned with a team wanting to remain small and focused in the pursuit of impact so embracing this idea and building on the tools that elixir gives us has allowed us to do the following the last couple of months the fastest whatsapp business api stack after load testing by whatsapp engineers despite being one of the smallest ones three of the world's largest whatsapp business api installations run on our elixir stack we've supported the vaccine registrations and bookings in both south africa and india helping millions of people get their kovid vaccinations like i said we're trying hard to make sure that we can have conferences in person again we've run social justice and equity focus summits and impact boot camps in partnership with whatsapp to help hundreds of social impact organizations scale their impact through chat and four more are happening in november this year we've helped humanitarian organizations like the unhcr which is the united nations high commissioner for refugees provide humanitarian relief for refugees in both latin and central america and the middle east we were able to respond to the crisis in afghanistan and help care international and international aid organization launch a national support line for kabul and afghanistan in four days so start with simple things that work complex systems are built out of simple things that work the elixir language is a simple language like i said at the start i'm so inspired by the team to say hey this is largely done but as a result of that the ecosystem has flourished and so if we can do this as a team then you all can do so too there's nothing special about any of this and so think of this like software patterns we're all familiar with software patterns that can be applied to solve a particular problem consider these patterns for your team that can be applied so if you're looking to use your technology skills for good i hope you're able to take inspiration from our story this far you do not need a large team you do not necessarily need a huge amount of funding and at the risk of sounding very very cheesy the truth is that the world is changed for good with less than you would expect what you do need though is determination focus and alignment on mission and vision and the technology stack that practically and philosophically aligns with your team and so for us at turn we found that elixir aligns extremely well and supports us in this vision and we're proud of the work we've been able to do awesome hi simon great talk can you elaborate a bit about your work with the vaccine registration in south africa was elixir used for this thank you yes it was um so our whole stack is elixir based i elaborated a little bit on that last year the back end is all elixir and the front end is react app and it communicates over graphql using the absinthe library it's just a pretty much a bog standard just phoenix application and for the vaccine registrations in south africa as well as in india many people were able to use a number on whatsapp to register their vaccines and get a notification when their age bracket became eligible for a vaccine and so in south africa it was staged different categories of the population depending on their age were eligible and so people could subscribe ahead of time to get a notification when their age bracket became available and when that was available through whatsapp it was possible for them to make an appointment and go to a clinic or a facility where they could get a vaccine registration and so all of that messaging happens via whatsapp it's um in many countries outside of america um whatsapp is almost the default telco for most people um yeah and so it is a utility function in in many sense of many cases and so in south africa people could just use whatsapp to register for a vaccination find out where there's a slot find out where there was a facility close to them and then get a confirmation of that registration um in some other countries it's also possible for them to get a vaccine registration certificate via whatsapp which basically is is proof that you have had your vaccination so that you can go participate in certain parts of society again as before does that answer the question thank you oh a stern working on the challenge of getting more depth into elixir oh so we're not i think the bigger challenge we're working on is for devs to realize the opportunity they have to change the world and our tool of choice for that is elixir and it's worked extremely well for us however many of our team members didn't know elixir before they joined and it was easy for them or easy enough for them to pick it up and become comfortable with it and start contributing so i think if anything we're trying to get engineers to realize that the the amplifying power that software has in societies in that each of us has a responsibility to create the society this is the kind of society that we want and for us turn is a vehicle where we can exercise um exercising that opportunity i suppose in creating a more society kind of society that we want using the skills that we have in code so sorry it's a little bit more of a high level answer i'm not really looking at recruiting elixir developers specifically uh if you have elixir skills then great fantastic but it's not a requirement it's more about realizing the potential for being able to do incredibly good work shaping society thank you is there uh is there some point or conditions when you might consider growing the team um yeah look i don't think we've hit our max i think for the kind of company that we want to be personally i'm comfortable with i like the idea of being able to know uh all of the family members names of the people that we employ um that feels like a good natural boundary um there's something about you know how moore's law says like the the power of cpu multiplies every so many years and i think with the ecosystems that we're building in software the power of a team multiplies every year as well like what we're able to do now with kubernetes for example or elixir is is incredible compared to what we could do a decade ago and so i think a team now can do far more than a team of four now for example can do like far more than a team of four could do 10 years ago and so i do think we're wanting to grow the team but not crazy um also because i do think that as we work on this the the multiplying factor of just technology and ecosystems comes into play that 10 years from now we'll be able to do more with the same team that we have now thank you uh if you're currently on a team that is talk on the big band mode how do you effectively effectively turn that chip around and move towards incremental releases without backing everything up even further oh that's a good question i think a lot of the stuff that i talk about hinges on trust if if you're not in a position where people can trust you that this is something worth trying then it's going to be hard and so rather than saying we need to do less big bang releases and more like incremental releases i would focus on how do you build the trust so that when you come with that suggestion um people trust you to to execute it well and last question for now uh can you speak more about the kind of social change opportunities you see for devs [Music] um i don't think there's a shortage uh it's just a matter of looking for them and aligning yourself with those opportunities so [Music] uh i would say there's plenty of tech companies that philosophically in their dna are closer aligned with companies like ben and jerry's than with large tech companies operating out of silicon valley and so i'd start looking there there's plenty of opportunities in health and i would say there's huge opportunities in climate change i think there's huge opportunities in humanitarian space and education space so i really think that once you start digging into it you will find some amazing uh opportunities um there is for our community specifically on thinking on on using chat for social impact there's a linkedin group um called a chat for impact group where every once in a while people will will post jobs there and say hey we're looking for someone to help out with this like around um women's rights in specific country or around climate change or early childhood development use cases for example um so i would say start there um there there's more opportunities that you would actually expect thank you very much

Channel: ElixirConf

Views: 880

Keywords:

Id: MwY7p64YEXw

Length: 39min 45sec (2385 seconds)

Published: Fri Oct 22 2021