Event-driven architectures in Elixir - Maciej Kaszubowski - ElixirConf EU 2018

actually I'd be talking about events and this is the topic that is really interesting to me recently and I found that other people feel the same and you have a lot of resources about building system using for example event sourcing which is I mean of fusing events as a main source of truth in your application you can find a lot of resources that describe how to build micro service architectures using events and how using events make this services more decoupled and you can have a lot of different message queues and even stars and this whole approach is basically amazing with Ireland character and OTP but then I started to wonder why I really like events and understood that this is not the point this doesn't matter if you have if you have some message queue it doesn't matter if you have this event processing a synchronous or not because you see he worked at a company called up unite and we've been using elixir for more than two years now in different project both small and large and of course we love the elixir because it makes our code much better and much cleaner and often hear that because it's very functional you don't have this additional complexity of our languages so you can often hear or read that in elixir you don't need any patterns because it's just modules and functions right but then I looked at the code of the project I'm working on and I used the new mix xref functionality that allowed me to basically render a graph of dependencies between the modules and I saw that and this is actually not the whole thing because the whole thing was so big that it didn't fit the screen it's about 1/2 of one of the applications in our umbrella and we have like 15 of them right now so it's messy and maybe it's our fault but you see building software is quite a complex task because because it requires not only a lot of technical skills and technical knowledge but you have to really understand the domain you're working with you have to communicate clearly with your teammates with the clients and the users and to be honest we are not so good at communication as developers mostly but even though building systems is so hard and so complex it turns out that changing existing software is even harder because we make all of these assumptions about what are the problems what are the way that we are going and what are the goals we think we understand our users and our clients but it turns out that we don't and we often buy them blame them for that and we shouldn't so this is not about thought about using some different sourcing mechanism this is not even a talk a talk about using LTP because I like to think that this is a talk about building small things that are really independent because I strongly believe that when you have such system system consisting of rings that are very small and not coupled with each other you can have this freedom to change and when the clients wants to change something or ask it want to change something because you start to understand your domain better you have the ability to do this without making the work harder than it should be so I chose an example product and we'll be thinking how to build basically an Airbnb clone so in this application we have some-some person that owns an apartment in the distri hadn't chicken basically announced to the world that the apartment is free then the guests can look for the apartments and if they like anyone they can make a request they can try to book this apartment and based on for example how how long the dictator the owner can accept this booking or reject them so how to build this thing and from my experience on what I have seen in different product I think that and not only our projects as a wrap unit but on different blog post our books I think the most common approach to build this system would be to start with some apartment context and you don't really have to really dig out because it doesn't really matter but in this context you'd have some function to create a new apartment he would have some other functions to acquire the the existing apartments and return some results based on the query that the user provided for example you want to create a query by location and the price and the number of etc so then I would have to make a booking so you'd trade a booking contact in which you have a function that has a guest allows you to send reservation to make a reservation and then you need a function to reject the reservation as the owner or accept it and it's quite nice because it sure is nice and allows us to write this code but there's a problem because when you make this thing when you accept the reservation a lot of things have to happen you have to first of all notified the guest that his or her reservation has been accepted you have to transfer the money from the guest account to the owners account you have to mark this apartment as taken in this period of time so it wouldn't appear in the search and it wouldn't be possible to make the reservation again so you have this disconnections and as you can see they're sort of couple because we might the D context connected together and apart from that we make multiple contexts depends on the same data because now the apartments context and have to use the reservation table to filter out the apartment that are taken so we have this coupling and most of the people I think think about only one type of coupling because you have the most obvious obvious one you might have a dependency on the code itself but what is often not considered as us so bad it's there almost all the time is the coupling on the database so in our case we have the booking context writing some data to divert the patient's table and the apartment context reading the data and where is that and I think that in most in most of the systems we have a case when we have few entities but a lot of different use cases in our case we have this basically free entities because we have the user we have the apartment and we have the reservation but on top of these entities we build a lot of different use cases and actually when with a space with each requirement you probably will make this entities bigger you make this entity more connected with the rest of the system and this means that when you want to change something you have to touch multiple multiple parts and actually if you look at this example that I showed you before in the middle there's a module with a lot of dependencies and it is of course the user schema so let's try to take a step back and forget about all the things that we know about computer science forget about the data and the databases and try to ask ourself what is really important and what is the problem that we are trying to solve and what is that our users want or our clients want and let's start with the use cases so the basic thing that we have to do in our application is to allow users to audit at the apartments and to search for them so let's start with that and let's create some piece of code that is responsible for managing the apartment so for example for adding them and what we want to do this the user can send a command then the comment is validated and if everything is ok we just publish an event saying this that this apartment has been added and we have the parts at the apartments completed basically because now we can when we want to search for the apartment we just add another module another piece of code that subscribe to this event and every time the event is added we just update our local state and the nice thing is that we have single responsibility here I hope you attended solute elixir tag before so if you did you know that there are two two first roles we have the single responsibility principle and open-closed principle and this way we can have both of them because our apartment management contact is responsible only for one thing and it is really closed for modification we when we want to add a new requirement a new feature we just had another code apart from this one that we have already created so I've said that we publish an event but what what are those and what purpose do they serve and events are basically fact they represent something that has already happened in our system and that's why they are available because once something happens you cannot really undo it you can published some other events to update the state but the fact that something has happened is there to stay and this is one of the reasons that that they are quite easy to understand and to reason about but the other thing is that they mop pretty closely to to Diggle that we are trying to accomplish because they they use the same terminology as our clients as our users they use the same words that we see in our application so for example we might have an apartment at that event and this is really important because what the users want to see that they want to know which apartment apartments were added and not how we store these apartments in the database apart from the events you have comments that are basically a way of representing the intentions and they have because events can have multiple handlers because when something happens we might have few different parties interested in this fact but comments are easy in this bug in this case because they have only one con 1 handler so we always know where these comments should go and they can be rejected if there's something drug so what about the code because I showed you these pictures and they were cool in my opinion but you obviously want to see some elixir and the thing is that it's quite simple and there isn't much to talk about because we now have such functions that are just pure logic without any side effect that take the command validate the command somehow and if there's something wrong we return the error and if there's filters everything right we just return the list of events and this is nice because we have despair functions that are really really easy to understand that are easy to test and you can clearly see what are the business rules that you have to follow until then you just rub this this module in in some other module that is basically basically a public interface and it's only job is to invoke this function and if there is a list of events returned it publishes all of them so then we can see that when we want to search for the apartment we just handles this event and we cite this information somehow in our own private database and the nice thing is that previously when we had this first version of code every time the apartment was added we saved it in the database and it looks unharmed full but actually when you save this information when when the comment comes you make a lot of assumptions about what this data is going to be used for and I often got get asked by my co-workers how should we store this data should we normalize them or store for example JSON and I try to always say well it depends how will you use this data and the last thing is about this approach is that you can decide later every time you want to use this data for something you just write a code that handles some events in the way that makes your task easier so to sum up the out part we have three functions representing the business rules that we have to follow and this is nice because it gives you all the benefits on a functional language and you can you have side effects only caused by listening and handling events so the other use case is that we have to to make it possible to make a book in the book web book around and we do this by creating another piece of software or code that accept the recommend if everything is right we just save the information and publish the event and then I think you'll remember but sorry not this slide then we have to can have some piece of code that notifies the owner and then when you accept the reservation as an owner you notified the guests and this basically solved the problem that we had before because now the the booking contact is no longer responsible for notifying the guests it's only responsible for handling the booking so we can easily add another handler that updates this information so I've said before that once the booking is accepted you can you have to make it so in the search so the search wouldn't return the depakene apartments and this way it's easy because the search apart the apartment search context know what its job is and can use the data that is flowing through the system to make it useful to use this and accomplish its goal so the next part is payments and it's also easy because it requires only adding another piece of code that is responsible for payments and its job is to basically know the the data that is used to be linked and every time the apartment the the booking is accepted it charges the credit card to the guest so now instead of this one booking context and this one reservation entity you have really small parts that are really autonomous because I've been talking about how the search context have to know which apartment are taken but the point is that it's not its job to to to make sure that the apartment won't be booked twice in the same period of time his job is to basically ensure that the results won't be returned but if we mess up this part and if we return the apartment that is Titan it's the the booking context job to ensure that that there won't be reservations twice in the same context so it's nice because it allows you to have all the rows in the one place you don't rely on the apartment search context to guarantee that the apartments won't be booked at again you have this whole business role in the one booking context so I'd like to basically I read it in some blog post but I like to for each part of the system forget about the data and start with the question what is the job of this part of the system and once I know what the job is I can wonder what data do I need to know to fulfill my my responsibilities and only then when I know which data do I need I can ask myself how can I get this data so for example for the apartment management my job is to verify the and the comments coming from the user so the only data that I need is it's right there in the code because I need the rules and there are encoded in the code itself as for the apartment search context I need to know which apartments are available so I listened for the apartment he had that event and I can also listen for the booking except that event to modify my style as for the payment context I have to do the same job I have to ask myself what is my responsibility and the job of the payment context is to transfer the money once the booking is completed so I have to know how can I charge this money so I have to know the billing information and I have to know that the booking was accepted and I do this by keeping billing information in my entire private state and I used the the event stream and use the events to know when to actually build this person so what about the pub side because in my total description there was a part about building the pub subsystem in the lecture using the design server and method passing but actually what I would recommend you to do if you if you want to start is to use something like that because every time you publish an event we take the static list of handlers from the config and just invoke handle event function of all of them it's fully synchronous so you don't have to deal with all the problem which hasn't Rinna's message passing but it can have its own problems so if you need something that is a synchronous you can basically use the new nuts on you know actually registry and its dispatcher functionality to to dispatch the events to different processes but as I've said this doesn't really matter because I strongly believe that if you want to use events to to model your system you have to start with the good domain design first and only there start to solve all the problems with how to actually deliver the events and this is the part that I would really love to talk about with you but maybe after the talk and of course if you need you can use something else like Africa like RabbitMQ to us your event stream you can use the knowledge that we learned today at the keynote to build some tasks system and use that but the the important thing is to model the domain first and one more thing that I would like to tackle is the that don't repeat yourself role because it's one of the most popular roles in our in our case and it tells you that you shouldn't have any duplication that every parts of the system should have only one representation and if you look at the diagram you see that we have we have an application we have the application we have the same information about bookings in the two databases so why and I think that the important part the part that is often overlooked is to understand what each row that we know is trying to accomplish and what this role is trying to prevent us from doing and if we try to look closer we say we see that don't repeat yourself role is about eliminating the need of having to change different parts of the system when something changes and when you look even closer you can find the content that is saying that oh sorry that it's okay to have textual an equal representation as long as you know which sort of the trophy is actually the manual and where you can go and look for the true information and in this part it's important to understand what we are doing and the data used by the apartment search context is used only to ensure that the results don't contain apartment that are taken and that's why it's it's not a source of truth because even if we don't have this information the system will still works fine work fine according to the business rule because the booking context will prevent the user from booking twice so this part is actually something like a cache that allows you to do your job but it's not necessary and the nice thing is that in the beginning we can start with one database and we can use one particle database to basically use it in all of our contexts but then our system can grow and it might turn out that we need to scale somehow so we can turn we can take this apartment search database and it you can use for example sharding to to have apartment apartments from different regions handled by different databases you can decide that Postgres is not longer enough and used something like elastic search and we see that sometimes having having this duplication allow us to change easy easily and allows us to basically change only one part without affecting the other so to sum up I think this approach has a lot of benefits because it helps you create parts of the software that are really small really focused and autonomous it reduces coupling between different parts of the system because they don't rely on their internal representation this approach creates better boundaries between the systems and between the components which is quite nice when you have to change and this approach is actually more scalable because if you do this right you can basically take these parts and extract them out of the mono if put them into a separate release and scale it accordingly of course you'd have to switch the synchronous pops up with something like message queue but it's fine and you can easily do this and it's also easy to change and this was my goal from the beginning to find a way that would allow me to change my software easier to adjust to the needs of my client my users so I think it's doing a good job of course any approach has its drawbacks and it's not the exception so it requires some more work because you have the duplication sometimes you have to do a more more initial work to make this happen so it might not always be worth it and you have to consider if you need it and it would help you it also introduces some something less familiar so you may have to take some time to adjust but actually when I talk to the new team members that are coming to the project are seeing these events their opinion is actually quite nice because they have this small part of the system that is responsible for one thing only so they need to just know the events and hear what they need to do and they can do this easily and if you take this approach and you you scale it and you start to use message-passing you start to use Kafka or RabbitMQ you would probably have to deal with message loss you have to deal with events being delivered more than once so you would also have eventual consistency so it's a bit further but at least you have this separation because you have on the one hand you have the demo model and its problems on the other hand you have the problems with message loss and you can tackle this problem separately and one thing that is heart and there's not a lot of resources is how to use even versioning because when you scale when you actually use micro services you have to think about different versions of the system running and what happens if some old part of the cloud sees a new event and this can be quite challenging and one more thing is that you would have to if your team is big enough and if your system is picking up you have to find a way to somehow share the knowledge about the events in your system so the other parts of the team can do its job but like I said all approaches have their drawbacks it's not the exception so you have to deal with that somehow but there's more as I've said you can use even sourcing to basically make the event log the source of truth which has a lot of nice benefits you can use this approach to build micro-services you have a lot of different storage options and this is the topic that is really interesting in terms of the the Ireland and elixir ecosystem because you can use database but you can just as well use applying files to store some information and this is actually nice and actually can improve your system and make it a lot simpler oh sorry for that you said that I shouldn't touch this button I would like you to think about the behavior and not to focus about on the details because we often start with the when we want to accomplish some goal we often start with the database schema and I think it's wrong and it leads to different problems that can be solved when you try to put yourself in the users perspective and think about the problems that you are trying to solve so as I've said for each part of the code that you are writing think about what it what is the responsibility of this part only then think about the data only then think about the source of the state because it's much more important than the representation of this data because users don't care about the way that you store data they care about what is happening and the last thing use your tools wisely and don't try to don't focus on the roads because when you try to read about even first thing when you try to read about using even first thing with secure areas when you try to read about micro-services you see a lot of people giving you these hard rules that you have to follow I think it's not a good idea focus on your users and I feel I think when you do that you will make yourself a much better developer you'll make the project better and you'll be much happier thank you [Music] okay anyone got a question hi thanks for your talk you know I wanted to ask about the event model and the coupling part where after successful booking there were three things that we needed to do to notification than the payment so what happens for example if they are actually coupled for example if the payment fails we need to cancel the booking does it mean that this event model will be much more complicated or do you think that in such cases should we actually take it like in one module yeah I actually hoped that someone has with the question about the payments because the traffic that there is no good answer because the way that you handle these guys depends on the business model it depends on the user experience that you would like to have in our your application so for example you might have a requirement to to charge the credit card and return the error right away you may have for example some discount codes that would make it possible to always succeed but one solution for example to what you've said about that you would like would have to return an error would be to not notify everyone when the booking is accepted but notify them only when the booking is accepted and the payment is done so it actually works quite nicely because it is now the requirement that you have so you have unless accept sorry in Detroit you don't send a notification when the booking is accepted but you sent when the payment is done and this is your business role and I think it's it might quite nicely and as as I've said there's no one right question right answer because it all depends on the business requirements that you have any more any more apricot question come on drop at time okay band of applause thank you [Music] [Applause] you

Channel: Erlang Solutions

Views: 8,854

Rating: 4.9622641 out of 5

Keywords: Elixir

Id: 8qDXG7tnl9w

Length: 36min 7sec (2167 seconds)

Published: Wed May 02 2018