CQRS and EventSourcing with Spring & Axon - Nakul Mishra

my name is knuckle and if you haven't figured out where I am from let me help you I'm from India and as you know in India we have tradition of stories or shall I say a lot of stories so being an Indian I thought what could be the best way to start dialogues than a story so let me share with you a story which my mother told me when I was six year old kid and this story made a profound impact in in my professional career it was a story about a group of blind man those more man's heard that there is a strange creature called an elephant which is brought in the kingdom but none of them were aware of the shape and form of the elephant actually they have never seen an elephant before so they thought all right we are curious enough we would like to know a bit more about this this creature so they went to the king and say oh dear King we would like to know about elephant and we would like to you know see what an elephant is but since we are blind we would like to learn about this creature by touching him so that King said all right we can make the necessary arrangements and next day in the morning they made all the arrangements and one by one each and every person started touching an elephant the person who touched the trunk of an elephant he said well this thing is like a snake attic snake other ones say well I'm afraid an elephant is a wall the third guy said well you both are wrong it's a it's a rope it's a big rope so so on and so forth each and every one of them started giving their own version of what an elephant is the last person said an elephant is something which is smooth which is hard and it's like a spear it took me a while to realize in my professional career that I was the same blind man because the code which I was developing which I was calling a domain model was nothing but a technical soup and a lousy one mind you I was developing classes which have bunch of attributes getter setter annotations and more annotations to make sure how the object can be saved properly to the database I always says to my friend that code is like water it reflects your dedication your hard work how much serious you were when you were developing it however in my case although I have the best intentions my code was not reflecting what business wanted it was reflecting more my perception blurred with the technical vision and the reality of business was something entirely opposite fortunately one day I came across this book called domain-driven design and when I finished reading this book it changed the way how I operate an approach software architecture this book told me to change my focus from technology to domain and if I have to summarize DDD in four points how I started applying it was first thing I did is I started placing behavior in the domain model I make sure that my code will reflect the reality of the business so I started working with the business people to understand what is the reality of business from the eyes of business and we started developing a common vocabulary which we can use throughout the organization all the way to my code I started using aggregates to provide unit of consistency across my domain model and finally I needed to define my bounded context I needed to protect my domain model because a customer for a finance department was entirely different than what player marketing would imagine and both of those definition have to exist in the same codebase so I needed to work on my bounded context I mentioned on just the previous slide about aggregate so if I have to define what an aggregate is in one sentence I would say they are group of objects or entities that naturally belong together and they can provide the consistency's unit across the domain model so you can update and delete things and changes at the same time that's all what I have to say about DDD now I would like to discuss about an architecture pattern in which events are considered as the source of truth and basing on those events some components can rebuild their own state at least that is what most of the people actress remember the story of the blind man which I told you there is one important detail about event sourcing and in event sourcing it's not about just publishing events the component which publishes the event is the same component who will consume the events so put it this way the event have no other the the component have no other state except the events stored in the event store let me walk you through an example so in a traditional app let's say we want to order a burger and at the end of the day we will have something like this in our database a row where we will have a order ID the amount what we bought the product which we walked and maybe the order was shipped we can have tons of more information but at the end of the day this is what we can have in a database row let's try to see how the same map looks from the point of view of event sourcing so if we have built the same app using event sourcing we can see what actually happened so first thing what happen is the person added a veggie burger then he decided to add a salad then he decided to add an ice cream oh and then wait what happened the person removed the ice cream the person removed the salad the order was confirmed and finally order was shipped as you can see from the point of view of my business there is a whole series of events which took place in a blink of an eye and if all this information would be accessible to the business people they could react on it and tell us what is the best way to engage the customer maybe the reason why a customer decided to remove the item was maybe the delivery time was too much or maybe the delivery cost was too high actually all this information is a food for machine learning and big data in real world companies like Amazon went to so much extreme and they are like utilizing this thing to such an extent that they will even ship a product before a customer have bought it they are so good in predicting it and not only for the groups they can predict an individual behavior of a person a natural question now which will pop in our mind is where do we store all those events and the answer is even store and even store make sure to store the events and make them available for the consumers as well as the publishers the publishers are the components which publishes the events the semantics of event store can do there is not a lot of things an event stores should do but it should do three things which it should excel in first it should be able to append things fast and since aggregates are the one which produced the events it should also validate the sequence number when it is writing because you don't want to get into conflicts if you are running application on two different boxes and finally which is very important is that you want your event store to provide you the access for information as fast as possible so you can replay all the events and a blink of an eye there are a bunch of implementation available one is by event store by drag young another one in 2018 is launched by ax and DB you can also use a relational database no one is stopping you however remember if you are going to do events sourcing at scale a relational database are not the best fit speaking from experience or you can roll out your own the final part of the puzzle which I want to discuss now before diving into the demo is CQRS so in a real world the problem is that domains tends to be quite complex they tend to evolve and and up quite being quite complex and we need different models to address those problems even in the same bounded context and CQRS is a common way to do that separation so in CQRS we can divide things in command and queries and in command part we want to focus on nailing the domain logic and in the query part we want to nail the domain data so what are commands well when you want to do something in the system you use a command so for example I want to send an email I want to order something these are the example of commands commands usually have a side effect queries on the other hand side are all about information the desire for information so when you want to have some information presented on the screen you use the queries so let's try to analyze how how command how CTRs looks like from UI perspective so you can have the commands routing to one side of the system which is command model but in that part of the system you want the people who have very intimate knowledge of domain to work there on the other hand side you can have projections or queries where you can afford to have people who don't have so intimate knowledge on the domain and they can still work and contribute there we can publish obviously command model can publish some events and they can be received by projections however rather than a meeting a event and forgetting about it we can start storing those events and by this way the command model can rely on those events as the source of truth so when command model don't store any state anywhere else it store all the state in those events I hope you'd ring a bell in your head because this is something which we discuss in a couple of slides before this is event sourcing so we can combine CQRS an event sourcing to build highly scalable systems it's all sounds so easy isn't it however the devil is in the details or shall I say the technical details when you are doing CQRS you need to think about commands you need to make sure that commands are routed properly events are delivered in the correct order you need to think about buffering of the events if you implement a knife implementation of CQRS let's say if you have multiple publishers or components which want to consume from the same source you can end up doubling the i/o and also utilizing CPU twice as much because you will deserialize also you can have some race conditions and I'm even not discussing here sink and a sink way to keep the things simple so at this point of time you can either start to think about maybe you know create that sort of a framework and there is no problem you can do that however you have to keep in mind that when you are developing this piece of code you have to divert your attention from business problems to the software problems or maybe you can use a framework like axon I hate to name-drop the frameworks in the conference's but in our case acts and helps to take care of all the non-functional requirements around CQRS and even sourcing and let us focus on the meat of the problem which is our own domain this framework started to develop I guess some time around 2009 and from that time it is constantly being developed and as of 2018 the guys behind acts and framework a form a company and they launched two products so ax & DB is a dedicated event store and axon hub is a product which they offer for messaging so if I have to say what axon is in in one sentence I will say axon is all about location transparency and every other thing is just you know other stuff out there what location transparency means in this regards is that you have a component which calls another component but this component is not interested to know what is the location of the other component and this comes quite handy so let's say if you have done DDD right then you might start building a monolith and as your needs grow you can start extracting those services into their separate micro services and location transparency plays a key role we can they implement it we can implement location transparency transparency using API is like using futures or using messaging patterns so this is the best part of the of the talk because this is where I'm gonna show you the demo so I have prepared a sample application but first I would like to walk you through what the demo application is about so we understand what the demo is all about and then we will dive into the code to understand the bits so the application is simple it's about betting person can place a bet and basing on the outcome they can bring some money and lose some money and there should be no shock that I am speaking about a demo with a batting gap because I work in a online gambling for an online gambling company so the first thing which we need is we need to put some money into our wallet so we need a digital wallet so let's create one so I can create for myself since we are in Poland we can use polish zlotys I can click Submit and the moment I want to click the submit what I want to do is I want to execute a command into my system which should create a wallet and maybe publish some events not maybe it should publish because we are going to see that I am using event sourcing as well so the moment I click Submit it seems the command failed because polish zlotys are not supported so let's go with euro all the Poland is in is in EU but still we use polish zlotys here so let's go with Euro so as you can see this time the command is validated let's try to deposit some money into our account let's put 50k again I'm going to execute another command and as you can see the command is executed and command published some event the event was picked by a projection and the UI is updated there we can use multiple projections can use and listen for the same event so let me show you what I mean in order to demonstrate it I created a management UI ok let's let's try to squeeze this thing in so right now as you can see in my wallet I have like 50k money here you can see the movement I am going to deposit a bit more 3 projections should be updated this this and this please note they all are individual projections so I am putting 50k and as you can see this projection also was listening for the deposit event it get updated this one get updated and this one get updated and there is no restriction on how many projections we should have we can have as many as we want and as much our business demands for so finally I can start placing some bets so since football is going on let's place a bet on France the moment I want to place a bet few things should happen in the system first the money which I'm betting should be frozen so I shouldn't have access to this money until unless the bet have a outcome it can be positive or negative and also my balance should be updated so let's see when we submit a bet as you can see my money is frozen some funds and I have available balance of 75 let me show you again Germany and let's put 25k so as you can see some amount of my money is blocked and my available balance is 50k and basing on the outcome this money should disappear from my account or it should double however I am not liking this outcome because I'm losing like 50k in just two spins so let's try on Brazil to bet and as you can see these projections are also updated the outcome of the bet is again lost damn I'm losing I'm on a losing streak let's try Poland since we are in Poland maybe threads will give me some mercy so this is my final spin all right there is someone upside who listens so as you can see my money which was frozen ended up being double and as you can see the projection also were updated I can also create another wallet so for players I can say for I don't know Adam and I can choose USD and you can see I have another player here listed and I can say maybe put I don't know 50 grand and as you can see that projection is updated so this is the first part of the demo let's let's dive into the code now so we started this demo with some commands so let's take a look on wallet here so as you can see commands are nothing but a simple bow Jose in my case I'm using Kotlin because it is concise and and I like it just for the sake of readability but you can use Java and we can have bunch of fields here in my case I kept it to the bare minimum which is volatility and currency when a command is executed it needs to target an aggregate how do we make this connection well we can use at Target aggregate identifiers an accent takes care of the rest it makes sure that the command will be routed and targeted to this particular aggregate in the similar manner you can see I have bunch of commands like deposit to request but as you can see they all are just simple pojo nothing fancy except this at Target aggregate identifiers so let's try to see what an aggregate looks like so the first thing you need to convert to a class to an aggregate is annotated with at aggregate annotation and it tells Spring and axon that this class is an aggregate I will show you how spring auto-configuration kicks in and an aggregate should have an identifier we can choose here fubar whatever as an identifier but please make sure the name of the identifiers should match the one which you use in the command so if you call here this thing foo for example then that thing should also be called as foo that's what the target aggregate identifier does you can see there are a bunch of fields in this class they are not here because we want to map this thing to a database they are here because they make sense from the point of view of business so in aggregate you don't just bluntly start adding fields you just add the fields which make sense from the business so put it like this if I don't need to validate currencies I should not have this field in the class I should not think about database or something then we can start handling the commands as you can see I am doing some validation that's why the polish zlotys was not supported and basing on the validation I throw an exception here I'm throwing a legal argument exception but in general it's good to throw a business specific exception but this is a demo so I didn't want to complicate it more and once the validation is successful I fire a event which is called wallet created event the name is in past because this is the convention if you want to name your event the event should be immutable and they should depict something which have already happened as you can see the event is nothing but a simple photo again you can use Java if you want to and in the similar manner I have a bunch of other events which have bunch of fields here so in a normal Java class what we will do at this point is we will do something like this in the constructor we will say command dot wallet ID for example this is this is what we are used to do in Java normal but please note that we are using event sourcing here which means that all the state change should happen when the event is applied so in an event sourcing handler so in my class I will have something like wallet created event I am handling this event initializing the wallet ID giving it a currency and setting the balance to zero in the similar manner when I am depositing here for example you can see I perform some validation issue or omit an event and the event is handled and I update the available balance as you can see in the code is really really simple although a lot is happening here but it is simple in the sense of I'm not doing some magic here I am just updating some field the final thing I want to show you and then we will move on the projection is this constructor YY this thing exists here so behind the scene what happens is you have an aggregate aggregate is not loaded in the memory yet so first time when you access an aggregate Accent need to load this aggregate load the definition of this class and all the events starting from the beginning till now gonna be replayed obviously you can have snapshots so then you don't go like ten years before and you know the replay become endless but that's why we need this empty constructor so our action is going to kick in create the instance of aggregate and apply all the events so this is all about command handling let's let's dive a bit on projections so I have here a wallet summary projection a projection is as you can see there is nothing fancy there are no annotation that this thing is projection except the name I can create some method and the important part is the parameters so this method accepts wallet created event and then I can annotate it with at event handler and acts and make sure that the events are delivered in the correct way Here I am using entitymanager I'm choosing h2 database in memory because we are all aware of relational database however you can use cassandra whatever and even you can publish more events from your projection so here I have a valid summary event which have bunch of fields and those events are listened by classes like player UI so what player UI does is when I've wallet somebody events got emitted it tried to update something on the UI and for management well I have a management UI and the same thing here when the wallet summary is created I try to update some charts and things on the UI the last thing I want to say is about Auto configuration so we have action configuration Maxon auto-configuration this is the class I want it to open so as you can see there is an auto configuration so if you are using spring all those things like event bus command bus you know they can be plumbed together using auto configuration so you just drop the jar in the class path and if you have spring these things get kicked in and you can start coding directly there is Auto configuration for many things so let me show you on the framework so those event bus and command but they are not like an enterprise ESB buses you can use particular implementation maybe you want to use rabbit or recently I added in the framework support for Kafka so what you can do is you can just drop in the acts and Kafka library or whatever in in the class path this auto configuration will kick in and it will try to set some sensible defaults for you and you can override them like you override in spring create a property file or yml file and just override the things so till now in our demo we have seen like commands cur simple commands which execute they emit some event and projection get updated however in a real world life is not so simple sometimes you need to coordinate between different bounded context what I mean is let me show you with a demo so back to my demo here let's login for naaku so then I want to withdraw some money from my account the scenario is like this I will request withdraw then I invoke some other service some third-party service and it can give me thumbs up and say hey you are allowed to let display withdraw the money or you it will give me thumbs down which means that I should not allow this player to withdraw the money in our domain it's called kyc know your customer or no I drain and in order to show you this scenario I created the video functionality I can click Submit the request is went to the external service obviously I mock the service and the withdraw is approved however the service is not necessarily need to reply in instantly it can take up to 72 hours for this service to reply and even the outcome not always is going to be positive so here I'm trying to request something which is very big amount like 6000 euro and my video is denied so the big question here in front of us is how do we manage complex transaction management with with this sort of architecture can we use acid I am afraid as it is not well-suited for this sort of architecture because the transactions are long-running and imagine a transaction which is running for hours in a database that's not gonna make your DBA happy and your and your database as well or we can use base which is basic availability soft state and eventual consistency so this model is a different form of transaction and we can use sagas they used to manage the base transactions they can react on event they can coordinate activities between different bounded context as well as between different aggregates saga typically live here in an event handler and what saga is is basically an event handler which can maintain the state that's it and it's a long-running event handler so just to show you how saga looks like I'm not going to go through all the details and so I have withdraw to Ville saga you have an annotation to make your class a saga you can start a saga and a saga can end in various ways so in my scenario I was ending when I was did getting validation okay I was approving the withdraw command and if it was ending in a negative path unhappy path I executed a new command in I have it command I highly recommend you to clone the source code and play around with it to see what sagas are since it's 2018 and we want to you know scale because every you know most of the companies have the requirement for scalability so I took the time to prepare a demo here which can scale here it is so the demo is deployed on Google Cloud I can log in here or maybe let's create a wallet I don't know for Adam oops I was too fast to deposit okay there is a primary key let's say foo okay wait I wasn't on wrong yes my session was expired sorry for that let let let's try once again okay the demon works like this here on the but that that was the front end part but I will like to show you how it looks on how we can see the commands and all those things so here I have a I am using axe and hub I can see how many valid instances I have here up and running and basing on those info I can see how many instances I have here I can also go okay I have problem with Internet let me use my 3G or 4G here to show this thing it was not something which I planned to do okay let me try to connect okay all right so as you can see the demo is up and running I can and maybe one some of you can also interact with this IP here thirty five one nine five nine nine four and nineteen so you can play with it I can submit here it is for my body as you can see I have the same app running on the cloud I have the management view I deployed it on Google Cloud here I have I'm using accent hubs you can see how many services I have up and running here so I can go to the overview see can you guys see this in the back are you guys listening anything okay and that that gives me confidence so let's try to scale this let's say I want to scale my wallet and I can scale it up to eight instances for example and maybe I need to login first let's login okay and let's scale so right now I'm trying to scale one of my services to eight replicas and as you will see the wallet instance are going to be scaled in the meanwhile yeah so now I hope kubernetes will kick in and and start to scale Oh as you can see the number of wallets are increasing life so I'm scaling it live on the Google cloud right now and I can scale any service any number of times here I can also see which commands are getting executed on which machine so I have deployed it in one region and the application is deployed on three data different data centers so I can see who is handling which command here so that comes handy I can look at the queries how many queries have been executed against my system I can also take a look on axon DB so here for example I can see all the events which are happening in the system so let's try to find some event aggregate identifier equals Nakul for example and you can see I have all the history so first I created a wallet then I deposited money I place bad I warn I place again I lost I did some requests and that approve the withdraw vollis was a proof so that is all about scaling at this point I will like to you know the demo was fun the internet disappeared for a while it was not so fun for me but the the thing is if you guys go home and you wanna start doing you know CQRS or even sourcing the question is how can you find commands aggregates bounded context domain events projections policy so on I'm afraid gooble cannot help you here and as much you like stackoverflow it's also not gonna answer you about your own bounded context and domain so what is the what is the answer then and we should do event storming an event storming is more about bringing people together with questions and answers even storming and brainstorming are two different things brainstorming is more about generating ideas and in event storming we don't generate ideas we try to go deeper into our domain and try to share the knowledge of the domain against which we're gonna build in our company I work for Kazuma so we do if I'm storming we just put some paper on on the glass we usually create this sort of a table I didn't have a full picture here because of the size but we usually put the stickies and say which sticking me mean what the people we give them sticky notes some marker people can start placing the events starting from left all the way to the right as the timeline the discussion evolves some of the ideas and up in vin and this is the most beautiful picture from the full presentation which i like because this shows how much amount of work i could have spent in building a system and then throwing it out and as the as the day progressed and as you come to know a bit more about your domain and dive deep you you can see that bounded context and aggregates and commands starts to fall naturally so in this picture those the paper with the thick line here are they default naturally on the bounded context the end result of event storming is of all filled with lots and lots of stickies but it contained all the valuable information so you know about your commands you know your aggregate you know the external systems you even know your domain events you have a better understanding of your read models and projections in journal you know also about policies so policies is something if you want to do some validations you know which sort of validations make sense and this brings me to the end of my talk so let's recap we should start to think about domain first and technology later DDD is not about perfection it's about domain designing it's about making sure that you design your domain properly CQRS and event sourcing goes hand in hand action can help to put theory to practice you can build your own framework you don't have to use action if you want to we can do a collective learning using event storming and at the end of the day the more we learn about our domain the better we can connect dots and as we know there walks is all about learning and connecting the dots so thank you and you can have your Q&A [Applause] before you have QA just sit here I have to take a photo yes that will be awesome because this is the important part of the talk you cannot screw this up yeah so are there any questions on which thing I have a question because this this ah github link okay here it is okay I have a question because what I found about working with events it shines a bit of shades connection between the components it's it gets cumbersome to navigate through the code do you have any solution for that yes that that's a good question so we also realized the same pain and eye-opener issue for tracing in journal in the accent framework so we can use something like Zipkin to understand how the events are flowing and have a bit better understanding I have a question about versioning of events how do you version your commands or events how do you handle different versions running or replying events of different versions so versioning is quite tricky with events so one of the things which people do is they can create version 1 version 2 and you can use up caster so you can cast the event from version one to version two however from my experience of casters are not very helpful when you have like billions of events it takes time and you need to evolve so what I usually try to do is and I'm still working on that stuff is make sure that you can make your events backward-compatible and also put a lot of focus on even storming because then when you are designing the events you are working with your domain expert so the chance is to screw up you know still exist but the surface area is very very small and you can use something like Avro for example so you can have a schema linked with a particular event and then you can evolve it or you can also one of the other things you can do is you can just if you don't need all that event you can just you know migrate from version one to version two so there are depending on the context you can use there there is some solutions available yes does axon have onion support for unit or integration testing for what for a unit or yes yes so I didn't show the testing here but you have they have the approach of given when then so you can create an aggregate and then you write tests like this given when this command is executed expect these events are emitted so you they have that support are there any more questions I've got one it does it seems to me that um events are something ephemeral and a database a good database which has been properly created I can reconstruct all the events in other words with your salad right if you add a salad and take it out I use like of a double counting entry type of system I can see I've put a plus to go in and a minus of the salad out so I could see I can infer the invents why would I want to focus so much on the events so the thing is I can see that you want to decouple systems using an event-driven architecture but it seems like you're just duplicating data which is ephemeral all right it's a it's a good question but you know the reason why the accountants don't use eraser they have a ledger does account and delete and plus and minus no people are doing this stuff from long time only we in IT are realizing it so to answer your question on a more technical level the thing is events are immutable and when you have an immutable system you can recreate everything so when you do plus and minus yes that times you know this thing happen but if let's say you have all the events from beginning till end of mind you can interpret eight that info in future as you want maybe you can feed it to machine learning maybe you can give it to AI or maybe your business want to enter in a new market and they want to do some risk analysis one more thing which you need to do is audit log in in various domains you have to make sure that things are audited properly so in our top domain because of compliance we have to do proper auditing and there are many bunch of scenarios which I can say does it answer your question all right any more questions all right so there is one you mentioned that the relational database wasn't is a no-go and this is out of your experience could you put more information about it all right I can say what went right was that we started very fast because everyone knew relational database but what went wrong is a long story actually I was thinking to do a talk on pitfalls of CQRS and that database was one of that so the thing what happens is relational database are quite good but when you are storing the events basing on if you use XML or JSON first of all the payload can become bigger or smaller but that system tends to produce lots and lots and lots of event it means your database grows a lot in size and if you want to replay all those events let's say you know I'm afraid the throughput of relational database is not there yet because they are meant to do something different and we are using them for something which provide fast right access and files read access and usually you don't want to maintain even indexes inside and when we reach at the scale of you know billions of events we started seeing the pain of relational database so you need a dedicated DP a DBA who can maintain the the database and make sure that everything up and running and even a small update in the event store let's say you are updating a version from X to Y only that DBA can do and the throughput is the real problem so for me the replay because all this becomes useless if you cannot replay your events in a blink of an eye if I tells you that you can replay something and it will take two months you already lost your business case so you need to have something which replay as fast as possible so replay time was the big problem so this brings me to the end of my questions and answers if you have more I'm gonna be around so thank you very much for attending my talk [Applause]

Channel: Devoxx

Views: 5,155

Rating: 4.9259257 out of 5

Keywords: DVXPL18

Id: hkJ29ER1EZU

Length: 50min 15sec (3015 seconds)

Published: Sun Jul 15 2018