Event Sourcing in Production - Real-Time CQRS/DDD .NET Apps with EventStore

thank you thank you very much guys yeah I can hear myself you must be able to hear me so we are I'm Matt this is Dan let me find my pointer what are we here to talk about we're going to talk about reactive trader clout which is something that a couple of my colleagues came to Barcelona a couple of months ago to talk about teaming and James they talked about reactive trader clouds and how it runs on dotnet core and how it runs in Google clouds and the front end and the WebSocket push and all that sort of stuff but today Dan and I are going to talk much more specifically about events awesome so reactive trader cloud has been put together with an architectural style called of insourcing and in an hour you're going to know all about that and hopefully you don't know too much about it right now and we are using of insourcing not just on reactive trader clouds a little toy open-source application but also a big angry production app that dan and i and a few other people that are here at adaptive are building at the moment which has got 2024 developers on a 50 odd people in the project overall many many years of development have gone on and some still more to come and so we're learning a huge amount about how to build an architecture like this and what we did in react to trader Cloud and how it scales up so this is photos but we're here in the flesh so it's a bit more exciting we are from a company called adaptive Albert gave us a pretty good example a good pretty good introduction thank you very much Albert we've been in Barcelona for a little over a year we've got offices down in tabla mouth and we are about 15 20 people in Barcelona at the moment and looking to grow more and more and we have a range of projects that are happening in our Barcelona office really exciting things and the style of thing we're going to talk about right now is exactly what we build so reactive trader cloud is yeah reactor trader cloud is a open-source application of ours that we put together so that we could come to conferences user groups evenings like this and talk about how we build what we build because most of the applications that we build are commercial applications that are behind MBAs or internal applications that we aren't allowed to talk about too openly so reactive trader cloud even though it's quite small it's open source you can go and look at it and understand it in I guess about half a day if you went through all the code you could understand the whole thing front to that it really does take the same architectural approach and style to how we build our big applications that have 20 developers on them working for a couple of years so a lot of the patterns that you see in the small when we scale them up they're still through and it still makes sense and so reactive trader cloud is what we call a foreign exchange single dealer platform so a foreign explorin exchange we probably all know what that is foreign exchanges the is just the act of taking your euros and going to Britain and buying right now a huge amount of pounds with them thanks to the British political system and it's that bat transaction that I did on the way down from London to here then I got very few euros for my pound arm but it's that scaled up to the entire corporate and institutional market so whereas I bought a couple of hundred pounds worth of euros people are selling and buying and selling millions it's not billions and actually the FX market is a trillion dollar business on a daily basis so you go from me buying a few a bit of spare change to buy some dinner and some drinks and I'm a Bologna to Apple moving all of its foreign exchange money or all of its all of the euros that it made in the last month back to America and the US dollars for example so it scales all the way up and down and we built FX FCPS for our files but what is an FX ftp so we've actually got one that we prepared earlier that we can show you a demo so this is an FX FTP it has some pretty simple components that you see in every FX FTP pretty much every investment bank foreign exchange firm will have one of these that they build and give to their clients so what does it break down to it breaks down to into a few really core key components we've got pricing here which is all the prices being sent from the server running in Google Cloud to the UI we've got the ability to to do a trade I can show you how to show you how to trade by looking down here so we do a trade you saw a flash in this part of the system down here of the apps down here which is the blotter the blosser is just the record of all the stuff you've done nothing more sophisticated than that and then over here on the right-hand side you have D analytics so this is keeping a track if you're a trader of when you purchased some amount of New Zealand Dollars for example there when I took my US dollars bought some New Zealand Dollars and then held onto my New Zealand Dollars presuming that I wanted to trade them back into US dollars at some point in the future this tells me if I've made money or lost inning for lost money if you're just like me coming to Barcelona to have some drinks you're not going to you're going to spend it I'm going to spend it on beer and food guarantee it so I'm not going to have any to take back with me but if you're a trader or an investment bank you're sitting in the market supplying your customers with slow with giving them liquidity and allowing them to do the transactions they need but you're also trying to make a bit of money out of it or else you can't stay in business and that is what we see on the right hand side so this is the UI and when we start talking about in the in a couple slides time how we map the UI and the functionality to an event sourced architecture you'll see that the components that we broke it down to map really closely to the UI and interestingly enough what you find is that these components are actually map really really closely to parts of the business that are inside the organization so if you trying to work a mouse so we've got three distinct components of the application and this is actually pricing and trade execution and they really do map to different parts of the organization so one of the really nice things about event sourcing and CQRS is that you get to split apart your application into the parts of the organization that you're serving that you're building the business functionality for and here you can see the app pulled apart into different pieces that really do map to the organization there is definitely inside very large investment banks a whole organizational unit probably 200 people responsible for getting those numbers onto the screen and nothing else then there's another part of the organization maybe 50 people that are responsible for what happens when you click then there's probably a thousand people responsible for all of the stuff down at the bottom of the screen in a proper application you could right-click on one of those trades and you could do a whole lot of actions to them what you call a post trade amendment so you could well I'm not going into what you could do but there's four or five different things you can do to an FX trade that require a huge amount of back-end support and processing and on the right-hand side well there's probably 500 or so people just responsible for generating the analytics providing market commentary like math libraries quant libraries all that sort of thing to provide you with a really accurate view of what your investment is going to do otherwise so we've broken apart the application and it really doesn't match to the parts of the organization that you you serve I've spent my life building my life I've spent quite a few years building complex applications and trying to figure out how to map but stop the stuff that I'm building and I'm sitting in an IDE and writing code to the part of the organization that is responsible for it and until we started doing using event sourcing I really struggled with trying to keep them all apart because everything kept getting smushed together and it was really really difficult so now that we've seen what the app looks like let's pieces back together so that Dan can use it later on so what does reactive trader cloud if you take it off the screen and stop looking at it a web browser and you look at it in in an architecture diagram well what you see is these different parts of the application that I just talked about broken out into components so we have trade execution which is where you click and do a trade reference data that the set of currencies that you're able to trade prices the price speed is the prices moving up and down the numbers that you can trade on reference data is again the currency so we've written into the into events or the currencies we can we can trade on now we can read them out the blosser is showing all the trades that have happened in the past and analytics is showing the PNL and the graphs and charts on the right-hand side so it's really nice that if you go to this github repository and you have a look at the code base which is in got net so that everyone here will be able to understand it the code matches the diagram matches the UI and it matches the organization which is pretty cool dan can folks hear me okay hello Amy now cool so my name is Dan and I'm a server-side developer at adaptive so I'm just going to go into a bit more detail in the topics that we're going to cover in this talk today so the app you've just seen we're going to talk about the server-side technologies that we use we're going to give some background on some of the design concepts that we're using reactive trader which are event sourcing domain driven design and CQRS we'll also talk about how we can implement process managers which also known as sagas if some of you may or may not have heard of to coordinate a business process in a robust and resilient way we're also currently working in a real world project at the moment which is very large and complex we are using these technologies so we're going to talk a bit about our experiences working on that project and some of the lessons that we've learned along the way so talking about some of the technologies that we use obviously we're using dotnet core so the back end is completely built using dotnet core version 1.1 at the moment but there are plans to move version 2 in the future we're using events tour so we'll talk more about that in a few slides of time but this is like a functional database and it fits in very well with the event sourcing and it has builds for both windows and ubuntu and has a dotnet core client API we use a tool or technology called cross crossbar IO which is an open source message broker we use this to sort of communicate between the different components of the system this is built using what's called web application messaging protocol or ramp and to sort of use cost bio we use an open source project called ramp sharp which is a c-sharp implementation of WAMP and it provides both the client and server roles we also use reactive extensions which a lot of you in here may have heard of these type of applications are all about streams of data and reactive extensions fits in very nicely with that technology or with those with design methodologies and the cool thing is that all of these technologies are completely open-source and they're available to start working with today okay so talking a bit about event sourcing so there's three things really that I wanted to talk about with event sourcing the first one is the state is stored as a series of events storage is append-only and it's a functional database so so in terms of state being stored as a series of events so in more traditional systems that are built on top of relational database we have what's called crud operations or create read update delete which many of you have heard of and worked with so you can imagine we had a customer entity we might have a table in the database called customer we might have a bunch of operations that might be stored procedures things like create customer or update address and things like that so the thing is with this style of system when you when you update data you are mutating that data so if we call a procedure called update address we'll actually we're actually changing the contents of that table and persisting that changed state and if we did have something like update address one of the questions that somebody might ask is well why did that address address change did the customer change it because there was an incorrect spelling and the street name for example or did they change it because they've actually moved house so in in domain-driven design we have what we call domain events and the idea is that we capture the intent of what actually happened so in a sort of event source system using domain driven design we might have a customer moved event or an address changed event so we capture the intent of what actually happened so going into a bit more detail of that so you can imagine that same data how we might represent that the series of events so the first event might be customer registered which is something that happened in response to the customer signing up to your system that event would have a payload and that payload would contain all of the data that you see there in the customer entity on the right-hand side so a little bit later the customer mint may actually move house so there might be some user interface in the system where the customer can actually record that they're moving house and then an event will be raising the system called customer moves and you can see that the address there is changed to the updated address and then a bit later on user may actually change the email address for one reason or another and you can see there that the email address has been updated and then perhaps the user got tired of the system and decided to close their account as you can see we don't go back into lete events we just append a new event called customer account closed so which brings up to the next point that storage in an event always depend only so when I say append only it's in a pen only log of facts new events I only ever appended to that log and nothing is ever deleted so for example with the customer account closed we just add an event there so this way we always have a golden source of truth in a system and valuable business meaning can be derived from that source of truth even well after the system was originally designed and developed for example the business may need a report that answers the question how many customers have moved house after three months of signing up and you can easily derive that information from this source of truth so and within traditional system you can of course do this as well but you know when using relational database management system you need to factor that into your design and development and there are many different approaches for example triggers for auditing as one approach whereas this is really a first-class concept with event sourcing it's also great for debugging so if you had a bug in production you could actually get a copy of those events and then play them through into a development environment with exactly the same code base running in a development environment and and just play those events in and actually watch what happened and find out why there was a bug the other point is that an event store can be considered as a functional database so if we have a function that takes two inputs the initial state and an event and then gives you some new state we can basically build up our current state by nesting those function calls so for example with those four events we just saw the very first function call would be the customer registered event with the initial state which would just be an empty customer object then that would give us a brand new customer object and then that plus the next event would serve as the input to the next function call you can keep neck nesting these function calls and build up to your current state and those of you who are involved in functional programming might know this as our left fold so taking a bit of a step back and talking about events that's really are a type of message in the system that that sort of says something interesting in the busyness happened and it's also probably worth mentioning at this point that events and commands are not strictly a DDD concept though many systems that are built using DDD will use events and commands so in DDD things that affect or change the state of the system are modeled as domain events so in that way events can be described as messages that represent something interesting that happened in the business and they're always named in the past tense so like customer moves or address updated or something like that commands on the other hand are an instruction request to our domain to do something and they're always named with a verb in what's called the imperative mood so basically through something like move house change address etc so that they're an instruction to our domain to actually do something and so that leads us to aggregates so how do these events actually get into the system in the first place so when we talk about aggregates aggregates are a group of related objects that form a consistency boundary when we say consistency boundary it means that everything that the aggregate represents must always be in a consistent state aggregates a sort of entity where business rules are enforced so any change that to the domain always happened through the aggregate so in other words there like a gatekeeper for our domain and we talk about events and commands before so aggregates apply commands and rays events and when using event sourcing those events are actually added to our event store and an append-only fashion so in that way we can view an aggregate like a function that takes a command and some states and then outputs one or more events so going back to our example it probably makes sense to model the customer or a customer account we might call it as an aggregate and we can see the command going into that aggregate registered new account new house closed accounts and then the events that would be raised subsequently after processing those commands cool thanks thank then so we just talked about events commands aggregate DDD secret do we mentioned CQRS yet not yet not yet okay that's coming so we've just introduced a whole bunch of concepts and it's really hard when I started learning about an insourcing two and a half three years ago I really struggled to put all these different concepts into something that made sense in my head that I could reason about what we've done with event or three axle trailer clouds is hopefully break down the concepts and make them really straightforward and easy to understand I think a good way to think about the benefits of event sourcing is to think about how it makes what looks like quite an intimidating and complex problem actually quite simple when you break it down to it put smallest part so if you think about how many events make up the application reactor trader cloud that's a that's a number we can go and look on the codebase and figure it out if I think about what the relational database structure would look like I end up with in my heads quite a complicated structure with maybe ten or twelve different tables if we think about how many events there are and reactor trader cloud is three different types of aggregates which we're going to talk about in two seconds and then there's about seven events in total so there's only seven events that make up all of the complexity of the application you just saw which to me makes it seem really unconfessed and really simple because you can really easily keep seven events in your head and reason about what they do and where they flow from what and and what how they interact so those events okay so if I stand right on the corner of the stage so you guys can see this frame so we've got currencies currently spear pretty clearly a first class concept than an FX single dealer platform a currency is euros or US dollars or British pounds New Zealand Dollars currency pair is simply an instrument a description of something that we do trade when you're buying or selling one currency into another and so we have a series of events that you use to put the currency pairs into reactor trading files for the first time so we have currency pair created currency pair activated and currency pair deactivated we need to separate out the concept of activation from creation because you can have currency pairs in the system that you aren't currently allowed to trade on so you need to be able to deactivate them which means you also need to be able to activate them the first time around if we look at what a currency pair is well it's a bunch a bunch of attributes and facts about it and then the activation and the deactivation is really simple it's just an event that literally catches the intent of what you're doing or what it means when you see it in the system that means euro dollar was activated or deactivated and enough information to tell you about the currency pair data that it applies to so really really straightforward we have events related to trade we have a trade traded event now this trade is this trade graded event is the most complicated event in the system and it wraps up all of the information you need to know about the trade so it has a trade ID so you can refer to it later on and other systems it has the name of the person that traded the currency pair that they trade us the rate they trade it out the date they trade it on the value date of the cut of the of the trade which whether you were buying or selling and whether how much you bought and solves and what that number is actually denominated in so that an FX trade like seven seven or eight there's different attributes and a few events so you can try to create a trade the trade can be rejected or the trade can be completed and that's it and from those small events not very complicated you could probably go leave this leave the meet up and go home and we'll go to work so my morning and write them down again from memory because they're not that complicated you'll be able to rebuild the system we have one last set of events related to credit accounts which are in reactivated clouds so that we can talk about process managers and the challenges of modeling things and events but these are three more events talking about the you can create the critters account a credit account is like the amount of money you have in the bank you can draw down on how much money is on your credit cards exactly that and when you try to buy something you either use up some of that credit so you have a credit reserved event or you've spent too much money and you can't buy it and so we record the fact that you've reached your credit limit you only had 100 euros on your credit card and you went out for a really nice dinner you would have a criminal's and a breach to bid when you tried to pay but if you went out from McDonald's you will be able to pay so you just have a credit reserve event pretty straightforward very simple concepts so those are those are events and I think the thing that we want to get across in this talk to people who are just getting started with the game sourcing it or maybe coming across it for the very first time is that these events aren't intimidating things if you think about how you the important facts about the business problem with the domain or the concept you need to model just put them into the event it's really straightforward if we were giving if we were talking instead about relational databases and someone was arguing about foursomes fourth normal third normal second normal form the fact that you have I've worked on databases databases before we're just this line the representation of a point in time would be five different tables one for the year one for the month come for the day and people take the stuff to the insta greet and make a real make the systems they work with really unpleasant to use but with event sourcing just write it down and don't over complicate it and what you're left with is really lightweight easy to understand event skimmers then right so those events you just saw obviously they have a schema so a well-known format to adhere to and really this is basically like a message contract so obviously you can define this message contract early on and I mean you know teams can work in parallel you can have a sort of a service component being built that consumes an event in parallel with the service that might produce that event important sort of decision to make when when designing your system is whether you have a code first schema or a sort of schema first schema so when I say code first you can use tools like JSON Nets and just use its object serialization to write out the shape of what those events look like which is great and that probably might make sense if you're working on a small system but something that we need to be aware of is that you can kind of break things very easily if you just know through you can resharper you can refactor something very easily and checking some code and then that is completely breaks a downstream component that's consuming that event so in that way you sort of even though you're you don't have any physical binary coupling you have an inputted binary coupling so it you know if you have a very large system it's definitely worth having a scheme of first design we've definitely done that useful on our project back to command query sponsored Liddy segregation CQRS so this is the diagram that matt showed before of rapid trader really it's just you can see on the top there you've got the do you want to introduce the command query responsibility graduation a little bit first yeah okay yep so we can see at the top there we've got bright and at the bottom we've got reads so we'll talk about that a bit more and so some of you may have seen this diagram from the Microsoft online book called seacoast EQ RS journey and really this is just highlighting the the flow of things from the right slide all the way through to the reed side so so secure s it's a natural fit with event sourcing but not all systems at UTC QRS need to use event sourcing and security isn't necessarily isn't necessary of when using event sourcing but but it's a really good fit so we secure our s we separate our commands from our queries using separate interfaces and that this separation allows us to use separate models to update data than what we use to query data in a more traditional sort of end layered type system where you kind of have your database then your data access layer they know you know your business logic layer and things like that quite often any changes you know you need to make an amendment to the way something is written you can quite often break the way things are read and vice versa we see curettes it allows you to completely separate the reading the right side and alleviate those issues and separating commands from queries can have several benefits so we can have separate streams working on completely different parts of the system for example somebody could write the blotter that you just saw and somebody could actually write this service with a trade execution service completely separately so it allows separation in that way it also allows for separate performance tuning so if we find that we're not writing very much but we're reading a lot we can scale our reads I completely separately to our right side which is really cool so back back to that diagram I mean it really is basic but you can just see very clearly there we've got our right side at the top there to actually write things into event store and our read side that reads things probably then store and over to Mack cool thanks man so when we started the big project that we're on at the moment down in iron at the moment we was the first time we'd picked up of insourcing and started using it an anger a couple of people on the team had worked there previously but in much smaller projects and so we were getting into it for the first time in a really big way and we had now on us the hind side of a year and a half there's a lot of things that we got wrong when we started that we would be able to avoid if we went back and a lot of them are about I don't think these things we could have done a lot we could have done differently there's some things we could have done differently but approaching a system architects in a completely different way is a journey by itself and it's something that the whole team has to go on so we use developers needed to learn how to think about approaching the problem in a in an event sourced way but when we were sitting there talking to business elements business analysts and even some of the more sophisticated users it was really apparent that everyone thinks in a relational database way everyone has been exposed to relational databases enough in the last 40 years 45 years that they think well and also Excel which is a relational database on screen everyone thinks about it in those terms all of their problems come to you as relational database representations even before they've realized they've made a solution out of it which means what you're doing when you're developing an event source solution to a problem is taking what you think is a business requirement that is actually a database schema that someone's implemented in their head and described to you and turning that into an event source system it's quite a lot of awkwardness around the impedance mismatch and we've spent a lot of time taking the the bas with us or understanding what it means to talk about domain events and the other attributes of an event source system rather than just talking about the relationships between customers and their orders or trades and they're the things that you need to deliver from the trade as an example people have got Emma hits all the constraints of that a relational database let you enforce so relational integrity you can make sure that if you make a change here that this other part of the system which has a foreign key linking it is updated at the same transaction and that the whole system remains consistent every time run someone runs a query against it if insourcing throws all of that out the window so event sourcing and CQRS give you eventually consistent systems if we look at this diagram you can see that when we make a change the system here by submitting a command the user making a change in the UI it flows through in this diagram four or five different boxes if you think about those boxes as different systems they definitely distribute it on opposite sides of a network from each other of a network connection that means that no matter how fast the system is and how low latency the the network connections are when a user clicks a button to submit the command it can take some number of milliseconds if you're lucky less than 125 milliseconds so it appears instantaneous if you're unlucky up to a second or a second and a half or two seconds to come back around and the user to see the result in that period of time there's an opportunity for someone to do something that affects the change on an updated version of the system in a relational database where you aren't doing C to RS and your view your query about the system happens on exactly the same path that you're right the system happens you will see nothing you'll see the system be locked if someone else issues a read they will get in line in the database to queue either before or after the transaction to get a snapshot of the system which means they prob they will they might sit mates mice make might excuse me they might see an out-of-date view of the system but we may go to issue a right they won't be issuing it against an out-of-date view of the system and that was really difficult for us to get across to people the constraints that people have in mind all of the relational database contains and then the menses world do any of those things it means you need to attack approach problem discovery so figuring out what the system should actually do and how it should be modeled in a completely different way you might have heard of event storming event storming is a way to model event source systems in a in a sort of collaborative way you get all the business stakeholders into a room you have a big long wall that wall is a good example of a big long wall and you give everyone post-it notes and you go out and stick on the wall the different things that happen in the system and you try to stop people from modeling the relationships and the rules and the bits of constraints that need to be enforced between the different bits of data you just get them to talk about what happens and you try to turn that of what happens in to your business events and try to stop people running down the road of just giving you a relational database answer to your question of what does the system do and it's really difficult interestingly enough if you can actually talk to just the users of the system who have never been exposed to SQL and maybe don't really like using Excel you get the right answers it's all the layers that our software development industry has put in the middle of people who have learned all about relational databases over the last 25 years of building software they put all this noise on the users that are just using the system and want to talk about things to the developers that mean something to them are really good sources of information but without access to they'll use those users in it it doesn't really work very well so once you have figured out what you think are the requirements and hopefully teased apart the bits of the problem that someone has told you about that just require a relational database and have got down to the event sourcing view of the world where you're just talking about things that happen and we need to record them because they're interesting you get to the modeling challenge so if we had had relational problem if we'd had a problem described to us in relational terms and now we needed to model it or we just need to put it into the database as people told us but modeling aggregates which are the unit of discourse in an event source world which then just talked about is really weird compared to modeling relationships and we've been doing it for two years and it still takes a while when you approach a new domain a new area of the applications a new requirement to be able to think about it and in the proper way and the way I think about it now is that instead of thinking about relationships between des data and what needs to be kept in sync or made consistent with each other we I think about the ownership of data and what date what other data it needs to be kept insistence istant with so who owns the sort of data what other bits of data will change with it at the same time which is easier than thinking about all the relationships between or between all the entities in your system as we were building the system we working on now we discovered that we got to delay a lot of decisions which made our life as developers really easy then made the point that you get to separate your right side so the side of your system that is concerned with updating the domain and handling requests to do things and book trades and update customer addresses from the side of the system that lets you view custom addresses and understand what trades you've already bought splitting them apart means you can work on storing them writing them in the appropriate way completely separately from representing them in an appropriate way from reading them back and displaying them to the user which means you don't have to make those decisions at the same time for every time if you if you had to make them at the same time what you are forced to do is model your relational relational database in a way that lets you query at at the same time to give you back the information you want to dispose of users and it really awkward to do that I'm sure if you guys ever done any SQL development or your work with the database and picked up someone else's relational database that has been built up over the years and tried to answer a seemingly simple query from a business user you in you can really easily end up with a query that has 5 10 maybe a dozen joins on it like this long lots of Weir's lots of silvering it's a nightmare to build up debug use with of insourcing that all of that goes away because you haven't coupled your ability to view the system from the structure you've stored the system in for for writing to is really really easy so you get to delay those things but you have to choose about what parts of the system live with the other system with it with other parts of the system and that can be quite difficult we found that for our project where we had built we'd split 20 developers into 4 or 5 streams so each development stream as we were doing we were using we are using agile so we have a scrum team of 4 to 5 people they are working on distinct separate deployable units of the application and we made those decisions about how to split things up on day 1 of the project well it's what you know on day 1 of the project about the project nothing so we got some of those decisions wrong which means that as time has gone on some of the boundaries and the relationships about the behavior of the system and what happens and why something happens in relation to something else are really awkward for us to work with now and we're now a year and a half into the project and some of those decisions are starting to catch up with us so we're thinking about whether to go back in and change the models and pull things apart and reconstitute them in different places but at no point do we think we're going to change the event schemas innocently enough we very rarely got an event schema wrong we've invented and vivid 50 event schemers maybe maybe more than that and I think we've got one or two of them wrong and had to go on back in and add something in the past which is that's really impressive so you get the event schema right but maybe the event lives in the wrong aggregate and as a representative of the wrong part of the system which is it again it's a totally novel challenge and not what we expected to happen if we was going through all this so the points are sorry for laboring the point that this is I wish someone had told me this a year and a half ago events and processes are more important than data and relationships so the processes are what happens in the system and it's what the business owners think about and your users are concerned about and the events are the facts about those processes and they are far more important than the data and the relationships and so talking about processes leads us on to process managers right so purchase managers um some of you may have also heard of these being called sagas so process managers are sort of entities or sort of artifact I guess in your in your system that coordinate business process let me say process that may be a long-running process like something that goes over a few days maybe or even months to a process that might be very short-lived maybe even less in a second so we talked about aggregates before and aggregates are very isolated they take care of themselves like for example the customer aggregate takes care of things like updating in the tress moving a house closing an account it's very separate so aggregates are kind of very sponsible for applying commands as we said before and then raising it events process managers are kind of the inverse of that so process managers are responsible for receiving events and then dispatching commands so they're kind of like they sort of sit between aggregates so if you have some sort of busyness requirement that spans multiple aggregates and you have some kind of process then the way you would solve its problem is with the process manager so so we talked about events and commands being messages and you know as all of us know in a messaging sort of messaging system we have some issues we have two main issues really because they are duplicates and ordering so there's no such thing as you know receiving a message exactly once only once there are you quite often you'll get duplicates and also ordering like there's there's no real guarantee that you receive messages in the correct order things are received out of order sometimes so with the process manager you can model that as a state machine which can handle both super kids and out of order messages so for example if an event is received out of order a transition to the next state can be denied until the first expected event arrives and for duplicates you know even if the same event if one event triggers a state transition receiving that event again and you're already in that state so it doesn't matter so in that way a process manager can deal with those issues in a messaging system so also when we use events tour as we talked about aggregates before we've essentially saved down the series of events that make up that aggregate so that's that's how we persist our aggregate similarly we can do the same thing for process managers so we can save the series of events that are involved with a given process so if we go back to the reactive trader example as Matt talked about before we have a couple of aggregates in our system so we have a trade aggregate which is responsible for dealing with booking trades and dealing with you know whether a trade should be rejected or done successfully and then we have a credit account which is responsible for checking you know does this trader have this account have enough credit in it to book this trade and if it does it will raise a credit reserved event if it doesn't we raise a credit limit for each event but how do you coordinate that process and this is a little box in the middle there it's called trade execution process manager and so you can see it sort of sits in between those aggregates there so those arrows in yellow are commands and arrows and black are events so the first thing that happens when you click on one of those tiles is a book trade command is issued that is applied to the trade aggregate and then we get a trade created event from a completely separate part of a system we have this trade execution process manager listening for events it cares about trade created events when it receives one of those events it will say Oh somebody's trying to book a trade I need to go off and reserve some credit so it will issue a reserve send the reserve credit reserve credit command and that will be applied to the credit account aggregate and this aggregate is we talked about before is is a consistency boundary so either that you know even at that count has enough money or it doesn't to make the book that trade so we can check whether we have enough credit available if we do then as a result of applying that command the credit account aggregate will raise a credit reserved event if not it will raise a credit limit for each event and our our trade execution process manager is also interested in those two events because it will respond to that event and then send the command saying to either complete or reject the trade and then in response to that the trade aggregate will raid those two corresponding events trade completed or check trade rejected and you can see in that diagram that the aggregates are very isolated I don't care the trade aggregate doesn't care about any of the credit related events or command it just cares about booking a trade and completing or rejecting rejecting the trade and that's it so it's sort of the process manager sits there and coordinates process in your system and so I'll give a little demo now of so in interactive trader we have modeled that interaction as a process manager using event store and I'll just give a little example of what that looks like so how do I change screens here how do we go they're with me unless that's all okay for this little console window from the right hand side I'm just going to use that to reuse Cuban Eddie's as our container manager we have doc containers for each of the components I'm going to run a few commands on the right-hand side here I won't go into too much detail of all of that stuff so this stuff is just what they're getting set up this is this web app is running on Google Cloud it's not running on Dan's laptop we are controlling the cubanía t's cluster that is hosting all of these executional execution herbal services then one thing I forgot to do earlier if you just put the mouse over the little green check in the bottom left hand side oh yes so yeah that's really hard to see but you can see here that we've got these are all the system components that are actually running in the environment at the moment so those things the blosser pricing reference data broker execution and analytics mapped to that architecture diagram that we saw before the right side and the read side and they are you can see alongside of a little number and that tells you how many instances of them are running so then going to through something I'm not going to ruin the surprise but you'll see that when we can when he does something we see the numbers change and the behavior of the system changes in response to a failure or something going wrong with some part of the system and they're no longer being an instance running of a particular component so I'm just going to bring up another screen here which is our web interface to events tool come back to that in just a minute so for sort of demo purposes in this application we've configured a few things like some delay to simulate something taking a longer time than usual and we have a specific currency pair here which I believe is euro JPY which we've configured to take five seconds the UI has a time out of five second so if I actually try and execute that trade there you'll see it's pending for a long time the UI times out and then eventually it will move to the done state leaders down the bottom there which you should have seen just down here so the idea of that trade there is one 7:02 so in terms of process managers let's have a look at the events involved in that particular process so if I unfortunately this part of the UI is not reactive so so we can see we have three events there that were involved in the processes of booking that trade so we have the trade created event that happened first down the bottom there a credit reserved event and then a trade completed event do you just want to click on the Jason yet link here to show people that we weren't making any of that up and so this is the event stored in event store trade ID trader name exeter exeter those are the samples that i showed earlier this is exactly what is stored in the database no more complicated stuff than that cool cook that off sorry so what i'm going to show now is the fact that our process manages as we should have managers step in the process we persist the events that we've saved into event store and that allows our system to be resilient so it a certain part of the system crashes we've persisted the state of that process and then when our system comes back up again we can just resume running that process so hopefully if i'm quick enough with my mouse clicking so on the right hand side there i'm just going to run a command that will scale number of trade execution services down to zero so there's no service running before I do that I'll go ahead and okay try that again Oh book that tray so eventually you should see that this little icon there at the bottom right has become amber and we can see that the execution service is not running we can see that at the very top row in the bottom left blotter there the trade is still in the pending state and our UI has responded saying that the claim that exceeded the execution status is unknown it's probably worth noting that everything else is still running so we're still getting price ticks come through and every other part of the system is still working we have disabled the ability to execute trades so I can't actually click and execute anything because that service is not running so if I go back over to my console now and scale back up to one instance of that service which includes the process manager which includes the process manager after some time the the icon at the bottom right should go green and because we've configured as trade take about five seconds after some time we should move to the dumb state and there we go so basically we show that our process managers can coordinate a business process that spans multiple aggregates and we can use events store to persist the state of our process and call resiliency so different parts of that system can crash we can bring them back up and that process can continue so that is all for the demo demo God smiled on us - and over to so once we had figured out which took us quite a long time how to get requirements and do the modeling we have we've seen quite a long time not working with process managers because we came to them a little bit later on and the project once we once we had them modeling things I would them came up became a little bit easier now that we were able to understand how to make these long-running processes work in an event source world but we still had a couple of other challenges so one of those was the architectural challenge so when we started the project the their client heads had no infrastructure no no development environment no production environment no software development environment or way to build and develop software at all so we spent about a month when we first started before we even got to the business requirements and started listening to users and figuring out what to build we've spent a month building our low-level infrastructure so if you think about the low level and structure involved in getting data into and out of a relational database and this is something I like to I like to think about but I honestly think this is such a small simple way to build applications than using a database and I hope never have to have to choose it as a normal database ever again but if you think about what an ORM is and how big and complicated it is people have written books about in hibernate which is the port of hibernate the java the java ORM that are like that they've written dozens and people have written dozens and dozens of different object relationship matter relationship matters all trying to solve their particular pain of getting their data in and out of a database and into an object-oriented world in a in a in a relatively simple way so when we started when we picked up a Vince store and saw the api's for writing events and reading events we were really amazed at how straightforward it is when all your data is is JSON documents it's kind of not surprising that without the API needing to understand anything about the structure of your events it's really quite straightforward to write them into a database into into a Vince store then read them back but that didn't stop us it up and doing all the wrong things the first thing we did was tilt our really simple lightweight api's to read and write data into events tour and wrapped it up in what you could call it the framework so a whole bunch of generic interfaces class classes factories all the things that you see in an ORM world that make it really simple to do hard things we made an event store it really hard to do simple things but that's all getting data in and out of event store is it's really really straightforward we spent a couple of weeks doing the wrong thing in a couple of weeks tacking back over trying to do the right thing and making the way we've got data in and out of the Wenzel really easy you can go and look at the wreath read and write X data passes happens in reactor trader cloud and they are probably about 50 percent of the codes that we have were compared to what we have an hour and hour big full project our big full project has 20 developers for different development streams millions of lines of code and the data access layer that we have in it is twice the size of the one that we have in reactive trader cloud and that's what we need and it works so the simplicity you get a persisting and reading back your data is absolutely amazing but once we had the patents for getting stuff into and out of it we still needed to build some layers on top of that on the right side we have talked quite a lot tonight about commands command handlers those are the things that take the commands map them to an aggregate to see whether or not you can perform the command and then tell the tilde the user the result of the command aggregate excuse me aggregates that's a consistency boundary that we've been talking about and process managers that we saw just a demo of that's all the stuff on the right side on the read side we have a range of different things that you see in reactor trader cloud as well so you have in process streaming dictionaries so because events store is its streaming beyond anything else you can set up a subscription to event store and be told about events as soon as they're written on the right side you have an axe I'm not like a message person that it all you back and give you updates that what's happening in the system you can take that into end reactivity and build data structures on top of it that let you push those updates all the way through to the UI so one of the things we have inside reactivate a cloud is something we called an observable dictionary and that's the library that is available on our github is a it's an open source project that takes a traditional dictionary so a key to a value and makes it a key to a stream of values so instead of asking for a customer and a dictionary and getting back to current customers in the dictionary what you get back is a stream of customers that will give you a new custom every time a customer record is updated so that quite small simple primitives that looks like the dictionary is needed easy to understand you can build really complex interaction patterns where the UI has been constantly updated in response to events being pushed on pushed in on the right side we also have out of process persisted remodels so we have got pretty much all of our application state running inside in-memory in our applications it's stored on disk and event sort and then when we spin up our application all of our different systems connect to events or read in all the data and store it in memory and start working on it sometimes it's really difficult to write the sort of structures you want to write to query and understand the data so a really good example of this is doing natural language search so if you have trade addresses users all this sort of quite structured data but you want to let users search it bus by typing in words you don't seem to go and write Google yourself that's a really slow way to get anything done what you do is you use something like elastic search the elastic search is a really nice natural language search engine that lets you customize and partition data and filter and query and do really natural queries it uses humans can understand right not just developers so what we have is elastic search that sets connected to events or we take a bit out of event store write them into elastic search and then we query elastic search for data so now we have a really complicated internal model inside elasticsearch that we don't care about at all we don't understand the complexity of elasticsearch we just use it like a little black box we feed of in CERN and query earth and do natural language search on top of it really really nice if you have a relational database you'll be doing the same thing you'll be taking things out of a relational database and copying them into elasticsearch to serve from them what you'll try to use the natural language searching capabilities of your SQL database Microsoft Excel actually has really good natural language searching capabilities but elasticsearch is a little better and that use of the right technology to solve the right query or read model side of your problem is a really good story for event or thing and you see it time and time again it works really well so we had requirements modeling architecture and then we had our development challenge so as I said earlier we've split our development into five streams to deliver the whole application in parallel people don't tend to build there's a lot of people building the sort of proper sort of product they're building for our client at the moment and they're very large complex things you need 2025 developers to do it in a couple of years you can imagine how big and complex it is but to do it in any reasonable amount of time we needed to do a lot of things at once I don't know how to have 20 developers all working in the same codebase at the same time over two years if someone does know how to do that please come and tell me because every time I've ever tried to do that people just in their whole life breaking each other's compiles ruining the build not being able to do anything getting frustrated hating their teammates what I think we needed to do is split the development apart so that one team or one stream going slowly didn't slow everyone else down but just because we've so we picked the way to break those things apart like I said on day one of the project because we needed to assign people to teams and start moving forward so we picked these functional areas very very early on in the project and the way people communicated between them was via events so we divided the world up into four different streams and communicated by them only with events and as I said we only if it's wrong a few times and we spent nine months developing with 15 to 20 developers all developing in four streams each building huge really complex parts of their system but only ever talking about how to communicate with other streams by a meeting we had once a week and events it looks exactly like the events that we showed earlier stored in a wiki we didn't have a dotnet contract or anything like that or a schema stored in a relational database we just had events written on written on confluence on a wiki and the thing that really blew me away and that made me think that maybe there was something to this event sourcing event style form of building systems was it after nine months of development of between fifteen and twenty developers we took a snapshot of all the four streams and integrated everything together and it took us one and a half days of one developers time to fix the different versions of events that people thought they had written to and have it fully integrated into an environment that works perfectly one and a half days of one developers time after nine months of 15 to 20 developers time I was really pleased that it worked but absolutely amazed that it had held together like that and that comes from the way we had a loose schema exactly what Dan was talking about we went schema first rather than kosis we defined the scheme is on a conference page on a wiki page and made it really easy and lightweight made them really easy and lightweight to understand and talk about and it held together for for that length of time and we're now another nine months and we're about to do another fully synchronized version and we're actually almost fully integrated at the moment the whole time and it again really straightforward so that was that ah just on time any questions I have here for Forks are before our technicians here is a whole bunch of links through the reading I don't expect anyone to copy those down and we'll put this up somewhere thanks very much question when do do you thing for your let's say your Nets project a big project that where you have different domains the next time you could you use different heaven sourcing database for each domain we separate schemas or will you do the same single single big stores I think it would depend on the organization that we're building it for if they appear organization breaks down and two really distinct business areas so there's an office in North America and a office in in Europe for example I would split them apart because I want the software to map to the organization but if they're all relatively in the same place and thinking about the same part of the domain then I would do I would do what we've done now another thing at the beginning you mentioned that the knowledge is you use layering store and like this one you say you use to communicate between service agent yeah crossbar IO what was the difference between using this and use event sourcing events there and sort that arrays for communication because it's events right so so so this is the language this is the persistence the stores the data on this this lets you communicate via from that from the front end so the web page that's running in a browser with the.net service that is the the right side or the read side if you write directly to this thing from the client which you could do you don't have any business logic to implement the rules so you need a layout that implements the business logic and aggregate everything together and individuals UI so the html5 page can only should only see my trades but the system as a whole has trades from lots of different people and they can't exist just in my system APIs exist and some service layer somewhere yep so that that crossbar i/o is UI and server and they talk directly to events or using a dotnet API cool any other question at the back sorry you were hello so this architecture leased out to design a system would you say that you think you can apply it in majority of challenges that you want to implement or would you hold back and think like oh this works well for example this trading system but other solutions you think like here it would not work because you're you know looking a little bit on the flip side like yeah what would it think I think that it has it adds a layer of complexity to what you're trying to do there's no doubt that just because of our big past experiences doing something that we've done before relational databases NBC RM is really easy but to do a system that this big and this complicated and maintain and manage to keep up velocity it's a really good set it's also a really good set for the problem domain so the system is a something called a commodity trading risk management platform which is something that big commodity trading firms use to manage their entire business operation so it starts with one end where you have a trader that puts the trade and you have at the other end with maybe a hundred different people doing different jobs all the way through the middle at the other end settling an invoice and paying someone coming so that's all the way through and in the middle you've got this peak you set that involve putting things onto ships and onto trucks and putting them into silos that is a very it actually match really well to in a big driven world lots of stuff happens after lots of other stuff and you can trace a lifecycle all the way through if you have a system where there's actually a whole lot of unconnected things happening that don't really have anything to do with each other and aren't linked to other things maybe this isn't such good I can't think of a good example but yeah it's difficult I should just take this hammer every now again before you approach with caution okay thank you yeah at the back sorry we're going to be the process manager I have a question because if you don't show there what happened is for instance you reserved the the credit and then for some reason because it's fail and then you have a reservation does that mean how do you handle the kind of rollback cuz I mean when things work is it it's clear but with things for some reason some of the company if you've returned some money did the King know for some reason complete the transaction the trade what's happened then who is in charge of rolling back and say okay free that money yeah rolling back up so so like we sort of have a best-effort approach of resuming from where we left on really of course if the service went down before the message was able to go over the internet and get the server then you've lost that message but whenever we sort of transitioned to a new state by applying you know responding to another event and transitioning we persist that and so we're kind of checkpointing what state we're at in that process manager so in the example that we showed there obviously the system the process manager responded to the trade creative event and assisted the state of that process manager we could have been very quick you know milliseconds I guess over the over the wire here and and then the service crash at some point after that so that state when we bring the service back up again we basically play through all of those events again and another sort of I guess piece that we didn't really get into detail was event store event store has sort of a thing called persisted subscriptions and this is essentially like a message cube which is persisted and so you can sort of set up this specific subscription we have one called trade execution and we've configured it to be interested in trade created credit reserved credit limit breached events and and event store knows what event is dispatched so eventually will internally keep a check point of what event it successfully dispatched and our process manager will either acknowledge that it's received those process and receive receive and process those events if there's some failure it will put it into what's called it's dead letter Q well it's called parking in event stores and or you can the server can also choose to skip a message that it doesn't know about so basically you have that sort of things in the event store as well which manages the whole message queue side of things so in our example there where we crashed the trade execution service event store knows that it didn't it was not able to successfully dispatch the credit reserved event I believe in that case so it just when it when the service connects again it knows what events need to be dispatched and the process will carry on from that point on does that make sense but the question is to answer a different slightly different question one of the challenges of in store when you have an append-only database is that if you make a mistake actually really hard to fix your problem so if you have a relational database and you do something wrong you just go to production you open up extra hole management server you log into the database you update the record that's broken you commit the transaction and you go home but in an event store you have to have created you have reporting built-in events that you can insert to compensate for the error you introduced before you don't know what errors you're going to introduce how can you create events to compensate for things you don't know you're going to do I might come back in a year time and tell you this was a terrible idea but right now and so ever it works so that is that there is a real subtle thing since Awards to manage mumble question of you I actually have two questions first how well events store scale and the second question are there situations when there is a race condition between two different events so for example like on the on the situation when you showed when you try to buy the currency and the worker was down could it be possible like at the same time the price of the currency like a relation will be change it this event will be processed while the violent will be not present yeah yeah so first question is how well does it scale it scales just fine for our system it's not super fast we have performance tested it on on the cloud deployed to fairly fairly impressive infrastructure running on the cloud it runs in a clustered environment configurations we have three nodes or all achieving quorum and has a raft Buster inside it and that was doing 40 to 50 thousand messages a second it is not a huge amount but it's still quite a few on the set on the on the second question could it get out of sync could it be a race condition that comes back to the consistency boundaries if it's really important that the trade that you do match is against the current price of the market what you have to do is bring those two things together in an aggregate and the process that we have that we were starting to build with the credit reserved amount is actually a good example of this so here we see we go out and try to reserve the amount of money on your account see if you can actually do the trade then we come back and see whether we still want to do the trade just cause you've reserved the credit doesn't necessarily mean that we still want to do the trade but that's the place where you check again a little bit of consistency to make sure it's still insane thank you hi I have one question related with this one in another with something else so are you somehow ensuring that when an aggregate it's processing a common is processing once a time in terms of achieving consistency or you get to that point when this is a problem you have to somehow check versions or something like that and a are you already in situation in which you have to manage time events in terms of time outs for example or in terms or events that you want to happen in the future because of some kind of processes do you have any kind of you actually structure for this sure so first question is do we have multiple so in our system we have multiple events multiple right instances so you can issue commanding in multiple instances if events or does give you some transactionality guarantees and it gives you them on the on the on the aggregate boundary basically so you the API lets you say when you write an event to a to an aggregate to what what have in store calls a stream you can say I expect the last version that I wrote that is on the stream to be version 5 so this should be version 6 if it's not tell me about it so it's like optimistic concurrency if someone else excuse me if someone else was gotten before and written and other methods and the streams now on version 6 when you write your event that you think should be being applied to version 5 the system will reject it and tell you that it was a problem so that gives you guarantee you've retried yeah yeah and you or you read the event you say you've what you do is you rebuild your aggregate can I still apply the command and it's still valid to go on top of this newly formed thing because when the user issued the command they were looking at version 5 of whatever it is it's a business decision now should i retry or should i go back to the user and say something's changed you need to take care of the problem for me have a look and hit submit again if it's relevant so that's that's one solution the other solution is to ignore the versioning say I don't care right over the top of it not right at the top of it upended anyway and now we've got two events that happen I give inconsistent versions of the identity is that a problem depends on your your business problem and sorry the second question was again the money oh you didn't ask the engine that's oh I don't remember it yeah well it was another time even so sorry timing is really awkward for things that happen on a large timescale like everyday what one thing we're not doing yet that we will probably do in the future there's actually inject time as an event into the system so that when you have a process manager and it says well every time a new day occurs we do these things it's not looking at the clock up there it's looking at the clock in the system and so you inject new day as an event and everything that thinks that needs to do something when there's a new day please that event goes off from processes all the things the things that are a much smaller frequency like every second for example it's just not practical for certain events into events for every second so you have to take more care about having a process manager that has state in it that is processing things in a serial fashion and that understands when it expects to have done something when it rehydrates itself after it's recovered so you can imagine that an event takes as you approach this manage your season event wait 30 seconds and if it hasn't seen something else happen raises an error so when it crashes and recovers it goes back to the beginning those through the events waits to see 30 seconds starts a new 30 seconds and sees if it hasn't seen you again come through so you have to balance the engineering to make sure that it all makes sense and it's quite complicated it to make sure these those sorts of time-based things are resilient as part of okay guys say and you cannot say afterwards thank you for the dog so give her much [Applause]

Channel: Barcelona .NET Core

Views: 25,951

Keywords: DDD, CQRS, EventStore, EventSourcing, .NET Core, C#, ASP.NET Core, Event Store

Id: t2AI9hODJ2E

Length: 80min 59sec (4859 seconds)

Published: Wed Jun 21 2017