Developing microservices with aggregates - Chris Richardson

so welcome to my talk on micro services and and domain-driven design or specifically aggregates and the goal is really to show how the concept of aggregate is super useful when you're building business logic or business applications using the micro service architecture and it turns out that the applicability of domain driven design to micro services is much much broader than what I'm talking about there's other concepts that are sort of out of scope for this presentation like bounded context and various sort of strategic design concepts from DDD but in this talk I'm just focusing on on on the on this concept of an aggregate so before I get into that a little bit about me so if you don't know me I'm Chris Richardson I actually live in Oakland California got my start and programming back in the mid 80s building Lisp systems so compilers everything from run times compilers garbage collectors all the way up the stack to IDE s eventually I ended up programming in Java then you know ten years ago sort of a lifetime ago my book pojos in action came out and that was all about spring and hibernate which ten years ago were these technologies that were absolutely revolutionising Enterprise Java development and it's super cool that ten years later you know spring is still innovating and still extremely relevant so I'm super excited about that and then back in 2006 to 2007 I started playing around with what was then a very obscure service known as Amazon ec2 I think this was before even the term cloud computing had had been developed and an evangelist from Amazon came and spoke at my local jug and we thought they were going to talk about api's for buying books because that's what Amazon was known for ten years ago and instead it was api's for provision servers and the concept that you could just do that blew my mind so much so that I created an open-source project which evolved into a startup called Cloud Foundry the name of which is somewhat famous today there are different technology so my Cloud Foundry the classic version which was a pass for deploying java applications on AWS was acquired by spring sauce which was then acquired by vmware and ended up being at vmware and then pivotal for some number of years and ended up leaving almost so I suppose three years ago now so since then I've been focusing on this area of micro services so I do consulting and training with Mike refer about micro services and I've also a founder of a start-up and we're building a platform to simplify the development of business applications that use micro services and in fact some of the concepts I'm going to cover today is supported by our product if you want to know more about micro services in general then go to this link learn micro services dot IO and there you'll find links to blog posts articles presentations videos example code that I've been also training that I've created over the years so so the one go-to place so that that's sort of me so here's the agenda so first off I want to talk sort of provide the motivation for the rest of the talk by describing the kinds of problems that you will encounter when you're implementing business logic using the domain model pattern and what in a micro service architecture and then after that I'm going to talk all about aggregates and how they they solve those problems so let's get going so first a little bit about micro services which you know just everybody is talking about them today so I'm just going to touch on them really really briefly and just to say that first and foremost the goal of the micro service architecture is to tackle the complexity of large app patience and the way of course that it does that is by using the ancient technique of modularization so in other words it takes what would otherwise be a large monolithic application and breaks it up into a set of smaller applications which go by the name of services and it really is just a form of modularization each more each micro service corresponds to a business capability which is something a business does in order to create value so if you apply that sort of thinking to an online store you would end up with an architecture that looked like this right you have various services corresponding to different business capabilities like the catalog service that's maintaining product catalog information the review service that's keeping track of reviews the order service that's responsible for processing orders and so on so we're just breaking the application up into these miniature into these mini applications each service has its own database which is absolutely necessary in order to ensure loose coupling and but in at the same time that's a double-edged sword because that's going to lead that actually is sort of the ultimate the root cause of many of the problems that that have motivated this this presentation so keep that in mind each service has its own database and then of course sitting in front of the services you're going to have an epi gateway that's acting as a facade providing a single entry point into the system of micro services and possibly doing providing a dedicated API to each particular kind of client and then you've got various clients that are consuming those micro services could be a web application could be a mobile application and each of these services you can scale independently but that's sort of our out of the scope of this discussion and so services enforce modularity because they have a very well defined boundary so a service is a process that ultimately and everything that is inside a process is private to that process unless it has been explicitly exposed through an API whether that could be rest or messaging so it's just a very very strong form of modularity something that's been really difficult to achieve inside programming languages right like Java packages are just not good for for building truly modular systems OSGi attempts to do that but I guess we all know how well that succeeded right except you in general with amongst application developers so that's a microservice architecture so it enables you to develop faster and build more modular applications that are higher quality and faster and so on but there are there are a whole set of challenges with the Micra service architecture and in this talk I'm just going to focus on some very on a couple of very narrow specific issues that if that sort of impact how you implement your business logic so the first problem you're going to encounter is that your typical domain model right because everyone knows that if you're building complex business logic you should apply the domain model pattern and build objects with rich structure and rich behavior the problem you run into is a typical domain model is really just a tangled web of classes inside a monolithic application there's actually problems with that in itself but there's some real problems with that in a micro service architecture because ultimately you want to be able to put different classes in different micro services right but objects point to one another so orders would have a reference to its customer order line items have a reference to the corresponding product so when you want to put the order object inside the order service it's going to have the order object itself is going to have this pointer to a customer what does that mean when it's going across the service boundary right that just does not compute likewise with the prints from order line item to product so that's one problem a typical domain model cannot easily be partitioned in across multiple services the second problem you're going to run into is that a typical Java application is just going to rely on asset transactions to enforce invariants in enforce business rules so in this particular domain model customers have a credit limit so when you place a new water you have to verify that the that the new order won't exceed the credit limit in a monolithic application that's that's trivial you begin a transaction you find the existing orders you find the credit limit for that customer you make sure that you haven't exceeded the credit limit and you might have bought at this point and then you insert an order and then you commit the transaction and assuming that you're using the right isolation level the serializable property of acid transactions will enforce that business rule even if there are multiple simultaneous transactions trying to create waters for the same customer those the the sort of the transaction manager will in will in essence serialize those transactions so they appear to happen one after the other so that that's that's a really nice simple programming model right you know we're just used to it and there's a lot of power but the problem you have with that approach is that first of all it violates encapsulation remember each service has its own database so the order table belongs to the order service the customer table belongs to the customer service so you can't access both of them within the same transaction you're violating the encapsulation of each service so that's one problem the other problem you have is that this transaction would would actually span multiple services and that of course would require some form of distributed transaction mechanism or two-piece you know or a two-phase commit mechanism and it turns out that in modern applications to PC is not a viable option on the one hand it does give you some kind of consistency guarantees though you actually have to read the fine print to see what kind of isolation level it gives you but because it comes with a whole lot of complexity and issues it's generally best avoided in modern applications it's not supported by a by many modern technologies like no sequel databases or or modern message brokers so even if you wanted to use it you couldn't and then also there's the cap theorem right which basically says you have consistency availability and partitioning and you have to pick two of those and today it's preferable to pick availability over consistency so there's sort of a theoretical sort of issue with with using two-phase commit in modern applications so we sort of have a problem this just sort of just this transaction model just doesn't fit the kinds of technologies and the kinds of systems that we're building today so that's a problem and then in particular it doesn't fit the capabilities of most no sequel databases which have which tend to have a very very limited sort of more transaction model basically you can update one record whether that's a document or a row or a key value pad in a typical no sequel database you don't have full assets transaction sort of semantics there so there's sort of a problem taking the traditional way with which we build applications or with which we write business logic to be precise and translating that to the modern world of micro services and sequel databases so yeah so that that's the problem so what's the solution so it turns out that many of the solutions to the problems that we have when we build micro-service based applications can be found in domain driven design which is kind of funny because this book was written like 13 years ago so it's sort of this ancient text as far as software development goes which is very philosophically motivated but it turns out that it actually has the solutions to building very very modern market which is quite quite ironic in a way so let's dig into that a little more so I'm probably many of us have read you know started reading some of the book right and it talks about the building blocks of domain models right there's the concept of an entity that's an object that has a persistent ID there's the idea of a value object that just sort of has value but no identity there's the concept of services that contain business logic that don't actually belong anywhere or don't belong in entities or value objects and then of course there's the concept of repositories that really are just abstractions over collections of entities that are living in a database and we've all pretty much internalized those right you know that that perhaps maybe not value objects as much even though they're incredibly valuable but we're all used to using terms like entity service and repository you know even frameworks today support that right so that the least the first four evented sort of modern kind of our dictionary but then there's the last one which is aggregate which when I read the book you know back in 2004 2005 it was like yeah yeah yeah whatever and I kind of ignored it and just focused on entities and value objects services and repositories but it turns out that aggregates are absolutely essential concept to apply when you're building business logic in a micro service application so what is an aggregate so an aggregate is a cluster of objects that can be treated as a unit so it's sort of a graph that has a root entity and possibly one or more other value objects or entities that are just sort of referenced by the root either directly or indirectly so it's a little cluster of objects that just reference one another and it turns out that most sort of business entities like order customer account product etc really actually are aggregates we sort of think of them as entities but really you know like in the case of an order yeah there's an order object but that's not really the whole order right that an order also has line items and it also might have a payment address and a delivery address and so on possibly payment info as well so you know so we have an order aggregate and and it is this cluster of objects so it's actually this way of taking a domain model and breaking it up into chunks that can be treated as a unit you know among other things it actually says it actually defines what it means to delete an order right so when you delete an order you have to delete the line items and the address and so on so that's just number first benefit it actually breaks a domain model up into chunks and anytime you can modularize your domain model that's quite good but there's actually some deeper meaning to it than that so there's a set of rules that aggregates have to obey and the first rule concerns how one aggregate can reference another so you know an order for instance can have a wreck would have wanna reference a customer right which is that which is itself another aggregate but what's really interesting here is that rather than having an object reference to the customer it actually just has the primary key of the customer so an order would would not have a customer field but it would have a customer ID field likewise an order line item would have a product ID field and not just a product field that would be a pointer to the art to the to the product object and it's sort of like wait what what you know you can't have foreign keys and object models right you know if you were to go back and look at a text on object oriented design foreign keys in your domain model would be considered a coat smell and should be removed instantly so we're actually sort of violating some of the traditional rule in very traditional object-oriented design but the benefit of this is that not only if we modularize the domain model which is good the actual chunks the actual aggregates are very loosely connected now right because it's just in terms of primary key and not a proper object reference so that makes it really easy in many ways to actually put the order in the order service the customer in the customer service and the product in the product service because there are no object references that are that are attempting to span process boundaries so in other words aggregates let you partition your object model across micro-services so that's a good thing so that that's one immediate benefit there's also another rule which when I sort of read it for the very first time like made no sense whatsoever and the idea and the rule is is that a transaction should only create or update one aggregate so in other words sort of an aggregate is kind of a unit of consistency and that that sort of shocked me as well because you know back then I was using my relational database inside my monolithic application I could begin a transaction update as much data as I want and then commit the transaction a night I sort of have wonderful guarantees and now there's someone who's saying right Eric Evans is saying no no it transaction has to be limited in scope and it was sort of like but that's silly it's not properly leveraging the technology that we have available but you know you could think back to one of the problems with with micro-services and databases right a transaction has to fit within a within a service we can't use distributor transactions in order to span service boundaries and so this constraint on what our transactions can do inside our business logic exactly fits the technical constraints that we have inside a micro service architecture so now because an aggregate is contained within a service and a transaction can only update or create a single aggregate a transaction is guaranteed to fit within a service boundary so we've sort of addressed the transaction problem that we sort of have to deal with in a micro service architecture so that that's you know huge benefit not only that the this sort of transaction scope actually fits the capabilities of a typical no sequel database right so in a no sequel database you can only update say a single document or a single row or a single key value pair so if for instance we persist are an aggregate as a document then we're writing transactions that are only creating or updating a single document at a time and so you know it's sort of a perfect fit at that level set of Burnt you have a burning question that's coming up later yeah so I I have I have intentionally narrowed the scope of what a transaction means which I think as you're feeling this sort of discomfort doesn't necessarily match the actual business requirements that you might have but I'm doing that to create suspense right so interestingly that there is this issue around like how big are your aggregates because you know in order for something to be gent to actually be atomic right like the unit of atomicity in this programming model is now the aggregate so if there is something that so happens to need to be added needs to be truly atomic and it turns out that many many things don't need to be atomic but if they do then what you need to update within that transaction has to fit within a single aggregate and it turns out that you often get to choose your aggregate boundaries so I for instance had been saying that orders customers and products were their own separate aggregates but that's a design decision that I consciously made you could for example put customers products and orders all within the same aggregate so your system just consists of a single aggregate and I actually talked with someone who tried to do this for some startup in the Bay Area because they wanted consistency right turns out that it was terrible in it and it did not work but yeah but something that could work would be to have a customer and their orders be an aggregate right so you could imagine you know a large MongoDB MongoDB document containing a customer and all of their orders and that would be a single aggregate and product would be another aggregate and that could possibly be a legitimate way of implementing your application if there was something that you needed to implement a transaction that needed to be atomic and in you know that involve customers and all of their orders but of course there's a there's a drawback is that that you can only update a single aggregate at a time right updates are serialized when it were update service of the same aggregate as serialized and that can result in bad usability in a bad user experience like when two orders two users try and edit the same order for a different customer the second one might actually not be able to see their changes because of like an optimistic locking exception so there's sort of problems that so in general it's better to have as fine-grained aggregates as possible but but you do have a choice so that so that so that's one that's a partial answer to how do you make things atomic but if you want to sort of solve this problem more generally you actually have to use an alternative approach which I'm going to talk about not right talked about next so yeah so let's imagine that you know orders and costs you had you know orders right you've got needs along to a customer customers have a credit limit and we need to always ensure that this credit limit is never exceeded so if as I showed if they're in the same database same monolithic application with the monolithic database trivial right but if they're separate aggregate belonging in separate services and you can't use two-phase commit it's like how the heck do you maintain consistency or in this case how do we reliably enforce this this business rule or invariant and it turns out the solution is to use events so we're actually going to abandon this sort of acid model and go to an eventually consistent event-driven model and the idea which is really quite simple is that a service publishes an event whenever it updates one of its aggregates I actually strictly speaking you could imagine the an aggregate when its state changes actually publishes the event but you can think of this happening at a service level so whenever something changes or something of note occurs you publish an event another service subscribes to those events and then reacts accordingly can update its own state so in order and in order to sort of maintain some consistency so the way this would work with order processing is as follows so request comes in to create an order the order service would create an order but sort of in this pending state like where hasn't been verified yet it would publish an event to say that the order has been created that would get consumed by the customer service which would perform the credit check you would actually in this case in the example code it actually updates the customer or attempts to update the customer to reserve the credit for that water and then it will publish an event which indicates the outcome of that credit check either a customer or either a credit reserved event would be published if it was if the credit was able to be reserved or if the credit limit was exceeded it would publish a credit check failed event the order so the order service would process one or other of those events and then update the order in some way either approve the order or reject it because there was insufficient credit so in this so that so that's how you'd implement something like credit check without using acid transactions so of course in this case you instead instead of one big transaction there's actually a series of there's three transactions one in the order service one in the customer service and then one back in the customer service and there they're driven by event at least the all but the first one is driven by event so here's a slightly different view of this right so create order gets invoked the order gets created an event gets published that get the event gets consumed by the customer service which updates the order so it updates the customer with the credit reservation it then publishes an event to say that the credit has been reserved that event gets consumed which then changes the state of the order which then in turn could publish another event which could trigger fulfillment and so on so that that's how you implement sort of maintain data consistency in a micro service architecture using events so we've sort of abandoned the acid model and we're now using this eventually consistent model which some which some people call base right basically available soft state eventually consistent and one thing you might note is gosh this is now a bit complicated right like you know in a more general case you you know like in a sort of regular acid transaction when a business rule gets violated you can just simply rollback the transaction and and all of the changes get undone right but in the in this model you could have a long sequence of steps and it was only when you got to the fifth step do you realize that some business rule would be violated and you actually end up having the publishing event which would then next trigger the execution of these so-called compensating transactions which would explicitly undo the changes that they had to make to in the first four steps of this process so that that's a bit complicated and you have to carefully think about the design in order for this to work you know like if you're transferring money between two bank accounts right you debit the from account and then when you try and credit the two account you find that it has close so you then have to credit the the from account with the money that you just took out of it but then you could have a problem because someone could have closed the from account and now you've got this money in limbo and sort of the bank wins I guess so so there's some you know so it but in reality the real world works this way I think if you work for a bank then you probably know that you transact money doesn't tend to get transferred using acid transactions at least the last time I talked to someone who worked for a bank that that's what they said so that that's the approach but in order for this to work there's an interesting little problem you have to be in order for it to be reliable you have to atomically and it's sort of transactional sense update the database so for instance insert an order and publish an event right you've got to do two things involving two different pieces of infrastructure the database in the message broker and they have to be a done atomically because for instance if you update the database and then you fail to publish an event because the message broker was down or something like that or you crash before you have a chance to publish the message which is more likely then the system is now in an inconsistent state what's really ironic is the standard way of doing this would be to use a distributed transaction where you know you update the database and then you publish a message to a JMS message broker it which participates in the same transaction but for the reasons I mentioned earlier two-phase commit is not an option so we need a different way of ensuring that we can reliably publish event so it's kind of funny right like we decide I'll monoliths too hard so let's use micro services and it turns out that once you go down the micro services route there's a whole bunch of complexity with maintaining data consistency sometimes when I've given talks people you long for their monolithic applications but yeah but then monoliths have big problems so it turns out that there's several different strategies that you can use and in this talk I'm going to focus on one of them called event sourcing but there are others so for instance eBay would use this to maintain data consistencies across different systems so it would the application event pattern an application inserts an event into an event table so in the database you're actually basically treating a table using a table as a message queue so you update say in this case the order and you insert an event into the order into the event table and that's just one asset transaction and then outside of that there's another process that's polling the event table pulling events out and publishing doing doing something else right in this case publishing them so that's an approach that eBay used and then there's others so like LinkedIn they actually tail the transaction log the database commit log or transaction log and publish events based on that and that is a really really interesting technology but the one I want to talk about is event sourcing and if by the way if you go to learn microservices i/o there's you'll find a link to an article that talks about these different approaches also on micro services dot IO which is where these patterns live your you'll find a write-up for most of this as well but what I want to talk about is event sourcing which is a particular way of writing your business logic that really is all about event in fact event sourcing is an event centric way of persisting your data and so in a way the application isn't updating the database and sort of publishing an event it actually then that basically the there is this concept of an event store that is kind of like a database and a message broker all rolled into one and so whenever something changes that state change and that all you're doing is publishing an event into the event store so it's a very event centric way of storing your data and now the details of which will become obvious in a minute and the way that works is as follows so for each of your aggregates you identify the state changing event there also with all the domain events to be more precise which might be state changing but they could be other things of significance like the credit limit was exceeded event which wasn't a state change to the customer but it just represents the failed attempt to violate a business rule so you identify that the any events of relevant and what's interesting is there's also there's actually a workshop format where you gather school events storming where you gather people in a room and you brainstorm what the events are within a particular domain and you and you use that to actually design your business logic but you figure out what your events are and then you define event classes so events become first-class citizens in your domain model so in the order system you would actually have classes like order created or to cancel or the ship to order approved and so on so that's the first thing you use you sort of treat events as first-class citizens and model them explicitly and then the second thing you do which is really different is that you change how you persist your domain objects so the standard way of course is to take an order entity and you'd map it using JPA hibernate or just straight sequel to a row in an order table right that's sort of the standard way that we've been persisting our domain objects you know from the beginning of time or at least ever since relational databases were in vented right but with this model with event sourcing you don't do that you don't do this straightforward mapping and instead what you do is persist the events themselves so conceptually impossibly in practice if you were using a relational database you would have an event table that would be the entirety almost of your you of your database schema because every state change ends up causing an event in this to be inserted into this table so when an order is created all you do is save an order created event in this table likewise when it's approved you would insert an event to indicate that fact and likewise when it shipped you'd insert another event and this is all you store for a given order just the sequence of event so there are no other tables the order table goes away order Lyon item goes away and you could imagine that the event data column is a blob of JSON just for the sake of argument because that's sort of just kind of a serialization of some bit of data and then when you want the current state of an order right like you think about I want to load the order entity so JP a slash hibernate it's going to wish you a set of queries and and reconstitute an order object in memory in this system what happens it you actually query the event table load the events for a given order or whatever entity it is you're loading and then you replay those events to reinitialize what a blank order object and so you're basically taking the stream of events collapsing them together in order to reconstitute this order entity and yeah if you have lots of events there's a technique known as snapshotting that sort of means that you don't have to go back to the beginning of time and load the order load the orders if you're functionally inclined you can think what you're doing in order to reconstruct the current state is a functional fold or or a reduce over the stream of event so that's the essence of events or sinks you're storing the event and nothing else at the event are the system of record in your architecture so your application now looks like this where you've got this event store you know each egg regard is now represented in the event store as a sequence of events and so the order aggregates are owned and entirely private to the order service the customer aggregate is owned and entirely private to the customer service but they can actually subscribe to events that one another publish so so that that's how you get the event-driven architecture here and then the way requests get handled are as follows so imagine a request comes in to the order service first thing you have to do is find the events for that order right so it's a select against the event table if that's how you're storing the event you instantiate a new order using the default constructor you apply the events that you loaded from the database to reinitialize the order later on you'll see that that an aggregate in this model has an apply method that takes an event and updates its current state you then process the request which gets represented by a command objects of aggregates have a process method that takes a command to say do this like reserve credit doesn't actually change the state of the domain object but what it does it returns a sequence of event that represents the state change those events are then applied to actually perform the state change and then the events the new events are saved in the event store so appended on to the events table that that's storing the event and this is done with optimistic locking to handle the scenario where two requests to simultaneously trying to update the same order so that's that's one part of request processing so that sort of in it you know how an order would be canceled or or so on or marked as shipped or or whatever the other part of it is that services like the customer service can subscribe to the events so that so the event store has a subscription API and then when those events are saved in the event store interested subscribers are notified so they would get the event so it's sort of kind of like a message broker in that regard so the customer service would get the event the order event whatever that would be and that would then cause the order service to then go and update the the customer for that order if it made made sense for that particular event which would then of course result in some customer event being saved in the event store which would then trigger some other event handler in some other service so that's sort of how this event how event sourcing fits in with the event-driven architecture you can have other subscribers that perform arbitrary actions right so when events when they get notified of an event an event subscriber could update a view there's there's a related pattern called command query responsibility segregation where in order for to facilitate easy querying or to support queries you maintain a denormalized view of the data so that that view would be kept up-to-date by subscribing to event so a great example of that would be using elastic search for instance right you want to add text searching on to your system where you simply subscribe to the stream of events that are being published by your aggregate and we index the documents inside elasticsearch other subscribers could send out notifications like emails or text messages or mobile push etc so sort of you know that you can do fairly arbitrary things so the events store itself really is like part database because it has an API for inserting events for a given entity and also retrieving events for given entity by primary keys so that's very database like and the database portion could be you know a sequel database could be a no sequel database but then there's also a message broker type API as well where you can subscribe to events that have been saved in the database so some people it's kind of a hybrid database message broker some people have implemented it themselves on top of like my sequel or Postgres orphan net land there's a dedicated event store called event store or get event store and then my startups work part of what we're building is an event store as well so there's various different ways of sort of building this built building and events store so event sourcing you know has various benefits right like it solves the date you know it lets you build an event-driven architecture that solves these data that supports this notion of aggregate based development so you can easily maintain consistency between microservices or more specifically between aggregates you know every state change results in an event those events could be consumed by machine learning algorithms they could be turned into notifications that get sent out to users this it also eliminates öor mapping issues as well because we were no longer saving domain objects in the database with saving events which generally are easy easier to sterilize also because we're verifying at state changes in other words every state change is represented by an event object it means that we have this built in hundred percent reliable audit logging mechanism right like you know systems I worked in on the past where was layer we got to implement audit logging and we just kind of sprinkled cools to some audit logging service throughout our code and sometimes it was reliable sometimes not right whereas this it's guaranteed to be reliable because you a state change by definition is an event in the event store which is quite good and then another benefit is well because you're saving the history of every domain object you can actually go back in time and execute temporal queries and find what's the state of this domain object two weeks ago which can be useful if you need to sort of do diagnosis or the regulators want to know how you've executed trades for example right you've got history that's accurate built in with this so that's pretty powerful and then also because you've preserved the history of everything that has happened in the entire system in theory all the way back to day one you could actually implement a feature today that sort of say this feature this module is analyzed sort of analyzes the stream of events and could for instance build up a view but it could sort of do interesting analytics you could actually have it process all of the events that have ever been published in your system since day one and the effect of that would be as if you had implemented that feature on day one right whereas normally it's like if you implement it today and deploy it today it can only sort of do things going forward so now you've got you've sort of got a time machine that enables you to go back and implement features you know as if they were done on day one but having said that you know there are a bunch of drawbacks right like sir requires an application rewrite you know the way you write your domain logic you've the way you persist your data is very different though it does tie in nicely with the migration to micro-services right so you migrate your code from a monolith using traditional JPA to a micro service architecture that's based on event sourcing right it sort of could fit in easily with that that you know there's under undoubtedly a learning curve right you're programming in a different style so there's you know that takes a little bit of getting used to another really interesting thing is that events never go away right so I mean kind of a joke saying it's a historical record right which hopefully will not hopefully you won't make bad design decisions with your events but there are some serious issues around you know evolving the schemer of your events like adding new attributes over time that you need to bikini to consider because once you have saved an event in your event store consumers might you know might have to proce that might always have to be able to deal with events with that particular structure for forever so you need some discipline also you just you but in general you can also plug in some like die effectively kind of take a version one event and upgrade it to a version two event when you load it in so so your domain objects can always be written in terms of the latest event so there are there are things that you can do there but there's there's some interesting challenges there's also an issue that this is a message based system and so you know consumers might see the same event multiple times and so in some situations you're going to have to do some careful coding in order to detect and ignore duplicate events so that that can be a challenge usually the solutions are quite simple though and then there's another problem querying the events or is not exactly straightforward so you know imagine that you want to find you want to find all orders that are say in a particular state right not in sort of a traditional schema you just go select from orders where state equals X right and you will find the or the rows that match but now I mean if this is a really simple query but in actually in order to figure out what the current state of an order is you actually have to go find the events for that order and actually sort of discard all but the last event that correspond to a state change for that order right so in this case yeah you'd have like where and then some nested select equals the desired state right but you could easily imagine that a more complex query against the event store would just become incredibly complex and potentially inefficient so there's some sort of real problems there and also some event stores only specifically ones that are based up with on top of no sequel databases only support lookup of events by primary key so you have no ability to actually execute queries anyway and so that's why as I mentioned earlier you need to use CQRS command query responsibility segregation and maintain separate views in order to support your queries but that's sort of a whole other topic and but it turns out that in general CQRS has value anyway right like it's just a generalization of having using like elasticsearch in order to support text queries it's just you end up having to use these secondary stores and in order to support any kind of query in your application except for the load this entity by primary key kind of queries that's a whole other topic okay so I just want to finish up by quickly talking about what's showing some example code for this sort of order and customers use case so there's in the actual application which you can find in github there's actually three services there's the customer service that has the customer aggregate there's the order service that has the order aggregate and then there's actually an order history service that's implementing one of these denormalized views so it's a CQRS view and it's actually using MongoDB and internally for each customer there is a MongoDB document which has an app an orders attribute that is the collection of the orders for that particular customer and that when MongoDB is kept up to date by subscribing to events so that's sort of the big picture view of the application if you dig down into the customer service right like yeah there's a customer controller the the domain logic consists of a customer service the customer okay yeah the customer aggregate and then there's some event handlers that are subscribing to the order event that are being published through via another service and that system actually is using Kafka as well because there's some other communication between the order the order service is actually publishing events or so the customer service is publishing event some other event or some other messages they should say that it get consumed by the order service via Kafka as well as the event store but that's sort of our you can look at the code later so here's the customer aggregate so it's got some state it's got a crit field that's a credit limit it also has credit reservations that's a field that's a map from order ID to money so it's keeping track of the fact that order one two three four has reserved $1,200 of the available credit there were also a set of process methods so those are the ones that take a command which represents a request to do something to a customer like create the customer or reserve credit and they return a list of events so they don't do the commands don't perform a state change they return event that represents that the state change there are also apply methods that take an event and perform the state change so there's a very sort of structured way of writing the business logic and in essence if you think about it so in the old way of writing code you'd have a reserve credit method that took some parameters sort of use some implemented some like business rules and then did a state update in this model that method is split into two it's split into a process method that's doing some sort of busy implementing the business rules and then returning an event representing the state change and then the state change is handled by an apply method that takes the event and just update State so it's sort of the same business logic we've just split it into two so here's the aggregate itself so it implements reflect so yeah implements a base class called reflective mutable command processing aggregate so it's sort of like following the spring naming convention but it's it makes sense because it's an aggregate that knows how to process commands and it's mutable because you can if you are was programming in scholar I'd have immutable aggregates but this is Java so I'm mutating them so they're mutable and it's actually using reflection to dispatch on the event type and the command type to the appropriate apply method or process method so that that's if you that that's the explanation I have unpacked the name for you so the pro process methods you know there's a couple of those the second one the process method that takes a reserved credit command is the interesting one because that has one line of business logic in it right so it's saying if the available credit is greater than or equal to the order total then return the credit reserved credit customer credit reserved event so we've reserved we there is credit for us to reserve but then if there isn't sufficient credit it returns a customer credit limit exceeded event which is a typo in that so that that's a you know that's a great example of some of a method that implements a business rule and then if we look at the apply method those are really simple right so the corresponding customer credit reserved event apply method just puts an entry into the map of credit reservations so that that's that one there so there's no business logic in the apply methods this that you know what to do has already been decided by the process methods the apply method simply apply the change and then and I should have mentioned those are used when reconstructing the current state of an of an aggregate when loading it from the events store so on the other one that's interesting is when the customer credit limit is exceeded that's not indicative of state change it's indicative of a business rule violation so the apply method doesn't actually do anything you know here's the controller that handles the post request to create a new customer so it's just calling the create customer method on the customer service and here's the customer service so the create method is just one liner and that's because it's written using this helper class known is called aggregate repository that has a save method which takes a command so in this case it's a create customer command and what save does is create a you know a a customer aggregate using its default constructor processes the command by calling the process method applies the event by calling apply and then it saves the events in the event store so that whenever you need to you know create a new aggregate you would repeat those four steps over and over again so the save method just kind of like encapsulate that boilerplate for you so it things end up being you know what you write ends up being really simple so that that's the service here's an event handler so it's annotated with at event subscriber and that's actually ends up being picked up by a spring bean post processor which then ends up subscribing to the event store using the subscription name that that's passed in and then you can see there's one event handler method that handles the order created event and remember the so when an order creative event is received what it has to do is tell the customer to reserve the credit so that's that's that's what the event handler does and it's calling the update method on the context that gets passed in and this is like the save method it's a helper method that simplifies how you interact with the event store so what it's doing it so it's saying you know you you tell update go find the customer find a customer with the specified customer ID and process the reserve credit command and so under the covers is actually loading the customer processing the command applying the events and saving those new events in the event store so it's just sort of encapsulating that piece of boilerplate code and then there's some there's some ugly code here which it's like so this codes written in a reactive style and using Java eight completable futures which have a hideous API if you ask why's that well at least they're useful but the names of the methods are all just wrong like handle async should really be called flatmap thought if you are or actually even better they need to recover with method if you're familiar with Scala but basically what this is doing is handling the scenario where an order was created with an invalid customer ID so there's no actual aggregate to go in the database so you get an entity not found exception and this piece of code here is actually had recovering from that by using spring cloud stream to send a message back to the order service saying you tell told me to reserve credit for this customer but this customer doesn't exist which would then cause the order order to be canceled and that so spring cloud streams is actually publishing a message via Kafka so there's sort of this secondary event based sort of back-channel that that's being used here and that's all abstracted away and this go you know you go online and you can see the the code for us in all its ugly details so so that that's actually pretty much my talk so in summary right so aggregates really are the building blocks of your business logic in your Mike for service based applications and in general it's like go read the domain driven design book because there's so many concepts in there aggregates and also the strategic design concepts like bounded contexts that are incredibly applicable to micro services architecture you want to use events to maintain consistency between your aggregates or between your services and events sourcing is a really good way to imply an event-driven architecture so that yeah so that's my talk thank you for listening and I hope that you found it useful and here's all my contact information so thank you and I guess there's time for questions if you cling if you don't want to go to the bar right away um yeah yeah oh that's a really good question yes so what do I use for the event store so that I actually so I actually have two answers for that so so if you go to eventuate dot IO so one there's a there's a hosted version so that's actually leveraging features of AWS to provide the event store but within the next week or so I'm actually releasing an open-source version that you can run on premise that's just going to leverage a sequel database and Kafka yeah yeah yeah um yeah yeah so so so DF so for the recording yeah the question is well if you're sort of refactoring an existing monolith what which what approach could you use right for in order to publish event and so yeah you had you have a choice so one is it works like number one you almost always need to do something especially because integrating a monolith with a microservice you almost need events going back and forth in order to keep data synchronized so that that's part so you like you need to do something and then yes there are different patterns that you can apply and I thinks a transaction log tailing is too problematic because you're seeing low-level changes to the database schema and you have to reverse engineer what the business events are so lets you know rule that one out so then the next thing is well should I use event sourcing or should I use application events and you know it's part of it is sort of what what you're it's like either one will work right part of it is sort of what your preference is and then also by the time if you're going to use events anyway you on top of a relational database you use might not you might end up discovering that using event sourcing is pretty straightforward actually oh hang on this is a spring conference I'm sorry yeah so so the question is yeah if you are doing a greenfield project well would I use akka and once again that's an interesting question on the one hand say that the you know light band formerly known as typesafe feet folks have a quite interesting stack what you know what what I would work what I wouldn't I like actors and I like using actors within services what I'm less comfortable doing is using actors ads sort of the organizing paradigm for an entire system right I'd feel much better using sort of more open and and and somewhat less proprietary technologies then say like actor remoting for example you know you know what I mean plus this is a spring conference and we love spring here don't we actually I'm told that I'm out of time but I'll take your question offline so once again thank you for listening and I hope you found this useful

Channel: SpringDeveloper

Views: 213,598

Rating: 4.9335084 out of 5

Keywords: Web Application (Industry) Web Application Framework (Software Genre), Java (Programming Language), Spring Framework, Software Developer (Project Role), Java (Software), microservices, spring cloud, spring boot

Id: 7kX3fs0pWwc

Length: 69min 50sec (4190 seconds)

Published: Tue Jan 24 2017