Keynote - Udi Dahan - DDD Europe 2020

[Music] good morning welcome hello it's great to be back at ddd europe specifically for this event sourcing talk just a quick question to sort of get a sense of the audience who's been to ddd europe before okay a lot of you who's this the first time for you i'm guessing that a lot of you raising your hands twice either that i just don't know what happened here because it looked like eighty percent raised their hands for both questions this usually doesn't happen when i'm in europe getting that many people that are raising their hands what about event sourcing itself i mean who's been doing event sourcing on their projects already okay and whose is the first time that you're hearing about event sourcing okay some hands coming up alright so for those of you that this is new i'm going to give you some of the background of what this event sourcing thing even is for those you that have been doing it my guess is that you've heard varying opinions on what it is how to do it what's included what's not included how that relates to other patterns this is sort of the natural order of things okay so for those who just sort of coming in and feeling a little bit confused about what's been going on with event sourcing the fact that this is our essentially first event sourcing conference is sort of a product of that as an industry we first start doing things in new ways then we come up with names for it then we start talking to each other about those names and then realizing that everybody is using the same names to mean different things then we have conferences then we get together then we sort it out and then roughly ten years down the road we kind of get to the point we're like okay now we finally know what we're talking about so for those who don't know me i'm moody dahun i speak at various events like this i spent a number of years consulting working with various companies I started a company called particular software over there we build service bus technology that enables building event-driven systems and I'm at Moody to hunt on Twitter if you want to talk to me over there so I know I'm not really old enough to pull off the whole Clint Eastwood grizzled get off my lawn type of thing but there will be a little bit of that in this talk a lot of the things that sort of get bundled underneath event sourcing they're not really that new and that's where it's going to help as we sort of go through this to put some order in these types of things my concern and it's not just about event sourcing but pretty much every new thing that comes along there's this sort of early phase where everybody's like oh my god it's the greatest thing since sliced bread I'm using event sourcing for everything and that's usually where a lot of the damage gets done I'm afraid so if you or your company is in this phase of event sourcing all of the things where everybody's all hyper excited about it again you know this is not the first time that we as an industry got excited about some new thing that's going to make everything magically better before that we had micro services which still is kind of a big thing that everybody's kind of going on and on about domain driven design for a while was that brand new thing service or in it architecture before that and it goes on and on so really you know every other technology out there has gone through the same type of thing and over time yeah they all kind of start to look the same and as we go through a bunch of these patterns you'll see the similarities and I'll try to articulate the differences between them so this idea of things getting sort of blown out of proportion it was documented a number of years ago from the fellows at Gartner known as the Gartner hype cycle so the very first conference on event sourcing means you are here the peak of inflated expectations this is where we are event sourcing all of the things some of you have already been through that and realized this is a lot harder than it looks and there are a lot of challenges along the way you might be down there in the trough of disillusionment saying you don't want let's actually rip all that out and just you know go all the way back to basics our objective with this event sourcing conference is to hopefully bring some of you up towards that slope of enlightenment of understanding what it is what it's good for what are the other bits that make it up and how to use them so that's kind of our story for the day obviously in this talk I'm not going to be able to give you all of the detailed information that helped bring you up over there but we got lots of great speakers and we won't always agree with each other all right so for those who that this is the first time that you're at a DDD conference or any type of industry conferences you'll be hearing from lots of speakers and not everybody will agree with everybody else and then that might leave you confused you'd much rather have something to say this is what it is this is how you do it here's a cookie cutter approach ABCD and then you just follow that and you're good we're not going to get that alright we're not mature enough yet with regards to event sourcing and Industry to get to that point but we will hopefully take all of this stuff that sort of falls in and around event sourcing where we've got SOA and domain driven design and bounded context and adventure of and architecture and micro services and pub/sub and all of this mess and hopefully get you to a point where at least you understand what those things are what they're good for and how to make use of them on various projects so let's start and bring some order to that chaos that we just talked about what is a vent sourcing anyway so let's start with this who thinks they know what event sourcing is raise your hands that's a loaded question isn't it huh I see the heads are kinda like mm-hmm let me think about that actually you wouldn't have given this big prologue if it was that clear-cut now I remember when I asked who's doing event sourcing at the beginning lots of hands were like yep I am and then indeed who thinks they know what event sourcing is it's uh mmm I think so alright so generally speaking what there seems to be broad agreement of is event sourcing refers to a collection of patterns it's not just one thing all right a collection of patterns based on the idea that you persist the full history of a domain as a sequence of events rather than just persisting the current state alright and that's sort of the big distinction from the way the traditional domain models were built when using object relational mapping technology of sorts where historically what we've done is we've taken a domain model in its current state and persisted that current state and then made changes to that current state and persisted that all right so that's the main difference that event sourcing is bringing in on top of the core domain model pattern let's talk about domain model patterns for a second so who's used the domain model pattern before didn't see some hands all right who has not who has not use domain model pattern okay less hands so just a refresher even for those who that have what the domain model pattern is and again it's one of those things that you think you understand it until you kind of go into the details so domain model is an object model of the dome that incorporates both behavior and data fairly generic so my guess is that a lot of you have a domain model even those of you that might not have heard of the domain model pattern you probably have some kind of object model of your domain now seeing as we're part of the DDD domain driven design conference it's worth including some more detailed information from the book that actually penned the domain model pattern so first thing when do you use it use it when you have a complicated and ever-changing set of business rules now like every where do you use it where do you not use it in that same book page hundred and nineteen it says if you have some simple not null checks and a couple of sums to calculate a transaction script is a better bet what that means in short order is that the domain model is not I repeat not a best practice you do not use it for everything there's going to be a bunch of complex stuff in a given system and some not so complex stuff this idea that all of your business logic is supposed to be in one domain model well that causes some problems that you might have heard about in various movies the one domain model to rule them all and in the darkness bind them now if you've built a system using domain driven design and you have a big complex domain model this is usually the reason why it's just putting too much stuff in that one place and when people are going to do event sourcing these days they're doing the same kind of thing they're taking what essentially is a big monolithic domain model then they're saying awesome now we're gonna make that event sourced the pain that we're feeling with our one domain model to rule them all when you put a vent sourcing in there that doesn't solve the problem it makes it worse and that's where people end up in that trough of disillusionment it's the expectation that event sourcing is going to solve the pains of the one big domain model so hopefully as an industry and I've seen speakers talk about this already saying you don't want to be doing monolithic architectures with event sourcing alright so having a vent sourcing in one big place not a good idea but that's true for all patterns alright so now that we've put that caveat aside about saying again event sourcing as compared to traditional domain models maybe we should start talking about what is an event in the first place I mean it's one of those words kind of like service weather micro service or service-oriented architecture or domain service that we tend to throw around a lot but when it comes to event what do we even mean by that so hopefully you're human me a little bit for some computer history because it's important to understand where things came from in order to understand what they are and what to do with them today so in the beginning of computers we had this thing called an interrupt an interrupt was a way to signal to the processor that something needed immediate attention in the early days of computer programs were essentially static void main you start at the beginning and you go one two three four five six seven eight nine ten and then you're done that's how code was written interrupts came in to say wait a minute something happened that's important we need to stop and take action on that so things like somebody typing on the keyboard or doing a mouse click or running out of space on the hard drive or any number of things like that were hardware interrupts that essentially told the processor stop what you're doing and handle this that was essentially the first concept of events in computers as we had them from that point in time graphical user interfaces came along and they created slightly larger concepts of things like a button that could be clicked at this point we moved from just a hardware level interrupt to a more abstract software level concern saying hey we had a button clicked event maybe we should do something about that and this is where we went from just sort of the hardware level to the software level where we created a different programming construct it wasn't a class it wasn't a method wasn't a pointer events were a new kind of thing that were built on top of all of those existing pieces in order to help us build in this type of reactive fashion saying something happened we need to handle it so when talking about these events we need to distinguish there is the event itself the button clicked and then we have the code that handles that thing this is where programming actually started going in an interesting new direction because you could have multiple callbacks for a given event this was a fairly new thing you got to remember in the early days of programming what we had was structured programming were essentially you had one code flow that you would go all the way through he'd have if-then-else but this idea that multiple pieces of code could be triggered and that the calling code wouldn't necessarily know what it was was a brand new thing at the time now there were people at the time that said that would create an unmaintainable mess that not knowing what code is going to run next is gonna make just you know big messes developers won't know how to handle that but here we are decades later and it's fine it really is so you got callbacks methods on the classes with arguments and this is where things started to get a little bit complex because when we're talking about these arguments that are giving us additional data about the event that's where we start to get into the sort of in-between state of saying well what is the event and what are the arguments and what are the callbacks so again in gooey type territory in a mousedown type of event you'd have a whole bunch of arguments you'd have the index if you're clicking on like a tooltip or tab strip type thing you'd have which button was clipped was this shift button pressed what's the XY coordinates of the of the mousedown event you'd have a lot of this stuff and essentially these were considered arguments this was not the event the event was mousedown these were all the arguments that were passed along with that a number of years later we as an industry came up with this thing called the parameter object pattern which essentially said all of those arguments are ugly it makes it difficult to maintain you end up having difficulty versioning your API because you end up with all sorts of overloads between the arguments so essentially we said let's take all of those arguments and package them up together into a single object you add a little bit of strong typing and an ID and you get something fairly modern that looks like this where the mouse event args is your parameter object and the mousemove thing that you're seeing over there that is the event the mouse event handler is a strongly typed delegate or callback type pointer that will tell you that your method to handle that should have that specific signature essentially we started to create some structure around these concepts of events so that's on the let's call it programming language side and computer history side of how are we getting to event based programming but now I want to come at it from a slightly different angle because there was this parallel thing happening outside of the gooeys called component orientation that made a big difference you see when building more complex offer not just simple GUI type interactions we realize that just having big structured code with methods calling other methods calling other methods doesn't work leads to an unmaintainable mess so the idea was if we create loosely coupled and highly cohesive components that are separate from each other that we can evolve independently that we have data hiding we have encapsulation all of those ideas from object orientation but slightly bigger that that is going to do good things for us off of the back of component orientation and that level of decoupling when we included the event based patterns that we were just talking about a minute before of saying you know what if we want truly loosely coupled interactions calling a method on an interface that's not bad I mean it's better than calling a method on a concrete class but event based interactions are even more loosely coupled than the request response style interactions they're more flexible and that's essentially where we the paths that we took that brought us to event-driven architecture so you'll often hear these two things used interchangeably of driven architectures and event sourcing but they're not the same thing at all right a venture of an architecture essentially is component orientation plus an event based interaction model please note there is nothing about event-driven architecture that indicates that it's in any way distributed across the network I'm gonna say that again there's nothing about event-driven architecture that indicates that it requires or is based on distributed systems having multiple processes running in multiple places the idea was very much from what we nowadays call a monolith from a deployment style how do you decouple that so that it is more maintainable over time it wasn't about scaling not at all not for like a performance scaling perspective it did have to do with how can we have multiple teams working in parallel on the same codebase and loosely coupled highly cohesive components with an event-based interaction were a better way of doing that now when we started talking about event-driven architecture that's when we started to engage in a little bit more languaging around this thing of events where the components that created produce those events we call those event producers the components that handled the events were called event consumers and there was some kind of mechanism for getting this thing called an event with its arguments from a producer to a consumer that was called a channel now when you have everything running in process it's not really that big a deal to figure out how to have these channels that are communicating with each other but in the early days of event-driven architecture things like how do you create a thread safe event dispatching model that was a tricky thing it didn't just work nowadays we kind of take that for granted oh absolutely you just raise an event and lots of consumers get callbacks and it just works but it didn't start there now one of the things that I see happening a lot unfortunately still is that people get hung up on these terms and consider a component either an event producer or an event consumer but not really both it is common to have event producers also consume events and vice-versa it almost always happens that you'll have some code receives an event does something and produces other events so in that case the language of event producers and event consumers is not really that useful architectural II speaking essentially in an event-driven architecture everything consumes events and everything produces events and this is hopefully where you're seeing where that's going to lead us over time towards an event sourcing mentality but it started first from that element of loosely coupled highly cohesive components now once you start building event-driven architecture based systems that's one thing when you have an organization that has a whole bunch of systems lying around and you need to integrate between them that's where you get the distributed worlds and the event-driven worlds start getting shoved together so integration in the early days was hard you had applications on different platforms this was before Jason this was before XML this is when we were for the most part using CSVs for everything unfortunate CSVs in their systems technology I gotta tell you it just lives and lives and lives in you're like I already died don't need this anymore and it just doesn't it refuses to go away and that's part of the challenges of integration now messaging technology which was originally called message oriented middleware was designed to make that easier to cross all those boundaries to simplify the integration between platforms now when you had an event-driven system that was built as a bunch of loosely coupled components integration with that thing ended up being a little bit simpler because those events were really good candidates for passing to other systems saying hey something significant happened in system a we know because an event was published in the event-driven architecture let's take that information put that into a message by taking the name of the event and putting that in the header of the message taking the arguments and putting that in the body of the message and then route that to some other system that's going to be interested now this is where things kind of shifted in terms of event based programming to this more modern concept of what we mean by an event because for the most part we distinguish between the event itself mouse move Mouse down whatever and the arguments as a payload with messaging because we couldn't really do the sort of transparent just different platforms publishing and subscribing in a transparent manner we required a message to bundle those things up together more explicitly now one other small piece of history in event-driven architecture as it's done in process almost all of the routing that you see all of the sub bribing and registering to events is done based on the type of the event essentially the name of the event you don't really see code that's saying I want to subscribe to the mousemove event only when the xy-coordinate is within this space right you can't really do that kind of GUI programming but when people started to do messaging based integration that's when this content concept of content based routing started to creep in of saying I'm not only interested in the type of the event I want to actually deal with things that are in the payload itself that was not originally part of event-driven architecture not at all it's only later on when doing message based integration that this concept of looking at the payload for where to route things started to become relevant so it's important to realize how things happen what the history were in terms of where that leads us today so where are we today when we say the term event essentially what we're talking about is not just the type but it's that name plus the arguments essentially it's a data structure that has some kind of name that tells you what happened and a whole bunch of key values that will tell you this is the data of what that thing happened now the important part is that it's about an action that's already occurred in a GUI type setup button click mouse move all that stuff is easy it's when you move into loosely coupled highly cohesive components territory when you're doing a venture of an architecture at a business level they're talking about what happened becomes a little bit tricky so we'll talk more about this you know what are these events what are the business events what are the distinction between business events and other types of events because that's where most event sourcing goes wrong it's essentially saying an event as an event as an event now again intuitively we understand the difference between a button clicked event and some sort of higher-level business event like signed up a new customer or a customer bought an item but in between there there's a lot of gray that we're gonna talk about almost in all cases you will have some kind of infrastructure that's going to move these events around historically that could have been a messaging based type infrastructure for passing those things between systems dealings with transactions and consistency and all that kind of stuff or in an event sourcing type way the act of persisting a stream of events and bringing that stream of events back and doing snapshotting there's going to be some kind of infrastructure that's helping you with that but here back to the where we are today before using event sourcing on a complex system stop a second alright so again there's that whole hype thing we're going to use event sourcing and it's gonna solve all of our problems just stop right there remember what we said about that domain model pattern that monolithic style of putting all of your logic in one place if you're if you see yourself heading in that direction making that mistake that's where you need to stop again even before you're doing event sourcing but also I'd say before you're doing DDD or domain models or that kind of thing so if you're building a complex system my guess is that sort of at an abstract level you have this kind of set of layering in your mind you have some kind of user interface invoking some kind of API usually remote which will invoke some kind of business logic which will ultimately influence data that gets persisted in some database you're thinking and most of the thinking in the domain driven design community is is that business logic layer right there is your domain model I have news for you it is not why because of that one ring fallacy in order to comply with our principles of what actually a domain model is event sourced or otherwise you need to take your original idea of what a model of your domain is that has a bunch of entities like customers and orders and products and employees and those types of things this entity relationship diagram and you need to decompose that into something less monolithic so how do you do that well it's hard but I'm gonna give you some ideas remember what we mentioned about domain models you use them when you have complicated and ever-changing rules but not when you have just some simple not null checks to and sums to calculate so let's look at let's say a given entity like customer customer has a first name is that a complicated and ever-changing rule answer is well really I mean we get some logic that says well the name can be 40 characters it can be 50 characters I changes a little bit for a while but then it just sort of settles down we'd never change it again so complicated ever changing not really customer has a last name also same category what about the name of a product the description of a product the image of a product a lot of the data that users see and interact with is not really that complicated it definitions don't change that much by virtue of the definition of the domain model pattern you don't need domain models for that data so you start separating those things out if we start looking at things like a product price or a customer status those are things that over the lifetime of the system will see that the business changes them quite a lot it says okay we're gonna start off with preferred customers and non-preferred customers great and then a year into the system the business as you know what we're changing that we're doing gold silver bronze customers gold customers get this silver at that okay great we're gonna change that behavior over that then they say okay great but we've got this other category of strategic customers so strategic customers are one thing and then you've got gold silver bronze which is another thing and you'll see that type of behavior and data changing repeatedly it never really settles down also things like how much of a discount does a certain status of customer get on a product well if the product is this price then they get that price or they get this kind of discount you start seeing a bunch of the logic changing together between the customer and the product but it requires time to sit down with your domain experts and kind of sift through what things really change together versus what things change separately so saying hey we're gonna take the customer first name last name phone number that kind of stuff we're gonna put that in one domain we're going to take the product names descriptions images we're gonna put that somewhere else we're gonna take customer statuses and product prices and the logic around discounting we're gonna put that somewhere else this is the first act of finding your boundaries and from there we can start asking the question saying which areas are complicated and ever-changing and we'd say hey you know that customer first name last name thing it really is just taking some data from the UI shoving it into a database and showing it again as it said in the patterns of enterprise application architecture a transaction script is a better bet you don't need a domain model for that domain model pattern wasn't designed for that product names and descriptions essentially you got a product catalog again you're taking a bunch of data from the UI in the back office storing it and then showing it to users in the front office data in data it's what we'd call crud again you don't need a domain model for that event sourced or otherwise only when you uncover the subdomain that is that kind of complicated and ever-changing that is where say okay that might be a place to use the domain model pattern so essentially instead of that simple layered architecture that we saw before we start subdividing that and saying hey it's not just one set of layers essentially there are these slices if you will that kind of start at that domain level going down in the sense of well I don't actually need the customers first name and last name in the same table as their status information because really those things vary independently of each other from a single responsibility perspective they have different reasons to change so when we think about it it's no longer just one system essentially it's these sets of boundaries we're not only going down but also going up we can subdivide them so this idea of what does a product look like I mean on the screen it looks something like this and it has a bunch of information like the product name the description the image the author the price the inventory but essentially that gets divided as well that the product catalog information the name the image the description belongs in one boundary the rating of that book belongs in another boundary essentially all that's shared between these two things as a product ID in both cases we've got product ID and then a bunch of data and then in another one we've got the same product ID and a bunch of other data and then that price product ID and how did we come up with the price inventory and the product ID essentially we end up with what nowadays is called micro services where each micro service has its own data but notice that it's not like we have a product micro service that owns everything about a product or we have a customer micro service that owns everything about a customer when looking for boundaries you'll usually find them going right down the middle of your entities where different parts end up in different boundaries so essentially instead of thinking about this in terms of layers what would happen if we took those vertical slices and said you know what those are our primarily primary architectural concerns essentially we've got these boundaries boundaries one two and three each of them can be built differently they don't all need to have the same layers one of them could be fairly simple based on a document database as a UI some basic validation logic as separate components but that's it another one because of scaling reasons dealing with lot more data needs to have a sharding type database so it has some different logic for physical scaling out yet a third one has more an event sourcing nature to it therefore we use an event store underneath it in order to support that but here is where I want to come back to that concept of events we have events passing between these higher-level boundaries in a publish/subscribe type manner and then we've got things happening inside the boundaries which may or may not be event oriented essentially it's a question of events on the outside versus events on the inside now some of you might remember probably about 15-20 years ago paper was written called data on the inside versus data on the outside really great paper if you haven't read it I suggested strongly this is the exact same idea boundaries matter things on the outside need to be treated differently from things on the inside so when thinking about high-level business events what do we mean by that well let's take an example of a kind of retail flow where we have these loosely coupled highly cohesive components interacting in an event-based manner meaning we're doing event-driven architecture where one boundary publishes the order accepted event multiple subscribers listen to that one of them takes action on it publishes another event saying order build shipping listens to both of those things and then decides to take some action on that here we're talking about stable business facts and not so much things at an internal level so if you're thinking of a domain that has a more create read update delete relationship to its logic and data so entity created entity updated entity updated entity updated entity deleted those might be internal events that happen in one of those boundaries but those are not the higher-level nice events that we're talking about when thinking about these level of events the concept is it's something that happened that is not planned to unhappen or to be overwritten a stable business fact now here I want to talk a little bit about terms or patterns that you've probably heard about in the context of event sourcing auditing and replay so when talking about auditing events essentially what we're doing is we're creating a copy of the events that were passed around and storing them somewhere almost all messaging infrastructure already provides this capability out-of-the-box so if you have this type of flow and you're saying hey in my business it's important to have a tracking of all of the events that happened that can be done without doing event sourcing all right event auditing is an independent pattern a lot of times people again lump that together because they think of event sourcing as one thing remember our definition event sourcing is a collection of patterns it's not just one thing then auditing is one of those elements that allows us to essentially record stuff that's going on and it's very useful in a number of systems once you have all of these events audited then you can replay those events if you want to now at the business event level to be totally honest with you there are so many challenges with event replay I gotta tell you you know if you're integrating with another system that's not event sourced and distinguishing the now we're handling real-time events versus now we're replaying historical events and make that you don't pass those events along but these ones you do but if you're correcting an event then pass that one but make sure that you don't do that when you're in the replay of the correction mode it's just too hard right people twist themselves up into knots saying if we're doing event sourcing we must do event replay and then while we're integrating with another system because we have to how do we make that work under the presumption that they must do event replay all the time at the level of these sort of high-level business events really I'd say 99% of time it's just not worth it there are some special edge cases where that might be interesting and some of you might have certain areas where that's happening but again keep those in a separate boundary than everything else events on the outside versus events on the inside so now that we've talked a bit about the events on the outside I want to talk about the events on the inside because again as an industry we do a pretty bad job at naming things we use events for everything so one of the places that events make a bunch of sense is when we want to notify a user let's say that their order was shipped all right now you might be thinking is that an event on the inside or an event on the outside there might be relevance at a higher level business events have order shipped but definitely on the inside to say hey when something happens inside shipping we're going to use a signal or we're going to be using Google Cloud messaging we're going to be using Azure notification services whatever in order to push a notification to our user a lot of times developers use the word event when they mean notification try to be specific about that notifications are things that are most relevant on the inside within a specific bounded context of pushing data from one physical location to another that's a notification now that might be driven off of some higher-level business occurrence like the order was actually shipped but distinguish the technical part of moving the notification between physical locations from the business events itself so that's notifications scenario number two that often gets bundled in that related to integration between systems inside billing we've got our retail system but you know the organization's got a data warehouse it's got a big s a PRP system the guys in finance are using an Oracle finance type thing and we need to get things that happened inside of our system into those other places again this is a situation that a lot of times people say I'm gonna use event sourcing for that but remember what I said one of the big challenges with event sourcing is the replay part when integrating with other systems if your domain is there's a big part of integrating with other systems there are other tools for that change data capture is an infrastructure ability that many databases have that will allow you to essentially stream out the changes to a given data source to other places this is a kind of data synchronization technology change data capture add some ETL on top of that for transforming the data from one format to those other formats and this is just you know in raishin 101 we've been doing this forever it works again 95 99 percent of the time you don't need to build eventsource domain models that replay themselves in history and all that kind of stuff in order to solve this problem this is mostly a solved problem already one other case that often gets talked about saying we need a vent sourcing to scale hammering the same database for both our reads and our rights it just doesn't scale for us we need to have event source projections creating read models somewhere else so that then we can read off of those things fast for example building a product catalog we need to be able to serve up lots and lots and lots of queries really fast but as we mentioned before the domain itself is not very complicated and ever-changing specifically the part of showing images and names and descriptions of products on a screen its crud and I don't say that in a way to sort of belittle it say this is fundamentally a data problem and you know what databases are pretty good at solving these types of data problems they have this concept called replication pretty much every database out there has it where you write changes to a write master node and it asynchronously replicates those changes to read replicas as many of them as you want essentially you have an event-driven model at the data level itself that already gives you a synchronously updated read models it just isn't event sourcing its database level replication its infrastructure and half behind-the-scenes you just open up a connection via your database client and say hey Postgres I'd like to read something says hey I'm gonna give you a connection to one of the read replicas hey Postgres I want to write something here's a connection to the right master and then you do sequel it's really not hard a lot of this stuff when we say I'm gonna use event sourcing for that it's not because we have to it's because we're just not familiar enough with the alternatives now again you know event sourcing has its uses we'll get to that but it's important to understand that there's a whole bunch of these subdomains that there are simpler and better ways of tackling them I'm gonna use one other one that's not also really that event based either but a lot of times it gets modeled using event sourcing so for example in the sales domain customers who bought product a also Bart's products B C D and E essentially we could describe that as a series of events where we're recording every purchase that a customer does and then we're building projections off of that to help create these types of recommendations per user to say what should we recommend to user number one versus user number two versus user number three essentially what I've seen unfortunately a few too many times is people are using event sourcing to build what is essentially a graph database because that question of saying customers who did X also did Y and Z and all of these relationships between them that's what graph databases are designed to do so questions like figuring out connections and strengths of connections and ordering them based on each other this is like the whole space of graph databases so a lot of the event sourcing stuff that you might think of in this domain in many cases are again inside the graph database you don't need to build that yourself the database technology provides those things for you I'm gonna give you one more example not so much a business domain but a technical domain monitoring my guess is that a lot of you need to monitor your distributed systems and you've got lots and lots of events that are coming through you need to do event logging you need to do event tracing and you know just the scale is enormous you're getting more and more events and as you add more systems you know just the sheer volume ends up getting to a place where regular databases don't handle it and you're saying hah now I'm gonna do event sourcing because events and I'm appending them and here I don't even necessarily need to do sort of update type of events but essentially I'm sorry there's another category of database technology it's called time series databases that is essentially what they do you put data into them that is time-stamped in this type of manner and there are a number of examples of that in flux DB is I'd say probably the better known one Netflix has built their own because well that's what Netflix does and they call theirs Atlas and there's all sorts of differences between them Atlas is a purely in memory sing influx is replicated and has some durability to it but essentially when I see people saying I'm gonna go build an event sourced monitoring type thing what they're doing is building their own time series database without even realizing it and having that intermingled with the actual logic of the domain so again part of the challenges around people doing event sourcing is that they look at anything that looks remotely event-driven saying on an event source that thing without taking the time to do a deeper analysis of the nature of the domain what part of it is complicated and ever-changing what parts aren't what kind of persistence technologies out there suit them and I gotta tell you I mean some of you're gonna be looking at this and saying yes but in flux I tried that it didn't scale for me atlas well it's not durable so I have to build my own thing yes at scale everything becomes custom when your Netflix yeah you're gonna write everything custom yourself when your Amazon you're gonna write everything custom yourself the point is to distinguish when are you writing infrastructure versus when are you dealing with the domain itself and hopefully through some of these examples I've given you ideas of various types of infrastructure that are out there the mapping to domain type of examples and essentially you know that element of thing you know what I don't just because something has events related to the domain doesn't mean I'm gonna go do a vent sourcing for that so here's the question what is event sourcing actually for right we're having the whole conference on it what's the point so there are specific cases like I said you know in the 1% to 5% of the cases where you're essentially dealing with a complicated ever-changing domain where there are sort of two main criteria first is it's by temporal meaning there is the time at which a thing happened as a transaction time and then there's a validity time if your domain experts are always talking in the language of this is valid as of or effective as of that's an indication that you have some by temporalities to your domain some domains have more points of time there is a valid from there's an effective from there's an effective to there is a corrected as of if that's a core part of your domain event sourcing could be a good fit for that however this is the important thing code as data becomes an important part of that once your data is as of effective of and all those types of things you also need to be able to run rules as they were at a point in time because that's the challenge with long-lived complex systems the code that was running v1 is different from the code that's running v2 which is different from the code that's running v3 which is different than the code that's running v4 if you're gonna be replaying history with all of this history of events you can't replay that history with the new rules essentially you need to replay history most of the time as it happened meaning being able to persist this was my logic as of a certain point of time this was my logic at another point in time so essentially your code is not just a whole bunch of if-then-else statements and loops that you're writing in a domain model your code is data as well this is complex this is not something that you should just sort of say oh absolutely the domain expert used the word event we're gonna go do event sourcing in order to apply event sourcing patterns to a domain this is essentially the bar and while you'll be doing that in maybe one sub domain in most you won't need to so there is one other category where it's not really domain oriented where event sourcing is used a lot this is what we were talking about before event sourcing is for building infrastructure the folks that are building in flux DB you bet they're doing event sourcing a lot of the technical patterns for saying hey I want to bring up a new node in the cluster I'm gonna take the history of the transaction log and replay that in memory using snapshots to make this thing faster or absolutely relevant when building a kind of database technology the folks that are building neo4j graph database you bet using event sourcing patterns left right and center but it is mostly in the domain of infrastructure that we do these types of things much less in the actual business domains itself so when do you do that when you're at Netflix scale when you're at Amazon scale when you're at Google scale when you're doing that type of thing that all of the existing technologies out there just don't cut it okay great when you have the time when you have the resources when you have the budgets when you have people who've actually built things that are this technically complex at scale then yes absolutely that's where you do event sourcing but for those of you that are not at Netflix Amazon Google scale I want to just shine a light briefly on sort of the what can you do so when you're not at that super duper level of scale Moore's Law is your friend folks I gotta tell you you can scale up far beyond anything that you've imagined so I looked this morning the biggest Amazon ec2 instance that you can buy has a hundred and twenty eight virtual CPUs and almost four terabytes of memory it's a beast you're thinking you know I might like that that could probably run all of my system and then some but it's probably too expensive that's people's sort of general assumption about scaling up too expensive if you were to look at the price of a reserved instance for a three-year term saying this is an important system I know I'm gonna have to run it for at least three years Amazon give me a price how much does that cost my guess is those you thinking that gonna be an arm and a leg it costs six thousand two hundred and fifty dollars a month nothing up front if you pay a little bit upfront it costs a little bit less than that sixty two hundred dollars for in just absolutely unbelievable amount of processing power that is cheaper than most developers by a longshot it is absolutely cheaper than a team of developers doing event sourcing and domain models and CQRS understand that this is today three years from now Amazon will have the yet other SuperDuper quadruple extra-large datacenter Enterprise Edition on steroids that's going to give you 4x what that is for the same price or less monolithic deployments where everything just sort of runs together can scale up pretty damn well now this idea again I'm putting our planets here as an example thing this is not you're not building the Sun you're not building the supernova if you're at sort of let's call normal real-world levels of scale understand that hey there's a lot you can do you don't have to go and eventsource all the things and keep in mind that at you know during this three turn the business is gonna change anyway right I mean you built this system and then a year later than say we're gonna want a whole bunch of changes and then six months after that they're gonna want a whole bunch of changes and then they're gonna want some more changes and then you're gonna realize that all of the architecture that you've set up it's not gonna fit anyway who's ever had that happen to them two years into production the rules changing you have to change the architecture yes yeah we need to accept that incorporate that idea of re-architecting a system while it's live that's just the way that things are going to be having the right boundaries in place can allow you to reaaargh attacked bits and pieces at a time and simplify that but for for god sake keep those pieces as simple as possible in the beginning scaling them up as we mentioned before and I know that you know all of us have a certain amount of professional pride which is why we come to these conferences to learn and see the patterns and see what's the best way to create maintainable clean code but after you've been working on a code base for five 10 15 years it doesn't look pretty right it's this kind of platypus it doesn't really it's not a DDD it's not a layering it's not a CQRS it's not an event sourcing it has a little bit of everything the issue is we tend to view that as a failure but it's not this is just nature itself this is what long-lived systems end up looking like it's not pretty but it's functional it allows the business to keep going and evolving now hopefully I've you know scratched the surface set the stage giving you some ideas about what's going on in and around event sourcing event driven programming event-driven architecture we have a lot more speakers coming today that will go much deeper into the details and an examples of things that I've talked about at the beginning you you are interested in more about things like how to divide up responsibilities how to get your boundaries right I have a bunch of videos online at this link Oh particular net slash DDD EU 2020 where I go into a lot more depth of various domain examples how to find those boundaries the practices around that and that'll sort of set the stage for ultimately building these types of more complex systems again that URL go particular net DDD EU 2020 now for those of you that are interested in doing event Riven architecture happened to be using dotnet my company particular software we build service bus technology that makes that easier if you like take a look at that we've made some recent changes to it from now on you'll be able to use it for free and development so please give it a try let us know what you think and with that I'd like to thank you very much my name is Judy de haan I'm a 2d dahana Twitter and have a great conference everybody thank you [Applause]

Channel: Domain-Driven Design Europe

Views: 9,688

Rating: 4.8842974 out of 5

Keywords: ddd, dddeu, ddd europe, domain-driven design, software, software architecture, cqrs, event sourcing, modelling, microservices, messaging, software design, design patterns, sociotechnical

Id: -iuMjjKQnhg

Length: 67min 47sec (4067 seconds)

Published: Mon Oct 05 2020