Event-Driven Architecture: I do not think it means what you think it means

with the popularity of microservices tooling like Kafka and event sourcing the term event is pretty overloaded it's actually caused a lot of confusion on what event driven architecture actually is there's different concepts like event carried State transfer events is notification and event sourcing that if you confuse them and conflate them together you'll end up with a lot of unneeded complexity I'm going to unravel what these concepts are so you have a better understanding of what event-driven architecture actually is hey everybody it's Derek Martin from codeopinion.com if you're new to my channel I post videos on software architecture and design so if you're into those topics make sure to subscribe this video is brought to you by eventstore DB the stream database built from the ground up for event sourcing cqrs and event-driven micro services for more on event store DB check out the link in the description so first let me talk about the context and some of the concepts that I'm going to be explaining and where they're really applicable and where I see them in most oftenly use so I'm really talking about large systems and they're being decomposed hence why you might be using them if you're defining services or microservices this is where you're often using some of the ideas behind event driven architecture so generally these are line of business and Enterprise type systems that I'm referring to so Gregor hope posted this a few weeks ago I'm actually more a month ago now from this video and he posted by the time a new it term gains mainstream adoptions it's guaranteed to already have lost its meaning and I think this is true in our industry in a lot of ways as well as I always view this for Martin Fowler the semantic diffusion which when a word is coined by a person or group it has a pretty good definition and then it gets spread through the wider Community it weakens that definition now the thing with event driven architecture it's not exactly that way it's just taking on kind of different forms and different ways of applying it kind of over time so I'm going to talk about kind of the various ways that really a venture of an architecture is applied so there's a few really good blog posts that kind of illustrate this I've covered this blog post this was from a wix.com developer Nathan still nitsky and they have this five pitfalls to avoid kind of in their journey to a venture of an architecture and a lot of the posts here like I love these posts because they do illustrate they really are kind of common things that people run into but it also has a part of it where it is kind of confusing different concepts that I'm about to illustrate there's also another post called don't let the internet dupe you event sourcing is hard and I'm gonna have a link to these two articles in the description but this post as well really kind of exemplifies if you confuse different concepts together you're going to end up in a bad place all right so event driven architecture you keep using the heart I don't think it means what you think it means sorry I had to post that Kevin on my private Discord server also recommended it when I was talking about doing this video so I had to so what are we talking about with Adventure of an architecture the four Concepts I really want to talk about and how they're used is using events as a way to persist state for data distribution and as a way for notifications if we break these down it really comes into talking about event sourcing event carried State transfer domain events integration events and workflow events so all these are all kind of the concepts I'm going to talk about and if you confuse these things like I said this is when you kind of get into a pile of hurt I think one thing I gotta leave out a little bit and I'll save for a separate video um is event streaming although I will talk about event streams and I will talk about tooling that's built around event streaming and speaking of tooling some of the confusion here not necessarily because of tooling but because some tools can do multiple things you have different things here like and service bus which is a messaginglibrary.net we have event store who sponsors a bunch of my videos thanks to them which is a database specifically for event sourcing and then there's different tools like dibesium which is a change data capture tool that can publish events to Kafka topka topics so I have Kafka here and then which is a distributed log then I have something like rabbitmq which is more your typical Cubase broker and then I have your Cloud infrastructure there like Azure service bus for messaging event grid event hubs and then SNS sqs Etc on the AWS side and there's many many more different tools not that any of these tools are causing the issue of confusion it's just that a lot of them can do multiple things so it feels like you should just do all of the different ideas and concepts with one individual tool so first let's talk about event sourcing so fundamentally event sourcing is a way to purchase State instead of persisting State like this let's say we have a relational database or you could think of a document store and a document let's say we have a particular product in a warehouse and it's skew that product identifier is YYZ 987 we have a quantity on hand of 27 we last received this on a particular date and we Last Ship that product out on a particular date now with event sourcing the idea is instead of recording current state like that rather we record the events the facts that we can use to then derive what current state is so we record everything in a stream of events that is kind of getting us to that place so what's what really happened how did we really get there well we probably actually received product from a shipment from our vendor so we received 30 on a given day then we received another shipment let's say on the exact same date of a quantity of five so now really if we're kind of keeping track of our quantity on him we're at 35. then let's say we had some orders that we had to ship out so we shipped out an order it had a quantity of six so I'm at 29 and then let's say we did a stock count which is typical in a warehouse accounting all the particular products and we realized oh two of them were broken so we need to adjust our quantity on hand so we're now should be at 27. so our representation of current state could be we know when we shipped it we know we last received it we know our quantity on hand this is actually how we got here the series of events things that have happened these business Concepts that have happened in the past that we care about that's how we can get to current state and that's how we record what our state is so when you're thinking about a stream of events it is usually for an aggregate or something unique let's say our product so if we're thinking about our relational model here we have different rows let's say we have another product der five seven six and it has a quantity hand the same type of information with event sourcing we would have another stream of events for that particular product to record all those different state transitions so those occurrences of things that happened through its kind of life cycle so we have product received for that other product we had some products shipped for it and shipped again so we have these two different streams for these two different products in our warehouse so what am I ultimately describing well I'm describing that event sourcing is about State it's how we persist state so Greg young who you might be most familiar with for event sourcing and has probably been the spearheading person to lead all this charge is finally apparently writing a book I follow him on Twitter he posted this a few months ago but I know he's been heavily into it and I took a screen capture because this what he said was one of the most important pieces so far that he had to write of the book when he started it and this is the section that I've highlighted I think is the most important is event sourcing is that we will store facts events which have occurred any state that we have is derived off this set of facts and any state that we have is transient I replied to Greg I said I wish you would have coined it fact sourcing because then we wouldn't have this confusion with event sourcing and Hell events but I still think it's very applicable but again the key here the key is event sourcing is about State why this is so important is because people confuse this with the idea of using events that are State as a way to understand what's happening in another system which I'm going to talk about as a form of communication but the thing is is just like any other service that you have that has its own database we don't do this anymore we don't have service a reach out to service fees database we don't do this we realize that that databases for service B that's its own internal implementation detail of how it persists State we don't go reach to it rather we have to reach out to service B through some type of API some type of contract that we have we can't just go middling and trying to reach out to its database specifically so if that's the case the same holds true if you have if you're doing event sourcing in service a and you have an event store you can't have some other service just randomly go and reach out to your database your internal implementation detail just because you're event sourcing doesn't change the fact that somebody else can go reach in to your service boundary and interact with your database directly so event sourcing is about State it's about persisting state of likely your Aggregates and it's in internal implementation detail why this causes so much confusion and Technical chaos and complexity is because people want to treat it as well as the other form of using event driven architecture which is using events as a form of communication don't conflate these two ideas here so when we're using events and event driven architecture as a way to communicate between different service boundaries what does that really look like well there's kind of two forms of this that usually play out there's more behavioral which are using events as a means of notification to notify another service boundary that something important has occurred and secondly which I think is probably becoming the most popular and I got a little asterisk here because I want to talk about this more this is a form of Distributing data and typically this is kind of more in a crud form so create read update delete usually done by the idea of an entity or a little bit more fine grain by crud by property and I'll explain what these are so why would we want to distribute data around to different Services why does this happen so typically why this happens is because let's say we have a client call service a and we have certain capabilities that we provide here but for some reason we need data from service b or for some reason we need to make a call to service B if we're doing that in a synchronous way VIA HP grpc Etc if we have any issue with service B the problem is that our request from service a originate from our client is going to fail and this is when people start realizing okay well I don't want to have the service to service synchronous communication all over the place rather what I would do is move to something asynchronous so that I have the data that I need when I need it when a call is made that way when a client calls our particular service we just reach out to our database and we have all the data we need we don't need to think about making that call to service B so how do we get all the data that we need that's when the idea of kind of propagating data comes up so the first thing that usually happens is people will create an event kind of more like I was saying that more in the crud sense and they'll say that a product has changed let's say we're talking about a product in a warehouse we're gonna have our product changed event that's going to get published to some topic uh for a particular SKU that a product identifier we're identifying what changed so what will happen here is that let's say service B is the one that's publishing that event and service a it's the one that's going to subscribe to that product changed event and what it's going to need to do then is it's going to then make a callback a synchronous call back to service B to say hey this product change give me the all the information I need about that product so that I can then update my local database which is kind of like a local cache about that product information so that's kind of the first step of really what people start doing here the second one really relates to well what happens if we make that synchronous call to get that product information the product change but uh oh there is a failure I tried to make that call back and I can so I have this event that I need to process I know now my data that I have in my local database is stale but I'm trying to make this call and it's unavailable there's a bug there's some issue I can't get that data so now the problem is we have stale data and what exactly are you doing with that data in your local database so what's the next progression of this is starting to include the state in the event so then what often happens is we have more maybe a more granular event something like the product price changed event and I want to focus on the names here because I'm mentioning they're very crud kind of centric they're product price change or or product changed Etc so now what we're doing is we're saying that's the event it's a little bit more granular it's just changed why it changed I don't know but here's the skew for that that identifier the product and here's what the current state is now when that price actually changed so let's say that it's 80 dollars you may go a little bit more of a fatter event which then becomes okay well I don't want to keep track of the Deltas because maybe based on our UI this is really difficult to do so we're just going to publish a product changed event include all the details about that entity because we're kind of energy Centric here of the SKU maybe the name the price all that details I don't care what changed we're just gonna publish to this event and let the consumers just deal with all the properties and whatever they want to update so our events kind of get started as maybe potentially kind of just these Delta changes which may or may not be hard to figure out a lot of this is derived by your UI and then it can get a little bit fatter where we have all this state within our events so what happens is when we publish that product changed event with all that data or even just the product price changed event now we don't need to do that callback we don't need to call us back service B we can just use that data directly within the event and then update our local database so there's various change data capture tools one of them being an example here of dbzm which can sit on top of MySQL or postgres it understands for example in MySQL the bin log and it can then publish the changes the Delta changes of your update statements Elite sensors Etc and it can make turn those into events to publish onto a Kafka topic which then you can have different sync connectors to then output that somewhere else like for example elastic some data warehouse wherever you want to get and basically be pushing that data to but why do you want to do this exactly typically this is because we're thinking about entities entity services and not necessarily at the Forefront thinking about behavior and then the data that we need for business logic so instead of having a monolith if we have services or microservices whatever that have their own databases the data that they own we end up starting to duplicate all this data all over the place when we're really kind of still in this distributed monolith even though we're not making request response from service to service we're still doing it asynchronously because still we're still treating it like we need data from another service now I have videos on doing UI view model query composition where you do need data from multiple Services the often things though is why do you need that data if you need it to perform some type of business logic realize that you have stale data because it's coming from another service even if it was a request response if it's still data it's local data that you have it's stale data can you perform that business Logic on stale data now like in that slide with the CDC and dibesium and mentioned data warehousing and to me that's actually a really good use case of where you want to have all this disparate data kind of conform into one place that you could actually do some reporting on some bi Etc the difference with this as opposed to what I'm getting next is you can still have the same type of idea of having these Services own their data but rather what you're doing instead of propagating and kind of Distributing data from service to service via events really what you're starting to use events for are notifications and they're business Concepts I alluded to this before in the name it's not that the product changed why did the product change oh well it's a product price changed well why what was there a sale did we increase the price did we decrease the price it's actually the idea behind this is driven by the domain it's the business it's things that they actually care about and you're going to be using these for notifications for Integrations with other service boundaries and workflows so there's different ways of thinking about these it's not to get into the nitty-gritty I have videos on this but there's the idea of generally about domain events which generally can be considered kind of inside events it's within your Dome your service boundary of that particular subdomain you may have integration events that you want to expose because these are a little bit different in the sense that you could think of them like your API through your contracts these are going to version differently than something internal to you like domain events but regardless both of them you're really using as a means of notification so let's talk about why you would use them as notification now I'm going to go through a transition here to really explain this because what I'm going to get down to is really the event contains really nothing more than an identifier so first kind of the way you could think of this is let's say we're going through an e-commerce uh checkout process so we have our basket our client goes to the ordering API and submits all the data related to let's say our credit card information shipping information Etc and then what we do is that we're going to publish an event called an order placed event with the details of the address the payment information all the items everything related to that order you kind of think of this as the event carried State transfer we have a lot of data in that event and then from there it gets published to our broker our request to our clients done okay thanks for your order your order's been placed asynchronously obviously through publish subscribe we have the payment service that's going to be subscribing to that event to that particular topic wherever that resides and then because it has all that information in the event it could then persist that to its database the payment details and let's say that at some point then it has to reach out to the API Gateway that it's using to charge the customer's credit card all that information was contained in the event yes something like credit card we could be encrypting those properties or the message itself but generally the idea what I'm trying to get across here is we still have these really fat events that contain all the information and the reason this is is because we're just not thinking about kind of the idea of that a service should have the data that it needs it's really about the workflow so rather the alternative here is when our client makes that initial checkout request to our ordering service let me just really kick off the workflow so at this point we could be saying to our ordering service okay here's the items here's our shipping address for example and then from there the next part of the process after it saves that data is a client then might send that payment information to the payment service here's that order ID or some deterministic ID that we've generated by a client or ordering that started it gives it to us here's the payment information here's my number here's the expiry and at this point what the payment can do it can just persist that information at this point it's really kind of temporary storage because we haven't done anything with it we haven't placed our order we haven't had to charge the customer's credit card it's not until the client and our UI tells our ordering place now yes I reviewed my order I actually want to place this order where we can then make our state change to actually place that order and then we can publish that order placed event simply with our order ID this is drastically different we don't need to have all that data within our event because because the boundaries that require certain data already have the data that we need so we're really just using this event as a means of notification to our payment service so that it can consume that event when it's published and you can say oh yeah I have that that payment information for order ID 123 then I can use that information that I already have locally to then go and hit my payment gateway to charge the customer's credit card so what I'm describing here is using events as notifications and a way to do this is event choreography to the goal on that example a little bit farther is that you're just having Services consume events and publish events and really they're unaware that they're a part of kind of a business process in workflow so let's take that one a little bit further where when we had our order that was placed we could then be publishing that order placed event to our message broker to a topic and from there we could have our payment Service as an example it's the one subscribing to that order placed event and from there just as earlier it has the details so it go hit the payment Gateway if that was successful maybe it publishes a payment completed event and that's going back to our broker maybe to a different topic that the warehouse is the one that cares about that specific uh payment completed event it's the one that's consuming it from there it could then create a shipping label with whatever shipping Services we're using but the idea here is that there was no defined Central orchestrated place of this workflow we just have different Services consuming events and Publishing events they don't really know that they're a part of a workflow but this is event choreography and really you're using events as a means of notification now if you're new to event driven architecture and you're thinking well how can I start I don't really have all these Services maybe I'm even in within a monolith and this is actually a great place to start is thinking of it the same way but a set of workflows between different Services think of just the ideas of things that you can do asynchronously when something occurs via a notification an event so the classic example of this let's say we have our service for sales and it publishes that order place that well within that logical boundary of sales maybe there's things that we need to do like obviously send the email confirmation maybe we need to have some text message SMS go out to our customers maybe we have a web hook system where we want to call some other third-party HTTP API when an order is placed all these can live within that same logical boundary that service and we could just have consumers for it meaning we're producing the message that event and we're also consuming it so we could have all these different consumers the one for email let's say that's talking to AWS SES to send the email we have this homegrown web hook system where we make calls to http apis third parties and then maybe we again we're talking to twilio for sending out an SMS so the idea here is that you can also be the consumer and the producer it doesn't have to be that you have Services you can do this within a monolith and this is an actually a really kind of great way to start is trying to look for those places where something occurs like an order place an event and then you want to do something else that isn't critical um in terms of happening within the same trans transaction especially like sending an email so back to tooling why does this matter because depending on what you're trying to do are you trying to use kind of data duplication everywhere and I hope for the right reasons are you trying to do event sourcing are you trying to do more pub sub messaging potentially also with the idea of commands and doing orchestration then you're going to want to use the appropriate tool for the job yes some of these kind of cover a bunch of the ideas but don't be confused that just because it's suitable for one that it's suitable for everything so as always just understand the tools that you're using and understanding the concepts and if those tools support it so kind of to go over this event sourcing what is it it's about State it's about persisting State and you're using events as a way to do that think of them as facts they're internal they're an implementation detail they're not used for notifications if you're using event carry state transfer you're kind of using that as a means to distribute data are you doing that for kind of query reporting purposes realize that if you're Distributing this data you have stale data yes data stale the second that you select it from a service or even from your own service but there's a lot of implications when you're trying to use that data to perform some type of business logic at that point I'd be asking really take a look at your boundaries where the data resides should it really reside there the idea of having events as a form of kind of communication as a notification for workflows for business processes there's different ways of kind of dealing with these workflows in orchestration which is a combination of commands and kind of Cubase work and Pub sub which I talked about doing strictly with choreography which again is kind of having these long-running processes that really don't have a central spot but you just have events be consumed and events being produced so hopefully this provided some insight so when somebody says to you that they're using a venture of an architecture you can ask a little bit more deeply specifically how are they using event sourcing are they using event carry state transfer are they using events for notifications at the very end of this video I'm gonna have two videos that I think are the most appropriate to this topic that will probably cover it on a different angle so check out those if you enjoy this type of topic these types of ideas and you want to talk to other developers about software architecture and design you can join my channel all and get access to a private Discord server the links in the description on how to join if you found this video helpful please give it a thumbs up and as always please subscribe for more videos on software architecture and design thanks

Channel: CodeOpinion

Views: 40,269

Keywords: event-driven architecture, software architecture, event sourcing, microservices, kafka, event driven architecture, event-driven architecture spring boot, event-driven architecture node js, event-driven architecture aws, event-driven architecture c#, event-driven architecture microservices, event-driven architecture kafka, event-driven architecture vs microservices, event-driven architecture patterns c#, event-driven architecture vs rest, event-driven archetecture c#

Id: iAA7PTqs4xY

Length: 26min 47sec (1607 seconds)

Published: Thu Feb 09 2023