Getting Started with Event Sourcing in .NET

hello everybody I'm Nick and in this video I'm going to get you started with event sourcing in software engineering and this is not a net specific tutorial even though I'm a net Channel and I will be showing things using net and cop it doesn't matter what your background is what I'm going to do here is break down the topic in so simple terms and such simple code that even if you're doing Java or python or JavaScript or anything you'll be able to take this video and adapt it to your own language event sourcing when being taught tends to be presented in a very complicated way when at its core it's a very simple topic but I think because it's a lot of fancy terminology people tend to get confused by it so in this video I'm going to ditch all that and I'm going to just explain it in plain English and just build on top of basically nothing and you will understand when the terminology comes in once you learn the concept now this video is sponsored by AWS and if you want to find out how you can get $25 worth of free credits on WS check the description down below if you like our content and you want to see more make sure you subscribe for more training check out my courses on do train.com okay so let me see what I have here and as you can see all I have is a conso application with zero dependencies and literally nothing that's why I said you can actually take this and adapt it to any language we won't complicate things the topic is fairly easy to understand if you bring the terminology in later and the main idea around event sourcing is that state in our application is not just a row in a database but every time something happens I store that action I store that thing I store that event in the data base in an append only fashion I never go and say update this thing that happened in the past because that thing happened in the past and in a given moment in time it was true so what I'll be doing is if I want to reverse that in the future then I'll go back and have a reversal event but that thing still happened the great way to explain this is if you pull up your phone and you go into your bank account and you see that you have $10 then those $10 are not just $10 it was Zero when you open the bank account someone sent you 100 so at the time it was 100 then you paid for I don't know something that cost $50 so your bank account was 50 and then 10 was added so 60 then 50 was removed and you're back at 10 so these are all events in a given moment in time where you got money or you spent money and a great example of reversing things is what if someone gives you a refund for a product well if you pay $50 and you have a take out $50 then eventually that refund will replenish those $50 and to get the final number all you have to do is say okay what are all my transactions in my bank account oh I have these 10 transactions let's go ahead and add on remove all that money from whatever it was in the beginning of the balance let's say zero now if that sounds inefficient to do every single time in a real application they're all way around it but the fundamental concept is that things happen and you build State out of the things that happened let's go straight into the code because there's no better way to explain this than actually showing you code so I'm going to go ahead and create an inmemory database to store my events I'm going to call that student database and I'm going to use doome train my platform as an example here so let's say a user is created so a student is created then they can update their profile maybe they can enroll into a course or maybe they can unenroll from the course or I can unenroll them and to store those events what I'm going to do is I'm going to create a new directory and call that events and I'm going to create a very simple class here an abstract class called event and what this class will have is two things first events are things that happen in a given moment in time so I need a way to store a timestamp so I'm going to say day time and I can say created at the normal terminology is timestamp I'm going to use created at UTC here and then I need a way to correlate a bunch of events because let's say that something happened for Nick and Nick has id1 in the system and then you have John and John has id2 we still need a way to identify those two users things can happen within those users but I still have id1 and John still has id2 my stuff my events and John's events will be distinguished by separate streams streams of events imagine it as a sequence of events about a given thing in your system in this case the student well streams can be identified by stream IDs so all of my events will have a stream ID here so stream ID here we go in this case I'm going to use a good to represent My Stream ID that really depends on what you're doing some systems use a string it can vary but in this case I'm going to use a stream ID and I'm going to remove the setter from here I'm going to turn that into an abstract good so here I have my base event now I want to represent first what happens when a user is created a student is created so I'm going to create a class called student created note that I don't have a student class this is very important so here I'm going to create my model what would a student created have well a student ID of course and you can argue this should be ID but because events are a separate sort of thing there's not so much ambiguity in terms of what each thing represents so I'm still going to keep it as student ID here then full name email and date of birth now this will be extending the event class that I just created and I'm going to implement the missing member which is a stream ID so the stream ID in my case will be mapped to the unique identified of the thing I want to represent so in my case it is going to be the student ID and I'm going to do that and that's that there's nothing else to touch I just say that the stream ID for my student is the student ID then all I need to do is add all the other actions so what would I have I would have student updated so again this would extend the event and what would I have here I would have a student ID and I can only update the full name and the email mail you can't really change your date of birth so you shouldn't be able to update it then I'm going to have the student enrolled so that would look something like this where I have the student ID and then the course name the course name could be a course object itself I want to simplify this so I'm only going to have the course name so when you enroll to a course all we do is we stall the name of the course and then I'm going to have the unenroll where someone can say oh I want to unenroll from this course maybe to get another one and so on so now I have all of my individual events so now I'm going to go to my program.cs and create a student database instance so new student database and I want to go ahead and create a student how would that look what you would say is that a student has been created so I would have a student created object using the student created object I just had now to have some persistence with the IDS I'm going to be using I'm going to hard code a good here for the student so I'm going to say student ID ID is that and then I'm going to pass all the other properties so email full name and date of birth and that's it and that represents now that the student was created and then in my database all I'm going to say is student database do append so I'm going to append an event in my database that a student was created obviously the append method does not exist so I'm going to go ahead and create it but it's not going to be a student created it's going to be an event so any event we'll call that append method to be saved in the database now of course I need to store these things in memory so I'm going to say private readon and I'm going to store it using a dictionary where the key is a good where the good is a student ID and then I'm going to use a sorted list even though it's not necessarily mandatory for this example but I want to signify a thing that's very important here and that is the ordering of the event and I have intentionally admitted if you're very familiar with event sourcing things like vision from the event maybe that's something to explain in a different video but the key here is the daytime when the event happened and then I'm going to have the event itself and I'm going to say student events equals new and that's that so what this sorted list is supposed to represent is that stream per event in this case the student events so what I'm going to say V stream equals and I'm going to try to get that stream for that student so I'm going to say event. stream ID and if you don't find it return null so if the stream is null then take the student events with this stream ID and create a new entry so equals new sorted list of this type and that's it and now I've created the stream for that student ID and I can start adding things into the stream all I need to say is I want to take that and say add and I'm going to store the eventor created at but I'm going to have to set that date as well to daytime UTC now so I'm going to store that here and then I'm going to store the event itself here we go and that's it all I'm doing is I'm storing it in a sequence in a stream I'm saying add this at the bottom of this sorted list based on the date it was added so now if I go back to my program.cs that's it that's as if I save something in the database now what if something else happens for example what if I also want to store that the user was enrolled well I would have something like this where I have that student IDE and I say enroll the student in the course from Zero to Hero rest apis in net and then append that and what if the student decides to change their email well very easy all I'm going to say is student updated and I'm going to pass Nick chapsas doet tr.com and then the full name and student ID and then click append and that's it and if I just quickly run this to show you what's happening behind the scenes if I just stick a break point here maybe and I say debugged let's take a look at what's happening in fact let's go through every single step so first I'm going to go ahead and don't want you go away GNA go ahead and just step over this and step into the append method so we're going to try to get the stream the stream is null because there's nothing about the student then I'm going to create it I'm going to set that date and then store that event and then if I go to the next thing you can see in here now that in the count I have one stream that has one event in it and I can keep going this time we had the stream and I'm saying just add it at the bottom because this happened later and this happened later so if I go in my database over here and I click stored event and expand that there's a stream for the student and it shows what happened in sequence student was created then they enrolled in a course and then they were updated I have everything about this student now this stream contains everything about my student and if I needs to know what happened with a student I also have a form of auditing by default but what happens if I want to represent my student in some way well there's a way to do that as well so I'm going to go here and and create a student class so I will have a student class but it's a bit different than what you might expect that student class will have all The Usual Suspects so we have an ID we have a full name we have an email enrolled courses and then date of birth we have the properties as you would normally have but then we're going to have a few methods because what we want to say is that the state of this class should only change through these apply methods now I know I have these as Setters so they're not really mutable there's a reason for that which comes into the picture later so for now I'm going to leave it as it is and what I'm going to say is I'm going to have an apply method per event that can mutate this student class so I'm going to say student created for example I'm going to say student created and then in the apply method I'll say okay what is this event changing about my student well I'm going to have the ID set I'm going to have the full name which is also a part of that event so student created do full name I'm going to have the email as well so email is set and I'm also going to have the date of birth right so all of those things are set by the created then I have updated which sets again the ID which I could skip but I'm not going to do that so the ID the full name and then the email because those are the two things we can change yeah we can skip that because we won't really be updating the ID so we can remove that and we do expect to have a student didn't created before updated ever comes into picture so there's that and then we also going to have apply methods for enrolled and unenrolled and you might be able to see where this is going because those are all private and they should be private and there should be only one public apply method in our scenario anyway which says that I'm going to accept that event and then I'm going to map it to the right method I'm going to use a switch here there's other techniques and there's other approaches some people call this evolve some people call this process I'm just going to call it apply and all we're going to say is take that event determine the type of this event using a switch and then call the apply method appropriate for that type and now you should be able to see exactly where this is going because now I can say that my student if I want to represent my student is a student database. getet student which doesn't exist but I can still pass the student ID in here and I'm going to go ahead and actually use the student object over here so the right method can be created so public student get student get student ID and I'm going to go ahead and show you how easy it is now now first and foremost if the student events does not contain key student ID then return null because just we don't have a student to return but if we do have something then create the Bare Bones of that student object and then get the student events using that student ID so now we have the events and we're going to say for each student event in student events go ahead and call the student. apply method per event and because that is a sorted list I'm going to have to call Value and in the end return the student so now I'm going to build the state that represents my student on the fight based on those events so let's go ahead I'm going to tell this into VAR and let's go ahead and see this again step by step so I'll go ahead and step over the database and create my student so created then enrolled then updated and then I'll say get my student back so I'm going to step in here does the thing contain it yes it does so I'm going to say new student get the events I have three of them they are sorted by the date the time that they happened so I'm going to go here and go ahead and apply every single one of those events setting the state of that object every step of the way and then in the end what I get is a materialized view that represents that student there's a few things now that come into picture when it comes to terminology so I have a stream containing all the student related things so I have a stream per aggregate to represent that student and then the student in this case is also the aggregate root we don't really need to explain that complicated terminology that much you understand the functional aspect of things and that's really what matters so I have the ability to append only what happens to that student and then build the materialized view to represent that student based on what happened now there's a few things that you might immediately think of and the first thing that I would think in this situation is that isn't it a bit costly to just go ahead and pull all the events from a stream for this specific aggregate for the student and build it every single time wouldn't it be easier to just sort of keep uh the state or a snapshot or a materialized view of that thing in the database or maybe in our system and then just return that and the answer is yes and that's what's called a projection so let me show you I'm going to go ahead in the append method and I'm going to show you how we can build what's called a synchronous projection and it's going to be synchronous because we're going to be updating that state as we append new events we're still going to append events but we also going to update that state now whether we're going to do it synchronously or asynchronously depends on the scenario you are into if you're in a read heavy scenario or if you're in a right heavy scenario and also if you can deal with some eventual consistency or if you absolutely need strong consistency I don't want to go too in depth into those Concepts because I do have a video coming for that as well I I want to keep it simpler so let's go ahead and see how we can build a simple projection what I'm going to do is I'm going to create a separate dictionary here I'm going to say private read only dictionary good and then student and that will have my students in here and then every time an event comes in what I'm going to say after I store that event is go ahead and the students dictionary use the stream ID which I know is a student ID here and set get my student State now to do that I'm going to use the get student method and I'm going to pass down the event. ID and that is definitely not null at this point because I have at least one event that I just added and then if I want to get that view what I can say is public student get student view and then I'm going to just return that student so I'm going to say students. get value or default I'm going to try to get based on the key otherwise return null and that's that this is definitely not null at this point so I'll go here and I'm going to say student and then student from View and I'm going to say get student view and I'm going to stick a breakpoint and quickly show you what's happening so I'm going to still append everything every step of the way but I'm not writing one thing I'm writing two things every time so I build first this state by just going through every single event but I also if I step in here here I can go into the students thing that contains all of my materialized views and I can see that I also have stored that and I don't need to go and get everything now and then rebuild it I can just go ahead and return it from that storage so that's all really really nice and fundamentally that covers a big portion of what event sourcing is now the alternative to all that is of course not doing this synchronously because remember if you're in a real database scenario you would have to wrap this into a transaction because the addition to the database of that event and the update of your current state should happen at the exact same time and they should succeed together or fail together so it should be Atomic now what I want to do to just give you a better understanding of everything is bring a real database into the mix and for that I'm going to use dynamodb I've built event sourcing systems in Dynam DV in the past and Dynamo DB and also things like Cosmos DB as well they are great for that sort of thing because of their partitioning strategy and Dynam DB especially because the terminology I think and all the features that it has makes this such an easy experience and if I was to build an event sourcing system I would use something like Dam be I know this librar is like Martin that use postest to do that I'm more of a Dynam person and that's why I'm going to use that here so what I'm going to do is I'm just going to quickly close all the tabs and I'm going to add Dynamo DB into the mix the V2 library and that is it and I'm going to keep it very very simple I'm not going to complicate at all I'm going to go to the student database and I'm going to remove these in memory representations and I'm going to replace with a private read only I Amazon Dynam DB and I'm going to say Dynam DB here equals a new client and that client will have a region of EU West to because that's where I am and I'm also going to say Prime private const string table name and the table name is students for this case now I'm just going to quickly show you how easy it is to create the table in Dynamo debut I'm just going to say create table I'm going to say students here and then I'm going to set the partition key and the sord key event sourcing and no secret databases play very nicely together especially because of the partition key so partition key is not necessarily like a primary key in a database because the uniqueness of an item in database is like Dynamo to be is the combination of the partition key and then the secondary key in this case the sord key which as you can understand by the name it is also sorted by default based on the value so what we're going to do here is create a partition key and we're going to call that PK and that's for a reason I will explain later and then I'm going to call the S key SK and that's it and I'm not going to customize anything else all I'm going to say is create that table and that's it we're going to give it a second but we're not really going to tou it until until we push some data into it and then I'm going to start updating this code to use dynamodb so let's go ahead and just delete everything from here but still implement the exact same logic so in a pend I just want to save I just want to write a thing in the database and that's it now because we're going to be writing a lot in Json and how calization works and inheritance works with things like inheritance and generics in cop we're going to have to change the append method a bit so I'm going to turn this into a generic ttype parameter and I'm going to say where T is event and I'm also going to change this to an async task because we're going to be awaiting things so this is now aend async and I'm going to do the same for all the other methods but for now I'm just going to delete them now because I'm using inheritance in my events I do have to change this event class a bit to let it know that it is inherited by some all the classes so the serializer and der serializer of Json knows how to give me the right type back when I say realize this event back into a student event to do that first we have to say that this is using Json polymorphism and then I'm going to have to add a Json attribute per type I'm storing and I'm going to have to also specify the name of that event so when an event is stored and then it's retrieved the right type is used to serialize it and distalize it that would look like this and again if you want to grab any of this code it's in the description down below now since I'm going to be using a part partition key and a sort key for everything I'm storing I'm going to need two properties in my class so first I'm going to say Json property name and I'm going to say PK and I'm going to create a property which is a string because that's what I created in Dynam DB and it's going to be called PK now the PK is getter only and it's actually computed based on the stream ID so the string version of the stream ID will be my primary key and like I said the uniqueness is the combination of the primary key and the sword key which if you want to get very very technical daytime might not be the exact thing you should be using as a sord key you might want to use a long time stamp but I don't want to go too in depth in this the idea is that you could have clashes if for some reason you have two events with the exact same created ad in the same stream it's an edge case but I didn't want to mention it and then the sord key over here is going to be SK and SK and it's going to be the created at using the iso format which is also string sortable why is that important because everything in that stream now in Dynamo DB will be stored by order and then all I need to do is in student database it just say write this event so first I'm going to set the created at UTC as UTC now then there's a bit of code to serialize everything down to attribute which is what Dynamo DB expects and then I'm going to to create a create item request which is a new put item request what we need to pass down is the table name as the table name and then we need to pass down the item as attributes which we already have and that's it then in the end Amazon Dynam B and put item async create item request and just awaited and that's that so let's go back here and update everything appending append they snc I wait it I wait it and over here I waited again I'm not going to check the students first what I'm going to do is just run this code and show you how easy it is to just store everything in that table now so we're going to create the database instance we're going to go ahead and say append and then I'm going to set the values turn everything into attributes put item request and then call the put item a sync and I'm going to do that for everything and as you can see everything run and now if I go back to Dynamo DB and I click students and I say explore table we're going to see three items and they all store the type name of that event so student created over here all the details about that event and I can go back and see all the individual things in order using that sort key so student enrolled and then later as you can see student updated so now it's a matter of retrieving it from the database and building that view and this idea of doing it on the Fly sometimes is called a live project jection where you don't really store it or synchronously or asynchronously you just calculate it on the fly so first let's go here and say create the method for get student a sync so we get the student by student ID now Dynam theb is a bit weird when it comes to quering but that's what a query would look like so we say we have a key condition in the primary key we pass down the primary key value like this s represents a string and our key is a string so we do that then we're going to fire that request to get a response so we're going to say I'm Dynam be query async and we're going to say if response do items or do count actually is zero then we just going to return null this means there is no event for the student so there's no student basically then we're going to convert the attributes which is the item as attributes uh as documents using the from attribute map and then we're going to have to convert the documents into the objects by going through a Json Midstate first that would look something like this where student events on the documents select Json serializer and then deserialize that as an event but now for this to work I actually have to change one thing I'm going to have to pass down some Json settings because that type attribute needs to be taken into account and I'm not dealing with the order this is a bit of an edge case but I'm going to add it anyway so all I'm saying is these are my Json cizer options and I'm going to pass them down over here when I say uh deserialize this object and then in the the end I'm going to do the exact same thing so VAR student equals new student and then I'm going to loop around everything and everything should be ordered by default already because I have that sort key if you don't trust it for some reason you can still have something like U order bu and you can say x.c created at UTC so you'd have that and then you would say apply that student event and then return the student that is not null and that's it so let's go ahead and actually update this program and in fact I'm going to delete all the students first so let's go here and say just delete all the items and let's go through the flow again so breakpoint the top and let's go and not only create the student and retrieve it of course I need some code to actually retrieve it so await a sync we have that let's do it so let's go through everything we're going to store all the items as you can see we're storing them one two 3 here we go and then in the end let's go in here get the request get the response we actually have a count of three because three were the items that we stoed three individual events we go back we get the student events and then we loop around them we apply them and as you can see created first then we have enrolled and then we have updated and in the end I'm getting my student as an on the fly materialized view and that's it and that's really cool but what's even cooler is that dynamodb actually supports transactions meaning we can actually store that projection that materialized view as we're saving the new events and they can fail together and succeed together so in here instead of having this put item request like this I can do the following I can say our transaction request equals new write transact write items or items request and then I'm going to have transaction items I'm going to create that uh list and then first I'm going to write uh the event itself so that is a put so put equals new put which contains the table name and you can have cross trable transactions as well in Dynamo DB um so if you stored your projections and your events in Separate Tables they could still be part of a transaction and I'm going to say that the item is the item as attributes then I'm also going to have the new uh write item and that would be still a put on the same table but the items I'm going to store is just the student the materialized view so to calculate that what I'm going to do is I'm going to go up here and I'm going to say VAR student view equals and I'm going to read to get the current state of the student now you can see where there could be performance issues with this approach too but I'm reading from the projection and in many cases if you need that synchronous strong consistency this is what you're going to have to deal with I'm going to say I wait get student async and don't worry we're going to update that get student async method to use the projection instead so I'm going to say event do stream ID otherwise return a new student and this is only relevant when we have the first event only but it's still good to have and then we're going to say student view. apply and we're going to apply that incoming event so apply that event and then serialize everything into Dynam format so that can be done like this to turn everything into an attribute map and then in the end I'm just passing it down here and I'm saying await Dynam DB transact right items async and pass that request and now if I go ahead and I just clear the database again let's refresh and just delete everything here we go bye-bye and then update get student a sync to read from that projection so I'm going to say just comment this one out in case anyone wants to grab the code and get the previous version the signature will be the same but the code in it will be different now the way I'm going to write this is a bit interesting so what I'm going to do is I'm going to say that I have the get item request so we not listing anymore we're not querying we're just getting a point read for a single item which is very very efficient and then I'm passing partition key and sord key because that's what uniqueness is and the S key and the partition key student ID and then this underscore view or you can say projection or you can say state or current it's completely up to you what you want to say I don't want to confuse you with terminology I just want to show you how you can do it conceptually so now that means that I can do a very efficient Point read get that response using the get item a sync method and then if there is nothing then just return null otherwise do this conversion D and return everything back as a student and that's it now as you can understand here this means that I have to also add in the students object that partition key and sort key logic because this will also now need to be stored in dyb same thing as before the only difference is that I have to add this view suffix to prevent it from being in the same stream as everything else to make it even more efficient so you don't have to read and ignore a file and once I have that in place again my database is clear so I'm going to say go ahead and just run it step over everything and I'm going to say create an pend that first event and I'm going to go to Dynam DB at this point so if I refresh that you see two items here one of them is the view the other one is the event so the first event student created and then the second thing is the view that shows me the current state of that student so date of birth email and Ro costes and so on then if I go and I say step over the enrolled event what's going to happen is I have now three items The View and the new enrolled event so that exists but also my view was updated and I have that enrolled courses over here and then I can keep going and going and eventually read that item and you can see I have everything but this is now coming from dynamodb from that projection from that view that state and that's how you can do it synchronously with a synchronous projection but you can also have an asynchronous projection and Dynam be is excellent for that because it has a CDC mechanism or a change data capture mechanism built in and the idea is that every time you change something in the database in a table you can get a notification an event that contains that information you take that and then you do anything you want with it so if you didn't have this transaction what you could have is you can have a new Lambda over here I'm going to go to the bottom and say that I want a new simple Dynam DB function and I'm going to say event sourcing tutorial. Lambda and what this Lambda can do is every time a document is updated in that table then I get a notification and this function Handler method is triggered and I have the records that were updated or changed or deleted or whatever so what I would do is I would say record Dot and I would get the Dynamo theb stream record and then I can get both the old image and the new image I'm going to get the new image over here and I can say to Jason in fact I can go ahead and update this client and these events to be able to do that here we go and I should be able to say to Jason and then convert that into whatever I want and then update my projection using that asynchronous flow and that's how you'd have an asynchronous projection to create that state and that's it fundamentally that is the concept you just keep appending events of things that happened and then you can create that state and the great thing with this is if you use case changes in the future and you need a different way to represent that data then you can just recalculate that projection based on the new requirements there's no data lost you can always go back in time and see exactly what happened and it's a very very nice way to build software I hope this was easy to understand please leave a comment down below if you want to learn more about this topic because these are things I've done quite extensively in the past but I just tend to not talk about it because I don't know how interesting it would be for everyone but if you think that's interesting to you leave a comment down below let me know and I'll make more but now I want from you what do you think about all this and is event sourcing now easier to understand leave a comment down below let me know well that's all I had for you for this video thank you very much for watching and as always keep coding

Channel: Nick Chapsas

Views: 50,895

Keywords: Elfocrash, elfo, coding, .netcore, dot net, core, C#, how to code, tutorial, development, software engineering, microsoft, microsoft mvp, .net core, nick chapsas, chapsas, dotnet, .net, .net 9, event sourcing, get started with event sourcing, event sourcing .net, event sourcing c#, Getting Started with Event Sourcing in .NET

Id: n_o-xuuVtmw

Length: 37min 6sec (2226 seconds)

Published: Mon May 20 2024