Clean Code with Entity Framework Core | Brendan Richards

[Music] so welcome to my talk clean code with entity framework core and so the real focus of this talk is to introduce you intensity framework core everyone's got different levels experience with entity framework in general but then taking the core concepts of entity framework for core and looking at how we then use that is within a cream code architecture so the introduce myself my name is Brendan and I'm a solution architect SSW and that means I spend most of my time at client work building websites of various kinds various sizes various scales and so most of my time is spent doing this stuff for full production but now and then I get to kind of take a break from that and do a bit teaching I do the user groups like I am now I do some act ace and the training the super power stores and they're fantastic for really cementing kind of your knowledge it really helps if you want to learn something explain it to somebody else so MIDI has helped you get your um your thoughts in order in a previous life though way before I moved to Australia I was a Java developer and when I was a Java developer I had some cool things and run about 2003 one of the cool things we had in the Java world was called an hour em the first one I had was called the object relational bridge and then a little bit later I started using something called hibernate and the idea of those was data relay libraries to talk to a database which I'll cover in more detail and then over time in 2004 moved to Australia and I found myself in a place where there seems to be much more dot Network about then there was Java works I started doing dotnet development but I found that I missed a lot of my libraries and I found that when I looked at their standard Microsoft stuff there wasn't no one was talk about dependency injection and no one was using RMS or open source loggers so I started looking around and find things on the Internet where people are taking java packages and paused them across to.net and it works quite well because it was a huge equivalence between Java and.net back in those days and so I started finding libraries like enloc and and in hibernate that did the work for me and then later on Anthony framework came out and I started using that instead and over time entered a framework pretty much replaced hibernate in the RM space finally coming to now where we've got the latest version of.net core about to be released I'm going to talk about a few things today and we'll start by introducing entity phone montecore and some really key fundamental things there even if you've seen it before it's worth going over because based on my experience I see various usage patterns used in the wild that I want to try and stamp out I want to try and you know get people to understand how it's working behind the scenes and therefore where it's the best way to use entity framework and cover the basic usage and then I'm gonna take a step back and start looking at clean architecture and then how we can fit clean architecture or fit into the framework all into a clean architecture setup then there very briefly cover testing EF core performance and finally I'm going to cover what's new in the latest features of entity framework or three so let's start for introducing entity framework core this is the blurb straight from the entity framework or home page and it says enter the framework core is a lightweight extensible open source and cross-platform version of the popularly of the popular entity framework data access technology doesn't tell you much there it says entity framework is a version of in C framework but the next lines does tell you a bit more it says entity firmer can serve as an object relational mapper ORM so I'm gonna start by highlighting that sentence those words object relational mapper and start talking about what that means so an object relational mapper ORM was created to solve what has been called the impedance mismatch and the impedance mismatch is all about differences between how a database handles data and how an object oriented programming language handles data so let's take a look at a database if I want to put something in a database I first have to create something called a table and in that table I'll create columns and each column has a name and a datatype then if I want to put in that table I just fit in those values I've created a table there of people if I want to take some more data and make it related to that data what I can do is I can create a second table fill that with values and if it wants a link the green table back to the gray table what I do as a crate what's called a foreign key so the value in the second table that matches the primary key value in the first table and that creates what's called a one-to-one relationship one record in the green table matches our links up to one record in the gray table if we want to make this a many-to-many relationship I can make sure there's no unique key on the foreign key there and that duplicates in there I can now link to skills it's in my people table if I want to make a many-to-many relationship though I can't do it this pattern what I have to do is put a relationship table in the middle so this type blue table in the middle was composed just of foreign keys and that links between the two so we're gonna have many people with many skills and it's a true kind of relationship table now an entity framework six you could point a system at at the entity framework and it would generate this middle table for you their entity framework or still can't do this but sometimes I find that's not too much of a drag because quite often when I'm building software I want to put some kind of auditing against this data so who created it and what time they created it if I haven't also generates table with just foreign keys in the middle I can't tell who linked these two things together once they do that so quite often they'll actually deliberately put a middle entity in the middle just to handle that relationship mapping so let's take a look at what happens when we start talking about object-oriented code so in object-oriented code you have this concept of an object an object can contain data plus behavior so I've created two dumb classes here person and skill that basically match the data types you've seen in the previous database but if I want to link person to a skill I can just put a reference there because I've got in memory reference pointers and I get a bi-directional relationship I'm gonna put reference on either side or put references in both directions so that the person knows what skill it has and the skill knows what person it belongs to if I want to make amends a one-to-many relationship I can use a collection then optionally I can put a corresponding reverse index reverse relationship going back or leave it out or from many-to-many I can create a link each side so all in all when you're talking object-oriented code it's richer than what you can do with just databases because databases are GS columns and tables of data with object-oriented code you have data plus behavior in object-oriented code you have classes inheritance polymorphism and encapsulation and you have pointers and collections that can model relationships so overall object-oriented code gives you a much richer toolkit for modeling data so we've talked about the the impedance mismatch and how entity framework works to manage that let's now start start by looking at some of the core features of entity framework and the first thing to unlock all the DB context so the DB context is the most important class in entity framework it's kind of the root thing you create to talk working with your data and it's your interface the data for another class called the DB set the DB context this is the most important thing is acts as a unit of work and tracks changes to the entities that have been loaded in stat DB context and we'll cover that in more detail a bit later it's also quite smart they can sort out all the relationships between the entities you've put into that DB context so it works out what's new what's changed what links to what which foreign keys have to be placed and there's quite a few things for you and it sorts it all out when you save at the end it's also really important to manage the lifecycle of your DB context because the DB context tracks changes today so you're working with it uses memory to do so if you have a DB context that lives too long it's gonna start filling up with more and more memory into your entire data base is there in memory though if you're working with a dotnet core web application that's really easy to manage because all web applications are naturally bounded by that request you have a request coming in from the web it pricks of a process to handle that request and finishes at the ends so generally you'll you'll work with a DB context with a context per request pattern and that's the fault pattern you get when you plug entity framework into your own web components so let's take a closer look at entity framework and how it tracks changes the data via various entities States so create a few one boxes here ones the database and ones it's big books called the DB context and inside it a bunch of entities and by entities I mean these are plain C shop classes that have been loaded from the database and hydrated wensley framework and they're kept inside the DB context so the first and most simple thing I can do is I can say DB set or add and what this does I can take this do this with a new class so in my code I can type new customer and they'll create a customer object I can feed in properties of that customer object and then I can say ok dbcontext take this object and it's put it in the added state so what happens there is DB context accepts that it keeps that interesting in the added States and then if later on I hit Save Changes it will for a database insert the key thing here is no database work happens when I say DB said to add it all waits until later when I call that save the next state to talk about is an unchanged so this happens when I load a database a load an object from the database it gets put in the DB context but I haven't touched it make no changes to it so it's now in the unchanged state this happens if I though DB set where and load a bunch of records with a matching okay predicate case or DB set defines to load them by ID and DB a set that I find is useful because it will also case results if it's all in the DB context if you don't find it by ID knows has already got it there it won't bother touching the database to get that that object for you into the state that modified this is what happens when I've loaded a customer from the database and I've got that object if I then in any way touch that customer objects go customer that name equals and then change the value the DB context intercepts that operation knows it's changed and marks it is modified so all I need to do to change data is modify it and then Save Changes later the DB context knows that I've changed it it tracks those changes for me density of state detach is quite interesting here because this is what happens when a loading object I put it in the DB context I keep hold of their objects but for some reason I close the DB context behind that object so now in this state I've still got that object I can still look at that customer I can read that customers properties I can modify that customers properties but it's no longer attached to a DB context context so it no longer knows has that path back to the UM database there's another feature of entity framework that was on by default instead of mmx6 and is available as an option an entity framework core and that's a feature called lazy loading so lazy loading is all about managing relationships between objects where say I've got an order and that order has a list of order items I can load the order but not load the order items yet what will happen is the DB context for lazy loading turns on will monitor my use of that object if at any time I go order dot items in my code it'll intercept that operation and say okay you haven't got that yet but I will for you go and fetch that data behind the scenes with an extra database query so it's really useful for situations where you've got a few layers in your application you've got a custom bit a cost a central bit of code that fetches things and you pass that down to the next layer and don't know what the next thing is going to do with it because then on the fly it will load that data for you but it's really bad for situations where you're rendering out a whole table of data because you might end up with what's called the n plus one problem so you want to display a table of 50 orders and I want to also display information about those order items I will make 51 databytes requests because i have one request to fetch the original list of orders and for every row I'm going to throw our second request to get the order items so it's called n plus 1 ones get the data and one extra one two it's right through the things so what you can do there is you can turn to the framework to preload those orders or you can turn lazy loading off and actually manage that by hand so lazy loading used to be on by default instantly found look six and made it controversial because there's lots of code being written by people that was inefficient loading far too much stuff into memory so now it's off by default which is much safer but you now have the option to turn that on so coming back to this diagram we've got a detached entity so what do I do to get there and back in place or what I can do is use what are considered to be the most misunderstood commands in entity framework it's called update the problem with the word update is update is also the name of a database operation that you want to do if you want to update data so naturally a lot of people see this thing called update on the DBA set and they assume that means I need to call that maybe a single time I want to update my data but it's not the case what update is for it's for taking a detached entity putting it in the DB context and marking it as in the change state if I have a entity that's always been the DB context and the content was never closed it's tracking the changes I don't need to tell any frame with it's changed in fact if I call updates I'm going to tell into the framework for the whole object is dirty where potentially NC family cuts the capability to only update the fields that have changed so if I use the update commands but I don't need to I'm going to do more database work than I should so the update command is specifically for taking the detached object put it in a DB context and calling it and and marking it as modified and in fact hibernate and then hibernate went through a whole round of this exact process where they had an operation called update and they realized it was misleading so they changed it to merge later on later on but density family if it went through that and that process yet and finally at the very ends I can call Save Changes and save changes are powerful operation if you're not interested in usually I could you just quickly rehash what you just see it again just quickly summarize this beautiful me about using their update command ya had how did I get my data that is detached is that side a query I did five minutes ago and I've kept as a memory yes and maybe if you if you for example put an object into memory case close the DB context cuz the request is finished but pull it out of memory case in us in another request and then try and work with it is now detached from its dbcontext sorry there'll be the sort of thing I dip eclis do with my reference data but if I then decided that I needed to update some of that reference data I would just update sorry if I wanted to update it in the database I would need to make the change to it I need to call update yeah they don't need to call save that's right yes yes so in the case you do have a defect stone so they like it's coming from memory cache and the DB context because it was in a different um request that was loaded then you can just that's the right time to use update that will then put it back and then they're like new DB context and you can carry on working from that point thank you no problem so finally safe changes comes in and safe changes the kind of a pretty powerful thing because like I've said before it works about a whole bunch of details about what's related to what and who owns who and which things need to be created which things already exists and things like I can create a new object but that could have a reference to something else that already exists in the database it'll work that out and know that the reference is just a reference to an existing thing and not a reference to another new thing and it kind of does all that work for you so it's mark so any questions before I move on and real basic concepts of entity framework I'll move forward and we'll cover you some basic usage this is the default 101 how do I go about using this so we've covered the engine and how it works a little bit let's talk about how we use it and how we use it it's really quite simple first off I'm going to show you some code this code is on github that's the github URL for it but this code is borrowed heavily if not blatantly stolen from some other code made by Jason Taylor called Northwind trailers the idea of Northwind traders is we took the classic Northwind database which was a sample database made for access and has made in there kind of mid nineties and basically just scaffold it up a whole bunch of code and a whole bunch of initializers so that we can create on the fly a database application hook that lengthy framework or that we can use for sample applications and I'll use this but pretty much all my sample code these days because I can get up known working you know a database that's easy to understand it's got customers it's got products it's got orders and order items all the things that are really easy to explain and it's got to pay the best full of data that you can write queries against so it's really useful so what I've done is I've taken that I packed it around a bit upgraded various bits and tried a few things out here so I did I did a kind of copy while in the fork because I wanted to really highlight certain features immensly framework that don't actually match or belong in a sample application like I'll show you the wrong way to do something and leave that in there so you can highlight that and then show you the right way that has no business being in Jason's perfect sample application so the first thing to do if one a work with entity framework is to find some entities and that's in this case is just a plain old C sharp object I've got a customer and I've got a whole bunch of fields most of them strength and I've got a list of audits that your customers made simple stuff then I have to create a thing called a DB context so a DB context is like I said the main class and in that dbcontext I can figure what type something when I manage and also I can assign some configurations it's my recommended way of doing configurations is to use this command you see at the very bottom there apply configuration from assembly so that will scan and assembly you give it and find all the configuration files you put in there and pull it on in at once because if you don't do that and you've got a big system with you know hundred tables you're going to get a lot of the configuration in this one dbcontext file so once you've set up a DB context you need to configure it against your into the unit's your dotnet core application and the way you can figure a DB context it's pretty much similar to the way you can figure anything in dotnet core because dotnet cool unlike the previous dotnet has a really good built in abstraction for setting up dependency injection so what that means is you've got this function here that's in startup it's all and startup is a kind of new file that's there in every single dotnet called web application and it builds this thing called I service collection and then if I want to add something so any dotnet program I just find that library look on the first praise the library may be a three or four lines down they'll be a say or install this new get package going new get install or dotnet adds and then they'll say add this to your services because what all the libraries do is they make an extension method on top of this our service collection and the extension method based services dot adds my thing so services door at dbcontext or services dot at mediate our services door add fluent validation all these libraries just wire themselves up and what these functions then do is register all the types you need in the dependency injection and just to really quickly explain what I mean by the quite a pendency injection with dependency injection you can figure this thing called a container that gives you a dependencies and your dependents these are the things you need for a piece of code to run like in this case we've got some code that will require entity framework then once you've configured that stuff inside a container like we've done here with these um add commands for they want to use that thing from anywhere in my code you can see the top few lines there are declaring an instance of the DB context and you see that in order to crate a scaffolded customers controller there is a constructor there and that constructor requires a DB context to be fed so that means I cannot create a version of this controller class without having an instance of that and context provided so what-what the dependency injection system does is the framework will make sure that every time I ask the framework to give me a controller it will war up a DB context from configuration that the idea of doing this is an alternative to doing it this way would be I could just inside the class somewhere go new dbcontext and start working with it but then have tightly coupled the code I'm writing to that dependency I need to work with so with dependency injection instead of me saying I want to create one of these myself I'll get the framework to provide what I need and I'll declare what I need hopefully as an interface because that means I can then replace the interface with test versions and do unit testing so one more thing you can do with entity framework is once you've got that set up you can even scaffold controllers this is inside Visual Studio you'd right-click and go add and you click controller and then further down you have this video awesome looking thing called API controller with actions using entity framework looks awesome so you click on that and that then generates you a controller based on your data in your database and based on the mappings you made up but I've got some problems with what you get when you do this it works pretty well for really simple applications but it's got some issues when you start writing bigger and bigger and applications and the problem I have with it is that one at the top of this line so what we're looking at here is a action inside of controller so this is the first piece of code that you write that runs when an incoming request comes in for your application so the point of this is to receive a request from the web probably a JSON request or a web request and then run your code inside this action but if I have my incoming requests here which is the top of my stack and then my database one line down which is the bottom of my stack we're going to put my business logic we're going to put my code that makes decisions or does any work it all has to go in that one little gap that it was sandwiched between those two lines there's mother problems here in what we're doing with this code is taking everything exactly as it is in the database and something it's straight over the wire to the client the chances are you probably don't want to do that you don't want to share every single piece of data in your database with everybody so my recommendation here is as soon as you get so a little bit of complexity you want to start thinking about not sending just the data as it is in the database straight down to the client you want to start thinking about creating view models that are optimized for what the client supposed to see and having data server side that is what the clients not supposed to see there's another problem here and that problem pops up if your database has bi-directional relationships there's nothing wrong with having bi-directional relationships in a database it's a really good idea because you want to start at one point and navigate around and if I happen to know the idea of this thing here I want to go and navigate around the next thing over there and what where my starting point is can change around different requests but if I have bi-directional relationships in my database then if I try and send that customer into the jason am sterilizer the jason serializer will go crazy hopping between those two entities until the memory runs out so once again the recommendation here is to credit view model that doesn't have that bi-directional relationship and then just render what the client needs to have and not everything Christian could you please explain what you mean by bi-directional relationship are you saying cast of orders for example yes sir bi-directional has asked what I mean by bi-directional relationship so bi-directional relationship is when this thing has a pointer coming back to this thing and this thing also that's the point of coming back so coming back to my examples right early on here okay so here's an example of a relationship where the skill has a link back to person yep so I can get from the skill with a person by following that reference but if got a reference to person I can't get back to the skill I'd have to query the whole list of skills and find the ones where that person is linked but in this example it's bi-directional the person knows about the skill and the skill knows what the person and if I was to render that in Jason it would start rendering the person and go okay here's the person here's the idea but the idea cross put the name across Oh quite a new rapper of a skill and they'll start filling in that listed rapper ID name for the skill and then I was another object so quoting for the person but that's the reference back so you can it will basically get stuck in a loop you're saying that's so stupid it's going to say I've got skill number 123 I've got person number four five six and then say oh I've got a reference to both of those I'll put those both in the guinea without to read them one jason by default in the simple mode will be dumb and just hook between them you what you can do is start hacking around with your jason serializer and so let me jason see what i start tracking objects is dealing with and so have the jason serializer start sir and pick those bi-directional relationships but for my money you're much better off building the structure you want to get into the client and setting explicitly you can hack around the adjacency Rison but then you jason serializer it's no longer just dumbly following your records it's having to keep a whole new section of memory keeping track of what i've looked at and what it's ID is and whether it's a Tina or not now it's definitely possible it's quite Jason where you I mean you can create Jason where you put all your objects in one spot and then build your structure with references but that's much more complicated Jason than just having the data arranged out in the structure it's in you solve the problem by saying view bottle earlier yes that's right let's take another look now so we looked at a get request um and what could put its back into context we're talking about and what I don't like about the defaults kappa raid controllers there's nothing wrong with them they work but if you're going to start with building bigger applications Enterprise Server applications that's the problematic though great demo you know this is how it works this easiest way of getting working let's start looking at the next bit so what we're looking at here is a put request so I have something has fired off a web request including data and I want to update a customer record so I'm calling a put request to put that thing up here now I've got this object here we're at a top of type customer and its name is customer and this looks a lot like it's an entity from my database but it isn't what this thing is it's the result of data binding on the web request that came in now don't want to put you off your pizza you've just eaten but that might have come from a JavaScript application it certainly is coming from the web so it's not to be trusted but what here is in this code it looks like it's when our entity from our database but it's actually just been nude up by the framework from whatever junk came in over the wire so that's problematic and then what we're doing in this code is throwing it straight on so that the database and saving it now what were you saying if look you've got plenty of options for wiring invalidation but it's still it's still got problems what if there was some missing fields in that data coming in well it would just set them to no there's another thing in here that I thought was quite amusing they're so afraid of the database TV set to update commands that right here the pretending it doesn't exists and I actually like this line of code because far more explicit about what's going on it's saying take this object put it in a DB context and marketers modified that's what the update command does but this is explicitly saying that rather than just calling update which looks like you kind of call out update commands but it's not what it's doing so I like that line of code but again it's a good thing to point out and it's the key thing here I don't like taking this object and putting it straight into the DB context so but a bit of a look about what I don't like so what do I like let's take a step back and take a look at clean architecture I'm going to bring up my classic slide now I use this side in every talk to justify absolutely yeah nearly every because for me the point of architecture is to avoid what's called spaghetti code so in spaghetti code the relations between the pieces of code are so tangled that is nearly impossible to add or change something without unpredicted breaking something somewhere else it certainly sounds familiar to me anyone seen this kind of problem quick show of hands and I see some nods yeah everybody that's right no one sits out to make spaghetti code everyone says to themselves right this is a nice new project I'm going to keep it clean this time everything's set up just right and everything pop roughly and then requirements change don't they and now because requirements have changed halfway through the project you've got far less time to build it and so you'll make a shortcut or you'll plug something wait doesn't do and then you've got to just make it work and suddenly you've got all this stuff that's in interconnected and you end up this spaghetti code system it naturally occurs so for me the secret sauce of spaghetti code is dependencies you get spaghetti code when you have code in one place that's used by two different places and you change it once to make that thing work but then you're broken that thing over there and so for really the best way of managing spaghetti code is to carefully manage the dependencies and that's where clean architecture comes in because clean architecture is all about managing those dependencies so clean architecture is known as a domain centric architecture because in the middle you put this thing called the domain and so the domain is all about modeling the problems you're going to solve in your application so in the case of Northwinds we've got objects like customers we've got objects like orders we've got objects like order items and the relationships between those objects are true regardless of what you're doing with those objects so a custom order a customer can place orders regardless of whether or not I'm using a sequel database a story or whether I'm just running it to a file somewhere that truths about customers and orders is universal regardless of the technology I'm going to use to save and get it back later so the rule of thumb here is we make the domain as complicated as we can with as much business as much as we like as long as we don't take on dependencies to any external libraries or applications so model that the vault data as much as you can and the good thing right entity framework cause it's getting better at better at dealing with more complex domain objects and I'll cover some of those features later then around the domain we have this layer called the application so you can vary this quite a bit I've seen some some people do clean architecture where they just put applications the main all together and that's you know for a smaller project that's fine but the idea of application for me is really about the operations you can do what are we actually do with this data so it's the interface to actually go and create or cut an order for our customer so you see this gray arrow here this gray arrow is the dependency so the application layer knows about the domain and knows we're dealing with customers the main doesn't know about the application layer it's just modeling that core data then on the outside and always bring these up together infrastructure so infrastructure are services we need to actually then do this work so for example I might need to send an email I don't need to know that I'm going to send an email later in order to model that our customer has orders but at some point in the application I'm gonna want to send an email saying yep you've placed your order so that do that I neither have an email sending service and the idea is that I put an interface about the email sending service in the application but the actual infrastructure that sends an email is then an implementation of the interface so the application doesn't know that details about how that sent so I might be sending a message via send word one week and I might decide to switch to SMS another week if I make the interface generic enough I can swap these things out and so with infrastructure we also have persistence so like I said before the fact that's going to a database always go into an XML file shouldn't affect the application too much but if we can make our persistence at a separate process and with that dependency flowing in words then we've decoupled or application a bit more we've got things that we can test in isolation away from all the infrastructure that helps them to work and then finally I thought we've got this thing with a presentation or layer and in other demos I'm showing today the presentation alone is just the network traffic layer it's taking requests in from the web from a JavaScript application somewhere and just pushing them in to the application so to recap on the clean architecture it's all about managing your dependencies got a question could there just as readily be a windows forms application it doesn't have to be a web application that morale would work just yes yes that's a very good point and this design is a pure design that applies to any application like you could take this on for Windows application you could take this on for anything you want to build some of the details might change obviously but you could apply this to any language and any application you want to build it's just a way of laying out or you could choose to choose different patterns but this is a great pattern so can I work with and build an application say yes so we start off with managing your dependencies it's also highly testable and the whole point of managing these dependencies is we can look at bits of the domain of the application layer in isolation and run unit tests with them disconnect them from the database and make sure they still work and make fast tests that work that way it's also independent of the user interface independent of the database and independent of anything external if you think you've seen this stuff before it looks a bit like onion architecture it's pretty much the same thing the new badge okay I consider onion architecture and clean architecture certainly at the higher level under I'm showing you here to be pretty much the same thing so when we talk about this type of architecture and we talk about the persistence layer we've put a start discussing what interface do we use between these levels what interface we put between the application and the persistence layer and for a long time across the history of onion architecture and for a lot of people still that an has had one answer and the answer has been something called the repository pattern so the repository pattern mediates between the domain and data mapping layers acting like an in-memory the main object collection now the repository pattern is one of the most widely used and widely understood patterns going quick show of hands who's at some point rather I've written the repository pattern few of you who's written a repository pattern and they're really closely stuck to this it's a domain it acts like a collection in memory who's done it but then found it started to grow over time having more and more features until over time it became the de facto application layer and all of the all the business logic in that whole application was there inside the repository it especially just map between your business logic and your database but actually became your business logic certainly happened to me so I'm not trying to add comprehensive enoch the repository pattern it is a good pattern I have used it well and I've actually written libraries based on the repository pattern but here's a question here do we still need to import these patterns when we're dealing with entity framework well certainly installation density framework or insulates your code to a point from database changes and as the phone looks six you couldn't answer this you couldn't claim this for a while because the only good database provider for instance box six was SQL Server so you could tie yourself down to SQL Server which you know probably good things to do know whenever actually changes their database I've done it I think in one project in my life so I just take that dependency and work with it but now with AF core you've got much better database providers and Postgres which is my favorite database is the top level and provider by default is you know it's kept up there side by side it is fantastic it works really well and actually one more thing I haven't got time to show you today bring those son that son be code I made I actually made a version of Northwinds where I created to persistence layers so that I could use either Microsoft SQL Server or Postgres to actually manage both them so I had one DB context and one D becomes its configuration but created two sets of migration changes the schema changes were specifically for the d-didn't databases but the actual configuration the layer up was the same and I could use both databases and switch them at runtime so check out the code base to see that and so the DB context acts as a unit of work so the idea of the unit of work is when you're dealing with repositories you're dealing with something it looks like a collection you see add stuff to the collection and remove stuff to the collection but we're dealing with databases they have the concept of a transaction so if you just talk into repositories to act like a collection I could talk to the customer collection I can talk to the order collection but there's nothing naturally there that wraps them up into a 1:1 package so here's a hidden event where I've created a customer and then there are new customer with a new order and let's wrap that all up in the one transaction so unit of work is that that conceptual pattern there some map changes across multiple repositories into one unit which then maps to a database transaction behind the scenes well the dbcontext acts like that and the DB set acts like a repository it's got all those actions you expect to see on a repository layer it's got add update those kind of things and finally AF core has features for unit testing without repository so if unit testing is your main reason for using repositories then unit testing fills that in as well now let's just say again I'm not saying don't use repositories what I'm gonna say is you've got a choice you don't have to and do what works best for you so we've talked about the repository pattern so what should we do instead and what I'm going to talk to you now is but a slightly different pattern and I'm gonna advocate for putting that pattern not this layer here in the arrow between persistence and application but actually take some concepts a little bit higher up and do a little bit more work in our application layer first before we ever talk about our database and the bit of work on when I put inside deprecation layer it's called CQRS you have a question you pass across the thing so as for the previous slide just wondering too much about Postgres so let's say sequel server doesn't wrap the whole unit of work in in a transaction and also what sort of isolation level doesn't run under default yes a good questions that I'll repeat the question there so the unit of work and does support transactions about a fault a Save Changes is a transaction it has the default isolation level with that transaction which I believe is read committed yeah I think it's read committed but you also have support in NC framework for adding a transaction scope around it and setting whatever its transaction scope you want so if you want to do work across multiple instances of a DB context for example and I have done this before when I've got a lot of processing to do maybe I'm you know processing a big batch of records because the DB contexts can fill up one good way of you know processing is you know do a thousand records at a time so I've dropped the context bring it up you want all that inside a transaction then you can put a transaction scope around it and that just uses that and context based transaction scope thing now I know that context based transaction scope thing is very hard to get working with async code because it does use things like threads local context and it's it you have to have code that's able to refer back to a magic context somewhere so in those that context is there and I know I think I can mess with that a little bit little question yeah okay grab that mark please sorry so the solution that I've implemented for that is to create some middleware on the request pipeline that has the dbcontext injected and you startin in the transaction on the request yep yeah that's a good way yep and so you can certainly yeah we can certainly wrapper requests you can wrap a requests in a transaction at the beginning and ends or you can you know create your own scopes one thing with dependency injection is you have this concept of a scoped object so if you're working outside of requests or you want to actually make your own scope inside a request just go and do so quite a new scope and then anything inside that scope will have that life cycle and get disposed of and finished off where that one scope has finished so you've got some very good api's for dealing with that now that were just weren't there by default you tagged them if it took on certain dependencies like if you're wide in order fact if you are in unity which was the Microsoft one but um now you've got a really good default implementation that gives you these concepts and you're also still free to pull in things I call their FAQ and get a much more richer and configuration system and that works transparently with the current default api's and adding all effec on top so you've got a lot more options there going back and forward so let's talk about CQRS and i'm along with our advocate here is taking a step back from their database layer and start putting CQRS as a concept inside our application layer so what do I mean by CQRS well CQRS stands for commands query responsibility segregation so the idea is that you split off your commands which your database writes from your reads and there's some good reasons for doing that because lots of applications are quite read heavy and there are different things you want from you reads versus writes in most cases you want your reads to be as fast as possible and you want your rights to be it's correct as possible because if you've written the wrong data it's a catastrophe but your reads you just want to make sure they're fast and so the number of ways of achieving this but we're gonna start with the simplest possible way and just by having one database but acting differently between doing queries against the data person doing writes against the database it's like I say you're separate reads from writes and you also take on this thing called the single responsibility principle so for me they require to almost competing concepts here don't repeat yourself means that you want to write something once and reuse it as much as possible because don't reinvent the wheel with a single responsibility principle is more about managing that dependency sometimes you're better off writing two different versions of the thing because then changing this one won't break that code over there so the single responsibility principle is what is all about making something contains that does a very well-defined job and reuse it as many times as that well defy job but what you should never do is start tweaking that well-defined job to act in multiple use cases because that's when you get the dependency hill of stuff that breaks and having knock-on effects and side-effects so it's CQRS it's easy to add new features just add a new query or commands and it's easy to maintain because your changes only affects one command or query and CQRS pattern can be used with repositories this is not replacing it this is something else to do a step above and we will look at that in more detail bit later so CQRS can be quite a loaded term it can mean quality if an umber of different things for different people there are a bunch of different ways of doing seeker s and for some people they quite sequestered event sourcing no event sourcing is a kind of next-level way of doing your data in your application by all means use it if you need to we don't have to well it bent sourcing is all about is it's really useful for really really high throughput work so and with event sourcing instead of keeping the database in its natural third normal form state and when work comes in at the end of any piece of work we end up with the database once again in its pure third normal form State and we do own the transaction and we make sure we do everything one at a time instead with the event sourcing instead of modifying the database as things happen what we do is we take a record of the event that happens then we think of an example and that's once a year they sell Splenda in the grass tickets now what used to happen is those tickets go on sale and 100,000 people would log on that instant and try and buy a ticket and when that was plugged into transactional database that was starts turning hot then start smoking and it was fall over and then would buy any tickets and it was always an embarrassment but when you're talking so let's have a think about the dates you need to know about selling tickets so traditional transactional system would take someone's ordering right that's the database update three or four tables here's the order table here's the tickets table there's a ticket in the there's Ono's and dad although items and so there's huge contention on the order items table because it being written to all the time but someone that wants to buy a ticket doesn't need to know about all those records in the order items table all we need to notice that a ticket is one integer that's how many tickets are left so instead of waiting for all this stuff to come in and then applying it all to the database and doing it really slow leaks is rotting data instead what you do is you have a single datastore that's called the event store and that event store is there things that happens I have sold a ticket bang goes in I have sold a ticket bang goes in and then as those tickets the price is the first thing you do is just look at that ticket look at how many tickets were on it or head out that order and you subtract one from that integer so that integer is what we call a read model it's not a permanent store for that data it's just a number that represents its exist is just for reading because the next one coming in does need to query the database and query how many have been solved it needs to look at one number in memory and so it's how many tickets are left so then later on and you've got all the time in the world you can process those orders and so your price that's those orders then push them into the database but I'll be able to sell these tickets because I've got this backlog of all of the process and that's where the idea eventual consistency comes in so instead of having the database transaction come in and we apply it to the database and wait for that to resolve itself we take the events we store the events and then in our own time we process those and finally make the database consistent with what we need it in the end so eventual consistency the what might happen in that situation is you might accidentally sell five too many tickets so what you're doing in that situation is have a business process because we've just sold 100,000 tickets so we can afford to have one person sat there on the phone and phone those three people up and say sorry but what you can't afford to do is have the whole thing fall over in some no tickets so it's a situation where being 100% atomically correct isn't serving the business any purpose at all where are being able to take in a huge load in one go is much better that's the or event sourcing and the key idea of event sourcing is this store of events and this read waddle in the middle so that's something that's available to you if you start going down the CQRS route but that's not what CQRS is CQRS can be much simpler than that but Seaquest allows you to take that thing on the go and it also doesn't necessarily mean separate data stores for region mites what it does mean is clearly separate commands and query operations and separating your processes for reads and writes you have a question yeah just a very simple one I think what you've left out of that and I agree with everything you've seen by the way it's a really sensible model yeah I is what you do if your server crashes you've got to every start proces and that means that you've you've got to look through that entire event table to work out where that number is we do that's right so you you restart is still going to be relatively slow but it still got to be blindingly faster that's right yeah you're faster but there are a whole bunch of when you're talking that kind of you know disaster recovery in a event sourcing model there's a bunch of a processes maybe you don't want to spend too long with a backlog events but actually do process those events you can at any time crate snapshots of the data store at that point in time and once you've created that snapshot you no longer keep that whole backlog the events you can throw those away unless maybe sometimes what you can do is do a wicked undo process but your roll back to the perceptual point in time prior events but choose to take them out because they turned out to be invalid and then you can actually roll back time against your database so there's some really powerful things become available you see from event sourcing this isn't a talk about event sourcing this is a talk about entity framework but event sourcing always comes up as being related CQRS so my idea is is start using CQRS because it will unlock the possible capability of doing this stuff in the future adding repositories doesn't do that let's take a step forward and start looking at how do we do CQRS in entity framework and here's a great library we've decided to use called mediator so with mediator we can define our commands and queries as mediator requests the application layer just turns through a series of requests and response objects and we also get the ability with mediator to attach additional behavior around their event hand handling so let's take a look at what that looks like here is an example of a request is called finds customers by name first thing to point out about it it's pretty clear what it's for I've given it a name yet the name is finds customers by name and it's got one parameter that name and because I've made that parameter a constructor parameter as well I can't quite an instance of find customers by name without giving it a name string so what it is it's incredibly clear how to build this request then it also tells me what it returns if it turns this thing called a customer's ListView model and that's a view model which is listing some details about some customers so the idea here I've created this thing this really simple class that clearly defines my input what I put into it and my output then I create a handle a class so the hands that class is going to do the work of fetching that data for me and I'll cover some of the details around here later but you see we inject in an interface in front of our Northwind dbcontext and we have our context code here that's going to get from our database and project it into our view model and I'll cover who I'm setting rejecting later so here is a piece of code that handles that so once again we have a clear one thing the definition of it this is the name of it here's what it returns and here's how I build one here is the piece of code that can handle that being sent in then finally this is what my controller looks like once I've taken this on because now I no longer have and pretty much anything of anything inside my controller my controller is concerned with taking the request that comes in from the wire so I've got a request coming in that says fine by name and gives me a name so that will be a URL that triggers that and then all that do is build this foreign customers by name request object and send it to mediate up so I have completely decoupled how that works from the controller and the controller does no work other than map this incoming request to a handler so what's great about this is any business logic that's handled that's done by the handler behind this thing I can call that from anything that's capable of building this object so basically any piece of code that has a string that I think could be a name I can build a fun customers by name query since it's a mediator and that will run it so I could have it in a startup class I could have it in a console application it doesn't have to be a web request that triggers this piece of code so made as repeatable unit of work the other thing that's really clear is the input coming in is this objects and the output coming out is this object there's no confusion where I'm feeling an object in and the hands that's gonna modify the object I gave to it I'm gonna have an input Bob jek coming in I don't expect that to get modified but I'll get a result output coming out and there's no side effects if I want to know what's happened I just look at the response of that method I don't have to look at anything else so it's really clear what I sense in and what I came out and if I've got an operation that changes 50 things that'll expect the result of the operation to return me there 50 things that were changed so with that let's come back to here because what I've got here is more interface to my data layer and once again it's a dependency injected thing so the question is what interface should be used here and we were talking about am repository pattern I spent a bit of time thinking about what is the simplest interface I could use to just go ahead and use entity framework but still make this an interface dependency that I could override if I wanted to and this is the interface I came up with so rather than explicitly listing out all the different types in my application I've got that in configuration anyway I've just exposed this DB set generic method system context set off type customer will give me the TV set for customers and context set up top order or give me the DB context for orders so this really simplifies the interface here and the drawback of this approach is it's far less discoverable so from using our context where I've got the things listed out then it's very discoverable to go context door and a bunch list of things in intellisense was that a question you want to grab that you may answer this a little bit later on but how do you go about I will cover that later on yes yep and so yeah one thing to point out here is I am still taking a bit of a dependency on entity framework certainly I'm taking a dependency on DB 6 which comes from Wednesday framework but I've done enough that if I needed to override something I could create a subclass of DB say and get that to work differently intercept the commands I wanted throw not input an exception but the commands are down and you know if it really needs a thing it's not clean but it's a bit of separation but certainly it's a lot easier than running all those repository courses and putting a an instruction between myself and the data and the database the benefit you get from doing this is in sleep omics improving there's been a new change recently quote I think innumerable your cover later but this can do is Iike noodles as well as our query balls so you get more for your more for your money than if I just don't use the repository that I've written five years ago and the same apology pattern then I wouldn't be getting those benefits flow through so let's take a look at see QRS and EF core and start looking at some read operations so in read operations we want to get from the database and get data straight into it view model as quickly as possible and don't track the changes we've made to it let's take a look at how we can go about doing that here is a query object that made starting more complicated this has a name search field and it has some paging fields a page index and a page size and it's going to turn a view model so let's take a look at the handout to work with that I'm going to start by going context upset that's fetching me the UM DB set class and a call where true and all that does is turn this into an interesting form of query iqueryable and it's going to contain all the records but the reason i put where true here is i have started to turn this into my queryable and the key thing with my queryable is it doesn't execute into the database until it needs to so now I've created there's our quibble objects and oh yeah and the things that will make it execute are things like two lists we're going to make a list of things so it's going to fix that from a database or counts because we'll account the number of Arkham Records or first or default I'm gonna load the first records but until it has to I can build up this object incrementally and add features add filters to this object so I'll start with a simple iqueryable and then I can do things like applied filters and apply paging so for example apply filters down the bottom here I can just if my query object contained a name it's going to apply that name filter where the name contains the name I said and if I had some paging buddy is set in I can use skip and take so over time I'm building up at more and more complex query to send into the database so that's built a query what can I do with that query well here's a very important Depp took to understand this dot select on the end of the arc we were able here fundamentally changes what goes on with that query because if I was to run that query like that without a select it would go and load those objects as they are and put them in the init in the DB context then what I can do is add a dot as no tracking onto that query and it will tell DB cut the entity framework not the track changes those entities so I'll turn the tracking system off so that's pretty good but if I do dot select on that I am no longer going to fetch complete entities I'm going to fetch specifically the fields that I've asked for in this land of expression so this block here it's called a lambda expression it's this thing with a fat arrow in front of it so dot selects see that's a reference the thing I'm selecting I'm selecting on customers and then I'm going to start projecting that into our view model and then this maps values from the database into the view model and because I've got select name customer ID and orders I'm going to fetch the gesture fields I asked for rather than the whole thing so I'm performing a much faster read operation there's one slight problem we're dealing with this though because how this works is this chunk of code here in yellow is a lambda expression that gets compiled into a link expression which gets sent to entity framework and compiled into a sequel statement the problem is the c-sharp code I can put in this yellow block as that much wider and reach a broader reach than the c-sharp code that can successfully compiled into sequel statements so the danger is that I'll write code in here density framework can't cope with what used to happen with earlier versions of entity framework is if entity framework couldn't deal with it it would give up load the whole table into memory and then do it all in memory now that's pretty bad but with EF core 3 now it will no longer do that if it can't do it it will warn you and won't just automatically fall backs are doing it in memory and being slow the problem with those sort of problems is it works fine when you've got five test records it's not fine at all when you've got a million records in production so but what can I do if I want to perform an operation in this mapping this projection that can't be handled by NC framework well here's what not what to do I've seen someone do this you go early on in the pattern you just type to list because that fixes it but what I've done there is I've loaded every single customer and every single order and every single early employee record that was linked to one of those orders and I've loaded them all into memory and then I start iterating through and build this small data set that I want now you can sometimes I'm not advocating for this you can sometimes get away with it if you've been really good at plying paging and filtering first you're filtering it down to five records at a time then you'll get away with it it's probably fine if this is an admin screen that two people are using then yeah just just do it because you've got more important things to spend your time on but it certainly isn't the right way to do it so be aware that's what you're doing when you use to list and so you want to list whatever the query is in there will be the one that executes so a better parent is what I call select anonymous so what you do is your first path through a select and you take fields and other database pretty much exactly as they are on the database you don't modify them then once you've got them as they were in the database you can then load them and then pass them through to a projection that actually does more work and more modifications so you can convert date times or you know work with you know perform operations that can't be done by the database what I'm finding is more and more stuff works in the database within the framework or used to be that if you want to do a time conversion from UTC time to local time that wouldn't run in NC fine work but now it does there's more and more stuff is actually being picked up and run skinny it's getting better but there'll always be stuff that doesn't work and yeah this is a very simple example stuff can blow out really quickly if a question those things mostly implemented on both sequel server and the Postgres provider or do you find there's the I really like the post quest with water I haven't done enough real word development against it post quest for my favorite database but most of my clients are Microsoft shop so I am you know I can speak for how they work on the Microsoft pan I don't know how much difference there is between the two I found it's been pretty good in my basic tests of putting down inputting data real but when you start dealing with those real life queries I haven't got a huge my experience with entity framework and Postgres a question why did the first example why was a first example not able to run to be run by the indifferent work these actually these codes you've got here are able to run it's if I had some feature in here that can't compile across okay yes this example you're saying here this does work yes yeah but it's easy to put code in here that doesn't you just you know make it too complicated I'll put in some operation any code that can't be run inside the database then that that will fail so the end ephemeral may not have been able to build the necessary query yes it might not be able to build that query if I'm certainly if I'm putting a reference to a library there isn't reference very fine work like I've you know I've made by an extension method so to work on the strings then it won't be able to do it can you test those things using like link pad or some querying tool that way of testing it and I'll come to testing writer I'm just a question around what you just said about capabilities that can be done in the database just following up on your example about date/time conversions yeah so entity framework supposedly should work again you know using your example you can change out your database provider an entity framework will work with sequel Postgres yeah Oracle whatever right yep so in theory it should provide the same capabilities across all those so let's say there is a good are those capabilities that are exposed an entity framework only capabilities that are supported across all database providers or is smart enough to offload some of those functions to itself if it can't be done there's really good question and I have to look into that in more detail my feeling is the idea is that here is the set of things we support and then we test a provider for that but the providers are open source so the ends the final map team can't go well that's not a provider I'm gonna you know they have a verification process for example I know the Postgres team of whom and the post quest NGP SQL Bovada I've been doing a lot of work and so that's definitely the first one to do I don't know much about the state of any other ones but um I can't there must there will be bugs where it works in one place and not others there's no way it can't and you know the way to find those is to have more users using them but I think from my experience of you know doing basic accreditation the Postgres trial has been pretty awesome I haven't tried it in anger with some complex queries so we quickly covered the unselect then projects another thing to show you and a good way of getting data out is this is a commands from auto mapper called projector - so what this basically does is pretty much the same thing also acts as it is from the database and then automatically use automatic configuration to then map up to the requested view model now automata is fantastic mapping between things are very similar in shape you have to decide whether it's easier to debug code you've written to copy values across or debug 1mf mappings because you might find auto map is great for 99% of your application but the cases where you have to make that configuration for it is it easier to debug all them up for configurations or is it easier to debug just I've written the code and write on the map and myself and that's up for you to decide and certainly I've seen people argue against the autumn open because it hides code that you can just write yourself but it can save you running a lot of code so you've got a way up those two two considerations one final thing we've got here and it's gherkin excape Palamon it's this thing called query type so what's called query types in AF two point one and two two and there's now been renamed to has no key it's the idea in both of these things and we'll cover why it has no key the idea both these things is to create read only models to work with so it was called query types you could have context you know that DB context not set to deal with data you could write to if they added the contact of dbcontext our query we just read any queries you can't change the data then they made the decision that well actually there's not much special about these queries they're just entities that don't have a primary key and therefore can't be mapped we'll call them has no key so here's an example of a slightly more complicated Examiners most purchased products there's a comma more complex query but this table this object sorry is the output of a queer oh and some customer details and the product details and how many of that product they took then what I can do with that is the very first thing at the top there is there has no key configuration lines that says this is an entity that has no primary key it'll load it in a read-only mode and then I can say to query and put either a view name in or a full sequel statement in there so here's an example of the sort of sequel statement where you can write this in linked but I would say once you get to this kind of this kind of spot you just better off writing it in sequel because you can then you'd open up sequel tools you can use your query Explorers and in this case I've got a few inner joins a group by there was a problem with group boy and as they find milk or one and that if it had any group boy in there at all it would do it all in memory so it didn't have proper support for group is all that's been fixed and they've made huge strides to not do stuff in memory anymore but at some point you need to be able to jump out and just run a query and this is your mechanism for doing so because you can then map up a query directly to and instantly to receive the results that query now before this feature came along or what I said using dapper to do that piece of work because I'm dapper pretty much does just that and dapper doesn't have any of that complex ends the change tracking stuff it's just here's a query map the results this objects and a way it go it's really super fast but this kind of just does what depe does with one exception with one once you've built that thing I can treat it like any other context and then add order by skip and takes onto that statement because it's sequel you can take the input of one single statement or one single statement and then push it into another sequel statement and so it lets you do this so this is the well but it has a new sequel around the sequel where the original statements sequel in the middle yeah yeah so pretty powerful stuff so I can apply paging and filtering and the order buying on that complex query so this is this is your escape plan basically to get out so they come back to CKS and the F core we've spent a long time looking at the reads but to make CQRS work well where you've call for those read operations you want to select and project into view models you can do it with a select statement and use link where you can when it gets too complicated you can use in those query models that the idea is you do it in a way we don't follow up the intensity tracking system you'd avoid the change tracking and just get that data out as quickly as possible then for your write operations they're the bits where you but I have some complex business logic there the you want to be correct so in those situations loadcomplete entities each time because you might have business object there's checking things against those entities and build a unit of work and have all those references properly checked by the entity framework system so you've got our fast read and a slower right so in summary one recommendation is always map those incoming requests to commands and crews the application layer and then you've got some choices what to put here you can use entity framework interfaces or interface based entity framework and just get on with your life and write some code you can create a repository layer and achieved true persistence ignorance that's not a bad thing to do like there's good reasons for doing so so I'm not saying don't do that but do what you think you need to do or you can create another layer of come and queries you can keep that separation of concerns all the way through the stack if that's what's appropriate to you and one more thing when I show you in this section and this is John P Smith he's written a good book on in Stefan mucke core it's also a good blogger and he's came up with a great example which to me is proving why Seacrest is the answer to go forward he's written a sample this is the picture from his blog from that sample where he's used CQRS it's got commands and queries and what he's done is he's using entity framework and for his commands he is writing voron sea foam at commands into a sequel database then when that right into that sequel database occurs as a second process kicks off that projects into a view model and this view model this read model sorry goes into cosmos DB so the idea of the read model is it's potentially throwaway it's not the the eventual storyboard data it's there to provide a very fast way of providing certain queries so say for example in your application everyone's hitting the front page and they're getting a list of products by category time and time again products black alligator products by category against the sequel database that'll be a joint across a couple of tables be it take the results of that query denormalize it doesn't matter if there's double ups in there get it in the fastest read model state put it in that no sequel database I'll put it in 20 no sequel databases I have a massive farm of them so that query service fast and then the right salmon farm well how often do you change two product definitions like you know they'll be three or four product changes a week because someone's made some content so fine they can go in it doesn't matter if it takes 30 seconds for that to push through so all your read models but your read models are fast and your SQL is correct so the key point to make here is if you want to use repository pattern great put it there that's where it belongs it's it's there between the domain model and your own sequel statement but it's putting commands and queries secrets here that allows that next level stuff of separate read models those extra steps to optimize allows those to happen and you don't have to take this all on an all-or-nothing approach we're going to move our entire location to event sourcing and spend the next two months building a framework for it what you can do is you start with what is the biggest pain point oh it's that front page make a read model for that front page and get that working first so it doesn't have to be the entire thing just make something better with a slightly different architecture the rest of my stuffs can be much quicker so I'll blast through it so the next thing is testing so we want to write some tests against the application layer we want to write unit tests I'm going to show you X unit and the EF core in-memory provider so if you're using repositories or general testing in general it's a good idea to write mock versions of all this infrastructure and resistance things so then the unit test can just test their functionality and just receive the data they need for that test to work but EF core memory provider for me is a provider for dotnet core that's designed for testing it's not for one time it's there for running unit tests against and for me it's not for testing persistence it's not for testing I've saved this thing and then I'm really back and that there it is it's it's working well it's for is to give you a quick powerful mock provider to test the logic inside your application and your commands and query handlers so it's not for testing the database right and integration test tested that actually needs to have at least a few but for testing your code AF core memory provider gives you a much more powerful faster mock provider than boiling all this stuff by hands it's fast because it's in memory and it's also allows you to isolate your tests because it's in memory and it's fast you can cry basically a new database of memory per tests you don't get any side effects of if I run my tests out of order my test will fail because of time insert before the read that checks it so let's take a look how we do that well we create a DB context using our DB context options builder and then what would have said use sequel server or we would have said use ng psql we say using memory database and we give our in-memory data name if we generate a good for that name then every time we run this to generate database it'll be a unique name if we gave it a fixed name it would be the same instance for the last time of the application then we just knew what that database we put some data in it and bang it's good to go okay then I don't have a command that wants to test that we're querying customers or query their filter on them I can run a test that does that and if I say conducts don't get customers I will get that test data for customers so very easy to write tests against it and that's basically it for am testing it's really quite quite easy to go forward but won't go a bit further now is EF called performance because EF core performance has had a bit of a bad rap over the years because of how easy it is to load everything into memory a problem with EF core performance is generated code is almost never faster than right and single boy hands unless you've really bad it right and sequel there yeah but you use EF core so make that boring cried there's quite update delete operations quicker and easier to write you don't want to spend weeks setting up your architecture and you know I like going into places and talking to people working out the domain drawing lines and boxes on the board here's this is a one-to-many that's and minutes of money you can very quickly from that board of one-to-many and many-to-many x' drop in some classes hit go scaffold up your donor base and have a working working prototype so that's what NC framework is really great for so when you're talking performance you should never talk about performance without a reference or something you need to talk about what is this system for and where does it need to be performance how many people are going to be using it what are they gonna be doing because you could have a really slow system if it doesn't have enough users anyway it's fine like there's no point spending developer time which is money to optimize where optimizations not needed and so you start performance by measuring your performance against the metric of what you want to achieve then you can focus your optimization work for the best business value you might be able to take five seconds of an admin screen but that admin screen has got three users you might better spend the same amount of time taking half a second off the homepage that's used by million users taking that half seconds of the homepage is going to have a higher business value than taking five seconds off that admin page so focus that that work where it's needed and then you also need to do integration tests to affect the actual and predicted load and be ready to bypass EF court when you need to here's the first super easy win um to talk about mediator is a certain example of a piece of code that you can inject into mediator to wrap every single media at a request so basically you wire this up to mediator and you'll pick this up and it'll go time roulette start time let's stop you'll run your request and if your request took more than five hundred milliseconds it'll make a little warning in your logs so straight away this takes no time to do just cut and paste this one it's still the one from my sample code there put it in your application you've got a measured metric go on use your application for a week come back and look at all the warnings and work out where it's being slow start looking at those and work out which ones are important straightaway you've started to collect even a data-driven approach to your work another thing to looking at is EF current support for our logo because EF court will log quite a lot if you turn the log levels up obviously if you turn the log levels up way high it's going to slow everything down and your production will be slow but you can take some options maybe come up with a smart way of just logging certain a certain percentage of requests but certainly it's easy to get logs a trendy framework so you know turn it right up on testing and take a look at what's going on um integrates well with things like seer logs seek an application insights so good for generating logs good for collecting those logs than never getting through those logs it's one thing that's good to do is measure the number of sequel statements executed per command or request if you're firing a piece of linker entity framework and you think it's doing one thing and you're seeing five sequel statements being executing at the server you know there's something wrong another thing you can do is from the NC farmer paper itself you can actually tag your query with a name then when you look for those that look through the logs you can find those names and actually map up okay this particular query was fired by you know I've got a whole bunch of stuff going on and it's only one operation here something that's going on so I've talked a bit about performance timebomb SQL can appear fine when developing it's an empty database they've got five test records everything's great is doing table scans for everything but I can't see because it's on my local machine and it's working but these problems can surface in production and you can have lots of upset users now having lots of upset users is better than having no users like if you've got lots of users then you know if you've got lots of users and it's slow because you've got too many users there's still a good problem to have for wanted that paying users of course if you've got a problem with loads of unpaid users you've got a problem that tech can't fix but in generally you know those problem can service in production it's harder to fix it once it's in production once you've got a whole user base so you can't red green refactor performance without a failing performance test so learn to generate a failing performance test you need to create a test that performs a similar sort of load you need a infrastructure for testing that looks a lot like production and your size of data set for testing that's as big as production or as big as your intended production so how can we do that well that's where this why we called bogus comes in BO this is a great library for generating huge amounts of human friendly test data that's procedurally generated and then you can use a seed value to make that data set repeatable so what I mean by that is I can using a seed value on my generation I can generate a whole bunch of test data on a server somewhere generate that same test out on a client and then use that clients a hammer that test system so I've done this to generate you know a million login records on the server then generate a client put that client in containers and spin up 20 containers all hitting the crap out of that server and seeing how many I can send through so how do we go about generating some test data so the first thing we do is we feed it a randomize seed so that's the thing that says every time I create a random value using this faker setup it will generate the same thing for the same input and for the same configuration on which the configuration changes then you'll generate different values but if I have the same configuration and the same seed it'll procedurally generate the same big data set then we come in here we've got some rules so we're creating our customer record so for every record when I create a custard test for we create a rule for rule this map this matches up to the company name and then we've got this if faker object here then against the faker objects is a whole bunch of objects that lets us generate stuff so f-duct company lets us generate things by companies there's a company name you'll make up a company name there's a company address they've got a field called company bs not quite sure what that's for I think it's for the mission statement and against names you can generate names initials full names and their dresses you can generate you know partial addresses and it's also got a locality setting so I had a German client so I said it's a German and it generated German names and German addresses and they look you know like German addresses in fact the data looks a little bit like this here is some here is the output of that bright piece of code you see in math approves every single field but the top two lines there in that am table are from Northwind and then the everything from 78 down is generate data now what this is great for if you're just generating junk into your database it's really hard when you find a bug go oh there's something probably wrong with user X Y Zed 5 7 but if there's something wrong with Boyd Harbor you can say hey there something wrong with Boyd Harbor across the room that person will be in their search and find them and you can start digging into the data so it's really easy to work with and you can generate vast amounts of it so here we are using that to generate a test and in this case we've now got an integration test context because we don't want to run this when we check in our code because it it won't run on our CI CD pipeline but we generate a rear SQL server then we initialize that migrate it and put in all the test data we can then create a whole bunch of fake users and then throw them in a database even though it's right about here I've got a commented outline this bulkinsert operation is available through a library that gives you a kind of bulk inserts penalty phonemic call that works great in - - but the version I was using didn't work for 4-3 just because hasn't been upgraded for version 3 yet so I'm doing all my samples in the latest version 3 this slobber didn't work but I expect that library to work again very soon when the upgrade new version and I mean that was fairly quick customers to add range but bulkinsert bypasses inserts and does you know SQL bulk copying and it's hugely fast like it's there's the bypasses the Holika busy that only works with SQL Server because it's using SQL sir bulk copy but I'm sure you could do with something similar if you really needed to with whatever bulk import and systems you got in other databases got a question okay yes the question there was using the bulk inserts and when you've got a more complex some data model and basically bulk inserts bypasses all referential integrity that's what makes it fast so the questioner how will it handle it badly but not if you have insert in the right order if you create the right related data set and then insert the table with no dependencies first and then the tables around it in the right order you can very quickly bulk insert but you need to understand the relationships between them between the tables you're inserting into and I mean you see this when you do restores as well you'll turn all the all um you've got commands you can do in a database to turn all referential integrity off do a whole bunch of insert then try and turn it on and see if it it see if it comes up yeah but yes there that doesn't have any magic yet the bulkinsert does not have any money to sort their stuff out ad range will sort stuff out for you and that's why it's a lot slower bulkinsert doesn't have the checking but it's fast you either have correct or fast there's yeah it's a trade-off the other thing we're Balkans is it switches off database locking while it's duets oh yeah really yes it is is the speed it takes to take them thing and put it straight into the disk basically so for summary on performance one of the biggest problems one of the biggest causes of bad performance I've seen in the world isn't to do with entity framework isn't to do with anything other than a developer's been working on library over here and develop put in the next layer is kind of using it and they haven't spoken to each other and they don't realize that the thing they're calling on the layer down is loading everything into memory and it's breaking it or some really simple things like I'm I've written a repository layer here I saw one client where they had a acronym setting that was global for the entire plication but then every it was also the root of all their permissions and every time something was coming in every requests would query this one string from the database like five to ten times for every single requests so we're going to do we just put a case in there and that one thing and suddenly the moment database hits went right down so it wasn't a problem in T phone work it was looking at the code pasting together the story and realizing they were doing work they didn't need to do so those logical errors measure and prioritize your optimization efforts you're never going to make everything perfect so you know everything needs to be good enough for the task is made for and then find the bits that need to be fast and optimize those you can then got a whole bunch of options for optimizing your queries you can rewrite your link you can write handwritten SQL when you need C like be ready to jump out and finally one of the most effective way to speed up most applications is just caching the stuff that can be cached like think about how often it changes how long can that case be sometimes a case of a second or five seconds can massively speed things up in a very easy way of getting good casing it's once again this CQRS problem because you're reading models are separated you can create a CQRS application layer the queries the database keeps a cache for a waiver long you want and then that's spit it up I find that one of the things it's really good for communicating that to a developer is to say the fastest database query is no query at all that's right yes you're quite right like databases on the scheme of things are slow compared to in-memory c-sharp databases are incredibly slow because to make a database query you've got the start by opening a connection now that connection might be pulled but it's still fetching a connection resource then you've got to send a string down that connection resource and that requires network traffic that's an i/o blocking operation that goes to a database if you're lucky the database has seen that query recently and a lot the table there but it's still a whole series of steps being performed by a whole bunch of different systems where if it's just already in memory is just to fetch it and send it it's it's you know better than order most you'd faster to return something from case like maybe a you know factor of 100 or so yeah or faster one hundred to a thousand I'd say so finally a quick run through some of my favorite features of or watch need an entity framework course not everything but some of my highlights one of the biggest things is the linking provements more stuff I remember that highlighted big yellow blob of link which gets turned into a query more stuff works when you put it in sequel there that's them the way to look at it there's more stuff works and more cases and also though we're getting rid of that horrible bomb which was client-side execution of stuff I'd much rather it failed with them looking at it then looks like it works but then has problem performance problems down in production course most DB support which is going to be really useful for doing things like CQRS as well like cosmos DB is a great answer for providing that read model support there's a provider for cosmos DB and because of that some fundamental changes are happening in the guts of entity framework they're breaking it up and Co because before that at that point it was assumed everything was going into relational database because most EB is the other relational database it's more like a lookup key store document DB that happens to support s - well as long as it's as one of its interfaces but to do that they were actually start breaking up and now a lot of the relational logic has moved into a new package because Microsoft entity framework or doc relational and the cosmos DB provider will enable most safe caller at most their EF core features so it's going it's going to be good for most operations like it's gonna and you know certainly for things like building a read model we're not to be querying it you're gonna be here's the data for this query hold on to that and then I'll have a whole bunch of web farms sending that query and when it needs to happen it's gonna make that stuff very fast c-sharp eight support we touched on this when I mentioned one reason for not creating your own abstraction around entity framework is you're not going to get funding new features so this fun new feature is called IA sync enumerable and it's part of the c-sharp eight let's trying to explain this one way back in the dotnet 3.5 days we had this new concept come along with collections called the yield operation and the point of the yield operation was I could have a collection that sends out values as it has them and I can consume that collection kind of asynchronous as a stream now the whole async kind of awaits stuff is all about non-blocking i/o operations so what you can do is you have a task which is that thing that's waiting for an eye operations happens I'm waiting for a database query to run for example when that database query is finished that task will then pick up the await bit and carry on so if I want to use and I stand at their standard async commands with a huge database of commands and I want to load a huge bunch of data if I just use async I'll wait until that whole thing has finished running and then get that million records ia sync innumerable joins the two together so it joins that concept of an async operation to a form the initial query then the thing I'll get back implements is in consumable so it will start feeding me values as they're available not waiting for the entire query to finish running so what can do an async commands defects the data about data and then start streaming that thing through so if I want us credit requests that wants to handle a million records it'll start getting the first value when I can then plumb that stream through all the way to my output and start streaming the data out as its read and so I don't have to have a million records in men Marie I can back that up one record at a time and it's sort of streaming kind of model so very cool stuff and yeah if you were writing a repository pattern you wouldn't have a I think innumerable but you get it by default if you just talking straight sir DB sits AF 6.3 and dotnet core a lot of people have applications they're looking after and some of those applications can you believe it they're more than two years old so one of those people going to do that's right you can rewrite but you know some of these are more than two years old and they've been written on top of EF 6.3 and EF core is not the same as 6.3 there are subtle differences that if you had to go through and a whole code base and then pick them it's going to be hard so until now those people are slightly stuck do I can't move for the dotnet core and take advantage of all the new dotnet core stuff certain the Web API is that way better because I'm stuck on 6.3 is my persistence mechanism well now because you will be able to put 6.3 on top of Nick or I can have a dotnet core Web API and other features there but then still have EF 6.3 running in that same process and don't have to rewrite everything so that's going to let you take advantage of all those you know APR controllers we have one controller class all the different types of controllers rather than fundamental different plumbing for API based stuff versus MVC by stuff which is what you had just before we switch dotnet core and all that dependency injection configuration stuff as well so you get all the benefits but still keep your EF 6.3 so that's going to be a good good migration story hence the framework core is no longer part of the asp.net core shared framework because what happens here what does that mean when dotnet core first came out it was really light had not much with it if I the idea was to make it really explicit we'll ship you this empty project template if you want to do anything with it you need to download those API yourself you know put add the nougat packages which was very clear everyone knew what they would you know everyone knew exactly what the input into their project because they put it by but everyone complains it was hard you had to spend all your time working out what combination of dependencies I need so they came up with the idea of coming up with this asp net core dot all migu package then included loads of stuff so we had to do is just open up a new net core project and bang asp.net call it all was there and you had all this stuff all embedded in everyone complains it was too big it contains a lotus even it even contains stuff that wasn't for Microsoft like Redis case engines and stuff all it was a kitchen sink approach bringing everything in now while they did this they improved some tree-shaking stuff around when you compile it it jokes it down but people complained it's too much stuff so we're seeing an ongoing refinement on from hardly anything so everything and the kitchen sink to small amount of stuff and what that does mean is that the ATF core tools are no longer available by default you get to install them you got to do the dotnet tools add commands to bring the evident tools into the command line but it also means though is for all into the framework users it'll be the same experience where the way it was is if using sequel server it's already bundled any have to do anything we're using anything isn't sequel server you've got to do extra work now everyone has the same steps because they've got a Goan install and Stefan mucke by hands and put in the provider they want to use so there's one one path to follow for all the users backing fields with a couple of things I'm going to talk to you now about various improvements in getting entity framework to map and more complicated domain model the idea of backing fields is instead of mapping to public getters and setters when you're mapping from an entity so your database you actually write directly to private fields on that object so you're free to put business logic on those getters and setters you might have a set command that when you change a value in an order item here then trigger a recalculation of the order total on the parent above so doing that with classic instead ephemeral core was hard because entity framework or would write to all those fields and you cannot but not mapped attributes on but if you start right into backing fields instead there's the private field properties it'll write to those instead of writing to the public and get set properties so it gives you more freedom in having more complex domain models owned entities and collections the idea here are you can have parent and child objects where it makes no sense to load the parent object without its children say for example an order objects and that author has order items and the total in the order object is calculated based on the presence of the order items in that situation it makes no sense that all load the order object without its items because it won't work properly so you can say in a zphone work that the items are our owned collection and then entity framework will never give you an instance of an order without its related order items so once again features that make entity family got better at mapping complex domain models to your database and kind of giving you some cement semantic control so that was some more highlights of the best things that watch knew and there's a load of stuff of what's new and the docs are amazing on this there's lots and lots of little subtle breaking changes that are kind of too small and he's a tarik to go into in a talk but the documentation has been fantastic and in fact it's um the whole documentation still with Microsoft has been amazing in recent times density framework or is one of those examples just go to the docs it's very well written it's very clear and not only does it show to what it's very good at explaining the why as well so we looked a whole bunch of stuff this lamb turned out to a long talk we looked at introducing SD framework or and the fundamental concepts and I really want you to walk away understanding how entity framework tracks changes to entities that it works with and the subtle difference around that update commands the fact that doesn't mean I'm going to update the database it means I'm taking a detached object and I'm putting it in my dbcontext we look to the basic using configuring how you can really quickly get into the fragment to work but then also the reasons you know the problems you get from that really simplistic approach and how they take a step back then we looked at clean architecture and how put in ceramic into a clean architecture context and we discussed the pros and cons of do I use a repository pattern or do I take some kind of more direct dependency on entity framework and how CQRS is actually a much more powerful abstraction to put into your application at an earlier level before you then go and talk about where your database layer is and finally we looked at testing EF core performance and then what's new in entity framework any more questions before you finish up yeah just schema migration in EF core as like using the features available there as opposed to tools like DBA or questions on schema migrations the schema migrations are a brilliant feature I've got a mate I've got a great part of the talk on it I just couldn't fit it in there's too much stuff yep all right I want to be really quickly yeah thank you very much thank you I had to get this down to 45 minutes to do DDD Melbourne so and that's the only time I've only got anywhere near 45 minutes but then I had to really hammer through it it's it's hard you gotta kind of bring it all in and a few seconds per slide but I do have some code here okay let's talk about migrations so I have an entity here and mentally here is a customer and this is north wind is from the 90s so in the 90s and there was no such thing as Twitter so there was no Twitter and data so I put an extra field here to put someone to put the customers Twitter handle so to make a migration I'd go and make a change to my schema I've added a Twitter field then I run some commands that we run on air AF migrations add and give it a name Twitter and that would generate for me a migration and the migration looks something like this you see I've got two ones here one for Microsoft one for Postgres and they both have two versions of the migrations if I look at the migration here I've got one here called customer Twitter so what it does automatically from just that change happening to that entity model is it'll recognize that I've got a new Twitter feed Twitter value and will create this kind of dsl domain-specific language for the schemer updates with add column give it a name give it table very readable stuff that's cool how the hell does that work if you look behind the scenes on these migrations you will see behind every single migration there's this thing called a designer file and this designer file contains a whole bunch of generate code now way back in the ef6 days this was a big ugly binary blob of data there was no way to read it but now an EF core it's this beautiful readable fluent syntax format so what happens is the version of this from the most recent migration is written into this thing called the DB context model snapshot so when I say add migrations it will look through the current configuration of all my classes all the configurations against those classes and compare it to the data that's in the last snapshot so it doesn't read the database at all to work out what's changed it takes that snapshot of data and then compares it to the new snapshot of what it's got and then from there generates this migration class then from this migration class let's go back to that one I've got a few options I can at start up with the application apply those migrations directly to the database and the database itself will have a table to keep track of which migrations have run all I can use this to generate sequel and send that signal to my DBA who wants to vert everything there's a few options there you can also one thing to consider with the application apply its own migrations as if you want to do a farmer service they might fight over who gets to upgrade so at that at that point it's a good idea to move the migrations into a deployment time process that I come and so program runs also that means you can run your schema changes with a higher set of privileges than your runs on web application gets so you have a little console program that runs during deployment that runs the schema changing privileges your application can run with scaled-down privileges another thing to consider is this code is the scaffolded code you are free to make changes here and in fact you can also just right I think it's migration builder that's yep you can run migration but other SQL and put any arbitrary SQL you want in that process or you can write your own of these things based this is a scaffolded thing what running your own code is useful for is if you're migrating a column type because the more generally to migrate a column type you need to create a column copy the data across and then delete the old column well this won't generate that copy the data cross operation because it looks at your schema doesn't realize you want to copy from one tables the other but you can write that into this process here so you're free to write as much courage you want into here but be careful you don't get into the state where you've written so much stuff your databases wait a sync with your actual schema configuration so I'd say generate first and only use this for simple operations we're not modifying your schema research you just migrating data inside that schema you've also got some options as well as writing migrations in here you can also you've got a operation to migrate data so there's a seeding mechanism that now works its way into the migration system and insert scholar inserts data or removes data with the migrations in sync where previously you woods had a seeding system where it run all the migrations and then apply a seeding system at the end it was hard to if you had seed data that was in sync with a different party you could get yourself in the situation where you had data that worked in history as you worked up a new developer coming in it'll probably all the migrations and not have ceiling in sync and and not work properly now that seeding system actually comes in with the migrations and the seed generate generate day that you generate becomes part of that migration and so the migrations and the seed data for those migrations stay in sync so yeah that's the migration system so much is vastly improved like this whole readability of this thing now it's much easy you actually look and say what what do you think it's actually there and you look at it and you can see what the things the configuration is and then you can go back to your you can also if you don't like what this is done deletes it go back change your configurations scuffle it again and all of these migrations also have an up-and-down version so you can roll back obviously you can't undo data that was put in there if you do a drop on your database you have to up the data but you can run those schema changes back and forth through time until you get it right and the tooling for that works just as well on Mac and Linux you don't really need Visual Studio to get those features who were talking about there yeah it's a very good point and yeah these features all work from the dotnet EF commands and so it works it just uses the dotnet to run so yeah you've gone of the days where you needed the there was a time when the way to run EF migrations was to launch them from inside a special PowerShell console for that was there for package management even at the time we're thinking this isn't this isn't right this isn't how console development works it was just the first time there was a good console support for anything inside Visual Studio and so finally there being a good console support for your studio they started to use it it was a weird decision it kind of worked but now it's properly working as its own an console now the power show options still do exist but everyone uses the dotnet EF because that's like you say cross-platform and it works fine yeah I've also seen some posts around new tools coming in so we're starting to save kind of migration tools coming in and richart's all those for your mappings but I like reading the code to see my mappings I don't need to see pretty pictures I'll draw pictures and then make it in code and I wanted either they were rounded open and then you can generate pictures from your code there's not a thing I really care about any other questions your questions are passed down there I guess they're eager loading is complete bypass of M plus one because with either loading you say up front I want you to load this customer these orders and these orders owns and you say it up front and it'll go and load them all so it's I mean with eager loading you are definitely doing a lot of sequel work should getting all the data but it won't be the end post one problem because you're not using razor loading at that point you're saying up front loaded also eager loading definitely isn't al al n plus one problem oh that's right yes with with eager loading your problem might be performance if you're kind of think I might need it so load it just in case you might be loading data you know needs so yeah the problem there it's always a trade-off eager loading is safer against the n plus one problem you're going to have it in just one back to sequel work but yeah the risk is your loading stuff you don't need yet so really the best way to handle it is for me it's that CQRS approach and making commands for a certain thing I think a lot of the problems for things like lazy loading and things like eager loading is when you have a piece of functionality here that gets a bunch of data and you reuse it in different contexts so in you know of getting a customer for this context and getting a customer for this context and sometimes I'll need to have more data about the customer and sometimes I don't I prefer to split out into separate commands and load just the deado I need to fulfill that commands and then if there's a different use case have a different load process to load the data for that command then there's no crosstalk between those two operations and everything's loading what it needs for that specific use case and we don't get this crosstalk or things that break because I'm changing it over here but it's also been linked over here Adam already brought this up but they've got this Don they call super power store so I think we'll finish by saying thank you for listening [Music] [Applause] [Music] [Music]

Channel: SSW TV | Videos for developers, by developers

Views: 102,736

Rating: 4.9338541 out of 5

Keywords: ssw, software, brendan richards, microsoft, advanced, dotnet, .net, .net core, clean architecture, EF Core, ef core, entity framework, Entity Framework Core, Object Relational Mappers, Clean Code, dot net core, dot net, .net core 2, clean code principles, C#, Object-Orientated C#

Id: LDRxo6wDIE0

Length: 115min 32sec (6932 seconds)

Published: Tue Aug 27 2019

Reddit Comments

[u/Trident_True]

I remember watching this before, it's a great overview for someone like me who knows nothing about decent architecture.

Here is the template repo. It gets updated pretty frequently and takes pull requests.