Using Clean Architecture for Microservice APIs in Node.js with MongoDB and Express

in this video we're gonna look at how to use Bob Martin's clean architecture to build a micro service in nodejs with Express and MongoDB and as an added bonus we're gonna sprinkle in a little artificial intelligence hi I'm bill sewer from dev mastery and you're watching mastery Monday a weekly show that helps you write better code and advance your career don't forget to subscribe and hit the notification bell so you never miss an episode so this is the clean architecture it's an architecture model that was developed by uncle Bob Martin who is a pretty famous programmer and author who you've probably heard of I recently used this architecture model in order to develop a microt service that's going to handle managing comments for my upcoming website in this video I'm gonna take you through the code for that micro service and I'm gonna show you exactly how I applied this architecture style to that code in over 20 years of developing applications I have found that this is by far the best style of architecture to use if you're trying to build code that's meant to stand the test of time when you use this architecture you end up with an application that is very very resilient in the face of change so as things change in the world around you your code can survive and adapt and change with it all right let me show you the micro service so you can see what I'm talking about we're using an app called postman to send HTTP requests to our micro service and see the response so we've got the micro service running on localhost and we can post a brand new comment so for a comment we need an author and in this case it'll be Wayne we need some text for our comments so Wayne will say Party Time and of course we need a reference to the post so here we've got Wayne saying party time on this post and if we send this we get back a posted object that contains an ID tells us the author when it was created there's a hash to uniquely identify the comment tells us when it was modified of course it was modified at the same time as it was created because it's a brand new comment there's the post ID that references the post that were commenting on we can see the comments already been published so published is set to true when we've got some information about where the comment came from so we've got a source with an IP and a browser in this case obviously it's coming from postman and then finally the text of our comment which is party time we can also reply to comments so if we grab the idea of this comment we can use that as a reply to idea and now we can have Garth say excellent when we'll send that and there you go same idea except now we've got a reply to ID so behind the scenes all of this is going into a database so we've got MongoDB running here and if i refresh this there's our two comments Wayne and Garth and you'll notice that each comment is its own record so they're not nested within each other however if I go back to my postman and we grab the post ID we can use that to get all of the comments for this post ID and you'll notice that the responses come back nested so we've got Wayne saying party time and then within the Wayne record we've got a replies array and that's gonna be Garth saying excellent and no replies to that so there's a bit of logic running behind the scenes in our code to make this happen so as you can imagine we can update or edit any comment that we want so by grabbing the idea of Garth's comment we can change and have him say party on Wayne so we'll send that and there's the edited comment so you can see the modified on date has changed that created on is the same and Garth is now saying party on Wayne so what happens if we want to delete this parent comment so here's a comment that's got a reply to it if we delete it what happens to the reply so let's check it out we'll grab the ID here head over to delete pass in the ID and you see we get this deleted object that comes back it tells us one comment was deleted but it has soft delete set to true and we've got a message that says the comment has replies so it was soft deleted so if we go back to get and run that you can see now the author is deleted and the text says this comment has been deleted so that preserves our little hierarchy our tree here if we grab this ID try to delete that you can see now the deleted count is too soft delete is set to false and it says comment and deleted so if we go back to our get see nothing so both comments were now deleted because there was nothing left in the comment thread so that's all logic running behind the scenes and it's important to focus on what the business logic is so that when I go over the architecture in the code you understand where the benefits are so a lot of these tutorials tend to focus on the tools and there isn't really much business logic at all they're just grabbing whatever comes over the wire and jamming it into the database here we're actually applying some rules and some logic to the data not just blindly sticking it in the database so there is one more small trick that we have up our sleeve so let's say for example that we wanted to have a new comment from Garth that is slightly inappropriate so we'll have Garth say yeah when monkeys fly out of my butt and so if we send that you can notice that our published comes back as false this time so the comment did not get automatically published and the way that works is behind the scenes I'm actually using a service from Microsoft Azure called content moderator and so you know if I was talking to a client I would say I have turbocharged or micro service by using a cloud-based AI you know between you and I it's just it's an API that I'm calling from Microsoft that tells me whether or not the contents of my comment may be inappropriate and then based on that I set the published to false so that the comment doesn't automatically get published with something inappropriate and it gives me a chance to go in and review it and by the way this is how I think for the most part most of us are gonna end up interacting with and using artificial intelligence there's going to be api's and libraries out there that do all of the real API kind of stuff and we're just gonna consume those libraries or api's I'm also behind the scenes using something called a kismet and Akismet is a spam detection service so that's going to make sure that if somebody accidentally while not accidentally somebody purposely tries to put spam on my side it's gonna get caught by kismet and again I won't publish it and so all of these features that I've just shown you are all built using the clean architecture from Bob Martin so let's jump into the clean architecture I can show you what that's all about and then we'll hop through the code and I can show you how I've applied the clean architecture to the code so that we end up with this result all right so this is Bob Martin's clean architecture and you'll notice it's made up of four concentric circles so in the center we have a circle labeled entities so entities are the primary concepts of your business so let's say you're building an application for an insurance company you might have an entity like a claim or if you're building an entity for some company that sells products you'll have entities like customer order product these are all what we call entities if you're building a social media application you might have an entity like a post for example or a comment next we have what are called use cases so use cases are interactions between entities so if we're in our insurance company example you can say that you'd have a use case like filing a claim or for our retailer example you might have a use case like customer places an order likewise in a social media setting you might have user posts a tweet next we have our adapters in green and so adapters are there to isolate our various use cases from the tools that we use so from the frameworks and drivers that we may be using to deliver our application and that's the final circle so that final circle is all about these frameworks and drivers that we all use to build our applications and so in the case of my micro service I'm using Express as a framework and I'm using the MongoDB driver for nodejs as a driver the next thing you'll notice is there are these arrows pointing from the outermost circle down into the innermost circle and they go only in one direction and so those arrows signify the flow of dependency they say that an outer circle can depend on an inner circle but an inner circle cannot depend on an outer circle and that rule is the magic that makes this whole architecture hang together and work and deliver all kinds of value so if you think about it if the flow of dependencies goes from the outside in that means changes are gonna ripple from the inside out right so if something changes in a framework or a driver since there's nothing depending on that framework or driver the change is isolated to that one circle on the other hand if something changes in an entity well the use cases depend on our entities and the adapters depend on our use cases and the frameworks depend on our adapters and so that change has the potential of affecting every layer and so what we do is we put the things that are most likely to change over time on the outside and the things that are less likely to change over time on the inside and that's what makes this architecture so resilient in the face of change of course you might have noticed there's a problem with all this so if you've got a use case like place order for example customer places order how is that going to work without the use case being able to access the database because chances are the order is gonna have to be saved into a database and so you would think that the use case would have to call our adapter and talk to our database which would invert the flow of dependency well that's what this little sub diagram in the bottom right is all about so when we zoom in we see on the left-hand side we've got a controller and a presenter both of these are examples of adapters and on the right hand side we've got our use case that has been split into three boxes so in the center we have the use case interactor and so that's the logic that actually governs the use case and then on the bottom we've got an input port and on the top an output port and you'll notice there's a tiny upper case I in the corner and so that tiny upper case I stands for interface that means that the use case input port is an interface that is satisfied by the controller and the use case output port is going to be an interface that will be implemented or satisfied by the presenter now if you've got a Java Script background and you haven't done a lot of programming in other languages you may not be used to this idea of capital I interface because in JavaScript these kind of interfaces don't really exist but the good news is to do what Bob Martin is describing here all you really need to be able to do is inject your dependencies rather than import them and that's pretty straightforward to do in JavaScript and I'll show you as we go through the code so let's do that let's jump into the code and have a look so here we are in github and I've put a link to this repo in the description so that you can follow along with me and the first thing we're gonna do is look for our entities so we'll jump into the source folder and then since this code is all about comments it's not surprising that our entity is going to be in a folder called comment inside of a module called comment j/s in here the first thing you'll probably notice is that there are no import or require statements at the top of our code so typically when you jump into a JavaScript module the first thing you see is lots of require statements or inputs or import statements but in this case we're not importing anything and we're not requiring anything and that's important because this is an entity and if you'll remember from our diagram entities are in the center so entities have no dependencies so if we were importing or requiring libraries or other code then those would constitute direct dependencies and we can't have that so instead what we do is dependency injection so you'll see that what we export here is a function called build make comment and that's going to receive all of our dependencies so somewhere upstream we've got some code that will call this function and inject all of the various dependencies that the function requires in order to do its work and that has a number of advantages the first advantage is testability so now by injecting the dependencies I can actually stub out any of these dependencies so that I can test this code independently of anything else and I can make my stubs behave however I want so that I can test different behaviors and interactions the other advantage is as I'm building this code I don't need all of the dependencies to be ready I can just write simple little stubs that I can implement later and focus on the business logic for a comment finally and most importantly the advantage of doing this is that I can change the implementation details of my dependencies independently of this code so what that means is I have a function here called sanitized and it works to make sure that there's no invalid HTML or like script tags inside of my comment text now I'm depending on a third-party library to do this work that's coming from NPM but in the future I could write my own sanitized function or more likely I can swap that library for another library that maybe does it better or that's better supported and I wouldn't have to touch this code at all and that's very very valuable because that means that I'm not risking any of the business logic of my application in order to upgrade or change a tool that I use from a third-party so if we jump into the code here we'll notice that ultimately we're returning a function called make comment which is a factory that can return us an instance of our comment entity and we receive all of the properties that make up a comment we put some sensible defaults in and then the code just validates all of the inputs making sure that it enforces all of the business rules for a comment and you can see it's very easy to read so a comment must have a valid ID it must have an author the author's name has to be longer than two characters it must contain a post ID the comment text must include at least one character of text the comment must have a source so the source is what we looked at earlier where we saw the IP from which the comment was coming from in the browser and then if there is a reply ID it's gotta be valid and so this is a very nice way of being able to see at a glance what all the business rules are for a comment and these are all enforced and handled by the comment entity so a comment knows everything there is to know about what it means to be a valid comment we have our sanitized text which uses the sanitized library that I pointed out to you before and this is coming as an injected dependency and next we make sure to validate that after we've sanitized the text there is still some text left for the source we use a make source Factory and we consider source another entity of our application it's a sub entity of comment and so this makes source factory similarly to the make comment Factory that we're inside of now knows everything there is to know about making a valid source so in here you would expect to have all the similar kind of validation for the various properties of a source next we set a constant for what we're going to use to mark comment as deleted so this is the deleted text that will appear when we need to self delete a comment and then we specify a hash attribute of our comment which will populate a little later so finally we return a frozen object that exposes a bunch of get methods and what's valuable about this is that it indicates to the consumer of this code that they are only allowed to read these properties and they have no way of directly changing the properties we do everything we can to make sure that there's absolutely no way to put this entity into an invalid state so if something is gonna break is gonna break on entity creation which means that we'll have a very fast and early signal that we've done something wrong in our code and that makes things super easy to debug a very very common source of errors and problems in code is when you can put you can have some code that puts an object or an entity into an invalid state and then you don't realize that that entity is in an invalid state until you try to use it in some other code down the line and then it becomes very difficult to trace the source of the bug so you know that there's an error happening in your application because your object is in an invalid state but you don't know how or where that object got to be in that invalid state so here we make it impossible for code later down the line to put this object in an invalid state so if you're gonna put it in an invalid state it'll happen on creation and if it happens on creation we'll throw an error and you'll know exactly where the source of the bug is so moving on if you do need to change or if we do need to change some properties of our object we can do that through funk so for example we have a function called mark deleted and under the hood that's changing the sanitized text and the author property of our Comment entity and so that's the only way to modify these properties you can't just go comment dot sanitize text equals something else so that preserves the integrity of our entity hopefully that makes sense so if we jump back out to our parent folder you'll see that we expose our comment via an index Janus and if we look at index dot JSON and libraries and so this index dot j s is responsible for building our make comment function and then it's going to just export a valid version of the make comment that has all of the dependencies embedded in it so again like I said this means that if we need to change some of these dependencies if we need to use a different crypto library or a different sanitized library for example we can do so without having to interfere with the business logic of our Comment entity and you can see here we're bringing in sanitized HTML from a third-party library and then we're writing our own little sanitized function that does some configuration of this third-party library on behalf of our Comment entity so we've really now cleanly separated the business logic from the dependencies so if we jump all the way back out to our source folder we can now look at our use cases so use cases describe all of the valid interactions within our application and you can see just from looking at the folder listing you can tell exactly what this application does so you can add a comment you can edit a comment list the comments remove a comment there's something about handling moderation so this gives you a very nice map and and a very nice indication of all of the features and functionality of your application just at a glance which is quite nice so we can jump into the add comment use case and see what's going on there so here you'll notice now there's one import and that import is for our comment entity so you'll remember from our diagram that use cases have a dependency on entities so it's valid to import a concrete implementation of an entity into a use case and then we use the same technique so we have a make add comment factory function that's going to receive the dependencies of this use case and these dependencies the comments DB and the handle moderation which we'll look at in a moment these are adapters so they come from the green layer and based on that second diagram even though we don't have interfaces we're using dependency injection to make sure that we're not tightly coupled to our dependencies at design-time we only become tightly coupled to them at runtime so our make add comment Factory returns an add comment function that receives some data which constitutes comment info and we use the comment info to make a comment entity and so in here this is where all of the business rules and knowledge and information about what makes a valid comment will come into play and so if anything is invalid in this comment info we'll find out right away here next we check if the comment exists by going to the database and using the comments hash property to find any similar or identical comments and if the comment exists we just return the existing comment so this makes sure that if a comment gets submitted twice we don't add it to the database twice we don't add any duplication so that's a bit of business logic that belongs in the use case so the use case is aware now of the interactions between the various parts of our system moving on we have this notion of handling the moderation of a comment and so that's a dependency that was injected here that's a different use case that understands how to deal with comment moderation so if the comment is inappropriate or if we need to do something to review the comment that's all gonna be handled in there finally we grab the comment source and then we insert a comment record into our database and notice what we're passing to our comments DB is just a plain object with plain properties and those are all coming from our comment entity so we know that all of this stuff has already been validated and is good but we don't need the comments DB which is an adapter to our database to know or understand anything about the comment entity it's just receiving a plain object and that's another principle of the clean architecture you need to pass information between layers in the format or structure that is most convenient for the layer to whom you're passing the information let's look at another use case as you can imagine edit comment would be pretty similar let's take a look at handle moderation and see what's going on here so again a factory function and we're injecting two dependencies so one dependency for some function called is questionable and another dependency of a function called initiate review and again because this is a use case these dependencies that were injecting these are adapters that are interfacing with some framework or library or other system external to our use case so our handle moderation use case basically checks to see whether or not the comment is questionable so we get a comment entity in here and we ask the question should this be moderated and we do that through this is questionable function that was injected if it turns out that a comment should be moderated we call our initiate review and we pass in the ID and the text of our comment and then we make sure that the comment is not automatically published so that's that functionality I showed you earlier where I tried to publish a comment that had some rude text in it and you saw that it came back as published false that's what's happening here we're calling unpublished on this now moderated version of our comment entity so moving back into our use cases we can take a look at remove comment and so again in here we're bringing in our comment entity we have a factory function where we can inject our comments DB and then in here we've got all of our business rules related to removing a comment and so the one that's probably interesting is this soft delete so here we're saying if a comment has replies we want a soft deleted and if we go and look at this soft delete function which is over here what we can see is that we go and we make a comment entity out of the comment info that we've retrieved and then we call mark deleted against our comment entity and so the interesting part about this is this code here has no idea what the implementation of mark deleted is it doesn't need to know or understand how a comment gets to be marked deleted that's left up to the comment entity and so here we have this nice separation of concerns where the business logic that's relative to the comment itself is handled by the comment entity and in the business logic that's relative to the use case is handled here inside of the use case and so down the line if we change what it means for a comment to be deleted we can go into our comment entity and change the code or the implementation in there and that would just all work so this code would not have to change and so we've created this nice separation where we can change things independently of each other okay so jumping back into our use cases now we expose all of these use cases via and index is and so this index j/s brings in all of our factories and it also brings in our dependencies it's going to create our handle moderation use case and you'll notice here this is an example of a stub so I haven't written or completed the initiate review functionality yet behind the scenes this is probably going to kick off a process that will notify me of the fact that I need to go in and review a comment because for some reason it's questionable but I can write all of my logic and all of my code without having fully implemented this initiate review functionality because it's just being injected in so here this is just a stub and that's one of the advantages I was talking about earlier then as you can imagine we make our add comment use case or edit comment use case list comments remove comments again injecting all of our dependencies and what this ultimately returns is an object that contains all of our use cases so a single object that will contain all of our use cases and then for convenience I'm also exporting all of the individual use cases so that we can pull out one at a time if we need to hopping back to our source folder we can jump into the data access folder and take a look at comments DB and comments DB is just a object that exposes a bunch of database interactions so this thing protects us from a direct dependency on our database from the rest of our code so the rest of our code doesn't know how to issue a MongoDB query to find all comments that's handled in here this is the only place that understands and knows how to interact with the database and even then we're injecting our database connection code so that in the future if we need to connect to a different compatible database we can do so without having to touch this code of course if we move away from a MongoDB style API then this code would be affected but that's okay because this code is completely isolated from our use cases and our entities so from our main business logic this code is only concerned with how to write the code to retrieve and put the information in and out of the database as required and so as you can imagine our index Jas is gonna set up our connection and make our comments DB a doctor and the nice thing about this is when we go if we go out here and we look at our tests so this is our spec file we can actually bring in a different make DB function so for testing purposes we're using an in-memory database rather than trying to use the instance of a running MongoDB database on our local system and that's really helpful so we can do all kinds of faster parallel testing etc so let's jump back to our source folder and now in here we can look at controllers so just like data access controllers is another example of a type of interface adapter so that's that green layer inside of our clean architecture that green circle that is the second from the outermost circle and in here we can take a look at post comment for example and so now you can see similarly we've got this factory function and now we're injecting our use case so this injects the add comment use case into a factory function that's going to return us a post comment function and post comment is going to expect an HTTP request and then it's going to extract data from the HTTP request and then pass that data to our add comment use-case then it's going to return an HTTP response like object and so here you can see for the first time we're dealing with actual constructs and parameters of HTTP type interaction and this keeps that cleanly separated from the use case so the use case has no idea what context it's running in whether it's inside of an HTTP application or inside of a desktop application or on the front end or on the back end it doesn't care it's here now where we deal with the HTTP concerns but similarly in here inside of this controller we don't really care about the implementation or the details of Add Comment none of the business rules for adding a comment exist here here we're only concerned with the rules around HTTP interaction request response and again going back out as you'd expect we've got delete we've got get we've got patch and then of course we've got our index J s and our index J s in the same way that we've exposed our other functionality or the other layers we import our dependencies inject our dependencies using our factories and then export an object that exposes all of our various controller methods and again for convenience sake we export them individually as well so moving back to our source folder so we can now look at the entry point to our entire micro service which is this index.js file right in the source folder so if we go in here finally we get to the point where we're using Express of the Express Jazz framework to communicate over HTTP and now in here we import all of our controllers that we just saw so delete get post patch etc and then we pass those controllers into Express's various methods so this is Express Jazz if you're familiar with it you've got a pose to delete a patch and then you pass in a route and then you give it a function to handle requests to that route and so our function that's handling that is our controller and then we've got one extra little bit of indirection here we're using this make callback function which you can see is imported up here from Express callback and all make callback is this is just an adapter that takes our controllers and turns them into an express Jazz style callback and so expressed yes expects a function here that receives a rec and a res object and so that's what this make callback does and so we can actually go look at that so if we jump out to source we can take a look at Express callback and we see here so it's gonna receive one of our controllers and then it's going to return a function that takes a rec anarres it's going to map the rec object to our generic HTTP request object pass that along to our controller after our controller has done its work it's going to map an HTTP response so that's what's returned from our controller it's going to map that to a res object and then use the res object-- to set the status and send the response so going back we see that in here we see that Express is actually depending on our adapters rather than having our code depend on Express and that's the last circle in our clean architecture and we've managed to preserve the flow of dependency so that the frameworks are depending on our adapters which are depending on our use cases which are depending on our entities hopefully you found this helpful and useful and I'd love to see you back on this channel again so don't forget to subscribe and hit the notification bell and you can also check out my website at dev mastery comm where you can sign up for my newsletter and be kept up to date on all the new videos as well as articles and other content relevant to developers like you so take care and we'll see you next time [Music]

Channel: Dev Mastery

Views: 206,441

Rating: 4.9613028 out of 5

Keywords: restful api, node js, express api, nodejs express, rest api, express rest api, node api tutorial, restful api tutorial, software development, dev mastery, devmastery, Bill Sourour, Clean Architecture, JavaScript, expressjs, clean architecture uncle bob, clean architecture javascript, clean architecture node, clean architecture nodejs, express js, node js mongodb rest api tutorial, microservices architecture, building a microservice based app, clean architecture example

Id: CnailTcJV_U

Length: 33min 49sec (2029 seconds)

Published: Mon Apr 22 2019

Reddit Comments

[u/jeanleonino]

Friends don't let friends use mongodb

[u/taciomcosta]

Awesome! :)

[u/neonphlux]

Great video, immediately subscribed and looking forward for more of your content. Thank you very much.

This is a great overview of a clean architecture example in node. Its also the first time I've seen someone abstracting out functionality like validation and hashing out, which is interesting and mind opening. It also helped me solidify the differences between usecases and controllers, making me realize the parallels with the hexagonal architecture; Controllers as the drivers and usecases as receiving the driven dependencies. Curious though as why you choose to receive normalized data as input in your entity gateway interfaces as instead of receiving the whole entity. Seems like that leads to coupling between the specific implementation of your repository with your interface.

Thank you again for the great content.

[u/bot_not_hot]

Thanks for doing this. I may refactor some things in my app after watching.

[u/xPerplex]

"clean" and "MongoDB" in the same title O.o

[u/Waryjot]

Have you thought about presenting this architecture using Typescript? I am very interesting, where should be keeped interface injected to the entities? They should be in a core layer?