Create A Paginated API With Node.js - Complete Tutorial

hello everyone in today's video we're gonna look at pagination which is the ability to return just a subset of the results from your API to the users so that they can go page forward or page back just the same as how Google search works and we're going to tackle this in three different ways first we're gonna build just the most basic pagination that you can build inside of an API secondly we're gonna take that a step further and we're gonna make a nice middleware for our pagination so we can apply it to any of the resources in our API and then third we're gonna hook up this simple section of our pagination to a database so that we actually have a real-world MongoDB database that we can paginate with our API let's get started now as always I just have a completely blank folder open in Visual Studio code and the very first thing we need to do is to run NPM an it with - why is the flag and this is going to initialize our repository with a package JSON next we need to install the dependencies we're gonna use and since we're gonna use a server we need to install Express which is going to be the framework for our server also we're gonna install a dev dependency by typing in - - save dev and this is going to be for node Mon and what node Mon allows us to do is it automatically refreshes our server every single time we make changes this way we don't have to manually stop and start our server every time we make changes now in the package.json in order to use node Mon let's just create a simple script we're gonna call it dev start and inside this script we just want to put node Mon and we want to put our server je s file just like this and this is going to be a file that we're gonna create now so let's create that server je s and this is going to be where all of our server code goes so let's just make sure we save our package JSON and then inside of our server je s let's set up the most basic Express server we can so we want our Express variable which we can get from the Express library we can just say require Express and then we also want to get the app portion of that which is just calling this Express function oops just like that and then lastly we can just say app dot listen and we can give it a port for example port 3000 and now if we come down here and we type NPM run dev start which is that command we just created you'll see that our server is going to start up and you see it's starting node server Egeus and our application is listening on port 3000 and this is great but right now we aren't doing anything with this server so let's set up our very first route which is going to be getting a list of users we can do that very simply by saying apt-get and we want to get users so we can just say slash users and of course we have our request and our response variable and in here we just want to return our users so we can just say response JSON and we want to return users so let's just create that users variable and as I mentioned at the beginning of this video the very first step is just a very simple pagination we're not going to worry about the database that's going to come later in the video so for now we're just gonna create a user's variable and we're just going to set this to our own array and inside of here let's put our user objects so our users are going to have let's say an ID of 1 and we're going to give them a name for example user 1 and now I'm just going to copy this down a bunch of different times so we can simulate having a bunch of users and I'm just going to increment all these numbers just like this of 10 11 12 13 and same thing for the names of the users just get all this out of the way 10 11 12 13 perfect now if we go down here save this you see our application restarted and we can create a file over here which is going to be called request dot rest and I'm using an extension in Visual Studio code this extension here is called work and pressed client right here and this allows me to make rest request inside of a file called dot rest for example we can say that we want to get at a local host 3,000 make sure you put HTTP at the beginning slash slash and we want to get slash users just like that and you see the send request button if I click that you'll see it makes the request over here and as you can see we get our list of users returned to us in JSON so we know that our application is actually working what's minimize all this go back to our server so we know that our users is working but we actually want to be able to pass what page and what offset we want so that we can get only the resources we want and not the full list because you imagine this clothes could be thousands of users long and we don't want all the users every single time so for example let's look at our request here and maybe we want to be able to pass some resources for example we want to be able to pass that we want to have a page let's say for example page 1 and we also want to pass a limit let's say that we want the limit here just to be 5 for example so it's the same give me the first page of users and I only want 5 users from that page obviously right now if we send that request we still get all the users it's not actually doing anything so let's go into our server and start implementing this so inside of this get users function we want to be able to get those variables we just passed up so we have our page which is equal to request query page and the exact same thing for that limit variable now we're actually processing this page and limit request that's being sent up to us and all we need to do is actually get the array that we're returning right now which is all of our users and condense that to only the specific page and the specific users in that limit range and the easiest way to do that is to get the start index and the end index of where we're going to be query in and the start index is actually fairly simple all we're doing is taking our page -1 and we're just multiplying it whoops times our limit so what it's saying is we know that our page that we're on is going to be a page we want for example one is going to be a page 1 page 2 is page 2 but since arrays are zero indexed and these pages are 1 indexed we need to make sure we subtract 1 because page 1 is essentially the same as starting at index 0 inside our application so 1 minus 1 would be 0 times our limit we start at index 0 now moving on to the end index this is just going to be page time as limit because we don't need to worry about subtracting 1 since we want to get the very end of our array and not the beginning of it so now that we have our start index and our end index what we can do is actually query our users to return just the users in between the start and end index and that's actually fairly straightforward all we need to do is say users dot slice and the slice method we pass it a start and an end number and it gives us everything between those so we can just say start index and end index and we can just use this as a result so we can just set this to result users and let's actually return that as our JSON let's make this request again and you can see we get users 1 2 3 4 & 5 these are just our first 5 users if we change our page to page to make a request you see we get the next 5 if we get 6 through 10 and we could maybe change this to be a limit of 3 and now if we make the request you see we get 4 5 6 which is the second page if you limit by 3 and now this is all great but usually when you have a paginate 'add api you actually want to return somewhere inside that api what the next page would be in the previous page so that the user knows if there is a next page or if there isn't the next page and same thing with previous page it just makes it easier for the client consuming the API so in order to do this we actually want to append extra results to here so let's create another variable we're just going to call this results and we're going to set it to an empty object and then I resolved users we're just going to say results dot results is going to be equal to these users and then we want to return our results so now in our request if we make this again you see we have a results section which is going to return our array of users and now we can actually put extra information inside of our object for the next and the previous pages so let's do that now let's set up next so we can say results dot next and this is just going to be equal to an empty or an object and the object is going to have a page as well as a limit property and now the one that property is going to stay exactly the same we're just going to pass the limit they passed us but as for the page we actually want to pass the next page so we can just say page plus 1 and we can do the same exact thing for previous and as you guessed it we just want to do page minus 1 so page minus 1 and we want this to be previous so now if we do our request again you see we have it next which is saying page 21 and this is actually a problem if we go back to our server this is being passed up as a string same with the limit so we want to actually parse this as an int so we can just say parse int and we can do the exact same thing with our limit so we want to parse that as an integer now if we save this make our request you'll see that our pages 3 are limits 3 and our previous pages 1 & 3 now this works great but what if we change to page 1 if we make our request you're gonna see our previous as page 0 and obviously this is not there is no page zero so we need to make a couple checks inside of our server let's start with the previous one first we want to check to see if we have a previous page and this is pretty easy we just want to see if the start index is greater than zero because this means that our array here is not starting at the very first user also for our next results we can do a very similar thing we just want to check if our end index is less than our users dot length and we can flip this instead of here so what this is saying is that our end is somewhere before the end of our actual users array so we have extra users to query so we should show them the next and now if we send a request you see the previous section goes away but let's say if we make our limit here to be 13 for example now if we save this click send request you see that immediately our next and previous go away because there is no next or previous section that we can go to we can even change this down to 10 for example save make a request and you can see now next actually shows up as a result because we have a next page since we have more than 10 users now this is the very basics of how to set up pagination but we have to copy this code every time we want to paginate a different user let's say we have a different model let's just copy this and let's say we have post and we're gonna changed here user to be post for all these just excuse all this so now we have two different models we have a post model and a user model and we want to have an app dot get for the post so we can come in here post and of course request response now in order to set this up we'd have to copy all this code like this and paste it inside a post and change all of our user variables such as here change those to post and that sounds like a lot of work so instead what we want to do is set up some form of middleware where essentially all we do is pass it up here we can say paginate 'add results and all we would do is pass it our post and this would actually do all the pagination for us we wouldn't need to do anything ourselves so in order to do that we need to create a function that will handle that so let's remove all this code instead of here we don't need anything instead of our post it'll just be completely empty for now just like that perfect now let's go all the way down create that function whoops function we're going to call it paginate 'add results and this is going to take the model that we want to paginate and now inside of paginating results what we need to do is a middleware function always takes request/response and next so we need to actually return a function which takes request/response and next so we can just return an arrow function like this so normally when you create middleware you wouldn't be able to pass anything to it you wouldn't be able to pass this post because you're passing a function that accepts request/response and next but in our case we're passing post to this paginating results and we know that the result of this function here needs to be another function that takes request response and next so down here we're making sure we return a function that takes request response and next so it will properly work as a middleware and the way middleware works is it actually executes before all of the code inside of your thing so we have all this code inside of our app kit and this middleware is going to execute before it and if we call this next function it'll actually move forward and call the code inside of the next section within our case is this get section for our users so what we want to do inside of here is just copy almost all of this code let's just get all of this code from our users copy this down inside of our paginating results and wherever we have the word user we want to replace this with our model so it's the users we're going to do model that linked and instead of slicing we want to slice our model so there we go now we actually have a results let's make sure we indent this properly so that we have this results variable but we want to save this so that we can access it inside of our app kit so what we can do is we can actually save this to our response what we can do is we can say res dot and we can call this whatever we want I'm going to call it paginate ADRA's ult's and we want to set that equal to our results this is just this variable right here and then we want to call the next function just like that and now we can delete all of this code instead of here and all we need to do is send down our paginate ADRA's ult's so we can just say we result dot paginating results and all we need to do is make sure that inside of here that we actually do paginate a results and we pass it in the users so just change this post to users here and we can do the exact same thing our post up here for paginate results let's save that and now let's query our users and make sure it works you can see we query and we get the exact same results our first 10 users and we can change this to post hit send request and you can see we now get our post instead and we didn't have to copy any code all of our code is in this paginate a results section down here and let's just kind of go through a little bit how this works if you remember we pass in the model we're using for example post or users and then we're returning a function which acts as a middleware and as soon as we call next it's going to go to the next chain of events which in our case is this function we defined here which is returning our JSON for both of these and we're doing just all the same stuff inside of here we were doing before the only difference is we're actually setting the paginating results to our result variable so that we can use it later on in our chain as you can see right here and now that's great for setting up the basics of pagination and then extrapolating that into this middleware function here but what if we actually want to use this inside of a database well to do that we're gonna set up a very basic MongoDB database using Mongoose so if you don't already have MongoDB installed on your computer check out the video linked up in the cards or the description and that'll show you exactly how to download MongoDB and once you're done with that you can come back to this video so let's quit out of our server here and we did to install a library called Mongoose and this is just going to allow us to easily interact with our MongoDB database through this kind of wrapper that just makes it so much easier now that we have that installed we can remove a lot of our boilerplate code let's remove these users and these post variables and we don't actually need this post get anymore now the very first thing that you need to do with MongoDB and Mongoose is to setup a schema which will allow us to actually access MongoDB and save records in a collection so to do that we're just gonna create a file called users jeaious this is going to be our schema so we can import mongoose into here which is just going to be called choir whoops require and we want to require mongoose just like that and what we want to do is set up our user schema so we can just do that here it's going to be equal to a new mongoose dot schema and the schema is going to take all of our data all of our properties for our user in our case we just have name because ID is automatically generated and we going to tell this that we want this to be a type of string and that we want to always require our user to have a name so that's our very basic user schema and the last thing is we want to export that user schema so we can use it so we can say module that exports is going to be equal to mongers whoops Mongoose dot model and we want to make sure that this model here has a name in our case we're going to call it user and our schema which is our user schema and with that setup we already have our user model created so we can actually start creating users saving users and most importantly querying and paginating our users list so now back into our server let's actually cook up our server to our database so we can have in here Const Mongoose is going to be equal to require of mungus just like that and also while we're at it let's import our user so we can just say user is going to be equal to require dot slash users just like that so now we have our user model and our database connector and to connect to our database we just want to say Mongoose dot Connect we're gonna pass it the URL normally this URL you want to save in an environment variable but for the purposes the application here we're just gonna write it directly in this connect string so all we need to do is save DB colon backslash backslash localhost and then another backslash and the name of the database we're gonna be creating if we're just gonna call this pagination it really doesn't matter what name you use and let's just save that and start up our server to make sure it works again npm run dev start and hopefully everything is green and we have no errors but you'll see of course we have an error so let's see what that is it says required is not defined and it looks like here I didn't put true for required I just forgot that property now if we save it and check you can see again we're having an error and it's saying users is not defined and that's of course because we don't actually have our users variable we're going to change this to our user model and now everything should work properly and we're getting some deprecation warnings and this is actually okay well it's pretty much saying is that Mongoose goes back and uses some older versions of MongoDB and we just want to tell it to use them version so let's just copy this code and paste it in here we need to use the new URL parser and we also want to use the unified topology so we can paste that in here also with the value of true now if we save you can see our deprecation warnings have gone away and depending on when you actually watch this video and implement this you may have more or less deprecation warnings that you need to worry about just copy over the code if you do have them and it should work just fine now the next thing we need to do is just populate some default data inside of our model the very first time our app loads up normally you wouldn't need to do this but since we have an empty database we need to add some data to it so mongers actually gives us a really great tool to connect the database we can just say Mongoose connection and this is going to be our database so we can just say Const DB is going to be equal to that then we can say DB dot once so the very first time that we open up our database so we'll say open we want to run some function just like this and this is going to be an asynchronous function so let's just put a sink in here whoops just like that and inside of here the very first thing we want to do is check if we have any users so we're gonna say users direct count documents just like this this will give us the count of all documents if we execute it and we want to check to see if that's greater than 0 and since this is asynchronous we just want to await this just like that whoops and now it's saying if we have any users in our database all we want to do is return because after here we're going to actually implement adding our users to the database so if we have users we just want to return so we don't add even more users we don't want to have duplicate users and that we're going to do is we're going to use promise dot all so we can just execute a bunch of code inside of these promises and we'll just have a simple dot then at the end which is just going to log out for us comes out log added users so one side here we're gonna add all of our different users and we can do this by just saying user dot create and we just want to pass it in a name and the name here is going to be user one just like that put a comma at the end and we're just gonna do this a bunch of times so we have a bunch of different users two three four five six same old same old ten eleven twelve just like that and now if we save this you should see after a bit of waiting that we should have printed out for us console dot log added users and as you can see it's added all the users for us down here so we know that everything is working for us and if we save this again you should see that the added users doesn't show up because it's actually returning out here because it recognizes that we have users now with all of that out of the way we're finally ready to actually implement the pagination for our database we have users in the database and now we can actually do the pagination and this is going to be very similar to how our pagination worked before we can actually keep all of this code up here at the beginning exactly the same we can keep all this code for starting and index exactly the same the only thing that's going to change is how we actually query our model so what we want to do is we want to find users and we want to find all the users so we're not going to pass anything inside of our find and then what we want to do is we want to limit how many users we have to the limit that we got passed in so this will only return five users if we pass a limit of five and in order to start at the right section we just want to make sure we skip and the Skip is going to take a number which is going to skip two in our case we want to skip to our start index and then only get the next however many we limit units and then we can just run the exec function and of course we want to make sure that we await this because this is asynchronous code and we can wrap this inside of a try it catch so instead of the try here we just want to put this actual code and we want to make sure that we call our next inside of the try if that executed successfully if for example we had an error in here all we want to do is catch that error and we want to send that error to the user so we want to just come in here and we want to say res status just like that I'm gonna send a status of 500 and we just want to have a simple JSON message so we can say message is e dot message and of course we also want to make sure we set our request or a response paginating results just like this and we don't need this arrow here I don't know what I was thinking with that now if we save that you can see that we have an error with the application it says a weight is only valid in async functions and of course we just need to make this an async function now if we say that everything is green down here so let's actually try to query our users and see if this works all we need to do is change this to users here instead of post so we add users and let's make sure that we query let's say a limit of three and we want to start at the second page let's send that request and you can see we have our previous being shown up with page one a limit of three a results being shown up for users four five and six but for some reason our next is not showing up we must have some kind of error with our next so let's go and make sure that we check why our next is not showing up inside of our server if we go all the way down here you can see obviously our model dot length there is no length to our model what we need to do instead is actually check the count of our documents and of course execute that and make sure we await that now if we save that go over here send this request you can see next is properly showing up and if we make our limit really large for example we make it 14 and we set our page to 1 send our request you can see that next in previous no longer show up because there is no next or previous page to go to and that is how you do everything from the most simple pagination to the most complex database related pagination if you enjoyed this video make sure to check out my other videos linked over here and subscribe to the channel for more videos just like this thank you very much for watching and have a good day

Channel: Web Dev Simplified

Views: 84,754

Rating: 4.9563885 out of 5

Keywords: webdevsimplified, api pagination, api pagination tutorial, api pagination js, api pagination javascript, pagination, pagination js, pagination javascript, pagination tutorial, pagination express, pagination node js, pagination api rest, express api, express api tutorial, pagination node js api, pagination mongodb, pagination mongoose, pagination explained, paginated api, paginated api node js, pagination api tutorial, paginated database, paginated node js tutorial, paginated

Id: ZX3qt0UWifc

Length: 23min 20sec (1400 seconds)

Published: Tue Sep 17 2019