Node JS - Writable Streams & Backpressure

what's going on guys welcome back to whoa dude i am i'm rusty on the intros what's going on guys and welcome back to a brand new video this is web dev journey and in today's video we're going to be talking about the readable streams counterpart which is the writable stream and also back pressure but before we get into that guys i do want to say happy new years and i hope you had a very very good one on your side as well and i don't think i ever said this but uh happy or merry christmas merry late christmas i guess very late christmas it's not late right like six days six seven days right right right all right anyways uh yeah so happy new years guys and let's just get back into the video so in today's video we're going like i said we're going to be doing a writable stream and also back pressure that's what we're going to be talking about and just like the readable stream writable streams are everywhere http client and server responses are writable streams the file system has writable stream process that's dot stud out and stud error are writable streams and just like again readable streams writable streams are published in countless npms so it's very important to understand writable streams as well that's going to become a tongue twister for me later on saying writable a lot okay so anyways let's just get on with this so this file right here is basically the same file that we did with the uh readable fs the only difference is that i did i destructured the create restream from the fs because we're going to be grabbing the create right stream from that as well and just like if if create restream is reading from a file right stream is going to be writing to a file to a specific file right so let's just get that out of the way so we're going to get the right well it's create right stream there you go we're going to grab that and then we're also going to create that so well let me just copy this over right we're going to do a restream copy d it's going to be right right stream there it is we're going to be creating a right stream so i did write stream create right stream right stream and also since we're right into a data this is basically you you're just going to give it a destination on where you actually want to you know write the data too so what i'm going to do is i'm going to say that i want to write the data in the current folder and i'm going to copy copy dot mp4 well dot mp4 so what we're going to be doing is literally copying this anime dancing and before and just copying that to this file called the copy.mp4 that's all we're going to be doing okay so as you know restreams are designed to read bits of data once one chunk at a time whereas right streams are designed are designed to write bits of data one chunk at a time so instead of county logging the chunk length all we're going to be doing is writing that chunk to that um copy.mp4 so all we have to do right here is just say write dot right because we're going to be writing right so right and we're going to be writing the chunk being passed in so chunk there you go right now once the read stream it does end instead of logging that and what we want to say that hey since the restream ended we want to tell the right stream hey there's nothing being read no more so we just want you to end so write oops right stream dot end that's how you just end it no not without an error just like so right i'll get rid of this because i'll copy the error anyways there you go now just like just like restream has event listeners right stream also has event listeners so in this case let's say that hey we want to do something once the right stream ends so we're going to say right stream dot on and it's called close so once the right stream closes what we want to do is i don't know console.log uh file copy but instead of console login we're actually going to be doing process dot uh std out stud out i don't know why i said std out stud out or a standard output standard output standard output so stud out this is actually just going to be a it's a right stream right it's a right stream so it's going to write stuff to the console we could just use this and matter of fact console.log just wraps around process.out to write your streams if you did not know that so yeah anyways we're going to just do write and say file copied slash slash new line new line and there we go control save this now if we run this file using node i'm going to say node what was called writable stream or writable.js you don't see anything in vgc file copied but you are going to see that copy.mp4 and if i do click well let me actually open this uh review on file explorer there it is so if i do open this it's going to be oh sorry about that god damn it it is going to be that little see that there it's right there cop god where's that right there copy done before and this is just the exact same file or exact same uh video that we had so now that we have writable streams you know well we know a little bit about writable streams and how to write stuff let's talk about back pressure right so sometimes the data coming from the readable stream is too fast for a rideable stream to handle using our water bucket uh you know example wait where's my pin at i got it got it right here so let's go back to that water uh you know example so let's say that we have that little destination bucket over here right this is a destination bucket and we also have a little funnel with a pipe going through it right and here we're pouring in the bucket of the source right our data so we're pouring in let me see we're pouring in this bucket in here and let's say that it's full right this hose is full it can't take anymore what should we what what should what wait sorry what should we do right should we keep on pouring and let it overflow no that's that's pretty bad right so what we do is actually just stop and get rid of this stuff what we we just stopped pouring and then they wait until this water goes to our destination now once this is empty what we could do is keep on pouring right we keep on pouring water and whenever we actually have a full hose right here this is referred to as back pressure and how much water our hose can handle is referred to as the high water mark okay and we can make our hose you know bigger like i said it's referred to the high water mark and we could just make that bigger or the size that we wanted to we could make it where it could like fit everything in we can make that hose extremely big if we wanted to but like i said your memory is going to take a very huge hit so you know understanding back pressure and high water marks is very very important as well so that's why we're going to be talking about this so let me just delete all this and let's just get right into it again and we're going to give it some examples using code obviously so what i'm going to do is just copy all this i'm going to just make a new file we're going to call this the um back pressure dot js so that way it's very different even though it's going to be almost the exact same thing i do want you to have a reference of the back pressure now right stream that right actually tells it can tell us if our hose is full or not if we capture the result let's say the const cost result we're going to cut we're going to capture the result and if we capture is going to return a true or false value false meaning that our hose is full so false means our hose is full it cannot take uh it cannot take um what's it called any more data or any more water right and by the way you could check if you're having if you have back pressure just copying this mp4 you could check uh if you do have back pressure by looking at your memory like i said this is all memory related it's going to take a big hit on memory basically everything does right but this is going to be a very big hit and you can just see if this uh node process is using a ton of ton of memory if it is that means that you do have back pressure if it's not then um you have some back pressure like what's it called like solution in your in your code this is what we're going to be doing creating a kind of a back pressure solution where if it's too full it's going to stop reading right so let's just let's just get on to the video and you'll see what i'm talking about right so right now reaches capturing the result true or false false means that uh the hose is full and right now we're going to say if if it is false like a result there is no result right so this flaws is going to turn into true so if it is false then what we want to do is i'm going to log out right now for right now log damn it log there we go count a lot log i'm gonna say back pressure and what we want to do is pause the restream since it is full we want to pause restream so restream dot pause right there right and there is an event that you could use on the right stream so like remember that little um example of the water hose once the drain or once the hose is completely washed out meaning that all the data is already into the destination when can we start pouring back right and we could actually use a event listener to check that out and that's called drain so i'm gonna say right stream dot on and if the hose is drained so drain if it's completely empty our hose what we want to do is i'm gonna again i'm gonna just oops i'm gonna log out log do i'm burping log uh drained and we want to do is resume our restream so read stream dot resume there we go right and now we could check that out and see if it's all good or not so if we clear this out and do node back pressure you can see that we have a ton of back pressure look at all this back pressure right basically what this is it depends on how you look at it right for me this is pretty good because we're using less memory get rid of this so if you have a lot of back pressure that means that your your memory isn't going to be used as much right so it's stopping the restraint meaning stopping not not memory intensive no more and then just waiting until the right stream is able and then pouring back in right so this could save your memory now like i said if you don't really care about memory you have all the memory in the world uh you you could actually set that high watermark mark like we talked about right how thick our hose is and you could do that in the correct restream right you could pass in an option so what we're going to be doing right here is saying hey uh what we want is to set our high water mark and this is going to be red and bits so i'm gonna just do a very large number i have no idea i'ma just you know type in some stuff that that's good enough right let's just let's call it this right so if i control save this right now and rerun that same uh back pressure you're gonna see that hey we don't have that much back pressure you see that right there that's because our high water mark how thick our hose is is very very thick and it could like you know drain it super fast uh but this like i said this is going to be more memory intensive if i take out some let's just take out a couple of numbers or you know just like that i have no idea right let's do it one more time well that's a little bit too much so let's uh let's uh go into i don't know something like that let's see if we could uh i'm trying to give you a good exam or a good example of something well you know what i mean right i'm trying i can't think of a number right now but anyways you know what i mean the higher the higher the higher number the higher water mark is the thicker the hose is going to be right and remember this is just bits so the more bits you put into this high water mark the thicker the hose is and the more memory you're going to be using obviously because you're basically just saying hey uh once we read we really just wanted to go to we want it to uh flow uh pretty pretty steadily right we don't want it to have back pressure or we don't want you to stop breathing we want you to just keep on going right if you have a lower number lower bits that means that hey we want you to just you know resume or do the back pressure matter of fact if you just take this out you don't even have to add this in this will actually just default to uh memory efficiency basically it's going to save your memory a ton right so if i do back pressure again you're going to see all these back pressures it's it's a lot it's a lot of back pressure right so again without setting this high water mark uh we don't want to take up too much memory with our application so we could handle that through handling back pressure that means pausing our restream until our right stream is ready to handle more data and that's it for this video guys i hope you enjoyed it learn something new um hit that like button and subscribe comment down below and like i said happy new years guys um thank you for watching my videos i really do appreciate that and expect more i really want to start off this new year's with a new series but i didn't even finish this one so i'm super sorry about that guys you know i just we had to do family stuff right so we just left without even telling you guys anything so i'm super sorry about that but anyways guys thank you so much for um watching my videos um stick around because it's going to get more and more interesting obviously we're going to have more and more series so um happy new years guys and i will see you in the next video bye

Channel: Web Dev Journey

Views: 1,073

Keywords: node js, nodejs, nodejs streams, node js streams, nodejs buffer, node js buffer, readable streams, node js readable streams, writable streams, node js writable streams, backpressure, writable streams and backpressure, node js backpressure

Id: FS2OWxS5P_E

Length: 15min 8sec (908 seconds)

Published: Fri Jan 01 2021