Node JS - Pipe, Duplex, & Transform Streams

what's going on guys welcome back to a brand new video this is web dev journey and in today's video we're going to be talking about a few things actually we're going to be talking about piping streams duplex streams and also transform streams these are essential streams that we need to know about and you probably messed around with some of these already you just didn't know right so let's just get right into video because this is going to be i'm not gonna say a long video because i don't think it's gonna be that long but it might be so i'll let you see it screw it just a hit this might be a long video okay so you might be thinking streams while streams is fun to work with or very interesting to work with this is a lot of things that we have to do you know just to get some streams out right just to read some streams and all that well to that i must say you could actually delete all of this and also delete this little option that we have right here we don't need that anymore and simply what we could do is literally just pipe the restream to the right stream right so all we have to do is just say restream dot pipe pipe and we want to pipe it to the right stream well so whatever we're reading we want to pipe whatever we're reading to this right stream that we have so write stream just like that right so so again what we're doing is the same thing instead of wiring up a bunch of listeners to listen for chunks of data and then pass those chunks of data into the right stream the pipe method is doing it for us the pipe method also automatically handles back pressure for us the only thing we didn't do is wire up an error message or an error listener and we could simply just do that by saying dot on error all we want to do is log the error that's it get rid of that semicolon and yeah that's it this is basically doing the exact same thing what we had in our back pressure uh but the only thing is that we only did it in this one line of code and i could show you that it's working i deleted the copy dot mp4 because we're going to be redoing them and if i do node uh pipe.js this is the name of the folder type.js oops right error was not defined i'm not meant to do long it's supposed to be dot error that's it so ctrl save this sorry about that guys let's do that again and there we go and you can see that the copy dot mp4 is right there it like i said it's doing the exact same thing so any restream can be piped into any right stream and we're gonna take more uh we're gonna look into that a little bit more so let's actually delete this or i'm gonna just comment this out so that way you have a reference to that and what i'm gonna do now is come this out because we're not gonna read that and we're actually going our restream is going to be from our input and we're gonna be writing to a text file instead so i'm gonna say uh file whatever and then it's going to be a text file so what we're going to be doing is reading from our keyboard from our input and then you know writing it to our file so what we're going to do is say process dot standard input so stud in dot what we want to do is pipe this whatever our standard input keyboard whatever that is we're going to pipe it to the right stream that one we created upstairs and that right stream is going to be creating a right stream to the follow-up text right so if i control save this and let's try to do this right here let's do node pipe.js if i hit enter it's rating is waiting for me to do a input right so i could let me actually open up you can see that i already created the file.text so if i do hello hit enter and then i say world hit enter wasp misspelled word get rid of that enter again there it is right there so as you can see it's actually just piping our text from our input and then just you know writing it to that file.text i'm gonna do one more time this is only going to show once file dot txt is closed then we opened right if i hit enter it's not showing i have to get out and there it is right there another way of looking at this let me actually close this out clear this out and i'm pretty sure you already know you've seen this before where you could echo some stuff from your terminal right hello world and it'll just say out hello world we could actually pipe this down to our node.js application right here and all we have to do is just say hello world and then pipe the unix pipe right you've seen this if you work with linux a lot you've seen that little pipe so we're gonna pipe that to the node uh pipe.js pipe right so if i hit enter essentially what it's doing is since we're we're piping the hello the text hello world is going to write that to our file.txt and if i open that you're going to see hello world right there and another quick example is clear this out is cat you know cat right cat is just going to give you the contents of a file if i do cat pipe.js right it gives me the cop the contents of that file what i could do is literally just pipe that again you know in pipe.js so what i could do is just say cat pipe i'm going to use a unix pipe we're going to pipe this to the node pipe pipe dot js right yeah there you go so if i hit enter right here you're gonna see that file.text is actually just our program that we wrote over here is exactly the same thing so the pipe method allows us to pipe data from any readable stream to any writable stream okay now let's actually focus on now let's turn our focus to the duplex stream now a duplex stream is a stream that implements both a readable and a writable stream okay it does both these streams allow data to pass through readable streams will pipe data into a duplex stream and the duplex stream can also write that data so duplex streams represent the middle sections of pipelines so what i'm going to do is get rid of this this and this back pressure i'm just copying um under this right here create a new file we're going to call this the duplex dot js paste that in there we're going to get rid of this high water mark don't need that and what i'm going to do is again just by simply just saying radiostream.pipe and we're going to pipe everything to that right stream there's nothing i mean you've seen this before we did it already i'm not doing a duplex stream just yet but this is what we're going to be working with so in order to do a duplicate stream what we're going to be doing is grab the most basic type of blue duplex stream which is called the pass-through and we could actually grab that from the stream module so i'm going to say const up here we're going to grab the pass-through we're going to set that equal to require and it's going to be from the stream module that's pre-built and we're going to grab the pass oops pass through that's why it didn't want to work man i was like why isn't it working so it now okay there it is pass through right so this is the basic uh duplex stream and you should if you don't know too much about it don't worry like i said we are going to look at this but you can look at it in the node.js documents as well and then after this we're going to create a new pass-through stream called report so i'm going to say just say cost report and call and equal to pass through or actually new passthrough sorry new passthrough and this is our report stream now or our new duplex stream which is called report and when when i say that the duplex 3 represents a middle the middle sections of a pipeline that means that we just put them in between readables and writables so right in between this readable stream and this writable stream what we're going to do is just say we also want to pipe our new report stream down here as well so a duplex stream can be piped between a readable and a writable so it means that a duplex stream can receive data from our restream and send that data to our right stream okay so you might be wondering what's the whole point of implementing a pass-through stream right and well there's several reasons but basically if you just wanted to to see something about the data or report on it okay now i do want to say that a typical uh duplex stream doesn't change anything about the data there is a type of duplex stream called transform streams which we will talk about later in this video and technically yes the pass-through stream or the pass-through is a transform stream but we're just going to record we're just going to use it as a basic duplex stream in this example so what i'm going to do is simply just say let we're going to just count how many chunks of data is being passed through that is it so i'm going to say let uh total whoa total we're going to set that equal to zero and what we're going to do now is is we're going to report on that data right since report this report stream is a readable and writable we could actually add listeners to this right so right here i'm going to say report dot on we're going to uh we're going to add the data event listener meaning that once it receives data we're going to report on that right so we're going to have a chunk we've done this before chunk and we're simply just going to [Music] increase that total so we're going to say total is going to plus equal plus equal chunk dot length we've you've seen this before this is nothing new and all we're going to do is log out the bytes oops let me just and it's going to be the total whatever the total is at that moment there you go ctrl save this now if we open this up and try this out let's try it out let's go over here clear this out and let's do node duplex this time and you're gonna see that hey look at that is logging out all of our bytes right and this is coming in the report like i said it's just something that we could do report on or see the data something about that data if the data is coming through right see something about the data that's coming through good god what am i talking about right so duplex streams help us compose streams into pipelines let's say that let's let's say that now you want to slow this whole thing down right you want to slow how much how fast the data is coming through this right and there's actually another type of duplex cream i said cream duplex stream called throttle and we're actually going to be building it out by hand you idiot so over here what we're going to say that we're going to create a new instance of something called throttle we have not done anything or we haven't built throttle yet but we're going to create a instance of that so we're going to create a variable called thraw draw toll there you go we're going to set that equal to a new instance of throttle which we still have not created yet right and we're gonna pass in 10 milliseconds for right now this is the time delayed that is going to this is the time delay i guess the the nation the the what the delay the delay okay god that sounds weird say it anyways so 10 10 milliseconds is going to be our delay and since we are going to be building this uh throttle uh using duplex we are going to need to grab that so we're gonna say duplex right here no throttle is not just a pre-configured no it's not something like pass-through we actually need to build this out so we're gonna say do plex we're going to just grab the duplex right here and up above we're going to create a class called throttle and it's going to extend our duplex duplex right there and we do need to have already some functions inside this duplex called read and write remember duplex are a combination of both a readable and a writable stream so it's going to have automatically the read function and also the underscore write function as well we're also going to have a constructor up here constructor and we are going to be passing in the milliseconds as you see right here 10 milliseconds inside this constructor we're also going to invoke super without the ms and we're going to just set this dot delay we're going to set that equal to the ms that we have upstairs and that's pretty much it for that so when we're reading or writing the data we want to throttle that writing right we want to throttle how fast it's going through right so in the read section or the read function what we're going to be doing is this dot push now remember this dot push is referring to the stream itself so the throttle stream so this is not push we're going to be pushing by the way sorry i'm getting ahead of myself the right method you already know that the right method is going to have a chunk these are already things that the right method already has right so the right when you're writing to this function is going to have a chunk the encoding type encoding the encoding type and also the callback encoding there you go so we're going to desktop push the chunk out and also like i said this is where we want to throttle that how fast it's writing right so we're going to set timeout on this you already know how to do that we're going to get the callback we're going to call the callback once this.delay has happened so once the 10 milliseconds has passed right so delay there you go and i am going to add in one more function called final and this is basically once there's not no more reading being done there's nothing to read no more so once this happens what we want to do is this dot push no so that way we could just stop reading and stop writing ctrl save this now let's try to let's try this out if i clear this out wait sorry we can't try it out the reason why is because we haven't even piped that so what we're going to do is literally just say pipe like i said this is the the middle section right so we could put this anywhere we want so we want to pipe the throttle let me actually do it like this dot there you go throttle so we're going to pipe the restream through our throttle first so basically right here this is what's going to happen right it's going to push the chunk into the report so we're going to see that console.log and then it's going to uh delay for 10 milliseconds and then redo this again push the chunk delay push the chunk delay chunk delay right it's going to do that all the way through so we're gonna see this right now this uh open this up let's do node.duplex js and you're gonna see that look at this you're actually seeing it delay right this is much more slower than what it did in the first time and to really get a clearer view of this we're going to actually delay it 100 milliseconds where is it at right here right and you're gonna see it's going to really really slow down look at that pretty cool right ctrl c so as you saw duplex streams are a necessary component when you want to compose streams into a complex timelines or complex pipelines not timelines pipelines right the duplex streams implement both a readable and a writable side of things so therefore they represent the center parts of a pipeline and the last stream we're going to be talking about now is the transform stream we which we could transform the stream coming in so what i'm gonna do is actually i'm just grab this copy this delete all that stuff new file and we're gonna call this trend form dot js there you go paste it and yes we could actually grab the transform from the stream module there is something called transform there it is right there right so transform streams are a special type of duplex streams okay instead of simply just passing the data to the restream into the right stream transform streams change the data transform streams are the center pipe pieces that can be used to transform data from readable stream before they are sent to the writable streams right so that's what we're going to be doing in this video or in this part of the video so what i'm going to do is actually create a class and we're going to just call this replace text basically we're going to write a text out and we're going to replace it with a character of our choosing right so replace text txt and this is going to the extend the transform there you go we're going to have a constructor obviously constructor obviously right right right right we're going to invoke super also we're going to grab remember we're going to so i do want to explain what what replaced text does so what this is going to do for us so every time we write a a string called let's say hello right all we want to do is replace all those characters into x's so this is going to spit out uh it should spit out just that just x's right or replace it with any type of character you wanted to so you want to do s's or vs's right that's too many yeses but whatever you know what we're trying to do right so we're going to be passing in in the character into the constructor and we're going to add this dot replace char and we're gonna set that equal to the char that we passed in now unlike unlike the uh typical read and write streams where if you want to create methods for those those have automatically the read method and the right method transform has different types of method they actually have the transform method and that's true right here and this is basically the same thing as the right method so we're going to have a chunk being sent in we're also going to have the encoding type and also we're going to have the callback okay and what we're going to do here is literally create a little you know we're going to replace all of our text with that character that we have so i'm going to create a variable called const um let's call it trans form chunk chunk and we're going to set that equal to chunk we do need to read it so chunk dot to string remember we've done we've talked about that before and i'm going to replace everything i'm going to use what i'm going to use is regex expression for this but i'm going to replace everything with the this dot replace char you play the rest yeah i must place there you go replace char and the red expression i'm gonna use is anything coming in as lowercase a through z [Music] or uh uppercase a through z or any numbers that are coming in zero through nine and i forgot to do my bracket right there so anything that's lowercase a through z or uppercase a through z or numbers coming in we're going to replace everything with the replace.chart whatever this character might be it might be okay so once we do that what i'm gonna do is we're going to this dot push we're gonna push that transform uh chunk into our stream and also we're going to call the callback that's giving me an error because it's const like that okay and another method that we're going to add is the flush method and basically once the restream is done we could add more data or more things to the right stream right if we wanted to so in this case it is going to have a callback and what we're going to do is just this dot push the final thing and we're going to just say more stuff is being passed through right and that's it and then we're going to just call out the callback once again so this is our replace text stream now all we need to do is create a new instance of this so we're going to say const i'm going to call it x stream because from because we're going to create a new uh replace replace strike stream with the character x we're going to replace everything with the why is the cat x right and all and all we have to do now is just you know pipe this through our restream pipe it and then go into our right stream so what i'm going to just do is just say process because i just want to read from the keyboard so anything that you type in the keyboard is going to be transformed into those x's so process stud in or standard input right dot pipe and we're going to pipe er we're going to pipe our x stream first of all xstream why isn't it doing it it should be grabbing it automatically right so we're going to pipe that we're going to pipe our standard input through our xtreme first and then what we're going to do is pipe that to our console right so i could say council log or i could do another type of way which we've done process dot stood out right and that's it so we already know that any restream could pipe anything to a right stream so let's see this in action right so let's go over here clear this out and we're going to node transform there you go so if we say hello you can see that it transform our string into all the x's right and we could see oh i can see is x's there it is now there are a lot of transform streams out there right no js comes with zlib package that's a transform stream that can be used to zip incoming data from the restream and send compressed data to the right stream crypto is a npm package that has transformed streams that can encrypt data chunk by chunk and then pass encrypted data to the right stream or decrypt data from the restream and pass the decrypted data to the right stream so transform streams are the essential part of the stream family all right so that was it for this video guys we went over the transform stream we went over the duplex stream and also the pipe stream all right and these are basically all the basic streams that you need to know or that there are right to do a lot of the things that npms do and a lot of things that well a lot of applications do right so in the next video we're going to be talking about http streaming right doing streaming through the browser that's what we're going to be doing in the next video which is probably something that you guys wanted anyways none of this probably none of this but yeah anyways thanks for looking into this guys looking into my videos you know uh watching my videos with your own time i really do appreciate that so thank you guys hit that like button comment down below and subscribe if you have not and i will see you in the next video which is more than likely i'm not gonna lie it's more likely going to be our last video next video right so thank you guys for watching this video i really hope you enjoyed it and i will see you in the next video bye

Channel: Web Dev Journey

Views: 4,899

Keywords: node js, nodejs, nodejs streams, node js streams, nodejs buffer, node js buffer, readable streams, node js readable streams, writable streams, node js writable streams, backpressure, writable streams and backpressure, node js backpressure, pipe streams, duplex streams, transform streams

Id: rQXaDH__Suk

Length: 27min 5sec (1625 seconds)

Published: Mon Jan 04 2021