Understanding Streams | Understanding Node.js Core Concepts

foreign let's Now understand one of the most important Concepts in note streams let me just say that if it wasn't because of how strings are handled and noted node wouldn't be as popular as it is right now not even close so it's of utmost importance as a node developer that you truly understand this concept and how node.js strings work still a lot of people who say it's hard to understand streams I keep going through different tutorials about streams and I feel like I know streams and at the same time I don't know streams well I have to say that it's not harder to understand them it just takes a little bit more time to understand them you first need to have a foundation of knowledge of note and some Concepts to be able to understand streams properly and that's exactly what we're going to do in this video if you're just here to see some examples of streams and just know what they are this video is not for you our aim here is to truly and deeply understand streams so much so that you could just use them in your applications and feel confident that you are creating highly performance and memory efficient applications and to be honest that will take some time so let's take a look at what we will do in this video pretty quick we're first going to learn about how to use an existing reliable stream and node now the stream could be in one of the Native node.js modules itself or it could come from a package that you installed so we're first going to learn how to use these rival streams and then after we learn how to use them and after we understand them we're going to learn about how to develop our own writable streams and Note and after we're pretty much done with the routable stream we're going to move on to the readable stream so we're first going to learn how to use an existing Rebel stream and then we're going to learn how to develop our own readable streams and these two different types of streams are the fundamentals of these streams so after we understand them we will be pretty much done with understanding streams at least for the most part and then we're going to learn about duplex and transform which are just variable and writable and but they are implemented in some special ways but we're going to get into all of them later in the video now if you don't know what his arrival stream what is a rebel stream what does it duplex and what is a transform don't sweat it we're going to talk about all of them in great detail later in this video this is just like a quick kind of like a table of content for this video so after we're done with learning how to develop our own readable stream we're going to learn how to implement our own duplex screens now with duplex it's just a stream that has a writable and a readable in itself so if you understand those two understanding duplex will be really easy so we're first going to learn how to develop a duplex and then we're going to learn how to use a duplex by using that duplex stream that we just created and after we're done with duplex we're going to do the same thing but now with a transform stream so we're going to develop a transform stream first and then we're going to learn how to use that transform screen that we just developed and the transform string that we're going to create is a decryption and encryption application so we're going to learn how to work with the raw binary data with these division ones and modify them in a way so that we're going to have an encrypted file and then we're going to do the same thing but in the opposite direction so that we will change our encrypted file into a decrypted file which will give us back our original file so we're going to write a lot of code in this video for each one of these we'll possibly create some simple applications and we will gradually enhance our knowledge of streams our understanding of streams step by step so we have a lot to cover there are a lot of Concepts that we need to unwrap and understands well which means that this video is going to be pretty long and please don't try to do whole video in just one sitting in just one day that would be really counterproductive please instead do it in different portions maybe do it in 50 minutes settings or something like that so watch the first 50 minutes right now today watch the next 50 minutes tomorrow or the next hour whatever feels most comfortable for you so do it in different sessions and different settings and not just all in one day because there's a lot that we need to learn about and doing it all in one day I'm sure it's really not going to be helpful and you we will really end up confused at the end of the day so please do it in different sessions and don't just say okay I'm gonna fast forward I'm gonna skip this part I'm going to skip this part and do it all in just one day and boom now I understand screams completely definitely not do it in different sessions in different days and I'm sure that you will really take your knowledge of streams and node.js to the next level after you're done with this video so do it in different days write code with me follow along with me create some app locations do the challenges that I'm going to give you and afterwards I'm sure that streams would be something really easy for you to use and the quality of the applications that you're creating will really increase all right and we're also going to write some codes that might not be really related to streams of the first sites but we will they will really help us to understand the bigger concept here streams so we're first going to write some programs that are not really related to streams directly what we're trying to do is to build a background and then really see how useful streams are I know exactly where you should use streams so we're going to use what we already know buffers and the file system module to do some things with them and Benchmark our programs and push what we can do with them to the limits and gradually we're going to introduce the concept of streams now I also want to say that buffers are at the core of streams and if you don't understand buffers to be frank you won't understand streams at all it will be really hard so please if you don't know anything about buffers go ahead and watch my video about buffers here in this course in this playlist and we're also going to build on our current knowledge of buffers right now in this video all right so let's get into it I'm going to open up my terminal and on my desktop I'm going to create an empty directory called streams and here in this folder we're going to write all of our codes related to this video we're going to create a whole bunch of different files so we need a folder for that so please make sure to do it and follow along with me so now that I've created my streams folder I'm going to see the end to it and first I'm going to create a file called uh let's call it write many dot Js and I'm going to open this folder up in my code editor now this file writes many we're going to write some code that is associated with writing something to a file many times right so let's see what we can do now first I want to give you a challenge I know we're just starting but I want you to think about it and see what how you can do this problem so I want you to create a file here in your directory whatever you want to call it it doesn't matter and I want you to go on ahead and write something to that file one million times okay so it could be some anything it could be just numbers from one two three four up until 1 million or you could just simply write one character maybe character a or a character b or anything or maybe a portion of a text a sentence it doesn't matter the point is that you write something to that file you could name it for example test.txt the point is to write something to this file one million times right so I want you to go ahead and see how we can do it you should be able to do it if you have watched my video about Define system you can just open the file and then you could just run a loop and then write something to it a million times and I also want you to Benchmark your code and see how long it takes and the way they can Benchmark your code is using console.time so if you just run Constable time and name for it for example write many and then at the end you should run a console.time and with that exact same name so if I just try to do one operation here say let a equals two plus two and run this code so I'm going to say node write many obviously that this piece of code so from this point to this point this code took this much .034 milliseconds which is a very short amount of time so I want you to write your solution and execute it and Benchmark your code and see how long it takes right so now let me show you how I'm gonna do it so I'm gonna first require the fs module so accounts fsicles require node FS slash promises because my code will look cleaner and right here because I want to use the syntax async weight I'm just going to write an immediately invoke the function like this and I'm going to mark this function as async all right so right here I'm going to first open a file so I'm going to open a file in this case I have created this file already so it's called test.txt I want to open it as right because we're going to write to it we need to write something to it one million times so definitely it's going to be right and I'm going to save it to variables and I'm just going to call it our file handle and now here I'm going to create a loop so for let I equals zero I lesson 1 million so six zeros I plus plus and now I'm going to write something to this file handle so I'm going to say file handle dot right and I could here specify string or buffer so I'm just going to actually uh specify this I variable so I'm going to create an es6 string and I'm just going to inject that in so I super simple now if I run this code I'm going to open this file test.txt and I'm going to write this string to it 1 million times and before I run this code I'm also going to Benchmark the code so right here I'm going to run one console of time and I'm just going to say cancel the time right Mini and then cancel the time and here after our Loop ends dot Time end and I'm also going to mark this as weight because it returns a promise and let's wait for it to resolve all right so now let's oops semicolon here and there we go all right so let's run this code and see how long it takes to finish this process so I'm going to say node right Mini and it's probably going to take some time so I'll wait all right uh so it's done I took eight seconds to run if I take a look at my file you can see that it starts from zero and it should end at one number lower than one million so six ninths and that's the end that's the last number and as you can see here it took eight seconds to complete let's run it a couple of times and see what other results we'll get so I just ran it Forward two more times and as you can see here the results are pretty consistent around nine seconds we're not going to get this exact same result maybe like uh 8.111 and 8.11 the next time it's going to be different the disk speed might be different from time to time and the computational power your CPU speed might be a little bit different from time to time but so but the point is that they are consistent around 8 seconds for something like this to happen right and just a note if you run this code one more time it's just going to write from scratch because we're not appending to that file we're writing from scratch so the content will be overwritten every time you open the contents will be overwritten but here in this for Loop because we'll keep we'll keep writing to it we're not actually overriding anything we're just adding at the end all right so let's try to run this application a couple more times and take a look at how much CPU power and memory usage this application is using so I'm gonna open up my activity monitoring application and let's try to run this application a couple of times and see what's happening now if you're on Mac I encourage you to do the same thing and actually on any other operating systems on Mac you could just open your activity monitor application on Windows you can open task manager which is a similar application to this one well it's a little bit different but it's roughly the same thing you could see the whole list of all your processes and you could I think you could filter and see how much memory and CPU disk Network and all the rest how much all these resources are being used by different processes and if you're on Linux you could make use of the top commands or you could install h-top which is a little more user friendly than top and I assume if you're on Linux you know how to install that so I'll leave that up to you but still you could just use the top commands and to be honest it gives you more information comparing to what activity monitor gives you but this one is more user friendly so I'm going to stick with this one throughout the course at least most of the time so what I'm going to do is to just search all my processes by the one that I care about which is note now if I open any applications using node meaning if I run node I'm going to get a new process here so if I just say node a new process now when there is nothing going on it's using zero percent of the CPU which is expected because we're not really executing anything it's just an idle application it's an idle process and memory usage is around 12 megabytes 11 12 megabytes so let's now run our tiny program and see what will change so I want to say node right and let's first monitor how much CPU we're using so if I run it the process is going to pop up here and you can see that it's around 100 percent now we might be asking why is it 100 here but and the idle it's only around 60 so it means it's 60 like CPU is not being used and only it's here it's saying 100 it's using well the reason is it's because my machine is an eight core machine and each CPU corresponds to 100 of the usage so if I see one c one process and using 100 it means that it's using only one of my cores but to its full power if I see a process that's using 200 it means that it's using two cores but at their fullest power so I see the process happens and I see 100 usage but I still have a huge amount of idle CPU on my machine which is something that you need to care about later on when you're deploying your application we're going to talk about this way more later in the course but the point is it's 100 but it means that it's only using one chord because node.js is single threaded and it's only using one of the cores if we want to use more than one core we have to use clustering or the word for thread module which are the things that we're going to talk about at the end of the course now let's monitor the memory usage so the CPU was 100 only one core if I run my application and I monitor here the memory you could say that it starts at 20 and it goes up to 40. so 50 it hovers around 50 megabytes so it's safe to say that it uses only 15 megabyte 50 megabyte of my memory so we could say here this piece of code it takes eight seconds to run uses 100 off the CPU and I could say one core and memory usage is around 15 megabytes now let me just make it more consistent so I'll say CPU usage 100 meaning One Core and execution time is eight seconds right what's better now let's try to see how much we can make this code faster now if we recall from my previous video we said that uh the Callback versions in node are usually a little bit faster and I mean in the fine system module and that's what the node.js documentation says so let's try to change this code to use the Callback version and see how much faster we can get so I'm going to copy paste this code and I'm just going to comment this first code out this first portion and let's try to do it using the Callback version so instead of using the promise version I'm just going to say f is set open same two parameters are the same but at the end we need to specify a callback function and the Callback the first parameter is our let's try it again because Hornets see this pop up because I don't really remember [Music] actually I'm not getting the pop-up because I'm not including the right module so if I want to use the Callback version we shouldn't include from the promises instead we should include from the regular FS module meaning it means that we need to drop this uh Promises at the end and now we're using the Callback version so if I so the first parameter is going to be uh the error and the second one is just going to be a file descriptor and I think we talked about in the previous video find descriptor is just an ID it's just a number that is associated with an opened file and it's Unique so each open file has a unique file descriptor which is just a number and to write to it we could use the function called fs.rite which accepts the file descriptor which is FD which is right here and then we need to specify a buffer or it could be a string so I'm going to move my for Loop here inside and I'm just going to use the exact same thing that I did previously in that promise version and this one accepts a callback but you can imagine that it's not really going to work because then we need to call the next function inside of that callback and because we're running it a million times it doesn't really make sense to do something like that and we don't really actually need to do anything in the Callback anyway so I'm going to omit the Callback and change the syntax to use the synchronous version so I'm going to say write sync and let's move our console of time ends after the loop and we actually don't need this piece but let's keep it there because it doesn't really matter because let's try to make them as close to each other as possible [Music] now let's run this code and see how long this one will take so I'm going to delete this test test.txt to just to be extra sure that something is actually happening so now I'm going to run the code foreign took like two seconds which is a lot faster than the previous version now to confirm that we're getting the exact same file I'm going to open this one up and it starts at zero and it should end at 1 million and that's right six ninths so we're getting the exact same file and it's it's actually executing 10 times faster at 10 times but maybe around five times or six times so it's a lot faster but we actually only use the Callback version Let me just show you what will happen if we use the Callback version so we're not executing anything at the ends but we still need to keep it right now if I run it well it roughly actually it didn't take 1.4 seconds because you saw a little pause because at the end after the for Loop finishes it's just going to lock this and we still have some callback functions in our event Loop so what if we reach the end the event Loop is still working now if we open this file you're going to notice that we're getting all the numbers I mean if we get the file size of this one and compare it to the previous one it's the exact same size but as you notice here numbers are not in order so here I have the last number but right before that one I have 90. and if I run it another time it will be different for example here I might have 97 and then here I might have 98 and then here 99 and the point is they're not in order and it's because we're pushing so many different events to the event Loop and they're not really actually getting executed at the same time so we can't really be sure that the first callback that we're going to push is going to be executed faster than the Callback version that's going to be pushed after this one so there's no guarantee in that now we can't be sure if the order of numbers which is different from our previous version that all the numbers were in order because we're waiting those promises now what is really interesting here is that if I run this code and if I monitor my application so I'm going to look for my Note and if I just run it let's take a look at him our memory usage certain memory usage is around 500 megabytes 600 700 it really jumped to 700 at one point so let's run it again and take another look here we could see 500 700 and it's it's a lot if you compare this one to the previous one which was only 50 Mech it's it's way more it's like maybe 10 times more so even though it's executing faster it's occupying so much more memory which is again different from the synchronous version if we try to do it using the synchronous way it's not going to occupy like 700 it's going to be like in again just 500. and if I change this one from 1 million to maybe 10 million I don't think so it is going to work let's try this and let me open my activity monitor see my memory jumped to 2 gigabytes now three it's going higher and higher for five I only have eight so I don't think so it's going to go higher than eight seven and let's just wait for a couple more seconds yeah as when it reached eight I now my application crashed and I see I reached Heap limit allocation failed and JavaScript Heap out of memory so we ran out of memory if we take a look at my memory pressure here these Orange Lines an indicate my memory pressure it's that period of time that I ran that application so be careful when you're pushing so many different events on your event Loop now I know it is a very extreme example but these things might happen you might end up in a situation to be putting pushing so much events to the event Loop and you might quickly run out of memory so I'm just going to change it back to the synchronous version and move with that one so I'm going to remove the Callback and I'm also going to change the function name from right to right sync all right now let's run it again so yeah around one and half a seconds now let's take a look at the activity monitor and see how much memory we're using so I'm going to run it again so I say around uh 30 or 40. it's definitely a little bit lower than 50 maybe just slightly lower than the previous version so let's take a look at the CPU usage if I run it should be the same thing as a hundred yeah it's a hundred we're not exactly seeing a hundreds because you know there are some latencies here and there and we are not even sure about the memory if it's like 30 it could be a problem here in my activity monitor because it doesn't get updated as soon as that application uh ends so what we can do is to just say the exact same things but the execution time is now a lot slower around so maybe only one let's just say two seconds just round it up CPU research 100 still one core and memory usage let's just say 50 megabytes doesn't really matter that much now let me also try to make this code a bit more explicit because I don't want you to think that we're writing a string to that file what we're doing here is that we're converting the string back to a buffer and then we're going to write it so what we can do is to create a new buffer here and then keep writing that buffer to that file so let's let me actually just run it a couple more times and just Benchmark it because I positively for a second and run it a couple times and what I noticed was that the results are pretty consistent and they're around 1.6 now here it got to 1.8 but there we could it's safe to say that this code runs at 1.6 seconds so I'm going to change it to 1.6 right but now uh to make it more explicit what we can do here is to either specify a character encoding and that means that converts this string back to version ones using that character encoding or I could just Define a buffer here so I say const buff equals buffer Dot from this string and then the character encoding which is by default utf-8 but nonetheless I'm going to specify it and then I just need to pass that buffer in here if I run this it should still be the exact same thing not the exact same thing it will be a little bit longer like 1.8 as you can see here and the reason is because now we're defining a new const variable every single time in each Loop run whereas before we would just pass the string and node.js itself would convert it into a buffer so the previous code is a bit faster because there we don't Define a new variable every single time but not something that we really care about I mean if we were to specify this here at the top and don't create our buffer every single time using node and just let's just say maybe a and pass this buffer here so we're not creating our buffer every single time in our Loop this code will now be slightly faster even to the previous example where we were specifying a string here so it's around 1.5 1.6 you can see that these are uh faster than this one which was 1.6 I didn't really get to 1.5 but here it did so if you don't Define so if you don't create buffers every single time it will be slightly faster but again these are not things that we really care about all I want to say is that I want you to know that we are writing a buffer to this file and not writing a string so let's keep it like this one and let's just also change it to 1.8 all right so we're creating our buffer and then we're writing it to our file and we're getting the exact same result nothing is really different in terms of the file that we're going to get exact same size everything is just the same thing and they're all in order all right now at this point do you think it's possible to make this code even faster what do you think do you think it's possible to maybe make it twice as fast or make it three times faster or I don't know just make it faster in any way that we can not maybe buy a slight amount but by a significant amount so I mean make it twice as fast or something like that well the answer is enough freely without using streams but most definitely yes if we want to use streams so without streams we can't really make this code any faster but what streams we can really make it a lot faster and we're going to see that in just a second so what I'm doing right now is to write the code to do the exact same thing but this time around using streams so we're not only going to talk about what streams are at this point I'm just going to show you an example and then we're quickly going to jump into the concepts and really understand what streams are so I'm going to copy this code out to the first solution using the promises version and I'm also going to comment the second solution out so I'm going to paste the code in and uncomment it actually these three lines need to be commented but now we don't have the time CPU usage is still going to be 100 so I'm going to keep it that but the memory usage we're not sure so let's see how we can do the same thing right one million times to this file but now using streams so the way you can do it is to First create a right stream of this file so let's do that so I'm going to say const stream or a write stream we have different kinds of streams but we'll get to them later on for now just know that it is a stream so stream equals file handle dot create string notice that we also have another function which is called create read stream you will use this one if you want to read and you would use this one create right string if you want to write in this case we want to write so I use the create write stream and now that we have the stream in place all we can do is just write to it all right so I understand that you don't know what a stream is at this point but let's see first how to use it so the way you can do that is just say stream dot right and here we have to specify our chunk what we want to write to the stream now our chunk could be just like before when we were writing to our file it could be a buffer or it could be a string so in this case I'm going to use a buffer just like before so I'm going to create the buffer like here where is it it's here I'm going to copy this code and paste it right here so we have now our buffer I'm just now going to write it to our stream and I'm going to remove this last line all right so now we're writing to that file but this time around we're using streams so let's run this code and see how long it'll take to execute and how much is the memory usage so I'm going to delete the file first and let's first make sure that it actually works as expected so I'm going to run the code and well it was quite fast but let's take a look at our file and see if we're actually getting the same file and well the answer is yes we're getting all those numbers starting from zero going all the way up to 1 million so we are creating that same file and well it's way faster it's only 200 milliseconds it's not seconds it's not two seconds like here just or eight seconds it's way faster than both of these Solutions way way faster I mean this one was 1.8 seconds and this one is just like a fourth of a second and the results are actually pretty consistent all around 270 milliseconds so we could say that the execution time is 270 milliseconds like this now what about the memory usage let's write and take a look at the memory usage so I'm going to open the activity monitor and this is going to be pretty fast execution so I hope we can see something oh actually we can't because my frequency right now is like where is it if I go to View and update frequency it's very often one second but that application is running only 200 and 200 milliseconds so I can't really get that process here but what we can do is to increase that number or maybe use the top commands foreign create a new window and just run top and I don't know if we can really uh filter but if I run let's see what we can get well actually view it here at my activity monitor and for just a second you can see that the memory is around 150 megabytes which is a lot I don't know if I'm getting it here but actually it looks like we got it here in the activity monitor so I don't care about top right now but if I keep writing executing it we could see that for just a second it appeared and was like 200 so let's just say 200 megabytes which is which is a lot more than this one now I really want to say something that this code here is not a good practice I'm just going to say it right here don't do it this way okay and this is not a good way to work with streams and one of the problem that you can see is our memory usage it jumped to 200 megabyte which is which is a lot if I just add one number to this one and make it to 10 million and run it right now you can see that my memory usage is now one it's around around two gigabytes which is a lot so don't try to run this code in production and we have to take care of some of the things with that stream to make it way more memory efficient but you can see that it's way faster than the previous examples way way faster around maybe 10 times or nine times faster which is a lot which is a huge deal when you're creating applications if we can make your code twice as fast or even 20 faster that's that's a lot let alone making it 10 times or nine times faster it's it's a huge deal so let's now try to really understand what streams are and then we're going to get back here and fix our problem with memory usage and make it both super fast and also super memory efficient so let's jump right in and see what streams exactly are well let's first take a look at the definition of streams in the node.js documentation now no just says that stream is an abstract interface for working with streaming data in node.js and that's it that's what streams are it's something that helps us with data streaming it makes data streaming way more efficient now we might be asking what exactly is this streaming data when we see streaming things like video streaming game streaming Netflix or in General Media streaming might come into mind but it's a lot more than that whenever we have data flowing we can say that we have data streaming the definition of stream in the English language is just a continuous flow of something it also means a narrow river which is also relatable but here in computer science it's closer to that continuous flow so if I say for example a stream of people is coming to the school maybe for the graduation party I don't mean that 500 people are coming in through the door at one Chunk in one second no I mean that people are coming in gradually for example five people will come in and then another minute 10 more people another second two more people and over time they will gather and then we're gonna have a total number of maybe 1 000 people in the school so that's the same Concepts and computer science data streaming just means that you don't move the whole data in just in one chunk you move it over time in different chunks so you break it down to different chunks and then you send those chunks over and then at the end you just combine them together or maybe not the point is that you're sending it in different chunks all right so let's take a look at one example let's say you want to do a copy paste so you'll have a file on your computer on your laptop maybe that's 10 gigabytes in size and you want to copy this file and paste it somewhere else maybe on the same laptop or on another disk on your server on a USB drive it doesn't matter you just want to do this action of copy pasting now what you could do and by that I mean your operating system or the program that's supposed to do this copy pasting it could just move this whole file in your memory first so your memory is going to be occupied for like 10 gigabytes and then it's going to do only one right and create this one file so that's one option of doing it it's like moving 500 people through the door all at the same time in just one second now you might be saying well that's it's really fast we don't we're just going to get all the people in one second rather than waiting for like maybe five minutes or one minute or two but the problem is that you also need to build this huge door that will allow 500 people to come in all at the same time which is really not ideal in a lot of cases and it's really not efficient to build that huge door just to be able to move 500 people maybe just for a graduation party or something like that it's not something that's really ideal in our world and the same is true in computer science it's not ideal to move this whole file in your memory and then do one right although that's possible and you could do it it's not really ideal let's say that somebody is on their mobile phone and they only have like one gigabytes of RAM and they want to do a copy paste of a file that's like 500 megabytes if if they do that they're going to occupy half of their memory which is already full in a lot of cases and that's really going to end up in a lot of different performance issues and it's not really something ideal you don't want to do something like that instead what you need to do in your operating system or the application you're creating you need to set up a stream of data between these two points so in our case of doing this copy paste rather than just moving this whole file into memory and then do just one right you're gonna just send different chunks maybe send them in chunk of 16 kilobyte 20 kilobytes one megabyte or something like that now yeah you are technically doing more rights but it's way more memory efficient and also in terms of speed it's close to that previous example it's not really as fast as that one but it's really close now in the case of note and our previous example of writing many times to a file what we did was that we wrote one million times to that poor file but after using a stream reduced that number so using a stream if we set up a stream we wait for the data to gather and get big enough to be equal to the size of our chunk and then we write and the size of our chunk by default is 16 kilobytes so that's what streams use in node.js by default so we wait for the data to gather and I mean each time we write we're writing to that buffer and then if it gets big enough then we're going to do a write and we're going to get into this concept just a bit later but this is how it's roughly done we set up a stream and then we keep writing to it so keep waiting for the data to become available and then we do a write now writing and reading from files isn't the only place where node.js is using streams a couple other examples would be for example here imagine that you have your node process and also another process called ffmpack which is an application you can use for editing files actually editing video files like cutting them out or adding a text just usually normal video editing operations so let's say that you want to make your node.js process to talk to this one you're creating an online video editing application and you're constantly receiving the data for example cut from this portion or change text here or change the effect here or things like that so keep getting those data and what you need to do is to set up a stream between your process and ffmpeg and keep sending those data in chunk this is going to be way more efficient than you know sending all these in one go like in our previous example that we did one million rights and then we managed to reduce it down to way less number of rights which again we're going to get into that later but you can imagine that this is another example where using streams will make sense we have a flow of data we keep sending information to this FFM pack and this is a very good example of using streams now you could also use many different screens in your node application for example you could have another one that will talk to the network cards so when you as you receive requests from the clients you read those requests off of your network cards and then you do processing on them or you could also send responses back to the network card to go to the outside internet so this is another example we're using streams is extremely efficient and in fact if you're using the HTTP module or the neck module or pretty much if you have created any network applications in nodes you have used streams now those packages are using them behind the scenes for you but that's what is really happening if there is a stream there and it's keep reading from the responses excuse me reading from the requests and writing to the responses could also have another stream to your storage so you would receive the files for example MP4 files and you save them to your storage using a right stream and then using another right stream you talk to ffmpack to say all right go ahead grab that file and do these operations once done F of the pack might use another stream to send you the information to say that I'm done and here is a new data now you would receive those data in a read stream rather than write stream and then you would just send a response back to the client using a right stream because you're writing to the network card so these are just some examples where node is heavily using screams and this is making your node applications way faster now let's talk about the different types of streams that we have available now mainly we have two streams we have a writable stream and a readable stream now a writable stream would be a stream that you could write to like that right million times example that we wrote we were writing to a file so we used a writable stream now a readable stream is a stream that you would read from so you can imagine that you might want to for example open a file that's very huge like 10 gigabytes and you want to read from it and then using another write stream you want to write that to another file so that's where you would use a readable stream now in our previous if I go back to the previous slide here you could see that this is a rattle stream because we're writing to FFM pack now there is another one from ffmpack to node which I haven't actually drawn here but you can imagine another stream from FF Mac to node but that is a readable stream because notice is reading from that now this one from node to network card again we could have both a readable stream and a writable stream if you're reading from the requests we're using a readable stream if we're writing to the responses we're using a writable Stream So if we send requests back to the client we are writing and if we are reading the requests coming from the clients we are reading now in terms of our storage because we're just saving those files on our hard drive it will be just one writable stream or multiple write streams because we could be handling many requests at the same time so we will be writing to different files simultaneously but yeah these two are the fundamental types of streams in node we also have two more streams and one of them is duplex which is just a screen that's both writable and readable we also have a transform stream which is just like duplex but it also transforms the data and we're gonna get into these two later if you understand the writable stream anywhere you will stream understanding these two will be really easy so whenever we have a flow of data streams are an incredible option to use so this flow of data could be between nodes and another process or node and an underlying resource like Hardware or it could just be really any kind of data flowing and why data would just mean zeros and ones all right so for example let's say that you have this piece of data and you want to encrypt it so we want to do some operations and make this data encrypted so that nobody could access it unless they have access to secret code now what you could do is to create a map and say whenever I have a double zero and a double one I'm just going to change it to a triple one and a triple zero so we're going to look for in the data and see where we have these patterns and just to note these encryptions and compressions and things like these are actually working with the raw binary data what these errors and ones in most cases so what you would do is to just find those keys and then just do a replace and end up with something that's really not meaningful to somebody that doesn't understand this map so this part will be something gibberish if someone takes this data and runs it through a capture encoding they're going to end up with some gibberish because we have done a replays here now if you give this to the right user and now they want to decrypt it what they could do is to just use that map and do another replace I mean go backwards and then get the original data now in real world encryption applications it's actually more complex than this there are a lot of prime numbers involved a lot of discrete math so it's not just doing a replace on one portion of the code and then leaving the rest as they are it's just more like doing an encryption on all of the data using some mathematical formulas so it's more complex and also way harder to crack but this is also a valid example of an encryption just have to make sure to keep an eye out on those keys so that you know that this is an actual key and you have to do the replace and it's not part of the data but anyway now this is an example of data flowing imagine you want to encrypt a huge message that's hundreds of megabytes or maybe a couple gigabytes or even more it's not really memory efficient to just move all that data into your memory and then do your encryption what you would do instead is to just try to get different chunks out of that original message be it a file or a response it doesn't matter just be it any kind of data it doesn't even have to be a string just some raw binary data so you want to get different chunks out of this file and then do these operations and make it encrypted and then write it to a destination and this is a good example of using a transform stream so just read the data transform it and write it to a destination another example would be if you want to compress a data now in this case it would kind of go the other way so you keep looking for patterns you keep looking for repetitions and you try to condense them down to something smaller for example we want to have a map like this and say whenever I have this many ones and double zero and then this many ones I'm just going to condense it to zero one zero one zero one and then you're going to look for those patterns which we have three here in the binary data and then you're going to do your replace so what's going to happen is that you're going to end up with a file that's much smaller in size comparing to your original file and then using this map you could just get the original file now you might compress some data and the file size might not really change because there might not be that much repetition or that many patterns to condense them down to something else but some cases especially if there's a ton of manual repetition you could really reduce the font size by so much but again it really depends on the data that you're working with but here could be another example of using a transform stream the point is whenever you have this data flowing you could use streams you could use the readable stream to just read you can use the writable stream to just write duplex if you want to do both and transform if you want to modify the data in any way all right so now let's understand what happens inside of a stream object how do we know what streams are let's now see how they work more specifically right now let's talk about a writable stream so let's say that this is our writable stream object and we can get this by running something like fs.create right stream something that we did in our previous example we wrote that code and then we got a stream object and this is our object this is our rival stream now inside of this writable stream object we have an internal buffer and its size is 16 384 bytes by default which is specified by the high Watermark value and we're going to see that in action later in the code but this is the default size and you can change it but this is by default how much of a buffer will be allocated to each stream and note now apart from this internal buffer each stream object each writable stream object also has some events properties and methods available on it now one of the most important methods is this stream.write methods now whenever you call this function this method what's going to happen is that you're going to push some amount of data into your internal buffer let's say you try to write a buffer that's of size 300 bytes now this data here is should be a buffer now you can also specify a string but notice later on we'll use the character encoding specified to convert this data this string to a buffer and then do the right so this data think of it just like a buffer although you could specify a string but think of it like that you could only specify a buffer just to make it easy for you to understand how this works so you push some amount of data into this internal buffer and you keep pushing so you push another chunk another one another one another one and you keep going until your buffer is completely filled so now that we have our buffer completely filled what the string will do in this case is that it will do a right so it will now pull all of this data out and one chunk and do one right this is how a rideable stream works endnote now the reason that it's so efficient one of the reasons actually is if you take a look we wrote eight times so if you count these they are eight different chunks so we wrote to that stream object eight times now in our example we have that Loop and we kept writing these numbers to our file so we wrote One actually space a number and then a space so let's say if that string is like five characters so five characters each character is eight bits so five times eight is 40 bits way smaller than this 16 kilobytes that is the size of this buffer so we kept pushing those small data to that screen so in this example we're on a times but the important Point here is that we only write once so we write to our stream eight times but we only write once now when you write your writing to an underlying resource like a file for example or your network card now this is way more efficient because you're writing now and your memory and your process in your node.js process and then once the data is filled you're just going to do a write now in our example of that for Loop and writing those numbers now without streams we're writing one million times to our underlying resource but if we use streams will happen is that we will keep Gathering the data and then do one right so we will significantly decrease the number of rights that we do into that file which is way more efficient now you might be asking what will happen if we try to write to the stream an amount of data that's way bigger than the internal buffer and not just way bigger just slightly bigger let's say that here our last right was 277 bytes now what will happen if we try to write a data that's maybe like 3 000 bytes for our last right to completely fill this buffer and what will happen is that node will push this amount of bytes to our internal buffer and then buffer the rest of the data which is going to be around 800 bytes that 800 bytes will be in memory I will keep track of it and once the internal buffer is emptied it's going to push that again to this internal buffer now if you think about it in this case we're going to have a problem what if you keep writing to this internal buffer without letting it to get emptied now what's going to happen is that no just will keep buffering all of this data and by buffering I just want to say this one more time a buffer is a location and memory that holds a specific amount of data so if you create a buffer that's 400 megabyte in size you are quickly filling your memory by 400 megabytes so when we have a buffer whatever its sizes you have allocated that much size off of your memory all right so if you keep buffering data it's going to be a problem because you're going to run into some memory issues for example if you try to write a data that's 800 megabyte in size what will happen is that well first of all the buffer will get filled and we're going to have this 16 kilobytes in our internal buffer but notice now we'll keep the rest of the data in memory right which is going to be a huge problem and in this case where our memory usage will jump up to 800 megabytes which is not something that we want we're using streams to try to lower this amount as much as we possibly can but if we try to do this if we try to run this code if we try to say stream.writes and put a data that's very huge we're going to have some memory issues or not just doing one right but you keep writing without letting the buffer get emptied like our previous example we kept writing to that stream but we didn't really let it to get emptied right we just kept writing and now just kept buffering all those incoming data so that we had that huge amount of memory issue I guess it was like 700 megabytes I don't recall but it was a huge amount which is something that we don't want so what you need to do is to First wait for this internal buffer to get emptied and then do another right now this time this process that the stream does to empty this buffer is called training and we have to keep an eye out on this event and it's actually an event so you say arrival stream dot on train and then you do your code we're going to see all that in code but I just want to say that you have to wait for this internal buffer to get emptied and after it's completely empty you are now safe to push more data into this internal buffer or I should say right so this is how a relative stream works endnote it's just an object that has some events properties and methods and also this internal buffer let's now take a look at the readable stream object let's see how this works so again we have our real stream object and we can get this by running something like fs.create read stream and what we're going to end up with is this object now just like a writable stream we have an internal buffer size of which is that high Watermark value which is by default 16 384 bytes or just 16 kilobytes now again just like a rival stream we have some events properties and methods available on this readable stream the way that we can push data into this readable stream is by calling stream.push and then some amount of data so we keep pushing into this internal buffer and what's going to happen is that once it's failed we're going to get an event and the event is called stream it on data so the event's name is data so you need to run this code on your readable stream keep looking for these data events so once internal buffer is full we're going to get an event called data and we're going to get this data all in one chunk here in this callback function so we're going to pull the data out and we're going to do whatever we want with it and this three types section maybe you want to write it to another writable stream or you just want to read it it's up to you so this is a readable stream and you use this real stream to make huge data a huge message into different chunks so let's go back into our previous example here let's say that you have this 800 megabytes of data let's say it's a message coming to your network card maybe it's a request or if it's a file you want to read so what should you do in this case what do you think let's say that I want to write a data to a real stream to it right we'll see excuse me maybe it's a file and this data is so huge what should I do in that case well what you should do is to First create a riddle stream off of this data need a file or a request anything you first create a readable stream and then when you create a readable stream you're going to get the data in chunks of this size this 16 kilobytes 16 kilobytes of size so you get the data and then you write to this rival stream so this is what you should do if you have a huge amount of data and you want to write it somewhere first create a readable stream and then you write it right so this is really how this rival Stream Energy will stream work now what about duplex and transform now they're just like these two a duplex is just a stream that you could both write to and read from so a rival stream and a readable stream they only have one internal buffer only one but a duplex has two a transform has two one for reading and one for writing so these are all the streams we have in a nutshell now let's jump right into our code editor and try to use these streams and let's see some interesting results let's first fix our problem with that memory issue all right let's try to work ourselves through this problem by some of the basic things that we know so first of all I'm going to copy paste this code because we're going to have a new solution to this write many problem and I'm just going to comment this portion of the code out I'm also going to copy this include line it's going to be the same but still all right so this is our stream object that blue rectangle we talked about in the slides so let's try to see what is happening here and let's see how we can fix this memory issue problem now we might be thinking is it possible to get the internal buffer of this stream well it's not really recommended and it's not really documented in the notice documentation so we're not going to do that instead what we're going to do is to use two properties that we have available under stream object that will indicate how much of that internal buffer is failed so I'm going to comment the for Loop out and I'm going to just lock these two properties first one is stream.writable High Watermark value actually right well dot writable High Watermark and this is the size of our internal buffer which is by default 16 kilobytes if I run this code I want to get that value that we talked about in the slides 16 384 bytes now it is possible to change that value but you don't really need to do that in a lot of cases unless you are implementing your own streams which is something that we're going to get into a little bit later in the video all right so let's try to log another value which is stream.writable length now this value indicates how much of this buffer is filled so if I log it I should get zero because I haven't called stream.write until this point so let's try to write and then get that value again so I'm going to say stream dot write let's just write this okay really simple and let's log this rival length after this string and actually you know what I'm not going to write a string to our stream because you are not limited to just writing strings to streams you could write anything any zeros and ones you could write an image a video file anything so what I want to do is just first create a buffer out of the string and then write it in that case I think it will be easier for you to understand what's happening so I'm going to create a buffer first and we'll say buffer Dot from this or actually just string and then I'm going to write this buffer to my right to my stream object let's run this code all right so this lock that is zero we haven't wrote anything yet but after we wrote this buffer to our stream now I will write the length is 6. and if you're wondering to know why is it six it's not because it's six characters well actually yes but it's more like that each character is represented using only one byte so if I just log the buffer so log buff you can see that I have a buffer that has one two three four five six so this Buffer's length is six bytes I write this buffer to my stream to that internal buffer and now this radical length is six if I try to write some more things to this buffer if I try to write the same thing but multiple times if I save and run it again now durable length is 30. so what's happening at this point is that we're just pushing these buffers to that internal buffer to this stream object so we are gathering data right so you can imagine what's happening inside of this Loop we'll keep writing to this buffer and we keep increasing this threadable length value but the important part here is what happens when this rival length gets the same size as the routable high Watermark value so this is going to be 16 kilobyte what will happen at that point so what we need to do is to check and see what this write method returns if it returns true it means that we are safe to write more to our stream it means that our internal buffer is not full yet but if it does return false it means that our internal buffer is now full and we should now wait for the stream to empty that data and do the final write which is to write to our underlying resource which in this case is just a file so what I'm going to do right now is to like first zoom in a little bit and also comment out all the logs that I have so I'm going to comments this for loop again and also this time ends and I'm going to remove all these rights I'm also going to remove our routable length at this point because we know that it's zero I'm going to keep the high Watermark value to remind ourselves what this value is and what I'm going to do right now is to create a buffer that has the same size as this High Watermark value so the way I can do that is to just say buffer.alloc and then specify size whatever size I specify here in bytes now my buffer will have that exact same size so this is how we allocate a buffer how we create a buffer this is one way of doing it and you can also here specify an option an optional parameter which specifies how to fill this buffer so if I just say character a it means that fail all of my buffer with this character or you could specify a number or another buffer or actually nothing so let's just do a quick recap on buffers let's create a buffer that has a size of 100 megabytes and filled with this character a or actually just a number let's just say a number maybe a 1 or 10 actually and I'm going to specify 100 megabytes now if you want to convert between megabytes and bytes or kilobytes to bytes or whatever just do a quick search on Google and that will pop up so just say mb2 bytes for example and here we get this converter on Google so if I want to create a buffer that has a size of 100 megabytes just say 100 here and then I get that number and bytes now if you're not familiar with this notation this is how we specify big numbers and computers it means one times 10 to the power of 8. so you could either specify this in your buffer to get 100 megabytes or you could just specify it with those zeros so you have to specify eight zeros one two three one two three one two now I create a buffer and fill each byte with a 10. let's select this buffer and see how much memory usage we have now if I log it you're going to see that we get these numbers and I'm just going to write these here at the top in case you're confused so 8 Bits [Music] is one byte and each bit is either a one or a zero now one thousand bytes is one kilobytes and one thousand kilobytes is one megabytes all right so if you want to create a buffer with the size of 100 megabytes you have to work yourself through this chart so 100 megabytes means 10 to the power of 2 times 10 to the power of 3 times 10 to the power of 3 which will be 10 to the power of 8. so 10 to the power of 8 and then you're going to get 100 megabytes or you could just use this converter here and get and converge between these values back and forth all right and one more point in this buffer when you like it you're going to get these numbers in hexadecimal and each hexadecimal value equals to four bits so we have this hexadecimal value of let's say 1A each number corresponds to four bits so one two three four one two three four and this is not the exact value if we get the exact value you could use a calculator or do it by hand if you will and you have to switch to a programmer calculator and just put 1A and value is zero zero zero one three zeros and one and then ten ten all right so this is our buffer its size is 100 megabytes and it's filled with this value with 10. so 0a is 10. you could just put a and then switch to digits you're going to get 10. so 8 a and hexadecimal is equivalent to 10 in decimal formats anyway so let's create our buffer and take a look at our memory usage we're logging it and let me just put in a set interval function here at the end so that our process does not exit right after it's done so now I'm going to run the application and I want to take a look at the activity monitor and sure enough if you take a look at the memory usage it's 100 megabytes around 12 megabytes is for the whole application and this buffer is allocating 100 megabytes and you notice that the memory usage jumped back to 16 megabytes and it's just this garbage collector thing that node.js does behind the scenes because we're not using this buffer anymore nodejs will delete that but if I run it one more time you don't even see that the memory usage is now 100 megabytes so whenever you are creating a buffer you are allocating some amount of your memory to that buffer we have a warning here well it says that we need to close the file and it's actually right we're creating a file here but we're not closing it at the end which is bad practice so I'm just going to close the file here at the end once we're done so file handled up close all right all right let's now create a buffer of that size of high Watermark value and then write that to our stream so I'm just going to say specify this value which is should have it in our logs which is 16 000 something so I'm just going to paste it here so I have created a buffer with this size and now let's go ahead and write this buffer to our stream so I'm just going to say stream.write buffer and what will happen at this point is that because our internal buffer is now full because we're creating this buffer that has the same size as that internal buffer this function now will return false so if I just log it say console don't like this I think we still got that warning but we're going to troubleshoot that later so let's run again and now we can see that this uh value this method returns false if I try to create a buffer that has even one byte less than this value so I changed four to three now if I run it this function it right now returns true [Music] so the important Point here is that we need to check every time this function returns false we need to wait for it to get emptied otherwise if we keep pushing to it we are buffering that data and what will happen is that we're going to get some back pressuring so it's something that we're going to talk about it's actually this thing that we're talking about right here in our for Loop if you take a look we'll keep writing to this buffer to this stream this buffer so we keep writing this buffer to our stream even though we reach this High Watermark value and we're not letting the stream to empty itself so we're having some back pressuring here and we're not handling it quite well we're putting a huge amount of pressure on this stream on that internal buffer inside of the Stream so how could we let the stream empty itself now what we can do is to just listen for an event called the drain drain and whenever this event happens I can't type right whenever this event happens it means that the buffer is now emptied that internal buffer is now emptied and now we're safe to write more to that stream if we want so let's say that we write this to our stream and then let's write one more thing to our stream and in this case I'm gonna allocate a buffer has a size of one byte so now we get to this high water mark value I'm just going to fill it in this case which is character a and let's write this one so this first right call should return true but the second one should return false because now we reach this value this high water mark value so if I run it I'll get true and then false but because it's now false the stream will now take some time to empty itself and once it's done it's going to emit this string event if I just like something we are now safe to write more we run it we're gonna get uh actually we haven't got that log actually we're not getting this log because we are closing our file right after this code so we reach this point and then we quickly close it and that's it we're not getting to this point and when we close this file handle we are actually also closing our stream so let's comment this line again and we're gonna get back to this close afterwards because we have to talk about some of the other events like finish and and so forth but let's now focus on the strain events if I run this again now I'll get that log that says we are now safe to write more so our first right returns true second one returns false because at this point we completely fill that internal buffer but now the stream will emit this train event if I don't do this if I don't completely fail my internal buffer and if I run this I'm not going to get that log because well at this point my internal buffer is not full so we don't need to wait for the train event so we should always wait for this train event if you don't and if you just keep writing to that internal buffer and if I log the stream Dot right length rival length this value should never exceed this value so this is our high water mark value and this is our writable length value and as you can see this is bigger than this one you should never let this happen whenever this one returns false you should immediately stop writing to it and instead wait for the strain events otherwise you're going to buffer this data and we saw what happens when we keep buffering something if you just buffer something if you just create a buffer you're just immediately going to allocate some amount of memory so you should never let this value get higher than this value that is a bad practice that's an awful practice and you're going to have some memory issues some performance issues so do not ever let this happen always check for this value and always always most importantly check for what this stream.write method returns if it returns false you should just stop writing to it anymore wait for the strain event to happen and once the event drain happens it means that now you could move on ahead and write more to that stream so I just at this point write a couple more things to my stream so I just write one byte to that stream and if I just log this random length value again now what I'm going to see is I want to get one because now we emptied our buffer so rival length which indicates how much of our buffer is full is now one because at this point we have completely emptied the buffer we have route to this test.txt file and now we could just move on ahead and write more to it we don't see anything here because this number it doesn't mean that we're writing 10 if it was a character it would mean the same thing if this this is confusing you got to learn more about buffers watch my video about buffers but just to make it easier for you to understand I'm going to change it to just an eight I have to scroll all the way down to get those Haze and these A's are these ones but if I just change 10 to a save run a code and open the swap I want to see a whole bunch of A's all right so let's recap this train event happens when our internal buffer is full and when it's emptied all right so if we do something silly if we try to allocate a buffer if we try to write to the stream a value that has exact same size we're going to get into some issues because in this case we're writing again to our stream but we're making it fully so this code will eventually lead to the stream object to emit this strain event again and then we're going to get into this callback and it keeps happening in a loop so if I run this we're going to catch a whole bunch of locks because this code keeps happening over and over again the stream Waits For That internal buffer to be completely emptied and once it's done with that it's going to emit The Strain events we're catching this drain event here and stream it on drain and then whenever it happens we're executing this callback but in the Callback we're writing again to that Stream So this code will again emit another train event and this callback function will keep happening and happening over and over so with this knowledge in place now that you understand how draining works that you should never write to a stream once it's full and you could find it out when streamline returns false let's now go back to our Loop and fix our problem so I'm just going to first remove all these codes or actually I'm just going to keep them for your reference later on now let's now work in the for Loop so I'm going to uncomment the for Loop outs and let's see what we can do here so what we need to do is to check for this value if it returns false it means that we need to stop the loop and then lesson for the drain events and then after the train event happens in the Callback function we need to do something to continue on with this Loop so if you want pause the video right now and give it a shot it's now all about your JavaScript and programming skills you need to stop the loop and then resume it if it's true actually you need to resume it and that drain event but anyway you get the idea see if we can do it and if you can't it's fine I'm going to show you my solution in just a minute all right so what I'm going to do first is to well if we want to check for this value so we need to have an if statement and if this if statement is false actually if this expression here returns false so what we need to do is to just break out of this Loop and then in that drain event do some stuff and somewhat somehow resume this loop again so the way we can do it is to move this code inside of a function and also move this I outside of the scope of that function so in that case in the drain events we could just call that function and because we have our I value outside of the scope of that function we could resume the loop so we could create a new function here called Write many and hopefully I'm not defining any other things with this variable well nope we're good so this function we'll have this for Loop inside of it and we also need to Define our I variable here outside of the scope of this function so let I equal zero and ILS then a million I plus plus and you know what because we are defining this I variable outside we could just change this for a loop to a while loop logic will not be different at all it's just instead of for loop it's now a while loop and now what we need to do is to increment our I value here after we do a write all right and also I'm going to call this function right after it's defined so write many and now we're good now this code is just like before nothing is really different the only two things are different is that we now have our I variable here outside of this function and we also have this function which we're calling right after it's defined now in this case what we can do is just break out of this while loop if this returns false and then in the drain listener which we're going to Define right here after this right mini we could just call this function again and what's going to happen is that we already have our I variable at that number that was previously so we'll just continue with the loop rather than just starting from scratch and start from I zero and then go all the way up so let's now Define our train events so I want to say stream dot on drain so notice when we get to this point it means that our internal buffer in the Stream is now full so we need to stop writing and here we need to wait for the internal buffer to get emptied so by the time that we get to the Callback and we are executing the Callback we are now sure that our internal buffer is now empty so what we can do is to just call that write many function again like so and now we need to do our if statement check so if stream.write buff if this returns false so now if this expression returns false we're just going to break and that's it or because it's one line I could just remove these curl brackets and do it in one line so this should now be our solution and it should work pretty okay so let's now run our code and see if it works so I'm just going to delete this text.txt file and now I'm going to run this code so node right Mini and now we have our test file which starts from zero and goes all the way up to six ninths here so our code is working pretty okay and if we take a look at our memory usage let's try to run it a couple times it's pretty fast so we can't get the process well we did it for a couple seconds but the Mario usage is around maybe 50 megabytes right so it's not 200 it's something that we saw previously it's definitely not 200 because now we're doing a drain but in our previous example here memory usage was 200 megabytes it was it was really awful you should never do this so we managed to reduce this number down maybe by four times and that's also Benchmark to code and see how long it takes notice we can't do this right here because if we do it's going to run sooner than some of these callback functions so a good place that we can run this time end function is when our stream is done so when we're done writing to this file we want to emit an event and say that all right we're done writing that's it this is going to be our last comac function that we're going to write and there it will be a good place to both call the time and function and also close our file handle so the way we can do that is to check for this I variable and say if it's equal to the last number which is going to be six ninths we're just going to end at the Stream so let's do that here so I want to say if I is equal to and notice we can't just say I equals to 1 million because we're not ever going to get to this point because if I becomes 1 million we're just going to break out of the loop what we can do is to just say this variable but -1 so six ninths so if you subtract one out of one million you're gonna get this variable so if we're in our last right let me just put a comment here this is our last right and I'm also going to say F stream dot right turns false Stop the Loop okay and for this one I'm gonna write resume our Loop once our internal buffer is it's our stream's internal buffer is empty all right so now let's do our last right which is going to be stream Dot and this is another method that we have available on the stream objects so using right you could just write but when you call the and Method you're going to indicate that this is going to be my last write so you can also specify a chunk which I'm going to so I'm going to specify that buffer and this is pretty much it now if we run this method on a stream it means that we're done if we try to write any more to that stream like this this is going to return an error so when you run the and Method it means that you're done with the stream and you should no longer write any more to that stream now this ends method is also going to emit a finish event which is something really useful that we can sometimes use for example in this case we could in that finish event we could run our time and function and also close our file handle so let's listen for that event so stream it on finish then here we're going to move our time end function and also close the file handle all right now let's try this code and see how long it takes to run so I'm going to run the code and well looks like we have an error let's see what the error is right after ends hmm well yeah because we need to return this because if it's I is this then we are calling this and then we're moving to this right so we are getting this error that says you should not write after and which is something that we just talked about right so just return and that's it I think this should be pretty much it let's run it again and yeah it's right so this function now takes around 300 milliseconds to run which is the same number I guess yeah it's slightly the same thing maybe this one is a little bit faster but we can't really say that much faster it's just by maybe two percent or something like that so not something that we really need to care about now this code might be slightly slower but notice that it's because our memory usage is not it's not that big so if you try to maybe instead of a mail you try to write 100 millions or maybe a billion I'm sure that this execution time is going to be slower than this one because then you are having a huge memory issue but also the most important Point here is that our memory usage is now 200 megabytes but here in this solution it's roughly around 50 megabytes and this is going to be more significance if you increase this I variable so you could now use maybe a billion 10 billion and you're still going to get a pretty good memory usage somewhere around maybe 50 megabytes or 100 megabytes really something really much much lower than our previous code now let's see how many times we are training so I'm just going to add a log here and say training and let's count and see how many drains we get what do you think the value will be so let's run the code and we're getting a lot of logs if I scroll all the way up so let's try to count them and the way we can do that is to just one way we can do is just select all these texts and do a copy and then paste it in our code editor and now here we could take a look at the line number which is 481 so it means that we drained our internal buffer our stream 400 times do you think this number makes sense well the answer is yes and you can get this number by doing a little bit of math so let's get the file size of our test and see how many bikes we wrote to it so it's seven megabytes but we don't care about that we want the exact size so I'm gonna right click on it and click on get info and right here you can see exactly how many bytes this file has or this file size button bytes so I'm going to copy this number and I'm going to open up my calculator and I'm going to switch to basic mode so view basic and I'm going to paste that number in so this is exactly how many bytes we wrote to this test file right now if we divide this number by that high Watermark value by the size of the internal buffer we should get to this number 481. so let's get that value I don't remember okay if you ever have it's in our terminal so it's 16 300 and something so I'm just going to copy this and paste it in my calculator and now I'm going to hit the equal sign to see the result of this Division and sure enough it's 481. notice that we also have a remainder which is expected so in our last right we are writing something to our file but that chunk that we're writing is smaller than the size of the internal buffer so we're not doing another drain so we are writing 482 times but we are draining 481 times because in the last time we don't need to drain because the value is less than the internal buffer and you know what this could be kind of confusing because we're logging training but when we get to this point we are done draining so it should be drained right because we're doing training before this event happens so right before this event happens the buffer is getting emptied but once this event happens and we get to this callback function at this point we are done draining so now we have our internal buffer completely emptied so it should be drained and then we could just do our rights again if we try to like stream that radical length here we should get zero so we just try to do it right here it should be zero but right before this event it should be 16 000 something size of that internal buffer or else it could be slightly higher than that because maybe our last right well our last ride that will fill the internal buffer it could have a value that's slightly bigger than the amount of space that we have available in data Channel buffer so it could be just slightly bigger but it's not something that we really care about because it's just a couple of bikes nothing really too crazy all right so at this point we are pretty much done with the rideable string in note and we covered some of the most important things that you need to know about a rival stream like the train events the end events right and that high Watermark value let's now take a look at the node.js documentation and try to do a quick recap there and then we're going to move on to discussing the readable streams so I'm going to open up my browser and just search for node.js and I'm going to go into the docs and also notice that now we are using node.js version 18. when we were starting out I installed version 16. but now at this point nodejs has pushed another version out version 18. so I also switched to node version 18 which is something that I recommend you to do as well now if you follow it along with the course you installed NVM and if you have NVM installed it's pretty easy to change your node version now that we are on version 18 if you want to install another version you just say NVM install 18 point whatever now in your case if you're in version 16 or something just run this command and we have installed 18 points 12.1 at this point this is the latest LTS version but go to the node.js docs to see what's the latest version of node for you and after you install the latest version you just run MVM Alias defaults 18 points that version that you just installed and then your default node version will change to that one because it's something that I just wanted to point out if you're on version 16 it's still fine I guess but this is how we could change versions and really it's not that difficult so in the node.js documentation I'm going to open up the section that says stream and here you could see that we have one API for stream consumers and one API for if I scroll down for stream implementers right now we're discussing the first one stream consumers but later in the video we're going to move on to the second API as well all right so in the API for stream consumers you notice that the first code that node.js it gives you is an example using the HTTP module now I just want to say this right now in this video we're just going to stick with the fs module but the concept is just the same thing so if you understand how to work with streams in the file system module if you then move to http module or the net module or any other modules node that use this stream object you're not going to have any problem whatsoever using those streams for example here if we take a look we have requests on data now this is something related to a real string which is something that we're going to get into in just a minute but this res object is a writable stream so if you take a look we are writing it to this response object and then at the end we are just calling the end method which you could also specify a chunk to specify that this is going to be the last chunk to write to the response object or you could just not specify anything and indicate to say that this response is now done so the same concept the same things that were both inheriting from the same stream module so if you understand streams using file system the rest will be easy now here's right old screams and here are examples where in node this rival stream is being used one example that we looked at specifically in this video is FS write streams but it's also used in the HP module zalab a crypto TCP socket and the child processes standard in and standard outs in the fs module we're reading and writing from files in HTTP and this TCP socket we are reading and writing to our network cards and zilob and crypto we're just working with some data in some cases it could be a file or just something in memory and chart processes and these processes that standard error standard out here we are talking to different processes so you understand the concept using the fs module you are understanding how to use streams in all the other modules now first you could see the list of events that we have available on this stream.writable now there are pretty self-explanatory if you just read them you'll understand what they do here it says this close events is submitted when the stream and any of its underlying resources have file descriptor for example have been closed so if we let me just jump back into my code editor so if we try to close this file handle this stream is going to emit an event so I'm just going to write it pretty quick so stream it on close and I'm just going to say stream was closed and let's just also comment all these code outs you don't have to do this just or just demonstrate how this event works so if I just close that file handle so file handle Dot close now the stream will emit this event and we're going to get into this point so let's run it pretty quick node right Mini and sure enough we get stream OS closed for backup okay so we could just come here and read these different events and they're really really easy to understand at least of most of them I got lost where were we all right here so yeah this is the close event and also I just want to say always read the description of these events and methods and properties before you use them because it's really hard to memorize them and in fact you don't really have to memorize them because there's so much just come to these documentations and read them and then you'll understand what they do so you could sometimes find some interesting stuff here in the descriptions now here we talked about the train events and this is example that we wrote right writing one million times to something now in this case it's not specifying the writer writer could be a stream that's off of a file or it could be a response object or a request objects or any other thing really so let's see what else we have we have the error event again pretty self-explanatory just read it and you'll understand what they do we have the finished events we talked about it this event will be emitted after we call the stream dot ends function and also all the data has been flushed to the underlying system in our example it means that it wrote to the file itself rather than keeping the data in memory we also have pipe now we're going to talk about pipe and unpipe after we talk about the real streams so let's move on then we have cork now this is if you run this it's going to buffer all the data in memory and it's not going to clear anything or do any final rights meaning writing to those internal resources so sometimes it could be useful here we could see that the primary intent of this method is to accommodate the situation where we're writing very small chunks very rapidly kind of like that for Loop that we did previously so you could make use of this one in some cases like these but I don't know I haven't really used Quark but if you want go ahead play around with it it's not that hard to understand it's actually super simple when you write it when you call this correct method you're just not writing to that underlying resource anymore and you're just buffering all the data in memory and also you gotta make sure to call the uncork function where is it it's right here call this uncork method after you're done we also have destroy which again destroys the stream and then you have some properties like rattle that closed it's true if the closed event has been embedded so we can see that some of them are really easy to understand and you need to actually come and read what these do whenever you want to work with streams Dot and we talked about it in great detail here it says calling streamlined right after calling stream Dot and we'll raise an error which is something that we saw we also have a writable.set default encoding which is really useful if you are just dealing with text like a text file or an incoming request but if you're dealing with other types of data that are not text like an image a video file an application anything that's not text it's not really useful actually it's not useful at all I mean you don't want to run this method on a stream that is dealing with an image file if you do that you're going to mess up your data so only use this method if you are just dealing with text information again Encore which is just like that Quark that quarks means that it keeps buffering all the data if you call this method it will just stop doing that and flush all the data to an underlying resource and do one right and we have another property which is rival.writable which is true if it's safe to call that rights method and that means that if the stream is not destroyed errored or ended you also have an experimental property here which is rival imported which is indicating it returns whether the string was destroyed or errored before the Finish event again easy to understand we'll end it rival correct errored finished this High Watermark value length now this one could be quite interesting rival dot rival D trained and it's true if the streams buffer is now full and the stream will now emit a train event so when we get to that point this property will be true and yeah I guess this is pretty much it and now here we're done with the Rival section and now we have to talk about the real streams so we're pretty much done with the Rival streams now we're going to talk about the readable streams and then we're also going to use these rival streams later in the video and also later in the course this is a pretty important concept and we're going to use this pretty much I think in every video from now on we're going to have an example of streams not in every single one of them but definitely it's going to be highly used from now on in the course all right let's now talk about readable streams so in real world stream we read possibly from a huge file and we get the data in chunks so an arrival stream it will write and in a real stream we read all right so let's see an example let me first clean up here so I'm going to close this right menu file and I'm going to move all these files to a new folder so I'm going to create a new folder and call it write Dash menu and I don't know if you guys have this desktop txt5 or not I guess I was just experimenting so I'm going to delete it and I'm going to move my test.txt file and also my rightmini.js file to this write menu folder and now I'm going to create a new file and actually a new folder and I'm going to call it read Dash big meaning that here now we're going to read from a file so I'm going to create a new file called readback.js and what we're going to do is to read a file that we created using this write many code so we're going to create some huge files using this code and then we're going to start to read them here in this file so I want to First include the fs module so cards of physicals require node Fest slash promises and I'm going to create one of these immediately invoked functions and here I'm going to first create my file handle object so file a handle read to indicate that we're going to read from this file so wait FS Dot open now let's copy this file and paste it here in this read back folder as well so I'm going to paste it and let's open this file first so test.txt the flag should be R because we are reading and we're not writing or appending or anything we're just reading and let's now create our Stream So constant stream equals file handle read dot create read stream all right so now this is a readable stream as opposed to a rival stream that we had before so the way that we can consume data that will come to the stream is by adding an event listener on this string and the event is called a data and once the event happens we're going to execute a callback function and the first parameter the only parameter that this function has is a chunk so we're going to get the data in this chunk so let's just try to log it and see if we are getting these chunks so constant log chunk I keep doing this I need a semicolon here so that my prettier works well and now I'm going to open up my terminal navigate to this read back folder and I'm going to call the and I'm going to run the application chunk and not chunks and as you can see here we are getting different logs so we're getting the data of this file in different chunks so I just add one more log and just say maybe add a divider like this and run it again you're going to see that we're going to execute this function multiple times so rather than getting the whole data the whole 7.9 megabyte data in one chunk which is going to occupy our whole memory we're getting smaller different chunks like this now one thing that's different if you log the length it's not going to be that default 16 kilobytes rather it's 65 kilobytes so if you use this great read stream to create a readable stream of a file the default High Watermark value is going to be 16 65 kilobytes so if you go to the node.js documentation in the fs module if I can find it here file system if you search for create read stream which is here if you take a look at the options you can see that high Watermark is defaulted to 64 kilobytes so here it says unlike the 16 kilobyte which is the default for a stream.readable this one uses a high Watermark which is 64 kilobytes now we could change this by specifying the high Watermark value here so behind Watermark and whatever you specify here is going to be the size of your internal buffer so if I just say maybe 400 bytes and if I run the code again so no read back now you can see that my chunks are coming in length of 400 bytes but I'm going to change it back to 64 times 10 24 which is 64 kilobytes all right so this is how a readable stream works you just create your stream and then you add the data event handler on that stream and then you're going to get the data now a real stream could have three different states when you first create your stream you're not reading anything as soon as you add this data events what's going to happen is that the stream will start to read from that internal resource and then you're going to get the data in chunks if I remove this event then what's going to happen is that the stream will be in a past State and will no longer give me any data and also I have one more event available on the stream which is stream dot on Ant and if I call that the Commack function will be executed when we are done reading that source now let's try to read all the file and then write these chunks to another file now let's see if we are getting the exact same file so let's try to create another file here and I'm going to call it test.txt and using a rival stream I'm going to write these chunks to that destination I'm also going to change the name of this test to source to make it more clear what we're doing so file handle read is going to be source.txt and I'm going to create another one so const file handle right equals the weight fs.open test Dot txt and the flag should be right because we are going to write so the flag should be w and I'm going to create my next stream so I'm going to first name it to stream read and I'm going to create the second stream and call it stream right file handle right dot create write stream and now that we have our rival stream I'm just going to write to it so stream dot right that chunk so let's first see if we are getting the file so our desk.txt file right now is empty our source starts from zero and it goes up to six ninths and the file size should be if I reveal it in my finder file size is 79 7.9 megabytes so let's run this code and I also need to use this here and this should be stream right.right all right so if I run this code well it looks like it ran without an error and if I take a look at my destination file well all is good and let's also check the file sizes to make sure that we're not losing any chunks and no we're not so both of them have the exact same size as you can see here but notice that this is not something that you should do you shouldn't just write now typically hard drives have a higher read speed comparing to a write speed so here we're having some back pressuring we're getting a huge amount of data or not getting a chance to write them so node.js will keep buffering the data just like we saw previously in our for Loop right so our memory usage will be really high using this solution so we need to use that train event and pause and resume reading from the stream now to demonstrate how much this pressure is let's try to create a very big file using this right mini code and try to read that and do the same thing and Benchmark error code so I'm going to change this and actually you know what I'm going to create a variable for this value because we keep changing it so I'm going to say const number of Rights and right now it's 1 million so one six zeros and I'm going to use this here and here and just say negative one so now here we're writing 1 million times but let's change it to 10 million times to get a bigger file so I'm going to run this so nodes which I have to first go into that directory and node write many it will take a while probably a couple seconds yeah I took two seconds but now let me take a look and see how big the file is the file is now 90 megabytes so now let's try to do this again but instead of using that small file and try to use this test.txt file which is quite big so I'm going to copy it and I'm going to paste it and I'm going to remove I tried to open it so now my computer is freezed because it's quite big it's 90 megabytes and if you try to open it what is happening is that you're trying to move it all to your memory and that's why it's not it's going to be stuck so I have to well I guess I had to open my code editor so be careful not to open this test.txt file so now I'm going to delete that source and change the name to source.txt so now we have a quiet Big File here size is 90 megabytes so let's try to do the same thing but let's also take a look at our memory usage so I'm going to open up my activity monitor and search for node here and let's run the code so I'm going to navigate to that folder and run the application well looks like that it was quite successful and was quite fast actually let's see if we are doing the copy correctly I'm going to reveal in finder and yep we are so the reason is actually we're not doing that many rights comparing to that for Loop that the for Loop took a little while here to complete and the reason is we're writing in very small chunks now each chunk is possibly just a couple of bytes maybe five bytes but here our chunks are very big are 64 kilobytes so we're doing a very low amount of Rights but still we are having that memory pressure if I try to create a very big file so this is right now 10 million let's try to add in two more zeros to it and what's the number right now so it's it's a million so it's one billion so let's try to write one billion to that file and get a very big file a gigantic file and try to do this again so I'm going to save and I'm going to do that whole process again and this is now going to take a while to complete so I'm now going to fast forward this video all right it's been a while I have been waiting for like two minutes so the CPU time with this process now two minutes and while we're writing it so many times so it's really going to take a couple minutes to complete but the good thing is now you can really see the power of the streams our my mail research is really consistent at only 50 megabytes so it's pretty good and it's quite fast I'm creating a huge file I don't know how big the file will be but if I go to my desk so far the process has written seven gigabytes of data to my disk so the file is probably going to be like I don't know maybe 10 gigabytes or something around that number so it's going to be quite huge and we're writing one trillion times to a file so you can see how powerful streams are and nothing is really lagging CPU is only at 100 it's not well 100 is a lot but I'm only using one of my cores and everything in my computer is just working as expected my computer is not slowed down my memory usage hasn't jumped up to something really crazy and everything is just working as expected so let's wait for for this process to finish all right so now we're done and you can see that the code took five minutes to complete which I know is a lot but well we're writing a billion times so it's expected it's definitely going to be huge now let's take a look at the file size gonna really be careful not to click on it accidentally and try to open it now we can see that the file size is 10 gigabytes this is a lot of data now let's try to move this file to this read back folder and try to do this process again but using this file so I'm going to change its name to maybe text gigantic so a very huge file and I'm going to delete the source and now move this file over to that read back all right I accidentally clicked on it I guess after I moved it it tried to open it and my code editor just crashed so I had to possibly protector so let's try this out so the source will be text Dash gigantic and this is going to be my destination I'm going to save and let's try this not even sure if it's going to work I think we are going to crash so let's see I'm first going to open up my activities Monitor and switch to the memory Tab and now I'm going to run the code so let's see what is going to happen so I'm going to run it that was quite intense and if you take a look at my memory pressure it jumps all the way up I mean my computer just freezed I couldn't even move my mouse hopefully it didn't tamper with my recording I don't know maybe it did to remove it in the editing later if it did but you can see that it did not work so it just crashed my memory usage really jumped up to one gigabyte and I think it kept going and my computer just freezed but we managed to copy seven gigabytes of the data so definitely not a good option to do it this way so what I'm going to do right now is to use that training and try to optimize this code so here's how we can do it now remember we said that this function will return false if our buffer is full and if it's false you should never ever write to this stream again so if it's false you have to wait for the stream let it drain let it empty the buffer let it do the right and then you're free to write again so I'm going to say if this returns false in my code I'm going to pause the stream so I'm going to say stream read dot pause so if our buffer is full and this stream needs some time to drain itself while it's happening I'm going to pause the read stream so it means that we're not going to get any more data and we're just going to pause and I'm going to add the drain events right here so stream right dot on train now here what we're going to do is to just simply resume the read stream so I'm going to say stream read dot resume and this is another method that we have available on a read Stream So using pause you could change the mode of your read stream to a paused state so it's no longer giving you data it's no longer reading from The Source it's now paused but when you resume it it's going to resume it's going to start from where it was paused and then start to read the data again and give you the chunks all right so now let's try this code and see if we could get the copy successfully so I'm going to save I'm going to delete the destination file and I'm going to run it again I'm also going to monitor it using my activity monitor so let's see if we could do it successfully so I'm going to run it now looks like we're doing the copy if you take a look our memory usage is fairly low and only 30 megabytes and if you go to the disk you can see that we're writing 90 gigabytes so it's pretty much done if it gets to 10 it means that we're done and yep we're done so the process was ended successfully and we managed to copy the file let's also check the file sizes and see if we are getting the right numbers so my destination is 10 gigabytes and the source is also 10 gigabytes so we're getting the exact copy so if you take a look at the bytes you could see that they have the exact number of bytes the end is 400 bytes 411 bytes and here it's also 400 and 11 bytes so we're not losing even one byte and this copy operation and you can see that it was very very fast it only took a couple seconds and also our memory usage was extremely low I think it was only 30 megabytes I'm not sure if it got higher than 30 megabytes but it was at that number way way better than that one gigabyte of memory usage that we got my computer just freeze but now nothing really happens I mean if you try to write this code if you create a file that's like 100 gigabytes in size so instead of writing a trillion times maybe write thousands of trillions of times and then try to do this again you could just let it happen in the background and then continue on with the rest of your day so you could let this application run and try to copy file it's 100 terabytes and then just do your daily things maybe check the news check your emails nothing is going to be slowed down that much so this is pretty good and you can see how streams could be very powerful now here we're just dealing with files but you can imagine that these files could be requests or they could be messages that you're trying to send to another process or they could be something that is coming out of your database so you could really easily deal with huge amounts of data I mean huge really terabytes of data hundreds of gigabytes and nothing is really going to be that bad so actually it's going to be really good yeah that's right your CPU usage will be 100 but that's why that's what you want the point is to keep that memory usage as low as you possibly can now with this in place let's also try to do something a bit fancier so what I'm going to do is to as we're reading these chunks let's try to extract only some values so let's try to only extract maybe prime numbers or even numbers and instead of writing the whole chunk only write those numbers so if you take a look at this text gigantic now I can't open it because my visual studio code is going to crash but if we try to take a look at the contents but using a stream so what we can do is to just try to open this file but just like a stream just like what we did so we need to move one portion at a time into our memory now if we use the cad commands it's just like that it's going to open a file but using a stream so I'm going to say cat attacks gigantic just to take a look at some of the numbers that we have and as you can see my computer is not crashed I'm not slowing down it's just everything is just working as normal and this application is also working for itself in the background so removing one chunk at a time in our memory and then long can get so we're not having any memory issues here whereas if we try to open it using visual studio code it's going to crash because it's going to move it all into your memory and that is not really going to work in this case because the file is huge now I'm going to hit Ctrl C and stop this process and as you can see we have quite a lot of numbers well we have one billion numbers here and we counted until a hundred million around 100 million so what we need to do is just read them just like here one chunk at a time but only extract even numbers only write these even numbers to our destination so we don't care about odd numbers and we're going to dismiss them we only care about even numbers and the way we can find out if numbers even or not you just get the remainder by two if it's zero it's even otherwise it's odd so the total number of numbers that we're going to have in this test file which will only have even numbers is going to be I think 1 billion divided by two because we haven't even number every other number right so if you divide all the numbers by two you should get how many even numbers you should have so let's try this out so I'm going to first well let's first log the chunk and convert it into a string so these chunks are now in zeros and ones so I'm going to first convert them into a string using a character encoding so I'm going to say chunk dot two string and the character encoding that I'm going to use is utf-8 which is by default but I'm going to specify it nonetheless and utf-8 means that each character will be represented using only 8 Bits so each one of these characters maybe a three maybe a space maybe another space maybe a tab or maybe a new line character that will all be represented using only eight bits so eight zeros and ones like this so each character corresponds to this many bits just eight bits so I'm going to first convert it into this and also I'm going to split a by a space because I want to convert each chunk into an array and have each number as its own individual elements so I'm going to say dot split space and the way split works is like this so let me show it to you pretty quick so if I create an array so let array equals actually I first have to create a string equals maybe one two three four five and if I say Str dot splits and you could split by any character that you want if we split by comma or you can get an array one two three four five we also have some spaces here which we need to count for now we could remove these Spaces by something like this so if you say maybe string equals space and then a number and then another couple of space and then end now if you just run the trim function on it like so it's going to remove all the spaces before and after it so we're going to use these two methods to populate our array so I'm going to first split by this white space and let's just log this and see if we are getting the right number so I'm going to say const numbers equals this and just log numbers all right now I'm going to run this code all right so we're going to get all the numbers but let's just hit Ctrl C and exit because we don't need to take a look at all of them but now as you can see it's not really working as expected we've got some of these empty spaces and wait what is this do here so it's not really working as we want it and if you take a look here while all these numbers are correct right but the number that starts at the beginning of each array is not really correct so let's first try to account for these empty strings let's see what what's going to happen if we try to split by two spaces I think yeah now we're getting the numbers correctly because if you open the file we are using two spaces to separate the numbers not one that's my mistake so we should split by two spaces and not one if we take a look at the write many functions we are here we're putting one space and then another space and then our number so the next number will have another space at the beginning and then its whole number and each character so one character and this number is going to be a couple characters and then one character at the end is going to be represented using eight bits or one byte so eight bits is equal to one bytes [Music] right so now we have our numbers but notice again that cell we have some problems here at the beginning of Sony race here we have a zero which is definitely not expected let me also take a look at the first race I'm going to run and quickly exit and nope we couldn't get it but you could see that all the first elements are not correct so we need to somewhat somehow fix these first numbers so if we do that if we fix the first elements of all these arrays or maybe possibly the last elements because we can't take a look here right and we could try to log it by saying numbers so numbers dot length negative one and this way we could take a look at all the numbers so let's also trying to lock them like this so these numbers should be in order if not it means that we have a problem and as you can see we definitely have a problem and it's not really expected okay so this is the one element before the last elements of in of our array and this is our last element right and as you can see all the last elements well this one is correct but most of them are not correct so if we could fix this and somewhat somehow get our first elements and last elements correctly then what we're going to end up with is an array of all the numbers that we need right then all we need to do is to just Loop through each array extract these values convert them to a number and then check whether or not they're even if they are we're going to write them to our stream if not we're just going to dismiss them so let's first try to fix this issue with these starting and ending numbers so let me now jump into my whiteboard and show you what is really happening here all right so we're getting our numbers in this format so we have our first array and our first number is zero goes one two three and all the way to a last number which we could calculate if you wanted to because we have the Phi we have the size of our buffer and we also know that each character is represented using eight bits so we calculate what the last number would be but I'm just going to say something randomly so maybe three thirty two thousand and then 182. but notice that this will not be the case because we're getting the data in chunks so what can we get the first two digits first two characters maybe we could get 32 and then we're not going to get the eight one eight two we're gonna get that in the next chunk right so we're gonna have a problem here if we take a look at the next array now this next array because we didn't get the last three digits in the first array we're gonna get it right here so we're gonna get we're gonna lose some digits and we're gonna get one eight two and then we're gonna move forward with our array and also here we have a problem because we have some Lost characters so the last elements of this array also is going to be maybe 62 000 and then some Lost characters so we're going to have a problem here as well and then next array just like before we're going to have a problem at the beginning and also at the end now we could also not have a problem we could get the last number but completely complete and without any Lost characters but notice that it will rarely happen in a lot of cases we're gonna have we're gonna have some lost digits at the end of the last elements of an array and also at the beginning of the first elements of an array like here and like here now the last array our first element again is going to have some lost digits and we're going to have a problem here but for the last elements we're not going to have a problem right so imagine that we're dealing with that first file that was 7.9 megabytes and our last number was six nights so we were writing one million times now the last array is not going to have any problems right because we're not splitting we're just getting all of the end of the data in just one chunk so our last element is just going to be completely okay just like the first elements of our first array that is completely okay the last elements also of our last array is totally fine we don't have to do anything about it so what we need to do is to somewhat somehow manage these missing characters now one thing we can do is to just get the last element of the chunk that we're dealing with and then append this last part to the first elements of the next chunk that we're going to get and we're going to see how we can do this now to really understand why we can do this and how actually we can do this so let's first try to talk a little bit about binary so keep this in mind all right I'm just going to put it here at the right bottom of my screen now let's say that this is our file that first file was 7.9 megabytes and we were writing one million times so we have a huge amount of data here now if you get the file size it is 7.9 megabytes right and if you get the exact file size if you just right click and click on get info or properties if you're on Windows or any other way that you can get the exact file size now I have it opened up on my Mac right here so it's seven so seven million eight nine two and then six nine three bytes so this is exactly how many bytes we have in our file now if you multiply this number by eight you're gonna get the number of butt and bits so if you do the calculation so seven eight nine two six nine three times eight you're gonna get the number but in bits because eight by one byte is eight bits so the number is roughly around 63 million I'm not going to write all the number but I'm just going to say roughly 63 million million million bits all right so we have around 63 zeros and ones in this file right here in our file we created now let me just say this because I know some of you might just forget it so one bytes is equal to 8 Bits right and also each hexadecimal character is represented using four bits so I'm just going to pour down four zeros now it could be four zeros or four ones or any combination of that but one hexes number is equal to four bits each hexadecimal number is representing four bits now the character encoding that we're using is utf-8 now this 8 here means that we're representing each character using eight bits so utf-8 means each character so one character is represented using only one eight bits so we could have eight zeros and ones now I'm just gonna again put down eight zeros but it could be eight zeros or ones or any combination of that again so also keep this in mind just gonna put it right here all right all right so now the important point to take here is let's let's take a look at one JavaScript array so let's say that this is an hour array we have one two three and that's it so this is our first index which is zero and our second Elements which its index is one and then the last element which is index is two so this is a typical array now a buffer is just like this now a buffer is a unit 8 array it's a subclass of Unit 8 or 80 means that if it is our buffer each element of our buffer holds 8 Bits I'm just going to put down eight zeros so this is our first element of our first of our buffer and then the second element is just like before now in this case I'm just going to say maybe zero zero one or zero zero one zero so let's say that we have a buffer that only holds two elements so it only has 16 bits or two bytes so here we have a two byte array now the chunks that we're getting are 64 kilobytes or 64 000 bytes all right so 64. I don't remember what the number was exactly but I'm just going to say 64 000 bytes so we're getting the chunks and these values but let's say that we have a buffer that is only two bytes right so say we only have this buffer and let's see how this buffer is representing our data so as we said each element of the buffer holds eight bits so this is our first index or our zero index and this is our one index so index one corresponds to the second element index zero corresponds to the first elements now if you try to log the buffer in you're not going to get the values like this and zeros and once for the sake of Simplicity and so that you could easily look at the data they are represented using hexadecimal numbers and we said before that each hexadecimal character represents eight four bits so this four will be zero zero because we have eight zeros and each one of these will be a zero in hexadecimal numbers the first one is quite different so this number if you do it on a calculator on a programming calculator you're going to get one and then two so this is how you will see the buffer if you try to log it onto your screen now you could do these conversions using a programming calculator or just using math if you learn how to do it using math I guess it'll improve your understanding of binary numbers and binary data and all of this stuff but you don't really have to you could just use a programming calculator and the rest will be easy right so this is how we are getting our buffers now I want you to really notice something here we are getting in data in different buffers and these buffers have this much size so we're doing some splittings but we're not doing the splits maybe add here right this does not work we're splitting maybe at here or maybe add here now if we take a look at our array we cannot split at half of one this doesn't work right it just doesn't make sense you cannot say split my array at index one and a half right you can't say that you could just split at concrete numbers maybe you could split at zero you could split at one you could split at two you cannot split at two and a half same is exactly true when you're dealing with buffers get split at index 0 or index one but you cannot split at maybe one and a half so you cannot get one of this elements and one buffer and then another one and another buffer if you convert this into a unit 4 array and make it so that each element of the buffer represents only four bits rather than 8 Bits then you can't do that you can split at maybe here but you first have to convert it into a 4-bit array like this and then you could do those splits right so here this is a unit 4 meaning that each element of this buffer is represented using only four bits but here it's a unit 8 array it means that each element of the buffer is represented using eight bits now something to keep in mind here is to we talked about this utf-8 and we said that in utf-8 each character is eight bits now what's really neat here is that in our buffer because each element holds eight bits it means that each element of our buffer corresponds to one character right so in utf-8 you can have maybe any characters really you can have numbers so one two Sony numbers and you can have letters so maybe AEP you could have them capitalized or lowercase it doesn't matter so many different characters millions of characters and also you could hold some special characters like maybe a space or a new line character or a tab character so if you put this character in your file it indicates that you need to go to the next line it's just like an enter so if you hit enter and your text editor you're actually putting a new character enter your file your text editor will not show slash again and then n will just show a new line but again the neat thing is that each character now in our firewall we have letters and spaces but each one of these are held and one element of a buffer so when we are splitting and we're getting the data in different chunks we are splitting by characters so if this is maybe one element if we were at this point maybe 22 and then three four and then some numbers before it and after it we cannot split this three in half it just doesn't work but we could split at this location and we could split at this location as well so I really want to make this extra clear that you cannot split maybe four and a half you could have done this if you were representing a buffer using four bits rather than eight bits and then you were using utf-8 then in that case well sure you could maybe do something to get half of the data of this character in one buffer and then have in another buffer which is really unpractical and then you would have to do a lot of work to add these numbers together but because we're representing each character using only eight bits and we're using utf-8 each element of our buffer is only one character and we're splitting at elements just like an array we cannot split at half of an index or a quarter of an index we're just splitting at each index all right so now that we know this let's take look back again at our array let me also raise these we'll try to make it a bit bigger all right so what we need to do is to first find out where we have a split so here we have a split so what we want to do is to put the splits in a variable so maybe we could Define a variable called splits and save that variable in it so in this case it's going to be 32. we're not getting 182 we're getting it in our next chunk and then when we are moving to our next array so we're getting the next Chunk we check whether or not this number is correct and we could do that by adding one to it and if we add one to it we're going to get the next elements if we get the next elements it means that we're not doing we're not having any splitting issues we're getting the number completely but if we add 1 to it and we don't get the next elements it means that we have a split issue so what we're going to do is we're going to take this 182 this is the first elements of that our first second chunk so 182 and we're going to add that split value to it which is 32. and we're going to do this in a string and not a number right so we're going to get 32 1 8 2. so we're going to get our complete number so this is what we have to do we have to check the end of the arrays if they're not complete and then if we subtract one from the end of the elements if we subtract one and get the elements before it it means that we are not having a splitting issue but if we subtract one now we don't get the elements one after it before it's it means that we have a splitting issue so we have to save the last elements in and variable maybe called splits and then in our next array do the check and see if we are having this floating issue if we do we're just going to add these two numbers together and we're going to get the correct number so this is how we can deal with those numbers are not complete this is how we can attach them together and then get the right numbers and then when we do get the right numbers we're just going to convert them into a number and then do our checks maybe check whether or not number is even and then write it remove check if the number is prime or anything we could just do anything that we want but the point is we're getting the numbers as we want them all right so let's now implement this in codes so what we can do is just do a quick check so I'm going to remove these lines and right here I want to say if the last number is not equal to the number before it in each chunk that we get it means that we're having a splitting issue so if numbers at length negative two now this will give us the last number before the last element of an array so if I add one to it and if I don't get the last number numbers adult length minus minus one now if this returns false then it means that we're having a splitting issue or if it returns true actually because we're doing not equal now these numbers that we're getting are actually strings so we need to convert them to a number as well so like this you just use this number Constructor to convert any string into a number so if I say number maybe this string it's going to give me a number so if I add 1 to it it's going to be this if we just try to do the same thing but Define a string and you try to add one to the string it's not going to work you're just appending one at the end of the string but if you first convert it into a number and then add one you are adding it to that number but you know correct math so if it is the case what are we going to do is just we're going to pop the last element so we're going to remove the last element of our current chunk and save it into a variable so I'm going to Define a variable here and I'm going to call it splits it's going to be initialized to an empty string if this is the case I'm going to say split equals numbers dot pop now if you run this the stop pop method will remove the last elements of an array this is this is our array one two three if I say array dot pop it's going to give me the less elements and it's also going to remove that last elements off of that array so I'm going to use this method to remove that last element and save it into my split variable all right so now using this we're taking care of the last elements now let's take care of the first element so I want to say if numbers the first element of the number is not equal to the elements after it minus one so numbers 1 minus one if this is not the case and I also need to convert them into a number like this now if this is not the case we want to do a check so if we do have our splits from the previous chunk we're just going to say that numbers the first elements of our array we're just going to replace it with our split and I want to go into around the trim function on it so with our split Plus the first number and also the trim function why not need it but just in case so now at this point we have our numbers exactly as we need them so let's just try to log the numbers so log numbers so and actually we're not getting the numbers correctly and well it's because we are we need to change the order of these two if statements so I need to cut this and paste it here so because we're checking this splits from the previous chunk and not the current chunk so if I put this after this if statements what's going to happen is that I'm going to update the splits with that current chunk which is not something that we want we want to check the split from our previous run and if that's the case we're going to do our splitting actually we're just going to add them together and replace the first elements and let's try it again and now in this case we are getting the arrays correctly now as you can see this number is exactly one before the next number I also notice that here we have a space but we're doing the trimming now these cases are because the numbers are correct in the first place so we're not doing this check so this will not return we're not getting to this point and therefore we're not really adding anything to them because they are already correct and we don't have to correct them but these numbers are the ones that we have actually corrected so here we have another space it means that this number was correct before we even got to that point all right now let's also try to take a look at the last two elements and see if they are correct and I I guess these trims are redundant if that's the case it'll be better to remove them because we're doing two less operations and we're also dealing with a huge amount of data so doing two less operations could save us some good amount of time so I live this up to you to figure out whether or not these trims are redundant or not if they're redundant just remove them we don't need them but I'm going to leave them just in case all right so now our numbers are exactly correct so what we can do at this point is to just Loop over these numbers and only write the even ones out to our stream so let's try to do that so I'm going to say numbers dot for each and this 4H will run and in the Callback we're going to get each number I'm going to first convert the number into a number because this number is now a string if you take a look this is a string so we have to convert it into a JavaScript number first we can do that using this number so number number lots of numbers hope I don't say number another time now we say if n if it's remainder by 2. is equal to zero it means that it's an odd number actually it means that it's an even number so we're going to write that to this string so I'm going to cut this and paste it here now rather than trying to write the whole chunk we write a space plus n plus another space and if it returns false it means that we need to drain so we're going to stop reading from our stream and then let it drain and then once the training is done we're going to resume reading again from that file so let's try this so I'm going to well actually let's try it with a very simple file because we're going to need to open it and check it out so I'm going to foreign switch to that right folder and instead of writing a billion times actually I had already done it before I recorded this portion of the video but you could go on ahead and do it and change this number to a million and then run it and then you're going to get this test.txt file which just like before should start from zero and ends at six ninths so let's try to copy this file and paste it in our read back folder and let's go with that file so instead of reading the whole text gigantic I'm just going to read the normal text the name is test so let's change it to text Dash small so text Dash small So This Is Our Big File very big file 10 gigabytes and this is our tiny file only seven megabytes so let's run this and then check our destination desktop txc file and see if we are getting all the even numbers out somewhere on the code node read back and it took a second and let's now take a look at our desk.txt file and as you can see here we're getting all the even numbers and I doubt that we were missing even one number so we're getting all the numbers last numbers should be five nines and then an eight and yep we get the number and then one before it is the even number before this number and so on and so forth now the file size should be roughly half of the text small file which is actually the case so this is if you round it up it's four megabytes and if you run this up it's 8 megabytes so it's half the size of this text smile so we're getting the even numbers correctly cool now if you want you could also just hit the numbers that are divisible by 10 just like this run again will run pretty quick and our destination we're getting all the numbers that are divisible by 10 meaning they have a 0 at the end of them all right perfect so this code is now working pretty good now this isn't the testing that you should do if you want to do this just don't try to look at it and say okay it looks good so the code code works fine you got to do some testing so maybe try to go over this file one element at a time and then add 10 to each one of them you should get the next elements and if you don't it means that we are missing a number or maybe something else or maybe we're doing some we have some logic errors so I encourage you to whenever you're doing this in production and you want to ship something to the real users really try to do so many testing don't leave your code up and say all right it looks fine and that's it I'm gonna go with this absolutely not you should do some rigorous testing and really make sure that you're getting the right numbers now we're not going to do this in this video but if you want you can do that and for example if we're getting the even numbers just get each number at two to it and then you should get the next elements all right so now we're pretty much done with a readable stream you could just get the data in chunks and then you could process the data and then maybe write it or just read it it's up to you now instead of just reading it we could have you know maybe added to our database or something like that it's really up to what you want to do the point is you're reading you could read a huge file now if you try to do this with that text gigantic it will work with no issues in fact let's give it one more shot to text gigantic and if I run it will take a while because we're doing some operations here now this is where I told you if you try to optimize your codes maybe remove these trims or anything else maybe find something that you could convert numbers that's faster than this number Constructor now I don't know if there is one I tried a couple one of them and this one came out to be the fastest but there might be some ones that are even faster so if you could just optimize this code maybe a little bit it will really add up and become something pretty big you know you could save maybe minutes seconds it could be a huge time server if you just you know try to optimize even a couple microseconds or milliseconds it's really going to be a lot but if you take a look at the activity monitor you can see that our memory is only 50 megabytes so we're doing a whole lot of operations not a whole lot this isn't really a lot but we're doing some operations and we're doing a lot of readings and writings so we go to disks you could see that by its read is eight gigabytes and bytes right is 800. now if this bytes read gets to 10 we are done because our file is 10 gigabytes so I guess we I have to wait just a couple more seconds and that's it but the point is our memory is pretty good only 50 megabytes and we're reading a file at 10 gigabytes and I guess we're done now yep we're done so if you take a look at the console we're done with no errors whatsoever now if you want you could time your codes and the way you can do it is to well there is no way for us to figure out our last right well technically we can but what we can do that's quite a bit a bit better is to try to add an and events on our read stream so this will happen when we're done reading so I'm just going to log and say done reading so first let's take a look at our destination file and see if all the numbers are even say cats.txt and let's stop it right now so and as you can see we are getting all the numbers correctly now in this case we are getting the numbers that are divisible by 10 but you could change it to any other number you could get the even numbers odd numbers anything it's all up to you but if you just take each number and add a 10 to it you should get the next number or vice versa if you take each number and subtract 10 from it should get the number before it so it looks like it's working pretty good and the size is roughly one tenth of the size of our gigantic file so it's 10 gigabytes and this one is one gigabyte perfect now let's try to time our code and see how long it takes so right here I'm going to say console dot time read back and here I'm going to say console dot time and read back and I'm also going to read from text small because I have to wait for possibly a minute for it to finish but this one is quite fast so node read back oops an error stream read is not a function yep I forgot to call the dot and let's try it again and cool so it's done reading and read back took 100 milliseconds so this is also a good thing to keep in mind now notice that in a real stream we used the Finish events but we don't have access to this one on a read screen and a read stream will instead have an end event now you could get to this point the Finish event by calling right stream dot end so streamrite dot end and when you call this then you're going to indicate stream is finished and then you're going to get to this point now in a real stream this is not something that you can do it's automatically done for you so when the readable stream is done so it will figure out whether or not it's done and if it's done it will automatically emit this end event for you now if you're trying to implement a readable stream which is something that we're going to get into in a couple minutes when you try that the way is that you if you push a null character to your readable stream so say stream read dot push null that indicates that your stream is done which is something we're gonna get into in just a little bit but yeah at this point I have to say that we're pretty much done with API consumers let's take a look at the documentation again and go over what we have so here I'm going to find the readable section so we're done with the writable if I can find it this is all about writable and here it is real streams now here again you could see some examples again they're just like before HTTP but there are different for example in the previous writable it was FS write stream but now it's FS read stream same is true with HTTP requests for example here Edge responses on the client now arrival stream it would be HTTP responses on the server so there are some things back and forth but you get the idea now here it says that we have two reading modes which is really important to understand one is flowing and one is paused now we did Justice we started to make our stream into flowing modes by adding that data events and then we paused it when we were draining and then we resumed it again when we were done draining and here we could see that it says all real streams begin in paused modes but can be switched to flowing modes if you do one of these so if you add a data event handler if you called stream.receum method or if you call the stream.pipe method now we're going to get into pipe in just a minute but keep that in mind and then you could also switch back to paused mode by calling stream.pause or by removing all the destinations now we could get these states by calling credible.gradable flowing you'd see that if it's null it means that you haven't done anything if it's false it means it's in pause mode and if it's true it means that it's flowing but how about was right I don't quite remember but I think that was the case uh yeah it says one riddle.reload flowing is null no mechanism for consuming data is provided so if we just create the stream and you try to log this you're gonna get null but if you start to make it into the flowing modes by adding a data event you will change it to true and when you pause it it will be changed to false now here also says that you should choose only one API style now there are two ways to read one is to add the data event and another one is to add the readable events and then read by calling a read function which is not something we're gonna get into in this video but you could read that here in the documentation and when you if you understand this data event this one is just like this one it's you will understand it's then we have some events like the close events which is kind of like before that close events for the writable then we have the data events which we talked about in great detail now you're going to get a chunk which is going to be a buffer but if you specify the set encoding you could also get strings we have the end event which is emitted when there's no more data to be consumed from the Stream and the end events will not be emitted unless the data is completely consumed and here it says sometimes that you are done with data but there is still data you're not going to get into the end now sometimes you might just read a portion do some work with that and then you're done but this event will not be emitted if you want to get into this emit end event what you need to do is to just switch it to flowing mode until all the data is done and then you're going to get into the end event now you can use this function if you call if you add that readable event on your readable stream we have the error events we have the pause events which will be emitted when you call that pause method now this is that real events which is just like that data events now we should only stick with one if you are going to use this readable you shouldn't use the data events again on this stream just stick with one now the way that you can use this one is to add the events listener first and then here's how you could read from it and you can also optionally specify size parameter to this read function and here it says that if you use both of them this one will take precedence and flowing but you shouldn't really try to do this just stick with one and he also talks about what will happen if you remove these listeners to resume events destroy now here we have some properties closed which is true if the closed event was emitted is paused as another method it's kind of like that variable we talked about but it only indicates if it's paused or not this is how we can pause the stream we talked about this one now piping we're going to get into it in just a minute now piping is just like that first thing that we did now before we add these operations what we did was that we just copied the stream over right so we read from that text and we wrote to that destination and we also took care of some draining things so we paused the stream when we were training and we made sure that we're not having any memory issues now piping is just like that you add a pipe to a real stream and then you specify a destination and node.js itself will handle all that draining and pausing and resuming for you we're going to see an example about this piping in just a minute but that's basically what it does and this read is if you add this real events then you can make use of it and specify a size option and we also have this property which is true if it's safe to call this which means that the stream was not destroyed or emitted error or end and we also have some experimental methods here we're not going to talk about them but if you want read them and use them Rebel encoding this will give you the encoding if you call that set encoding function real Earth flowing a real high Watermark value which is how we can get that the size of that internal buffer and pipe so you could pipe and then maybe after a while you could unpipe or shifting we're not really going to talk about it because instead you could just use a transform stream and it's it's usually better so skip this one wrap again we're not going to use this now here it says it will rarely be necessary to use real good wrap but we still have it to make it more compatible with older node.js applications and then again some more experimental methods I encourage you to use them but we're not really going to talk about them because they could change and they're right now experimental but they're really easy to understand for example using this map it's kind of like mapping on arrays so we could map over your real streams and see what you have when you can filter again like an array for each and two array you could convert it into an array and a couple more so it's still more experimental methods reduce and we're done so we're pretty much done with a real stream and then we have to talk about duplex and transform but let me first show you an example of piping so I'm going to create a new directory and I'm going to call it actually right here so new folder and I'm going to call it copy so here we're going to try to copy a file from one source to a destination create a new file called copy dot Js now let's also try to put a file here to copy now this one is quite small maybe let's try to make this test a bit bigger so actually I'm just going to copy this text gigantic over to my copy folder and let's try to copy this did I paste it come on copy and paste that looks like it'll take a while so I'll wait right it looks like it's done it tried to open it but it can't so now let's try to do something let's try to copy this file to a new destination maybe in the same folder but change your name to text gigantic Dash copy or something like that let's try to copy the file but without using streams so let's try to put aside all these string Concepts that we learned about for just a second and try to copy the file with the current knowledge that we have from the previous videos so we now have to read the file and we also now we write to a file so we could technically be able to copy this file over so if you want positively right now and write some solutions write some code and try to copy this file and please don't use FS dot copy because if you use this one it's just going to copy the file using a stream for you so without using a stream let's see how we can copy this file so I'm going to first include the fs module require node FS slash promises this function so I'm going to first read the content of this file and then write it to a destination so let's first open a destination so const test file equals weight f is dot open I want to call it text Dash copy Dot txt and I'm going to flag it as W to indicate that we're going to write now I'm going to read the content of this file so I'm going to say const result equals the weight fs.read file and I'm going to use this function now now this function will automatically open the file first for you and then return all the contents in one buffer so read file go to specify my source which is text Dash gigantic and now this is going to be a buffer let's first try it with a very small file so this one is fairly simple so it's only 70 megabytes so let's first try it with the text.txt file and not stat text gigantic so I'm going to change it to text.txt all right let's Now log the results copy and run the application but not again my semicolon all right there we go no such file or directory I don't know what's happening here oh it's test so let me rename it to text right so as you can see here when you call this function it's going to return all the value in one simple buffer now we could specify an encoding if you want but well let's just deal with this buffer first now what you can do is to just say desk file dot right and then specify that buffer and that's it it's also a promise so I'm going to say wait and our text copy is now empty but if you run the code you will now get the data in the text copy so this is one way you could copy a file over but it's not really practical it's actually not practical at all if you're dealing with very tiny files it's okay you could do it this way it's but still even in that way if you use streams it's way better and not just streams this is this is really problematic with big files so let's try to copy the text gigantic file so I'm going to say the text Dash gigantic and I think it's going to crash I doubt it's going to work so nodes gigantic now this will definitely crash because we're trying to move a buffer that is just copy that is tank bite in size so definitely we're not going to be able to do it but let's take a look and as you can see well actually we're not crashing so notice we're turning an error that says the file size is greater than two gigabytes so it's figuring that out for you but while the point is if you try to do it with a file that's less than two gigabytes what's going to happen is that you are going to your memory usage will be the size of that file which is not really good so we just add in two here and run this code so right mini and then node right Mini right it's done and it took 27 seconds so let's try to copy this one I'm gonna reveal this in my finder well it's one gigabyte so we should be okay so I'm gonna paste it here and copy and change the name to just text Dash Maybe not small but to maybe text Dash thick all right so let's try to copy this text big and see if we can do it so text gigantic text Big and we should be able to copy it but let's also add a set interval function just so that my process doesn't exit so interval come on interval and I'm just going to run it every second and let's open up the activity monitor and check this one so node copy and well you can see that our memory usage is around 900 megabytes and it's still doing the copy operation actually I think it's done with the copying operation it's not exiting because we have to set internal function but you could but you saw that our code was really bad I remember usage was at one gigabyte I was also try to time it so console.time copy one right here and one right here so console.time and copy and I'm just going to remove the set interval because we already know how much memory usage we're going to have run it so the copy operation was pretty fast only at 900 milliseconds but our memory usage is really high and we can't deal with that plus we can't even copy a file if it's over two gigabytes so what we can do so let me just add in some comments here memory usage is around one gigabytes and notice we can't now we could also add our CPU usage but this is not how we usually do it because when you're running an algorithm a solution or any code your CPU usage will be at 100 so adding CPU is such 100 is pretty useless now what we do usually is to report our speed complexity now we have two things to Benchmark our algorithm and our code one is speed complexity and another one is space complexity space means how much memory your code is taking it means how fast your code can do that operation now these are usually topics of algorithms and data structures so I'm not really going to go over them but in speed the way the reporter chord is like this so we say it's Big O maybe n and if it's Big O N it means that if we have an array of 100 000 elements we're going to do a hundred thousand operations now if we say it's Big O of N squared it means that if we have an array that has a hundred elements in it we're going to do 100 to the power of two operations so if you learn about these you're really going to improve your programming skills so I highly recommend you to learn about algorithms and data structures if you haven't already there are some great algorithms out there to learn about and that's it's not just all about interview questions it's really all about how good you could code you know how fast your Solutions are how well they run and things like that so definitely learn about algorithms and data structures the really important every software engineer out there needs to know about these I don't really teach you how to create highly performant code and how to really try to make your code as fast as possible so definitely keep that in mind if you haven't already just go ahead and check it out there are Great Courses out there great videos tutorials books learn about algorithms and data structures so I'm not going to put my CPU usage here because it's definitely going to be 100 but I'm just going to say execution time which was around 900 milliseconds okay now again let's try to improve this code but still not using streams so let's try to come up with our own streaming solution let's try to use streams without using node.js streams so the way we can do it is to create a buffer and keep filling it and keep writing it so that's something that we can do to really improve our memory usage so let's try that out so I'm going to actually just copy the code out and comment this portion and let's start from scratch so I'm going to first get my source file which is going to be waitfs.open text Dash Big Dot txt and we're going to read from this and I'm also going to create my destination file which is a file that I'm going to write to so it means that I'm going to copy from this location to that location so FS dot open text Dash copy Dot txt the flag should be w so the way that we can do this copy operation is by calling this function source file a DOT read now this function will return a small amount of buffer it will return us a chunk so let's try to log it it also returns a promise so I'm going to wait and let's run it so as you can see here we're getting a value now this would look familiar to it's 16 kilobytes so this function is returning a chunk of our file it's not returning a whole file like here this read file is returning the whole file in just one buffer and if it's more than two gigabytes it's not gonna work but this will definitely work if I just try it with text gigantic it will work no problem but it will only return me a small chunk a buffer that has a size 16 kilobytes so what we can do is to just keep executing this function and keep writing the result the chunks that this will return to our destination file and we basically are setting up a stream of our own so let's try this out so I'm going to save this into a variable so I'm going to say const read result equals this and I'm also going to get the bytes read out of it called read result dot bytes read or bytes red now I'm gonna check and see while this is not zero just keep reading and keep writing to our destination because notice now this is how many we read now when we get to the end of the file it's going to be zero which indicates that we're done reading and there's not nothing else to read so I'm going to wrap this code inside of a while loop and say while White Street is not equal to zero just keep reading and keep writing and I have to well the find is right here so bytes read maybe I say negative one because when we're starting out we haven't read anything so I'm just going to specify it as negative one and right here I'm going to say let bytes read equals that result so while it's not zero we're going to keep reading and then what we can do is just write this buffer into our destination file so Dash file.writes read results well it was called buffer so we're going to get the buffer out of it and write it to that location all right so this is how we can actually set up our own stream now notice that if you're doing this in production definitely take care of all the our handlings wrap it around to try catch and handle the errors we're not going to do it in this video for the sake of teaching and simplicity but in production definitely always wrap your code around to try catch whenever you're using this weight and handle all the errors so let's try this code but using a text first let's try it with text Dash dot txt small file I'll get sometimes confusing so text Dot txt this is our small file right yeah it is I want to change the name to text Dash small okay so this is our seven megabytes file this textbook is our one gigabyte file and this gigantic is our 10 gigabytes file someone say text Dash small so let's try to copy this all right it looks like it was successful if I take a look well you see some weird characters here some nulls but apart from this one the copy was successful as you can see now the reason that we're getting some nulls is that our last read is going to be a buffer that's not complete so we're going to have some zeros at the end of the buffer so let's also try to lock the buffers and see if we could get to that last log well the last buffer is just zero because we're not reading anything in that last read but also if you take a look at the buffer before it we're having some zeros at the end of the buffer which is not really indicated here because we're just logging some of the elements but if we try to maybe log this elements maybe add location this you can notice that even in our last buffer we get some zeros so what we can do is to remove those zeros from our buffer so that we don't get this weird null characters so what we can do is to add an if statement now if I like this read result again if you take a look we can check this byte's read property and see if it's not this value which is the value of our buffer it means that we have some zeros at the end of the buffer so we need to remove those zeros so that we don't get those null characters so let's try this one so let's check this bytes read so I'm going to say f White Street is not equal to 16 what was the number I'm just going to copy it white Suite is not equal to this number now what we're going to do is to first get the index of not failed so we're going to get the index of the last element that's not filled actually the first element that's not filled in our buffer and the way we can do it is to say read result dot buffer so this is our buffer now dot index of zero and this will return the first zero the index of the first zero that happens in our buffer now we're going to create a new buffer I'm going to say buffer.alic now the size should be this one index not filled and now we need to do a copy in these buffers so I'm going to say read result dot buffer dot copy as it means to copy this buffer to a destination so whatever I express right here is going to be my destination so it's going to be a new buffer and then I'm going to specify 0 which is going to be Target start which is going to be zero Source start which is definitely going to be zero and then source and now source and is going to be this index not failed so I'm just going to paste write that in so using this copy operation we're copying this buffer to this new buffer that we just allocated with the right size which is the size of those failed elements so we're dismissing all these zeros and we're copying that buffer over to this new buffer from these locations so we're starting from zero all the way until index of not failed and then we're going to do our right so test file dot right new buff I'm not going to write that buffer that came out we're just going to write the new buffer that we just created else and our else statements we're going to do our normal rights where our bytes rate is equal to the size of that buffer that's going to get returned if we run this code well looks like we have a typo somewhere so here it's called new buffer so I'm just going to paste that in what's called new buff so it should be new buffer if I try it so we can get an error which is a good point if we take a look at our copy now we can see that we got rid of those null characters and we're just doing our copy with no problem whatsoever great now let's also try it with that text back file now this one took nine milliseconds and I'm also going to indicate the file size so fine size copied was was one gigabyte right yeah it was one gigabytes so let's try the same thing with our little streaming thing that we set up so I'm going to copy text big.txt only just confirm that the file size is one gigabyte yeah it is so I'm going to save run the code will take a while well it took around two seconds but let's see if we it was successful I definitely can't open it but if we take a look at the file sizes so text copy is now one gigabyte so we copied this text Big over to this text copy and if you take a look at the font sizes they do match up but what we did is that we significantly reduced our memory usage so let's try it again and take a look at our memory usage you can see that it's around 20 megabytes and it doesn't really go above that point we're not sure because it exits quite fast but it's around that now let's also try it with text gigantic subject mechanic now notice that actually this took two seconds so let me just add these in so copy it is one gig memory usage around 20 megabytes or let's just say 30 megabytes and execution time was 2 seconds now it's definitely slower than this solution because here we're moving all the file in our memory and then we're just doing one right so it's quite fast but the memory usage is really awful but here we're doing multiple rights we have set it up a stream but well our memory usage is significantly lower and we're not limited to the flash size so we could do this on a file that's like one terabyte in size 100 gigs doesn't really matter it's also to try it with text gigantic and see if it works or see if we're going to get an error well it's definitely going to take a while but memory usage is around 30 megabytes and it doesn't really go higher than that my disk usage Yeah by three six if it gets to 10 we're done let's also take a look again at the memory so it's around 36 megabytes 30 megabytes I should say and we're done let's take a look at the file sizes text gigantic is 10 gigabytes and text copy is also 10.9 gigabytes so we are copying the file correctly without any issues all right so this code little code here as you can see is a little streaming thing although we set up so we keep reading from a source and we keep getting the chunks and then we write them to a destination now technically we don't have a buffer that we put the data in and then we pull it out but you can imagine that this is this is someone somehow similar to that streaming thing that we did now let's also try this but using streams I'm first going to copy this code and paste it and also I have the comment it out so we're getting our source file it should be text let's first try it with text big and then text copy now we don't need any of this logic instead we need to create our read stream and our right Stream So read stream equals weights actually just source file dot create read stream and a const right stream equals now this is going to be where we are copying our file in so it's going to be test file dot create stream all right so with these two in place now rather than trying to use the data events on this read stream and get the chunks and then write them to the right stream and also handle the draining and pausing and resuming the street stream what I'm going to do is to just use something that energy is already gives us which is piping which just does the same thing that we talked about we'll read from the source and it will write to a source and it will automatically handle all these back pressuring all these training and resuming and passing so when using pipe let me just show you how it works so you just put in your reach stream first and you call the pipe method on it and for the destination you have to specify a rideable string you cannot specify a readable stream it has to be writable now you could also specify a duplex stream and a transform stream which they both also Implement a rival stream but the point is it should be a rival stream and if you put a label stream in the pipe destination it's not going to work so I'm going to say right stream so what happens here is that this is Read Street so this pipe method will automatically read from the stream and those chunks that it gets it's going to write them to this destination now you can also chain these pipe methods like this and then put another writable like this or another duplex or another transform and then keep piping like this so if you're doing this again if you want to change you can't do it this way because this arrival is just a writable this should return a real Stream So if you want to do this these streams that you have in the middle they should be transform or duplex which they both Implement a readable and also a readable so I can't put a route stream here it's going to return an error because there's nothing to read from a write stream only to write to so let's just quickly run this code and do a quick benchmark so I'm going to copy our previous solution and just paste it here it's a file size is the exact same file size it's text Big memory usage let's take a look at that and also execution time [Music] so I'm going to say node copy.js and we're done so it took around half a second which is faster than our previous solution our string thing that we set up and the memory usage should be fairly low so if I try to copy it a couple of times now in this case it's actually way faster looks like that my computer was just freeze for a couple milliseconds it's actually five milliseconds 30 milliseconds which is way faster than what we were hoping for right and sometimes it gets a bit higher well it really depends on a lot of things I mean what's going on in my computer so results vary but you could say that it's pretty fast sometimes we get to five milliseconds let me just delete the file and try it again soon it deletes and let's try with text gigantic so text pick gigantic 2D monitor and if I run it well it was well actually looks like our timing function is not right and yep it is not right because we need to do a callback function on this pipe take another look you know it's also first to make sure that it works and then we're going to handle that callback thing so we can say for sure how much is the execution time because we need to see when this piping is done there are various things we can do so we're going to talk about them and if we take a look at text copy sizes yep they do match up so here it's 411 and it ends here with 411 and they were both 10.9 gigabytes cool so you can now come to the documentation and read the pipe thing piping is kind of easy to understand nothing too crazy is going on well it might look like magic but at the end of the day what's happening really is those things that we did previously so it just handles these trainings and pausings and resumes for you behind the scenes so here we could see that readable.unpipe you could also unpipe maybe after a while it could be some cases that you might want to do that and here's how you could chain and let's see now we could use this on dot end on our readable to figure out when it's ended and there we could run our time end function so let's do that let's read stream dot on and here I'm going to move in my time end function and let's try it with text Big just to be able to compare it to our previous solution here so I'm going to run it and while I took around one second so not that much faster comparing to our previous solution but yeah it's it's twice as fast as it had something so I'm not saying it's our previous solution is better definitely not this one don't try this in production definitely make use of these things that are already built for you there's a lot of care going on behind them and a lot of good people are working behind them so always using something that is there could be better not always but in a lot of cases so execution time is around one second I guess it varies but that's about it I notice it's not five minutes as we saw previously was because we had this time end function somewhere that was not correct so it will take a while to complete so I'm going to just change it to one second file size copy one gigabytes and memory usage is the same is it the same let's try it well it could be maybe a little bit lower 20 I guess sometimes it gets to 30 but yeah let's just leave it at that let's leave it at 30 megabytes just like before it could be maybe a little bit lower but a couple bytes really doesn't matter so here is the definition of this pipe you specify your destination and it always have to be a writable and you could only execute this pipe unreadable you cannot execute it on a writable stream and here we could see at the last sentence that the float data will be automatically managed so that the destination is not overwhelmed by a faster readable stream so this is what we talked about that the node.js will automatically handle those things for you and here we can see allowing chain of pipes if it's duplex or transform so you could chain these methods like here now this Z should be a transform or a duplex let's see its definition it's this one zelob.createzip well which is a transform screen so it makes sense and this is how we can be notified that the piping was done by default stream dot end is called on the destination writable when the source emits n so this is how you can figure out whether or not the stream is done and yeah that's pretty much it let's take a look at the other things about pipe and also I just want to say this again we talked about this choosing one API style and we talked about this data events and real events but also piping is in this one of these API sites so if you run this on data on a real world stream do not try to pipe it again or use this unreadable just stick with one and that's it don't try a couple of things at the same time it will lead to some unintuitive Behavior as it says here in the documentation and he will read more about these states that if there's a pipe destination and if you remove them what's going to happen is that I think this state will be changed to Post Yeah will be changed to pause if you remove one of these piping destinations from a readable stream so as soon as you add this so I want to just do a quick recap when we create our stream up until this point this is not doing anything it's not reading from the underlying resource it's not giving us any data whatsoever it's just sitting there and nothing is happening and if I log that variable which was read the scream Dot readable flowing yep that's it if I like this and run it you see now it's null because we're not doing anything but as soon as I start piping it will be true and if I unpipe so I'm going to say read stream Dot unpipe and unlock this again now we get null now null is our first lock now at this point we are not doing anything with our read stream now at this point we start to read from it so this property will be changed to true which as you can see here and right after we're going to unpipe so we're going to remove the destination and we're going to change the read stream into pause State again so now our readable flowing value is false if I start to pipe again and like this again now will be null false null true false and then true because we started to pipe this stream again and here's our final file so if you pipe and unpipe it will resume from where it was passed and to really demonstrate it start with a text small let's run it now we're doing a couple of piping and unpiping like here so we're doing an unpipe if I run it it's quite fast and if you take a look at text copy starts at zero goes all the way up to six ninths so by piping and unpiping that's how we can pause and resume reading from this read stream you could also get this on pipe events you can read definition here on pipe event is emitted when stream.unpipe method is called on a readable stream so here when we were calling it we could also make use of that unpipe method on our rattle stream and some more methods that you could just take a look here and understand easily what they do and we also have this pipe event on a writable stream which will be emitted when we call this stream.pipe on a readable screen and we add that writer to its destination and here it's just at the beginning of the node.js documentation here it says a key goal of the stream API particularly stream that pipe method is the limits the buffering of data to acceptable levels such that sources and destinations of differing speeds so we could have a source that's faster than a destination so the key goal of streams is to make it so that we're not overwhelming our available memory especially this stream nut Point that's what this method is doing all right now in production you shouldn't really try to use pipe and the reason is because of the poor error handling that you have available I mean technically you could do some error handlings but you have to keep running the dot on on each one of these streams and then manually destroy your right streams if an error happens now what you can do instead is to use a pipeline so I'm going to show you the documentation first here we could see the pipeline method and you have to include it first and here's the only place that it was mentioned in the documentation it's way better than piping well it's actually basically just don't get the piping for you but it will also handle the error handling for you and it will destroy those streams if an error happens so you could read the documentation that says a module method to pipe between streams and generates forwarding errors and properly cleaning up which is very very important and we get the all of this happens in the Callback when the pipeline is complete so if in production you just run this piping and you leave the streams as B and some errors might happen and actually they happen quite often your files might just get removed for whatever reason something might happen in your system and maybe you're just dealing with a network and your network might just go down for a couple seconds or just for a couple milliseconds or a couple microseconds so errors do happen quite often in production and using piping is not really recommended because in that case you're living a lot of streams open and you're going to run into some memory leaks and memory issues so always clean up when you do have an error so you can just use Pipeline and automatically let nodes handle those things for you this was recently introduced I think it was version 10. and prior to this pipeline that it was introduced we were using something like pump yeah if I remember that correctly so pump this was a package that we're using to handle these errors and you can see it was published five years ago so if we scroll down you can see that when using standard source.pipe destination Source will not be destroyed if dust emits close or an error and you're also not able to provide a callback to tell when the pipe has finished well technically you can use the end events but yeah that's right you cannot use a callback to indicate that but pump does these two things for you so it will automatically destroy the source for you and you can see it's a very quite popular package 20 million downloads weekly so when you want to use pipe you should really take this into account and install this package instead of just using pipe because you can see it's a very fairly simple package all it does is that you call this and you specify your Source transform another source and destination just like a pipe syntax is quite different but you also get a callback function and you could also and it will also automatically destroy the source for you if one of these return emits and error events so this is what you should do if you want to use pipe or just this pipeline now let's also take a look at the pipeline Pipeline and pump they firmly work in the same way so I'm going to use this and here we can see that stream pipeline will call the destroy method which is really important when you're done you should destroy your methods now if it's successful it will automatically be destroyed for you so when you and that's a close Events maybe I'll read a stream or write stream and then it's done it will automatically be destroyed but if it's not successful it will not happen but pipeline will handle that for you is that something else to keep in mind in our previous codes that we wrote you have to keep in mind that if something happens you have to manually destroy your stream now if it's all successful it will be automatically destroyed for you but if it's not you have to take that into account so maybe add in stream dot on error and then manually destroy your stream but anyway let's include that pipeline first it's going to say const was it lowercase or uppercase lowercase pipeline choir stream and notice we actually haven't really used this module in this video so far and it's because these streams are used like create read stream are inheriting from this node stream module behind the scene so if you take a look at the source codes you will see that they are inheriting from this so if you want to implement your own streams which is something that we're going to get into in just a minute you have to include the stream maybe if you want to create a real stream you have to include the readable and then inherit from this readable which is something that we're going to see in just a minute but yeah this is the module we forgot to talk about it until this point but we're going to see this module a lot more from now on and the video so let's just use pipeline I'm going to comment all these out all you have to do is just call the function specify your source which is read stream and just chain you can add in as many streams as you want here but you have to make sure that these streams that come in the metal they should be either duplex or transfer you cannot put a writable screen here because if you do it will try to read from this and write to this which is not going to happen or if you try to put readable string here now this works fine from S2 to S3 but from S1 to S2 it will not work because it's a real stream and you cannot write to it that's something to keep in mind [Music] so let's try this and our error I could also use it like a promise so you have to include it from this so node stream slash promises just like this file system module that we did but I'm going to stick with callback right now so I'm going to just lock the error all right let's run it and also let's move our time end like here and well cool so you can see well we get undefined because we don't have an error right now so you could just say if error do something with the error and if not just log that the pipeline was finished this will now handle all those cleaning ups for you so if something goes wrong here it will destroy those streams and you're good to go you could also use pump which is quite popular and quiet light I encourage you to always take a look at the source code at the GitHub account and see what's going on so you always go to the issues and always take a look at the source code so we can see that the last commit was two years ago which is not really that bad because well it's now really stable it was published many many years ago five years ago so really a lot of products are fixed but the source code is quite simple it's just one index.js file and yep it's really really simple so we have some of those destroying things going on around Destroyer we have this Destroyer function which destroys a stream and search more you could see that something happens it's going to destroy these streams for you automatically as you can see here so something else to keep in mind now something else that I forgot to say another useful function I can use for error handling and cleaning up is this finished function so using this one whenever a stream is no longer readable or writable or has got an error or was closed unexpectedly you could use this function to handle your errors and do all the cleanings as you can see here so it's also something else to keep in mind when you are working with streams in production error handling and cleaning up is very important when you're dealing with the streams so do keep in mind this pipeline function and also this finished function and also this pump all right all right so that's all that we need to know about piping definitely always check the documentation I don't mention all the things that way the video could be really long but we covered the Core Concepts so you should now feel free and feel really comfortable to just come to the documentation reads everywhere that we have this pipeline piping read stream write stream and we're pretty much done with these sections now we have to talk about duplex stream and a transform stream we just mentioned that a duplex is a stream that you can write to and read from and a transform is a screen that will just like a duplex so it's inheriting from duplex but it's also transforming the data somewhat somehow but we're not going to use these two right now just like we did here we're not going to use the API consumers because honestly there's only just the xelop and crypto modules that use these two so the fs module is not really using the duplex and transform but what we're going to do is to implement them ourselves so we're going to include them and then inherit from them so we're going to move on to the next portion of the documentation so API for stream consumers this is how you could use a stream that's already written for you be it and you know just documentation or a package that you install that will give you some streams and the next API is how to implement your own streams so we're going to first learn how to implement our own rival stream and then readable and then we're going to learn how to implement a duplex and a transform and then learn how to use a transform and a duplex stream by learning how to use our own screens that we created all right so now let's move on to this section [Music] all right so now let's talk about how we can Implement our own rival streams so I'm going to close all these open files and I'm going to create a new directory here and I'm going to call it a custom writable and inside of it I'm going to create a new file called custom writable.js and just to be consistent let's just change its name to use a dash instead of camo case like this all right now the way that you can influence your own stream is first by including that string that you want to implements so in this case we're trying to implement a rival stream so we have to include the Rival stream and it's capital W from that stream module so node FS note actually notes stream like so now we have four streams available we have readable we have writable and we have dupe lights and we also have transfer so we're going to use all these and Implement our own streams but right now let's focus on the rideable stream so you first have to include that particular stream that you want to implement and then you have to create your own class now if you're not familiar with JavaScript classes please pause the video right now and go and learn how they work it's really important to understand that and if you don't know JavaScript classes it's still fine as long as you understand JavaScript object per typo inheritance that's all you need to know now we're going to use a class syntax which is just a syntactic sugar for that object protocol inheritance now if you go to the implementation section you usually see that if I can find something here you see the example first using a class and also without using es6 features so you could also Implement your own streams just by using normal javascripts so you first have to Define your function and then make it inherit from this writable that you're going to include so we have both options in this video we're going to focus on the class-based syntax but feel free to use this one if you feel more comfortable with this one doesn't really matter so I'm going to Define my class and I'm just going to call it simply write stream which extends or inherits from that writable object that we included right here so now that we've defined our class what we need to do is to call the parent Constructor so we need to use this super function now if you're using pre-es6 syntax it just means that in your function you need to call the writable object that you just created so you need to call it and pass this which refers to this function to that routable Constructor this is basically all that this super dust it just calls the pattern Constructor and passes this to that so when you use dot call what you're doing is that you're saying that change the this difference of that function to this this reference it is really something related to JavaScript and will take a while to explain so I'm not going to do it so if you are not really understanding this part just read about super and this and you're going to be good to go all right so I'm going to Define my Constructor and I'm going to accept some options now in this stream what we're going to do is to just Implement that FS create write stream that we just did so just to make it more easier to understand I'm going to change it to FS actually file right stream so we're going to create this stream and then just like before here in our write many codes we should be able to use it like this so we should be able to call Dot write Dot and use the train events and so on and so forth but now we have more control over how our object works so what we need to pass is one high Watermark value I could also not pass it and the default one would be that 16 kilobytes but let's just do it and also a file name because we're going to specify what this stream should write to now in the Constructor I'm going to call that super function and rather than just passing all the options that are passed to the Constructor will be better is to just specify specifically what we want to pass to our parent Constructor and if we want to take a look and see what are all the options you could use you could see them all right here so you could use this High Watermark value you could use default encoding object modes emit close or knots and a couple more so if you want to see the complete options take a look at this portion of the documentation so now in this case I'm just going to specify the high Watermark value and that's it now let's also save the file name so this Dot filing equals finally and let's now open the file up first before we try to write to it all right when you're implementing your own streams you first have to well do this inheritance and then set up your Constructor but after you do these two things there are some specific methods that you have to implement and you could see the list right here they must Implement one or more specific methods if you want to implement a writable stream for example you have to influence maybe write or write n or maybe both and also the final which is for our final right now for reading you only have to implement this read you cannot implement this read when you are implementing a writable stream and for duplex you have to specify all of these but we're now focusing just on the writable stream so you have to first implement this right method which will be called whenever you write stream dot write and that's it so if I initialize My Stream here so I say const stream equals new file write stream like this and I could optionally pass my high Watermark value or I could just not and default will be 16 kilobytes and I'm just going to specify maybe 1800s like so now just like before we could write to our stream like this so stream.writes and then our chunk so we could say buffer that from whatever so this is some string [Music] and then we could say stream dot end maybe this actually buffer from our last right like so so when we do this when we write to our stream this method here will be called right now this method accepts three arguments one is the chunk that we're writing another one is encoding which if we don't specify it's going to be a buffer by default and the last one and which is quite important is our callback so here we're going to do our right operation and then when we're done we have to call the Callback function so when we're done we should call the Callback function now the Callback will accept only one parameter which is either an error or just no so if you don't specify anything and just call the Callback it indicates that you are done writing and it should move on now when you run this right and then you immediately do another right now while this is happening so before you call the Callback it's going to buffer the data so when I do double writes and I don't let this right to finish and that means not getting to the Callback it's going to buffer but if it does get to the Callback and then I write again it's not going to buffer so if I maybe put this call back in a set timeout function and that's going to write maybe after five seconds and that five seconds until I call this callback all the things that I write to my stream is going to be buffered so when we invoke this callback function what's going to happen is that we're going to get that train event on our data on our stream excuse me all right actually trained so when you call this callback what's going to happen is that node.js might emit a drawing event for you now one thing that I really want to make a huge emphasis on is that you should never ever admit these events from your child classes so I shouldn't ever do this so say this Advanced maybe error or or drain or things like that it shouldn't do this at all right if you do this you're going to run into some issues so never ever admit events from your child classes let the methods do those things for example here when you call this callback it's going to emit this train event for you and you shouldn't do this explicitly now the same is true about overwriting these methods so you shouldn't really just say write like this and then overwrite the original method instead you should use underscore write okay so this is how node will be notified that this is going to be your implementation for right never ever do this okay this is really bad practice and I don't even know that it's going to work so it's really advised against in the node.js documentation so do not do this and also one more thing do not try to call these methods explicitly in your code so never do this okay it's again super bad and it's not really advice you should just use those methods that you have available when you were talking about those stream consuming and do not ever try to call these explicitly in your code when you're using your streams just just use it as you would with any other normally stream object saw a couple points that I wanted to put an emphasis on now let's try to do our right now before we do our right there is one more method that we can Implement which is called constructs which has a call back now this is a pretty important method and this will run after your Constructor runs and before the stream gets to the right method or any other methods that you are going to implement such as the final and destroy we're going to implement these in just a minute so this construct will run after your Constructor is complete and before you call this callback so if I do a set timeout and in that set timeouts I'll call this callback and to set time map might have this five seconds duration so what's going to happen in this case is that this right is not going to happen after five seconds so this will call my Constructor and then after my Constructor is done my construct method will kick in now until I execute this callback all the other methods will be paused now some of the things that you can do in the construct method would be initializing your data Maybe opening a file or opening a resource or whatever and also do some asynchronous codes so let's try to open the file that we're trying to write to here in the construct method so I'm going to first include the fs module so const FS equals require [Music] node fs and we're going to use the Callback version so I'm going to say fs.open I'm going to first specify my file name which is right here and the flag which is going to be W and my callback function so we're going to have an error and our file a descriptor which is just a number and we need to use this file descriptor later on when we want to write to our file or read from it actually in this case we're only going to write but also we want to close a file we have to use this file descriptor so when we open our file what we're going to do is to first check for error if we have an error we're going to call the Callback function and pass the first parameter to be that error object so if you just call this callback and don't pass anything it will indicate that the construct was successful and we will move on with our rest of the codes but if you do pass an error to it the stream object will automatically handle that for you and exit the stream and indicates that hey we'll have an error we cannot construct our stream so else we do not have an error what we're going to do is to save this FD variable to our stream object because we're going to need to use it and all the other methods so I'm going to define a new variable here so this dot ft and I'm going to initialize it to null because when we're just starting out we don't have a file descriptor but when we get to this point we will have a file descriptor given the fact of course that our open was successful so this dot ft equals that ft then when we're done we're going to execute the Callback like this so I'm going to write some comments here this will run this will run after deconstructor and it will put off all the other methods until say it will put off calling all the other methods until we call the Callback function and here I'm going to say no arguments means it was successful and if we do pass an argument it means that we did have an error so if we have an argument or if we call the Callback with an arguments means that we have an error and we should not proceed all right so this is our constructs method and now in our rights method let's just call the file descriptor and see what we have all right so this dot FD and it's not a function so let's just log it so console.log this dot FD and let's also comment out these final and Destroy just write to it once and see if we could get the file descriptor so I'm going to open up my terminal I'm going to navigate to custom Dash writable and I'm going to run the code it looks like we have an error let's see what the error is path arguments well yep we're not passing file name to it so let's do that find name should be let's also create a file and I'm going to call it text.txt but this is going to be the file that we're going to write on so file name text.txt all right now let me run it again and now we do get our file descriptor which is as I said before is just a number now in our right what we can do is to just write to that file now notice that if we do this it's not really going to be helpful at all I mean it's just like that first solution that we wrote right here right we're writing every single time that we receive some amount of data so what we can do instead of just writing every single time is to maybe write whenever we reach the high Watermark value so let's just do this pretty quick so in my Constructor I'm going to define a new variable and I'm going to call it maybe this stud chunks and which is an array now we're going to receive these buffers so this chunk is a buffer and we're going to push it into this chunk and we're going to check its size and say if the size reaches that high water mark value then we're going to empty it and write all the data down to our internal resource which in this case is just a file so let me also Define another property so this starts chunks size maybe and it's going to be initialized to zero so we're going to increment this value by whatever the size of the buffer is that we're pushing to this chunks so let's now do a bit of checking so actually let's first oops let's just push this chunk to that so this chunks a DOT push chunk [Music] and then this dot chunks size I'm going to say plus equal to chunk dot length so I'm going to push that buffer into my chunks and then I'm going to increase its size all right and then I'm going to do a check and I say if this the chunks size is greater than this dot writable high water mark so if that's the case we're going to do our right and empty this chunks so I'm going to say FS dot right so we have to first specify our file descriptor which is this dot FD and with that in place we have to specify our buffer which is going to be all those chunks but we need to concatenate them together so that we have one single buffer so we can use the buffer dot concat method to do just that so I'm going to say buffer the concat this the chunks and then my callback function which is going to have the error as its first parameter and what I'm going to do here is to say first check for the error if we do have an error just like here in our construct method we're going to call this callback and pass that error in it indicates that we have an error and we should not proceed so I'm going to say if we have an error just return callback and error so just call the Callback function and pass this error object in now now if we don't have an error we're just going to say this dot chunks equals an empty array again and this the chunks size equals zero and then we're going to call our callback function like this and if we don't get to this if statement so else if our chunk size is less than the high water Mark we're just going to call the Callback function and that's it all right you know what let's just for fun also figure out how many rights we're doing for that file so I'm going to define a new property here and call it this dot maybe a number of Rights and I'm going to initialize it to just zero or let's just name it writes counts right and I'm going to use it every time I do a write before I call the Callback function I'm just going to say that plus plus this Dot rights counts now I'm going to say plus plus because this is slightly faster than this not that much but because we're doing a lot of Rights we have to try to save as much as time as we possibly can so saying plus plus I is faster than saying I plus plus I'm not going to explain why Google it if you want to know but well that's a fact it's really related to JavaScript and nothing to do with node.js right so our right function is pretty much done so let's run the code and see if we can get this buffer into this file so I'm going to run it first check if we don't have an error so no we don't have an error but let's try to open a file and see what we have and as you can see we don't have anything and the reason is we're only writing when our chunk size gets to the size of this Watermark value which we defined it to be 1800 now something else we have to do is to implement this final code so this is going to run after our stream is done so here we're going to grab all this value and just write it right so I want to say and actually the first argument is callback so we need to call this callback when we're done otherwise no it will not be notified that we are done and we'll just keep waiting or return an error or something like that so I'm going to say fs.write just like before I want to specify the file descriptor and then buffer dot con cap and it's going to be this the chunks and then a callback function and its first argument is going to be error so if we have an error we're just going to return and call the Callback function but with that error object like this otherwise we're going to set the chunks to be zero and also call the Callback function like this if we run this if you open that text.txt file well we're still not going to get it and the reason is this is only going to get codes when we call the end function so we have to do this right so we have to call this Anza methods I want you to it's going to notify this final method and then we're going to do our right so if I run this now now we're going to get our rights now notice that this is slightly different than crates read stream because in that case when you're using Create root stream every time you write even if you don't call the and function you're going to write to that internal resource which is the file in this case but here we're not doing that but you know it's uh it's up to you you could do something like that maybe we're trying to do this to make it faster but I don't really recommend doing this in the real world just use the fs create read stream it's more it's better than this code that we're writing but this is just for educational purposes only it's only meant to teach you how to use the stream objects themselves and create your own streams now we can also implement the destroy method and here it indicates how to destroy our stream and do some final things if we want so here I'm going to log how many rights with that so I'm going to say a number of Rights and it was this dot writes counts and then I'm going to close the file so if we have the final descriptor we're going to say fs.close this.ft and then error now this destroy will be called with two parameters the first one is error and the second one is the Callback so you have to call the Callback otherwise it's not going to be done and here as the last callback all I'm going to do is to just say callback either this error or that error so if closing returns an error I'm going to call the callback with that error object otherwise I'm going to call it with the error object that was passing of now our object was passing it's totally fine we're just going to move on and else we're just going to call the Callback with that error object now I'm just going on that's totally fine if the stream is destroyed completely and without any issues this error object will be null but we do have to call the Callback now also one more point you should never do something like this maybe say Throw error or new error or things like this never ever do this if an error happens what you should do is that you should pass it to the Callback as its first arguments and that's all that's how you handle errors where you're implementing your own streams so every time you have an error just pass it to the Callback and notice will handle it for you and do the rest of the things that it needs to do with the stream itself again the same is true here every time you have an error just pass it to callback and do not ever try to throw an error or emit an error event like this this is very bad practice and it's not really meant to be used this way so never ever do this now this final events will be called before the stream is about to be closed and after it's done running when you call the Callback it's going to emit a finish event so we could say stream dot on finish and I'm just going to log stream was finished so let's run it and we can see that stream was finished if we take a look we have our rates and we get number of Rights zero because we were only incrementing it here in our right but we're actually doing the right here in our final so it's expected we're going to get it to be zero and I need to remove one space here to make it more consistent all right now we get this finished events because we call this end and we implemented this final method if I don't call this callback so if I just comment this out and run a code again you can see that we're not going to get these logs because if you don't call this callback this finished event is not going to be emitted and you're not going to get to this point and also you can't even get to this destroy function because this destroy will only happen after you are done with this final method so we saw that we didn't even get this number of Rights block so let me just uncomment this callback and I just noticed that here it should be two of these characters and not one the or tag should have these two all right so at this point we're pretty much done with our rival stream so now this stream that we're getting is just like if you run something like this so const stream equals FS dot create write stream so technically we should be able to do everything that we can do with this stream using also the Stream So what I want to do right now is just run our first code which was this right mini but now using this custom class that we just created so I'm going to go to this write many file and I'm just going to copy from this point up until actually all the file so I'm going to copy it all and I'm gonna paste it right here let me just zoom out a little bit so I'm going to paste it right here I'm first going to remove all these codes I'm going to paste it and let's remove all these logs and also move the require statements at the top and I'm just going to name it FS promises and let's also do an update here whatever we were using a promise which is the only place and now rather than trying to get my stream like this I'm just going to paste this code there so I'm going to cut it out and paste it here all right so this should just work fine and let me also change it back to let's just not specify the high Watermark value to make it to the default value and this is our number of rights which we're doing how many it's right now at let's just try 1 million so six zeros to remove one more and yeah this is pretty much it stream Dot and stream dot right train and finish and we're also timing our coat so we're writing to this test.txt file which is right here and I'm going to delete it and run the code and see what we're going to end up with all right so let's run it node custom writable.js and we're done and number of writes you can notice that it's 481 and where is our oh we have two so what's going on yeah we have an issue here actually we don't need to open the file at all why am I doing this why did I change it to this so we don't need this we don't need a file handle we're just creating it like this and we don't even need to include this FS promises my bad sorry so let's delete all these files all right and now let's run it we have file handle close yeah that's right we don't need to do it here we're closing the file handle here in the destroy function actually methods now every time a function is associated with a class it's called a method and every time it's just a standalone function something like this this is called a function so I keep using these two terms interchangeably but that's really how it should be so a function is just a standalone function that doesn't belong to any class but a method is a function that is for a class so that's something that you probably need to know but don't really focus too much on it I might just use these terms interchangeably all right so now let's run it again should work now and yep cool so let's now take a look at our text file and we were getting that same exact file and it's it's working pretty good we implemented our own stream rather than using fs.create right stream we're using our own find write stream objects and if you want to convert the speeds well here this code ran do we put it wow we forgot to write these for this one but I assume they're pretty much the same you know what let's let's try it so I'm just going to put the execution time it's going to say execution time and number of writes how many rights should we do well let's let's not do one million because it's quite fast so let's instead do 10 million so seven zeros right so three another three or so I just have to remove one and now it's ten million and I'm just gonna put it right here let's move them down here so number of writes this and execution time so let's measure how fast this code will run so I'm going to navigate to this write many function actually write many files so right many and I'm just going to execute that code it will take probably two seconds yep so this took two point let's run it to three times and get the average so 2.7 2.6 and well we could just say 2.7 seconds like this accurate running gets for 10 million times but here we're running it for only how many for only one million times so let's do the same thing with our custom writable and we should have seven zeros here so I have to put one more so now it's 10 million and let's now run it and see how fast our code will be so I'm going to navigate to that folder custom writable and I'm just going to run the code should be fairly similar yeah it's 2.3 seconds I guess it's a little bit faster but still stick with that create read stream rather than trying to do it it's like this it's probably faster because we're saving them here in this chunks array and then we're doing the concatenation we're not really making use of that internal buffer that much but I don't know let's also measure the memory usage so it's around 56 megabytes so memory usage is also the same thing as before now if I just put one more zero here to make it run a bit longer so we would probably be around 20 seconds now so let's measure my memory and yep that's around 30 megabytes 50 megabytes so we have a good memory usage and also a good speed time take a look at disk you can see that we have written like 800 megabytes and if it gets around one gigabyte it'll be done and now it's done so this is our custom writable and let me what did I do right so I'll save and close this all right so this is how you can implements your own writable class now some of the points that you have to learn is that you first have to extend from this rival object and then you have to implement these methods now there are some best practices and something to watch out for for example not throwing an error here or not calling the functions here I mean for example not calling I don't know this dots rights or here you shouldn't run this dot underscore right you should always use your stream just as you would with any other streams so there are some best practices we talked about them there are also mentioned in the notice documentation so I highly recommend you to read that as well and I guess we're pretty much done and also one more Point here if you are implementing your own streams you won't find a lot of packages out there that are using this so they are implementing their own packages something that I can just say off the top of my head would be busboy it's a package that is one of the dependencies of malter which is used for uploading files so if you take a look at the source code you're going to see that it is implementing its own writable and readable streams we're going to take a look at some of those packages at the end of the video but right now this is all that I want you to take from this part of the video and also here we're writing to file but you could really do anything you want you could maybe write to a network card you could I don't know write to another process it's all up to you and we're going to see more examples later on in the course as well now let's also take a look and see how many times we're draining here so I'm going to just use a variable and Define something here or maybe let D equals zero and then just increments it so each time we do a train we're going to increment this value by one and right here in our finished events let's just log maybe number of trains and lock that the variable like this and if I now run it let's also change it back to 10 million actually just 1 million if I run it you can see that number of trains 480 and number of writes for 181. now a number of writes should be one more than this actually you know what let's just put something else here so every time we write let's also increment that writes count as well so let's also increment it in our final so now it should be 482 yep that's right and we're training for 180 times which is the same as before if you recall from here on this right menu we logged how many times we were training and it was somewhere 480 I guess 481 so we're training with no issues whatsoever if you want you could uncomment this training and then I'll try to just keep writing to the stream and you're going to see that we're going to get some memory issues so you have to also train these custom streams as well it's not really that different from a consumer stream something like this FS creates read stream or FS create write stream or things like that so stream is a stream and you always have to take care of things like draining error handling and things like that all right so now we're pretty much done with custom writable and now let's see how we can Implement a custom readable now uploading a custom readable is quite similar to this one so I'm not really going to explain much we're just going to write code and it should be fairly easy for you to understand because we're having the same functions we have the same Constructor the same construct methods now we don't have this right instead we have read and we also have and I think we don't have this Final in readable stream but we do have this destroy method so let's quickly Implement our custom readable as well so I'm going to create a new actually let me first close this open file and now I'm going to create a new folder called custom readable and here inside I'm going to create a new file called custom readable .js so I'm just going to speed up it's fairly similar to that previous code that we wrote implementing our custom rival function so this one is fairly easy so we first have to include the object so const readable equals require node stream and let's also include the fs modules of const FS equals require node FS and now I'm going to Define my class I'm just going to call it file what did we call it here it was file write Stream So file read stream stands readable so what we're trying to do is just like before we're trying to implement FS create read stream but instead just using our own custom class so we're going to have our Constructor here we're going to specify the high watermark and also the file name and here we have to call the parent Constructor and pass in our functions actually our options whatever we want and then our file name which should be the file name that's coming in from the instructor and then our file descriptor which is just and all from when we're just starting out all right now I'm going to use the construct method that will accept a callback we have a typo right and here I'm going to first open so open this dot file name and flag should be read and error and file a descriptor just like before not here if we have an error let's just return callback and call the callback with that error object like this if we do not have an error let's just set the file descriptor and also call the Callback which is very important so you forget this if you forget calling callback you're going to have some issues all right with the construct method in place let's now implement the read methods now when you are influencing a rattled stream he would implement the rights method or the right NF method or both but when you are implementing a real stream you have to implement a read method you cannot Implement a write method here in fact my visual studio code will not even give me that suggestion but if I try to add this read method there here I have it now the three method accepts one parameter which is the size so we could say read four bytes or read maybe 10 bytes or whatever so here we have to read from our source which we're going to create right here and let's create a file actually let's move this text.txt file over to that file so it's our small file so I'm just going to copy it and paste it right here so we're going to read this file and get different chunks out of it so I'm going to first create a buffer I'm just going to call it buff and it's going to be a buffer dead Alec now what we're decides is we're going to create a buffer of that size and then let's now read the file so fs.read the file a descriptor to specify what is that opened file that we're going to read from and then our buffer to fill it and with this data that's going to be returned and then we have to specify an offset which is going to be zero now the length is going to be size because that's how many bytes we need to read and position should be null and then we have our error objects and bytes red so here what we need to do is to just push something into our internal buffer so let's first say if we have an error now in this case there is no callback to call it and pass the error as its first arguments the way that you need to handle errors here and this read method is just by calling the destroy method like this and passing the error objects and I'm going to return it right so this is one difference that you need to keep in mind which is different from that writable stream and that right method that we were implementing here now if we don't have an error what we need to do is to say desktop push now whenever I push here so if I push a buffer that is going to get added to my internal buffer and most importantly this event the data event is going to get fired so let's first create an instance off of this object so I'm going to say const stream equals file read stream and actually new file read stream and I'm going to pass in the file name to be text Dot actually in a string so text.txt and now we could add the data events right here and we're going to get different chunks so let's just lock the chunk for now and let's just push something here let's push Maybe a simple string so buffer it's up from maybe string something really really simple and let's actually put it right here and compare these codes now if I run a code let's see if we're going to get an error so I'm going to navigate to custom Dash readable and now I'm going to run the code well it keeps happening over and over again and the reason is we are not indicating that this is going to be done keep pushing every time we push what is going to happen is that we're going to emit the state events and then when it's done then we're going to run this method again and we're going to push again now what we can do to indicate that this method is done and get to the was the event and yeah it was sometimes it's confusing between end Finish close events and all these that's why I don't recommend you memorizing them and just take a look at the documentation every time you want to use them but yeah it was dot end if I'm not mistaken yeah don't end actually not dead end but an event at its name is and so here we're not getting any arguments and it just indicates that we're done reading so I'm going to just log in stream is done reading if I execute it we're not going to get to that point because well we're not pushing null to our internal buffer we're just pushing pushing and pushing so it's going to run infinitely let me just exit out of it hope it's not freezed I can't exit I guess it was freeze I don't know yeah here we go all right so if I just push a no so I say this stuff push null and let's do it after maybe a couple of seconds so set timeout and I'm gonna push something so I'm going to push the snow after one second now let's run this code and after one second we should now get that done event actually it won't work I don't know what I was thinking because every time this whole method is running and we're not really going to get to this point so rather what we can do is to you know specify a variable here and then add it and then check if we get to that point it just push now so far instead of this just push null and now we're just going to get that and event so stream is done reading we're not really actually reading anything so we're not getting this data event at all so this is how this reads method works there's no callback to call you just indicate the state by pushing and that's all if you push null in the case that we're done otherwise it keeps going and it keeps emitting this data event and yeah Life Will Go On so let's just go back with our codes now at this point if we don't have an error and we read a chunk of that file and now we have it in our buffer because we filled it here in this function now we need to do is just push this buffer into that internal buffer of this screen so I could say this dot push now what we can do is to say if bytes red which is this argument is greater than zero it means that we've read something we're going to do something on this buffer because remember in our previous code that we wrote here in the copy sometimes we get this buffer and sometimes we have some zeros at the end right and if we just try to write it we're going to get some weird knob things in that file so what we can do is to just remove all those zeros and what we can one way we can do it is just use the sub array method and it accepts a starting index and also an and index which is going to be bytes red right now otherwise if bytes read or bytes red is not greater than zero or it's zero we're just going to push now to indicate that hey we're done we're not going to push anything else so let's just take a look at this method so I'm going to run node and I'm going to create a buffer so buff equals maybe buffer it up from text and now we have our buffer which has one a two three and four bytes in it if I call that sub array method I can specify starting index and also an ending index so if I specify three now my buffer is going to lose its last value if I just let's find one I'm just going to get the first item so it specifies to get the elements starting from zero all the way up to this index so that's what this sub array does so if we have a buffer like maybe this and then a couple of zeros so in this case bites red will be three right so we specify that we want from index 0 up until bytes red which is three so up until this point and then we're going to dismiss all these zeros so we have a buffer that's just the right size okay and well actually I just I think we could have done the same thing here and copy so instead of doing all these crazy things we could have just used that sub array method and make use of that white thread and you know just subtract those failed bytes yeah we should have done that but anyway it's also valid so if you want feel free to try it again but using this method and I guess our code will be even faster just using subarray but anyway so now that we're pushing to our internal buffer and we're actually reading something let's also try to actually let's just run it and see what we're dealing with so I'm going to save and I'm going to run code and Yep looks like we have read all the file let's just run the two string so two string utf-8 and yeah all is good we read the whole file and at the end we got this lock so we also emitted this end event right here now in the destroy method here just like before we have an error and a callback and here we could say if we have the final descriptor just going ahead and close that so fs.close this dot ft a callback with an error object I'm just going to do it like this so return callback if you don't specify those curly brackets it's going to automatically return this expression that's going to come after it error or that other header from this first parameter here and what else if we just don't have this let's just call a call back that error object like so notice it's really similar to that thing that we did previously and custom writable we have destroy we don't have final but we have destroy we have construct and it's fairly similar to that one the only difference is this reads methods and also this push and I was just going to put down a comment here and say null is to indicate the end of the Stream all right and yeah we can see that it works pretty darn similar to that create read string that we used before now feel free if you want use it with this read back code just like we did before with that right Mini try to copy this whole code and instead of using Create read stream and create write stream use our custom classes so use files read stream if that was the name yeah fine read stream and also file right stream so you could just change these two and that's it all the rest should work just like before and actually well there's one small difference here we're using the high Watermark value which is by default 16 kilobytes but f is create read stream uses 64 kilobytes so that's one slight difference but you could use both now again in production don't try to reinvent the wheel this is just for educational purpose you sometimes you really have to implement your own streams and it's really useful but if it's already implemented especially in the node.js source code just use that one I mean we could have some issues here you know we are humans we make mistakes their code is really a lot of people are watching the codes and a lot of people are constantly working on it there's a bug that keeps fixing it but if you just come up with something it could definitely be worse than that solution that many people have been working on and it's been there for years people have been testing it so that's something to keep in mind but well yeah here's how you can Implement a real stream if you want now you can come here and read the documentation and you should be able to understand all of it so whereas it's here implementing a real stream here you can again see all the options so you can specify in the Constructor and if you scroll down again you can also not use the S6 and not use the class syntax and then just do it using the classical JavaScript formats here we have constructs and here you can see that it really emphasized that this construct method must not be called directly okay it may be implemented in by chime classes so it's not really mandatory to use this construct but you could and if you do that this is what will happen so you shouldn't ever do something like you know say stream dot constructs or something like that it will automatically be done for you so after you call the Constructor after this is done the construct method will kick in and here's that method that we talked about you only specify your size which is number of bytes to read and when you're done you just push a chunk destroy be able to push chunk and this is that thing that we saw so we have that read and then we push something we push a chunk and then you could also read What's Happening Here and if you push null it means that it signals null chunk and it means that our stream is now done or we can get that end event now we have an error while reading we should just use this destroy method and here's one more example that you can see all right now we're pretty much done with this custom readable class and now let's talk about a duplex and a transfer stream now a duplex stream is just like a readable and a readable so let me just first quickly create a new folder and call it custom duplex now we're not really going to spend a whole amount of time here because we already understand all the things we need to understand now we only have this duplex in JavaScript because if you see here JavaScript does not have support for multiple inheritance so this class is there which extends both a readable and a writable class for you so if we had something like maybe class my custom duplex extends maybe writable and readable so if we had something like this in JavaScript we wouldn't need to have this duplex in the first place all right so when you're implementing a duplex you have two different streams and these two parts work independently from each other and so when you're implementing a duplex stream you have to specify the right which is here and also in addition to this one you also need to specify your read method so you're reading from one place and you're writing to another place now these two places could be completely different from each other all right so let's see how this duplex stream really works so let's say that this is our duplex object now this duplex stream has a readable internal buffer and also a writable internal buffer so a duplex stream has two buffers comparing to a routable and readable stream which only have one buffer a duplex has two internal buffers now these two side of this duplex readable sites and writable sites they are totally separated from each other okay they don't have to be related to each other at all so they could maybe read from one source and write to another source they don't have to be any relation whatsoever now in the real world site you push data like this so you call stream.push data just like that we saw in that underscore read method that's how you push data into a readable stream and the data could come from maybe source.txt so we read from this source and then you push and then you get the chunks here in the chunk variable and you could really do nothing with that chunk you could just let it be there or you could maybe log it to the screen or do whatever you want so you don't really have to do anything with this you don't even have to call this on data you could just keep it in a past State and that's totally fine now again in the Rival side you write using stream.writes so you push data in that internal buffer and then once it's filled you're going to get the data out and maybe you could just write it to whatever.txt file or any other destination so these two sites are not related to each other at all they could be related you could set it up in a way that they could be related but they don't have to be you could read from a file maybe and you could write to a network cards or you could write to another process they don't really have to be related to each other at all so you could maybe change it change this real to a positive State and while it's happening right to your writable site or maybe keep it flowing and also keep writing to it these are totally separated now there are some use cases where a duplex would be useful now in node.js there's one module that's using it the net module TCP sockets and we're going to see that later in the course but I guess that's the only place that we're using duplex in node but something that's really interesting here if we say that we are getting the data into our readable internal buffer like this so we're reading from One Source now rather than just using a data event and just maybe logging to chunks or something like that if we try to write those chunks to our writable internal buffer like this so we get the chunks out and then we write that to our writable internal buffer and then we get the data out now this is a special case of a duplex stream and it's called a transform stream so if you do this with a duplex stream you're going to end up with a transform stream now a transform stream is just inheriting from duplex but it's underscore write and underscore read methods are set up in a way that would just pass the data data around between these two internal buffers now technically if you just pass the data around and don't do anything with it you still have a transform stream but it's usually called a pass through stream because you're just passing the data around and you're not really doing anything with it but if you do some operations here in this passing around maybe compressing the file or encrypting the file or I don't know maybe if it's a text file just making all the letters uppercase or something like that well in that case it would really really be a transform stream but it doesn't have to be like that so you could just pass a data around and it's still well called a transform stream but this is how a duplex and a transform stream work and note now let me try to quickly Implement a custom duplex stream now I'm going to do a lot of copy pasting because it's really same to this custom readable and custom writable feel free to just watch me do it or if duplex is still not really making sense to you you could do this as well so I'm just going to copy this portion paste it in and instead of including the readable I'm going to include duplex and let's now Define our class right here so class I'm just going to call it duplex stream let's see how fast I can do it extends the duplex like this and then we're going to have our Constructor [Music] and what else we are going to have our construct method but our Constructor will now accept some different things so in our Constructor we're going to accept maybe a writable high water mark and also a readable High watermark now again you can see these options here in the implementation where it says implementing the duplex stream here you can see that we have readable hind Watermark and writable High watermark so we're going to pass these two and also we're going to have two different file names so readable maybe rival file name and actually just read file name and write file name and in our super we're going to pass in this one and also this one [Music] and we're going to Now set our variables so let's start read filing should be read finally and this dot write file name should be right filing and we're going to have reads ft and write FD like this and we can now also have the chunks and chunks size so let's also grab these two from our writable so I'm going to grab the two we're not going to do this right count we're just these two chunks and now we have our construct now we have to first open maybe our readable it doesn't really have to be in any kind of special order and oops if we have an error just return the error else just save the read ft into that ft and I'm going to call it read FD like this I'm going to pass it in and then if it is successful let's let's have some callback helps so FS dot open it will be better to use Promises at this point but who cares let's just move on with it and implement this real duplex stream as fast as we can so this dot write file name mode should be right and we're going to have our callback error and right FD if error just call back error otherwise set this right ft equals right FD and call the callback with no error at all I mean I mean I with no arguments whatsoever all right so this would be our construct methods and let's now also Implement our rights I'm just going to copy the rights method from here so I'm going to copy this and paste it right here we're gonna get the chunk encoding callback and let's see rival High Watermark value writable hi I just want to make sure that we are getting that correctly yeah we have this property available on it now we're checking for the writable length we're writing we should write to write ft buffer dot concat this the chunks error just return otherwise it just do the Callback and that's it and that's just pretty much it about and also we don't have this rights count so I'm going to remove it so this would be our write methods let's now copy the read methods so I'm going to copy this one over as well we're going to read but from read FD like this we're going to get to buffer and we're just going to push it to it we do have access to this push method just like before and we also need to well we don't need to do anything else here and read let's also specify our final which is for the writable side so I'm going to copy the final and I'm going to paste that in we have our callback we should write to write ft the buffer the chunks of an error we don't need to increment the rights counts we no longer have that variable and this is pretty much it for the final methods let's now influence destroy now in this story I'm not really going to close the files I'm just going to you know just call the call back and that's it let's just keep it as simple as possible so I'm going to say call back maybe error now technically you should now here close those files but we're just trying to create something as fast as possible so let's now use this duplex stream objects I think it should be working fine but we might have some errors but let's give it a shot so const duplex equals new duplex frame and here I have to specify read file name so let's create two more files so read.txt and another one and I'll call it right.txt so read file name should be read Dot txt and the next parameter is Right finally which should be right.txt [Music] all right now we have our duplex let's just run it and see if we get an error so navigate to custom Dash duplex and I'm just going to run the code right now yep we have an error let's see what this error is the path arguments must be of type string or oh we have a problem here in read file name so see what's happening yeah it looks like I have a typo yeah I have added an e hero which shouldn't be and see if we are done yeah we're done so we're now creating our duplex stream but let's try to write some things to it so I'm going to say duplex.write so I'm going to say buffered from this is a string maybe tried a couple I'm going to save zero one two three and let's also end it I'm just going to say end of right all right let's give it a shot now ER and if you take a look at file now we have some string in it we should also be able to insert a character like this yep we can't so yeah we're writing some things to our duplex stream but at the same time we can also read from it so I'm going to say duplex dot on data we're gonna get the chunks and let's just log it so I'm going to say it log chunk we've got two string utf-8 now we're not going to get anything at this point because well this read file is just empty so if I just put in something so this is some text and more and more and more let's try it again all right now we're reading and we're also writing so reads and we write so this is an example of duplex now the three destination could be maybe on another hard drive really or in any other location they don't have to be in the same directory or anything like that or we can also not write it all to a file and instead write to our network cards or write to any other places it doesn't have to be related to read they are totally separate all right so this is one quick example of this duplexed stream I know it sometimes might be confusing but now that we have implemented this it should now be pretty darn simple for you to understand all right so that's about custom duplex and duplex streams that's pretty much all you need to know about duplex and now let's create an encryption and decryption application so I'm going to create a new folder and call it encrypt Dash decrypt come on all right encrypt decrypt and let's create a new file and just call it encrypt decrypt actually just you know what I'm just going to call it encryption .js all right now here let's try to implement a transform stream and that will encrypt our data all right so now I just want to make something extra clear encryption and decryption so these two are the opposite of each other so encryption and decryption and compression and I don't know what it's called the other way around maybe unzipping so oh we have a helicopter passing by so let me just wait for it to go by hopefully it's not grown in my voice all right it's pretty much gone all right so where are we encryption and decryption and compression and hashing and salting and decoding and encoding these are completely separated from each other now they are really similar to each other because we're in all of these we are dealing with the binary data we're dealing with these zeros and ones now if we encrypt the data we're making it into something that no one else can really decipher what's going on unless they have access to what we did now the decryption is just the other way around so if you encrypt a text to some data so maybe you know this is a data if you encrypt it we're going to get some gibberish so if someone tries to read it they're not going to understand what's happening unless they run it through your decryption application and then they're going to get the original data all right so this is what encryption and decryption is now compression what you're trying to do is to compress a data so if you have maybe a pattern like this so we have 0 1 1 1 1 1 you could do something and say okay I'm going to convert all of this into maybe just a one and maybe a zero and a one so you are technically reducing down the number of bits that you have in that file so you were compressing the data now we usually compress data when we are sending the requests and receiving the responses so that's one good example to compress so we're dealing with zeros and ones now in encryption it's the same thing maybe you could just take each value and add one to it and then you could end up with something that is not really decipherable by a normal person now hashing insulting it's different from encryption but I'm not really going to get into it too much right now you hash something you make a password into something else and you also have this code that you use to you know switch between that's actually you can't go back so if you hash something you cannot go back what you can't do is to get an input and hash it again and if you do get the same values it means that you know your password was okay and salting is something related to hashing it's not the opposite of hashing but it's something that you use in hashing now encoding and decoding now here when you're in code or decode you are getting some zeros and ones and you are converting it to something that is understandable maybe you are converting it into an image or into a text using a character encoding or into a video file or into a PDF file or whatever else that you want so you have to create an encoding application say all right every time I see these maybe display a pixel at location maybe 23 400 maybe 409 so every time I get this binary value I'm going to display this at this location and whatever binary comes after it that is going to be my color code so I'm going to display here maybe a color quote that's red so it could be something like an image encoding or decoding now for string and characters it's just like that one you say that every time I give this value I'm going to display t on my screen I'm going to display G or 3 or a new line character or a new tab character or just a space or whatever but this is the concept of decoding and encoding so all these are some important computer science topics and you should know about all of them pretty much all of them are really important now a note encryption and decryption and hashing and salting these two are done using the crypto module usually and compression is handled using the zalib module so here we have zealip here we have the crypto now you don't really have to use these modules there are some other modules you can use as well but you could also use them and for encoding and decoding well we saw the infamous buffer so buffer is heavily used here but actually we just have text encoding in node.js right so we don't have an image decoding and encoding nodes we don't have a video encoder a note maybe you could install some packages out there or you could Implement your own it's really up to you and it's not that hard as well yeah it definitely takes some time but no just can handle it if you write the code so you could say the buffer set up set the buffer in a way to you know return an image to you or something like that so you need to understand these and also be able to distinguish between them but right now we are trying to create an encryption application so let's see how we can do it so I'm going to first include the transform object from node to stream like this and then I'm going to create my class and I'm just going to call it encrypt which actually extends transform like this now when you are implementing your transform stream well we talked about that the transform is just inheriting from duplex but you're not going to really implement the rights methods and the reads methods and construct methods and things like that if you do that there is a really good chance that instead of using a transform you should use a duplex instead of a right method and a read method and you know destroy methods all of these are done for you in the transform class so they're all done for you you will inherit all of them and you don't really have to worry about them too much all you have to do is to implement the underscore transform method and that's all you need to worry about now the transfer method accepts a chunk and that is what we write to it right so just like before you say streamloud writes if you say encrypt.write you're going to get it here in the chunk and well you're also going to have an encoding just like before which if you don't specify it's going to be a buffer and just like before just like an underscore right we also have a callback now what happens here is that when you're done with the data you can do two things you could either push so you can push the data in and if you do this you're just passing the data around right so if you do this it's just like that underscore reads method we pushed so now this is going to become a source of reading for the next stream that's getting it getting its data from this encrypt class all right so you could either push or you could just call the Callback and specify the chunk and some errors uh what was the parameters let's just take a look at the documentation whereas that's transform so it should be somewhere here I mean duplex so transform yeah it's right here so we could either push or we could just use the Callback right there so if you use a callback your first parameter is going to be your error parameter and your second one is going to be the data that you're going to pass around so you have both options so either this or this feel free to use whatever that feels more intuitive to you I'm just going to use push here in this case so yeah just like that we have our transform stream now we're not really encrypting anything but let's just give it a shot and see if it works so I'm going to create my bankrupt class I'm going to say encrypt actually in this case it's more like a pass through so but let's just give it a shot so encrypt equals new end credits I noticed that we don't even have to call the construct method all of this is done for you all you have to do is to just specify this transform methods and assets now something else that I'm going to do is to just put two files here one to read from a one two right two so I'm going to create a new file say read.txt this is some text to read later and I'm also going to create a file called write.txt all right now let's include the fs module node FS slash promises and that thing async and immediately invoked so const read file handle wait efforts.open and the file was called read.txt we're going to read from this and next one is going to be right file handle if I set open right.txt w ow these two employees let's create our Stream So read stream equals read file handle that creates read stream and write stream is equal to right stream right actually right file handle dot create stream okay and now what I'm going to do is to just pipe these two together let's also move our encryption and site and now I'm going to pipe them together so I'm going to say read file handle dot pipe while in production remember you should use the pipeline instead of just pipe so I'm going to pipe it into my pass through stream or or precisely my transform stream and then I'm going to pipe the result out to the right stream so right stream looks like it's not recognizing this so we have a typo somewhere yeah it should be read stream not read file handle so we're getting a data from one stream we're piping it to our encryption transform stream and then we're getting the result out and we're piping it to right stream if I run this my right text should be should have the content of this read text file so let's give it a shot so I'm going to go to encryption encrypt decrypt folder and just find the codes so you know encryption.js we have one error which is read file handle pipe it's not a function it looks like I forgot to save let's try it again cool so we don't have an error and if I take a look at write.txt file you can see that we passed the data around from read to write so it's kind of like copying now what you can do is to just log the chunks right here so chunk a DOT to string and utf-8 and here we can see the chunk so this is how you would Implement a transform string which is implements this function actually I should say methods and you get the chunks and you push it around to become available to be read for the next stream and then you do the pipe like this but now let's actually do our encryption so the way you can encrypt the data now if you think about it we're getting our buffers and values like this so this could be one buffer so something like this so A4 maybe one a one one or Twenty and so on and so forth so this is our buffer that we're getting because this chunk is just a buffer right and each hexadecimal number you know that corresponds to four bits and also it's actually a number so this 34 is just a number if I open up my calculator if I switch to page 16 numbers which is basically just hexadecimal and if I put that number what was it yeah 34. so I just put in a 34n now here we can see the number of bytes that we have occupied which is eight of course because each hex system number corresponds to four bits and also if you just want to fill all of them you should use the F because if you use it it means 15 and actually just one f it's 15 and it occupies four bits and it fills all the four so if I just keep putting F's I'm going to occupy all of these but our number which was 34 now this is 52 in decimal right so 52 the way you counts right one two three four five six and all the way to 9 10 11 and so on and so forth so this hexatill number is just another way to represent a decimal number okay hopefully that's really clear for you right now now number what was this number 52 okay so 52 as a number is totally different from 50 to as a string so these two are not equal to each other whatsoever they're totally different this 52 oh there's another helicopter passing by someone just wait up for a couple minutes until it goes away all right we're clear so 52 as a decimal number is totally different from 52 as a string right 52 corresponds to number binary 34 and everywhere you go in any computer any Linux machine this is the number that corresponds to 34. it doesn't have to do anything with any character encoding with any operating system with anything this is how it is but this string it could be many different things it depends on how you want to represent character 5. so on Linux this could have one binary value on Mac OS it could have another binary value but actually we should say on different character encoding so if Mac OS and Linux they both use utf-8 they these this 52 will have the same binary number in both devices so here it depends on what character encoding we use and then we're going to get some service and ones but as a number it does not have to do with any of these things it's just a number so hexadecimal 34 is equal to 52. all right so with this in mind one way that we can encrypt our data is by modifying these values so something that we can do is to maybe say add one to each one of these I remember there are just numbers right so we could add we could multiply we could divide or anything now if we add one to all of them technically what we're doing we are encrypting our data right so let me just show you how this works so I'm going to just do this pretty quick using our buffer so rather than just logging my chunk I'm going to say actually I'm going to Loop through my buffer so I'm going to say let I equals zero I less than a chunk that length [Music] plus plus I and what I'm going to do here is to say if we're just going to add 1 to each element of our buffer so I'm going to say f chunk I now the highest value that a buffer could hold is two F's right and two F's is equal to 255. so if it's 255 let me just show you on my terminal nodes I'm going to create a buffer I'm just going to call it B and I'm going to say buffer it from you know so this is now my buffer and it holds these values so something I can do is to add one to the first elements and say buff 0 equals to buff 0 Plus 1. right and if I unlock the buffer again instead of 73 I have 74 and this value that is returned is the value but in decimal and not in hexadecimal all right if I try to convert my buffer back to a string so to strength you can see that it's now TT's ring so it doesn't make sense at all it's not string it's modified right so that's how one way we can do to encrypt our data now if one value is 255 which is the highest value that the buffer could hold because we only have eight bits in each one of the elements now if I try the same thing but instead of adding one I try to add maybe one thousand what is going to happen is that this number will be returned but if I lock the buffer while I'm having 5c now 5c is so five see it's 92. so it's disregarding all value that it has right so I'm going to get something that really doesn't make any sense so you shouldn't really try to add a value to each element that will make it bigger than 255. now generally if you transform the buffer into instead of using 8-bit for each element you try to use 16 bit for each element so now we have eight bits but rather than eight bits you start to add eight more bets to your buffer so each element is now represented using 16 bits now instead of 255 you could use 65 000 for each number so now we could really do some crazy things it could so you could really multiply a number by a prime number maybe a huge prime number and subtract do some divisions and things like that and really come up with something that no one could you know figure out and break and get to your original data so let's try to First add one but I'm just simply going to say if it's not 255 because we don't want to change the base of our buffer at this point if it's not 255 just simply add one to it because if we add one to a value that's 255 we're going to lose some data and then later on we're going to have it we're going to have some hard time recovering that data in our decryption class so if this is the case just say chunk I equals chunk I Plus 1. really really simple so this is basically what we're doing we're adding one to all these values in our encryption and the result is that we're going to have a data that is encrypted we're done just push the Chunk in and that's it and you can optionally also call the Callback function so let's let's try this and see if we are actually encrypting our data so we're reading from this this is our text this is some text to read later and we're writing to this so let's run our code and first make sure that we're not getting an error so node encryption [Music] encryption all right it's done and if I take a look at right well you notice that we're now getting some gibberish it doesn't make any sense at all but the neat thing here is that if you run it now through a decryption so if you read the zeros and ones and if you subtract one from those values so you read it in a buffer and a unit 8 buffer and you subtract one from the values you're gonna get the original text so let's do that pretty quick so I'm going to create a new file and call it decryption .js and I'm just going to paste this in so I'm going to copy all the codes I'm going to paste it instead of encrypt it should be decrypt like this [Music] and so new decrypt read stream now we should read now from our encrypted file so I wish I had called them encrypted file and decrypted file but let's just do it I hope it's not really that confusing so we're reading from right now and we're writing to actually let's not write it at all let's just no let's write it so we should point to our decryption transform string now this will decrypt my value and I'm going to get the original data now I also need to do a check here so I have to say if chunk I is not equal to 255 is if it's 255 we haven't really touched it but if it's not 255 we're just going to subtract one now this way we're going to get the original data so we're going to subtract 1 from all the values actually now we have a different value so we should technically say -1 and FFF because it's the last value we're not going to do anything with it let me also change it here so we shouldn't add one to FFF because then we're going to lose some data and then a 4 if you add one I guess it should be a 5 negative 1 and 11 should be 12 negative 1 and 22 should be 23 negative 1 and adding hexes of numbers is a little bit different so let me see if I did it correctly so 2 23 plus one should give me 24 yep that's right but if we try to add maybe 1 to a you're not going to get a one you're going to get a number that's after a which is B all right you don't really need to know all of these just understand how this transfer method works and that's it we're doing this pretty much for fun but the concept is the same so now let's run our decryption application and see if we can decrypt this data so I'm going to say node decryption and well looks like it's not working let's change it to maybe decrypt it corrupted.txt and we're going to write to it so read and write and let's also create a file and call it d repted am I swelling at correctly yep actually should be decrypted Dot txt it's empty now we're reading from this right actually it's now lost so let's quickly generate that data again so node encryption now we have some encrypted data here in this right this doesn't really make sense but let's now try it with the decryption so notecrypt dot encryption by run it now we're getting the right data and honestly here we're getting some weird characters but that's totally fine we don't really care what they are right you could get a value that's not really defined in a character encoding now here we're trying to open an encrypted file using a character encoding so you could end up with some values that don't make any sense whatsoever to in character encoding application so you don't really have to get everything as a string right you could get some weird stuff like these and here you can see that says actually it doesn't have anything to do with that but yeah so now we're reading from this read we're encrypting the data and writing it to this flight.txt file using encryption.js as you can see here we are subtracting one from all the values and you know what I think I'm doing something no incorrectly uh an encryption now when we were encrypting we are just removing one from each value like this and then when we're decrypting we have 34 now this FFF is untouched we have 84 and then we just add these ones to it and then we're going to end up with the original data right yeah that's right so this is our original data 34 after this encryption which is act one and then we're going to end up with 34 again which is our original data now you could try it with any string you could really let's try to encrypt one of those big files so text small not text gigantic because I don't want to sit down and watch this happen for minutes so I'm just going to try this with text small I'm going to paste it in and I'm going to change the name to read so I'm going to first delete this reads and change the name of text file to read actually yeah read Dot txt now let's run our encryption first node encryption.js it's done now if you take a look at right it does not make any sense whatsoever right but let's try to decrypt it so let's try to read from this right.txt file and write to decrypted.txt file so now I'm going to run my decryption application so now the decryption.js I'll take a second and now if we take a look at decrypted.txt file now we have the original data so it's really actually we don't and we have a slight problem here because we're not getting the exact values you know what I think it's because here in encryption we should either check for one and then so we should do it the other way around if it's not 255 we should add one here right and in decryption we should subtract one right yeah that's right otherwise we have to check for one and not 255 so my bad so now if I try this so let's first create our encryption file so note corruption now if you open up the encrypted file which would be in write.txt well now we're not getting those weird characters but even if you do get some weird characters again that's totally fine now we can read these values so it's not really that crazy but well let's now run the decryption application all right now if I take a look at my decrypted.txt file all right so I had to positively over a sec to really figure out what's going on now this is really not going to work because you have to call the Callback function after the push or you could just use callback and pass null as the first parameter and then data as the second parameter like this it will work without any issues so instead of just pushing it I'm going to call the Callback same for encryption so callback null and then chunk and let's now try it now this one was also our problem we should add an encryption and remove from decryption because we're checking for 255 um but also this was another problem so my apologies so if I remove my decrypted file and also my right file all right so now we only have this read.txt file which starts from zero and goes to six ninths now let's try to encrypt it first so note encryption.js now we have our encrypted file right here and let's now also decrypt it so now the decryption.js now we have decrypted file right here which is exactly the same as the file we started with right if we go all the way to the end that's six sides and if you check the file sizes so we encrypted to read we got the right and then we decrypted the read and we got the decrypted text so these two should be exactly the same and they are so six nine eight and six nine eight and they're both 7.9 megabytes alright so I know that in this encrypt decrypt application we got confused a little bit because well this was a bit confusing first reading from a right.txt I should have named these better but now while we have our encryption and decryption application now in the real world you aren't going to decrypt or encrypt the data just by adding one to it a hacker could just easily figure this out so they could just play around a little bit and quite fast they could figure out hey you had added one to all the numbers so they could reverse that process now real world encryption and decryption applications would really use some gigantic not prime numbers here and then do a lot of discrete math stuff to really be able to decrypt the data so that no hacker could really encrypt it or really break it down and figure out what was the original data so https is one example of encryption and we have so many more out there like so many vpns they are encrypting the data messenger applications they are also encrypting so when they send the data they encrypt and then they decrypt on the Target mobile so if somebody somewhat somehow attacks this network and be able to sniff these packets that we're sending they're not going to be able to figure out what was the original message because we are encrypting all right now sometimes I could just figure out how to decrypt but that will take a lot of time to do now I just ran the application but with text gigantic so this red dot text is now 10.9 gigabytes and it ran well pretty good it was quite fast I think around one minute and the memory usage was really low around 30 megabytes and I also took a look at this decrypted and as you can see I just ran this command cat decrypted.txt and we are decrypting and encrypting without any issues so it's working really good and you can really see that we could really get crazy with streams and do some really impressive stuff if you will you could do compression encryption and so many other things as far as your creativity goes and we can you can do a lot of things with streams now one idea that I could give you right now it could be quite good to work on maybe take it as a challenge also try to log to the user how much of the encryption is done so if you try to encrypt a file that's 10 gigabytes like that text gigantic it will take roughly around a minute to complete so I want you to take it as a challenge and lock to the screen how much it's done so you could like maybe something like this decryption decrypting Maybe and 60 65 percent and then decrypting maybe 69 and then decrypting maybe 72 percent and go all the way up to 100 and then locked in the screen that decryption was done and you could also do it the other way around for decoding so decoding 65 and while definitely you're going to start from zero and you're going to go all the way up to 100 so take this as a challenge and do it now you should be able to do it with the current knowledge that you have now I'm going to give you a couple hints what you can do is to get the total file size out of your read file handle so this read file handle has this stat I guess yep it's called stat and in there there's the file size of that file so you get the total file size so let's say the total is 7 million this number okay so it's a pretty big number but let's just make it something reasonable so I'm going to say 300 billion if I'm reading that number correctly but it doesn't matter so you're going to get the total file size and what you can do here is to in the Constructor to find new property and call it maybe total bites red something like that and here every time we get the chunk get the length and add it to this total bytes red so you keep incrementing this value every time you run this transform method and also in the transform add and if statements so say if you know something like that that says every 1 000 times that I run this method I'm going to log to the screen the percentage I shouldn't try to do along for every method call because in that case you're going to get a whole bunch of logs which is really not useful instead you should you know Define a variable and say how many times we read this we're calling this method so it should start from zero and every time you call this function increment that by one and also do an if statement to say if it's divisible by maybe a hundred thousand just lock the screen the percentage and you can get the percentage like this so you get the bytes right let's say it's this value so this is how many bytes you rent you're divided by the total file size and then you just multiplied by 100 and then you're gonna get the percentage so take this as a challenge and do it and do share me the results I'll be glad to take a look at your code also I might post this lotion on my GitHub or on my LinkedIn or I might not I don't know but take this as a challenge and really try to do this it could be a cool project to work on to you know do an encryption and decryption and also keep logging to the screen how much of that encryption or decryption is done all right so we're pretty much done with screams in this video and definitely from now on we're going to see a lot more of streams in the course and not just in the course actually and other videos that I might publish here on this channel who will definitely use streams where we'll have some videos that we're going to build some cool applications but and then we're going to use some third-party packages so they're not really technically part of this course because in this course we're just focusing on the native notice modules and on the notice itself without installing anything really but we will have those kind of videos as well where we will install some packages and build some real world applications and both in the course and in those videos we will see a lot more of streams all right before I end this video I'm going to just give you some final notes and also talk about how you could learn more about streams and build on your current knowledge of streams now there's something that we didn't talk about in the video and that streams in object mode now typically when you're using streams you're handling some binary data and a buffer but you could also have streams in object mode which is another mode for a stream now if a stream is in object mode rather than using buffers to move data around and by data here I mean series and once we will move JavaScript objects so think of it for example like a database that you have and you have some large Json data some large objects maybe gigabytes of objects and you want to move them to your node process now that would be a good place to use streams in object mode because you are moving a huge amount of data that is a JavaScript object now you could create a stream and when you were creating a stream if you specified the object mode it will be an object mode otherwise if you have a stream and you want to switch it to an object mode it's not safe and also when you create a stream in object mode the high Watermark value will now be instead of 16 kilobytes it will be 16. I guess it will mean 16 number of objects and yeah here we can see for streams operating in object mode the high Watermark specifies a total number of objects so we're not really going to talk about object mode but I just want you to know this and keep this in mind because sometimes you will see some examples of streams and object Mode Nothing is really different everything is just the same you have readable you have all those methods you can Implement stream that will operate in object modes but now instead of just some zeros and ones you're having some JavaScript object so I'm sure otherwise just you reading the documentation and creating some simple applications for yourself and doing some experiments you will also understand streams and object modes now I could also give you an example where this object mode is used uh if you know this PG package and node which is here now you could use this node postgres to interact with a postgres database in your node application so you'll just install this and well we don't have an example here but if we take a look at its website you'll see how it's used so basically just run some queries like select all from users table or something like that and then you will fetch some data from your database now typically if you want to fetch a huge amount of data like 100 megabyte of data or a couple gigabytes of data off of your database what we would do is to use some things like limiting and offsetting and keep making different queries to your database and get different data at different times and then maybe do something with them but you could also do that using streams now don't know postgres doesn't use streams but there is this package I think it was called PG streams node npm let me see if I can find it PG query stream I think it was this one [Music] um yeah I think yeah I think it was this one I'm not quite sure what the name was but here we can see that receive result Rose from PG as a readable object stream yeah I think this was this one PG query Stream So if you install this one when you run a query like this you're gonna get the results but in a stream so this query here is now actually you get the query and then you create a stream off of this query like this and then you are going to get this stream and you could just use it like any other stream so you could use the end you could pipe it you could do anything that we really talked about in this video so it's really similar to all the other stream consumers objects so we have a node and in other packages that we have available and also I encourage you to read the source code and see what is happening if you take a look at the issues and you could take a look also at the source code which is a great way by the way to learn more about streams because just read these Source codes and some of the well-known packages out there that are using streams and you will see how they are implementing their own streams and also how they are using streams I think this package should be heavily dependent on the streams object let me let's just try to search for stream see what are some of the places that it's being used concat streams now here we also have a lot more packages that are there to help us deal with streams better so that's also something else I encourage you to take a look at something like concat stream stream which is here it's pretty popular 15 million downloads each month we're not really going to talk about them but I'm just going to give you some hints where you could Explore More and see some of the packages now a good way to learn about these is to you know find these packages and try to read the source code and see how they are using streams so that's something I highly encourage you to do now this one let's go back to that PG query stream here we could see some other packages maybe like stream spec I don't really know what this is but I think it will be pretty helpful to take a look at it and see the source code and how they are implementing their own streams and whatnot and you can see that streams are really used in this package here it's using this readable from stream and I guess it's it should really be inheriting from it at one point so if we take a look at this file I'm sure you will find something like that so let's search it readable and yep here it is class query stream extends readable and this is how you could Implement your own stream something that we talked about a Constructor destroy read and yep that's pretty much it now another thing that I can say is busboy I'm not quite sure if Express is using streams in itself so I'm not quite sure whether Express is implementing its own streams or it's just using the response and request or streams that are implemented by the HTTP modules so I don't really know but if you take a look at the source code it's just a quick search you'll figure that out in no time so it could either be just using those responses and request objects or it could be implementing its own and one shape or another but definitely something that's for sure is that a lot of dependencies of Express and a lot of other packages that are kind of in the standard Express Library they are for sure a lot of them using streams and actually implementing their own streams in a lot of ways but one thing I can just say about Express if you try to handle file uploadings using Express you most definitely have heard about malter which is something that is really I think it's also an Express standard library or something like that so the smallter is here that will help you to handle file uploads but it's most important dependency is a bus point now which did I search it yep we did so busboy is also another package the source code is pretty minimal and you could I guess you could just read the whole source code in just a couple minutes and it only has one dependency stream search which is another streaming package so you can also read about this and this is pretty popular 5 million downloads weekly so stream search is a module for node.js that allows searching a stream using what is this algorithm honestly I don't know what this algorithm is but I think it's a search algorithm we have so many of these out there so Yep this is another cool package you could look at it it looks interesting I haven't used this but I might one day I mean being able to search a stream it's kind of cool but this busboy itself if you take a look at these Source codes so I'm going to search for busboy GitHub here it is so basically you could use this busway package to handle file uploads and if you take a look now I'm just wondering it's kind of lower level so it's harder than malter but this is how I usually handle my file uploads and my node applications rather than trying to use monitor or something really high level I really try to implement everything myself something like busboy would really allow me to do that so if you want to use this you need to have some good understanding of event emitters and also streams like here.ondata dot on fail dot on close but it's really a cool package malter is using this it's incredibly fast and you can really control everything that is coming through your web server you have pretty much full control using this package and also most importantly what I really like about this is that its source code is really minimal so if you go to lib there's just this index.js which is not that much like only 50 lines and you also have this utils which has some functions but this is pretty much it and then you have types which you have one for URL encoded and most importantly for multi-part which is the part that handles these file uploads and you can see that at the top it's including the readable and writable and then it is extending these and inheriting from these so if you look here you'll see the header parser but there should be something that says extends right here so file stream extends readable and this is how busboy is using streams to handle file uploads the reads method and what else The Constructor and it should also inherit from this writable which is right here so I highly encourage you to read these source code they are really well written and you could get so many Inspirations by reading the source codes and just experiment and use them and see how different packages are implementing their own streams and you know you could I'm sure you could learn a lot this way so definitely don't stop your journey of learning streams and mastering streams at this point definitely read the documentation for yourself it's a great recap definitely check some of these packages out check their source code try to install them try to use them maybe try to even modify the source code and then see what it could come up with and yeah this could be some good tips that I can give you right now and also also take a look at the node.js source code as well and see how node.js itself is implementing streams so I have clones the node source code here my computer and let me just show you where the source code for streams are located at now when you're just starting out and if you want to know where the code is implemented just go to the module itself right at the top almost always you'll see something that says residents right here source code is here in this file and then you will open that up and see some of the other include lines and well you'll you'll go to different Pathways and figure out which part is implemented in what file so let's also take a look at the node source code so lib streams.js now in the node source code things that we really care about one is D source which has the C plus plus code and another one is lib which has the JavaScript code so let's first take a look at the streams implementation which is and streams.js it's right here and you could read it right here it's it's a pretty small file but it's actually including a lot of things so that's why it's super slow model you'd see require this require that and here's a pipeline here's a pass through here's transform duplex writable finished destroy and so much more so many things are getting included from this file and well what you can do is to take a look at the other source codes like here now for streams so part of the code here is in the slab so stream.js is one of them another location is right here these underscore streams but they are just including something else so nothing really is going on here as you can see they're all including from this stream folder which is right here now here we have this consumers and we also have the promises version just like FS promises and streams promises and we also have this web stream which is different from streams but it's really related to streams and so if you go to the documentation all the way down you could see something that says web streams API now we have this in node because this w-h-a-t-w-g which is kind of like a standard it's just an organization that imposes JavaScript standards and things like that nodejs also has this web Stream So this organization said that we need to implement streams in no in JavaScript because we didn't have them and now this web screen was introduced and now node just to keep up with this JavaScript Community also introduced this but it's not going to really replace streams it's not something else to learn it's actually really similar to what we discussed so if you understand streams you will also understand web streams okay nothing is really going on here so a readable stream and well a transform stream just like before you call this transfer method chunk now it's a little bit different you have a controller but but it's really the same thing here you change the stream and this function but as you can see it's really the same thing so we also have streams and javascripts so for example if you're using react you could also now use streams and you could read the documentation if you search for maybe streams Mozilla and streams API so this one is now for just JavaScript and you can see that you could also now use streams if you go to Concepts I guess we should we should go to maybe maybe use Rebel streams and see some examples so here we can see that we have this read method we have start this is also how you implement your own streams High Watermark we have the same thing here so I could really say that with high confidence that I'm sure that you could now really easily understand the streams API as well on the front end and actually let's let's take a look at this stream API and just see the definition here so if you go to streams API and try to read this see it says that's the streams API allows JavaScript to programmatically access streams of data received over the network and process them as desired something we saw before right we received data and then we process them and we did whatever we want with them now here we can see that it says streaming involves breaking a resource that you want to receive over a network down into small chunks right so just you have a huge file or a huge video file maybe a huge image you break it down into different chunks and then you receive the data rather than just trying to receive it all in just one huge chunk so which is really useful and we saw how to do it using the real world stream so if you try to read a file using real stream basically you're just breaking it down into different chunks and receiving the data in different chunks which is incredibly incredibly useful and very powerful so you receive them in small chunks and then process it a bit by bit okay so you have full control over every single bit in your data which is amazing I think it's it's really really great you could do so much work with this so this is something browsers do anyway when receiving assets to be shown on web pages now this is kind of cool so when you are opening a video file and your browser like maybe playing a YouTube video or when you're just actually opening a web page you're waiting for it to load you're receiving the data in different chunks and I can show it to you right now so if I open up my Google developer tools and I go to network I just disable cache and change my network speed to slow 3G I just do a quick refresh you know definitely take a while but you can see that I'm receiving the data in different chunks right for example even this one this Javascript file I'm receiving it in different chunks so you could see how much resources I'm getting here 1.1 1.2 1.4 so I'm not just receiving the whole web page and every single last one of these and just one chunk I'm receiving them in different chunks even each one of these items so this image I'm receiving this in different chunks driving just receiving it in one huge chunk so streams are used pretty much everywhere in computers and your web applications in your web browser and your operating system pretty much everywhere and you can see here that it says sometimes you'll see images displayed gradually as more is loaded right so I'm sure you have seen some of these examples that you open a web page you receive the first portion of the image and then it keeps going all the way down so you're receiving different chunks right that's that's where streams are used here and here most importantly it says it but this has never been available to JavaScript before right so we don't have this streaming Concept in JavaScript notice introduced it and now we also have it in JavaScript using this web stream concept which is pretty new to this JavaScript world so previously if we wanted to process a resource of some kind maybe a text file or a video file would have to download the entire file wait for it to be deserialized into a civil format then processed a whole lot after it's received so it's moving the whole data into the memory and then process it which is super super limiting and we can't deal with huge files but now using streams it is available in JavaScript this all changes so it's a huge concept and you definitely need to master it both as a node developer and also as a JavaScript developer you can now start processing raw data like the JavaScript bit by bit as soon as it's available on the client site without needing to come on I keep opening this document dictionary by doing a force touch on my trackpad but okay so without needing to generate a buffer string over pull up so now you could just receive data in different chunks without needing to you know create a huge buffer or a huge blob or things like these so this is also something else that you also should be able to feel comfortable using streams on the front end so we can now use it with react angular View and whatnot and I'm sure you will have very easy time incorporating it into your application well I shouldn't say easy time but you'll definitely know the Core Concepts and with just experimenting a little bit you know creating something receiving different errors I'm sure that after maybe a couple hours you'll really get up to speed and start using strings in your applications all right so where are we so we were here in this web.js which as we said before it's just something that notice also implements to be more standard comparing to other JavaScript communities but it's not going to replace that node streams it's not going to be a new thing still that stream is heavily used everywhere in node and also if we say that it's going to replace it which is not really going to happen we're not going to have any kind of troubles incorporating it might be some of the syntax might be slightly different but the Core Concepts are the same thing alrighty so we also talked about this streams folder and all the files that are implementing it but there's also this important thing so so far it's not nothing really crazy right it's just some small files but if you don't want to see where the core implementation is you should go to this internal which is here in this lip and then internal and then we should have something that says uh web streams well not web streams but streams itself so let me see if I can find that here so internals and then streams now here you could see the core implementation of those strings that we talked about for example duplex if you open it up you could see right off the bat that it's including the readable and writable and it's inheriting from those two right so here is our duplex class and it's calling the readable Constructor and rewritable Constructor so when you're creating a duplex stream you're just creating both a readable and a writable but in one single object we have to plexify which I guess will receive two streams and then it will give you one duplex stream if I'm not mistaken I'm not sure but yeah here we could see the more detailed implementation of those streams here we have pipeline we have readable.js and these ones are kind of long 1000 lines but yeah this is where all the magic is happening and also apart from this one so we talked about this lip right now these files that we just saw are the only finds that are implementing this stream module but using JavaScript there's also this other part that's implementing the even more core things of nodes but now it's really a low level and now it's in C plus plus so if you go to source and if you scroll all the way down you should find something that says streams right here stream base stream pipe stream wrap so these are more lower level but if you are familiar with C plus plus you could also read them and find some cool things and some interesting things about them and see how a node is really implementing these streams which is quite interesting in my opinion so here we could see that we have all these files here from stream wrap and stream base so these files are implementing the streams but using the you know CP C plus plus side of node and the header filess.h files are header files which basically just include the definition of the functions and the dot CCC files are implementing those function definitions that were defined in the dot h files so also feel free to read these codes if you are familiar with C plus plus and you can see that node buffer.h which is the C plus is also really used in streams so yeah you could see the source code and implementation of streams and nodes and all these files so here are for C plus plus and then in lib and that internal and also on that stream.js file you can see all the rest so this is kind of like a map for you to be able to navigate this source code of node and read more about streams if you want and most importantly I should say try to create applications for yourself so now that you understand the Core Concepts go read the documentation read the source code of other packages read the node.js source code and most importantly I should say it's for you to go out there and create your own apps we saw how powerful node streams are so you really could get creative and do some really cool stuff with them try to handle some huge amounts of data and don't really limit yourself to just files you could also use networks and talk to other processes so you really have a huge amount of power at your fingertips right now so do some cool things with this and I'm sure you'll get a lot more comfortable with the streams as time goes by and we will also see a lot more of these streams in the course and in the channel as well so now I'm sure that your knowledge of streams from now on will keep improving and improving all right that's it about this video thank you so much for watching and I will see you in the next one foreign

Channel: Cododev

Views: 14,255

Keywords:

Id: e5E8HHEYRNI

Length: 375min 11sec (22511 seconds)

Published: Sun Jan 08 2023