How To Stream Large Files Over TCP In Golang

what is up boys and girls in this video I'm going to teach you how to stream large files over a network and golang because if you don't stream your files you're basically going to read them into memory and that is pretty fine if you have small files like simple images or something but if you're sending videos or make it bigger chunks of documents you're going to bump into memory issues because let's say you're going to send two gigabytes of a video and you have a server and everybody is doing the same you're going to bump against the memory limits of your server pretty fast so you need to stream them but before we continue if you liked the videos I'm providing to you please consider subscribing to my channel give me a thumbs up jump in to the Discord Community let's go so basically what we're going to do we're going to build this from scratch so I'm going to say let's make a file server read it's going to be a structure for now we don't need anything and it's going to attach a function to that and it's going to be Avis file service I'm going to call this start let's make a listener first uh gonna be net listen we're going to use TCP uh let's hardcore the port we're going to say 3000 to hold a grill of ports if there is an error and that's not null we're gonna I'm gonna lock fail out actually that's fine so we have a listener here I'm gonna say a for Loop where we're gonna accept connections so we're gonna say connection Eros is going to be Ln except with a t uh let's copy this error handling paste it in so we have a connection heat I'm going to make a function to read from that connection I'm going to call this FS from file server I'm going to call this uh read Loop just like that and it's going to be a loop so we're going to say fort and the first thing I'm going to do is we're going to make it non-streaming so you can see what the problem is going to be and then we're going to make it streaming right so non-streaming going to say a buffer which is going to be uh make me a slice of bytes and let's say 2048 or something some kind of a cool Buffet most of the time I'm using two for the eight what am I doing each uh you could make it small you could make it bigger whatever your needs um just like that we're gonna it so I have a buffet we're going to say here that n is going to be actually we need to read loop from the connection which is going to be a netcon interface so we're going to say con and read everything into this buffer if there is an errors and that error is not nil classic coach Shenanigans uh we're gonna lock fatal out we don't care if we don't give a uh for now so like this and then we have this n and this ad is basically the number of bytes we are written to the connection so we're going to say that the file is going to be the buffers n right everything we're gonna because it could be that we don't fill up the complete buffer so we're only going to grab uh the number we have written from the buffer right it's simple and then let's basically just do a printable here um the file you could print the file out we could also say fmt here uh print Allen actually let's do it at the bottom so we're going to say written percentage D bytes received actually received percent D bytes over the net work a new line make it a print f and specify the N inside of this thing that's fine so here in our start so we're gonna basically make a listener uh boot up uh or accept loopy and then we're going to say go avas read loop from that connection right and another good routine so we can handle multiple requests asynchronously yes I think that's good uh fine fine fine so what we're going to do is we're going to make ourselves we're not going to make a Constructor we're just going to say that the server is going to be and server just like that I'm going to say server start up it's going to be fine it's going to be a file server by the way like this okay so that's gonna basically boot up a server the next thing we're going to do is make a simple function we're going to mimic somebody that's basically sending a file to our server so we're going to say func um send file and we're going to say size right so we're gonna make a valuable size so we can test it with variable with with different sizes so we can see what's going on let's make an error here just like that so what we're going to do is we're going to say that the file we're not going to read it from from uh from disk that's very easy you guys should already know that what we're going to do is we're going to make a random file so we're gonna make a slice of bytes right we're going to mimic we're gonna construct the file here uh we're gonna make it as big as the size we give it then we're gonna say the error heat is going to be IO readful readful we're gonna say the Rand reader we're gonna make a random file rant read it what am I doing into that file here that should be good um if at not nil let's return this guy boom now we have a file heat um send file wait let me quickly think yes okay cool so the next thing we're going to do is we're going to dial right so we're gonna say the connection errors is going to be net dial what's going on ish do we need to specify TCP probably TCP and we're going to dial to our guy which is listening at three thousand let's copy this other thingy paste it then now we have a connection then we're going to say connection right and we're going to write uh the file and add come right let's copy this thingy paste it in and then we're going to say fmt print f uh written percentage D bytes over the net work a new line and look at this and then we're going to return nil and we're already done so what we're going to do is we're going to basically do something very simple we're gonna say uh go funk a gold routine just like that hop and then we're gonna time sleep let's do four seconds maybe time second and now we're gonna say send file and we're gonna make it thousand because we have 2048 as a buffer so we're gonna make a thousand to just to Showcase this now we're gonna say go run main.go I'm gonna wait for seconds boom and we received thousand bytes over the network this is our file uh this guy said written thousand bytes of the network this is the file a complete random bytes hey it's all working fine so that's basically reading it into memory right so the problem is here let's let's make uh so we know that we have each um a buffer Heat or the server side from 2048 bytes so basically what happens if we send uh 4000 bytes right let's try that go run main.go at high speed Vim of course let's go boom so now you see what already happened right so let's go back so it said written four thousand bytes of the network of course because we are actually written four thousand bytes the problem is heat you see received 2048 bytes over the network that's the first chunk of bytes and then we received the next of bytes right so the problem is that uh it's very hard to handle on the on the server side because how do you know how larger file is and and all that stuff and it's basically a non-streaming way you're going to get into issues and the best way to fix all of this Shenanigans is to stream the files some kind of a stream a copy streaming files over the network and I'm gonna teach you how to do that so basically let's start with this send file real quick so in able to do that instead of doing con right heat what we're going to do is we're going to say n is going to be IO copy we're gonna say copy heat and we are gonna copy uh to the connection and we're gonna copy the file the problem is if you say file it's not going to work because the file is a slice of bytes and we need a reader so we're gonna say you're going to convert that to a reader we're gonna say bytes new I read it this file and that's already good and then we're going to delete this and we should have the same effect that's already been fixed because copy what copy is going to do it's basically gonna copy buffer let me show you here copy Buffet so it's basically Gonna Keep copying uh with a specified buffer which is this gonna which is this size right and it's gonna keep doing that until you stop it or until it reach end of file right uh so we're also gonna run into issues with this you will see and we're gonna fix that but I'm gonna show you the whole shebang so you won't choke or come into problems when you're trying to replicate this so we need to do the same thing this read Loop right so we have this Buffet uh we're gonna replace this preferred with a a new bites Buffet and instead of doing it all Ash Bank actually Gonna Leave the error deed we're gonna say IO copy and we're gonna copy two or perfect and we're gonna copy from the connection a streaming file right they are both copying their bytes at the same time receiving and writing and reading right this is bad then we're gonna say the file is going to be the buff right we have the buffer here we're going to copy everything into this Buffet and been able to get the bytes from the buffet we can say bytes and that should be fine um yes okay cool so let's test this go run main go four seconds waiting but so you see we have written four thousand bytes over the network but there is a big problem because where is the rest of our logging nowhere to be found the system is hanging right so because uh look at this read Loop heat and then we are copying here so basically if we have Panic here should panic should Panic right but if we do that you will see that it will never panic some cup some coffee always good so basically you see it's not panicking because we're never reaching that code why is that well that's basically because copy will um keep copying you see basically uh where does it teach copy copies from source to test from the source to the destination until either end of file is reached on Source or another occurs right is it an end of file there is no end of file because it's a connection and it will keep waiting until there is an endo file received but nope there is no end of file nobody's sending an end of file there is no such thing as sending an end of file right so there's a problem how can we fix that well there is something cool and that is called um copy n pretty simple copy and basically means a copy but only copy this amount of bytes right very simple we could say here copy n and we know how many bytes we are going to send it's basically uh if you the size right that's the size we're going to send of course this is an end we need an INT 64. like this and that's all fine the problem is it's still not gonna work why because we here we also need to copy n let me show you go run I mean go we still we also need to copy n on the server side but we don't have a copy n on the server side basically we still have the same problem are we going to fix that do a copy n but now we have a big problem because where is the file size we don't know it right in the server side we don't know how big the file is going to be so we could say yeah it's going to be 4 000 right let's do it hard-coded right but actually we don't know that but hey so we're going to say um Goran may not go you see red and four thousand of the network and now we have that Panic so if you remove that panic and do it once again um maybe three seconds was already enough but hey you see now it's working perfectly fine we have streamed this file over the network right received 4000 bytes and uh written four thousand bytes without reading the whole Shenanigans into memory right we are basically if I I told you the biggest thing we are actually gonna copy is um not quite sure but I can find it but it's basically the buffer that copy allows this and that's the biggest uh bytes the biggest size we're going to lead into memory and send lead into memory and send like a streaming a streaming um Synergy okay so basically we have this 4000 which is a problem so how can we do that well it's very simple what you could do a couple things uh but let's make it simple so what most of the people are doing is if we send file heat right heat we have this copy n right but we're going to say something like a binary we're going to write the file size first right so we're going to say binary right we're gonna write into the connection right I'm gonna say what kind of order it is you're going to say binary a little endian it's a bite order big Indian little Indian is just the order of the bytes most um most of the time you will usually and look it up what it means I think some of my videos already covered that but hey just use little onion and uh set Margaritas on the beach so binary a little onion and we are going to send we are going to write the size right we're gonna write the size and it's going to complain because we need an n64. so we're gonna write that to the connection we're gonna first write it and then we're gonna start streaming so what do we need to do on the other side here is basically before we copying we need to write a read we need to read the file size we just wrote on the other side so we're going to say uh size is going to be an end 64. then we're going to say binary reads we're going to read from the connection right binary not big Indian little Indian and then we're gonna specify a pointage that a memory address of the size which is you could do it with n right we're gonna it needs to read into a memory and then we have the size heat so instead of saying 4 000 we're going to say size and we're gonna do that once again go run main dot go and everything works perfectly fine so we could actually make a big file each uh to demonstrate not too big because my PC is pepega so we're gonna say for example uh let's make a big file not quite sure if it's going to work but hey should work may not go so we're going to make a bigger file uh that's going to be sanded over the wire you see and it's nicely streaming this uh chunk of of bytes between the two servers right not quite sure how big I made it probably big enough so it's still streaming um are we gonna wait until it's done not quite sure hey let's let's leave it there as a nice background so you can see uh this is basically how you should a way to stream big files over TCP connections over connection so you don't read everything and to memory if you like the video I'm providing to you please consider subscribing to my channel give me a thumbs up leave some questions in the comments and very important jump into my Discord community so we can help you with a 24 7 support if you have questions and for the people that want to learn more about this distributed file stuff I have on my patreon I have a complete series on how to make a decentralized and distributed file server where you can store literally everything I call it the Google Docs the decentralized Google Docs it's all on patreon check it out uh if you're interested in that and I'm looking forward to see you in my next live stream or future videos thanks for watching bye

Channel: Anthony GG

Views: 30,605

Keywords: tcp, golang, go programming language, golang file server, golang streaming server, learn golang, go lang, go language, golang tutorial, go programming, golang 2022, golang for beginners, golang tcp socket, golang tcp, golang tutorial 2022, go tutorial, go, golang programming, golang tutorial for beginners, go programming tutorial, golang course, golang beginner tutorial

Id: 82oFmY-Qeok

Length: 17min 50sec (1070 seconds)

Published: Fri Dec 16 2022