Building Microservices with Go: 10 Handling files with the Go standard library

[Music] hello and welcome back to building microservices will go with me Nick Jackson and it's always pleasant scene so what are we gonna look at in this episode well what I want to look at in this episode is kind of continuing along the lines of what we've been looking at previously we've been working through our API now where I kind of want to go now we start to look at it file uploads actually I can't look at file downloads I want to show you how we can use things like gzip compression with our services so that we can compress our files and make them just really really quick response from from the server we can do all of this with the the gold standard package which you know it's always pretty pretty amazing so that's where we're gonna go I was thinking if we have time we'll look at both multi-part and the ability just a kind of upload binary I think what I'll actually do is concentrate on just kind of simple file uploads in this episode and I think we'll do a separate separate episode on multi-part files it's a little bit more complicated but should we dig in why not let's go so what I'm trying to teach you with this course is is going through the entire process I want to kind of introduce the concepts slowly slowly to kind of get used to using go as a programming language for building out micro services but kind of teaching you the concepts there's a lot of great stuff in the standard library which i think is really really suitable and and and I think it's worth spending time to consider that rather than just kind of diving in and just choosing packages there's some kind of questions around will I look at things like SQL X yeah I actually really like SQL X which is a kind of a package I wouldn't say it's it's full-on object relational model as of flan / M like gone or something like that but I think it's a really nice blend of using kind of just the kind of the base as SQL packages but having the ability to annotate struts and we'll see all of that and in an up and coming episode but for now it's always wonderful you're here and you're joining us and let's look at our old friend package HTTP so what we want to look at today is a way of uploading files and and actually uploading files and go is is pretty straightforward and the reason it is pretty straightforward is because we have as part of any handler this HTTP request and inside of the HTTP request what we have is a stream the request body let me just find that in the docs for you so body is a stream and the nice thing about the fact that body is as a read reader is that it can be used it's very generic obviously read closer as an interfaces combination of the i/o closer and IO reader but the benefit is that it's very versatile having this this thing from a kind of a technical standpoint it's also very efficient because when you send a request to a go service go isn't immediately kind of buffering all of the data that's sent to it it can you gives you the capability of doing it gradually you can read the data as its kind of sent up to the wire which means that we can do a lot of nice things like making sure that we we don't accept too much data you don't stream 10 gig of data when actually your maximum file upload is 3 megabytes and using the reader and being able to control the number of bytes are read very very nice way of doing it it's all very standard you know the kind of the same tool that you will use when you're trying to read and process Jason is actually the same thing that you can use for files so let's see how that works let's kind of dig into that simple simple example so we have our old service so what I've actually done is I've started to scaffold out a new a new service and I've called this one product images it's an exactly the same code repo you you'll find it just where where the other stuff is and the reason that I decided to kind of go and take a new service is that I kind of want to be able to teach you micro services and the kind of to start to show you how we can actually connect these so by being able to kind of build up our estate from a number of maybe smaller simpler services I think in the long run you'll see is very very beneficial but for now we're back into our main go and we're using exactly the same pattern that we used in the product API the responsibility of this service is that it's just going to allow us to to upload images so a couple of kind of just simple things we are going to have some configuration we're gonna be able to configure the bind address the log level so do we want debug logs or staffing what kind of just building on that pattern from earlier and we're defining through environment variables the source of our of our images hands up in the air in reality this probably wouldn't make the perfect file service because we're actually storing files to the local disk with an exception that I have thought about that and the component that actually does all of the file writing and the file reading well it's a it's an interface so we could stub it out so let's go so the first thing that we we want to do is we want to look at how we can send files to a service and and the kind of the type of files that we want to read are not anything cents complicated it's literally just posting binary data with with curl so we do that in exactly the same way as we would you are kind of previous so stuff where we're going to define a handler so I'm gonna call this my post handler and again I'm just gonna do SM so my Rooter I'm gonna do get HTTP dot method post and I'm going to return a su brooder so once I've got my masseur brooder and if you haven't seen the the kind of the previous videos where I've been running through the guerilla marks then I'll put a link up here and you can kind of go back and have a look at those but it's it's nice and nice and straightforward so we're going to do handle funk so and what is the path that we're going to use now with the gorila framework what we've got the the ability to do is to specify the the kind of the Reg ax and other elements that are going to be in our query seseri in our you know URI so we're gonna be able to follow the restful process of being able to kind of define controller so let's say we're gonna call this I don't know images sounds sensible enough and what I want to be able to do is I want to be able to upload images for a product ID so it's gonna relate to our previous service so I can define my ID and I can say that my ID is gonna be a number so naught to 9 and more than one of those so I kind of use that reg X there now what I also want to do when I'm posting my my file is I want to be able to specify the file name now because I'm not using multi-part request I can't get the file name out of the HTTP request so what I'm gonna do is I'm just gonna use a parameter so there's a kind of an argument I suppose around whether you would use multi-part or whether you would just kind of do the ability to handle files as an input stream and and I suppose it depends who your user is if your user is as a human web browser then multi-part like we'll look at in the next episode is probably going to be the the best approach if you're dealing with things like pure API to API then I think actually just posting straightforward body could the data be stored in a database yeah I mean absolutely it could there's no reason at all you couldn't do something like that you can store the data in in a blob in in a datastore I think one of the kind of the more I suppose modern or traditional approach modern non traditional or modern approaches that folks use these days is that they they kind of use cloud storage so if you're hosted in AWS gcp or a Cheryl or on digitalocean all of those clouds have the ability to to give you access to cloud storage which is very very inefficient very very inexpensive and very very simple to use you basically just have an API and that satisfies the dependency of a micro service where state can be an issue for you the as I say the approach I'm taking today I'm storing it on the file but I've already thought about the abstraction ahead of time you could rewrite the interface that I'm using for file storage to s3 or wherever you like very very very very easy so we're gonna post a file name and I want to call that parameter file name but what I can do is I can actually use in gorilla one of the nice things about gorilla is that I can define a regular expression for my my gosh I did not want to do that a regular expression for my parameter so I can say that file name must be must satisfy this regular expression which is made up of letters Ada said lower case and upper case more than one letter and then it must have a dot and then they must have a three character extension between a and Zed not the most perfect file name I will and of I will give you but it's um it's it's absolutely fine for for this simple example you can kind of change that out to whatever you want and then what we we want to do is we're going to pass it our handle a funk so I've kind of set up the sort of the the basics and we can we can have a look at that and start implementing our handler now so we have a file handler and our file handler has some parameters it takes a file store so previous episodes I've talked to you about dependency injection and how how useful that is I really like using dependency injection of my handlers because I'm passing in this files well files in this case is a strut but it could be an interface that means it's replaceable so I could replace files which anything satisfies the same interface and it gives me the ability to test ability so when we look at testing you'll see how this pattern I use very effectively I think to be able to write unit tests for the logic inside of our handle ism so we've already got our kind of our survey GDP set up there and we've got a logger and then we're not doing much but like what are we gonna do well we can we don't need to sort of validate that the file name and the file path but we can kind of just then save out this file because we've defined that Reggie axe in main go oops here we don't need to validate again that file name as a parameter is gonna match this regular expression if file name from MUX VARs is not null it will satisfy the regular expression so the same as the ID so I can write a you know a very very simple check here I can just say if I D is not equal to blank string and well I can actually say if I D equals blank string whoops if I can press the equals key or but file name equals blank string and we want to return an error so I'm going to do HTTP dot error and I'm going to do RW and I'm just going to do oh well look at this I could just call this function here invalid URI nice and easy F dot and violet URI and that's just writing out the the log so we'll we'll see that in action let's give this a little a little bit of a test so r dot u RI dot string that takes the response right okay and we'll return all right so let's give that a test let's take that that incremental approach and give our application a quick test so this you can find this this service in the product product images folder okay so we've got our our service always the keyboard never anything else so I've got my my file service now so I can I can technically kind of upload a file and I've got test dot PNG there so let me let me show you how I'm gonna do that I'm just gonna use curl so I can just do curl and I'm gonna do localhost 1990 which is where my app is is running out and I need to quit a path so I'm gonna give it a path of 1 and an invalid file name and it doesn't doesn't really matter but let me let me just specify my file tests or PNG so I get 4 found because we said we were gonna put this root this is an images it's not gonna find it and the reason it's not going to find it it's because I'm using an invalid invalid syntax so if I I kind of go through that and this is kind of one of the benefits of of using something like gosh go Rooter but it gives me the ability to to kind of do all of this this really nice nice stuff you can see there I'm now validating my files but I'm getting an error because I've I don't have that that image corrected so we're good so far so good so let's take a look at what's going on so I'm uploading my my file and you know my file upload handler is very very straightforward there's not very many lines of code here and I'm going to this function here on calling save file well what is what a save file do it's a it's a really kind of straightforward and simple simple implementation all say file does is there's a file path join and then it calls my my file store component so I'm just building up my path I'm taking the the ID and the the original kind of file name and I'm just building that up into a physical file location path and then in file port path in my file store there's it's a pretty straightforward pretty straightforward component and if my cool language server was actually working we'd be able to see that there we go so we have this interface it's a storage interface it has save path file and then if we look at the implementation all we're doing is writing a file to disk so it's it's not particularly clever there it's just using the standard i/o to to write a file and I think that's kind of nice so again because I'm doing this injection I've got this substitution I'm basing things on an interface I'm using local here I could use cloud storage I could use whatever I want so let's um let's let's take that and give that a little bit of a whirl so and finally okay sorry about that so if we look in file store we can see that we've created the folder WAN and we've uploaded that file there so the the file we've uploaded which was actually tests dot PNG we've we've just kind of given it a random a random path so if I just let me just open up a new terminal rather than kind of going to that one just do that curl again but whatever I kind of choose the the file to be as here it's gonna save that into this folder here and it's just doing that because of the way that I've got that set up so that's that's in its absolute simplest how easy it is to to upload a file now if I wanted to do something like restrict the size of the file again if I'm using like a multi-part stream then I'll show you next time how that is is automatic but all of these things the kind of one I'm when I'm saving that I'm passing the response I which is sorry the read closer and in my file save all I'm doing is using OS dot file create and with the file path and then I'm using copy if I wanted to to restrict the number of bytes instead of using copy I'd be able to do something quite quite simple they're just doing kind of a bufferedreader and then just looping over that file reading a certain number of bytes at a time once it exceeds my maximum that's it I can bounce the connection kind of one of the things I suppose is ask is why not just use the content length you can use content length but not all servers will actually validate the content length of the body equals what's defined in the the kind of the age we had a it should that's kind of the protocol but I find that it's always best to validate the amount of bytes that you're trying to write to a disk as opposed to trusting something like the content header and validating that what the content header said is actually what you've got so we've got that so far so the next step that we want to to kind of look at is how to get files back and and I think with gold this is actually really really really easy because we've got some wonderful sort of methods inside of a net HTTP so let's define our get file server okay so we're gonna create again a new handler so I'm gonna call this the get handler it's gonna be SM methods HTTP a dot method GATT I'm gonna creates a brooder some views using the standard gorilla get handler dot handle func now what's my my path going to be well actually my path is going to be exactly the same I want to be able to have my kind of controller set up so that you specify images is the kind of the route path and then you specify the ID of the product for the which the image relates and then the name of the image so I can do that and then I'm gonna pass my my handler now the nice thing about using kind of the default methods is for serving files I don't need to specify any specific handler I don't need to create my own because there's already one that exists for me and that is HTTP files server so let's just have a look what that is in the documentation so files server when we find it there we go so what file server does is file server is an HTTP an implementation of an HTV file server so it it returns you a handler which allows you to write files to the response file hand server automatically will read the file from from your disk at the the location that's given it will automatically determine the content type and set the content type header for your file and it'll it'll just automatically wrap it up in a response stream for you and it's it's really really easy to use so I use file server and file server expects a directory so the the directory that file server is expecting you can use HTTP directory and what HTTP dir does is it's gonna kind of give you the location of a directory during file system it kind of converts all of those URI paths into filesystem paths it's a nice nice and simple piece of code we can see here actually the implementation and I really recommend you you kind of look at the implementation where possible to some of the goal code it's got some great ideas of getting some some nice patterns and practices for your own stuff but let's see how we can implement that so we're gonna do HTTP file server and what we will do is HTTP dir and we'll do the location of our our directory so we're gonna we're gonna use our base path and we can do that like so now when I kind of save all of this up and I go and run it then what I can actually start to do now is serve those files back so remember we've got all of these files we've got tests we've got a staff we've got whatever's let's run our server and see how easy it is to do the the file server run that there there we go running okay so we we kind of show how we can post files there but we can also retrieve file so I can quite simply just do that curl again but this time I'm not gonna specify my data and what do you mean file not found why am i why am I getting file not found well that doesn't make sense does it there's a little bit of a gotcha here you see what is going to happen when you're sort of issuing this request is that file server is going to request the location of the file from HTTP dir and the value that it's going to pass to the file is going to be the path here so it's going to be images / 1 / test of PNG now if you look at the the route where we've specified the route to be base path so in effect what is happening is I'm trying to do a lookup of a file path images / 1 / test PNG from the the kind of the base path of file store well there is no images folder in file store so that's going to going to fail but that's kind of just the way that it works if I'd routed that into kind of a most simpler just like a dots path and there everything worked it would be fine thankfully there's there's a way around this we can again use something called HTTP strip prefix so let's take a look in the docs what a strip prefix do don't want to print that I want to find it so what strip prefix does is it returns a HTTP handler the room that serves HTTP requests by removing the given prefix from a request URLs path so let's have a look at an example here or there's a surprise look at that the example is actually what we want to do it's HTT be stripped prefix around an HTTP file server I wouldn't be surprised to find that sort of strip prefix was was very very specifically built for this purpose but it works beautifully so what is the prefix we want to strip while its images because that doesn't that doesn't exist in our path and then we're just gonna pass it the handler so that's HTTP there so let's run our file server again so we're gonna run our file server and we are got product images we're starting our server and we're gonna bind address 99 t90 let's do that curl again now this time the curl is working because we've done that strip prefix on the file server so that's that's actually really really nice and what you're seeing there is is obviously a load of a load of garbage it's it's some just the kind of the binary representation of that file we form what is the file type file types image PNG so it's actually detecting it's correctly so the nice thing about the HTTP file server is HTTP file server has read the file from disk and it's it's automatically kind of determined that it's a PNG and therefore it's it's it's kind of written that out to our folders and our disks and I'm out of time but we've kind of started to make some head rows into this I think we've kind of seen some interesting things we've seen that you know it's pretty straightforward to be able to upload files because respond stop Bodi is just a something that you can read from as a stream it's just an IO reader Rio Rita closer and we've seen that the HTTP file server and the HTTP directory gives a lot of flexibility around just that kind of simple ability of well just like you know serving very very straightforward and very very simple files but what I want to look at on Sunday in the next episode is just polishing this service up so let's look at how we can implement gzip compression so let's let's zip those files before we send them to the user let's speed up all of that process and let's look at how we can implement multi-part requests because a multi-part request is very different from the way that I've been uploading files I've literally just been uploading the file as part of the response body the browser doesn't necessarily work in that way the browser's gonna want to send an HTTP multi-part forum post and we'll see how that goes got built-in handling for those as always I thank you so much for watching I let it up this video and I will post it up in the links I hope to see you again if you enjoy like and subscribe smash the bell button for updated content and please in the comments below req any content that you want to see as always have a great evening

Channel: Nic Jackson

Views: 8,018

Keywords: go, microservices, golang

Id: ctmhYJpGsgU

Length: 30min 28sec (1828 seconds)

Published: Wed Feb 19 2020