Building Microservices with Go: 11 HTTP multi-part requests

[Music] so hey welcome back to building micro-services with go with me Nick Jackson I hope you've had a really really great week and I'm kind of gonna continue the series today so what are we going to start to to look at well we're gonna kind of start wrapping up restful services over the last sort of few episodes what we've done is we've looked at some things like JSON decoding and we've we've kind of looked at how we can do validation and documentation what I want to to start to look at is well how do we kind of handle some edge concepts around this so multi-part form requests and is one and the other well that's going to be gzipped data so I think one of the the important things about multi-part is that I think it's somewhat deprecated let's take a look so multi-part requests this is kind of I think back when everything in the browser was static HTML so you had a very kind of standard approach of doing things you didn't have a the opportunity to use a lot of the JavaScript frameworks like you did now so what you would do is you would basically just kind of put a well a file element into your HTML and you'd have a button so you'd be using HTML forms you'd push it and the browser would do the rest what the browser would actually do is it would send forms in a data format called multi-part form data a multi-part form data what that was it was kind of that HTML oh sorry HTTP data so it's a combination of of text and binary information but it would separate it using these boundaries so you can see here that that here's a request and you know you could have read this straight off the the wire in this format but you can also see that it kind of it's separated so we've got things like a field and that would correspond to an input box on your HTML form and then you would have things like a file and a file would have you be using the input file box but you get things like the file name and then you would get the binary data for the file and all of that would come through the request so this is still technically valid today but I think commonly people aren't necessarily using plain HTML to post data it's also not restful but just on the edge case that you need to take a look at it let's see how we could implement handling multi-part no API so over to our API so what we are going to do is we're going to add a new route and it's gonna be a post route but what we want to be able to do rather than handling the files in a kind of a restful way we want to be able to handle them using multi-part so let's first just create our new route now because multi-part the request is actually gonna have the file name and it's gonna have the ID and all of that sort of stuff in it we don't need to to necessarily capture that here so we're just gonna have a post which goes to the route path and here what we're doing is we're handling that with the serve HTTP function inside of our handler that we wrote last time let's dig into there and start factoring so the handler is designed to specifically just deal with a sort of a restful approach so let's first rename this so let's just call this let's call it upload rest and we can again we're just gonna preserve the signature but we're just renaming this for the time being and then what we're going to do is we're going to add a new function and we're gonna call this one upload multi-part okay so have files upload multi-part and again it's gonna take the same signature just the kind of that standard handler handler functions HTTP response writer and an HTTP request which of course a reference not like that okay I'll put the other comment in later upload multi-part something right so now we've got our function defined what we can do is we can kind of update our handlers to reflect that so let's do that okay so we're gonna do upload rest when it's a restful and when it's multi-part I'm gonna call upload multi-part so what do we need to do in order to handle multi-part request so as we kind of saw in the the sort of the documentation you can see that a multi-part request is going to have a body which is containing all of this information so does that mean that we have to parse out all of these boundaries ourselves not too much there go has a bunch of stuff for doing this on the the HTTP request what you actually have is two methods pass form for dealing with sort of URL encoded forms and you have pause a multi-part form so we can use these methods so pause multi-part form what that what is that going to do well it's gonna read all of that multi-part form data for us it's gonna automatically start breaking up the body on the boundary and it's going to store it for us in to some end to a collection that we can later reference so this is where we want to start so let's put this into our code alright so we're gonna look at what we're gonna pause the form so it's gonna be HTTP request pals form multi-part form sorry because we want to be dealing with a multi-part now max memory this is the if we look at the documentation this is the amount of data which will be held in memory the rest is gonna be written to temporary files on a disk so reading from memory is obviously a heck of a lot faster than reading from a file so you need can make that decision of well how much memory do I want I think if you're expecting a file a large file you probably don't want to allocate 10 gigabytes of RAM for every request well maybe you do maybe you just got a lot of RAM but I think no.4 in terms of max memory what you want to be kind of thinking about is what sort of size of feels of my anticipating if you've got small boxes text boxes and things like that that's probably where you want to be thinking around how much memory to allocate everything else you want to dump that out into a file so I think I don't know what do we want maybe a 128 K or something like that I think that's probably a a decent sort of place - decent place to start some max memory let's do I don't know what 128 times what do for oops something like that now this returns me an error if I'm unable to pass the request so in the instance that we don't have a multi-part former that we can't actually return this information we're gonna get an error back so we can return that response back to the user so we're just gonna do standard pounding if error it's not equal to nil and what we're gonna do is HTTP dot h TT TT TD p dot error response writer our message expected multi-part form data and then we can actually just do our error code so it's gonna be HTTP dot Status bad request and then we'll just return the the kind of the the other thing we probably want to do is let's just log something out to our RAM our error log we can just say expected oops let's just say bad request we can type anything in here later on alright now let's make an assumption we do have the form so what can we do with this object well now we've called pars multi-part form what we have available to us are the the sort of the the functions which exist on the request such as form value and form file let's take a look at those in the docs so form value and form file just gonna scroll up wishing I had an index on the left hand side but I don't so I'm not gonna complain too much there we go form value so form value is going to return the first value or the named component so again if you remember that multi-part then you have a name here okay this is part of the the kind of the header for for each of these these sections so form value is going to return the value this bit here for that given name now it's only going to return you when the type isn't a file so it's just going to return the texture ones in the instance that we do actually have file content well then that won't be returned by this collection so let's implement that so we can get our ID because our form we're gonna look at submitting with an ID actually let's take a look at the form so what we have in terms of a form it's just something in react jazz I really dig react jazz I find it very very easy to to use and and it's actually really really complimentary to restful services so this this HTML snippet here this is kind of what a standard form would look like and I've just written some some rack jazz to do that I'm not gonna dig into that I'm tempted to do a video series of rack js4 go developers but maybe stick a comment in the thing below there if you if you'd like to see that but I just want to run it so I'm just going to run my form I'm using yarn so I can just you can start that's gonna run there and you can see the our little rack Jess you I so this is the same UI as we were looking at last episode it's retrieving the the products from our product API but let's take a look at the the admin so our admin is really simple we're gonna have a product ID and we are going to choose a file that won't do and then when I submit this what I'm going to do is that will create a multi-part request and it'll submit it to my back-end server so back over to that back-end server so we're gonna we're gonna be able to get these these values so we're gonna be able get the ID and we're gonna be again the ID from our dot form value oops home value and then the name it's called ID that's what I have the form identify here just didn't react and we can use this ID so let's just for now let's just log it log dot info and we're just gonna say process form for ID I can choose my ID like this okay so now if I run my code so let me select my priority P I let me just go to the files API product images and I'm just gonna go run that okay already in use because I'm running my other API on port 1990 so let me just change that in the defaults there tonight you know what and I'll run that again okay so the API is now running let's go over to our form and let's hit the button okay let's check out what's going on here so we're gonna go over and we're gonna look at the console and we can see that access to blah blah blah localhost 1991 from origin localhost 3000 has been blocked by cause remember this old chestnut from last time well that's very very quickly implement that into our file server so again you can check out that video where I explain cause in more depth but what I'm going to do is I'm just gonna import a hallo pan handlers and down here we're gonna do a new cause handler go ahead dot cos and that takes a cause option and we want the origin allowed origins and I'm just going to pass that a string of star because I'm happy for any origin now I've defined my course handler I can just make sure that that wraps my main Rooter okay so let's um let's run that again and back over to our web UI so submitting our four okay so that's worked okay so you can see with we've we've submitted the form let's take a look at what the browser actually sent there so you can see the headers you can see that it's um let me zoom this so you can see the headers you can see the the content type multi-part form data there's the boundary that we were using and then you can see the form data itself so I have an ID one and I have a binary file nice and easy so this is working great so so far so good let's go over to our code again we're going to stop that so we're going to make some more changes so what we need to do is finish the the implementation of this what about the file itself where are we going to get that file from well we're getting at the file from the files funnily enough right so files let's have a look at this so our dot poor file and again a key now form file returns multi-part file F file header and an error okay so file multi-part header and an error so I'm guessing well if it it doesn't exist we probably want to just return an arrow so let's just put a generic error in there but what is what is this okay so file header let's take a look so file header is actually in the mime a multi-part package and if you remember back and let's just take a quick look at that but that that header that's gonna be sent with each of the parts how's things like the content-disposition it has things like a name so what was the name of the field that was on the form then you have the file name so I'm getting all of this information inside of the the the file header struct here so in terms of my my code what I need to do now is I need to save this file well I need to do one thing first because unlike when I was defining things with my I kind of my my restful approach was my restful approach what I can actually do is I could specify that my ID was actually a number and the validation that's built into the mocks Rudo is going to ensure that that was the case I'm not getting that same validation here I can't trust the input that's coming in so I need to validate that I do actually have an ID number let's do that let's do that first before we start start processing I'm just gonna comment that out for the time be right so we need to convert this into an ID so how can we do that well what we can do is we can use string cannot a 2i and string convey to I is is gonna return us an error and an integer actually I can I can look at this here and I can say stray error or call it ID error right and then we're gonna get this error here right so now we've got both our bits we can do another bit of validation so instead of just saying if this error is nil why don't we when we do a another error here as well and we can kind of say well if ID arrow is nil and then we can give a nice a message here so instead of a multi-part form data we can say expected integer ID so I'm actually having to do quite a lot more work than I did when I just looked at the the corollary sample and then we can say here expected file so we were kind of once we get through all of that we can actually start to to save our file so we can use the the same sort of function that we used last time for saving the file and let's let's just implement that so we can then just say F dot save file and we're gonna pass the the ID and we need the IDs a string because it actually forms part of our path so we can convert it back or we could just read it again I didn't think about that very thoroughly did I never mind and then we have the file name so where we gonna get the file name from well the file name comes in this header collection so there we go we can see I've got the file name then ice pacify response writer in my request as normal so this is now going to to save save my file there is kind of some protection in this so if we can look back at the the kind of the set up for for multi-part reader then multi-part reader will will actually the form parser will only read requests of a certain size if we want to read requests greater the default which i think is about 10 megabytes and I'm just gonna find that in the in the documentation for you then so okay so there we go max byte reader 10 megabytes then we can we can override this behavior but I think 10 megabytes is is fair for this application so let's just leave it so we've got that form file there let's save this up what is complaining about cannot use as files that's oh of course of course because into a little bit of refactoring so in our save file what we are doing is we're passing the request stop body we don't want to do that so we're gonna pass our file and here let's just refactor this HTTP requests and take an i/o read closer let's just call that yep oh of course the more than one f FF we're just gonna read a IO dot read closer we don't need the body back so we're now explicitly doing that and then here we've just modified this signature so we just refactor that ever so slightly but this some this is pretty much all we need to do so we can look across let's go battle let's run our application and then let's look at this code again so go run main dot go I'm running my API server again I go to my front-end application in my web browser and I'm just gonna upload so there we go uploaded form over here and you can see that I've saved the file for Product ID one and the path is holding so if I look in my folder where I'm saving those files there you go that's the file that I've just uploaded so that's all you need to do to implement multi-part form parsing in Ingo some of the things that you might kind of be be thinking about when you're looking at this and you might you might be like well okay this is fairly straightforward but if this sum is pause my part for this is writing all of that information to disk what if well what if I get a bad ID what if the the ID of of the request is a I don't know a string doesn't correspond then actually go is going to the effort of writing all of that that information file to the disk in the background yeah I mean that that's um that's kind of true this is kind of one of the I suppose the disadvantages of using the the multi-part form approach you can of course implement a lot of stuff yourself so everything that's going on inside a pause multi-part form is is actually pretty much is using the mime multi-part package under the covers so all of the things that's done automatically if you wanted to you could parse yourself so for example rather than reading that file and getting in as a file which is already been written to a disk you can pick up that file as it it's read in as a stream from the actual content body there I kind of thought about doing this and I'm thinking well isn't that a better pattern but but when you kind of start to look at it the ultimate problem is that you end up with the same problem you'll receive a file you don't know what the order is you can't guarantee that you're going to read the file ID before we read the file body so you end up saving them all both anyway so actually multi-part form is a necessary evil maybe but you know I think I think it's pretty straight so just as a quick recap so what we've seen in this episode is we've we've just seen how easy it is to use the to upload a multi-part form in all honesty is a sign of a summary I would still recommend taking a restful approach I think with with kind of modern code you have more control you're not just doing a very straight forward post what you actually have is the ability to manipulate and define the request yourself so you're doing in xhr requests in JavaScript is kind of the typical way that things are gonna work so in this instance calling one or more api's to be able to submit and upload the data well it's it's gonna be it's gonna be pretty standard all of the data is going over HTTP so it's gonna be all HTTP so it's using standard TCP as the kind of the the base level and then of course you're gonna be using either the HTTP protocol or the HTTPS secure protocol so none of that is is any any different yeah I think that's it for now next time and the very very last of the the kind of the series on rest it's gonna be gzip so I'm gonna show you how to - gzip these files how can we make it more efficient - to download our files and then we're gonna be digging into G RPC which i think is gonna be interesting now what I'm also going to start doing and I'm not gonna do this next week because I'm kind of away next week but starting from the week after I'm gonna start looking at kind of a side series it's not gonna be building micro services with go but it is gonna be all things micro services and in those videos I'm gonna look at some things that folks might find interesting such as how would you load balanced multiple services from a kubernetes ingress so that's actually gonna be the first in that video series I'm going to take you through what I think of the the kind of the the two of the nicest bits of software out there at the moment that you can run on kubernetes to load balance your well not load balance your service route traffic to your various different services and we're gonna kind of continue along looking at some of those things and that'll be the the midweek series we're obviously going to continue with building micro services and there's a lot lot lot more to cover but doing it this way it means I can kind of give you a little bit of side information because you may already be micro service practitioners but maybe not necessarily so much with a go for now I want to thank you again for watching I want to encourage you to like and subscribe so that you can keep up to date with the latest videos and yeah it's it's all gonna be great I'm all looking forward to teaching you so for now thank you so much

Channel: Nic Jackson

Views: 5,699

Keywords: go, golang, microservices, multi-part

Id: _7-IhHMptNo

Length: 32min 40sec (1960 seconds)

Published: Sun Feb 23 2020