Building Microservices with Go: 12 Using Gzip compression for HTTP responses

[Music] welcome and today's stream we're gonna be looking at gzipping audio so one of the things that that I kind of been trying to look at is take you through this this journey kind of trying to introduce these concepts little by little so that you kind of build up an appreciation of what's what's going on with the go programming language but but I don't wanna kind of introduce a lot of frameworks immediately because actually I think goes got some pretty decent basics now we've kind of covered a few things in the previous videos now I'll put a link up there of those but we've kind of been on this journey looking at restful api zone and I think when Ellie done a restful api is well I know when Ellie done with restful api because in the next video we're gonna start looking at G RPC so let's sort of dig into this so gzipping why should we care well I think there's a kind of a reason why you want to kind of gzip information and that is because data transfer so if you think about it transferring data and zip or a compressed format is gonna continue less bandwidth it's gonna get to the user quicker so what are the drawbacks on this well the drawback is that zipping and unzipping a file or a piece of information consume CPU it's not free but I think the key thing is that actually the CPU and the time it takes to zip content is less than sorry is lessen than the time saved it's worth the effort the time that's saved in terms of the data transfer is is a lot greater than the effort it takes to to zip a file the interesting thing is well like what he could be thinking about is mobile networks and and I think when you're designing a system if you're building it for a consumer-grade you've got to be thinking about your audience because if somebody's using a browser on a computer a nice fast computer with broadband internet you know things are very very different from when they're using a mobile internet and you could even have good mobile internet so you'd be running a 4G connection or even maybe a 5g connection if you you're lucky enough to get that but still the connection is is is not as great the pings are high or it takes longer to kind of transfer things connections take longer there's a lot more dropped packets you know you want to create a great experience you've got to be thinking about mobile users and this technique that I want to show you today just very very simple gzipping stuff is is how we can how we can do that it's just one of those steps all building the ghost standard library so let's take a look at this so what we have is our source code that we were looking at last time so the this is the same same code that I've been going through and we building building up this this kind of this file service type thing so one of the things that we've done is we've implemented this ability to download images and upload images so if we if we kind of look at how how this how this works let's just quickly run run this application so I'm just gonna do go run main dot go and I'm not compiling or anything like that I'm just doing go run for the moment and what I want to be able to do is I want to download an image so I have an image here it's ID one so if just quick camp on how the guerrilla framework works you have images so this is my root path and then I have an ID an ID is gonna be a number at least one between Norton 9 so that's represented by this regex here and then I specify the file name my file name again I'm going to use guerilla parameter and what I'm doing is I'm saying that the file name must be a letter was contained uppercase or lowercase letters only more than one and it must contain a dot and it must contain an extension of three characters between a and Zed lowercase so with with this what I'm doing is I'm kind of setting up my my file server and we kind of went through this last time but we're using the basic HTTP file server to pick those files straight up off the disk and stream them straight out so let's take a look at that so if I curl it and I'm just gonna curl localhost 1991 images one holding ping and that's going to forgive me this file here I'm gonna output it - OH file dot PNG and there we go I see so that downloaded and there's our file so that's what we've just downloaded really really simple let's just run that again but this time let's take a look at that in in verbose mode with curl so what we have is a GATT request for holding PNG we are going to accept any content types and then the server response is a 200 it's giving us a content length was that 80 81 K or so and we are getting the two content type image PNG all of this is is happening thanks to that HTTP file server and in that's just sending the data now the data as it's being sent is is literally just as a straight old file its streamed straight off the disk here and it's sent straight to my browser there in a completely uncompressed State now what we want to be able to do is we want to be able to implement the capability to have this configurable as gzip so if the client can sub sect zipped content then what we should do is we should zip the file before we stream it out to the browser so let's see how we do that we're gonna do that by implementing a new piece of middleware so let's just add our middleware over here in the handlers I'm gonna add a new file and I'm gonna call it zip middleware dot go okay at the package it's my pack in my handlers package so I can save that well maybe it wants to save eventually okay so I'm going to define a strut again and I'm going to kind of just well the usual thing that I would do with the handle I'm just gonna kind of do that little basic setup so type gzip handler it's actually middleware but doesn't really matter type strut and then I'm going to add my serve HTTP method to make it compatible to the HTTP handlers interface so douji gzip handler thank you correct out serve HTTP and if you remember the signature here it is response writer HTTP dot response writer and then a request HTTP request that's a reference response writer as an interface whoops doesn't return anything so this this is our kind of our basic handler signature so what we want to be able to do is we want to be able to serve information as as gzip so how do we how do we do this so the first thing that we need to do is we need to be able to respond so when when a client an HTTP client requests that it can handle in content in in gzip what it's going to do is it's going to send the header except encoding so if we quickly just open up a web browser here and do sept and coding hopefully it's gonna take us to there we go develop it up Mozilla great resource but accept-encoding request is an HTTP header advertisers which content and coding usually a compression algorithm the client is able to understand so when we don't have this except encoding or we we have something which we don't understand and what we're gonna do is well we're just gonna send plain text back so let's have a look at how we would do that so again we we're in here so we can get that header information from the response writer so let's do let's say well if and we're gonna say well let's say if adopt header and then tele sense is not working as usual not had it or get except coding and we're gonna say if it equals gzip and then let's just do do something simple here and when we find that we are just going to write and we're just gonna write let's just write any old stuff just sort of test this working that'll do now what's this complaining about this is all fine right now you're gonna see this in a second but there's an interesting thing about header in that header can actually take a comma-separated list so if I just try and do a direct comparison looking for gzip then that isn't gonna work it's gonna it's gonna fail potentially or it's gonna be certainly unreliable because if you look here you can see that except encoding we can do this sort of comma separated set up so we need to be able to kind of specifically just just dig into this it's an important thing to to kind of remember just in case that one catches you out we can do this really straightforward we're not gonna do any fancy we're not gonna split it or anything like that we're just gonna do string start contains and we are just going to look for that and we're just gonna say gzip like that okay this is our header if we go over to our main function let's just set that up as some middle web and there we go so on our get Reuter what we do is we're just going to use middleware and we are going to use our or header so we just need to create a new middleware I'm gonna call the MW and lers dot gzip handler I'm just doing this quickly and I can do use MW this of course needs to be middleware function so remember back it's going to be using this middleware function so why do we use that middleware function quick recap for you the reason that we use a middleware function not just a kind of a straightforward approach like this is that we need that next header so we need to refactor this so what middleware function actually looks like is let's say let's call this gzip middleware and then it takes next if you remember back from how the signature for go Zilla essary go real a middleware and that's going to return an HTTP dot handler so to get an HTTP handler in there what we can do is we can just do return and then we can just do HTTP dot handler funk and that will convert a function into an itchy to be handler right there we go so just a kind of a quick change there and then now what we can do is we can use that middleware correctly so again let's let's just run run our application so I'm going to go run main go I'm just gonna correct that error message line 16 so where are we here so we're closing that bracket we are closing there we go closing that bracket oh gosh I thought I'd changed that 30 seconds ago right okay so everything is is up and running there so let's um let's just in a new terminal let's take a look at that again so if we just curl that you can see that that's you know perfectly perfectly standard it's nothing nothing unusual there now if I use a different flag on curl so I'm going to use - - compressed and what - compressed is going to do is it's going to tell curl that we can accept gzip compression and it's going to automatically unzip the the content for us as well right so again we we have this content you see the accept encoding now we're sending deflate and gzip we have written that out and we've we've kind of well we actually just output a bunch of stuff there right so what we we want to do next is we want to to be able to kind of stream this file in a Z it's gzipped data and we can do that because goal has the ability to to wrap a stream in in a gzipped stream so one of the things that that we we're going to do is so if it's if it's gzip the first thing that we're gonna do is we're not gonna write that anymore we are going to create a gzipped response if we can't accept a cheese if format then we're just gonna call next okay so we're cool next oh sorry next served HTTP right now this gzipped response so how are we going to do a gzip to response so the thing is that what we have here is a response writer and a response writer is an interface it has certain methods on it so you can see there we have header let me open this and the docs be a little bit easier to a little bit easier to see response writer had a it has right header as a method as a write method so basically anything we do we can as long as we we're kind of implementing this response writer interface we can create our own response writers now to convert the response writer I'm getting here this HTTP response writer what I can do is I can just wrap it so let's let's take a look at what that's going to to look like so I'm gonna create my own response right I'm gonna create a gzip response right so I'm gonna call it a wrapped respond response writer that's gonna be a struct and what we're gonna have on there is we're gonna have a HTTP dot response writer so we're gonna just have the original response writer and we're gonna have a gzipped writer so the gzip rider is in the compress gzip package of go and again it's one of the standard library features so if we look at that compress gzip there we go you can see that actually you just have a reader and a writer so it's just implementing the standard go reader writer interfaces now the the kind of the nice thing about this is wherever you have a reader or writer you can use this gzipped gzip to read a writer and it will automatically decode or encode the the contents of the stream or the sorry the the the reader or the writer into a zipped format so we're gonna put a gzip writer on there so let's create that sort of idiomatic approach and let's create a new response writer so we're gonna do func new whoops wrapped ris bonds writer and the parameter that it's gonna take there is going to be an HTTP dot response writer the kind of the standard one that we get in the header and it's gonna return a wrap to response writer so in here we can construct that so we're gonna create wrapped response right it whoops not like that we're not tally sounds getting ahead of itself a wrapped response writer let's set our original response writer but let's create a new gzipped writer so let's GW so it's gzip dot new writer so writer takes an i/o writer well what also implements an i/o writer HTTP response writer which is really really nice so now what we have is this response writer and we also have the g zipped response writer now the the kind of the nice thing about this is well we can we can start implementing the interface HTTP response right because it's it's complaining here and saying well you can't do this because well we don't we don't have all of those things but we can we can implement those so what are the methods that we we want to implement well we want to implement the header method so we're going to implement func do wrapped response writer header which is the interface method for response writer and that returns an HTTP dot header so what we what we are going to do with this HTTP header is we're basically just going to return the [Music] original header so where we're not kind of doing doing anything anything strange there where we're literally just going to return the original so we're just wrapping that that function we also need to implement the the right method so again func RW wrapped response writer right and then the signature for that is you're gonna write a byte of information sorry a slice of bytes you're gonna return the length of the the data that you've written and an error if everything went wrong thank you for that intellisense right what are we going to do when we do our right are we gonna just do our w r dot r w dot right no what we're actually going to do is we're going to use the gzipped writer and we're gonna use RW sorry w RG w dot right because if you remember the gzipped writer wraps the response writer so when we call eww right any data that we are writing out is now going to be gzipped and that's kind of like a really single line nice a nice quite little elegant sort of solution we need just another sort of another method here for the satisfy this interface right had a wrapped response writer right header and right header is gonna be your HTTP status code and that is an integer and again we're just gonna do wrw but right header and we're just proxying that straight down lastly let's just put a flush function on there and the flush method is just going to do things like write anything so flush anything which hasn't actually been sent out on the underlying strings streams there so func WR wrapped response writer flush and what we're just gonna do is we're going to do WR dot G W dot flush I'm gonna flush that gzip writer and I'm also going to flush the whoops RW the the original whoops I'm sorry I'm gonna close the close the stream right so this is now gonna make sure that I mean any lingering data is gonna get sent so back up here so we've create that wrapped response writer so now when our string contains gzip what we can do is we can create an instance of our rewrapped response writer so let's just call it well I'm calling at anything here you response writer I'm gonna pass it the reference of the the response Reiter that I'm getting in here and what what I what I kind of want to want to do with that is then just use it in the normal way so when I call next whoops next dot serve HTTP what I'm actually gonna pass is our wrapped response writer and our request rather than sending through the the original one so now any Content that any function upstream of this middleware well it's going to automatically when it writes its data it's gonna be writing it to that gzipped right or rather than the kind of the standard plain text writer and I'm just gonna put a defer on there and I'm just gonna say wrw top flush so I'm gonna make sure that I'm flushing that stream before I exit and that's it so that's a that's a I mean it's a really simple example but let's um let's just see that working hopefully maybe you just never know this show what do we got we've got pointed to an interface that makes absolute sense zip middleware 36 I know exactly what's going on here the cars response writer is not a reference response writer is an interface so you don't want a reference to an interface oh there we go so then yeah yeah exactly so there's a great message there on chat saying that we can double this up to handle deflate too this is this is absolutely true I'm just showing a very sort of simple example of how to use gzip but you can handle any of the kind of the common compression methods in there so the kind of delayed deflate bzr if you're gonna use that and all of those wonderful things right where are we back over here so now let's curl that data and we're going to output it in compressed there to output PNG so now what you can see is that I'm kind of getting this this content there it's going to be gzipped if you if you kind of look at the size of the data there we've we've got three hundred and three thousand nine hundred and twenty three bytes of data which we've which we've written this and if we kind of just get rid of compressed there you can see there there's a massive difference so the we've kind of almost took down the file size by well nearly nearly a third we've reduced that that file size to just by implementing the the gzip on on that data so that's a kind of a I think a really really nice and and simple simple technique but it's a it's a really nice way to kind of just simply zip the the content oh sorry yeah setting the response header to gzipped I thought I was setting that did I not do that no I did not my absolute bad to tell the dot set content gosh my brain content encoding there we go we getting that that data out now so we can see the content encoding DS gzip we can see there the content type the file server is still correctly reporting that it is it is an image and you can see there that the image is is displaying correctly so thank you so much for chat for pointing out that little omission I made there but this is a dramatical reduction right so we've implemented something very very simple just using simple middleware and we've reduced the data size by 1/3 now but that can make it a sort of a pretty substantial difference when we're dealing with poor connections mobile connections the overhead on the server yeah for sure we're gonna be using a little bit of CPU because we're zipping content but I think you'll find that the kind of the balance is worth it like everything you can in need to sort of experiment with these things and don't just kind of blindly take an approach but I think in general using gzipping for your your outbound files is is a really useful thing to do oh we kind of only restricted to two gzipping files no I mean we can we kind of gzip anything that can be written to the to the body the kind of the the benefits it's a bit about whether the client cam can decode it I think that's all for this episode and I want to kind of say thank you so much for watching if you're enjoying please like and subscribe next episode which we're gonna do looking at in a couple of days I'm gonna be starting to look at G RPC I think you're really gonna like to look at this so until then thank you so much and I will see you all next time let's get that outro music going [Music]

Channel: Nic Jackson

Views: 5,415

Keywords: gzip, golang, go, microservices

Id: GtSg1H7SU5Y

Length: 32min 9sec (1929 seconds)

Published: Wed Mar 04 2020