GopherCon 2019: Mat Ryer - How I Write HTTP Web Services after Eight Years

hello well it's great to be here I want to say hello to everybody and I'll defer the goodbye now because we don't know how it's going to end and I want to make sure that I get out of the way it's a it's an honor to it's an honor to be here and thank you everyone who's chosen to come and spend some time with me given that Dave Cheney's speaking at the same time in another room and so what's it it's too late sit down so I've been doing go for a long time and I really love it you know and I most of that time has been spent writing HTTP services and I've sort of changed things as I go along across the years and some things have really stood the test of time and and these are the things that make it into this presentation I do the go type podcast so you can listen live on Tuesdays put that in your ears and I've got a book and oh yeah blah blah blah so I'm gonna I'm gonna crack on so last year I wrote this blog post and by the way you should write blog posts I really recommend it you don't fall into the trap of oh no one cares what I think it's not true so please do write blog posts in fact I I wrote my first ever blog post and I got a book deal from it and I'm not saying everyone's gonna get a book deal although packed but please share your experiences because we are interested in them even if you feel like no one would be so this blog post went a bit viral it got a hundred and fifty thousand hits which for me that's massive I mean obviously Jesse Frizzell tweets I dropped a bagel she gets that many likes but for me this was big and I was interested in why that was and that's why it's sort of I'm bringing it on tour the blog post it was also the oh I've gone too far it was also the that the most clicked link in the go weekly newsletter in 2018 and check this out look what they say amazingly I mean like thanks what do you mean it'd be trolling me on it want something no but I know what they mean you know there were lots of more interesting and deep technical blog posts some very beautiful ones and very brilliantly written things but at the end of the day you know we do have a job to do and so sometimes just very practical things are interesting and useful so I think that's what that's what this is it's someone put it on reddit and I didn't realize this but you actually can't type nice things into reddit now I don't know when they added that but so I got some criticism some questions some comments some ideas and I've really sort of taken all of that and folded it into this so it's not really all coming from me I also more or less exclusively pair program at the moment with David and Anders and a lot of these patterns we've kind of figured out together as well so it's not it's not just my thinking here it's really been influenced by the whole community and so it really belongs to everyone I guess so how did you use this talk then I think this applies to every talk it's not meant to just be taken literally it's not a rulebook that you have to follow it's not Scripture it's there they're valuable things because as I said they've they've stood the test of time so there is something useful about that but it a lot of this depends on your context as well and you might be operating in a different context and you might have great reasons to do things differently so absolutely you should do that it's really I'm gonna try and focus where I can on the philosophy behind some of the thinking rather than the literal little code snippets and stuff although hopefully that's all useful too software is about trade-offs and it's not a right or wrong so if you do disagree with things we prefer doing things a different way absolutely totally fine and I also recommend if you can try things out for yourself eating your team if you're a manager or a tech lead or you're responsible in some way for the time of the team building some buffer let the team explore these ideas and explore different ideas it pays dividends it's it's really valuable and it's not a waste of time so please do that so when I selected these items that I'm going to talk about and and and really how I come up with and how we end up in this situation I think he's driven by a few sort of philosophical points I just want to highlight them now and then we can and I'll point them out throughout as well so the first one is maintainability maintainability doesn't get enough attention I don't think we spend a lot of time obsessed with how long is it gonna take to build in the first place for us to then deploy right which makes sense because then customers can use it but I mean there's a cost to maintaining code and sometimes it's worth doing things differently if it means you get better maintainability sometimes it's worth taking a bit more time on something if it means the maintainability is going to be easier so yes maintaining code in in some contexts that cost of maintaining can be bigger than the cost the initial cost of actually deploying it in the first place so it's definitely something to bear in mind glance stability also I like talking about this one this is the idea is how quickly from just glancing at some code how quickly can you understand more or less what's going on you know we we kind of I think we have landscape kind of navigation software in our brains from our early evolution which I think we use when when we're thinking and mapping out our projects I don't know that I'm not scientists I mean a computer I suppose but so yes I think dance abilities like is important how how quickly can we figure out where we need to go to do things and so sometimes decisions are made that improves glance ability I think coach should just be boring as programmers we we want to be creative all the time and we want to find cool creative ways of doing everything but actually I don't think we need to do that in our code the code should be boring and obvious because then it's more maintainable it has better glance ability and more people can use it and you think about this conference and how big it's growing and how how Go is growing in general there's a lot of new people around and so we want to make it as easy as possible for those new people whether from other programming languages or if goes their first programming language you know we want to make sure that they can take part as quickly as possible so sometimes decisions are made in that vein - and also self-similar being consistent consistency in self-similarity is helps helps with all of this - so sometimes I will have a pattern that I use and it might be a little bit over engineered for this particular case but in the wider context it might just be similar enough to other things that it's worth doing for that for its own sake so there's some code in this talk and I just want to highlight a couple of bits I don't I use this little is framework for doing assertions it's like it's like testify but tiny testify is enormous and is is the opposite it's tiny it's like testify off steroids that's what is is there's a comedy the source code is in some cases more illustrative but I did try and make it all real so hopefully it doesn't look weird - weird I want it I want to try make it look as real as possible and I mostly use the standard library although I'm not a standard library fundamentalist I'm more than happy to use code that other people have written and see I really see the value in that and Julie Q has a great talk on how to how to pick those dependencies and what you can look for to make sure you being wise in your choices of what you bring in and with that it's code time so let's get in let's get stuck in okay so the first one this actually applies throughout anything I'm writing not specifically HTTP services but I do this tiny abstraction from main I really like being able to return errors as things if they go wrong and you can't do that in main you know sometimes you might have code which is is gonna do something like this where you're gonna you're gonna print a standard error and then exit if you if you have a lot of things being set up in that main block you're gonna have to repeat that lots of times so this is a very simple way of allowing yourself to just return error so it's very simple I stole this from franceska campoy I have a server struct so I have a type which kind of represents the component and the dependencies go into that type so I like this because it makes it pretty clear what this server needs in order to do its job and also it removes the temptation to have these in some kind of global State so you know you instead of having this just as a var in in the package space these dependencies you have them in the server and it makes a whole raft of things a lot easier Peter Bergen said said this he had five words it would just use four of them which is that's very Peter you know but very efficient yeah but he's right so he says if you if you don't use global variables and you don't have this global state then you you cut out a whole class of potential errors and bugs and difficulties when testing and things like this so this having the server stroked is nice because you keep that almost contained I like to avoid constructors generally because really you don't know what's happening if you call a constructor without looking at the code it could be spinning up go routines it could be allocating all kinds of memory whereas if it's just a type you know if you're instantiating that yourself then you know exactly what's happening so there's a kind of very clear expressive readability advantage to not having a constructor but I always end up with one so and it's essentially in here I just create the instance of it and I I set up the routes so I do something that the work that I have to do every time but notice I don't set up dependencies here I don'ts like connect to a database and I don't sort the logger out and do all this other things because I want to use this server and this constructor in different ways so if you have a couple of dependencies you know you can pass them in as arguments here that's ok too but usually you end up with quite a few and I don't really like I mean I refuse to call a method if it has four arguments or more but I don't really refuse to do it obviously I don't chase if I need to call that method so but it defeats the object a little bit of having that server struct right we want that server strokes to list out the dependencies we don't want a big function that has all those arguments in there you can make server and HTTP handler anybody that hasn't seen or read about HTTP handler and HTTP handle of func I recommend checking it out but essentially HTTP handler is an interface and you just have to implement this one method and that's everything you need in order to deal with web ingo notice it has a HTTP response writer and an HTTP request so that's your request in your response that's how you in with the outside world in go so if you put this method on it it means it becomes an HTTP handler so then you can just use your server but notice all I'm doing it in this example is passing execution to the router I'm not doing other logic in here and that's important because it would be a strange place to have other kind of logic so if you do have if you want to do logging or tracing or things like that it's better to use middleware or something that's explicit rather than sort of hide it inside this because I wouldn't think to go to look here it would be the last place I look probably I always have a rootstock go file so generally the advice in go is to group things up by responsibility so you could have comments in a file and everything you need for comments is there and you can have people in another file and everything you need for people is in there and it's I think initially I used to do routing in each of those places too but having a single file that maps out your services like this is extremely useful and if you think about any time someone's mentioned a bug or asked for a feature ask for a change you usually get a URL when it's an HTTP web service that's that's kind of what always comes in so then you need to be able to just go and find the handler that's that's dealing with that request this is a very easy way to do this and it's very glanceable you can see the you can see the whole map of your API in one place so I think that's a real kind of advantage when it comes to maintainability as well my handlers hang off the server like this as methods and that's how they can access the dependencies inside the server so it's kind of it's it's just normal go code isn't it it's just a method and we know we sort of know how methods work and so you know we don't have to use other ideas or other concepts in order to solve this problem when we just have methods so it's nice because it's simple anybody that knows go already knows probably how to do methods and and it's nice you do have to be careful because it's it's worth remembering that every HTTP request that comes in to your server gets its own go routine so you have to be careful about what you do with that server because you could have concurrent code running against it and you want to avoid data races and things like that I name handler methods like this despite how I used to do I I like to think of I like to be more expressive and say for example you know create task handler because that's how I would say it in in real life and flipping it over actually gives gives a nice advantage that the IDS will group these up and the documentation if it grew if it orders alphabetically it groups these things up by responsibilities it's very useful so that's a pretty simple little tip it works in other cases too and I quite like it it's very handy when you want to actually start to find these and look for them it's sort of really easy to find one where you need to be so I actually returned the handler rather than the method being the handler itself if the method was the handler itself it would take the response writer in the request in it doesn't it instead returns this HTTP handler func and that's just an anonymous function that you return after calling the method and what that gives you it's a slight indirection so you know I usually would prefer things to be just I'll just have the handlers as normal handlers but but the advantage to this is you get a little closure to environment where you can do some cool things including setting things up for example we have this thing so we prepare a thing you can do some work to get your hand get your handlers dependencies ready if they specific and of course you can use that thing inside the anonymous function and that's how you can access it remember data races also applies here you do have to pay attention to that if you have any specific dependencies for a couple of handlers that maybe you don't want to put them in the server type you can have them as arguments on this handler method instead and then you can access this in this case the format you can access that inside the body inside the little handler function again makes it very easy to see what this handler needs to do its job and in fact because of the type safety and things we get with go you can't get the handler if you don't provide the dependencies the compiler won't let you do it so it's it's just very useful sort of no-brainer and if you compare that to just having things in global space then you don't really get any of that help here are some more examples of things you can do for example the handle random quotes needs a quota and a random number generator and Tim Raymond actually pointed out that in testing you can create a random source with a predictable seed and therefore you stop it being random and make it deterministic which is it is a nice side effect to you know having the the random thing taken in as a as an argument and again more type safety and compile time checks which we like if your handlers grow too big and you end up with a server that has lots and lots and lots of things on it then it kind of starts getting a bit messy and a bit noisy so you you can have multiple of these you know you can create a server for people and the server for comments and you can decide if you how you want to name them if you've only got a couple of them and the grouping things not necessarily that important but but what's nice is again you can list out the dependencies that each thing needs and they don't have to be the same so it's a great storytelling point of view and and kind of allows people to discover at a glance what's going on I always use now handle a funk / handler actually and if you if you look at the if you look in the source code you can actually see how this works there's a handler interface that just has the serve HTTP method on it then there's a new type there is a function that happens to match the same signature takes the response writer on the request and then that function has a method on it which is the serve HTTP method and it just calls itself so essentially what that means is if you want to create a handler you don't have to create a type in order to do it you can just use an anonymous function and just cast it to HTTP handler func like this now sometimes you have to do some switching between handle funks and handlers that can that can happen a few times and if you find yourself doing that too much then maybe you just want to use handlers if you're using a third-party packages most of the time they sometimes they provide both but most of the time they just implement the handler interface and you have to then do some magic do some switching not magic just some switching you know casting and things which it does look a bit messy to be honest so if you do find that as that's happening too much maybe you need to switch over a little bit but it's not too bad so middleware would just go functions this has been talked about before it's actually I think really nice because it's just normal go code but it turns out to be really powerful so the idea here for anybody that doesn't really know about middleware and go is you have a function you pass in the handler and then it returns a new handler that can then do different things it can do things before calling the original handler can do things after calling the original handler or in this case not call that original handler at all for example if the current user is not an admin we're just in this case say Oh HTTP not found you know we just pretend the routes not even there if they are an admin then it'll fall through and the handler will get called and then they can access that endpoint so yeah this is really nice you can solve lots of problems with this - there's you can you can use this for logging and tracing and things like that you can you can do authentication things very common and what's nice is it's very explicit - and notice that this is also a method on the server and so therefore it can access the shared dependencies as well and it also has its own little closure environment too so you can do middleware specific set up if you need to which is kind of cool I wire these middleware up in the routes go and this this is part of this idea of having a single place that has everything you need that explains the API footprint of your service so you can see very clearly there that this slash admin route requires you to be an admin because it has the admin only wrapped there and that's how you call the handlers - we just that's how you call the middleware which just called the method pass in the handler the one that gets returned is the one that then gets routed by the router and there you go it's all wired up nicely so dealing with data how's it going so far it's alright yeah it's alright alright thanks mark Bates ladies gentlemen if you needed any motivational speaking done here's your man okay dealing with data so before I get into this actually here's a thing that happens a lot we all do this we're all guilty of it we abstract too early as soon as we smell that this thing we're doing could be useful elsewhere then we want to abstract it we're obsessed with dry code and we find any opportunity to not repeat things wherever we can is very natural I think seems but it's worth resisting it's better in my opinion to solve that a few times yourself first and then have a look and see if there's an abstraction that emerges from that rather than trying to imagine it up front however I always find that this is useful and essentially I have a common way of responding so it's really a helper here and it's just called respond passing the writer the request the status code and the data and usually these kind of these helpers usually start very basic and very simple and if you looked at this in isolation you might think I don't know maybe just have the JSON code throughout but of course the advantage is if you want to change the dates of formats you support you can do that in one place and then all of the service gets the improvements so for example is common to check the accept header from the request when deciding what to send back to the client and it allows you to support different formats and you can do that you don't have to do it in the beginning in fact it probably shouldn't but you can do it it's easier if you've abstracted this little bit here I do the same for decoding as well and you see this one see if I wrote this in a project normally David and Anders would say no we're not no nope no that's how he is that's my life the nicest thing you ever said to me was that's not a terrible idea and I think it is Spanish is his first language so I I hope it's just a language thing but his wife assures me no no so this is the same thing we have a we're gonna decode and all we're doing is is asking the Jason stuff to do its thing but later we could check the content type and do different things so you know we could support other formats like not XML but you know other formats now you can future-proof any help as you write with a simple rule of always taking both the response writer and the request even if you don't need them now because this is all you need to deal with HTTP and go so it's all you're ever gonna need and you can yeah you can future-proof your helpers by taking these two arguments always which is kind of cool I love this I absolutely love doing this so here I put the request in response types inside the little closure environment it's very common to just have these in the package space you know alongside the server type but then you have to everyone has to be uniquely named you're gonna clutter up the package space a documentation gets big and maybe you in the IDE when you're typing you type dot and you see the little list of things that's going to be massive if you have not even that many a few different handlers so doing this hides them instead inside this handler so you know as well that they're only relevant here because they're not accessible outside so again use storytelling here and imagine if a junior developer or somebody knew to go comes along in their ask can you go and work on the greeter endpoint well look they have they go to routes go find the greet thing go to the handler go to this and they have everything they need right there so if I have fun it'll be great I love that sometimes if you if you're doing handle a specific setup in this way sometimes it can be quite slow for your service to start up now if you're running this in a situation where it spins up once and it just stays running forever then maybe that's okay but I deploy a lot to Google App Engine standard environment and there's other services that scale down to zero and if they scale to zero it means when the first request comes in then it spins up an instance ready to go and and go starts up very very quickly on App Engine is by far the fastest in fact you don't even notice from an experience point of view you sort of hit the or you hit the URL and it's just various as though it was always running but it wasn't but that can you can start to affect that if you're doing all of your handler setup every time so we can use sync once instead to do this when that when this particular handler is first called rather than when the program first starts up and you just you do it like this you have some variables in the in the little closure environment and one of them is a sync once and then you call sync once you call the the do method and pass in anonymous function and the sync package will guarantee that that only gets called once so remember every HTTP requests get its own go routine so you have to you don't want to be Duke it's possible to have a few come in at the same time and you've got a careful that you don't get data racism and crazy create some of problems so this is a lovely little way to do that and it also has the added advantage of if this handler never gets called this setups never done so in those scale to zero situations depending on your situation that that can be a big save a big time-saving and a big resource save saver testing show of hands who writes tests okay does anyone write tests first TDD much less I can't I can't see you at all I was just pretending I feel bad like I'm lying so just tell you so testing okay yes testing testing is a great tool for maintainability by the way in fact you need it you need it so there's a few helpful things I'm gonna hopefully share here that will talk a little bit about testing HTTP servers and and I think what we'll see okay so the first one is the HTTP test package if you haven't seen this you should definitely check it out it's insanely useful you can create HTTP requests with it and unlike the HTTP dot new request function that can return an error this this one doesn't so makes your tests makes writing tests easier so you can create requests you can create a response recorder by calling new recorder and that is essentially it implements the HTTP response writer interface but it just stores all the basically the response so if you when you call right header to say I'm gonna respond with this status code the recorder just keeps keep the little note of what status code was returned it also catches the body so if you write the body out it catches it in a buffer and you can just a directly access the body and make assertions make sure that the handle is returning what you expect it to return and there's also a server type in there which is extremely useful if you want to stub out third-party api's and I'll show an example of that too but check this out at the docks it's not a very big package but it's an insanely useful so our server that we've made is very testable because it's just a function so we call the new server and you can see here that I'm also then going to set up a test database in this case you might choose to stub it out or mock it depends on your testing strategy but the point here is this is very quick all it's doing is creating the server and setting up the routes you know it's not doing all the setup if you've if you've got that elsewhere if you've got do it inside the sync one so you're doing it inside the anonymous function instead so you cut out everything that's kind of expensive and that means your tests run really quickly and that's what you want you want your tests unit tests in particular should just run very fast because you want to run them all the time and the advantage to that is you sort of get this live feedback from your code as you're going so it's insanely useful and then now look I have aged I'm using the HTTP test package here to create a request create a recorder and then I just ask the server to serve this request and it's kind of simulating a real HTTP request coming in and and then I use the recorder to catch what this is what the handler ends up doing and then I'm making an assertion at the bottom there to see that the status code was ok and you could test error cases here too you could pass in some invalid data and make sure that it's not an okay response you know so you can start to use tests to describe the functionality of your server and the other advantage of having good test coverage of course is you can deploy with confidence so you don't have to to worry about having very careful kind of release schedules and things like this you can sort of release continuously which is not a very nice situation to be in if if the only promises you've made in you about your service are encoded in those tests you know that you can deploy it if the tests pass and actually we we often will just have it anything they get pushed into master we use the continuous integration in the pull request so it runs all the tests in the server then it gets merged into master and then it gets deployed automatically so that that flow is very nice and you sort of if you start by a if you start doing that you it's kind of easy to keep going it's quite difficult to retrofit tests into this so that's why having testing and caring about testing from the very beginning is important the other nice thing is we only have to set the dependencies that we need so if this server had an email sender but this endpoint doesn't use it we don't have to set it it'll just be nil now if we do decide to change the functionality and we didn't update the test first I mean I would update the test first in that case and see it fail but if you don't work like that and you do add the email sender running this test with them panic because that email senders nil and then you have to go and fix the test so either way it's more kind of compile time or it's it's not a compile time is but it's more help from the machines which that's that's what they're for isn't it so now remember if you put the request and response types inside the handler environment then one of the criticisms that was in that read it was well then you can't use those for testing and that's true but you often don't need to and it's not worth it I don't think it's not worth having it cluttering up the package space just just for that and you can do things like this with a little anonymous struct and I'm coding some JSON here so you can see that I have this P and it's a it just has a name and I set the name to my name and then I encode that to Jason and then I go on and do my test so what that's doing there is making it very clear what this end point cares about so it's got great glance ability and storytelling properties of it you know this this P might there might be a person in this project that has a name and address profile picture number of something arms number of arms why not whatever it is you might but you might not need it in this case so we have this greet endpoint it doesn't doesn't care about all those other things oli all it deals with is the name and this makes that extremely clear now you can test this server in a couple of ways and they cut you can think of them almost as integration tests versus unit tests so in the top case here I am calling serve HTTP and so that is testing everything on our server you know it's gonna go remember it's gonna go to the root first then the root is gonna decide which one gets called any middleware on that handle is gonna run all of all of it will run including then hopefully the handler you're doing and so you're really testing the entire stack of in this little world the entire stack versus just directly calling that handler as a method and if you call just the handler it's literally just gonna run only that piece so this is a very nice way of isolating your tests right if if there's a bug somewhere the perfect case is a single test it fails because that's like a laser pointer to what's gone wrong if something breaks and you only use the kind of integration thing say something breaks in the middle where somewhere all of your tests are gonna fail and then you still have the work to go and figure out what's gone wrong so I always kind of prefer unit tests for that reason it helps and if you do want to stub out a third party service you can use the new server here with a little a little fake Handler and what this actually does is it binds to a random port and gives it an it's a real server that you can then make real HTTP requests to locally so that's almost like a level above them the integration testing you you're not just testing just your stack you're actually - you're actually running real HTTP stuff and that's very useful if you if you do want to stub out a third party service you just you just do it like this so you can see that I use I create this SRV it's I use HTTP tests new server and I give it a handler so that handler is what mimics the third-party thing and then you just access the URL from the server to build up the the URL you're gonna hit you have to do that because it gives it a random port so you have to know where to go and get it very useful turns out middleware also testable in a kind of similar way you create a dummy Handler and you then ask the middleware to wrap your dummy Handler and then if you're if your handler if in this case all it's doing is increasing a counter we just want to make sure it got called or it didn't get called then you just do that in inline and it's kind of again all in one place it's just a test function that has everything that you need for that for that case now of course if if you find yourself doing this a lot it might make sense to create a constructor for it for your test code but as you can see it's very clear and you can see that I've my first test here is to see that it's not an admin so I send the incorrect authorization header and I check the counter state at zero then I make it valid and I check the count as one and all I'm doing is testing the middleware it's isolated and it's laser pointer so that's it right boring code if you can use a server struct holding dependencies in there and avoid global state and avoid a whole array of other now a whole slice above the errors that was definitely sarcastic that laugh you're rooting in one place means you get a nice map of the entire service very useful for debugging and maintaining and I use anonymous functions instead of actually having the handlers there directly to create that closure environment and you can do things interesting things within that and you know I use the HTTP package HTTP test for testing there are some other frameworks and things out there that you might like so check them out if you want and I you can also read this on the blog and I just want to say a quick thank you for coming thank you to the organizers you know these conferences are not easy I mean just look at it it's not easy is a lot of work and I don't even know how much work goes into it cuz I don't I don't help but it's a lot of work even from what I see so and the other thing is thank you to everyone that's helping make this community so diverse it really is important and if it wasn't a diverse community I probably wouldn't be involved in it to be honest so I came for the go but I stay for all this diversity these interesting and different and strange and weird so but thank you very much

Channel: Gopher Academy

Views: 42,951

Rating: 4.948566 out of 5

Keywords: golang, software development, gophercon, programming

Id: rWBSMsLG8po

Length: 44min 11sec (2651 seconds)

Published: Tue Aug 27 2019