How I Built Microservices In Go In 30 Minutes

like to welcome Florine from JetBrains thank you for coming here my name is Florian I work for jitteriness as a developer advocate currently doing mostly go lands or go ID but today I'm here to speak about something else I'm here to tell you how I build micro services in an NGO in about 40 minutes or so half an hour and like I'm just going to jump right through the code because this is going to be a live coding session and it's also the first time I do it so let's see how it goes right so this is not micro service right it's just the hello world basic application that you get and see and as you can see you can actually run it and it will print hello world at some point just note my laptop is one of those ultra low voltage laptops that doesn't understand CPU or anything so like go compile takes a while but it works so yeah skipping onto the actual code like usually when you have the hello world right like the first thing that you want to do here is like oh I don't want to too hard to-- this right so you'll you'll see something like oh i just convert this store constant and i call this let's say the message that i want to have right so now I have that okay so this is still working compiling and Sun so let's transform this into an actual micro service something that uses Pyrus I need HTTP which learnt recently so first thing you would need a MUX in something that actually takes your requests and routes them accordingly right so we can say oh I want a new Macs right and now I want to actually use it so it's gonna be something like that like a variable right and then I want to say something like I want a handle func to actually handle the requests and for this example we'll just use something like I want to handle the requests to the home page and I want to have let's say the classic interface - like the classic function to handle this let's move this to the internal and instead of like writing the hello world will just write the message as we normally do right and [Music] now this one needs still a bit of work because we need to do HTTP listen and serve right so we need to give it an address and we need to give it an address on 88 let's say on all addresses on my machine and I need to give it the marks and now like this is the basic hello world when it comes to HTTP servers if I run this it's going to work there's nothing printed on on the screen anymore but if I go here and if I do run I can see if so I can see hello world written by HTTP and I see the the headers that the request generated right so so far so good let's make this a bit more useful and I'm one trick that I picked up while writing though is that it's always a bit faster for you to provide certain information about your response rather than not so this is sort of like a micro optimization if you want but it's a good one so let's say we do the right header because we know it's going to be a status okay right and we also do something like we want to give it a hint about the content type the content type detection ingo takes quite a while if you don't provide it especially on like bigger responses because it needs to detect like okay are you writing a JSON stream or are you writing a an HTML page or XML or whatever so in this case sorry HTTP there where I forgot my shortcut for this one I've done play where's my HTTP had a sorry this one later content type so we can provide it the content type and like I said this one usually speeds up your request when you'll see it if you benchmark it on my machine where it took 36 milliseconds to actually write hello world it's not gonna be that obvious but we can try so we compiled and we let's say we run this again sorry my laptop is not meant to do like that so it you can see some some difference it's 25 milliseconds now right right okay so next this is not really a micro service and there are some things that we are not doing here so for example how many of you didn't know that listen and serve returns an error okay so you should handle that error in case you don't know about it mainly because you can you can have cases where for example your port is blocked by another application and if you don't handle that then your application will be like oh I can't start and like just crash but you won't know why like purification can be perfect like 100 percent coverage 100 percent tests working it just has to start right so let's say we don't want to do that and let's load this as a fatal error and I said something like I know server failed to start right and we can provide it in the error as well right so now we handle errors because that's what we don't in go okay so the next step would be let's transform this into like a more resilient web server how many of you have have seen the blog blog posts from CloudFlare about how to serve pages on the internet especially when you are using TLS ok one person I expected more but it's good so I'm not gonna write like I have to predefined configurations for this one because I'm like I just can't remember all of that but it's gonna be in the final repository and you can use it this is the address where you can actually find the blog post it's brought CloudFlare and it's called exposing go on the internet and if you go there you can see all the options that I'm adopting here and like it tells you how to set your ciphers what's your minimum TLS configuration and so on to actually make it the secure web server and the next step so once I've done that so now I have my TLS configuration to be able to serve HTTP HTTPS traffic I want to also have a custom server because the standard built-in need HTTP package does have a already created server which is the one that's behind listen and serve but you don't necessarily want that all the time and you don't want necessarily that one because you let's say I want to set a different time out for reads or for rights and those timeouts are by default quite large I think ten minutes and I'm not sure if you want to have a timeout on a request and in on an inbound requests for ten minutes because then someone can actually mess up with your server quite badly so let's let's move the service here right now we have the server and we can change this to a SRV and then SRV doesn't take any parameters so we pardoned our server already in few lines of go more importantly like if I if I look at these these a couple of lines of these few lines of configuration you can also be abstracted so I can put these in a separate package that I can always use so let's say like I move this into a new server right so for now this kind of works it's it's in some function and Sun one thing that I would like to do now is to add an environment variable here and read it because it's going to be easier for me to run it for example from from go from a docker container and we'll jump to the container later on and let's just add some environment variables right now and we'll jump into them in a second so I've already predefined these right and I have this address now I would like to pass that address always to my builder function so that I can change it just from the configuration and I don't have to like carry where I do the change from and server address and this one is a string okay so now that I have a server address here this one still needs something so I said TLS and HTTPS right but we use where's the configuration so we use the semester which means still HTTP so let's convert this one to a listen and serve TLS and this one accepts the certificate that you want use for the personal purpose of the demonstration I'm actually using some self-signed certificates because yeah there's I'm on localhost that it's fine and I'm gonna read them from from the disk rather than like fetch them from somewhere else like a net CD storage or something similar but yeah that that one is a new depends on how you provision your infrastructure and we can do something like this so we need the third file and we need a key file as well so now that I have those two in place hopefully if I've done everything correctly this one still works and let's change to let's do all requests on HTTP this time around okay and that's it I'm on HTTPS now you can see I've made a request to HTTP localhost I've used the Etsy host file to actually map that host locally and it works so in just a few lines of go code like what less than 68 you can have a secure web server already in go and it's configurable to how you want it I also said earlier that probably you want to use this as a separate package so let's just move this one to a different package unfortunately the resolution doesn't seem to help me here okay fair enough we'll do this manual way okay so let's create a new package here called server so you can reuse this pretty much anytime and because we are in the server package we can just call it new there's no there's no need for that so now that I I've done that change right everything should be back to normal and like I can reuse that package now safely if I want I can test it but like right there's not much to test there right it's just configuration if you want to test ofcourse its structure values correctly and yeah that's fine if I'm not sure if that's going to be helpful or not okay and the other thing is we still have this built in function here right disclosure that we want to move so first let's move this somewhere else outside let's call it home handler right and yeah this starts looking a bit more like a go service and I'm gonna apologize for this but I actually think it's gonna be easier to exit the presentation mode for a second because I really want to move this one into a different package so you know the idea of packages is that you want to structure your code on functional units and you can see blog posts from Peter Bergen from I know her by by the name jbd or rocky hill if you if you know her by the Twitter handle there Bill Kennedy wrote a couple of blog posts on how to structure and how to think about your packages and the idea is that you want your packages to express units of work not necessarily type of work so I don't necessarily want to have a package called handlers I want to have package called homepage I mean it's a bit forced as an example but you can think of a package named checkout right where where you have all the checkout logic and that doesn't mean necessarily just the handler the HTTP handler that takes care of the request but you can also have their the database interaction part or you can have the template rendering part so that everything is encapsulated in a single package and it's easier to see and that's that's what I want to do if this one so let's let's move this onto a package called I don't know homepage right and I'll move it wait I need to move the message as well because we actually depend on the message constant and I don't care about that anymore to be in the main package so now I have that package right and it's easier to [Music] group here everything that's related to the homepage however what I found useful when when I was working with refinery services was that it's better not to have functions directly because you probably will as handlers because you probably will have dependencies on other packages so for example you have a dependency on the on a database or on the logging package and you want to somehow inject those functionalities in it I could go and say something like oh I want to log here I know let's say print line I'm starting the server right so I can do that and then I can do something in the home handler which says oh I've received a request right so now I use the log package in two places but I don't have necessarily control and what happens with that logging library if I want to now let's say log to standard error instead of standard out or if I want to use a different format for the logging I need to be careful to either the package to be well-written enough so that it can process that request from me as a user or I need to do the work twice right so I was saying earlier I want to inject that dependency on the logging package and the built-in logging library has something called a can actually create a new logger so you can say log that new right and you can choose worked right for example and I want to write to SD out I want the prefix let's say graphic on UK and I want the standard flags flags and I want the short filename because you can actually get a file name of of the line that prints that calls the log function so you can make your life easier debugging and generally I found that in my code or I can trace quick enough from where the log line came from by the file name so I can do short log short filing yes so short file name right but again how do i how do i send this to the to the home handler let me check this here and change this year and now if like everything in main is fine and i have to prefix i have the flags but i can't use it in home handler so exchange this a bit in home handler as well first and foremost we can create a type structure let's say and in this case we can say something like I don't know handlers it's not a very pretty name but that's the best I can do right now and we know we want to use a logger inside here which is of type logger I want to move this one so I can remove the handler now because it's going to be redundant and I want to move this on on a pipe so I'm gonna have H handlers and to use a pattern that's pretty frequent in go like you will see functions being used as constructors and in that case let's say we want to write a constructor for our handlers you can call it new handlers and wine you and not make for example is that if you look at the standard library new is used whenever you return a pointer and make his return when you return the actual type not the pointer to the type and that's a pattern that you'll see for example if the new and make functions and if you look at other packages they use the same naming convention for them so you can do this and then I have the handlers here I said I want to dependency on logger and I want to do a return here logger right so now I can build the package oh and that's a good point so now the package is more usable I need to build it here as well so I can say something like Oh give me a H which is a home page you and I can send my logger and that's how it the dependency injection go and especially when the request has so when the request has a dependency on on a data that's trust request boundaries right I can do as many requests as I want to log sorry to tell homepage but those requests will always use the logger instance like the internally the new instance created then I will I don't need to insert that at request time because it's going to slow down the request right all right so I want to have that instance always present there and the final step that I need to do here now is just say hey log logger and print line and that's it like that's that's the way to do dependency injection what else can we do here well we felt about middlewares in the in the previous talk right and we can write our own in the middle where's quite efficiently in ingo using the standard library as well and you can do something like let's say the logger middleware right so this one accepts a handle func which we've seen right and returns the handle funk because that that that's how go works for HTTP package and we can I don't want to return a null I actually want to return a function so now we can actually do some somewhere here and what I'll do is I'm gonna move this from here right so my request is still there let's put it actually as a method on handlers so we have access to to our dependency right and then we can call okay I want you to call the next one for me and let's make this actually a bit more useful let's say we want to time the request so start time should be by now and change this on a bit in how many milliseconds let's say I now sub and start right so that's how it can quickly instrument your request and - sorry okay I can put it in the first sure so I can do this if I want okay so yeah there's plenty of ways to approach this line another thing that that's going to be interesting to notice now that I'm working in in my package right and now I need to jump back in the main package so to do some extra work plus if I want to write tests for this it's going to be a bit trickier because my my roots are initialized in a different in a different package right I don't like this package doesn't have control over where it's called and I don't want that like generally I found that for me it's easier to to have control over the whole package and what it does in the same place so let's move this part as well and to do so I usually use a pattern like this let's say set up roads and I send the marks there and what I'm gonna do is I'm gonna create the method here right and I can I can basically cheat a bit on this one and now all my package logic is self-contained I I want to change this one from / - I know homepage right I can do that and by don't need to jump back in main to change it but I want to do logging for this that's what we started doing so I can do this and that's it that's how you do middleware management for your app you can use G for example which simplifies this or like gorilla marks for example which has the built in middleware logic but if you don't need something more complex if you need something like a lambda function running on Amazon or on DCP or asure then yeah like this this would be a minimal example of running a lambda function without having to have external dependencies let's see if it actually still works right we've done a bunch of refactorings here and like everything should still work but let's see so we have actually let me see if I jump back in presentation mode now so where's my console so now we have the server running we see the the date prefix as well we see that this one actually was written on mango line 27 so that's correct so it's easy for me now looking at log file to see what's going on there right and if we do a request so the request still works and we've seen that well it's processed in zero seconds things yeah I guess I can add a time out there but like that's that's how we do it right and what I don't have the time but probably in like 20 minutes we've we've written laid solution for how to to serve HTTP pages right the other thing is that we probably want to write some tests I said production right so unless you do test what was the the saying yesterday the TDD one test during deployment yeah so unless you do test during deployment yeah you probably want some tests right so let's say we use the classic home test function and we can how many of you have have you heard of the table tests okay a few so basically when when you do a table test right your tests look like this and you basically have a struct a slice of abstract where you define the name of the test and then you define your inputs and outputs and then you actually arrange for the test and what I've done here and you'll see there's a warning that oh no this is shadowed this actually is an intentional shadow because if I want to run in parallel the tests this will fail we've got that line it's gonna mess up the order if anyone saw the presentation yesterday in the same room from Sean Kelly about the message queues he basically explained the problem there I do to the pointers and how they work when you run the go test - parallel it's gonna mess up your test so I'm not gonna write the whole test right now because it's too lazy but I have the the best FREE define here you can understand why I'm not going to subject it to all of this the only thing that I need to do is probably change this a bit because I've changed the names right so I have new handlers here and I don't want to provide the logger and I have a instant of handler let's call it home right so this one actually tests the actual handler doesn't test the the workflow that your request to do and that's something for you to explore if you want on how to like you have all the logic here to set up the routes in order to be able to inject a test server in it which basically will follow the same approach we all let's go for this one and explain so we want to have an HTTP request as an input to our Handler and we want to have an HTTP response as an output that's how it works and we have some expected pipes right we expect a certain status and a certain body and [Music] there is a package called HTTP test it's in standard library it's under net HTTP HTTP test so this one is the import and it has some cool features in there if you don't if if you don't know about it or haven't looked at at it yet have a look because it it allows you to do some interesting testing using the test the HTTP interfaces and you can create a new HTTP test request so you can actually have a request created there with all the correct fields and so on and in in this case it's going to be a simple one right it's going to be get me the home page and there's nobody so this one is a get request not post and I want to have the response recorded told to me but without using the actual HTTP response that you normally get because that's that's an interface and we want to know more about our response not not just what the interface us to do if we go here the interface that that we have in HTTP would give us the right header right and header which are not useful because we want to actually see what happened in the requests right so this is where HTTP test new recorder comes in and gives us the HTTP response recorder and we use that here like we run our handler so we actually call this with the input and output parameters that we expect and then we say okay tell me if the response code was different from what I expect and tell me if the body of the string is different or not right so if I run this it's gonna fail or not is it no it's not so yeah I I reference the message directly so that that's why it doesn't fail but in your proper test like you should probably write here your actual test case right so if I would run it again it will fail way right so now we know it failed and we know why it failed as well which makes it easier for you to debug it as well so that's that's kind of it like writing tests for for this is a lot easier than bending clothes and you don't need to try test as I said for example for initializing the HTTP server but there's no point in that what else can we do we can do a docker deployment so if demo works I already have to demo the docker container right and this one is using the like the latest features available in the curve or one point no sorry yeah 18.3 or something like that when they introduced the multistage building process where it allows you to have a single docker file you can do both the build step and the actual container step at the end and what I'm using is the go 111 beta container just because why not test testing go right I'm actually also using case you haven't noticed it go mode here I'll come back to that in a second and like yeah that's that's how you usually do like you add your dependencies I prefer to vendor my dependencies and actually commit them with the code it's not necessarily going to be needed since go 111 but I generally advise for that because it makes it easier to do the build step and it makes it easy to track what changes also you don't need to depend on github or internet to do your build so like I just need to add a directory this step is actually needed in order to like run without running as a root user inside the container by default your container runs as like the application inside your container runs as root and maybe you don't want that it's probably not a good idea and I want to disable the Segoe part because if I use the net package it's going to automatically try to use the system resolver which is rely which means interacting with a sea library but I want to like have a pure go build like I don't want to depend on any Lipsy from the system so then I can just say okay disable the seco part and then run just go build as I I do it before and in in the actual container so where it says final stage it's it's a classic container like you you build it based on Alpine and my copy whatever you need in this case I need the SSL certs and the build binary that we we've got from the previous build expose the port and that's it so let's see if it actually works oops wrong button right so they actually delete that no we won't see I actually deleted the the configuration by mistake earlier sorry okay I can skip on this one but you have to trust me on it it it depends on because my binary now depends on some environment variables it will scream at me if I run the build process so it's not that exciting to run it what's interesting is that I was also using in case you haven't noticed this is the path to my project so I'm using the new go 111 go module feature or Vigo if you've heard of it so I have actually packages I'm not referring to them in a relative path like my paths are like normal packages I should probably format this right so that's what go 11 brings to us you can start doing this and stop depending on go path and build your app anywhere and yeah that's that's kind of it I guess I'm sorry for the docker demo but as you can see it kind of failed to start because it doesn't have permissions and so like I need to spend a couple mins to configure it database work there's a last step like inserting the database would be as simple as inserting the logger now because you would say something like let's say sequel X connects right so you would have your sequel connection that you get from the driver and then you just need to say oh I need to do the database part right so I add it as a parameter to my function I add it all the way so sequel X DB I prefer sequel X because it adds a few more convenience features to that and then I I want to have a DB which abb right so I actually sequel DB right so that's how you do sequel work afterwards or anything else that you depend on I like if this one would actually give me a second where do I have the import sequel X so now it's going going to the internet the question is it's going to is it gonna work or not it worked okay surprisingly the internet works in a demo yeah and like now you could literally start writing your your query straight into it and you have like in order to mock the database you'd mark it only once and then provide it to all your handlers for example and because you inject it like that you don't have to worry oh am i passing it to the request or not and so on it's going to always be on the handler struct and I also advise you to use the functions that have context at the end because you can do this so you can provide them with the context from your request let me make this a bit less red so if you if you use this type of work like everywhere you need to connect to somewhere or like execute the query use the context from from your request as my previous colleague spoke about like you'll be able to cancel your request if the client cancels those so you don't need to do extra work right and that's pretty powerful and I I've seen very few packages that actually is this but if if you're interested in saving some money eventually or making your application more performant knowing when the client cancels or times out or anything happens on the internet until it reaches your server and like the connection terminates that's a very useful thing to do not to mention your tracing and so on I think that's it I think we are running out of time if I'm correct yeah okay so if you have any questions please let me know by if you want to talk afterwards I can show you some some more things yeah just feel free to grab me around here yeah so five minutes of questions or you can get an early start to lunch okay you're using your colon yes and using a lot of some packs or something like that cuz it's very neat so what kind of configuration you are using I'm using the default configuration so it's basically a lot of generation of code yeah so it comes by default with like I can show you after a bit more if you want but like it comes by default with a lot of built-in configuration and so on and earlier when I said that I kind of want to move it to a different package and it didn't fit on screen it's because Windows was scaled already but like you can do a lot of refactorings out of the box with it so for example if I want these two to be in their own function I can say oh I I just want this to be a function right yeah so that's a plug-in and there's another plug-in for it to actually teach you shortcuts I'm not sure how should I do it let me see no no no I I know how to do hints what what I wanted to show you was the other thing like if I go here like if I don't use shortcuts there's going to be a different plug-in here which tells me oh you're not using shortcuts so I it also there's a different plugin which allows you to get help for for the shortcuts and like teach you and so on and you have statistics for them what yeah the other present the other plug-in is called presentation plugin and allows you to have the shortcuts pop on on the screen Heide it's always the case in like homepage packet but imagine a scenario that you have users package which you're you have an endpoint for your users you have the user model defined there and you have an author's package which has an API endpoint for orders and in order model us you're actually referencing to user model right and you can also refer to the order model from the user and in this case you will find yourself in a circular dependency hell I've been wondering like in this kind of structuring was like workaround for that I mean for services it's kinda in a way that might be abstracted by using some sort of like interfaces like local interfaces that abstract it and you will inject it but especially like when you are doing the modeling best tricky to get right and you can easily find yourself in this circular dependency chain so yeah I actually bumped into that quite frequently because it's like that desire of like using let's say as you said are the users from the order and water from the users but if you think about it you can also have a different package which actually combines the two of them into giving you for example order history right or something that's related to those two units of work without having to have a circular dependency so if if in this case you want to see I don't know let's say order history right you'd create a package or order of histories and like combine the functionality there so pull the data from from your two other packages in in that package it sounds a bit more work but on the long run you can still use the user packages is you can still use the order packages and then you can combine them to do pretty much whatever you want with them right and the logic will always be clear because it's going to be contained in that package so you won't have to think about oh where did I place the function that tells me how to retrieve the order list for a user or the other way like where did I place a function that tells me to retrieve the users that place an order for this item it's going to be clear because it's in order history I think it goes be like mind shifting if you are coming from like classical regul see MSE background because in there you're just relying on activerecord to give you like user dodi's yeah thank you okay anyone else any questions [Applause]

Channel: Coding Tech

Views: 126,809

Rating: 4.7744994 out of 5

Keywords: go, golang, microservices, software development

Id: bM6N-vgPlyQ

Length: 44min 37sec (2677 seconds)

Published: Fri Aug 31 2018