Go Tooling in Action

hi this is go tooling in action I'm Francisco a developer advocate for the go team at Google and in this video I'm going to show you some of the go tools that I use in my regular day to day i'm going to start with a completely empty directory this director is completely empty as you can see there's nothing inside really really nothing inside and it is under the environment variable name go path and i'm inside of this director here cool so how do i right go code well let's start with something very simple and i'm gonna use no tools at all to the point that i'm actually going to use cut a go program always start with a package statement and then a capital of import segments and then my function is going to be function main and when you call it it will print hello youtube and that is a whole go problem how do you write it now well very easy go run man go go great it works now if you know anything about go you'll notice that that code right here it looks pretty ugly it's not exactly what i could expire when i right go and that's something that go thump can fix so if I do go thumped main logo I will get the output that this cut should look like if I do go from dash D I will get the difference between both this is basically just having those two blanks right there and if I do go from dash W min there go I will read the file so now I mean the girl looks as it should for the rest of the talk I'm going to be using vs code vs code is an open source piece of software created by Microsoft and it's pretty awesome it has some really cool go extension that allows me to basically every single time I change my culinary save go thump will be executed directly so I can forget about go fun now that we have this code here we can pay attention a little bit more to what it is it's actually doing you can see that from the println is calling the function println defining the package fun and we're importing it here what happens if we remove it well go will complain if we try to compile this I never will say that font in line 4 is undefined is always unknown so how do we fix that we need that import fund statement but actually there's a tool called going import that every single time I save is also run so when I save it will appear automatically same way if I add another hello world but now I'm using the log package that will print it to to the standard error instead when I say we will appear to and if I remove one of them they will removed again so basically with this going ports you're able to forget a little bit about the import statements they just manage directly let me go back to the previous version with hello YouTube I showed that you can run go run may not go and get the output that we desire but there's more than that you can also do go build and when you do go build it's actually generating that demo there that demo is a executable binary and if you run it you get exactly what you expected that binary is a static binary that contains everything you need to run that file in in a Mac architecture so I could compile this code and send the binary to someone else that has a Mac OS like me and be able to run it directly without any extra dependencies but what if my friend instead of using Mac was using Linux or Windows well very easy just say go s equals Windows and go build again now we're doing press compilation of our code and voila we just created demo dot X e so demo dot X C is indeed a Microsoft Windows a mono slash net assembly and we can try to run it but - you know work very well because it's actually only working for Windows not for Mac that is pretty cool another option to compile your code is to use go install and what go install is doing it's compiling your code and installing the binary the resulting binary in the go path slash bin directory so if I run go pass slash bin slash demo I will get again hello YouTube cool what else we could do well go get go get given a package in this case I'm just giving the current package but I could give something else like github.com slash golang slash examples slash hello you would go to github download that code example there you go it will it went to github downloaded the code the corresponding code compiled it and installed the binary right here in my bin directory inside of go path so far we've seen a lot of the different things like the code tool can do there's more than that let's see one more goal is different here it just says the import path of the current package not really cool handle much but you can find more interesting things like you could say you know I want to print and this is a go template I want to print the name of the package and the name of the package is main because the binary I could also say you know I want that and the documentation and you will see that there's no documentation let's fix that so I'm going to add some documentation saying demo is an awesome demo and now run it again and you see that that fermentation appears here demo is an awesome demo I could also do all the things like I want to see all the import statements and it says that my package my package only depends on thumped which is normal because that's the only thing that we're doing from that println but we can also see things like okay so what does fum depend on another s'more of them that's very interesting let's join all of those all of that list with break glass in the middle there you go now we got a list of the packages that fund depends on pre Co you can imagine how easy it is with this to construct a tool that good generate a dependency graph for your own code okay so I've shown one way of accessing the documentation for the current package which is right here doc and you can do the same thing for thumped and see the documentation for the thumb package but there's better ways to do it than that my favorite one is gotalk so if you do go doc of this package you will get same message we do go doc a font you get much more than that you will get actually the full documentation which is pretty big with all the different functions that appear in thumped what if I care about a specific function I can do go doc thumped printf and that will give me the documentation for that specific function another option is to run go doc the binary with - HTTP and give it a port now when you're running this what you're doing is actually listening on port 60 60 so when I visit it you can see that the packages we can see are all the packages email machine not only the packages in the standard library that is cool but all the packages that I've ever downloaded so for instance we can find gorilla and gorilla max which is something that I regularly use which means that I have access to all of this documentation even when I'm traveling around and I mean an airplane without connectivity okay so let's make our demo a little bit more complex we're gonna modify our main logo and instead of being just hello world it's gonna be a webserver so I'm gonna say that all the requests are handled by this handler that I'll define in a minute and I can do listen and serve to start listening on port 8080 and I finally need to define my handler so that is a response writer and then the chip your request and basically everything gonna do is just print hello YouTube on the web cool this compounds it looks pretty good so let's try running it go run main there go huh nothing happens that is surprising well actually not really but it's going on is that something here is failing and go doesn't have exceptions go uses sometimes panics when something exceptionally bad happens but if there is a problem that is something expected instead what happens is that we return an error so one of these functions has an error that we're ignoring and that's why we're failing silently but which one well fortunately there's a tool that that tell us exactly that and it's called error check when I run it it will tell me that on line 11 HTTP listen and serve is not being checked and as you can see here it's listenin serve has indeed the type of function that given a string n HCP handler returns an error and that error is not handled so if the error is no nail whoops misspell nail log dot fatal of error cool that's right again now hopefully yeah there you go so now it fails but at least it tells me what's going on and the problem is that this there's already one server that is bound to 8080 so I'm going to go here kill that server run again and I'm gonna get this message saying hey are you sure you want to accept incoming connections which is which means that our server is actually running so I can go to localhost 8080 and again hello YouTube on the web cool so now I got my first web server already completely written and everything works for the next step what I'm going to do is I'm going to add something that is probably a bad idea but I'm going to check if the path that I'm getting is an email and that email finishes with AD golang.org how I'm going to do that I'm going to do it with regular expressions so I'm going to import Raggett and I'm gonna write it first so then I get the talk completion which is always nice so rackets Dodd I'm going to use math compiled and that gets the string itself so I'm going to say that this starts with the beginning of the line then whatever which is what I care about ad Golan dot-org and then the end of the line that's going to be my regular expression my path I'm going to get it from path URL the path that is the path of the request and I'm gonna check if something matches by calling fine or string sub-match there you go this has two parameters the first one is the string to be analyzed so in this case we call it path and the second one is how many of them you want to match since I'm going to be matching all of them I can pass just minus one and now I can say if match is not nil then it means that that that was actually an email and I'm gonna print hello gopher and the name the name is gonna be from actually from the regular expression so I'm going to get the second part the first part is the whole thing the second part is this piece here which is the username in the email cool and I save that does not compile that is sad let's see what's complaining that's not support indexing oh yeah not regular expression match there you go now this complains because it says hey possible formatting directive infringe MF println call and this is not a compiler if it was the compiler we could get a grant a squiggly line but we're getting green that is because it is not veiling to compile but go vet which is differential is complaining about it and go vet is basically something that it's going to give you hints on code that might be wrong it could have some false positives but occurs rarely enough that for me it's something that you should always have as part of your editing tool cool so how do we fix that yeah that's actually f printf there you go and let's add one slash and there also if I'm putting this one I don't want to print the next one cool so now we have our whole code our cup what it does is it says okay I'm going to handle off a web request with handler and then listen on port 8080 and then the handler what it does is it creates this regular expression it checks if the path of the request matches the regular expression if it does it says hello gopher with the name otherwise it will say hello dear and then just use the name and oh the same problem if printf there you go I'm going to have the - end right here cool so now we have our code it compiles it runs that's it run because is already running here so let's run again it runs and it says hello dear nothing or if I try camposgalan dot org it says hello dear Campo I go like the rorg oh so that is a match and that is match because there's that slash there let's drop that slash so I'm going to do it by skipping the first character of the path we know that all the paths will start with slash so with this safe let's run it again hello dear camp label on the road so that's still not good what is going on oh ho that's the end of the line not the beginning so let's kill it to run it again oh that is worse okay let's write without if I say okay dear food but when I buy something that much is the regular expression it crushes and it crushes with the panic something pretty serious says index out of range somewhere you know what instead of trying to find what's going on by written the logs why don't we set a breakpoint so I'm going to set a breakpoint I'm gonna start the debugger and the debugger I'm gonna say going to block some go and I'm gonna launch it it's gonna emot-- the bug okay so that is my configuration that is now stored here and I can try to just run it I'm gonna stop the server first and then run it that should request for the permission to whoa that is new okay that is requesting for the permission to connect that looks that looked good for now and now if i refresh there you go I got here and we're on the first step we're going to go next step I'm going to go next step and I feel like match here there is something wrong because that looks like the line that is failing so what is much well match according to this it has two elements oh no it has one element it has one element and the first element has two elements and that is the problem we're actually accessing the second element of as lies that only has one element so actually what I'm going to do is we want to access the first element then the second element of the first element kind of confusing but if you look at this make sense we're trying to get these campoy here so zero and then one cool so let's just play it will this work there you go now it's on let's write again loud refresh and hello gopher campoy awesome okay so that works okay so testing like this but basically just sending requests to our running server and see what goes on is a way of testing but it's definitely not something that we want to have to do every day so let's build something that actually verifies the valid video of our code and that's what we call unit tests so to create a new unit test I'm going to create a new file an angle I call it main and the score tester go am i saying it's underscore test this file will be ignored by go build go install go get etc but it will be taken into account when we do go test so that test was going to do is it's going to be package main - and I'm going to create my function test humbler that receives a testing duty and for now if I do go test it runs a test and it says that everything went well cool well it won't work there's nothing it did not do anything but we can make it fail RF something went really wrong and I run it go test will fail saying oh yeah something really wrong cool so let's use that to verify that the code is doing what we expect so first of all I'm going to want to call handler so I need a response router and HTTP request okay so let's start with the request I'm gonna create a new request so new request and I have a couple parameters to pass the first one is the method so I'm gonna use method get' and then the path it's gonna be HCP localhost 8080 this could be whatever you want I'm gonna try then compile and org and then the body the body is going to be empty so I can pass email okay we have the request and now what about the response writer well let's see a little bit more about this so handler has this definition we can go to the definition and then we have this response writer and a response writer is an interface right the interface that has Heather right and right Heather so basically what I want is a fake response writer and luckily for us this is one of those and it's in HTTP test and what we call a new recorder so that recorder will return a response recorder I'm going to call wreck and Rec has a couple methods but the important ones has right right Heather and Heather which are the ones that response rather needs which means that rec is a response writer so now I can go handler rec and Rec both pronounced exactly the same way and that error should be handled if there is no new valid test saying could not create request and that is there so now we were able to call the function handler and that is pretty cool and now we want to make sure that the response we got was actually correct if the status code is not status okay or 200 I'm going to air the test by saying expected status 200 god these are the one and if gonna check that the response which is body string if that response doesn't contain the word gopher actually go for campo that should include that then I'm gonna say RF unexpected body in response and I'm gonna say that is the body response so that is this one here cool so we got our test our test basically what it does is send that request and then verifies that the answer is correct let's test it go test and it fails because it says unexpected body responds hello gopher campoy oh it not string contains there you go so it doesn't contain hello then it should fail right so now we have everything we need we have our test and what is even cooler is that you can see both pieces of code one next to each other and run the tests by running tests go test package now we'll run all the tests in the package there's only one so that's pretty much it but now I can do tests with coverage and when I do test with coverage you can see that we're running these tests handler under on the left side and we're measuring what are the lines that are covered by the test in the main don't go in this file right here so we can see that we actually never execute in this line so let's add one a case a more test case to make sure that that use case is also covered so I could copy paste this code but instead what I'm going to do is I'm going to use what we call and go table-driven test so I'm going to define a new variable which is a slice of structs those drugs have input and output which are strings and the first one is going to be exactly what I had so the lender org and I expect to find gopher campoy the second one is going to be something else and I expect to find actually let's use something like this and I expect to find there is something cool now I'm going to iterate over those cases and here instead of comparable on the org I'm going to use C dot in here instead of gopher campoy I'm going to use C dot out that's it so now we're executing the same code but we're checking for two different values and we can easily add more unit let's read run our tests with coverage you will see that that line change color then now everything is winning okay so now we have something that runs and it is valid because our test pass that is pretty good but is it fast well let's see how we can tell if something is fast enough what I'm going to do is I'm going to run my server go run main dot go now my server is running and I'm going to use go work that will do basically five seconds of load testing over my server and I'm gonna try with gonna go echo on the Lord so now it's going to do in five seconds going to use ten go routines and after five seconds it would gives us it will give us the result it will say okay so you were able to run thirteen thousand requests per second which is a decent amount of them if you think about it but maybe you can do better but how can we do better well we're not really sure but we're gonna save this for later let's save this as perfe and we will check that later so how do I know how can I get more information about what my server is actually doing one of the ways is by instrument in your code and my favorite way of instrument in the code is by importing HD P P proof and as you can see I'm importing it with an underscore as the name for the package because I want to make sure that go imports doesn't deleted so now without doing anything else I'm just going to run my code again and everything looks the same let's see if it's slower than before so before we're able to do 13,000 requests per second now we're able to do twelve thousand eight hundred twenty five so there's around five hundred requests less per second but that's actually really not that bad not a good thing about this is now all of a sudden we're able to understand better what's going on something that we can do is we can visit localhost debug slash P proof and when we visit this page you can see that you can see that you can list all the routines all the girl routines and you also have access to the heap and the hip at the end you have a lot of information of the memory how many flips freeze how many hip allocations but also how many times the garbage collection run so far which is two thousand four and what were the latency is for the pauses in nanoseconds for the last last 256 pauses that is cool and you could probably write some script that parses this and obtains the result but there's actually already something that does exactly that and that is people so you can run go to p prof say i want to run it for 5 seconds and the endpoint to hit is debug p prof profiles i'm going to run for this ah let's see what did i do wrong - the seconds profile there you go now we're running it so go to p prof during 5 seconds and in 5 seconds we will get to p prof p prof is a tool that if you've never used it before basically what it does is it identifies this some binary the execution of some binary and then it allows you to investigate what happened so if you the top o the profile is empty so it actually did not measure anything and that is normal why well because there was no traffic going on so let me run go work but I'm gonna run it for one minute this time and let's do it again for five seconds I'm going to record I'm going to get the profile in working and now at the top there you and I can see what pieces what functions are actually the Moscow or we're spending more time running them but it is actually not that useful for me instead what I would like to see is this graph here by running web we get this huge SVG and we can log into we can zoom into it and see what's going on okay so that is a cisco parce que ok that is main the handler this one here this is the one that we wrote is the one we care about you can see that it's calling for a printf and compile no matches and start with that so that actually cannot make sense that that looks like pretty much we wanted but it's kind of hard to understand exactly how much every single one of the those box is taken something that I really like is if you run that again you can do go torch for five seconds and go torch is something that developed by by uber that will do something quite similar that go to pee proof it will measure all the performance of your binary and basically taking snapshots of what your process is doing and every single time and then it will generate that torch dot SVG so let's open that torched our SVG I really like this one because it's much easier to understand what's going on the horizontal axis it's basically how much time everything takes in percentage so we know that connection serve takes 81 for the one percent of the whole time that we measured but the one we care about is this main dot Handler and now I'm in the handler we can see that there is three things F printf Finals all strings are match and mask on pal mask on pal calls compiled that calls combat like us compile one path that goes on and another not go so now we understand better what we're spending our time doing the first thing that seems pretty obvious is that we're doing this compilation of the regular expression every single time and that's probably something that we don't need so I'm just going to go to my code and say you know what this regular expression I'm just going to move it out and basically do it only once the rest of the time we're just going to reuse it so gonna run my code again I'm gonna run some requests and gonna do go torch again now hopefully torture SVG the new one will not contain that compilation step anymore let's see it okay so we have actually became much smaller which is pretty good made that handler is here and now we don't have a completion step instead we have F print F final string so much and then some comes T to e that we can we'll see later cool so is this faster well we're not really sure but go work nose so let's run it so we run it for only five seconds we get then now we are able to do 20 almost 22,000 requests per second well before we're able to do 13,000 so we won nine thousand requests per seconds that's pretty good okay let me save this for later too I'm going to save it as per that one okay at this point I could continue doing this but instead what I'm going to try to do is I'm going to try to understand better what's going on with that specific function the main dot hammer I want to do a benchmark for that function rather than running the whole server how do you think go well actually it's very similar to what you could do if it was a test right so everything that I'm going to do is I'm going to copy this test and I'm gonna use always the same one let's move to what we had before so that was here and he started repeating it per case we're gonna call this benchmark benchmarks don't get this benchmark skin get peace and we're going to repeat this B dot n times that's it now this what we what used to be up that's not it we need to name the t as a B okay so now it not that is that is it we have our our new benchmark handler which pretty much does the same as other record but it does it as many times as we need and if you want to do it if we do go test or bench with this I'm gonna run my benchmark and get the five that it's taking 4367 nanoseconds per operation which is pretty good so can we do better again we need we need profiling so I'm gonna enable more profiling CPU profiles gonna go property P you and I could open it with go torch binary name is demo test and the input is property P you I can create my torch now you can see that this is running my demo dot handler many times and you can see exactly how much time that is taking so find all shrink so much is what is taking the mass is taking forty 1.3 percent so how could we improve that well at this point probably is when we should reconsider using regular expressions at all because basically what we're doing here is we're making sure that this is something that finishes with gallon droid so what if instead with it thing like if strings has suffix so we're going to check that path has the suffix I going the Lord then we're gonna use this could be faster but I'm just going to use trim suffix and we can drop completely this so now what I'm doing is if my path finishes with a goal and O'Doyle then just remove that suffix I go down the road and use it actually let's declare it as a variable why not cool first things first go test go testing passes which means that I could to correct that is cool go test that bench and let's see before we're able to run it was taking form and I set four milliseconds now taking three milliseconds that is pretty good that's a pretty good improvement can we do better than that well let's see a little bit more in detail let me create that CPU profile prodigy PU and this time I'm going to use the tool P probe to open demo test and property view and instead of doing web I'm going to do a list of my handler and I can see that I'm spending four hundred twenty milliseconds on this function here but I would like to know how much time I'm actually spending on string trimmed suffix you know what let's go back to our go torch so we have our demo handler F printf is the one taking the mask now right header here sticking a very long time and most of it is taken by the tech content type so why is that well go when you don't set the content type and the web responds it tries to figure out so it's actually analyzing what we're running to figure out if it's text or if it's JSON or events in image so instead of doing that why don't we just say it so we can say Heather set content type is text plain cool let's see if that is faster or not so from 3000 to 2600 and now we went down to MU 1900 that is really good with under 2,000 now that is great why don't we open the torch again and you can see there our handler now it's mostly spending time running the header but that's actually really fast so I think that we're pretty much okay let's see how much faster it is so we're going to do it as we did before by using go work let's see while that's running we start with 13,000 requests per second then 21,000 workers per second and now we're up to huh that's not crazy 22 almost 23 thousand requests per second but still we went from 13,000 to 23,000 so we can 10,000 requests per second just by doing some those little tricks so does that mean you should always be using these tools to change every single piece of your code to make sure everything is faster well no only if it's needed only if you actually need that extra performance but it is important to think about the fact that you should measure that performance before you start trying to make any other tweaks why because maybe you're going to spend hours improving some piece of code on which you're only spending 1% of your time so the increase on performance can be pretty negligible so it's better to first do your monitoring just go towards us go to P prof is whatever to works for you but at the end what you need is first analyze your code then improve it I hope you enjoy this talk thank you very much for paying so much attention to such a long video and you can ask me any questions on my email which is by the way campo a colander or or you can also find me on twitter as franceska thank you

Channel: justforfunc: Programming in Go

Views: 103,999

Rating: 4.9809847 out of 5

Keywords: golang, pprof, flame graphs, go-torch, performance

Id: uBjoTxosSys

Length: 41min 50sec (2510 seconds)

Published: Thu Jun 02 2016

Reddit Comments

[u/feketegy]

It’s unfortunate that francesc doesn’t make videos more often.

[u/Emwat1024]

Note that you don't need go-torch anymore for flamegraphs. You can directly use pprof for the same. Just run

​

go tool pprof -http=":8081" [binary] [profile]

[u/Stopher87]

Can we get a tldr of tools?

[u/chocobor]

Thank you, very nice!