13 Introduction to gRPC and Protocol Buffers

[Music] hey and welcome back to another episode of building microservices with go and in this episode what we're gonna do is we're gonna look at G RPC services so in previous kind of episodes what we've been looking at is we've been looking at how you can build out a JSON based service and using that restful approach well with Jason you're using HTTP and you're using the kind of JSON messaging as the kind of the message types and that works great it's easy to use it's very widely understood but I think the key thing around Jason and rest is that it's not particularly performance not massively optimized and you've also got to go to quite a large amount of steps building clients and doing the integration and things like that so Google came up with a new approach and they call it G RPC and the intention behind this is that you're still using standard protocols in this instance it's gonna be HTTP - as opposed to HTTP but rather than Jason what you're gonna use is a binary based message protocol called protobufs so with protobufs because they're binary based there are obviously quick AIT's faster serialize and send it over the wire and with them also what you do is you end up defining these interfaces these proto files and because you define a proto file anybody can generate a client based off your proto file so let's take a look at how that looks so the kind of the basis of this is going to be that proto file so I'm gonna specify that I'm gonna be using protocol buffers three proto three is my syntax and is the latest version now in a proto file you define services and you define methods for services and you define messages for methods a service definition looks like this so I specify service as a keyword and I'll give it a name I'm gonna build a currency service currency and with that then I'm gonna kind of start that parameter block and I can add the methods to it so a method is gonna use the RPC keyword and I'm gonna create a service called get rate now when you define a service or sorry a method in service for protobufs you specify message types so you need an input message and an output message so my message types what I want for my request message I want to be able to to define a way of getting a rate between two currencies so for example if my base currency is British pounds and my destination currency is US dollars I want my get rate service to return me just a number that I can use to convert between the two so I'm gonna kind of specify message again using the message key word in protobufs calling it rape request and then I can add my fields for my message so what what I want is a field well I want to be just a basic string and I'm just gonna call that face now with protobufs what you need to do is you need to specify the the location in the message format and this is important because you can't change this if you kind of want backward compatibility so I can add new fields to my messages and maintain backward compatibility but I can't change the indexes and maintain backward compatibility so I had my string my base and I want to also add destination so I've got my my request to fields base and destination so I can refer to find that there on my method cable get root rate request is my message type and my output will I define that by using the returns and then I wrote my return message so let's define our return message so return message is going to be rate response single field and this time we're just going to use float I'm going to call it rate and the index position inside the binary protocol is going to be one here I can just specify my rate response so now I have my service I've specified that it's an RPC based service I'm using the name get rate the input message is going to be rate request and it's going to return a rate response I'm defining my messages here rate request and rate response and they're gonna have string and float so let's just dig into this so I think one of the things that that's a little bit not confusing but you've got to think about protobuf since that the protobuf has its own types because a protobuf is designed to be independent of your underlying language so if I define a protobuf in go it should work the same as it does in JavaScript or Java or Ruby or Python so I'll put all the links to the documentation below but the the kind of the proto buffers website here defines this kind of syntax it tells you about the message types and the field types so when it comes to field types and if I scroll down through the docs here you can see that protobuf two times a type such as double and a double in c++ is gonna be a double in when it's converted to java it's gonna be a double in Java and Python it's a float and go it's gonna be a float 64 Ruby and float etc so you can see across all of those different languages a proto buff defined type double will convert into these various other things exactly the same applies to things like integers and signed integers fixed for kind of numbers where you're gonna kind of wanna a set number of decimal places boolean x' strings and strings of bytes so these are kind of the base types anything else you can also define your own message types and I can embed types and types so same ways you can kind of do this sort of approach with with struts now why it kind of does all of this is because you end up with this binary serializable format and again if we take a look at the the sort of the way that things are encoded whereas in Jason you're kind of encoding down or string with protobufs it's binary so for example this message here tests one it has an int 32 called a field now when this encodes what it's going to do is it encodes us to this this absolute sort of binary protocol the way that it kind of is efficient in the way that it packs is it's only gonna use than the required number of bytes to store the number so an sort of a 32-bit number you would ordinarily have a set number of bytes with with protocol buffs it's only gonna use what's needed so if you have a small number like 1 it's only going to use a single byte to store that data a larger number then it uses more bytes but because of that sort of flexibility it means that the the actual message sizes can be very very very small transferring all of that over the wire is obviously then very very flexible and I recommend you kind of take a look at this it's it's useful for reading just understand how the message structure of protocol buffers works when it comes to using it you don't need to worry too much about it because the frameworks gonna deal with with a lot of these things for you so we've defined our our service here and what we need to do is we need to generate some goal code from this so to generate go code from a protobuf you've got to use some tooling and the tooling that we are going to use is twofold we need the proto command and what the proto C command does is it generates code files for particular languages based on a proto file so it's gonna kind of take that proto file and it's gonna generate go or it's gonna generate Java or C sharp or Ruby Objective C also things like that for G RPC and we want to be able to code Jennsen GRP see what we need to do is use a plug-in along with the proto command so when you eat need to install proto C and you can get that from apt or you can get it from brew number of different sources and we also need to install the G RPC plug-in which is a go package and I'll put all of the links below where you can get that but the the G RPC package is maintained by Google it can be found in the sort of the standard github location okay so once we've got that installed so it's installed proto see we go get the G RPC and what we can do is we can actually generate our code file so what I'm specifying here in the command line is that I'm gonna have an input folder of proto's this is where I'm storing my protocol buffers for my protocol definitions and I'm gonna specify an individual file that I want to look at so my my currency service here and then I'm going to specify my output I want to use go there's an output and the plug-in I want to use is going to be G RPC g RPC is the framework which uses protocol buffers as a message and i want to put that in proto's slash currency so we're gonna see the generated code in this folder here so let's run that so make I've just got a make file here make proto's so that's generated and we can see now that this code is there so we're gonna kind of have this as part of our build process when we change this protocol we can see that what it's done is it started to generate the go struts based on the proto buffers definition so I created rate request here is a message as a string base and Astrid estimation string and you can see that the cogeneration has created this strut so it has a base against defined as a string which is the go type destination string as a go type and also it has a number of different fields here the access know keyed and unrecognized these are kind of used by the internal mechanisms serializing the objects you can mostly ignore this with a protobuf you also got the capability that you can serialize it to jason and that's quite useful to be able to interchange between a sort of a binary or a text-based protocol but you also get a number of different methods so it's gonna set all of this up for you this code gen file is not particularly easy to read I'm going to admit but you don't have worry too much about it intellisense generally does a little bit so we need so again there's our response we're defining rate float32 and we can see everything is being set up there again a number of different kind of things used for marshaling but the the kind of the important thing we really want to be looking at as well is the G RPC content so the plugin is also going to generate all of my G RPC code it's can generate me the ability to create clients and what we're gonna look at in this lesson is how we generate a server so that it generates me an interface to create a server a G RPC server using my service definition that I had in my protobuf I need to create a strut that implements this interface so let's see how we can do that now so the first thing we need to do is kind of understand like why we need to to do this why do we need to sort of set up the the various different service in it it's the way that it works so the G RPC framework what you're going to do is you're going to use a function which is generated for you inside this currency and it's a register service so this helper function which is code generated it basically Maps your implementation of the interface how you want to handle things like get radar the other methods that you define for your service to the G RPC server and and it kind of feels a little bit different from the way that you're managing Jason but if you kind of think of it in maybe in terms of that your currency server interface is your handlers and the g-o-p server is the same as your HTTP server HTTP r2 but you know so think of it that way you're kind of just mapping routes to a server so to set that up well we're gonna need a longer so I'm just gonna create a quick logger I'm just gonna use HC log it's going to use the default one and then what I need to do is create a new G RPC server so the gr PC server is actually in the the base G RPC package and I can set it up just like that I can specify number of different options on there I'm just going to err vanilla for the time being so once I've got that in order to be able to do my my register currency service what I need to specify is the G RPC server but I also need to specify an instance of a currency server so let's let's go create that currency server so I need to implement this interface and I'm just gonna copy that so let's do so okay so how do i implement an interface well I'm going to create a strut so type currency strut I'm gonna need a logger so I'm gonna define a logger on there and then I can implement that method so see currency get rate so context up context and we need our rate request and our rate response okay so the rate request in the rate district ponds we're gonna get from this auto-generated class file here this is put it into the package currency I like to a leus this stuff into package called proto's you don't need to to do that I just find it sort of easier for me to kind of understand what's going on so let me just grab the import there and we can get that out it rose currency Frodo's currency okay so we we have this rate request and we have this rate response so let's just implement this here so we're gonna do let's let's just log some things so we're gonna log to say handle get rate and we will specify the base currency which is going to be PR let's call it ah tele cents is wanting to work there we go get base so get base and I'm going to use destination that's going to be our don't get destination so then what I need to do is I'm always gonna have this signature I need to return my message and any error so my message let's just return a new proto's rate response we're gonna set the rate and for the time being we're just gonna set that 0.5 and no error so that's the bare minimum that I need to do to implement a currency service interface as defined in yep so now that I've got that what I can do is I can just do that currency dogs oh we should probably create a constructor so let's just do you currency a cheat log I'm just gonna use that idiomatic go approach Wow so much typing currency log all right so we're gonna define new [Music] currency and we're gonna specify the logger and I just want that to import oh it's Sarah wrong package there we go so we're gonna import that we're creating our logger and then we can register our register our currency service implementation against the G RPC server so the next thing that we need to do is to start the G RPC server so G RPC server has a method called serve and serve is similar to HTTP listen and serve as the you would use on an HTTP server the difference being that what we need to do is we need to specify a net listener so we need to specify a port that connection that we're going to listen for so we can define that so we're gonna do L error because net listen has that signature so net dot listen takes two properties network so what kind of network TCP is what we want to use and then address so for now my dress is just going to be anything and I'm gonna use 1992 as a port so if era I'm just gonna handle that just in case I can't create a listener it could be that the ports already allocated dot I'm just gonna do unable to listen all right then we'll exit OS exit okay but if everything's working then we get down here so let's give that a test so we can just do go run may don't go undefined proto's main don't go let's have a look [Music] okay I just need to do my important let's try that again see where are we complaining out window co19 register currency serve okay all right so now we're up and running so we've got we've got that service up and running and this is running now so how do we test it so previously what you would have done is you would have curled it so you would have just done curl localhost 1992 you would have specified the route and you were just done some data yeah like so now you can't do that with Geo PC because it's not a restful service using Jason so what we need to be able to do is we need to be able to test it so curls out of the question we can write some functional tests or something like that we could write some unit tests for testing but there's an incredibly great program called G RPC curl and what GRP C curl does is it allows it's kind of like curl like but for G RPC services really really useful tool super super useful for testing when you're building out your services you can install it just by doing go install again I'll put the link for this down below or you can grab it on brew if you're in a Mac but what we can do with this is we can kind of just do that setup so instead of using curl we can do G RPC curl we can specify data and we can specify that data as Jason because if you remember when we looked at the the definition for our requests it also has Jason encoding and we can specify the location so where our server exists and then we do this so instead of the root using a kind of a URI based path what we're doing is the name of the service and the method so let's take a look at G RPC curl and how we can kind of use that we'll use that quite a bit one by kind of testing through our services and working through this example so I've already got G RPC curl install G RPC we can see that there so one of the things that you can do with G RPC curl is the GRP sees curl can actually look at your G RPC service and it can kind of do some reflection you've got this list command so I can do geo geo PC curl and I can specify the I'm gonna use the plain text option because I'm not using HTTP the location localhost 1992 and I can say list now this is giving me an error here and saying well I can't list the services because the server doesn't support the reflection API that's because we haven't set that up so in order to enable our service to support the reflection API we can just use another package on G RPC called reflection and we're just going to do reflection dot register and our G RPC server it's as easy as that you might want to disable this in production but you could always put it behind a switch so let me just quickly restart my service I'm just gonna accept that cuz I haven't got a go firewall entry that yet but now if I run that GOP seeker list you can see the services which are coming back so GOP see everything is composed in terms of services and messages so service we've got currency so let's have a dig into that so now what we're seeing all the methods for currency so I can see that currency has a method called get rate well let's describe currency to cap rate and we can see that the signature for currency don't get rate is an input of rate request and an output of rate response so what is rate request look like again we can use that described and we can see here there's the the definition of the input message and we can see that it has Jason name so I can use the jason name of base and i can use jason name of destination and that's gonna map to the protobuf string type of base and destination this is kind of nice so GOP c-curl is allowing us to kind of from the command line we're not having to write anything structured we can we can just use jason to test our service so let's give that a go so we're gonna do plain text we need to specify the data so data is going to be base we're going to do GBP british pounds and the destination is gonna be us deep United States Dollars location is localhost 1992 and we need to specify which method we want to call so we want to call the GATT rate method on the currency service currency get rate and you can see there that that call has worked and we've got a response back of rate zero five if you look at the log that we put in there you can see that the gap rate message decoded base and USD so that's kind of going through our implementation there and it's really that simple so in the next episode what I'm going to do is I'm going to show you how we can use G RPC clients and how other people can create your PC clients from your protobuf definitions we're also going to look at some of the more advanced features we're going to look at things like error handling because error handling is different from the way the restful service works you can't use HTTP status codes because we're not rest so we're going to look at the differences there and we're going to also look at the differences in types of G RPC service because what we defined here is a simple unary service this is what it classified is something which has a say it's a kind of a single request response model G RPC also has the concept of bi-directional streaming or streaming protocols that means you can have a constant stream of messages the connection stays open very very very efficient and we'll we'll look at how we can build out some functionality in our currency service and maybe do updates when a currency changes I hope you're finding this stuff useful I'm kind of wanting to kind of introduce this slowly and kind of take it bit by bit but you know if you've got any questions or any comments please leave them down below if you do like the content like and subscribe and hit the bell button for notifications I try to release one to two videos per week stay safe I hope you're all well and I hope you're well maybe enjoying a little bit of isolation it's a great time to maybe do the things that you wouldn't ordinarily be able to do for circumstances okay but until next time stay safe and I'll see you again [Music]

Channel: Nic Jackson

Views: 25,621

Keywords: go, golang, microservices, gRPC, protobufs

Id: pMgty_RYIOc

Length: 32min 7sec (1927 seconds)

Published: Tue Mar 24 2020