Complete Golang and gRPC Microservices (Project Course)

so a couple of days ago I've made a post in the community asking you guys what you guys wanted to see and grpc was by far the most voted so here we are and honestly the combo of go plus grpc is crazy because if you want to build efficient and scalable microservices you should definitely consider adding grpc to your tool belt now in case you're not familiar with grpc it is a performance framework for building remote procedure calls RPC and combined that with golang which is already a fast compiled language it really creates the perfect Duo the PowerHouse for building micro services and this is really where golang TRS honestly when I was researching for this video I couldn't find no one in the internet doing further than the basic clients to grpc server that you find on the official documentation so I also wanted to make this video to go a little bit further um and this drawing here is what we're going to implement we going to go to that in a bit but basically we're going to do server to server communication which is how you most likely do it so have multiple Services communicating with each other and so in this video we going to be stepping out of this basic tutorial hell and we're going to be building a simple MVP product that we can ship to our users by the end of this video/ course and we're even going to do something a little bit different which is we're going to expose grpc between interest service communication and we also going to expose http for the client browser to communicate with the external services with the um with the browser right and one of the reasons I wanted to make this video is to have the foundation slides uh first because I'm cooking something big for you guys so stay tuned for that so this video is going to be the basis for building more complex microservices in the future so what we're going to be building is the beginning of an order management system imagine those um those McDonald's when you go to McDonald's you have those big screens where you can interact I think it's called the point of service uh let me show you so this is what you're going to be basically I'm telling which is this big screens here basically what you're going to build is something similar like this Self Service where you interact with the user interface in this case going to be for example a browser and then when you make an order we are going to send that to the order service and then the order service is going to communicate with the kitchen basically it's going to send that order to the kitchen so they can have inside the kitchen they should have like a screen with all of the orders to prepare and this is something similar that we're going to be doing and I think this is a very fun project to use microservices and to show you how the multiple Services interact because you can expand this on your own really fast you can create um more services you can create the user management system you can create anything you want because there are multiple missing pieces here but I just want to show you the order to Kitchen communication with grpc and and go now before we dive in let's make sure that you are up to speed with grbc and honestly all you need to know is you don't need to be a god with JPC n of that I am not definitely uh all we need to do is that it is a way of communication to server and clients just like you have been doing with HTTP and Jon apis right and the advantages of using drpc over HTTP for for in service communication is that grpc is more performant at least than Json because when you're doing uh Json you are sending all of this data and you need to First Marshall the data and Marshall the data before it can send to the user right and grpc uses protocol buffers which are by default binary so it's going to be more performance than sending the the raw text here and real protocol buffers is what we're going to be doing in the bit we're going to be writing uh protocol buffer files which is definitely more Compact and efficient than sending text like any text format like Jason and finally grpc is a really good candidate for INT service communication now when you are talking about uh server to browser communication that is not so much so that's why we're going to use both we're going to have two transport players Json and grpc grpc for server to server and Json for client or browser to the server Gateway and if you want to better understand grpc uh I recommend you going over the basic tutorial here on the grpc official websites now I have been talking over this diagram but I didn't show you anything so far so basically what you're going to be building is this diagram here so these microservices which is going to be another service which is going to be a big one and then we're going to have a simple kitchen one the kitchen is going to act has the first gateway for the users so it's going to have an HTTP transport just like the orders is also going to have an HTTP just to show you that you can have both uh but then we're going to have a browser which the user is going to have like a simple application we're going to do that simple front end as well I'm going to show you in the bits and finally I'm going to show you a simple common package so this is not going to be really a service per se it's going to be a shared package between the order and the kitchen and any future uh service that comes because we are going to generate Proto files and this is going to act as an interface between the service and clients and service and servers and basically we are going to have generated code that needs to be shared among them this is one of the advantages of drpc which has generated codes for our clients and servers so I'm going to show you how you can achieve a reusable package in goang so before you write any codes I want to start by showing you the protocol buffers so here I have the official documentation on the protocol buffers and the cool thing is that it's going to be language agnostic so we're going to write stuff like this the services and all of that and you can compile this to Java JavaScript C++ wherever so let's start by implementing that into a new folder which is going to be for example Proto and here I'm going to create a file called orders. Proto this is going to contain the signature which is going to act like an interface for the order service so the syntax starts with saying what protocol version we're using so it's going to be the Proto tree and then we need to say which package it is so for example this is going to be uh the goang package we need to say which package it is so here is going to be for me the uh common SL orders so basically here I have created my go init file which has the name of the module which is just kitchen by the way I'm using gank 122 as well then let's start by creating our service signature which is going to be called orders service and here we can Define the meth signature so this is very similar to what we do with STS and interfaces in go so I have created here the create order RPC method which is going to receive and respond with the create order and the create order request so for example the create order request is going to look something like this so it's going to receive the customer ID product ID and the quantity and you assign these two different numbers they need to be different and then in the response we could just say the status which is going to be a string and the same for the get orders and the get order response but here instead of returning an object we're going to return a repeated orders so it's going to be a slice of orders so let's define the order here as well and this is how the order is going to look like so it's going to have so basically this is what's going to show on the the screen of the kitchen so you're going to have the order ID the customer product ID and quantity um this structure could be improved but for demonstration purposes we're going to keep this simp now that we have this file here what we need to do is generate this into go codes so the first thing we're going to do is create a folder called services this is going to hold all of the services so let's just create them so the first is going to be the common we're going to also have the order service and the kitchen now inside the common we are going to have the generated code for this Proto so I'm going to create a gen Proto file to hold all of this and and I'm also going to create a folder for orders because we might have different Proto files and I want to have them separated by domain so the way that we generate this is that we need to install a couple of tools so if we go back here to the official documentation here I'm under go on the quick start and I'm going to leave this link in the description Because by the time you're watching this this might change but very unlikely uh but basically we need to install this here so I already have that installed but let's try to install it either way now you need to update your path with Proto I think I didn't need to do this but I don't recall because it was already some time ago and I think this is pretty much it here is just the example so what I'm going to do is that I have created here a mic file which is going to hold the generate command so I'm going to call this gen so here I have just copy pasted from another project that I had this is one of those things that you do once and you just change the the the name the thing depending on the project you're doing so I'm going to explain to you what this means so you know what to do if you want to change so the first is the tool that we have just installed it should be on your path file then we have the Proto file this is the path for the definitions of the Proto files which is the protu orders for me then the go out is the output of the generated code that this is going to spill so it's the file that we have just created under the common service and then we have here the RPC out which is going to be the same and we just say that the source path is relative and basically this is it so let's try and hit this generate this codes so if I go here and make gen and if we go here to the common we have here codes that's generated from this command now you should not change anything from these files this is just generated code that you don't touch uh basically if you want to change this you come here you change the signature on the Proto file generate it again and this is going to be generated again into new code so take that in mind this is just a bunch of jabber you should not read this but there is one thing that we need to install here which is this package so since we're here let's just install this otherwise we're going to have these errors so let's copy this here do go gets and the name of the package so it's the grpc for going it's added now the error should be gone they are gone so we have everything ready to develop and if you take a look at this you actually have like the order strict and you see the order ID the custom ID which everything that we added and this is just implementation of grbc behind so we are going to consume those files but we're not going to write nor modify them so we that out of the way we already have everything set up so let's go ahead and start building our micro service and I'm going to start with the order service so let's go ahead and start creating a grpc server so I'm call this grpc Doo this is the transport because we're also going to have the http.com this is another transport if you have drift or any other transport you not you can just do it here because we're going to build this in a way that is going to be scalable and basically we're going to follow like the the clean architecture little bit but this is going to be a division between Services handlers and any storage you have we're not going to do any storage for now so all of our business logic is going to be on the service folder and all of the handers is going to to be on the Handler and all of the store is going to be on the store we not going to create one and all of the types is going to be on the folder called types as well so let's just create this file as well it's going to be on the package types I like to have this General package you can just import the type from anywhere else so if you are familiar with any of my videos you know that I like to do this pattern where we have the stct for the transport then we have a generator and then we have just a run method that is going to initialize it so that's what we're going to be doing um for the grpc let's have the St which is going to receive the other then I have here a Constructor which basically just takes the address and creates a new instance of this Str and then finally we have the run method which is a method from this uh grpc server basically what we're going to do is that um let's go here to the mine actually don't have a mine so let me create here a mine on the orders and basically what we're going to do is that we're going to have the grp uh servers you're going to create one new JPC server going to pass in an address for example uh 9,000 and then we just call grpc server. run and this is going to start the server we're going to do the same for the HTTP just like the same structure so it's going to be very clean here on the main if we need to pass any database we can just pass it here initialize it here on the Main and we are good to go so to create a grpc server we need to go to the grpc package that we have installed so it's going to look something like this here now what we need to do is just return the grpc server grpc server. serve and the serve takes a listener so we need to pass in a listener and what you're going to do is that we're going to create a TCP connection and send that TCP connection to the serf method so just like so here you should be familiar with the net package uh we can create a DCP connection and we just send here the listen to the serf then I have just add here a simple logging on the console and after this we need to register our grpc services now we're going to come back to this in a bit but basically what's going to happen is that we're going to create a service here we can honestly just create it here already honestly it's going to be the orders service. goo it's going to be under the service package this is going to hold the business logic so whatever business logic you have it's going to be separated from the transport from the database lawyer from the HTTP layer JPC whatever layer so this is going to be just business logic here and just like the pattern that we did for the JPC we're going to create a service truck just like so so here you could have a dependency injection of the store um if you want to see this in action I do this in pretty much all of my videos check the free course I have on building a rest API in goang I use that all over the project there and this is the basic structure for now you're going to have a couple of methods that we're going to implement from here the Proto it's going to be these ones here the create order we're going to also have another one for the listing but we're going to come back to this in the bits but for now let's go back here to the uh grbc so as I've said the services is just going to be injected into the handlers so this is service is going to be just dependency so let's go ahead and create our first Handler which is going to be the orders now in the orders we are going to create multiple handlers we're going to have the first one which is going to be the G RPC and then we're going to have a different Handler for https so whatever transport you have there's going to be a Handler for that so we don't have to do any funky stuff so this is pretty much the gist of what I'm saying which is we're going to have the grpc Handler a new Constructor and here we are going to do the service injection so we're going to pass in the service that we have just created but we need an interface for that and then we also going to implement the order service server so that is a an interface created for as from the grpc so here on the generated codes there is this interface there that we can use and that is going to allow us then to register our order service here on this function and then of course here at the end we're going to have the implementation of those RPC methods so I'm here at the types and what we are going to create is the order service that we need to inject to the Handler and basically we have these method from the RPC and we need to basically replicate this here in golang um but the input and the output is going to be whatever we want because this is going to be the service this is the business Logic the functions that we have on our sites and then the Handler is the one that actually needs to uh have that structure right so let's go ahead and pass here a context and we pass here at a context so that we are compatible with the whole world basically if any method implements context receives context we can pass it around and the second parameter is going to be the order Pilots to create the order now this comes from the generated code here and this is the one that we actually want so these are all the methods so we can re actually reuse this TR and these ones here are of course private so they are not going to be shown so I'm going to call this um actually can delete this here and do it like so so this is going to be a pointer to the orders do orders like so so this is the structure now let's go back and we can copy this name here and let's call this dependency has the service or actually order service and types from order service like so and here we need to compose this interface here so let's do orders dot like so so basically if you're not familiar with the uh stct and interface composer ability and go uh basically you can get the structure on interface and then compose it with another one it's different from in inheritance um I have a video on interfaces as well if you want to check that out but yeah so we're implementing this implemented which then we have to implement because we are implementing this here we actually need to implement all of these uh Services here which we are going to do that right now so the first one actually you can go here and get their signature really quick so this is the one to create order and here we we have the Handler instead this is our Handler which is the orders JPC Handler and let me just remove this implementation this has to be orders and orders as well let save this and here I need to get the context and this is the request so now that we have this we can actually start implementing and consuming the service so let me first um create an order structure so it's going to be a pointer to orders. order and basically here we can get the order ID now I'm going to use I could use random data but I'm just going to use some mock data then we can come back and actually implement this uh it's going to be customer Zoom uh product ID one and quantity um 10 for example and now we have the order and basically just need to send this to the service now the service is going to maybe store this into the database validations offet it is the business logic so let's do h. order service and let's create the order where you passing the CTX in this case it's this one and we pass in the order this returns an error we should handle this and I'm just going to return nil and an error then we need the response so the response we already have the type so which is this this guy here so let's copy and put it here and all I need to do is return a status so I'm going to just say success and return the response and a NE error pointer uh let's now implement the create order on the service so if you go here to the service orders we need to implement this method here so we don't have a storage um and I'm going to keep that way because uh let's see let's see uh service order service this is going to have to implement the the types so is here like so and basically um I'm going to create here a very simple orders in memory so so basically if the server dies we lose the orders but at least we have a way to start them just for um running purposes first we make it work then we make it beautiful so let's um append this guy to the slice because we already have the order uh here's the CTX and here is the order we just send it in and we just return a nil error because we don't have one and this is the service and what is this error um guess so we have to make orders um DB let's say for example it's conflicting with the orders package like so so we don't have errors everything is working um if we come back here to the Handler uh we need to implement this here so let's register the service with the grpc basically we need to First receive the service so let me just pass here as a parameter and then we need to receive as well the grpc um what's the name of this it's the grpc do server I think which comes from the main here so whatever this guy here is um it's a grpc server so whatever this is we need to pass in so we can actually come here and start registering our service so are service which is going to be new um let's go to service package and New Order service no dependes and then we have the order handlers handlers do New Order service and here we pass in the grpc server and then the dependency which is the service like so and actually you don't need to do anything with this this is correct so basically here on the grpc where is this is already correct and we need to do the G um send here the service so order service we pass in the order service so we have this structure and now so we go to orders so this is the generated code package and then we do register order service server here we pass in the grpc and then we pass in the service with the implementation so if your grpc Handler is missing some implementation here this is going to have an error here so like if you would try to implement an interface and go you would have here an error because you're not implementing the correct interface for the service which is if you take a look this one right here so you need to implement this crate order here with the same exact structure so this is pretty much it for the for the servers so are we ready to actually run this so let's try uh and run our projects see if everything is right so I have just created here on a make file two commands just to run the order service and the kitchen uh let's go ahead and try it so let's do make run orders orders and here we have a grpc running we don't have compilation errors but we are not sure if it is working because we need to create a client and one of the clients that we're going to have is going to be the kitchen so the kitchen is going to request for the order service if you remember here basically the the order service is going to request the kitchen for the orders and then the orders are going to be displayed on the kitchen uh screen so that is what we need to do so actually before we go and implement the kitchen service the last one I was just forgetting to implement here the HTTP transport for the order service so has of promise you uh we're going to have the grpc transport and the HTTP side besides uh so I'm going to teach you how you can have both so yeah I'm going to quickly move to this transport here because this is the same way that you do to create rest API so I have my free Cs on the description uh if you want to see how you can create the restful API in goang with the good practices but I'm going to show you as well here very quickly how we can do this so here it is uh we have a stru which is an HTTP server so this might be very similar to the grpc transport and it really is so we have the same structure we have a struct con Constructor and a run method and here we have the same thing so we have a Constructor and a run method but here we are starting Serax instead of starting a grpc um TCP connection and and JPC um service and here we are using the same service that we are using here so if you take a look here it is the same order service has this one so you can see the beauty of this pattern and this division of concerns of domains because if I need to change something on the business logic I I don't need to change anything on the transport layer I just need to go to the business logic here on the service and change it there without conflicting with anything else and this is how you can really build complex applications with multiple people working on it without having conflicts and all of that which are going to stay the test of time so we need to go ahead here and also implement the Handler for the orders because we need to go here register our routes so we need to create a a Handler an endpoint basically that's to be like a post to create the order and then we are having that implementation separate from the JBC so basically it's going to be very similar we're going to have a Handler Constructor as well and we're going to have our Handler uh but with a different signature because it's going to receive an HTTP request instead of um grpc 1 so again very quickly here I am at the HTTP of the handle and I have just implemented it so very similar to the same grpc but we are uh returning here different conract so which is the order HTP Handler and here we register the rout so we're going to have a post Handler now again this receives an HTTP serve Max no dependences at all we're not depending on any other framework this is just with go 122 and here we have the end point for the create order this is the Handler receives a writer and a request and then here we pass the Json this is what I talked about that we have to do the unmarshaling and marshalling and here we consume the same service the same way that we do just like we do on the GPC so here again the beauty of this separation of domains and here again we respond with the same interface with the same struct and but instead it's going to be Json and then we need to go here to the HTTP transport at the main and we need to register our Handler so order Handler and instead we pass here the router um actually no we don't pass the router we pass in the service so the order service and now here on the order Handler we do register routes and here we pass in the router that's the way so we can just save and this is how we have the HTTP I'm going to show you now with requests so if actually we need to go here to the main yeah I did some changes as well here on the main um we need to go here and add the server so it's going to be the HTTP server new http server we pass in the address so it needs to be different one I'm going to use 8,000 and then I'm going to do HTTP not server do run and here because this is blocking I'm going to add this into a g and now if we test this we have two servers running on different parts so here I have my Thunder client open and I'm going to make these requests to this end point here so it support 80,000 and if I hit sense we have success and we are just creating ERS like so so we can see that we have the grpc and HTTP transports and we any other transport you can do it very easily like so just create here a new file and you're good to go how beautiful is this right so let's go ahead and finish up with the kitchen and here on the kitchen uh this is going to be simpler I want to move a bit faster because now that we have seen how to create other service you now have the knowledge to create any other service that you'd like so it's just the same thing and I'm going to show you how you can consume the grpc server that we have just created so if you remember from the beginning we are not communicating with grbc from the browser so the user when makes the request is going to be an HTTP request so at the end we are actually not going to expose any grpc functionality to the world so we are just consuming the other service with JPC so we need to create an HTTP server here let's do nhtp go and we can basically just copy what we have done here on the orders so here I have just created basically what we did at the orders but here we need to implement the um the route so I'm going to expose the Slash and basically what's going to happen here is that we're going to go create a grpc connection to our order service and then we're going to get the orders with the function basically then we're going to display to the users in an HTML the orders displayed in a table and here is the template that I'm going to use so just very simple here we have the order ID custom ID quantity this is the templates engine in go so we're going to use that'ss as well uh you could use any front end project you would like honestly could have a react project here that is consuming the kitchen service but we're going to keep this simple so the first thing that we need to do is actually do the connection with grpc so let's go here to the main and let's create here a function now this function is going to receive the address and client connection for the JPC so here we're going to do a connection to it's going to be grpc do new clients instead of being the server we create the client we pass in the Target which is going to be address uh and some options here we need to do the unsafe um thingy which is GPC which unsafe or uh let me see it's this one I just pulled it from the documentation uh and then we handle the errors as well and finally we just return the connection and a n error because there is no error here uh actually not returning the errors because this is going to be a top level function so let's go back here to the to the kitchen and let's make here a connection so new grpc clients we pass in the address for example I'm going to use um we need to connect to the client which is at the 8,000 I think or 9,000 let's see um it is 9,000 so we need to connect to this server here so let's do the same thing here then we need to defer the connection so let's do connection. close very important so we releas the resources now here we have access to all kinds of stuff so I'm going to show you how we can do this um I would probably create a service and the same structure I did here but because the video is getting big I'm going to show you I'm going to put everything here at the Handler but if you're building this for your own uh I would go ahead and create this structure or something similar so now that we have this we need to pass this connection to the client of the order service so we need to create a new ORD um we need to go to the generated code actually and get new clients New Order service clients and here we pass into the connection and this returns uh let me see the service clients so so here we got the clients now we can do all kinds of stuff has create an order so this is the same thing as you do in a rest API so this operation here that I'm doing is calling the rest of API like doing a post to the API but here we just calling a function this is what it means to do RPC remote procedure calls and yeah we just sharing all this code between the server and the client and it's amazing um basically here we need to pass the context and what I'm going to do is that I'm going to create a context with the timeouts so I'm going to go context. s timeouts and I'm going to use the r that context because we have access to that and here's going to be a time that seconds I'm going to make two seconds for example and pass in the context here and the next parameter is going to be the order payloads so here I have just pasted the dummy payloads you could of course this could be a payload from the HTTP request from the from the browser uh this Returns the response and an errors okay but we're not going to use the response so we can ignore it here we also need to handle the error from the context actually my bad this is not an error this is the canellation and I need to Def it as well cancel like so here is where we need to handle the errors and here put it like so so now that we are creating an order it would be nice because actually let's try and run this um it's not going to happen anything visually but I want to see if the client is making the connection to the server so let's go ahead and make run kitchen and as you can see it's not showing an error so I'm I'm positive that it is working so let me just close the server now what we need to do is actually create a new RPC method here here so let's go to the Proto let's imagine that we want to get the orders list which is what we actually want so let's create the get orders and I'm going to show you how it's simple to add new methods in the future so and I'm just using what we have created initially here so down below we have already created the structure we just didn't create the function because I wanted to be uh simple the way that we created the first uh service and handlers so let's go back here to the service actually I'm going to show you what we need to do uh let me just close this terminal here and if you do make gen we're going to generate new code and we're going to have errors here on the kitchen on the order service so if you go here to the service and Handler we should have uh problems because we are not implementing the RPC because there is a new method I'm actually not sure why we are not having the error but trust me we're going to have the error um so let's just go ahead and let me show you here the the Implement see if yeah it's here weird okay let me just go ahead and implement this really quick so here is the structure so I have just replaced the uh the packages so let me do uh requests is that what we it is and here basically what we need to do is go to the service get the orders and we actually need to create that part of the service so let's go here to the types now here on the order service we need to implement the get orders um which is going to rece the context as well and it's going to return a slice of orders order and I don't think we need anything else I mean we could pass the customer ID but for Simplicity what we doing is here on the service we are just returning this database here so let's do that here so thank get orders this is going to receive the context and return the ORD so basically what you could do honestly is just return the orders and I slize here so the service is implemented so let's go here to the RPC and what we need to do I don't know why I have a function here but we need to go to the handlers order service get orders here we pass in the CTX we get the orders and then at the end we just need to return this structure here so let's do response like so and we need to change the name of this I'm going to call this uh like so pass here orders because of the conflicting name with the package and we just return the response and a new error like so so this is how we implements a quick grpc method after just adding it at the Proto file um I think this is pretty much it so let me see if I have any errors here I don't think so now here on the kitchen we need to go to the clients so basically this client that we have here and just get orders and this is as easy as you can have it um we also need to pass in a pad here which is of this type I think this is the custom ID but although we are not doing anything with it this is just to show you that you can actually uh do this way let's get the orders and we also get an error and I'm going to just copy this error handling here paste it here and now I'm going to send these orders to the front end so here I have just implemented the template passing so I'm just creating new templates and I'm passing in the orders template here with the variables of the orders that I'm executing it um this is how I do it now the last missing part is that we need to go here to the main on the the orders on the kitchen service and we need to create this service so we need to instantiate the the HTTP here so let's do the server just like we did so uh servers or actually HTTP servers let's create a new one now this needs to be a different address from those ones because this is a different service this a different um running server and we just need to do run and this is pretty much it uh we should handle the error here but I'm confident enough that it's not going to fail so let's go back here and do let me just generate again I'm not sure if we did changes but just generate it and let's run the kitchen server so run kitchen uh I think it's going to fail because we need to run the let me run the order service and then let's run the kitchen service now we are serving on Port 1,000 here on two different parts now let's go back to the browser and try to connect to this URL here so let me go here to local hosts 1,000 so something looks to be wrong here in our templates uh let me check this here on the HTTP so I see what we are doing wrong basically is orders this is not a sice of orders this is a get order response so basically we need need to I'm going to call this response and then down here what we can do is response. get orders and here we actually get the SCE of the orders let's try this again uh run kitchen I'm going to restart the server or the service orders as well and now if we go to the browser we get the list here so as you can see this is just a very minimalistic UI but you can imagine this would be what the kitchen the the screen on the kitchen for McDonald's for example would have displayed and if you refresh you can see orders being created and orders being updated here so this is nice it's working and basically with that we have implemented everything that we have designed so here we are exposing by httv then the kitchen service has an grpc connection to the order service so this is an interest service communication and then I also taught you how you can do the uh common package so that we have your grpc code that is generated reusable across all of the services that you have on your microservice project and so guys this is pretty much it we have achieved what we said um let me know if you liked this video and give it a thumbs up consider subscribing if you are not already and as I said I have something in the pipelines uh if you're watching this in the future check the description or my comments on the comment section below because there is going to be some something big for you guys so see you on the next one cheers

Channel: Tiago

Views: 12,147

Keywords: golang, go grpc, grpc in go, golang grpc microservices, golang micro services

Id: ea_4Ug5WWYE

Length: 42min 2sec (2522 seconds)

Published: Thu Apr 11 2024