Bi-Directional Streaming With GRPC In Golang

what's up everyone in this video I'm going to talk a little bit about uh streaming a bi-directional streaming in grpc I have my GG poker project open for the people that don't know what DG poker is it's the decentralized poker engine that I'm building for quite a while right now and uh which basically we rewrote a lot of stuff a couple times on the stream uh going from our own uh peer-to-peer implementation to a grpc version and with protobufas and all that stuff and um right now we have this RPC way of handling things right um but I'm willing I want to explore grpc by directional streaming for a couple of reasons uh which I'm going to show you and then we're going to try to export it a little bit and see how it behaves but first of all if you like the videos I'm providing to you please consider subscribing to my channel if you're not already leave some questions in the comments give me a thumbs up and of course jump into the Discord Community all right so basically GG poker is a decentralized poker engine completely decentralized and distributed uh with no Central Authority yada yada there are a lot of streams already on my channel so check it out uh but this video is going to be a little bit more focused so basically let me open up um my Proto my prototing real quick so we have this gossip server and uh we have a handshake we have a handle deck we have handle take seat of course we have uh in in our own implementation our own peer-to-peer protocol we have a lot more right uh basically uh we can do a lot of stuff we can we can bet and fault and check and everything is working so good although we have a little bit of a complexity problem there that's why we basically um that's why I decided to rewrite this uh the gossip thingy and and grpc right and we are using this RPC stuff that's fine but I was thinking um basically that you need to understand that each time right we are in a distributed environment right we have all these piece all the space connected with each other and each time we are going to send um there's a lot of message communication going on right and it's all in a it's all asynchronous right there is no way to determine when a message is going uh to be received and from who that message is going to be received first it's completely that's basically the complexity uh we have when working with these distributed systems right so each time somebody's going to send a message to one of a pitch it's basically with RPC it's going to handle that message on its own right so if somebody is basically sending 10 messages at the same time we have one player and that player is going to receive 10 messages from all the plays right of or maybe a couple from the same plate whatever then um basically what's going to happen is our server or not is going to handles all these matches at the same time right or trying to do that at the same time kinda at the same time and that's where that's why we need to have in our in our uh in our game State and all that stuff right let me open up game State uh here right we have we have we have a lot of atomic stuff going on a lot of mutexes this player lists uh let me quickly see that it has a there's a mutexes and locks all over the place and still most likely we're going to have sometimes I can see it I can spot some some race conditions here and it's a very very very hard to keep track of that um and to debug that and even to test it right it's just it's a hell that's why distributed uh systems at scale well we cannot call the scale but that's why distributors in general distributed systems in general is um it's just a pain in the ass right it's a com it's complex it's complex so a little bit of coffee right all right so basically what uh why do we what why can can um grpc streaming help is right so you need to understand right now we have this RPC method so each everybody can can just send a message and we're going to handle these messages as they come but if we open up a stream if we open up a stream between between player a and play it B and then with all the other plays of course if you can open up a stream then we have a single point of um a single point of handling these messages right this so the message we open up the stream these messages come in right so we're going to have a general message and then we're gonna basically have a one-off type uh in in our Proto right where we basically going to assert or going to yeah we're going to check what what's what kind of message being sent and based on that message we're going to handle that and and by doing that we we can basically handle each message although we are going to receive one home and message at the same time we are going to be 100 percent sure that we only gonna handle one message at the time and the reason why I come up with this is uh because I'm working right now for trading light and they're using uh an actor model right now for the people that don't know what an actor model is look it up it's it's a completely different Paradigm uh the actor model it's basically how erlang erlang handles things right and it's the same principle right so there you don't have the concept of of course you have the concept of States but you don't have the concept of of keeping to to synchronize your state because you're going to handle all these message uh one by one with some kind of a queue system right so even though in a concurrent distributed environment you're going to receive 100 messages you're going to be 100 sure that this message that these messages are going to get handled one by one hence your state can be freed from all these locks most of the time of course if you're if you're if you're doing things in a good routine yourself inside of your program inside of your notes of course then you need to be careful because then you can have these these these things but still we don't do that right the only the only concurrency we have is the handling of these messages coming from all these different players but this can be boring but uh hey it is what it is I needed to do this I needed to explain this a little bit right so basically what I'm gonna do is I'm gonna try to export this real quick so you guys can also see uh how this streaming could work right I'm not quite sure if it's how I'm gonna do it I'm only going to implement it completely um finished right now I'm just going to explore and and show you guys how how that gonna gonna work good work actually so the first thing we're gonna do is basically heat uh I'm gonna do an RPC and I'm gonna call this um I don't know maybe a gossip stream or something and that's gonna basically man this font is huge so first of all let me make we're gonna make a message a general message which is basically gonna call message isn't that easy right we're gonna first of all basically what's gonna happen here here we're gonna have some kind of a one hour or something I don't know I need to check uh with a list and enum actually a list of of things that can happen right uh check race fault and all that uh take a seat and all that Shenanigans right it's basically a rapid an envelope right it's what they call it sometimes so we're gonna have RPC gossipstream that's going to be a stream of message just like that and we're gonna returns also is that a stream probably a stream message not quite Church then we're gonna basically do make protohe boom and that's gonna be perfectly fine so then in note we're gonna we need to do our implementation right so we have this handshake which is basically also an implementation but an implementation of an RPC instead of a stream oh it is what it is right so we're going to say heat Funk and a pointer to our note and we're gonna say how is it called gossip stream probably gossip stream it don't take a context uh I think it takes in a stream what they call it and it's gonna be also no pointers it's gonna be a Proto gossip stream server I guess gossip a stream server that's what you're gonna do it's gonna return an error we're gonna say a return null here that's that let me quickly do Micron to see if the implementation is right yes it is this this we are Gucci uh what we could do here this fmt parental and we're gonna say received message something like that and it's going to be I don't know receive message with no message not quite sure how I'm gonna it doesn't matter you can explore it later on so we have this gossip stream then we're going to open up our main thingy and we do a lot of stiff heat with what we're going to do is we're gonna actually return heat leave that open I'm gonna join this so I'm going to quickly make a note each at level uh at Port 3000 and this if you're wondering this 3001 is basically the API post it'll be spin up and a server a grpc server and then we spin up a Json API server uh to do some stuff right like I said if you want to know more how this thing is being built there are a ton of live streams um yeah I know live streams not not maybe the best to rewatch all the time I know but hey it is what it is right take some Popcorn some coffee and have a good day uh make server and start so what we're going to do is make a client real quick and I'm gonna join some code from notes real uh like I said we're gonna do this quick and dirty right that's who I am that's how they call me make grpc clients I'm gonna copy the strings in here because it's basically not exported what is this all right make glpc client connection uh that's fine I'm gonna say that the client Eros is going to be make grpc client com we're going to connect to uh or a server heat not one right don't don't mind these things behind return I'm just doing a return so I don't need to comment it out right so we're going to do an if uh what's going on each is not nil and we're going to do a log fatal maybe lock uh fatal this adult right now we have a client right uh and then it's a bit weird I guess but what we need to do then is basically say that the stream right we're gonna make a stream and uh it's gonna be interesting if we're going to really handle this this stream and and how we gonna keep track of these connections from all these players as a stream so I need to think about that later on and that's probably going to be in a stream or maybe if you guys like to that I stopped streaming and I make these these more focused videos and that are a little bit shorter than two hour stream which is probably hard for you to navigate uh what you want I understand but hey please help me please help me make a decision about this so we have a stream each and then we're gonna say uh it's probably also an Eros and that's going to be a client and now we need to call this gossip stream right and that's uh there's going to be a context I guess not quite Church yeah it is and of course like everybody we're gonna do a context to do here real quick of a background maybe I don't know I'll do a background then we have an Eros uh we're gonna handle this like always now we have a stream right so what we could do then is basically [Music] um we can send and receive from the stream right we can make it a a one-directional stream and a bi-directional stream whatever you want to call it so we're going to say stream sent and we also have sent message I'm not quite sure what the difference is there's going to be an interface and sent it's going to be a Proto message that's actually interesting doesn't really matter so we're going to say sent um actually that's interesting can we actually send whatever we want is by generated code okay I see we need to stay away from that classic grpc um there are a lot of stuff going in that nobody knows what's going on right grpc is like a black box or full of magic and sorcery and Wizardry um you have something that is called drpc which is basically a stripped down version of grpc because in grpc basically everything is outdated it's deprecated and uh it's crazy uh yeah that's why I'm only using the simple functions and just stay away from all the rest and you should do too maybe I should make a video about Dr PC what is your PC it's basically I don't know it's it's a fast version and a stripped uh a strip down version of grpc All right so we're going to do stream sent and then we're gonna say uh a Proto message he that's our message which basically is nothing in this case uh what's going on in this phone guys I'm so sorry uh sent why is this not working what do we need nothing right message we cannot do that we need to do this of course and the question about this was it returning an error Okay cool so we're going to say if uh if still risky what's going on we're going to basically do a log vital Heat fatal just like that right so we're gonna open up the Stream and now we're gonna send basically an empty message which basically is going to trigger on the other side each is going to trigger um wait it's going to be gossip let me search for for what I need heat right it's going to trigger the received message right uh if if all goes well and basically what I'm gonna do here is do a select so we actually block because if we don't do that we're going to return and most of the time we're not going to see the Lockheed from the server because we're already returning right let me see make run if that's going to work uh boom so we received the message it's working that's fine so of course what I think we should do is each time um like I said I need to think about that but how it's going to work in in our book credential right how are we going to keep connect because how are we going to keep track of these connections because we need to keep them open these streams right we need to keep them open and um so we're going to have for example in our message message envelope is going to have an internal we're going to have a message in there which is basically a connect message which is just a one-off message where we where people notify each other that they are trying to connect now we need to basically put them inside of a map of connections and all that stuff so we can then if it was a poker engine for example if you want to fault or something we need to broadcast that fall to all these connections right some kind of a chat basically a chat system uh well not a chat system but like the broadcast itself is basically the same uh you want to broadcast the whole ship bang right everybody in the party so basically uh you could do something like this right uh Forge you can have a for Loop otherwise um it's not gonna work yes we're gonna have a for Loop and we're gonna say that the message probably Eros is going to be stream receive right uh of course we're going to say if there is an Eros and that's not nil we're gonna return the adult here in this case can you guys yes you can see that we're gonna return the adult then we have a message so we could say an fmt rental and of course we know this message for now is just going to be an um a simple empty uh Proto Dot message right uh received MSG something like that actually let's delete that uh we're gonna Loop each that's fine this will never be triggered that's all this is what it is the same thing heat um so in our client so what I'm going to try to do is basically I'm going to send off a message from the client to the server server is going to respond actually we don't respond right so we're going to receive a message Heat and then we're going to say stream sent we're going to send a message and it's going to be let's send it back all right let's send it back right now and uh check the edit of course right so I'm so sorry that I'm a little interested in typing but it is this font is like it's insane it's but yeah um and I take that for the team all right so we're gonna do the same thing we're gonna make this stream heat and then we're gonna do a for Loop right and uh basically I'm gonna delete this and I'm gonna actually to be honest I'm gonna copy the whole shebang boom paste it in of course this can we need to unlock fatal heat because we are in the main uh like this let's copy this up paste that in delete this call it a date so received message we're gonna say he received message uh from the client let's call it Pete find Pete and then I receive the message uh from the server piece something like that right yeah and another thing is basically the problem is that in stream receive uh is basically going to block a little receive so we're not gonna send it until we are receiving so the question is do we need to spin that up in a good routine and maybe I don't know but for now to make it simple I'm gonna copy this right I'm gonna paste it in heat we're going to send a message which is going to be in this case we need to do a Proto message right here then we're gonna [Music] receive the message from the server and then I'm going to sleep here otherwise we're going to ping pong like crazy so we're going to say time sleep and we're going to say time second or something and that's fine of course the select Loop is not needed anymore because we are basically in a for Loop and um yeah I think maybe that's gonna work not quite sure so we're gonna say make run yeah so I received a message from the client and then we say received message from the server because we are ping-ponging of course that other logging is basically the game state that basically locks every I think five seconds the state but don't worry about it so that's basically uh streaming and grpc uh so like I said tutorial code right that's something you guys need to understand the tutorial code is easy right you see it's working and wow but actually if you're going to implement that at scale if you're going to implement these things in a real case scenario a lot of other troubles rise up right that's that's the thing that's why a lot of that's why I don't like to do these simple tutorial most of the time because it's all going to work perfectly fine until you guys are going to use that and Implement that in some kind of a system um and you're gonna choke because a lot of things are going on that's what it is but hey I just want to make this video to show you guys um this and I'm still working and I'm still thinking about our beloved poker engine that I'm not gonna dump that project into the dumpster so basically what I'm what we're done what we're going to do then of course we need to do a lot of stuff in this note so we can keep track of these connections and disconnect and connect and all that stuff but that's for later on but the main thing to take away is that from now on we're gonna have this one stream open we're gonna delete basically if we open up our Proto we're basically going to delete these stuff right all these things are going to get commented or deleted and we're going to handle everything from this centralized gossip stream thingy and and make sure we basically handle all these messages one by one so we are so we have a real so our status released our state is basically released uh of course unless we are doing some Shenanigans ourselves inside of our note and we're spinning up our Guru routine and we are adjusting the state ourselves that can cause of course uh data condition that's that's that can pollute a state right that can cause inconsistencies but if we're using messages coming from everybody else and we're handling them in a centralized synchronous way there is no way we can have problems with risk conditions and problems with our game State and everything and because you need to understand that the polkish engine or a distributed a distributed system itself is going to be there's a lot of State going on a lot of State and with all these locks and all that so at a certain point of time it's going to be so slow and it's going to be so hard to keep track when to lock when to unlock uh what's safe to to adjust without it's just a mess and we don't want to have that complexity so that's why we're going to use this stream right so if you like these videos I'm providing to you if you have some questions hey please feel free to jump into the discourse or jump into one of these streams where I'm where I'm making this live and we can discuss a little bit more further if this is a good approach yes or no consider subscribing to my channel if you're not already give me a thumbs up leave some questions in the comment to jump into the Discord community and I'm looking forward to see you in one of my live streams cheers

Channel: Anthony GG

Views: 7,866

Keywords: grpc, grpc tutorial, golang, golang tutorial, grpc golang, grpc protobuf, go lang, learn golang, programming tutorial, golang grpc tutorial, go lang tutorial, golang grpc, protobuf, golang grpc streaming, golang grpc bidirectional streaming example, go programming language, golang programming, golang examples, golang for beginners, golang beginners, golang grpc api, golang rpc, protobuf golang, grpc streaming tutorial, grpc client streaming tutorial, grpc golang hello world

Id: IHQNiQlQVXo

Length: 22min 11sec (1331 seconds)

Published: Fri Jan 06 2023