Protocol Buffers Crash Course

protocol buffers or protobuf for short is a method of serializing structured data useful for transmitting data over the wire or storing it on the desk in this video I want to build an application with pure Jason and then I'm gonna rebuild the same application using protocol buffers and finally I'm gonna compare the results in the context of this storage in this case because I'm storing the data locally on disk right the size of the messages and all that stuff right and finally I am going to go through the pros and cons of protocol buffers because guess what no matter how fancy the term sounds there's always something bad about any take out there there's nothing perfect if you're interested stay tuned if you new here welcome my name is Hassan and this channel we discuss all sorts of software engineering by example so if you want to become a better software engineer consider subscribing and hit that be like and so you can notified every time I upload a new video with that say let's just jump into a protocol buffer so I'm making this video because it's a prerequisite to another video I'm planning to make which is called G RPC and I found myself cannot make that video without actually talking about protocol buffers so here it is right so protocol buffer is a message format right it's it's nothing really fancy it's not as intimidating and it sounds so I'm gonna demystify it right now but in order to do that let's talk about how we usually do things okay so I have here I'm gonna build a JavaScript application using not GS - I don't know build a list of employees and think of this maybe you read them up from a table on Postgres and you're populating them in your no js' express server so you can send them over the wire right but this part is only gonna be a pure employees list and there is a bunch of objects so what we built I'm gonna go ahead and create a new brand new project brand new spanking project here I call it pro tool bath so you guys gonna get that code later and it just created a new thing yes file index DOJ's so areas we're gonna how do we how do basically create an array let's just initialize npm and all that jazz npm in it that's why all right and i'm gonna create a bunch of new employees how do you create an array of employees very simple it's jason it's beautiful just like that you have an array no type nothing and that's what made Jason a very attractive and and and very popular all these years right because I don't need to specify any types and we're gonna come to that okay so how can I add a new employee I can do this right all sorts of fancy stuff I can get a curl create a new employee get a name and his name is Hussein that's me and people pay him $1,000 because he's poor a month and a what else ID ID number 1 or 1001 how about that so that's how we add employees right you can do it another way let's create another one called Ahmed and I don't know you can do it this way right so this very very similar approach right you can do it this way and this guy's a man enactment is right way richer than the same because his his dad have like three oil refineries or something like that right so he's rich alright so let's add Ahmed to the list right and then you can you can add another it's gonna add another employees here did I have it still copied that thing yeah I have it a copy rec okay Rick owns like five thousand right and his idea is one thousand three beatiful stuff guys right so this is an array and how do you print stuff right we can let's go ahead and just print it so you can show you guys right so he can just do yeah Jason don't string if I employees right we can just print that stuff let's go ahead and debug that stuff okay you can see it's printed Jose what are you doing I I don't get what you're doing here exactly we'll get that out to the minute let's say I have this result now I have this employee array and I got it no matter where how I got it I got it from the wire as an HTTP to request right or I write it from a file or ready for my database and I convert into representated now I have this array I'm gonna write it to disk just to show you guys how big this file is it's a bunch of string of stuff right so how do we do file stuff require FS all right and then we can do FS dot right file synchronous because I'm not in the mood to do callbacks right here unless someone volunteer and do a promise based file system library I'm not gonna use the callback I hate callbacks okay Zamora I rather do that so go ahead and write this stuff as a file and what is the callback was the file name let's call it jason data don't jason and just throw it there okay hopefully it makes it it get thrown in the correct folder all right so something got written let's go ahead and check the file here there you go jason data beautiful stuff right how big is this file guys how can I check how big is this file let's go ahead to the command terminal real quick so if I go to the comment amount do LS dash L edge you can see that Jason my Jason file is around 125 awesome bytes right so just for three employees I have 125 bytes and that might not be big but think about going to the thousands this goes large and large and if thinking about sending this data across the wire across the HTTP protocol yes HTTP 2 does compress some stuff but despite that it's still quite large right compared to what we're gonna show now okay so now you saw me and few quick minutes I was able to write an application with essentially JSON and JavaScript how about we do the same thing but with the protocol buffer so here's how we're gonna do the same example but with protocol buffers so broker buffers are structured data is representation of structured data Jason is not structured data you can just add anything you want here and Jason will not yell at you you're responsible of the maintaining the correctness of a schema right it's not like it's not gonna tell you that hey your didn't you send me a name here but you said name two here or name capital you're responsible for doing this it's not gonna yell at you this is a valid Jason as long as it stands you don't have a specific hard rigid schema unlike protocol buffers which forces you to use a structured and a schema well to do that we need to build a schema so you have to build this protocol which we are about to build and tell that hey an employee looks like this it has three properties and the first property is this and the second property is this and the third property is this and you have to you can do it this way you have to tell the proto file with the protocol buffer how the file looks like so how about we do that so let's go ahead and create a new file and this file is gonna called employees dot proto okay that's the extension of row brocco files and I want you do that Visual Studio code will yeah who says what the heck is a proto file I never heard of this thing I said just you you go to search marketplace okay it's really good you don't have to but it's a good idea to install this Visual Studio Pro 3 thing if you're gonna stall it it's gonna do an auto and I can intellisense and tell you if your if your syntax is wrong why are we writing profile a profile and it will help you a lot so how about we do that so the first thing we're gonna do is you have to specify which version of the protocol buffers you are using okay because this technology has a fold across the years okay along the years it's being developed I think 10 years ago maybe 11 or 12 years ago right so it's been beginning better and better and better okay so the first thing you do is syntax and that's what I'm talking about right tells you they syntax equal proto 3 and you have to add semicolon at the end ok so that means I'm using the version 3 of the protocol and that's that's the latest and greatest stuff which there are some changes I'm not gonna go through them but essentially this is the very latest version and what we're getting interested in is everything in protocol but for most of the stuff is labeled as a message okay so my employee it will be turned into a message okay so we're gonna create a message called employee and this is just a single employee and this looks very Jason abuzz no really adjacent so don't get confused so my employee in this case I will create an ID the first property of the employee but it's a schema it's a structured data you're gonna tell me what type it is so I'm gonna say integer 32 so that's the type of the ID that means the largest number is I don't know four billion wherever no it's actually more than that okay that's the 32-bit right it might be actually there right it's a four million right so ID and you say equal one and that's the serialization sequence where we every property you label it with the position of where the exact and and this is where the protocol buffer maintains the compatibility as you upgrade your file okay so if number is number one it knows that this is the number one property this is number two property and so on you have to let bits sequentially okay unique key also the second thing is a string and it's called name right and this is the second property and the third one is salary right float salary equal three okay so just like that I created a single employee here okay and if you have noticed this is a single employee how do I do an array because I need an array this is just that single employee object right we're gonna create a single own current object but I need to do also an array so how do we do an array in protocol buffer the same thing we have to create a completely different array a message right and this mustn't in this case is called employees with an S and this guy when we create that class called employees will have an like an ad function that we can add employees to that thing remember guys this is a very similar to the push stuff we did here for the employees right you have to rebuild the entire thing with proto but the beauty or the protocol file protocol buffer is the beauty of our protocol buffers it is language neutral so whenever we built it once there is a magic fairy one that Google built and they say whew abracadabra poof it won't convert into any language you want and we're gonna show that magic now so what I wanted to do is I want to create a repeated a property of type employee employee and the name of this thing is called employees and this is the first one okay and let's explain that so the first type is here right so that the type of the property in this case I want to create an employee object and this is the name of the thing and that's the first location but I want this to be repeated that means I want as many as possible so that's they are equivalent right alright guys here's a proto file how do I include it in my JavaScript file because we're gonna create a brand-new javascript file you know guys you know that right you know we're gonna create a brand new jaws of double scrubber file index to the GS okay and that's just yeah we're gonna create a new I'm but I have no idea how we can import this proto file in my JavaScript how do you do require proto new it's not that simple son here's the thing let's let's talk about this a little bit the proto file is just the file is just a schema definition about your messages and and and your structured data okay you are responsible based on the language of choice in this case it's JavaScript to convert this protocol into a JavaScript file that literally build the class that is called employee and build a class that is called employees and add properties already and I property near bad probably slash salary and all that stuff and there's all of that stuff but that just defeats the purpose if I'm gonna build this thing might as well I'm not gonna use the protocol buffer that is why Google built a cool thing called the proto C which is the protocol buffer compiler you feed it a proto file you feed it what language you want the output as and it is going to output the equivalent language if you give a JavaScript it's gonna give you a JavaScript file covalence with this if you give a Python if you tell us they give him Python it's gonna give you a Python file okay with with the same schema if you give it a C++ output is gonna output a C++ file and that's why it's language neutral it's not agnostic a little bit of a difference you can just use it right you have to do some work so the work here is to do the compiled list thing into a Java Script file how do we do that I'm gonna have the link below and the description guys this is where you want to go this is the official stuff protocol buffers protobuf releases and then as of making this video this is the final release but I want you to go and find ignore all that stuff find the proto see the compiler for your operating system has nothing to do with the language here okay you're looking for either Linux Windows or an OS in my case it's an acro I'm gonna download the proto C which is the compiler of the protobuf file right on my Mac so let's go ahead and download this sucker okay and if you're ruining Windows you download this if you're using Linux you download this or you may there might be an apt kid that does all that stuff for you so I'm gonna go ahead and download that stuff right let's save it and then extract it okay it's gonna extract it somewhere okay and it's gonna throw it in the downloads and that's fine all right you can do that and here's the thing guys this is the file we want proto C okay we want to execute that and obviously to execute anything you have to be in the same path you have to do you can do some fancy stuff and and add this as a system link I'm gonna have the I'm not gonna do that just to make the video a little bit shorter okay so I'm gonna just go to the location literally and just execute myself how can I do that okay go back to my terminal here in the same location because I have the proto C right and here's what I'm gonna do where's the proto C file it is on my users the same Nasser downloads right proto C that's extracted file and then then and then proto C that is my proto C right you can as simple as just have the same file now this C and your windows are gonna be out ax C whatever right okay the next thing we're gonna do is do - - je s underscore output I want the output to be a JavaScript right that's the output but there are multiple import formats there is the Google way of importing stuff in JavaScript there is the common J's and hopefully one days the protocol buffer will do the module the import/export es6 stuff right it's not there yet I really hope that is it's coming soon because es6 is being that is there gonna be the standard finally right but until it comes we have to use this common GS thing right so we're gonna do import underscore style and then equal common J s equal common J s comma binary and then : dot dot space employee dot space employees dot proto so that's the input profile right and this is what we want the Apple one the common J's and hopefully let's just click enter and hope for the best and it's done look what it threw up it threw up it threw up and employees underscore protocol buffer dot J's which is the same name as the profile and it has code and it's a it is everything that we need to do this thing guys it is everything we need so start building all that stuff let's go ahead and use it guys how about you we use it okay so there will be an employee class there will be an employee's with an S class so let's start using it guys so I'm gonna do a change here on the debugger thank so we can write the debugger is pointing to index two so there's our new brand new stuff right there is what we're gonna do one thing we need is we need to install this thing the Google product I'm right and it is very simple to do you can do NPM install Google - proto buff right so that's just the runtime that Google needs to install that stuff and I promise the last thing we're gonna install and that's it right so runtime protobuf right a lot of the commands and the description guys for you as well alright so why do we need to do we need to import now now I can import this stuff I can import my beautiful schema so how can I do that I'm gonna create an object called schema I like to call schema because the whole thing is a schema right and what do we do it dot employees perk up buffer that's the exact same file that we're importing okay and here is we're gonna do I'm gonna create a new employees write employees what I'm gonna create an employee called sane and the employee now he's just gonna use new because you're creating a new object of this thing just like creating classes exactly like classes here right guys so schema dot and blowy right we just created a brand new one all right so what this is essentially you will have a bunch of functions that will be auto-generated for you on this object and unfortunately Visual Studio for some reason is not showing that stuff but it's it's functions like there's will be a function called set ID and you give it the ID like that okay there will be a function called set name and you give it the name and there will be a function called set salary and what 1001 that's good and just print this thing my name is I don't know just saying don't get name right this is the first employee I just want to make sure who actually works right then ten ten ten ten there you go my name is sane still nothing is fancy here who say what are you trying to do all right I could have done this myself I could have created a class myself but you will have to do a lot of work to do that right the proto see does the this most of that stuff for you and the proto C is the template where you can create all this beautiful stuff okay we need more than this guys this is not enough this is not enough right we need to create a med we also we need to create Rek let's great Ahmed and Rick mad rich boy jeez all right where is the rich boy Israeli thousand won okay why is my son if I was annoyed this guy's like nine thousand okay and then Rick we've bouf bouf bouf Rick son three and then Rick our sort five thousand wasn't okay whatever cool now we need to create a bunch of employees array and then add the stuff to how do we do that guys we do Const employees equal who guessed it who can guess it schema.org is we do have a class or a message that is called employee right so I can do that and when you create a repeated thing you're gonna get a function for free called ad and then employees right and if you ever you can always look at the code like I can oh I cheat all the time right - because I don't know the code right I had employees so that's the function right there okay with other languages you'll always get intelligence so you'll not have this problem but sure add employees I'm gonna add Rick doesn't matter the order we'll just add them in order so we don't have anything here any problem mad rich boy all right all right and we have a bunch of employees that's good I love this I love this a lot so the final thing we want to do here guys is showing you how can we deserialize this or actually serialize this into binary because binary is what we will store it to disk because binary is where we will send across the wire and this is the power of protocol buffer this is the most powerful thank you cool stuff let's do that so what we want to do essentially is let us serialize this now into a binary write binary plus and we're gonna do employees but serialized binary hope I spelled it right and that's it literally this will return this will convert into a binary and we return the Oba there is another function called deserialize which will it will deserialize the binary that you give it a bunch of bytes in to actual data right in actual classes and objects right that you work with and just for fun we're gonna do this right so we actually know that we're actually we're getting a bunch of bytes so if I do this and run boom look at that it's a bunch of bytes 0 1 0 1 v 68 and are they essentially the the decimals right here by default but let's write to disk that's our test right Const FS equal require FS right and then we're gonna do a star right if I'll sink and then just write bytes and a scholar what did we call it let's call this employees proto bytes just binary I don't know is that what we call that okay employees binary so as you go ahead and both write it ok so now we have the employees binary we have the employees the JSON data right that's the the JSON form version that the employees version and we can't obviously open it price just a binary data but let's look at the size of this thing guys and compare it right maybe just rename it to binary data so it just look it side-by-side go back to the terminal if I do this let's look at this employees proto I don't care about that binary data is 52 byte that jason data is 125 byte oh my god how much this is like almost two times smaller okay so that's the benefits of the protocol buffer is so compact so if you send it you will avoid this extra round trips especially these TCP round trips that it does right these annoying slow stuff right so protocol buffer very very useful right because you're gonna get a smaller format and then all right so let's actually just deserialize these bytes back to actual objects and print them we can do that definitely right so we know that the object that we service is the actual employees class right so that's a very important thing so we're gonna do here is const employees to an object right equal schema dot please write with an s dot d serialize serialize always misspelling thing man bad english deserialize binary and then you pass it the actual bytes that we received whether we receive them from an HTTP call or read them from a desk right you might want to consider protocol buff buffers are a storage mechanism because they will save you huge amount of data all right desk as as we showed right so it's go ahead and print that thing console dot log employees - let's just print them and then if I do that go go go go go go go go Boop yeah we don't have to do all that stuff yeah yeah yeah yeah yeah yeah look at that we got it back print it look at that beautiful stuff a bunch of employees that's what we got back well God has an object because we didn't convert them to a string right so we can just do it like Ana force it to convert to a strength by either doing to string right right that will actually throw them right alright guys that was the example aspects of this let's talk a little bit about finals this video by talking about the pros and cons of protocol buffer because nothing it's perfect let's just discuss this thing okay so one of the benefits of protocol buffer is actually having a schema a lot of people prefer this stuff especially back in the days there only languages that it was guess they are strongly typed languages like C++ and C sharp and vb.net right and VB sexy vb v was was strongly typed most of the languages were stored java right all strongly typed they need to hey you cannot declare just a variable by just using VAR or Const right is the beautiful days of Python or JavaScript right this didn't exist back in the days so why the the schema people a lot of people prefer schema because they need to their data structure not everybody like that but it is dope you just satisfy those people by giving them a schema because schema when you know the schema you can do so much optimizations so much optimizations you can make your data efficient you can store them more efficiently you can store them you know how to read them right okay you need a schema and you get more and less less errors as well with time it is the second advantage of protocol buffer is it is compact sizes it is extremely small right footprint it has a small memory footprint it has small desk for prank and and network for fourth brain right if you send it across the wire its binary and it's so small so you can send it and E HTTP to actually will add an additional compression which will make it even smaller right if you if it can who I am which is awesome right and the third one is actually language neuter so this profile that I built here right if you are if you don't like Java skin you hey right I can give it to you and we can talk and we can agree this is like almost like a language by itself right because I give you this file you use proto C right and you say output Python and it's gonna output very similar file to this but instead of employees and just go P be the J s is gonna be employees and just called P be 2 pi right and then that's it you're gonna start using that stuff right and then just build your classes very normally so and if you Java or JavaScript or any other language is the same thing so it's a language neutral protocol format and that saves a lot of trouble ok so that's another beautiful thing and G RPC is built on top of this thing and that's the key guys here that's the key the being language neutral is very very powerful yet still Pro goggle has took it upon themselves to build us the compiler that actually spits out the language of our choice right if that didn't exist this thing is useless to be honest nobody's gonna use it if broader seated and exists let's be honest right the cons what's bad about this what's bad about this well I couldn't find a single thing that is bad about this right by searching the internet for some reason people really love it but right if I really can squeeze my head and think really hard what I don't like about protocol buffer is look at this compare the time I took to build this example versus the time I built this example right almost three times slower right because I needed to do all that garbage of grading a proto file and compiling it and bad right just to write a beautiful code right if you know your language right you have no you will not news all this extra files and put it on your can make sure you have to obvious upgrade that stuff every time because this can have bugs right and Google are responsible to to mission to maintain that but guess what it's in your code so you have to come do a proto see every now and then to to first you have to update your policy file right and when you do that Google will will essentially patch some bugs security stuff right and any inefficiencies they are they are doing with the compiler and they're doing all that stuff for you so you have to regularly update this code by doing a recompilation of the proto file to generate then you J's and I might break your code that might do certain things right it's not perfect right nothing's gonna be perfect obviously no so might generate some undesired behavior for you that you do not like right so just pay attention to that right proto see maintaining a lot stuff so it was easier to if you will know your language and you're doing small stuff you don't have to use protocol buffer okay so how brings us to the second point right the second part is I don't really care about having a schema you are forcing me to have a schema I do not like that mind you guys reminding you that why people moved from soap to rest there are two reasons people moved from soap the simple object access protocol to rest the first one is that choices XML format that that the creator of that protocol forced us to use which is so big it's it's just as twice as big as Jason okay but the second thing is also the schema some people don't like to use schema they just they want to build a small application look at all the tutorials on YouTube for God's sake alright it's just if there of all those to-do lists are built with schemas people hate themselves if you're if you're building really small application sometimes you don't really need a schema and using a protocol buffer will be an overkill for you okay so if you really need to think about it do you need a schema yeah then use protocol buffer if you really know I don't need a schema I'm finally chasing and I'm not gonna take the bloat of protocol buffer and maintaining all this extra file and having to maintain a proto file remember because you're responsible for the schemer I do the motor you have it's not having a schemas not easy right having a schema is hard you have to maintain it and you might change the format and some people might yell at your clients will yell at you I mean your clients will yell at you regardless what do you do it's it's the circle of life there is no escape from that all right guys I think that's it for me all right so we kind of throw some shade and protocol buffer but I kind of like it I love it I love it I like it but depends I'm not gonna use it all the time okay because this is the terrible thing I have to go through just to do one thing it's just a lot of work I think yeah yeah some people might disagree with me says a who saying you're just there that's just yeah you're just lazy alright guys hope you enjoyed this video see you in the next one you guys stay awesome

Channel: Hussein Nasser

Views: 75,337

Keywords: protobuf, protobuf tutorial, protobuf explained example, protobuf pros and cons, protocol buffers, protocol buffers javascript, protocol buffers google, Protocol Buffers, protocols, soap, grpc, google protobuf, python protobuf, javascript protobuf, grpc protobuf

Id: 46O73On0gyI

Length: 36min 7sec (2167 seconds)

Published: Fri Feb 14 2020