Fast and efficient data serialisation with Protocol buffers (protobuf) in .NET

hello everybody i'm nikki and today i'm going to show you how you can use protocol buffers in dot net protocol buffers are a language neutral platform neutral extensible mechanism for serializing structured data it's quite different in terms of how it works as opposed to something like xml of j or json i'm going to show you how that works um but it's a very efficient and a very fast way of serializing data it's efficient because it actually doesn't store ads much information and it's also faster because while it's not throwing as much information but it also works quite quite differently in terms of how it is realizing the data in this video i'm going to show you how you can use it in your project and i'm going to use it on an example that's probably something you are using right now especially if you're using something like redis this video is part of my general.net tutorial series so if you don't want to miss any episodes please subscribe there's a notification bell to get alerted when i upload a new video so let me show you what i have here all i have it's an api that has a single end point the get weather forecast it accepts a date time which we only use the date part of and then it's using a randomized generator to provide you with 10 days worth of forecast from the day you requested it that's all it really is now because the weather doesn't actually change so frequently and the forecast doesn't change that frequently as well we're caching that response for one minute because imagine if we had like thousands of people calling this api you don't want to regenerate that and call apis to get the forecast every single time so we're caching it for a short period of time this caching happens in redis i have provided a docker compose in the description down below alongside this whole project so you can spin up your own radius in docker with redis running i can show you that the cluster is up and alive and now i can run the project and show you how it functions so as you can see i have a single endpoint and if i click that endpoint i'm getting the weather back and if i click it again the response is very very fast because we've cached the data and if i refresh that you will see that we have one key and this is the json body of the response that we are caching and in a minute this will be uh deleted and as you can see this is 813 bytes i don't know if you can say it's a bit blurry but it's very very small it's almost one kilobyte fairly small however we don't actually need to store things like date or temperature or summary protocol buffers or protobuf will actually only store the values because you have a contract for how this should be serialized and deserialized so let me show you how that would work in my cached weather forecast service this is where the actual serialization happens we have the database connection to redis and we're getting the cached value and if the cache value doesn't exist sorry if the cache value does exist then we're doing a json digitalization using the built-in json serializer and we're returning that value if it doesn't exist then we're calling the actual service and if this looks a bit weird to you um this is using the scripture decorate method i have a video dedicated to that i'm going to link in the description down below as well you can check that to see how that works injecting the same service within the service but we're hitting the actual api to get the forecast then we're storing it in redis and then we're returning it so every other time either we call this the value will be coming from the cache and we won't be calling an api to get it and the idea behind this is that it's very fast but we can make it faster and we can make the data that we store smaller by using protocol buffers the first thing you want is you're going to go to the nuget package manage packages and you want to find protobuf hyphen net and you want to add this unit package normally protocol buffers use their own contract that looks like this i'll put in screen right now and then you generate your classes out of this however we don't need to do that because protobuf net will actually provide you with a lot of c sharp um attributes to generate that all behind the scenes without even having to worry about it which is amazing so for the purpose of simplicity i'm going to use the same um contract that i use on the api level hold the database level and in a realistic scenario you might want to separate them but for now i'm just going to use the same so with that imported the first thing you want to do is you want to use the proto contract attribute on the class level so that is step one the second thing you want to do is you want to add the proto member on each property and you need a number as well and this number is very important it's called the tag and you need that because like i said before we don't actually store for example date or temperature c or summary in the data that we store we just store the values so protobuf needs this to know that okay first i need to deserialize a date time which is a string in protobuf i think then i need to deserialize an integer which is an int here and then i need a string again it doesn't know what the name of the string is it just knows the order and that's how it works and you can also have nested things so you could very much um have a nested object of some type i'm just going to use the same type here forecast and have it as 4 and then this object internally would also need to be decorated with protocol protocontract and its own proton members and whenever you make any breaking changes well you can't make any breaking changes because they will break the data that you store but whenever you add something you can add it with decorating it with a proton member and provide a tag and then you won't mess up the ordering so in that way that's how protocol buffers will do the serialization without actually storing the name of the property it's very useful so with that out of the way you currently are just simply able to swap out the json stored data with protocol stored data and let me show you how we can do that i'm going to go back to the cached weather forecast and this realization is a bit easy or easier than serialization so first you want to use the serializer class make sure it's coming from protobuf and do deserialize and we're going to delete that and almost there this requires a stream so we will need to convert that cached value we're going to use the reddish value which has an implicit operator in a byte array because if i show you what this digitalized method uses it uses a read-only memory byte which is translating to a byte array with operators behind the scenes so that's the idea with visualization now serialization is a bit more complicated because we need to use a stream and or a byte array and we can't simply say serializer dot serialize and then replace that because a serialized method accepts a stream so we're going to convert that weather object to a stream the way we're going to do that is we're going to write a small method here private static byte array of proto c realize of type t and then t record where t is let's say class and then using var stream let me just do this new memory stream did i just say memory steam well that's embarrassing and then you use a serializer class which is what i have here to do serialize on the stream and then the record and this will write the record to the stream and we're returning stream dot to array and now we turn the record into a byte array through a stream and we can use this protoserialize method here to serialize the object and now that's all we need to do to get started it's so simple now i'm gonna restart this service i'm gonna go back to um protobuf and i'm gonna run this endpoint and as you can see returning again and then i'm running it again and again again again and again still very fast but in redis we no longer store json we now store something that you cannot read however protobufcan and it's only 280 bytes before we had 813 so it's almost one-third or one-fourth of the size so your redis instance that has limited ram to store data can store three to four times as much data depending on your object by just changing the serializer so you're making more for the same amount of money i think that's brilliant and this just shows you the memory from footprint in terms of size but let's do a quick benchmark i have one here to see what that would mean in terms of performance as well so if i uncomment on this what i have here is a benchmark that will instantiate the service and do adjacent serialize and json this serialize on the exact same object that we've been using and then we also have a protobuf serialize and deserialize using the exact same method that i have in the class right now and i'm going to run this just to show you the results because they're quite impressive remember this will do well i totally run the wrong project but if i select the right one this will now run a bunch of tests benchmarks to see things like the memory but we care about the performance because we know it's smaller for protobuf already by checking redis and we'll show you how much time you spend in these operations and you might say well we're talking microseconds so it doesn't make much of a difference but when you have thousands of requests coming in you're using less data in your storage but you also are executing things faster and it just adds up and again this is just one use case of the scenario so you might find other use cases and as you can see it's 17 microseconds for json and 5 microseconds for protobuf which is again one third the speed i'd say this is quite significant even though we're talking about these small numbers you're gaining something and i think the biggest performance personally is the actual storage game not so much the speed gain but understanding how this works seeing how you can use it yourself in your projects and also using that video as a segue to glrpc which is something we're going to talk about in future videos i think this is a perfectly valid use case for something to uh for something like this you'll find all this code in the description down below please download and play around with it there's so much you can do with protocol buffers this is just scratching the surface and it's meant to be an introduction that's all i had for you for today thank you very much for watching special thanks to my github sponsors for making these videos possible if you want to support me as well you're going to find a link in the description down below leave a like if you like this video subscribe for more content like this ring the bell as well and i'll see you in the next video keep coding

Channel: Nick Chapsas

Views: 16,143

Keywords: coding, asp.net, .netcore, dot net, core, C#, how to code, tutorial, asp.net core, js, csharp, rest api, lesson, dev, microsoft, microsoft mvp, .net core, nick chapsas, chapsas, asp.net core 3, .net core for beginners, .net 5, dotnet 5, Fast and efficient data serialisation, Protocol buffers, protobuf, protobuf in .net, protocol buffer in .net, redis, dotnet, .net

Id: sdKBhDl-dCs

Length: 12min 55sec (775 seconds)

Published: Mon Nov 23 2020