Stream: Basic Redis Client in Rust

from everybody to what has been a very long break from from streaming for me but I'm back and hoping to get back into the flow of things I sent out a tweet on the Twitter's asking people what they wanted to see and out of the three choices that I gave showing a Redis clients built on top of a sink standard was the winner so we're gonna do that today just a little bit about how I hope to conduct the stream and things like that there there are other streams that exists out there that I think do a great job of showing both beginner and more advanced or advanced intermediate and I'm hoping to hit the sweet spot of of intermediate you've gone through the rest of programming language book before you've probably written some programs but you wouldn't call yourself an expert and particularly for today probably you've never looked into acing standard before maybe you've not even looked at any of they think stuff in rust before but you're wanting to learn more that's kind of what we're going to be talking about today Redis creating a Redis client is actually a super simple thing to do it's not that complicated and so though there won't be a ton going on but hopefully that's a good excuse for us to to go down and look with this what this looks like just thinking about how I wanted to write this and I thought perhaps what I will do is also go down some wrong path doing some things that I've done in the past that I know will end up in bad places just to show you know what troubles you might run into particularly with with async await since and so new there are definitely some corner cases where you'll run into that will kind of stop you dead in the tracks if you don't know what exactly to do otherwise it looks like chat also has some suggestions there like acing standard versus Tokyo comparison I can talk a little bit about that but I'm by no means an expert and either one of them so I can just give you my personal opinion and first slots having used both not for production use cases and then a little bit more about runtimes and stuff like that so let me switch over to my desktop so hopefully everybody can still see me down here down below and what we're going to be looking at is is of course Redis but what we're going to be building on top of is this thing called a sync standard and just for for those who don't know what a sync standard is a sync standard is a a runtime for basically asynchronously executing rust code and as it says here in the tag line it's an async version of the rest standard libraries so basically it tries to mimic the API of the standard library and just give a synchronous versions of it rust unlike other programming languages does not come with its own kind of asynchronous runtime built in or our event loop or anything like that like you might expect if you're coming from node or something like that so we kind of have to explicitly set up a runtime and and execute our futures on top of that that runtime and chat was just asking is this an alternative to Tokyo or in addition on top so this is an alternative to Tokyo they they kind of hit the same spots are trying to do similar things and really the difference between the two is how they expose their their api's so async standard is is built to mimic the rust standard library Tokyo has any P I that doesn't really do that and has some some some different choices and how it exposes its API also acing standard is built on top of futures RS which is the the crate that was kind of originally set up to experiment with futures and then futures the the future trait was moved into the standards library but there's some the future trait inside of the standard library is very bare-bones it doesn't really do a lot and so the futures RS crate is still comes with a whole bunch of other stuff built on top of of those futures that add functionality to the to the standard futures trade an async standard is is kind of running on top of that whereas Tokyo is sort of doing its own thing with that as far as why I picked to do this one an async standard and not in Tokyo is basically because I think there's already been a lot of stuff on Tokyo before so let's show off the alternative that's that's one thing I also personally like the philosophy of acing standard better for the reasons that I just said mimicking the standard library I think it's just a kind of obviously good choice for how to expose the API and futures RS there has to be some kind of crate that makes talking between runtimes compatible there was that was talked before that that was supposed to be futures RS it seems Tokyo will not be going down that route I haven't personally seen them offer an alternative of something that the whole ecosystem could build around because you know you may want to use Tokyo for one project you may want to use async standard for another and also if you want to build libraries you want to build libraries that can work with both and there needs to be some sort of common language so that they can talk between the two of them and that has not really been settled on for but if you today will kind of be drinking from the async standard kool-aid a little bit and assuming that the world is going to be built up around futures RS and we'll see how we can maybe start creating a little bit of a distraction so that if other runtimes out there were to use futures RS then our library possibly could work on top of them so in some ways we'll be creating a lot of our code will be will not know anything about async standard at all we'll just know about futures RS okay so looks like and chat with I'm not quite sure you can just use one or the other can you can you explain a little bit more about what you mean there if you're building an IND application you sort of you probably want to choose one or the other because you're not going to want to have to run times at the same time that's going to be a lot of overhead you probably just want to settle on one which is why it's important that libraries kind of have talked to an abstract layer that doesn't care about what's runtime it's running on so then you can build up a bunch of of libraries on top of this abstract layer and plug it into any runtime that you want whether it's Tokyo raising standard and there are there are other runtimes as well that are maybe not as general-purpose as they sink standard or Tokyo are more focused on specific guarantees I think one of them for instance I just heard about is a vast Bastion and you'll have to excuse me to my mighty key on my keyboard is not really doing well right now and no I don't have one of those 2016 Mac books with the with the screw up keyboard it's an older MacBook but you know hey if you eat while typing sometimes food gets stuck in there but anyway this is bass II and this is another runtime that is sort of an alternative tasting standard or to Tokyo it's it's meant to be highly available and fault tolerant so if you don't really need those those particular or what it's offering up to you then you maybe don't want to use Bastion but at the same time it's built on top of futures RS so if a library works with futures RS it can work on top of Bastion or LAN async standard and I guess it could work on Tokyo but because I believe they have compat layers but from what I've seen they're moving further away from that which is a bit sad so yeah that's that's another runtime that you could could look into okay I've been talking away too much let's get into it so we're going to be building a Redis client today and for those that don't know what Redis is it's basically like a a what do they call it an in-memory data structure store so that comes with a whole bunch of data structures like lists Maps you know it's basic mostly it's sort of the key value store that a source of memory and of course you can set it up so that you flush that memory to disk but it's basically meant to be non persistent so a lot of people use it as as a cache and things like that and it has a whole bunch of features like built-in replication scripting you know stuff like that but we're going to be building a client for this that basically supports the the bare minimum of what you might want to do with with Redis and so we're gonna look real quick at the commands that we and I think today the the commands that we're going to be kind of focusing on are underneath this strings group so we have for instance this git command here it says get value by key so if you have a value stored at a key you can get it and if you want to set it down here you set the value of a key and so real quick and I hope everybody can see this chat let me know if if you can't see this so you can there's a CLI that comes with Redis that you can use to interact with it and we can basically try some stuff out here so if we want to let's say get I heard I saw a good Twitter thread that was saying foo is not the most inclusive way to come up with random variable names because what the heck is foo so variable so if we try to get some variable and it can be anything like get maybe we're storing a phone number by somebody's name so if we try to get Brian here we'll be be nil but we can set Ryan to be let's set my age then how old am i I'm 30 and then if we go ahead and get Brian again 30 comes back so for now what we're going to do is basically support a client where we can talk to read us in this way we can set values and we can get values and there's a whole bunch of other stuff that Redis can do here this is just the strings thing if we look at all there's a lot of commands that we can do because of how Redis works most of these most of these are not that interesting to do once you know how getting set work because most of them work in a simple why so we don't really have to do a lot of these to really understand how revs works I I should mention now that there are plenty of other Redis libraries out there I think this Redis RS one is is kind of the most common one this one has a synchronous and asynchronous API built into it and so if you look at the cargo tom well here you can see that it depends on on Tokyo so and not a sync standard or it does have future 0.1 we won't be using that but we're going to be going down a different route and of course when this was first developed asynchronous rust didn't exist at all and then you know it came along and Tokyo was there so it makes total sense that they they went with Tokyo for this one I think there is another Redis implementation that I saw that's new let me see if I can yeah dark rez this I just found and it's fairly new and being worked on and this I believe is the only kind of serious looking Redis client that is based on a sync standard so we can see here and you can use Tokyo as well so they actually have some features where you can switch out which runtime you want to use which is pretty cool so I haven't used it so I don't know like how feature complete it is and stuff like that but what we're building today isn't necessarily meant to be a real thing it's more for learning um I don't know if I necessarily have time to actually write a real Redis client but I'll put it up on github and if people want to work on it then we can build a little community but yeah maybe some of these other projects could use your contribution as well cool so more talking real quick before we get writing code we have to figure out how to Stratos actually work and Here I am at Redis do slash documentation and down below here here is Redis protocol specification this is showing how the RESP read a serialization protocol works and it's this is bullet point number one simple to implement which is why we're doing this because it is simple temple and which makes it really nice and basically they have like what is it's five datatypes here that you have to worry about it's very simple binary protocol is it is a binary protocol but it's also human readable as well they don't they only use characters that can be represented with with ASCII characters which is really nice and so this should be pretty easy to do so I think with all of that I'm going to take a break have a drink an ass chat if there's any questions that they have a do see there was one more async standard does async standard support message framing I don't believe that there is built-in support for message framing in async standard itself I could be totally wrong but there is nothing about async standard that would make that not possible and there's probably I would assume there's a library for that but I'm not sure if anybody knows feel free to to write in the chat cool alright I guess there are no questions about what we're doing here so without further ado then I think we can go ahead and get get started as far as like what I expect you to know I expect you to know like intermediate like basic rust so you've read the book before if you have any questions though feel free to ask if if I type anything or say anything like feel free to ask totally cool cool so we're going to great I have fan club which is awesome and there's a question the rather seal I gave the string version of 30 instead of 30 yeah that's true that's that's because getting set our string api's so it's going to store it as a string it will store 30 as the character 3 and the character 0 not as you know a single byte number 30 although to be fair I'm not sure if that's actually really true and the deep dark implementation of Redis but at least over the binary protocol that's true it will return everything as this string cool so we're going to get started first we got to come up with the name real quick so cargo new Redis CLI is very boring but we could go for that if anybody has any suggestions please write them in now otherwise read a CLI it will be gonna wait for the chap to catch up but I assume no one's got any ideas I'm gonna type it in Redis CLI going once go entice Redis a synchronous asynchronous seal a syncretic CLI is as boring as write a CLI but it has a sink in front of it Redis async all great suggestions I think I'm gonna stick with Redis CLI for now we can change it and for now we're gonna do a binary just because it makes it a bit easier but of course we can talk about how to remove you know and put it into a to library flamethrower nice pretty cool that would be cool all right so right a CLI and then I'm gonna open this up in code for now just because all right and then let's just go ahead and run this real quick and get us hello world and my my computer is definitely struggling today I'm gonna put in a sink off the Tiki is gonna kill me and I believe 1.1 is out now so we can do that oh come on my computer's oh it is it is chugging hard nope oh there we go might have to switch to them if if things go this way yes we can use cargo ad if you want to use cargo I had to add it feel free I don't have cargo ad installed and I just put in the numbers because I always forget about cargo ad but yeah do we need a parser library like Nam we could use Nam and when we get to parsing I'll ask the yeah chat let me know if you want to see see Nam it's been a while since I've used it but we can certainly give that a try or we could just try and parse it by hand and people are asking for font to go up is that better all right and I'm just gonna go ahead and run again just so that it builds and actually I should probably should do let me do cargo check because we're probably gonna use that more all right cool okay so what's the first thing that we need to do well the first thing we need to know is how we actually talked to the Redis server and by the way if you don't have Redis I think on Mac you can just install it brew install Redis same on on Linux I'm sure I don't believe that there's an official support for Windows but yeah I think there is something that you can run Redis on Windows and I have read us running here if I do see yeah here it is and it's running it's running a server on local host port six three seven nine it says here on the networking layer client connects to write a server creating a TCP connection to port 637 them so I think that's probably the first thing that we're gonna do and I'm being asked to go up one more and the font size is getting pretty big so hopefully that's good let me know if that works yes the Fanta is wearing sorry about that I do I need a new MacBook this this this laptop is yeah going on almost five years now so that's that's not good yeah I need a new one but hey I was waiting for them to fix the keyboard and they did so maybe I can get one now all right yeah I'm using i use a adele for my personal computer but i was not very happy with it to be honest but that's neither here nor there cool we're going to have to connect to a TCP socket and the first thing that we're gonna need for that is an async standard and then I think it's in net TCP stream I'll come on keyboard you can do it so we're gonna get a keep TCP stream here and I think TCP stream has a Kinect on it and then we can just pass in local hose six three seven nine like that and it will complain at us that we are not using it just row so you can say come on stream so this works I am using a PS code with vim bindings I'm not using them but I am using vim bindings I am a long-term dumb user but I work at Microsoft now so come on gotta give a shout out every once in a while so the TCP stream connect doesn't return back a stream it returns back a future that's because connecting to the TCP stream actually takes a bit of time and so we need to wait a little bit for that to happen normally and synchronous roughs this would just block block the main thread until the TCP stream actually was connected and then it would return back our stream struct to us but this is a synchronous rust and we can't block and so what we need to do is wait somehow a wait for for this future to finish so we can do that by using this right here if we call Tata wait on it and of course it's not going to work and the reason for that is in the error message a weight is only allowed to be used inside of async blocks and stuff and so if you've not done async before you might go ahead and go okay I can do this then right and that's all well and good but this if we go ahead and do this this should compile now but it's saying we're not using it okay so we can go ahead and say this but like okay we get a future back here but what if I run it like will this actually run let me go ahead and do print line hello here just to see and if we go down here and do cargo run and let it build for for a second then we will see if this thing actually goes ahead and and runs hello and spoiler alert since this is taking too long to build it will not actually go ahead and run that and this is the most important thing to remember when you're thinking about futures is that by default rust futures are inert they're not like promises in JavaScript or some other languages that have this built in runtime you can see here nothing happened rust doesn't have a built in runtime because it can't come with a runtime it is a systems language if it had to ship a runtime in every program then it wouldn't be a very useful systems language and so instead we have to take our futures and give it to an actual executor to drive our futures to completion and that's exactly what async standard is and of course also what Tokyo is so how do we actually go ahead and do this and there are multiple ways to do this you can see we're gonna do that one real quick but before somebody send to pull pull request this is a 404 now but what we can do and I believe let me see need to find where task is inside of cool and I think we can do yeah let's do that so what we can do is say use async standard tasks and then instead of assigning this here we can say task block on and what this does this block on is it takes a async block and it drives that async block to completion blocking the current for thread until that async block is is done and so this should work now if we go ahead and run this then ultimately we should see hello and we do and so this works great and this is totally fine I think there's one way that's a little bit nicer for phrasing standard into a Toki also has this feature as well is to go ahead and mark our main function as being asynchronous here and this will basically just do what we just said what we just wrote but it will do it for us and so we can do this and we don't need tasks anymore which is great of course we have to mark our main function that's async now and so we're getting a failure here because main actually async standard main doesn't exist because as you can see here we have to we have to use async standard with the attributes feature and this gives us this attribute here to work with so let's come back in here version features and too many T's in this word and when we have that then we can come back here and we have main running then hopefully it's compiling now and we have a question about connection errors and how to properly ate that up we'll will touch on that in just a second it's a good thing that this is not included by default because anytime you put a macro proc macro and you're crate is going to explode compile times and so they only they put it up behind a feature gate so that you know you have to explicitly opt into it and we're paying for it now yeah and currently what chad is asking if we're only using async standard as a dependency and yes we are we are only using async standard as dependency of course async standard has a ton of dependencies itself but we are only explicitly using async standard right now I'm gonna go ahead and and get that running right there so we get another error that says well if we look here stream now we can finally await on the stream but the stream is not quite yet a TCP stream it's a result because connecting to a TCP stream can actually air out right and so how do we how do we handle that and if you've used the weight before you know that it comes with a very nice handy feature where you can just use the good old question mark for it and that will return back out the air to the calling function but of course we're in Maine and Maine you know doesn't take a result but the nice thing is that we can do this if we change our main to be this and down here oops return back okay of of unit then up and of course I need to do that and it doesn't know what I owe is a nice thing about acing standard is that it mimics the standard of library so you can basically could in theory have a completely synchronous library and switch out all your standard imports for a sync standard imports and then just put dot awaits after everything that used to block and it should work kind of like that's pretty cool and so where's I oh it's the same thing that at the same place that it's in standard set a sync standard i/o and so this should work we're now connecting to the TCP stream a waiting on it if it's an error we return back the error and if not we get back the the TCP stream so now we have it a TCP stream cool alright so hopefully this makes sense to everybody we now have a TCP stream which is really great and what do we want to do well I think the best way to actually see if this is working fine is to go ahead and use the best Redis command and I think is documented in here and that command is ping so what ping does is simply returns back palm so we're gonna send ping get back pong and the ping also takes an argument so you can provide an argument that it will echo back to you but we're we're not going to do that for right now it looks like some people in chat were struggling a little bit getting this to work you do have to remember that for dependencies async standard you have to on the attributes feature which gives you this async standard main here so make sure if you do that great so we need to somehow send ping and here back pong from it so how do we do that well in order to do that we're gonna look and see how commands actually get sent and since I looked at this before I know that they talked about this all the way down here sending commands to read a server so it says we should be familiar familiar with the serialization format we are not quite yet but we'll get to that in just a second and it says simply a client since the Redis service and our ESP array so we send some data type array containing both strings which is another data type and we'll look at how you serialize those in just a second so presumably you send an array where every single thing in your in your command is is a bulk string whatever that is and then the read a server will simply return back something of our ESP data type so we have to learn how to serialize arrays and and bulk strings so if we go up here and take a look at arrays then we can see what the actual serialization format is and a razor quite simple so our es P arrays are sent using the following format first a asterisks character it's the first byte followed by the number of elements in the array as a decimal number pretty easy and followed by crlf so that's the character turning new line and then an additional RTSP type for every element of the array so for empty arrays it would be asterisks then we have zero elements and then we have newline our character turned into line and for an array of to bulk strings which we're going to look at in just a second we have asterisks to carriage return new line and then we of our data so we need to look real quick at how to serialize these bulk strings as well and that's also pretty easy book strings are included in the following way we do a dollar sign which is great followed by the number of bytes composing the string so we have a prefix length and then we have crlf then the actual data and then a final crlf okay so we're just gonna write out this ping command by hand so we're gonna say command equal and of course we have to remember this is binary data right so we can't just put a string because then that's going to be encoded as as utf-8 and in this case that probably would be fine except we don't want it to it might be fine but just to be safe we're going to explicitly encode this is binary right here using B and then we're going to send that command off so going back again we have to send down here it tells us in sending commands to read a server right here a client sends the Redis server and our ESP array consisting of just bulk strings okay great and we know what an array looks like it is first the asterisk character pretty simple then it's followed by the number of elements in the array and we're just gonna be singing sending ping so it's going to be one element and then we're going to follow it by a crlf and that is going to be like this so that starts off our our array we're good to go there and then we have to send the elements as bulk strings and so again bulk strings start with dollar sign followed by the number of bytes composing the string and let's think here P ing is 4 so we send for terminated by crlf like this and then the actual string data so P ing and then a file final crlf and that is the ping command that we're going to be sending to the server and there's a question in chat so the content between the quote with a B will be the binary content of what is in the source file depends on the encoding of this source file and no it will be the literal binary of each does it depend that's a good question if you were to put non ASCII into those those things what would happen I'm not sure we can take a look later and see what happens but for now we're just going to send off this ping command so we have our command and we need to send it over the TCP stream and for that we're going to take our stream and call write all on it and send the command and let's take a look real quick at TCP and so TCP stream has a whole bunch of stuff on it here but what we really are interested in are these reads and writes these are extremely important these reads and writes are async reeve and async right that they're basically re exported from the futures futures i/o library that's part of the fifth part of futures RS and what these represent is basically the asynchronous version of the read and write trait that's inside of the standard library and so if we look at writes here are kind of the low-level things that you have to implement if you want to implement write and then you get a bunch of these things for free like right flush write vector right all and we want write all because it writes an entire buffer into the byte stream so if we give it a byte stream and it will just take it and keep on calling right it will keep on calling this this call right until it writes everything in it if you just call right the plain right and not right all it writes some bytes over many the the operating system wants you to allows you to and then it returns back the size of the bytes that's that's written and so if you want to write out everything you have to loop over and continually call right until you basically get back zero as your result the write all does that for us so we're just gonna we're just gonna use write off and we are getting a an error here the error is not quite as clear I believe as it would be if we ran it down here with cargo check so we can do that real quick but basically write all is coming from a try here and so we do not have these methods available to us unless we bring in that trait and here is its tell me exactly what we need we need the following trait to be in scope for us to be able to to use this this right extension trait unfortunately the right extension trait is private or right here so we can't just copy-paste this unfortunately the best way to get at it is by doing this async standard Prelude and this just has a whole bunch of helpers in it that that you're going to need and that allows us to go ahead and write it seems like on chat some people are getting some buffering on Twitch I'm really sorry about that might be my computer is not keeping up today and if that's the case I'm I'm super sorry but there's not much I can do about it and I am in I am in Europe so your mileage may vary if you're far away from Europe but all right so we're getting an error here and it's saying hey you're trying to mute ibly right into the stream you're trying to change it but it's not marked as mutable so we need to go ahead and say it all right and it looks like that is actually working the issue with this though is that this right all is asynchronous so we are writing to the stream but then we're kind of forgetting about it and what we would like to do is kind of say right to the stream and on the next line only right hello when you've written to the stream completely and so in order to do that of course we need a weight like this and now it's telling us hey write all can fail which is true writing to a TCP stream doesn't always work so we need to handle that error but for us we're just going to propagate that error back up into main and then it will splat out something to standard error if something goes wrong so now we've we've waited there and write all actually returns back unit so we don't really have to like store anything from it so we're good there so hopefully this works let's go ahead and run it and see if at least it doesn't explode okay cool so presumably if we wrote to Redis and it wrote back something to the TCP stream well probably not because the TCP stream got disconnected soon afterwards but what we would like to do now that we've written onto the TCP stream the command we want to read back from that TCP stream the response that we get back so let's go ahead and do that and of course that what we need is the dual to right so instead of right we need read and you can see read here and read has kind of similar ish methods like we'd to end and stuff like that so what we can do then is but before I get into that when we read to end it's important to note that we're reading into a buffer of some sort so we're going to need to allocate some buffer and then when we call read to end we'll take the bytes from the TCP stream and we'll write them into that buffer and all that we tend returns back is how many bytes that wrote into into our buffer so in order to read what we need is a buffer of some sort and the easiest way to get that is by do creating a Veck of some sort and we're going to if you have used vectors before you know that there if you just call Veck like this it creates an empty vector and what we need is a vector with some some room in it right we need we need a vector that has place for us to write into and read to to end will not grow the vector if it's not big enough and so we actually need to initialize this to be something like a vector with zeros in it and how big is it going to be we can a common thing to do is like a kilobyte 1,024 so now we have a vector it's completely empty we're going to be mutating it and we need to write read from our stream and we're going to call read what is every to end passing in I'm eatable reference to our buffer just like that all right so Reid to end is very similar to right right all up here it's asynchronously reading from the TCP stream so of course if we want to actually see what was being read we need to await on it we need to say hey wait here until this thing is completely done and the next thing is it also returns a result and so we want to do question mark again okay unlike right all we to end returns back something it doesn't just return unit it actually returns back yep it returns back you size which is the amount of bytes that it actually read from the stream and put into the buffer so we can say bytes ret and this is important because we have a vector with a thousand 24 bytes in it a lot of those are zeros and if we want to know exactly just like give us the data that is important to us then we need to know where our data actually ends and so now I think the most important thing to do is to go ahead and print out the result that we got back so we're going to print line and then we'll look at our buffer and we'll look from the very beginning until bytes read and see what it gets back to us see now we should shouldn't need that let's see sing here let's take a look here and it's saying that we're reading from the buffer but we are getting back an unknown size of bytes here which is a bit strange I think what we need to do instead instead of having like an owned buffer here we need to go ahead and read from the hmm what is going on here well not exactly sure what's happening let's just look at the buffer then and that's true so we can look at this so now we're just gonna look real quick the entire buffer now if we go ahead and run this then for some reason it is just stuck so we have an issue here we are sending something to the server and we're getting back out and it's it's just hanging when we run it it's never completing so if something is not working right here and my suspicion is that we are using the wrong thing here we're using reit end which is basically trying to fill the entire buffer instead what we need to use is is read so read to end is literally trying to read as much at every single as much from the stream as I can so a completely fills buffer and read instead reads back a certain thing from from the byte stream so let's just call this instead and they have the same type signature so we should be good there and we can run this now and you can see it worked we didn't really get a very good output here obviously because we're printing the entire buffer and of course we're printing the bytes as decimal numbers which is also not that helpful and so we what we probably want to do instead is go ahead and read only the first 1 2 3 4 5 6 7 bytes or so however many bytes read are and just read that as a string and so reading something as a string is quite simple if we call standard stir from utf-8 then we hand and a buffer and this basically tries to read this string as as utf-8 that and and this should work of course we're going to get a whole bunch of garbage at the end again but at least we'll be able to see the beginning and we can see we got palm yeah so that's good this should work to do this instead so now we're just taking the the buffer from zero until bytes read and reading it out as a string so and then of course when we reach right here stream will be dropped and when stream is dropped then the TCP connection is closed which is good so we we are almost done what we're going to do here is say is we're going to now parse our our response so let's do that real quick so instead of printing it as a string oops instead of printing it as a string we're gonna head it and pass it to a function that parses that and gives us back some kind of meaningful response so what we're going to do is call a like parse parse response and pass and buffer its read and for now what we're assuming is that the what is available and buffer is a full response that's not going to always be true so at some point we're going to need to decide somehow like do we have a full response or something like that or do we need to actually read more so like we need the ability to basically chunked the reader chunked response all right so let's call parse response then and parse response is going to take and a buffer of u8 and it's gonna return the result and let's have it take a return string or string as an as an error then and we can clean this up in a little bit and so we need to look what what these responses actually it looks like so we go down here again to hear this sending commands suits to the rightest server you can see that the red server replies to client sending any valid RESP te RS RESP data type as a reply so any of these values here can be returned back and I think that the first thing that we're going to do is there this errors type is kind of special so we want to look at that and see if there and if it's an error then we can return back the result error type and otherwise then we can parse it as something else so all we have to do here is what we can do is just say if buffer is empty then we can so we can just get rid of that possibility and say return an air and D buffer debugger so that will at least handle an empty buffer that we're getting past and then we can say if buffer 0 equals and we'll look at B and we can see here that errors the first byte of the reply is minus sign or - so we can do that say if the first thing the the buffer is this - then we can return back return air and we can say but we say format error response and of course this is not the best way to do this for now it's fine and we can say this is this is cheating and doing a whole bunch of stuff here that you know we are assuming things that we can't really assume like that the the buffer is full and stuff like that and that we have a complete response buffer here well what we can do is say take the buffer from one so skip over the first byte that we already know what it is and then go all the way up to buffer dot Len then I believe this would be minus two right here so we want to skip over the are the crlf at the end of our response all right and then we can say okay buffer we want to do standard stir as you TIFF from utf-8 and for now we'll just go ahead and think for now we'll just unwrap if that fails and say from one all the way up to buffer Len minus two yeah we're gonna add tests for this for sure cool all right so now we need to pass in reference there and we can't display it so but we can do this much work mmm and here what we can do think this should work but it might not yeah I know that chars are actually represented as four byte wide always so it's not represented internally as as utf-8 cool then we can pipeline here and we'll just go ahead and take a look at it see what it returns back to us oops having all kinds of typing issues today sorry about that great it worked so let's review real quick what we did here we on line seven here connected to TCP stream we awaited until we actually did the connection if it was an error we propagated that up we hand wrote our command here then we wrote it out to the TCB stream and awaited then we created a buffers that we could read into we call read on it and at first we were calling read to end but we learned that read and tries to read until it completely fills the buffer we don't want that and so we see how many bytes that we've read then we parse the response that we get back just from zero until the bytes are read and parse the response we take a look at the buffer if it's empty then something's gone really wrong if the first byte inside of the buffer is - then it's an error and we go ahead and return back an error and otherwise then we return back this string that is contained inside of it not including the first character and the last two characters so that should do that question for for a chat what would you like to see now should we create a more complex command like set for instance that takes arguments or should we focus more on parsing here and see if we can parse things instead I think probably working on set would be the right thing to do how does everybody think about that can also start creating a command struct as well all right we'll do set for now and I think we're gonna hold off on a command struct for now but it might be interesting instead of having to write out our commands like this to have an enum that is this RESP value enum so something like this that can represent all the different types that we have so we have a simple string which for now we can put into a string here and we have errors and we'll put in and what we're going to be doing here is have RESP value own all of its data so we'll be inside of strings it will be inside of X and stuff like that this is not necessarily the most performant way to do this but for now this is how we're going to be doing it and we have integers and I think integers are just gonna be 64 or they might be i-64 you have to look bulk string and that's going to be binary data and the race and rays are just going to be a BEC of rest values cool so now what we can do is have something that basically builds these commands for us so we can do something like in full best value F and I don't know serialize and it's gonna take a rest value and it's going to return back effect of you eights so it's just going to return back a vector with all the bytes set into it for that value and so of course we can match on self and for ping what we needed was two things so we'll do those first we needed array and there should be a raise another ring and not integers array oops oh come on computer you can do it so if we have an array we know what we need to do first is up here we needed to write this the star and then the length so we need to create our buffer that we're gonna write - we'll just create it like that and then we'll say but we can do both dot push mmm how do we want to do this might actually be good to do because buffer also takes also implements read or write sorry we can we can just write to it so we can say write star like this and then we can say write the values Len and we need to make that an actual string value so we can format that like that and then of course this is going to return back to string and what we want instead is bytes like that oh sorry it's not there their fights okay these should never fail I believe they should never fail as long as we have enough room hmm this is tough yeah let's do this for now and we can take a look to see what it looks like yeah Chad is asking to do to do push instead that's actually the better way to do this because it grows the vector along but I just wanted to to try this out real quick with this with this what we would have to do instead is you know yes at what the size would be and if we wrong that it would panic and stuff like that so okay fine we'll go back to to doing that instead we'll do push the issue with push though as that push only takes one element at a time right so we and I don't believe the back has like a push all or anything like that does it oops last back yeah don't believe that there is any kind of thing to push a whole collection of things and to we could do append but that takes a vector as well and what we really want is to pass an immutable slice and I don't believe that there is something to do that but if take chat let me know if you can pick again I see chat now it's recommending append I don't know if we want to do a pen but we certainly can do it it's I mean we're being probably too careful about performance for this little toy Redis client here so if well in some areas and another area if not so all right so we're pushing this on then we can go ahead and think there's a two back and we want to append and we don't get to it rap anymore great by the way underneath here let's just do this now so we can get rid of the air will return buff all right so see real quick need to turn turn string into oh come on keyboard is killing me we need to turn a string into into a vector there's from utf-8 in two bites tears in two bites alright I'm not expecting mutable reference here okay what are we getting now saying we're getting that cop yes because we want I believe we can do this cool yep good thing when I cover everything so we need to say I'm implemented and that should take care of it all right so we're getting somewhere now so we have the the array we're pushing on the the asterisks we're pushing on the length if we go back here down to array we can see that we need the crlf now and I'm just gonna do this the first one is the character turn and then the line feed so that's good and then we need to go over for value and values and call duh serious so this will reversibly serialize the this will recursively serialize the values inside of the array and this brings up a good point like we're going to be creating new buffers at each time and of course we can append the two buffers together and can you do this fabulous awesome great thanks Chad so I think maybe what we should do is take in a buffer here and say that this is going to be like this and then we don't have to return and we can just recursively send down the buffer called the buffer before keep you look buff recursively send down the buffer whoops needs to be a Veck sorry cool and then i want to say we don't have to end the me so we don't have to end it here so that's good and then we need a bulk string right so RESP value bulk string come on T you can do it and we're gonna have to do the same thing so just like we did above the bulk string here is dollar sign then the length of the string then crlf then the data then CR Allah so let's do that so it's basically gonna look very similar to this so instead of asterisks gonna do dollar sign then data here and then crlf and then we need to basically append the data oops and here's a here's another interesting question that we have here we want we basically want to take the data and write it into our buffer the way that append normally works is that it takes a mutable reference into to another vector and basically like takes the stuff from one vector and puts it into another but of course serialize it's just taking the respite value by immutable reference and then we're trying to like take the data out of it which won't work right and there are several ways that we could fix this we could clone we could clone the cop stuff we can you know take mutable reference but then that's weird like I think the best way is just serialize well we'll own will basically take the rest value by by ownership and then it can do whatever it wants with it then we forgot to keep him unimplemented here so now we own data we can do whatever we want with it and as long as we mark it as mutable whoops come on T you can do it cool so that's pretty easy and then I think the last thing we're missing is after we write out the data we got to do another CLR F like that cool so then now we can replace our command here with buffer Dec and then the command will be rest value array and this is going to take in a Veck of rest value bulk string and we're going to do ping like this and fortunately we got to do in two back on it visit to back with this alright and of course buffer we got a pass in there and it's a in here pass in as a reference and of course this wouldn't work we created a buffer we created demand but we gotta like right to command into the buffer right so what we can do is say command dot 0 lies and that send the buffer like this now we will serialize the command we're not marketing it as mutable very sorry rust we will do that for you and that should do it and hopefully what happens now is that we just get back pong again let's see yeah pong cool ok that's basically just a convenience for us so that now we can build our can our commands in these kind of more declarative ways may not have to build up like you know build up the byte stream or the byte buffers like by hand but we can just have this serialize method do it for us awesome okay so now let's do set real quick so we can take a look at set inside of our commands we have commands again commands and what does set do you set sit sit so set takes a key and a value and then some optional arguments we're going to to just send peen value for now so our command is still going to be an array like it always has been and this time what we're going to do instead of passing in P is we're going to pass in set if I can type T and we want to have another arrest value bulk string and we're going to [Music] what are we going to do let's do Ryan that's the key guess I'm being narcissistic today two back to back and rest value both string let's see that's a good one Berlin that's where I live so this is gonna be key value store of first names two locations to back all right so now that's set and we should be able to run and see what we get back from and we just got back to okay which is great if we we expect we expect it okay which is good that means it's been set and this is great we can start building up let's let's do like a little function here just a set and it's going to take in P and mmm we'll do key as a string and value as a string for now not sure if keys and values and right us have to be a skier or not that would be something that if we were actually building this we want to look into and we are going to return a result of nothing what was the success or let's just say air for now on we're gonna have our own arrow type cool and so what we want to do is actually just take this command and put it down here so we have to ask ourselves what is going to be the the actual API for what we're doing here and do we want presumably we want set to actually go ahead and make the call right but we need extra information like we need the TCP stream to be able to send on and stuff like that so really we probably want a struct called client that's going to hold on to that's that TCP stream like this and then we have access to it and do whatever we want with it right but of course we know when we send when we send something off over the TCP stream that we we want it to be asynchronous right we don't want it to to block our current thread that's the whole point of why we're doing this so we can go ahead and mark it as as async and now all of a sudden it's asynchronous of course and return okay here and now that we have that we can actually make this a method instead of a free-floating function and that means that this can take self now instead and this is all I can good so then all we need to do is take what we have here right and this should should work great yeah we're getting a real quick message here saying that when when you call question mark what that does is it tries to convert the error that it's seeing in the error result into the air that you have up here and so the error that we're getting back is an i/o error but we have no way of turning an i/o error into our custom error types that we have here and so it tells us exactly what we need to do we need a standard convert from so we want in full standard convert from i/o error for error and a boovies it should just be like this and for now since we don't really care what it is we can just return that great sorry just make this destruct for now we can think about error handling and in just a second so this seems to be seems to be good now all we have to do here is we should probably have a convenience method on client like new and I think the people client the best thing for us to do probably is to take in what TCP connect takes in so this two adder saw address soccer socket and that way we can just create a can create a TCP socket on our own this is gonna be oh my gosh io error this is great well that means we can basically just take all this stuff here not all that just this just copy it in like that cool so we need to we need to where is it to add earth where is that thing - soccer - goose-step async standard net yeah cool great and of course NASA our alright and this gives us a stream and we can say client stream and the last thing we need to do is wrap it in okay everything has worked out cool so this is our new client we can create our client so let's do that up here and say instead of this will say client my new local host but is it six six seven three nine is that it six three seven nine six three so now we have our our client here we can call client outset and say Ryan Berlin and because of because we've said we'd take strings for now we have to turn these these string slices into strings but we can take care of that in a little bit of how to make that nicer and of course we need to await and we need to await let's keep this thing alright so and the last error that we're getting is again we can't convert from our error back into an i/o error so let's just go ahead and for now we can just print that out just take a look at it all right yes we need it to implement debug just in case it's an error that we want to show on the screen so derive debug and hopefully that takes care of it great all right so let's finish up our set real quick so we once set like that we want our key now to be key in two bites and we want our value to be value to bytes and then we have our buffer or we see air live their command we need now to actually make this mutable here we can take care of that so in the future we don't have to do that but for now let's leave it like that because we're actually changing the stream and thus changing our client every time we call set then we write it out presumably we want to read back from sorry we want to read back from the stream again to make sure that we to make sure that everything is ok and so what we could do is stream read could try and reuse the the buffer that we have here and see if that works let's do that for so to get back bites red and of course for now what we can do is take a look at our buffer so if buffer zero equals do we we could now we could use parse responses do that you've already written loss already so parsed response are sponsz and we're gonna pass them the buffer now all the way up to by it's read and basically parse response that's just gonna check I close it an error or not so we can just pass it back up and let's make this take a air and for now our our error handling is obviously leaves a lot to be desired but we can just do this for now and add variance for all this stuff later on alright great so assuming everything is okay then what we should see here is just okay everything's good like okay tuple so let's see awesome and then if we say read a CL I get I am it set as Berlin awesome so we have done a pretty good job there and we're starting to get into like having an actual implementation that we can expand and stuff like that so there's a lot to improve on here I think what we'll do real quick is just implement it and then we can talk a little bit about how we would improve this and we'll call it a day mess in short stream but there's lots that we can do to improve it so let's just do and chad feel free to ask any questions or comments or anything like that while we do get so good it's just gonna take a value here oops and it's just going to be gets here and there's gonna be no value and we still have a buffer here we see realize that we write it out and we read it back and then we want the actual response to be here and what we're getting back from response here is based on the buffer so again this is not the best way to do it but we can do this for now take that buffer and just copy it onto the heap as a string and that should do it so then up here we can just say we're going to set and climb get of course and we probably won't cut that out this we can see what it is this alright cool so let's go ahead and change this to I'm originally from Tampa so we can go ahead and say that and if we go ahead and run this we'll see what it says Tampa yay awesome great we have another interesting I think our parser is slightly off we're getting five here so there's there's something to be worked on there but but this is already getting to a point where it's it's sort of working so that's wonderful so let's go ahead and in the stream with what we can do to improve this so we have the stream and we're just we're just holding on to it normally like this but that means that any time that we mutate the stream we have to say that we're mutating the client but really this stream is kind of just a implementation detail that we're never going to pass back out and so what we want to do instead maybe is to wrap this thing up in in something that allows us to to check the the runtime rules or check the Baro trekker rules at runtime so going ahead and putting this inside and sort of inside of a red cell or something like that there was a question about unwrapped as well and wrapped is basically just taking an error and if it's an error of panicking and if it's not returns that value yeah other than that there's there are some we obviously need a proper parsers so we need to go through and parse our responses more correctly for instance we probably should jump through the response and if we don't get back a full response then read again on on the TCP stream I believe that Redis I mean Redis can chunk its responses back to the client so we probably should be prepared from that right now we're just assuming that we get it all and one go and the nice thing about the way that responses are returned is that for arrays and stuff like that you are getting the length upfront so basically you can just go to the length jump ahead to that ensure that there's a a crlf right after that and then you know you don't have to look at the values in between so you don't have to scan the entire response in order to parse it yeah obviously other than that we need a nicer error type we need to support more of our recipes and more commands and stuff but in the end what we have here is more or less a working reddit line there's not much more to it we already support pipelining and the fact that as long as the client is alive the TCP stream will be alive and so if you do we call set and get here and we're sending those both of the same TCP stream we don't create a new connection which is somewhat of an advanced feature in a lot of other clients Redis clients that's kind of just falls out of our implementation here almost for free and that's about it so think if there are no more questions from the chat this is it I'll get feedback I think online to see what you thought about this and if you would like to see more of the rightest client I had a couple of more ideas for streams like creating a small binary dependency free binary for displaying pngs on a Mac so really just calling only into the the operating systems API is directly we might not be able to do it I'm on a Mac if they're only exposed through an objective-c API so we might have to do out in Linux instead but that would be one thing there's a ton of other things that we can do as well so I hope this was useful for you I hope it wasn't too fast or too slow and it was definitely a lot of fun so thank you very much

Channel: Ryan Levick

Views: 6,787

Rating: 4.9593906 out of 5

Keywords:

Id: 8TfjFZ478Rs

Length: 108min 8sec (6488 seconds)

Published: Fri Nov 22 2019