The Why, What, and How of Pinning in Rust

everyone welcome back it's been a long time since last time but finally I'm now back I also have a new microphone which is really cool one of you gifted it to me and that is really exciting that I can do something where people feel excited enough that they will contribute to help make things better I am John as you can see from screen I've been doing these streams now for over a year and I do streams on all sorts of rust related things some of them are live coding streams some of them are open source contribution streams and some of them like this one are more sort of educational streams and so there's a Twitter where I announce upcoming videos I also have a YouTube channel where a bunt where I post all of the all of the sessions we have after the fact so you can go back and look at any of the old videos if you want to um I also for various annoying reasons I can't actually have a patreon in the US and so instead I've recently started this Amazon wishlist that's linked to from my Twitter so this is how I ended up with this microphone if you have something to spare and you want to show your appreciation for what I do then please find something here that you think could be fun to fun to buy to help with help me and help with the with the streams so today's stream is gonna continue on from a stream that we did a while ago which is this one the what how the what and how of futures and async/await in rust so this stream we talked through like what our futures how do they work what is Tokyo what our run times how does all this machinery fit together what is async/await how does it interact with futures how does it change the way you write code and as part of that video we also touch on this notion of pinning and we talked a little bit about what that means but I feel like there's a lot more space for being able to talk more about pinning and that's what this video is going to be about it's gonna be about what is pin what is unpin why do we need them and how do you use them I'll try to keep an eye on chat for questions as we go these educational streams it helps a lot to sort of observe what you understand and what you don't as we go so if there's anything you feel like I'm not entirely sure what he means here please ask a question because that way there are probably other people who wonder the same thing as you do and I can then address it and also your the question and the answer will then be recorded in the stream for other people to learn from right so the reason we're doing this I have this live livestream voting site where a lot of you have voted in the past and if I can just pull it up here so this you can vote for what upcoming stream IDs ideas are most exciting to you the current winner is this open source contribution stream which i think is really fun we've done one in the past but I had this idea I really think it's important to someone do a stream on pinning and so I decided well I should do one and so I did this poll on Twitter to ask people what they wanted to see and as you can see the the overwhelming majority of the people who voted wanted to see like a deep dive into pin and unpin and that's what we're doing I will try to also take a few of the practical considerations into into account when doing this so my hope is to also get to look at some code and how code changes when you're moving to pin but it will be primarily about sort of the the ideas of pin like why do we have them and how do we use them great all right no questions before we start that is great so let's just dive right on in so as part of recent rust release pinning is something that landed in the standard library and pinning is usually talked about in the context of async/await because that is primarily where it comes into use in rust today but it's not really about async/await pinning is a more general concept that happens to solve some of the issues that async/await ran into so what we're going to do today is sort of look at why do we need pinning in first place why do we need the concept of pinning and what does that even mean for something to be pinned and part of that is gonna be discussing the pin type so pin is a is a struct it's not a trait or anything it's just a struct and also part of it is gonna be talking about the auto trait unpin so auto traits are things like send and sync and unpin which are automatically implemented by almost all types in the rest type system and that the compiler will auto implement for your types if all the things it contains are already unpin we'll also look a little bit about more sort of tools for dealing with pin but we'll look at those later so what is like the fundamental problem the paninis trying to solve well pinning there's a lot of text here that you can read if you want to but we're gonna look at this from sort of a code perspective which is as follows let's say that you have some struct foo all right let's call it you have some struct parsed and parsed in parsed you want to have a bunch of bytes and you want to have like wrapper which is actually let's do this owned parsed and you have a struct parsed let me just type this out and then it has a bunch of fields that reference into a so for example like name so the idea here is that you write a parser and your parser is gonna generate these structs called parsed and they're gonna have references into the byte stream that was parsed in the first place right so this is totally fine it means you can write like yes if we wanted to write it this way it takes like a a bunch or it takes an input string and it returns something like a result parsed right but notice that there's a lifetime here who knows what the center tecna's there's a lifetime here in parsed because it refers to data that is in the input and now imagine that you wanted to write a version of this where they called owned parts that didn't have a lifetime and the idea here would be that you have a buffer that has a bunch of bytes and you want to also have the parsed representation stored with that buffer so now you can sort of think of this as this parsed its lifetime is really sort of like self right it's this parse lives for as long as this buffer lives sort of right it's a pointer into it has a bunch of pointers into this buffer I guess it should be you a just for consistency so this has a bunch of pointers into this buffer so as long as those bytes don't move this struct is fine and you can move it around you can keep using parsed and you don't have to worry about the fact that there's sort of a implicitly a lifetime tied to it as you might wonder well this seems like a very nice use case and it's true but it happens to be that this pattern is pretty common in fact for async/await if you write something like an async FN foo and let's say it takes no arguments and it doesn't really return anything and you write something like what's it doesn't really matter but let's say we have a buffer of something here and then we do something like I don't want write out the things here but let's say we do something like read you buf thought awakes and then we do something like right wait fine I guess this will take in mute gin a sink read plus a sink right or something it's not exactly important what this does but the idea is that you have an async function you use some awaits in it now this might not immediately remind you of this there's there doesn't seem to be anything similar but if we dig in a little bit more we can actually see that these are the same and give me a second so here when you call food the first time let's do this a little bit first right so this is really this right these are the same mostly I guess technically there's something like an async move here sort of like this right but the question is what does a wait did the sugar - well really what happens under the hood here is that is that the compiler is going to look for every a weight to point and sort of turn it into a yield you know I'm gonna tell you what yield means in a second so let's imagine it gets D sugar to something like this the idea here is the first time foo is called it starts executing until it hits the first yield yield gives you a future right this produces a future and now it's gonna it's gonna continue executing that future until it completes but the question is what happens when it completes well when it completes and foo sort of continues from the previous yield point it needs to continue from here it can't start from the beginning because then it would do the read again it has to sort of continue from here and the same thing when this yield happens this produces a future when it eventually completes foo is gonna continue from here but what does continue from here mean it means that any of the state that the that we have already it's gonna have to be kept in particularly this buffer here has to be stored somewhere right because if you're gonna continue from here you're gonna reference buff right so so this sort of has a pointer up to buff similarly this future also has a pointer up to buff so really what gets generated here is something like a futuristic there's a start and in start you just have this buffer I guess this would be you ate something and then you have a like step one which is a IO read future right but this really sort of has a lifetime it has a reference it contains a reference to the buffer so it sort as a reference to start in a weird way and similarly this right future here also has sort of a reference to start you see what I mean because this future is trying to read into this buffer similarly this future is trying to write from this buffer so both of these types are really the problem and that the word struggling to express both in this tick self and this tick start is that they're sort of self referential right they want to have pointers into themselves how would it know the yield it got is the first one so the the compiler doesn't actually generate yields but you can think of it as it remembers where it last awaited right so the compiler is gonna generate code that almost looks like a state machine right so that foo is gonna be called again but when it gets called that it resumes from the right place you could imagine this is some something like an enum right where the next if if you're in start then you execute from here if you're in step one then you execute from here and if you're a step two you execute from here right so it's really just the compiler keeps track of where you are in foo and where it shouldn't continue from regardless though all of this has to generate self referential code code that has pointers into some stuff that it itself owns all right so why am I telling you this well who's the caller from the code the caller is whoever whoever tries to await this future it's the caller so why do we need pin for this well pin the idea behind pin is that the moment you place something in a pin you are promising that you will not move it again and why does that matter well let's stick to the simpler example of our parser here once you have parsed into once you've parts this buffer you can no longer move the bytes in this buffer if you did then the pointers the parser keeping would be invalid right if you move these bytes then the pointer in here that points to those bytes would be invalid and the whole struct would become invalid so if you have so this is where we get to the pin type so we can look at the definition for pin so pin is pretty straightforward it's as a struct that is a P where we learn more about P is when we look at the impulse for P so pin P the P has to implement D ref so think of this as P is a pointer type and the target of pin is the type that we want to talk about so for example you can have a pin of a mutable reference right so then the target type of the DRF would be a t so for example let's see what's the best way to exemplify this imagine that actually let's bring back our a sink here a sink foo bring it back to the way it was alright so here so here the weight points right here and the weight point right here require that buffer is pinned right the moment you have gotten to one of these futures you are no longer allowed to move the buffer because this future and this future both have pointers to it and so this is why if we look at the future trait let's bring that up here so the reason I talk about future here is more because it's the currently the easiest application of pin not because pin is necessarily related to futures so if we look at future what this so the poll method of future is the thing that drives a future forward is saying that self needs to be pinned and why does it say that well it says that because if you have a future that is like an async function then only when it is pinned is it safe for you to keep executing it or let me mmm trying to find the right words to say this ok let's let's think about this just from this perspective first of all so this is sort of going to generate well at the point where were the the thing that we're waiting here the future that we're waiting here all right let's try to write that out so it's an i/o read and it's a future it has a reference to a buffer hey somehow has a reference to a buffer that is the future that gets generated and it like keeps a stream which is a mutant async breed right and we're gonna implement future for future for IO read output is I guess something like an IO results.you size but the exact type is not important I'm going to leave off some of the less relevant parts here but oh man I can't type today yeah okay so this future is written like this but the future of Fuu needs to include buff right because that there's two returns of future so what does that future look like well in this case so this does some stuff it's not terribly important but the larger future that is generated for all of Fuu is something like buff which is a you 8024 and an IO read which needs to have some lifetime really it has the lifetime a buff now you can't write self-referential structs in rust today you can write them with pointers but you can't really you can't write them with normal lifetimes right so this future has a reference to buff sort of internally you could imagine this is something like if you wanted to approximate this you would use like raw pointers or something but really what we want to say is that this future is tied to buff but this would not be okay in and of itself because now if you moved Fuu future that moves buff right buff is just bites directly embedded in the foo food future struct so if I wrote some code like this I would say let's say that I had a mane I said let F is foo so f is now of type foo future we're imagining that this is the code that the compiler generates right it it magically knows how to make this buff lifetime work out because the compiler can do that so when we call foo we get back a future because it's an async FN and we're gonna imagine that that future looks a little bit like this okay now imagine that I do something like F dot F dot pol okay and now I'm gonna do let Z is F and then I'm going to Z dot F dot pol you see a problem here so here if if we think about memory right the memory for the buffer is stored in F it's sort of particular memory location and we call pol this s is gonna try to read into this buffer okay so it reads some bytes into that buffer and that's fine because this buffer points into F right an F hasn't moved the problem here is at this point we're moving F buff is no longer the same place in memory but we haven't changed where the buffer in the i/o read future points it still points into F but we've moved away from F and so we now call pole ons on the i/o read that we moved in to Z then now it's gonna try to read some stuff into buff but this pointer points to buff which was in F which is no longer there like you can imagine that here effectively F is dropped so if if we if we yeah it's gonna be an invalid pointer at this point that is not allowed to read into and so at this point our program would like crash with like invalid pointer or use after free if you're sort of from SeaWorld you could imagine it's something like that so before we continue I'm gonna take some questions because there's a lot to unpack here I know yeah let's see how do they schedule the execution at this point I'm not imagining that there's any executor I just imagine that pole is called manually for now that's still sufficient to explain what the what the problem is shouldn't the buffer be a field in a structure it's a separate memory location yeah so that's why I moved it out to here really what you can imagine here is there's there are some fields that are like the the ongoing state of the async event future and there's some stuff that's like there's like an enum here which is like progress which is like an enum which is something like like step one is an aisle if we wanted to flesh this out a little then this is really more what it looks like and in fact let's let's write this out this might actually help part of the problem with explaining this is that you can't express self referential types in rust today and so even writing the code is tricky there's gonna be some return type here so it's gonna match on progress let's ignore the pin for now because we haven't really explained what it what it what purpose it serves here if we're in step zero then what we're gonna do is we're gonna do self dot progress is you can't really say this either you can't have anonymous enums but let's imagine that we do step one I'll read new of I guess this also has a read write all right so buff and s here are the the fields of foo anything that's not a future basically any local variables and such right so it's gonna create s and then notice here that it takes a mute to buff to the buffer right so step zero is just creating the i/o read in the first place and this is where the pointers into the buffer in foo future is taken and if you're in step one then what you're gonna do is mu F or like IO read future then I our read future dot pole and here that you can imagine it's doing something like it's gonna match this and if it's pole ready then it's gonna something something something step two and if it's pole let's do this in a way where the compiler won't yell at me and if it's pulled pending then it's gonna return coal pending this roughly make sense for why this would be the future generated by this async event right that this is how it keeps track of how far through Fuu it is right so here if there was another future we waited on down here then that would be the step two part and then there would be a step two here which is if you're now in step two then this is the future to pole it sounds like we initialize a second buffer to store a temporary value until the a sink is finished would that be pretty expensive to do yeah so the the reason we want to be able to write the code this way is because there's only one buffer rather than I'm going to allocate a vector or something we can actually have a buffer that's like embedded into the future that we do the reading and writing for so that we don't have to do any dynamic memory allocation at this point not gonna deal with things are off topic right now okay so does it make rough idea that this is the kind of future we would get or would get generated by to the compiler if you wrote something like this just to sort of check the round the same page there and does it also make sense that without any other considerations this would be a problem right because here in fact we could just do don't need dot F here right the when we first get foo future here where it's step zero right here we're now at step one and both points into F yeah or I guess I owe rebuffed points into F here we're still in step one and I read buff points into F which seems like a problem because F has been dropped right so this seems like a problem so the question becomes how do we fix this in fact let's just for for now this is the old future trade the sell futures used to look right that you just took mute self and then you can run into exactly this problem assuming you had a way to write self referential starts right this would not be okay well so how do we fix it so this is where pin comes into play so PIN is your establishing a contract you're promising that once you've placed something into a pen pin it will not move ever again this is not saying that you can't move foo future it's the moment you've given out a pin to a foo future that foo future will not move going forward so this is a it's a little bit important right if I am here in step zero then it's fine for me to do this right and then let G Z and let F is key and then pull off totally fine because at these points I haven't called Paul yet so I haven't given out a pin okay so let's talk about why this matters so with this type signature instead pole is now saying in order to pole you must promise that food future will not move again and the reason for this of course is that we have references into the food future once we've moved past step 0 right once we start pulling then it is no longer okay for foo future to move because of this problem right now once pol takes pin look at how this changes so now if I try to write the same code so let's copy that down here okay so this is all still fine I can't call pol on F right I have to do something like X is pin new at mute and now I can call F the pole right and that's fine our read is now going to point it above just like we did up here I can't do this though if I move well I can if I move if I move F then I'm really just moving the pin right I'm not moving F and so this is still fine it I can still pull here so this is fine so what if I really tried hard to move F if I did something like mem replace and I did so how do I get at the F it's behind a pin if I really wanted to like I wanted the program to crash how do I do this well I really want to move the F behind the pin so how do I do that well you can imagine that I did something like when a dereference the pin and try to get other thing inside it and I'm gonna replace it with a new foooor something so will this work well let's look at pin so pin dear so this is the critical part this is basically what makes pin work DRF mute is only implemented for pin if the pointer is D or F mute so that makes sense right you can't you can't get to the the thing the pin is getting at without the type in between being D or F mute but also it only is implemented if the target of the pointer is unpin all right what the hell does that mean well remember I mentioned at the very early beginning that unpin so this marker trait is generally implemented for all types it's an auto trait so you will get it for pretty much all your types if your if all the things inside your typer on pin then your type will be on pin and most of the things in the standard library are unpin some things that are not unpin is anything that contains self references so for example foo future would not be unpin the thing that gets generated for this async FN foo is not unpin specifically here the compiler will generate something like not unpin for foo future so notice that this is opting out of the unpin thing what does that mean well it means that I can't dear F mute pin that will work this pin to F I can't I can't do this this won't compile it'll tell me that the F I'm trying to get at isn't unpin which was required for the DRF mutant pin and so therefore I won't be able to get at this I won't be able to move the F behind the pin which was the whole purpose I have no way of invalidating this pointer and so let's look at how this changes the implementation of future that the this pole so when you're given a pin to self you can no longer get at the fields right if I do match on on self dot progress all I get is a immutable DRF which seems like a problem right I can't I can't change the step by Matt and this is because we've we are saying that were not on pin so it's not okay to mutate self and then you can imagine that this is because he if I could just violate the pin if I was given if I was given the ability to mutate self in here I could violate the pin contract immediately by saying my member play self-taught buff with or just member place all of self with some new food right this would violate the contract because we've given out a pin to this food future so we are not allowed to move it alright this is still a lot to chew on and we will go through many more examples because it takes a while for this to click so don't worry if this is if there's a lot here does that mean a big impact of compilation times not sure what you're referring to as you are stating that for each of weight point another step is created oh I see so it is true that the longer your asynchronous function is the larger or struct the compile compiler will generate for that code right because it has to keep track of all the progress points so to speak of that function that doesn't really lead to longer compiles it does mean that the that async function is going to generate a larger future but it's not clear that that's really a problem how can you mute itself if you only have a pin right that's the big next question so let's deal with that so in the case of poll here we know that the reason waifu future isn't unpin is because moving the buff wouldn't be okay we know that it's fine to switch steps right going from step zero to step one is totally fine step one just has a pointer to buff so buff can't move so we can read we can basically reason about our future implementation or about what other type we have and say that in this case I know that it's okay for me to modify self dot progress it's not okay to modify anything else but it is okay for me to modify self dot progress and so this is why for here and this so you'll notice that they're unsafe methods called get uncheck mute and map uncheck mute which are unsafe for this very reason right here so for example let's use this get unchecked mute to demonstrate something that's wildly unsafe right if I did this so why is this not okay why is this unsafe because if I did this there's nothing swap there's nothing there's nothing stopping me from doing this right nothing stopped me from doing this but this would not be okay right and then let's say I draw X here all of this is totally fine for me to do and here I've moved something that was behind the pin which I've promised not to do and the the reference to buffer inside f2 would no longer be valid so this is why this is unsafe because normally it's not okay to do in our particular case we know that it's okay for us to access access self-taught progress so what we can do is we can do something like progress is this progress right so what this unsafe is guarantee is not guaranteeing but with this what we have to promise in order to make this unsafe safe is that there are we're not breaking any references we're not breaking any self references by getting immutable acts getting mutable access to progress here right and we know we're not doing this because we know that all the things inside progress refer to buff but there's nothing else that refers into progress so that's why this is okay and now of course here we can match on progress we can change progress and that's all fine and this now would refer to here we're mutating the here we're giving out a mutable reference to the buffer which is also fine because we know and that the reason this is okay is because we know that the buffer won't move because for future is is unpin now of course we can actually write this code because you can't write self-referential code and rust today but you can imagine that this is basically what's going on under the hood with when the async when the compiler generates a type from this is that it finds a way to get a mutable reference to this on a sign at some lifetime all right does that roughly make sense why we need the unsafe here if not I will happily explain it again and just leave some time for chat to catch up in particular you see how this this solves the issue down here of now there's no way for you to for you to cause the IO read inside of the Fuu future tab an invalid pointer because it's hidden behind a pin can't the language team add reasoning in the compiler about internal references to avoid using unsafe yeah so internally you can think of this as the code that the compiler generates it knows when it can use this unsafe right it knows which things are referring to which and so this is why when you write async FN you don't have to write any unsafe but the compiler is gonna generate some code that probably uses unsafe but it doesn't have to generate any unsafe itself so you don't have to write any unsafe when using async FN alright so that brings us to things that are unpin so one thing you'll notice is that if you look carefully at pin you will see that there are it's look good again this one maybe so we talked about get uncheck mute so this is you are promising right so this is where you see under safety you must guaranteed that you will never move the data out of the mutable reference you receive when you call this function because that would in fact that would that would violate the invariance on the pin type all right so that's what we said we if we do this then we will not move T that is what we're promising by calling this function or by wrapping this it's unsafe so if I don't use async I would have to use unsafe with pin ah no so not quite this is what we're getting to now known as that there's also this function get mute goes from a self where which is a pin a pin with a mutable reference to T and it just goes to a mutable orifice to T why is this okay why is why is this unsafe but this is not unsafe and the critical thing here is where T is unpin so if the thing you're pointing to is unpin then it's totally safe to get mutable pointers to it this is the same thing that we saw below right there's an implementation of D refuge for pin if the pointer type so this would be like a mutable reference or something is the ref mute to T and the target is on pin then you can basically just ignore the pin right if the target is on pin then it's as if the pin wasn't there why is that okay what's because if you have a type like if you have a type like if you have au 64 right or au size or something then you know that a you size doesn't have any self references so you know that that type doesn't care about being moved so even if you've promised that you won't move it again it's fine to break that promise that is what unpin means if a type is unpin it means that it doesn't care about that promise so for example foo future is not on pin because it cares about being moved it needs to have this promise that it will not move once you call pol right however a type like let's take a Oh read our i/o read is on pin because it doesn't care about you moving the i/o read you moving this type is fine even after you've promised you're not gonna move it again it doesn't care about that promise so what does this mean in practice well let's go to also the there's nothing wrong about this line I'll explain that later so if I do something like alright here's a type that you're gonna see in the you'll see in like all of the few the futures crate like I think it's in futures util or something and that is the ready future this is called ready and it takes a tea in a knee and internally actually it doesn't even do that it just takes a tea and it has the value which is a tea and there's an imple tea future for ready tea this is a real future that you'll see in the futures library its output is T I'm gonna leave off the context just because it's annoying self output and what does this do actually I think this is like an option and this does pole ready self value say alright so this is the real implementation of ready and basically so there are a couple of things you might ask yourself here the first one is how is it okay for us to call take on this option take take some mutable reference to the option why is that okay well remember pin implements DRF mute if the target type is unpin and the target type here is self so the target type is ready so is ready unpin well actually in this case it's not unpin although it's unclear it matters so in this case if T is unpin then remember unpin is an auto trait right so if all the things you container unpin than you are on pin so in this case ready T is unpin in here because T is on pin so in that case pin implements the ref mutant which means that we get a mutable reference to value which means that we can take it so ready here is really just like a future that immediately yields the value you give it and it is on pin which is what lets us basically ignore the fact that there's a pin here at all and this will very often be the case if you ever have to deal with pin of a type that where the type is something you have written usually that type will be on pin right usually you don't have to worry about whether things are pain or whether things are on pin or not because usually they will be on pin and then you can just use the value as if you had a mutable reference and so now the question becomes wise is okay well so R is a ready where the value is I don't know 888 right and then we call our dog Pole and then I'm gonna I guess [Music] Pinu right because I need to pin it in order to call pol and in this case the unwrapped is gonna fail but let's ignore that let's talk about the the pinning aspect of it if I now do mem replace new char with another ready this isn't a problem right moving this ready even after I called pol isn't a problem because there are no self references here there are no references here that would be invalidated by moving the ready at all right this is just not a problem and the way we express that in the type system is that ready is on pin ready says I do not care about the promise you made for pin and so you can pin and unpin me however you want if you have me in a pin you can get me without a pin if you have me without a pin you can get me with a pin it doesn't the pin doesn't matter to me is what unpin says invite we can see the extreme version of this if we look at pin here where you say see that there's even a safe function on pin into inner which takes a pin to a P and gives you the P so just removes the pin right and here crucially the requirement is that the target is on pin because if the target is on pin it doesn't care about the pin as long as you have unpin types nothing matters that they're rather the pin doesn't matter and you see that this is explained in the docks to the requires that the data inside the pin is unpin so that we can ignore the pinning and variance when unwrapping it so the only time we really need to care about pin is when types aren't unpin because then we need to deal with this business of their self references that we have to be really careful not not to violate if I understand correctly the only requirement to be unpin as it's an auto trait is to not be self referential basically yes now are there bigger requirements the other time you will run into it is if you are generic over things because if your generic over things then they might not be unpin so this is why for ready for example we have to bound ourselves by t being unpin because otherwise we couldn't get a mutable reference there ready because Reddy wouldn't be unpin of course the example of this is imagine that T here was a self-referential struct then if you have a pin of mute of self then taking here would move that self referential struct even though it was pinned and so that wouldn't be okay can we construct a pin from an object or do we need some kind of pointer to call pin new right so that is that's an excellent question that's what we're getting it next so I told you there was something wrong with this line if we look like a pin we'll see what it is so pin new is totally safe but it requires that the target is unpin and why is that right it's because creating a pin around a type that doesn't care about pin is obviously safe right it is totally fine you can create pins left and right you can remove them and add them whatnot however if you have a type that's not unpin really i want the dogs to be like this if you have a type that is not unpinned so this is the block for things that are not on pin then putting it in a pin means that you are making a promise you are making the promise that you will not move it again that it cannot move again or rather that you will not move it again right and so therefore the constructor 4-pin when the type is not unpin is unsafe because you have to make this promise and if you look at the safety here there's sort of a pretty long argument for why this matters so this constructor is unsafe because we cannot guarantee that the data pointed to by pointer so the the P type is pinned meaning that the data will not move or its storage invalidated until it gets dropped if the constructed pin does not guarantee that the data P points to is pinned that is a violation of the API contract and may lead to undefined behavior and later safe code by using this main method you are making a promise about the DRF and D ref mute implementations if they exist specifically they cannot move so let's dig into why this matters right why is new unchecked a problem well imagine that I wrote a malicious implementation of D ref mute so my evil pointer right so this has a chi and I'm going to dear F for my able pointer target [Music] alright so I'm gonna have a main here I'm gonna do let X is so X is gonna be a foo future which we know is on pin right and then I'm actually going to do my evil pointer all right and then I'm gonna pin that so this of course is not okay right because you can only pin when the target of the pointer is unpin which for future is not so we need to use new unchecked now let's imagine that you didn't have the requirement let's imagine that it was fine for the target of new to not be on pin right but we still had the requirement of unpin to get ad mutable references and such right so I still couldn't move out of X I couldn't I still couldn't move this because I wouldn't be able to do say mem replace X with this right this I still couldn't do because this pin wouldn't implement the ref mute because future is not PRF mute right so why isn't this okay why can't pin you just work for any type why does it need to have this bound on unpin to be safe so the reason is imagine that this code all imagine that that pin you accept at any point err type it did not have any restrictions on on on pin then notice that if I do [Music] notice that the D ref mute from my evil pointer it is given a mutable reference to self there's no pin here even though down here X has been pinned right this D ref mute is still given a mutable reference to self so if I had this evil pointer we're inside of here I'm gonna replace let's say like T is default or something is not important I replace it with something here then now this line violates the pin guarantee right this wouldn't be okay and I think they get up this here yeah by using this method you're making a promise about the DRF and Dieruff mute implementations if they exist they must not move out of their self arguments right pin as mute and pin as ref will call them on the pinned pointer and if you look at as mute so as mute goes from so this is critically the reason this is necessary there's this function it's easier to read it here so what does this function do you well it lets you go from say a pin box T to a pin at mutti so why is that useful what is useful for cases like you want to call future pol so a future pol if you recall has a type that requires a mute self so if you have like a pin box T and you want to call Paul use as mute to get a pin mute e alright so this function is pretty straight forward but it is safe for all types right so it internally basically just calls self Dieruff mute pin new unchecked internally this is what it does right and why is that okay well because if I have a pin to the outer to the to the box T then the target is also pinned like there's that this should be totally safe but the exception to that is I have to go through D or F mute and so if the deira from you did this this would not be okay and so this is why you eating a pin in the first place you are making a promise if the type is not unpin you're making a promise that the DRF mute doesn't abuse the power it gets from getting mute self in fact we have a similar problem with drop right so so recall that the the guarantee we give when we put something at a pin is that if I've ever given anyone a pin around a given type I can never move it again I am not allowed to move it again at no point in the future if I've ever given out a pin I can't basically I can't undo a pin promise unless the type is on pin but when I drop a type drop is given a mutable reference to the type even if I've previously only had pins right when the pin is dropped then the pin is gonna call drop on the inner type and that drop is given a mutable reference to self so it can move out of a thing that had a pin around it not okay and so this is why you'll see in the documentation for pin up here yes I see they have this like big example about self referential structs so here they talk about this thing called the drop guarantee and so concretely for pin data you have to maintain the invariant that it's memory will not get invalidated or repurposed from the moment it gets pinned until one drop is called and specifically there's also requirements on the drop implementation right if you if your type uses pinning you have to be careful when implementing drop because the drop function takes Mew itself but this is called even if your type is previously pinned right so this is the issue we just raised and exactly like this it's as if the compiler called get uncheck mute for you which is unsafe so if you've ever given out pins to your type right if you basically think of it this as if your type ever implements something where it takes a pin self then that means that your drop implementation you must treat as if you're implicitly taking pin mute self and so they give this example of how you could do this safely which is just in drop pin yourself and only then do your write your drop implementation yeah that make sense cuz otherwise you could do all sorts of bad things in here which is not okay I don't think ready T actually needs unpin for tea why don't think that's true why do we need a wait Kent compiler automatically do a way to encode code like this right if you didn't write the await keyword then it would mean that I would have no way to do something like this is a bad example but sometimes you want to wait on multiple futures so this is often referred to as a select like I have two futures and I want to wait for either of them to complete I don't want to wait on them in sequence I want to wait on them in parallel and if the compiler automatically awaited for any future it's all then you wouldn't be able to do that [Music] what is the context behind this what is pin and why do you need it right so this is what we've sort of been discussing in the past but I think it's useful to summarize it at this point so why do we need pin we need pin because some types are not okay with being moved right at some point in their execution they're gonna take a reference to something inside of themselves and if the type was moved that reference would become invalid which would not be okay it would cause undefined behavior and so pin is a way to to fix this which is if you ever give someone a pin of a type then you are making the promise that it will not move again and they can rely on that promise right so for example foo future relies on the promise that it will not move again to create a self reference if it didn't have this promise it couldn't safely do this because if it were moved this pointer would become invalid how do you deal with panicking then I thought that it was calling drop on remaining memory so panicking will also drop types but the the requirement is still the same right the requirement is still that it's gonna call the drop method on your type and whenever the drop method is called on your type then you need to make sure that it doesn't invalidate any pin requirements you made in the past so let's maybe this will help let's go farther down here so let's say I have a start foo and I implement drop for food here do I need to care about pending no because at no point have I done anything that deals with pinning right what about now if I have this do I need to worry about pinning and drop no because foo is unpin so it's always safe to go from a pin of food to a mutable reference to foo and so drop still doesn't have to take any special considerations what if foo internally contains the foo future so now who's not on pin I have at least one method well actually how about now do I have to do make any special considerations and drop no because at no point has there been created a pin of foo or if there has I haven't taken advantage of it so that promise may or may not have been made to me but I haven't used that promise in a sense and so I know that I can't be in violating any invariants that I have made around foo future because I haven't used it it's only if you have a type that is not on pin and that type is ever placed in a pin that you use so basically if you ever take advantage of that promise right it's only if you rely on that promise that you now have to be really careful and drop and so the recommendation that they make in the docs is basically that you do this you write this inner drop function which takes a pin to self and then you give it pin of selves you basically pin yourself before you call drop on yourself you have a specialized version of drop so really the solution to this would be for drop to always take pin self but of course they can't do that because the draw that wouldn't be a backwards compatible change right the drop trait has already been stabilized for a long time you can do in Polti unpin for ready let's see for ready yeah so here I could imple t unpin for ready in pole implementing the unpin trait is totally fine actually you're right you're right uh so the reason this is okay gene let's uh I don't want to keep all the others but let's try to actually compile some code just to see why this matters this is actually an important point so let's let's deal with it I use standard future future tasks all contexts and see that we can actually compile this code this technically takes a new context all right so this code compiles right there's no unsafe in it and so you might wonder well why is this okay right why is this okay well remember that the pin promise only matters if you make it so here we are moving T right we have a pin to self and we're moving T why is that okay well it's okay because we've never given out a pin T right this T that we contain we've never given out a pin to so we've never made the promise that we'll never move it and so therefore it's fine for us to move it here yeah now this might make you feel a little bit worried why is this okay right why can we why is it not unsafe to implement the unpin trait and it's because this implementation in and of itself is not a problem it is only a problem if you ever write unsafe code elsewhere so specifically this implementation of unpin is totally fine right it's more if we now try to give out a mutable reference to T right if I here tried to do something like pin you mute self thought I guess self taught imagine that I wanted to write this right this won't work why won't that work well okay let's say that T is future right right I just want to show you that this doesn't compile for the right reasons so you'll see that here it's saying that I can't call pin you because unpin is not implemented for T right so the only way that I can pull this value is I need to have a pin to it and so I would have to call pin unchecked and this is not okay right so this unsafe is not okay because T isn't done pin so even though we have this in pull this implies only problematic if we have unsafe code elsewhere and so that's why that's why we don't need the unpin bound here it's because we never give out a pin to tea and so therefore moving tea is fine yeah alright so we've we've talked a lot about the sort of theoretical underpinnings of pin and unpin so let's look at some interesting cases from the standard library I won't actually we can talk about structural fields so the recommendation spends a lot of time on structural pinning and the basic idea here is imagine that you have a wrapper type so I want to write a struct my future and it's just gonna hold an F and I'm gonna impose future for my future where F implements future just write out the signature here so what are we right here so notice that my future is entirely unnecessary right all it's doing is forwarding the future implementation to the underlying future that's all it does but in this case here we have an example of structural pinning and so basically what structural pinning means is that if you move the outer type the field also moves if you move my view my future then F will also move so structural pinning means so in this case we have a pin to self and we want to call self dot F dot whole right this is what we want to write what will the compiler say it will say it will say there's no method named pole found for so this is a very unhelpful compiler error basically it's saying I can't find this method pole you're talking about and the reason for that is the pull method is to find on pin of a type not on the type itself okay so pin new and we always we always try pin you first just to see whether it work okay so now it found the pull method but it's saying that the trait bound F implements unpin is not satisfied right so remember pin new in order for it to be safe it has to require that the the type that's pointed to is unpin okay so why doesn't this work well we could of course say we only support forwarding to futures that are unpin okay so that compiles why does that compile well if F is unpin then my future is on pin so self is dear F mute through the pin right the pin dear F mute sort of enables when the target type is on pin and so now we can get to the F and we can call pin new because the target type is on pin because the target type is F and so therefore this code networks but this is a little sad this means that you couldn't do something like have a main let's do an async FN food I couldn't do this all right so here I'm trying to set F to be some async block where presumably I'm gonna write a bunch of code in here right and then I'm gonna await that future this won't work why is that well we want there to be an implementation of future for my future right because otherwise we can't await it we can only await things for futures and for that to be the case for this for future to be uploaded for my future then we've already said that we require that the F is unpin right and so we need the jet so gen future here is the the future type that the compiler generates for async blocks and so okay we need Gen future of T where T is like the actual contents of the async block we need that to implement on pin but that requires that this so see how it says static generator at source lab RS line 20 right so what's that line 20 at line 20 is this async block so it's saying that the bound Gen future implements unpin right so this async block implements on pin is not satisfied and so now we're saying that this async block isn't on pin because no async offense or blocks are and therefore my future doesn't implement future and this is a little sad right we would really like to not have this bound we would like my future to forward any future because why not so how can we write this code so that we remove this bind bound well we're gonna have to use new unchecked right because our alternatives are either use new and require unpin or use new unchecked so how do we know that it's safe to get access to a field well on pin there's map unchecked mute there's also get on check mute but let's use map and check mute first let's look at what it actually does so map uncheck mute goes from a self so that is a pin with a mutable reference to some type T and it takes a function that goes from mutable T to mutable you and there returns a pin of immutable reference to you right so this is going from you a pin of some type and you have a thing that converts from that type to some inner type and then it gives you a pin of the end result and so why is this so in our case what we want here is this is basically a reference to self so let's call it this and we're gonna give a mu to this F okay let's first see whether that compiles okay that compiles we don't need this anymore let's make this pub so it doesn't warn us about it okay so that compiles but is it safe right here we have an unsafe block what does that mean well map uncheck muses evolving this function is unsafe you must guarantee that the day you return will not move so long as the argument value does not move and also that you do not move out of the argument you receive to the interior function okay so there are two things that can go wrong here the primary one is that here self is a pinned mute self map uncheck mutes gets a mutable reference to self right so in here there's nothing stopping us from doing moving self which might not be okay right it might not be okay to move self because F might be unpin so this is not okay so this is one of the reasons it's unsafe the other is if if map and check from you returned a reference to something that might not move that there might move anyway but let's ignore that for a second I'll give it if it like dereference the raw pointer at the point of some other thread or something silly right like who knows what this does it could take a raw pointer to somewhere on the heap that it just like generated randomly and that would obviously be unsafe but ultimately we need to figure out why this line is okay or whether it's okay and so here what we're saying is that first of all our closure here does not move things that aren't unpin right it doesn't move anything in fact and second of all the guarantee we need to make is that as long as this doesn't move this won't move and that is the case right we don't move F we never move F and as long as my future doesn't move right which is this then f will not move yeah and so this is because here we can rely on this thing called structural or they call it structural borrowing structural structural pinning basically they're asking whether pinning is structural for a field and in this case F F is a structural field of my future because if my future moves and F moves if my future does not move then F does not move and so it's okay for us to make this promise because we have been promised that we won't move therefore we know that F won't move and so therefore this unsafe is okay yeah it's a little bit complex there are a bunch of different ways you could write this code this is the one that uses the least unsafe right map and if you can use map and check music you should because there's less opportunity for accidentally abusing it one alternative here right would be unsafe get unchecked mute right and then we do something like this dot F dot Pole but now we'd have to also do new unchecked to this with unsafe and also now someone could imagine this was in the middle of like a long complicated function nothing is stopping somewhere doing like oh I'm just gonna set this to none or whatever and the compiler wouldn't warn them about this line because this line promised that the code beneath didn't do anything wrong and so this is why we're possible you want to not generate these right or you can do something like F is and then this right that way that this wouldn't accidentally be used further down all right so if we go back to this now one thing we have to be really careful about is it's not okay for us to impel unpin for my future as here now right this is not okay why is that not okay because now there is nothing stopping me from writing this function why is it annoying to write this code ah you think F and I just need something to stick in here right I can now write this code there's no unsafe in this code right I create an F of type my future I pin it so I'm guaranteeing that it should not move again and I'm saying that F should not move again right so Baz is an async FN so it's not on pin so I'm pinning it and that's totally safe and then I'm swapping out I'm just straight-up swapping out the my future in fact I could do dot F swap it out with a different buzz and all of this is safe you know clearly I'm breaking the pin guarantee right I made a promise with this pin pin knew that I would never move F again and yet here I moved F again in safe code and there's a subtle interaction here the reason this is a problem is because this implementation of unpin is saying that you can move my future however you want but this unsafe is relying on the fact that if you have a pinned my future then it will not move this unpin is opting out of that guarantee for my future it is saying that you can free we moved my future no matter what you like you can just move it freely and so this pin means nothing therefore this unsafe which relies on this promise also means nothing and so now we've broken this so this is why this even though this implementation is not unsafe this adding this safe implementation breaks the requirements of this unsafe did that make sense someone asked why is it safe to implement pin and it's because this on its own is never unsafe the only thing it can do is it can make things that are already unsafe be unsound might be incorrect so the when you write this unsafe what you're really saying is I have thoroughly checked the code and I know that this is safe I know that this pin guarantee means that like if this will never move then this will never move adding this changes the guarantees of pin on my future making this unsafe no longer correct and but this in pull itself isn't unsafe because if we didn't have any unsafe code in here so for example if this was like plus unpin and let F so this is what we had before right of pin you need self F poll right this is totally fine now this unpin isn't bad so it's not really the unpin that's the problem so this is why it's not unsafe it's only that by adding it you are violating an invariant that other unsafe code in your code relied on why doesn't the compiler always do it the compiler can't add this input for you always because it doesn't know that you have this unsafe then relies on this not being unpin so always adding this would be wrong because if you have unsafe code elsewhere it might rely on you not having the simple so therefore the compiler never adds it unless all of the unless your type is unpin if it's not generic or if all of the fields of it are types that are on pin then this unpin implementation will exist from the compiler alright so that should give us a decent view into unpin and why it's necessary and also how to use this with unsaved right so now if we remove this now you'll see that this code no longer compiles right I'm not allowed to create this pin you right the compiler won't let me create this pin new because my future is not on pin one thing that took me a while to wrap my head around the generic way to make a value unpin is to call box pin on it yes so that was what I was about to get to next so one thing you will see and this might strike you as surprising which is why we're gonna talk about it is if you look at box so box allocate something on the heap you will see down here this hidden in the darkness so take take a little bit of time to read this and see if it makes sense to you crucially why is this okay notice that here we're saying that for any type no matter whether it's pin whether its unpin or not um pin if you put it in a box it is now one pin or rather the box of the type is on pin [Music] all right so let's dig into it why is this okay well this is okay because if you have a type that is on the heap right so for example if I have here box of s then if you move my future then F is still not moved right so now you can move my future all you want I can still always keep pinning it if you pin my future and then unpin it and then pin it again not a problem all of that is fine because F still hasn't moved so it's not because it's not separate a self-referential right well what what unpin for box is saying is regardless of whether T is unpin or not on pin it is on pin when it's inside of a box and this is because now the structural guarantee is very different so can we now do this alright so by making this box given the implementation we just saw this means that my future is now unpin regardless of what F is because it was a blanket implementation so can we now do this right now that self self is now our pin of self is now gonna implement D ref mute so we can get up this does that not work we get rid of this broken thing down here this is now compiled how I need to yeah that's fine let's wait a second with that too so I can now correctly get to F and in fact I can even create a pin to it but really what I want here is this right I want a mutable reference to the target of the box to the F if I did this I would just get a mutable reference to the box which doesn't really help me I want a mutable reference to the target of the box and then I would pin that because I want a ref mute F right that's ultimately what I want to call pull on so this is work no because unpin is not implemented for F so notice that boxing in F just means that my future is now on pin it doesn't really solve this issue we still have this this requirement that we can only guaranteed that F won't move if we won't if the box how to explain this okay so why doesn't this work well I can write a function that does this pin let's X is box new a sink right so this is not on pin mute X you start with I need I need my async pass back right so a box of F makes that the box itself is unpin but we still that that's not sufficient for pin you to be safe on it because otherwise I could write code like this right where I still get to move the thing that's behind the box so boxing something in and of itself doesn't really help with pin it all this is saying is that if you have a type that contains a box T then that type is now fine to move even if you made guarantees about pin if you've made guarantees about pin of T on the other hand then box won't help you write specifically in here we want a what this code really wants to produce is it wants to produce a pin mute F right but in order to give out this we need to give given T's about whether F will move and so it doesn't really matter whether my future moves right all this box is saying is that if you move my future that does not necessarily move F but that's not saying that F won't move which is what we require here we require being able to say that F won't move and so in fact if you want to use box this way then what you can do this store here is a pin of box of F now what we're saying is that if my future moves then F won't move and a pin of box of F that's fine the or is it I close it what's annoying [Music] yeah so app in a box of f you won't be able to get a mutable reference to F through this so you won't be able to move F so if we know that my future won't move then we know that people want to just like switch out the F that's fine but if we now want to get a mutable reference to the F so let's first let's see what this compiles box does not implement structural pinning yes that is one way to say it that's right yes we still can't call pin you right and the reason for this is if we if we could just get a mutable reference to F this way then anyone else can as well and they might just like replace it entirely and so we're gonna have to use new unsafe new unchecked all right what am I missing here give me a second there we go so we still need this unsafe right even though now this is no longer than that no longer structurally pinned right if you move my future then box will still now move let me just tidy this up a little all right so we still need this unsafe map and check mute right and here what we're guaranteeing we're still saying that if my future doesn't move then F won't move so this is still the same guarantee that we made before however now my future is unpin right so you can freely pin and unpin my future and that's fine but this guarantee can now be violated pretty easily imagine that I wrote some code like this not knowing about the stuff above right unsuspectingly I didn't really look carefully through the file and I wrote this implementation well this is now a huge problem because now I can do this do what I do my future F this and then I pin that just give me a second to tidy up this so we can what are we missing Oh get F there's a get mute on pin as well and this alright so now I've written completely safe code that moves the F even though we pinned my future this is not okay right imagine here that I did like excellent pole and then I did X dot pole here again yeah right this is totally safe code it calls polska gives out pin but now this unsafe is being violated why is that well this unsafe is saying if you've pin self then it's fine to give out a pin to the F but that's not true right because my future is on pin and so now you can write this function totally safely and this writing this implementation is gonna violate this unsafe because this is giving out a reference to the to D to the F seems like if there was a kind of reference that is like mute but you can't use member place on the like on it then things would be safer so that is what pin is right if you put something in a pin you cannot get at the underlying contents mutiply unless you promise you won't move them it just that it turns out that there are a lot of ways to move things and just saying like you can't do mem replace isn't really sufficient the the standard pin docks has a bunch more examples of how you might accidentally move something but just saying no memory place is not gonna solve it it really it really is you cannot give out mutable reference to things unless they're wrapped in a pin so pin is exactly that promise and that has to rely on things like unsafe remember this doesn't have to be remember plates right this is the same just assignment remember place is just a convenient way to write it okay so this is a problem so how do we make it so that people can't accidentally write this code well we can write a pin box F here so there's a box pin which returns a pin of box and now you see let's get rid of these because they're not important and see here right so now this function you can't write safely anymore because what we're saying here is we're gonna put F in a box and then we're gonna pin it we're gonna say that this is going to be used as pinned and so you can't just like arbitrarily give out mutable references to it and so now this get F if you try to write it you would have to write unsafe code which wouldn't be safe right so this is now no longer possible and now it's gonna complain about this get F right so this this you can't call anymore so that's great and in fact this is comment out this little thing we don't read it needed right now what you know terminated its terminated right there oh it's awkward and now look at what we can do so I forget I think it's this so this compiles and there is now no one safe code here whatsoever so how did we do this well the Box guarantees that moving my future will not move the F the pin guarantees that there is no way to move the F anymore unless it's on pin right because if the pin pin wouldn't implement D or F mute to the F and you can't so that means you can no longer get a mutable reference to the F unless it's on pin which means that pin box is this neat construct that you can use to take things that are not unpin and make them unpin and the reason is the Box makes sure that it will never move right the the box make sure that it has a stable location so to speak and the pin makes sure that you will never be able to move F unless it's on pin so in general pin box is the way to go from things that are not on pin to things that are on pin of course the downside is now there's a heap allocation here which may or may not bother you but this is such a useful construct that if you look at box you will see that box has a function called pin you can give it a T and notice this is any T it's not bound by unpin at all and it gives you a pin of box of T if T does not implement on pin if it importance on pin you could move it just fine right because the pin is gonna implement dear fu but if T does not implement on pin then X would be pinned in Mariya able to be moved right the box is gonna put it on the heap somewhere and the pin is gonna make sure you can never get a mutable reference to it or you can never get one safely of course unsafely you can do whatever you want yeah okay so let's look at the implementation of this because our intuition here is that it's actually a little bit subtle why a pin box is safe to construct for any T so let's look at what internally what it does so there's a function into pin that takes a box T in fact we can look at this here first into pin so this takes any box T and gives you a pin box T and remember anything that's wrapping a pin comes with the promise that the target type will never move again right so how can this function be safe right this is what we just discussed it is safe because the box makes sure that it has a stable address and the pin makes sure that you can't get a mutable reference to it anymore and you will see that indeed internally it is it uses unsafe it says new unchecked it is not possible to move or replace the insides of a pin box T when T is not on pin so it's safe to pin it directly without any additional requirements and this is basically the short version of the long roundabout way we've we've got at this now yeah okay so writing all of this unsafe code should make us all feel a little bit uncomfortable right so let's see if there isn't a better way let's sort of save this by putting it in a comment because that's how we save so let's go back to what we had where we did let F is unsaved self map unchecked mute this F X great and we have this async food that's gonna use my future right so this is remember this is how we wrote it without having to use box and it required this like one line of unsafe but ideally when you write code like this you don't really want to have unsafe if you can avoid it and so this is really neat library called pin project which basically takes away much of the pain of dealing with this these invariants on your own so let's go to our cargo Tamil and let's add pin project equal 0 4 0 alpha bunch of stuff great alright so what does this let us do well pin project let's look at how you use it first and then we'll look at how it achieves what it does so you add an annotation on a struct and you say which fields you want to pin and then in here you do self dot projects dot F dot poll seats so notice that I added some annotations and then I replaced the unsafe code with this project call and it compiles there's no one safe code right no one safe code and we get the same thing that we got before yeah so that seems really neat it would be great if we could just always do this and in fact pin project you can i've uses in a lot of situations already but it's useful to dig into what does it really do under the hood and also you'll see something weird here like let's say that this have a t and this also holds a t notice that we've not added the pin annotation to the t maybe places over here we'll just build up a little bit of an example here my future future o FN 4 so this is gonna create a where's my example of this down here X dot get T T is gonna be I don't know fact T is also gonna be something that's on pin sorry give me a second I will explain in a second I just need to write some code to talk about firsts in fact I don't even need this code 34 that's annoying fine ah because there that's not ideal I should file that as a bug I have fine not worth it fine um in any case if we have a field that is not marked as pin in fact fine fine fine fine fine then let's do this just to demonstrate what happens I can do this dot T is bar and then I can do this dot after poll I meant buzz oh man fine I don't have a type to use here option you can mostly ignore all the various back-and-forth I'll explain at the end there we go okay now it compels great so notice here that I have a tee that is also totally unbound right it may be unpin it might not be on pin and inside of pole I'm allowed to change T even though I have a pin mute self and this F so remember this in order to call this I need one of these in order to call this I need one of these yeah and that works right this compiled just fine if I try to do this if we made this also an option which is a little bit annoying but some to this this requires a mute F and this will not compile right it's saying you have a pinned reference to F so you can't do that so pin projectors actually generated a thing where I can safely touch I can sail safely mutate fields that are never pinned right because remember the pin guarantee only matters the pin promise only matters if you ever give out a pin to that type here we never give out a pin to that type to T right and so it's totally fine for us to modify it directly however we do give out a pin to F and some mutating F is not okay but getting a pin to F is totally safe right because we've never given out a mutable reference to it and so therefore this code is fine and so pin pin project really gives us exactly what we want if we go back to the the ready example from earlier right remember how we had this whole business right if we remove this on pin and now this won't work right because T isn't ready isn't necessarily unpin so pin won't be dere F so here one thing we could do we can pin project this we can we never need to pin value right we never need to pin the T and so now pin project adds this project method here and now this will just work so pin project is actually also going to automatically add an implementation of unpin for ready that is going to be guaranteed safe so if we now do cargo check I guess this still doesn't work right because it's not okay but this compiles and notice there was now no need for any and say for ready either and we didn't need to write the manual implementation of unpin even though ready will now in fact be unpin so pimp projectors really handy for this but how does it achieve its magic right that's the real question here and so in order to explain that let me first undo this like option F business because it's a little annoying and then let's do cargo expand so cargo expand is this really handy tool for cargo cargo install a cargo expand that lets you show the expansion of all macros so let's scroll up here a little bit and see what we got generated okay so we have my future so we recognize oh man so we recognize my future and you'll see that it generates this unpin scope so this is just so that it doesn't pollute the type space with lots of names and there's some really like subtle stuff going on here but basically what it's doing is it's generating a type called my future projection and for any fields that are pin it's gonna hold a pin to a mutable reference to the field and for anything that's not marked as pin is gonna hold a normal mutable reference to the field right so notice here F was marked as as pound pin and so therefore it the mutable reference to it is a pin and T was not marked to spin therefore we get a normal mutable reference to it and then this project method is unsafe you notice that it doesn't get unchecked mute to self which gives it the underlying future and references to all the fields and then it pins the fields that are marked as pin and does not pin the fields that are not marked as pin and based on what we know from before why is this okay why is this unsafe block okay well as long as we never move self which this does not do and we never give out mutable references to fields in self that are pinned right so F here is something that we pin there is no way through what pin project gives you to get a mutable reference to the F safely and so therefore we're not and we're not invalidating the guarantees required by new unchecked and so therefore this code is safe giving out a mutable reference T totally fine because we never give a pin to it or a pin project never gives us a pin to it if we wanted to create a pin to it it would either need to be on pin or we would have to write unsafe code and there's also so this little bit nasty thing here this is basically the Impala unpin from my future right so it generates an an implementation of the UMP in trait and it does some magic so that in our case we will be unpin as long as as long as f is unpinned right remember that the reason why it was okay for us to have a blanket implementation remember back here we had an Impala unsafe T unsafe for ready T right remember this was totally safe this should say on pin and the reason for that is because we never give out a pin to to the T and so it doesn't matter whether T is on pin for weather-ready is on pin and that is what it generates here so it generates this this intermediate struct that any fields that we any generic fields that we never give out a pin to we don't care whether they're unpin for the purposes of whether the wrapping struct is on pin whereas we do care that my future is only pin if is only unpin if FN is on pin it doesn't matter what he is and that is what this generation does so pin project will give us safe pinned references to any fields we need to be pinned self safe mutable references to any fields that are not pinned and a correct implementation of unpin for our type even when it has generics it also does the same for for like enums where you have pinned fields and variants and stuff the other thing that's sort of neat that actually pin gives you is if I have and this might strike you as weird remember this F here so this dot F is a pin mute F right I have a set method that lets me do this oh man I'm gonna have to make this an option again as pint mute that's funny [Music] alright so this might strike you as funky right so this F is a pin of F and I'm allowed to change it so remember how we said you should never be allowed to change to move something that is behind a pin unless it's unpin well F here is not on pin there's no unsafe code here and yet I'm allowed to change it to none and the reason is because you are never allowed to move it until you drop it and setting is totally fine I don't give out immutable reference to the F it's not gonna get moved it is only going to get dropped because I'm gonna replace it with something else and I don't get back to the old value and so it's never moved it's just dropped and the new thing is set in this place so this actually this actually works totally safely trying to catch up in the middle of the video is hard yeah I believe it this is a deep deep like nest of things are we talking about pin as in circuit board no we are not talking about pin as a circuit board we're talking about pin as in the type used to give guarantees about self referential structs in rust so a very different type of pin yeah so pin projectors really neat for these types of things it handles basically every case after drone thrown at it and the author is also very receptive to bug reports and such although I've found relatively few and so this is a very nice way to deal with pins it the to deal with pins without having to write unsafe code and this has nothing to do with futures really right like you can use pin project for anything that requires pin it's just that currently you don't really require pin for many things that are not futures because most things are on pin unless they happen to contain a sink offence which are futures now that said these are things that might change so for example if Russ gets generators so generators would be something like I don't know what the syntax will be but like numbers so this is something some other languages have where you could imagine something like this so the idea here is that this would construct an iterator right so every time you call it it continues from the previous yield point and so it needs to keep any state so for example it was the exact same thing right I have a buff like this and I'm gonna like I don't know I don't know man like buff I'm gonna yield like buff this or something something like that so here this generator which is an iterator is now not unpin because it has self references and this has nothing to do with futures but this will still require pinning because it's any self referential struct and here you can see that this is the same as what we've been talking about before if you remember at the very beginning we also talked about this parser type see if I still have it here somewhere no apparently not we talked about this parser type that once where you have a parsed representation of a byte stream so such a type would also not be unpin so if you think of a type like Papapa Papapa still talks RS oh what's that's weird owning ref so the owning ref crate is basically gives you self referential structs if they have a good example yes the idea here is that you can return something that has a reference into itself the owning ref crate lets you do this and currently they do this with a bunch of hacks through the type system and things like and rest rates and such it is pretty reasonable to assume that this could be represented through pinning it currently isn't but here you sort of want to imagine that this type now requires that it's pinned otherwise the self references would no longer be valid mozilla is a nice explanation for iterates and generators with javascript yeah I mean generators are just yet another example of where you would need pinning because they're they so easily get self referential so currently this is also worth knowing any async block in any async function is Mark does not unpin even if it isn't unpin sorry even if it is unpin so for example our little boss down here this has no self referential fields there's no internal pointers here right it's empty this the return type from this will still be not unpin even though it doesn't need to I don't know others of them that all change it'll it's probably pretty difficult to figure out whether a given generated type should or should not be on pin and getting it wrong would be disastrous and so it my guess is what we'll see is that these will all just remain not unsafe when they're generated okay so the the next and probably last thing I want to do is look at some changes from the old future stuff which didn't require pinning to the new future stuff which does require pinning in the context of a particular library that recently made this move so we'll look at basically some of the PRS so that's what I want to do next but before we do that are there any questions about the stuff we've discussed so far so now we've gone through basically all of the all of the underlying theory and hopefully it's now clear what pinning is needed for and what unpin is needed for I'm not gonna deal with things that are off topic right now just because this is a sufficiently complex topic that I don't want to be here too far from it we can do some Q&A at the end maybe so probably when we demonstrated examples of code that is not okay by using memory place we needed to call poll on the returned value not on the new value yes you're right the poll would be on the actually they've both been moved so they're both invalidating but yeah I mean the problem doesn't arise until you use the pin again probably but nonetheless like you shouldn't do it in the first place if is okay yeah the reasoning why set on pin is okay is a little bit subtle but the basic idea is that set doesn't allow you to move anything right nothing moves when you call set or rather the thing that was in the pin gets dropped and the thing that is not in the pin and B it gets moved into the pin is moved into a pin so it's now pinned so regardless of whether it was unpin or not on pin it is now pinned and it wasn't before because we were given an owned version of it right so that that that's sort of the informal argument for why why said is okay all right so let's look at some actual changes so use this and you will recognize a lot of the patterns that we've built up to so far in those changes so the library we're gonna look at is a library called tower so tower is basically a it's an API for services like network services but all sorts of services in like like the idea is basically anything that goes from any function that goes from a request to a response is a service and then it defines various both traits and helpers structs to manage these like rate-limiting and logging and all sorts of things it's not terribly important what it does but it is a library that had a bunch of code that was written for future is 0.1 and that has recently been converted to using standard future the question was if there's a difference between set and replaced where the return value is immediately dropped mmm I think those are equivalent but I don't want to say for sure okay we could look at this so sorry for the slight here set yeah so set is this is basically a memory place it's just setting the target to be equal to value which implicitly drops the old value which i think is equivalent to memory place or rather it's equivalent to memory place where you immediately drop the return value I think so a warning for those of you in rooms with like where it's all dark now my github is light so this will become very bright for you all right so as you can see from the close pull requests like a bunch of these are update like particular module to standard future so let's take a look at some of these let's start with some of the early ones start with like tower load so first of all this bun this bumps a bunch of the dependencies to use the futures core preview and like Tokyo and tower previews notice that it also uses pin project because it's handy and so here we have a bunch of types and some of them we never give out pins to right so we add pin project to them just so that we're able to mutate their inner fields even in a pin context right so we might in fact if we scroll further down you'll see here that we want to we want to implement this no it's a bad example well actually I can explain it without that there are cases where you have a pin to one of these and you want to mutate either in or load and mutating either of them is fine because we never give out a pin to a tee or a pin to an M and so therefore we sort of basically we want to implement unpin for this type but if you recall from back when we were when we wrote the the manual in pull of unpin for ready it's so easy to like someone will write an unsafe block somewhere else in the file and that causes that everything to be broken because of that impolite so this is why I prefer even in those cases to add pin project to it because that will guarantee that no one will break it later some of it is just adding these contexts things that are now required you'll notice that service like this might be worth talking about separately though it's a while ago so if we look at Doc's towers service so this is an interesting trade-off the sir the tower service trait is sort of the core of tower and it has two methods it has pole ready and it is call now pole ready is gonna be called in an asynchronous context it's it's gonna be called from a future basically but known as that it takes a mutable reference to self it does not take a pin to self and the reason for this is because call which is used when you want to issue a new request we really want call to take a mutable reference itself because otherwise you would have to like pin the service every time you wanted to call something on it which effectively just requires that the services unpin and so taking pin here and not taking pin here is pretty weird right because you're gonna be basically alternating between calling these two different methods right you're generally gonna be calling Paul ready and then you're gonna be calling call and so if we mark this as taking pin self and mark this is taking mute self then now that is effectively the same as saying that services all have to be unpin because otherwise once you have a pin self you wouldn't be able to get a mute self unless self so the service was unpin similarly if we made and so this is equivalent to saying that service must be on pin but just without requiring the user to write like pin you in the first place we could make both of them take pin mute self right that way Paul Reddy would just be pin as normal and call would now require you to have a pin but that makes it a little bit awkward too because it means that first of all anyone who wants to use a service needs to pin it before using it and second of all and you want to implements the service trait would have to deal with the fact that call can't mutate self like inside of self so it just makes the API a little bit more awkward like implementers would have to use pin project project and callers would have to pin things but this is actually an ongoing debate this is part of the reason why tower the the tower on standard on standard future is still in alpha I think there's an open issue tracking this basically we're here so I opened this this is actually partially in preparation for the stream trying to figure out why is it shaped this way and what is what are the trade-offs and the observations here so option a is we keep things the way they are option B is we add an unpin bound to service right because requiring these to both be Miu itself even though this is in an async context basically implicitly requires that the service is on pin so we could just make that requirement explicit we can make pol ready take pin mute self and then call take mute self just to make it even more explicit but then we still that's effectively still the same as making the service on pin or require the services are on pin we could take everything make everything a pin but now the API is a less ergonomic so there are trade-offs like this that affect real code bases and if we look back at our load you'll see that at least for the time being they both take me itself let's see if there's anything else you'll see here he's another like this is a similar to my future right where we have a thing that internally contains the future and then it also has some other things so in this case we use pin project on it we pin the future because we know that we're gonna have to we're gonna need a pin of F in order to call poll on on the future but we won't to have mutable references to both handle an instrument because the code if you see here the implementation of future for it it is going to modify handle and use a mutable reference to instruments we want to have mutable reference to both of them but we also want a pin to the future which pin project all gives us and you see this pattern just repeat over and over and over again right pin project only some of the fields are pin and those are the ones that we're gonna need pins to and for the tests what this means is that for the tests we can now use async blocks right because they generate things that are not on pin but because all the methods now take pin self it's all fine you can see also in this so here we had a trait that so this trait discover so the idea with discover is that it's a trait that lets you discover new services and it has a pull method sort of like future pol accept and sort of like stream right so stream is another example of a trait where so stream this trait has pull next and it also takes a pin mute self because we want to be able to support implementations of stream that are not on pin because they might internally contain like an async block or an async function right you can imagine a stream for a stream rapper that takes a function that produces futures and just keeps calling it and keeps producing the thing that it gives we're gonna have to pin the resulting futures and so stream pull next take spin in general basically any poll method any method use seed that starts with poll is gonna take pin of Moo itself and so here we have this the straight that used to take commute self in the old future zero point one world and in the new world it now takes a pin self in this context and that also caused all implementers of that trait so for example here we have service list they now all need to take pin self and you need to use pin project in this case to access the thing that we don't even pin so the field isn't marked as pin so that gives us mutable references to it because we never pin it so we never give that promise and here is an implementation of discover for stream so here we're gonna have to pin the inner stream so therefore we use pin protect with pin and that lets us project the inner which gives us a pin of the inner stream which we can then pull so in general for almost all feel free to look through these PRS are pretty straightforward and see if there's anything you spot that's weird in general I think what really helps is the pin projector it is great I do think it's important to understand what it's doing under the hood and why what it's doing is safe which hopefully the explanation today has done remember how back when we started talking about pin how yeah so we talked about this drop guarantee in the drop implementation so you might wonder well what happens if you spin project on a type that implements drop right remember how drop you you have this additional requirement that you don't move even though you're given mute self and drop and the way pin predict handles is is if you try to use pin project on a type that implements drop then it won't compile so here let's get rid of this generator which doesn't work there's still compile is great so imagine let me now do imple ft drop for my future ft right so here imagine that I did something like a member place mute F fast mute actually just F dot take what am i doing and then we did like yeah I mean who knows we did there right but here I'm gonna move F inside of here and this is obviously not OK right because we've given out a pin to F before and so we're not allowed to move it now well if I tried to do this this code will no longer compile this should say self dot F so see here that the pin project attribute is saying conflicting implementations trait my future must not import drop right we're not allowed to do this because it knows that if you tried to do this you might violate the guarantees and soap in project will not even let you do this and so there's this pin drop thing where if you do this if you really want to implement drop for your type then you do this business is like a whole invocation you do and my future ft and you'll see here that basically what they've done is they've added their own trait pin drop and pin drop has a method drop that takes self by pin to mute of self just like the documentation for pin recommended right and now of course this won't let us mute ibly get access to F because the only way we can get F is through a pin so even if I did like self dot project here all I would get would be I would get a pin to F I would get a I would get a pin of F which doesn't let me take right I get a pin option F which I can't take because F isn't unpin so project is saving me here and if we look at the the expanded code up for let me just get rid of ready cuz it adds a bunch of noise so we have it up here so here's my future and you see there's an Impala drop for my future which pins self and then calls the the pinned drop drop method on self so this is if you recall the exact same pattern that they recommended that you implement drop you immediately pin self and then call a drop that only gets pin self so pin project is here implementing that pattern for you and it's doing such a way in such a way that you won't accidentally violate it there's also they also have this project which is for accessing fields with invariants and there's also unsafe pin when you want to generate an unplanned implementation that is sort of unsafe according to the rules the pin project knows about but that you for some reason know that it's safe so I highly recommend you take a look at pin project if you have to do anything that does with pin there is also I recommend you read through some of the the tower changes because that they're almost just mechanical changes to move to pin very little bit has to do with like the implementation details of power it's all about just moving things to pin and so that might help just to get a lot of that repetition and I also recommend that if you if you find old crates that were written for the old futures like future 0.1 try to port one to viewers future 0.3 or to the standard future some of it will be things that like traits have moved and stuff but much of it is just going to be dealing with pinning and now at least in theory you're equipped to deal with that and understand what the underlying issues are that you're you're working with there is for example there's one more thing that's handy to know about for the purposes of pinning which is the futures utils crate which has a bunch of like the basically all the things that used to be in futures like all the methods on future for example are moved into this crate and one of the things that provides is this macro called pin mute so pin mute is actually very straightforward but it's just a handy thing to know about which is actually maybe I can demonstrate down here with [Music] ya basically if you have something like this and you want to pin it you can do pin mute X and now you have X which is a future is sorry is a pin of immutable reference to my future it is basically equivalent to doing this but it's all just a tiny bit nicer it avoids you having to write this code yourself but that's also one to know about it's particularly useful in tests but at least now you don't know that it exists and with that I think that's all I wanted to cover with pinning and unpin are there any questions let's do questions about pinning and unpinning first or things you would like me to explain again this is like fairly technical content there's a lot of just subtle interactions that can be hard to follow and even though like we've been doing this now for like almost three hours it is hard to put words to exactly how these interact interactions work and so explaining some of them again might help I'll do off-topic questions afterwards I do think that one thing you'll find with with pin and unpin is it helps a lot to just to just use them a lot I'd like try to port something that needs pin try to read changes that are related to pin try to read like the documentation of standard pin and see if you follow it and if you don't then like just keep at it because after a while that it clicks a little what what is needed and why but it just takes it takes exposure really well this stream be available when it's not live yes I put the recordings of all the streams on YouTube afterwards I also if you want sort of a mental challenge so to speak I would say think really carefully about why we remember when we talked about box pin like why is it okay for box T to implement unpin unconditionally and why is pin box of T always fine to use like why does that give you pins of T no matter whether T is pin or on pin all right I think if there are no more questions about pin let's like pin that for later and do like general QA I'll put so in the when I post the video I'll put time codes too when we covered the different parts of pin and unpin and also a pointer to like basically now and when we do general QA so if you have general Q&A about rusts the ecosystem async me anything now it's the time I'd like to start writing some CLI tools in rust do you have specific crate recommendations for that well it depends what you're trying to write I think struct opt is great clap is also really good but I don't have any recommendations sort of beyond that it really depends on what you want your tools to do see those one further up here too this is neo of em it's not Emacs and it's not I mean it is vim but it's not them it's near them and I have someone asked me about my desktop environment earlier I have a separate video where I go through like what my desktop environment is like and how its configured and all my can fix are online literature about database internal implementations I mean the the sequel light documentation is actually really good explaining how the internals of a database work or sort of a traditional database for the internals of my research database I would read the paper it reminds you of Scala I mean so keep in mind that the code that we wrote today you very rarely have to write yourself so my guess is you won't ever have to write pin unless you're implementing your own futures and most of the time these days you will just write async functions and async blocks would deal with all the pin stuff for you so you won't have to think about this that would be my guess it should only matter if you have to implement your own futures how do you get error messages in vim is that ale yes that is ale ale is super handy for this is al plus RLS although I actually mostly find that I don't I don't want warnings while typing I would rather switch to a separate terminal and get the errors and warnings there oh I'm glad you think so I mean the topic we cover today is really challenging like arguably more so than the the stream on on how futures work because this is like very technical and nitty-gritty and subtle and like theoretical detail so if you followed along I'm very happy it's hard to to to talk through it because so much of it is just I don't say ingrained but I've just I've done so many of these changes to pin now that it's it's almost like in the fingers and it's hard to put words to why things work the way they do so hopefully this made sense in the end rust completions I'm using RLS for that and language client neova Mia do you recommend any specific repo to get exposed to a basic usage of pin besides Tower yeah one thing you can look at is there are a bunch of crates that give sort of Handy combinators like smaller things so for example you could look at it's an example of this like a while ago i ported there's a crate called stream cancel that I wrote a while back I have a I have a branch now that implements standard future for it but you could try just doing that yourself in theory the the change that was pretty straightforward of moving to pin there are other things that are sort of fun to think about like you could try porting buff stream to be a sink but in general I would just look you can almost go to crates IO and search for like a sink and look for smaller crates or crates that haven't been updated in a while they are likely to be on future 0.1 and they're likely small and then you could just try to move those two to a standard future and pin do you think the pin enables GUI engines like flutter to be implemented in rust in an easier way um I don't think so so pin in and of itself only really provides a way to give you safe self-referential structs or types it doesn't really unless you need self-referential structs for things it won't really help you now that said it enables you to get async functions and async blocks and await and so in that sense it's really useful for ergonomics but that's more about async await than it is about pin even though pin is what enables it in the first place so it could only be that async await helps with flutter but I don't think pin in and of itself does I've enjoyed your videos but as a novice a lot has gone over my head yeah so these videos are very much targeted at people who are relatively experienced with the language unless so at like beginner content and part of the reason for this is because I feel like there's a bunch of there's a bunch of material out there for beginners like reading the rust book for example is really good I know there are some streamers who also do more like introductory rust programming content I don't have any off the top of my head I know Ryan Levesque did some I don't know if that's the surname no if that's how you pronounce it but he's done a bunch of streams a while ago and I think hopefully starting up again has done some stream that are a little bit more sort of introductory or beginner-friendly so that might be something to check out but if you search for like Russ live coding I know there are also more people who are excited to do it now I don't think I will be doing beginner videos anytime soon because I have enough advanced concepts to tackle you might find for example the open source contribution stream better or the one where we write a hashmap those might be easier to follow if you're in the earlier stages and the in theory the next stream will be an open source contribution stream so that might also help lock free needed for sing signal slot communication for GUI I know that doesn't really relate to pin though will you still do go programming have you thought and doing videos on it so I haven't done go programming in a while now I used it a lot many years ago but for the past like many years I've almost exclusively been using rusts except for like coursework I don't think I'll be doing videos on it I also just really enjoy writing Russ code and I think it's fun and interesting like the fact that I can talk for three hours about one type and the language is really interesting some might argue that I should like go into formal verification and that kind of type stuff but that's too much for me how'd you get into rust was it just through your research I I got into rust because when I started my current research project I had to pick a language for it and I was like well I couldn't use go but go annoys me a little and the system language from Mozilla looks kind of cool so I might as well try and now five years later I have like eighty thousand lines of code written in it much of it from like very old versions of rust so yeah I mean I just sort of started and this is maybe a luxury of academia is that you can you can just sort of do that and pick pick a weird language and go for it and it's fine it's actually worked really well I had a bunch of co-authors on this project to and and I think collectively we've found that rusts while there's been there's certainly been alarming learning curve and some things feel more frustrating to do in rust than other languages like you have to be more meticulous about your programming but I think it has probably saved us time overall and refactoring is a lot easier than I think it might be in other languages when will you tackle lifetime C ultimate advanced rust topics so what about lifetime says you want covered because too many lifetimes aren't that complicated all right I should rephrase there's like higher kinda lifetimes which we don't have yet but the end is like the four constructs so there are some lifetime things that are somewhat complicated but in general I don't know what I would say about lifetimes for like three hours we we just finished for today but the topic for today was pin and unpin all right I think Oh implicit lifetimes makes learning and understanding lifetimes hard at first maybe so I mean implicit lifetimes are they are a little annoying because it it hides you from things that will show up in errors later so that's possibly true oh yeah the TCP series was pretty fun it's a good combination of like real world but also a lot of rest oh it's true lifetimes have become like a sort of canonical advanced trust topic even though I think when you work with the language everyday lifetimes are just not generally my issue like and they don't come up that often really I think they come up more in when you start out with rusts almost and then after a while it sort of you just implicitly program such that they aren't that much of an issue maybe is a sink stood in combination with spin equivalent to normal stood um that's a good question so the question is basically if you take that like a synchronous constructs we have in rust and you just spin on them like you just keep calling pol until it returns is not the same as just using the standard library the answer to that in the general case is no so for example imagine that you're trying to read from a TCP stream if you use a sink and you just spin on Pole and then what's gonna happen is you're gonna be doing a read to the operating system so you're gonna do is this call the response is gonna say not ready you're gonna do another read it's gonna say not ready you're gonna do another read you're it's gonna say not ready and so you're gonna be spinning on the CPU and you're gonna be spending a lot of resources crossing the the kernel boundary whereas if you do a blocking call like you do a read that's blocking the kernel is gonna basically Park your thread until the read is ready to continue or to proceed with some data which is far more efficient in terms of and because you're not spinning and you're not doing lots of system calls now if you use a run time then that changes things a little because the idea of a run time is basically when you do the read and the operating system says not ready that future will be parked and the the run time is gonna go ahead and run other futures and where there's no more work to be done it's gonna tell the operating system here are all the things I'm waiting on wake me up when any of them make progress and then the operators gonna say you can now read on stream three and that it gets woken up and then it continues on the future that was passed previously so when you have a run time you get sort of the best of both worlds what's generally your daily issue with rusts programming um I think my daily issues are a little bit different from many people's daily issues we have a code base is large enough now that there's some really subtle like business logic interactions that come back to bite me I think the biggest issue I have that's like a general rust issue is when I have as I'm using Tokyo as my runtime and I have lots of futures running all over the place many of them are manual implementations and this is an older codebase they used to be entirely synchronous that's been converted to be asynchronous and that comes with some issues where there are some pieces of code that are still blocking and you have to be really careful with blocking and asynchronous context because you're gonna block the runtime and that can lead to weird hangs where suddenly your application is doing nothing it won't make any progress it's using no cycles but it's not finished and figuring out like why futures are hanging is a pain so I spend too much time on that when will the next live stream be if you happen to know no pressure that's a good question I don't know yet my parents are gonna be in town next week so probably not next weekend I would say probably two to three weeks from now it depends a little bit what it ends up being - if it ends up being an open source contribution stream which is what it's looking from the votes then I'll sort of solicit some opinions on what people want to see me contribute to and then we'll take it from there but my guess would be two to three weeks do you think it's worthwhile to learn rust if you already know see and Haskell yes absolutely I think C is a very handy language for doing like very low level work it's also a handy language to know just to know sort of how the works in some sense at this point there are very few things I would use c4 instead of rust like I feel like I would just use rust instead and Haskell so Haskell you can't write performance software in-house calor it's very hard and I kind of like the imperative style I don't want everything to be functional so I mean I think there's absolutely space between C and a skull for another language I think rust is closer to replacing C than to replacing a house call but then again I don't really know what software you would write in Haskell in first place like I've I've dabbled a little bit in Haskell myself and I don't think I would use it for anything although I do use it for my window manager but that's separate have you ever looked into the embedded working groups stuff that's being done in the rust ecosystem a little bit it's really cool I like that we're developing these these tool kits and rust for building embedded devices there's been a lot of work on robotics for example which i think is really neat I think rust fits well into the space too because it has the same low level control as C does where you can actually work on embedded devices and have tight control over memory but at the same time you get higher level constructs and and more safety guarantees which are necessary and embedded contexts the biggest thing missing there I think is better support in libraries for operating without the standard library basically a lot of crates rely on stuff from the standard library from Stud which makes it hard to write things in embedded because you can't rely on any libraries or crates that requires stood and so I think having a push in the ecosystem to move people towards no stood or in particular to move towards only requiring a lock for example so only recording the allocator and not this is called interface for crates that can do that would be really good I'm the person who's been testing your PR - my sequel async oh yeah that's been really cool so what they're talking about here is the my sequel async rate is future 0.1 and III did an effort to port it standard future and that now basically works you can write like Aysen can await around my sequel connections which is really cool and it's sort of like the my sequel async has sort of been uh the the my sequel library has been the primary like focus for a long time and the my cycle async has sort of been a little bit of a not an afterthought but it hasn't received as much love I think now with async/await really coming though it's gonna be the next it's gonna become a lot more important are you interested in X modded like we am in rust yes I mean I already use X monad I would be happy to switch one to one that uses rust because I currently can't really understand all of my own config which is a little weird I don't really like having to recompile my window manager to change the config though so I don't know exactly how those would interplay I would really be interested in a general debugging video like with gdb or your perspective on it and maybe more particularly on a future context yeah so debugging in rust you can do with gdb there's a lot of stuff out there there's actually I did a MIT lecture series a while back with some people from my lab where we covered a lot of just useful topics to know in computer science and one of them was a video on debugging and so you might want to take a look at that that covers gdb in some detail and applies the same to rusts code mostly doing some performance profiling might be fun although there I would look at the the stream where we port flame graph to rust that has some of that in it but debugging in future context I think is a big back where I don't have the answers yet either I think it's an ecosystem we haven't figured out exactly how to do it the tracing crate is really promising here so I've been following that development a little bit and because it's gonna be integrated in or Tokyo is gonna be using tracing as well that's gonna make it a lot easier to do tracing in future contexts and see which futures are stuck where so tracing is as a general logging like crate but that has a lot better support for things like keeping track of the context where the log statement happens and that's gonna help a lot but I don't know exactly how far along that is there's a work on something called the Tokyo console which is basically like a way to get insight into what the runtime is doing which i think is gonna help pr2 but I think we we haven't really seen how that whole part of the ecosystem you got this deep into and you think mastering a language ecosystem offers a similar growth opportunity to learning a new paradigm so I think so I got pretty involved in the PHP ecosystem back in the day I wrote a bunch of like libraries that God used for things I got involved in the NGO ecosystem in that I built a bunch of libraries there too I just really liked building libraries and I also tried to contribute to the language but both in PHP and in go that was pretty hard at the time and I think to some extent still is like go is not a community language and so it's hard to be involved whereas with rust I also started writing libraries but then I also got more involved with the community in part because community or the language is more community oriented and so I actually felt like I could contribute to the language as well and so that was good so so in that sense yes it's the first one I got this deep into but it's not the first time I get into ecosystems and I think mastering a language ecosystem offers a similar growth opportunity to learning a new paradigm I don't know how to parse that sentence but I'm gonna go with yes I think getting involved with an ecosystem is good for you as a programmer that's what I have to say about that did you take a look at the Bastian project I don't think I know anything about Bastian where's the lecture series the lecture series are called hacker tools and it's at hacker - tools thought github do oh sorry this is light sorry people at home and it has videos on a like short lecture videos and notes about a bunch of topics and you'll see there's one on where is it program introspection so I'm like debugging and profiling so I would watch that video are there any problems or tasks for which Tokio runtime isn't a good fit and why so the Tokyo runtime is multi-threaded and so there are some workloads where you really want single threaded operation Tokyo has support for it but it's less of a focus but I don't really know where that would matter I think in embedded environments you might not want to use Tokyo because it relies on a lot of basically OS integration so in embedded context you might want to write your own executors for example apart from that I've been pretty happy with Tokyo I mean it has the advantage that it's being used in production in a lot of places so it has a lot of maturity and so for the database like the research database I'm building we've found it to be like very performant which is helpful and I know that there are many people who use it who care a lot about performance and so it is very focused on like being fast for the for the cases that matter in some sense like empirically have turned out to matter I also know that they're pretty devoted to backwards compatibility which is useful because it means you don't have to like break your code all the time although the move to a standard future is pretty breaking I don't think I missed any all right I think with that we're gonna call it a day if you have thoughts about the episode like feel free to leave comments I'm gonna upload the video shortly after this or if you have like follow-ups I've had a bunch of streams now where people have posted really helpful comments and follow-ups on problems we ran into during the stream where they find solutions references to explanations of particular concepts find any of those let me know if you have ideas for upcoming upcoming streams that you want to see it just like ping me on Twitter send me an email and I'll try to add them to the voting site or if they're really great I might just do them and if you want to hear about upcoming streams just like join the just follow me on Twitter or I'm also on Mastodon and that's where I announce all upcoming streams they will all be on both YouTube and twitch just like this one and if you want to support the Amazon wish list is there how does one get begin to get into all of this you start in the beginning you read the rest book it's very concrete and practical thanks everyone it's been great and I will see you next time so long farewell auf wiedersehen goodbye

Channel: Jon Gjengset

Views: 30,127

Rating: 4.9780221 out of 5

Keywords: rust, live-coding, pin, unpin

Id: DkMwYxfSYNQ

Length: 183min 53sec (11033 seconds)

Published: Sat Sep 14 2019

Reddit Comments

[u/U007D]

Love your stuff, Jon!

[u/Programmurr]

A very timely overview.

Excellent audio quality with your new microphone. Thanks to whomever gifted it to you.

[u/from_yourperspective]

Great video! What configuration do you use for Vim to get those inline error messages? Rls + something?

[u/Bromskloss]

The meat begins at 5:26.

[u/SCO_1]

I suspect i'm never ever going to understand Pin, so pin_project is very welcome and i wish it was in std.

[u/kosayoda]

Great video! I enjoy your work, hope to know enough Rust to be able to better follow along soon~

[u/CantankerousV]

Awesome! Good choice of topic.

[u/yaymukund_]

It's a real treat to hear you break down Rust code and concepts. Thank you!