Rust Stream: CLI Project - Part 1

all right welcome everybody to another rust stream been trying to get into the habit of doing this on a regular basis and perhaps through Fridays I did some streaming's but on kind of specific rust topics today we're gonna switch it up a little bit and start doing a project that I need to get done anyway and I thought hey why don't I just go ahead and stream all I'm working on it so that others can see I don't know a project being built from nothing instead of just kind of a contrived example so hopefully this is helpful for four people I hope that we'll be able to make good progress today I highly highly doubt that we'll be able to get the whole thing done probably going to be streaming for about two hours or so and then call it quits for the day and before we get started with that I'm gonna describe a little bit about the project itself what what it's for why I'm doing it to give some context of the whole thing so hopefully that's helpful and of course chat keep me honest if there's something I say that you don't understand or something like that then please ask the the aim for this stream today is to be relatively follow a bull by basically anybody who has programming experience even if you're just starting out learning rust hopefully there's nothing here that that will be kind of too over-the-top or too advanced it should be relatively straightforward but of course chat ask any questions that you might have about that and we will make sure to get those answered all right so switching over now to my browser to talk a little bit about what we're going to be working on today so a couple of days ago I tweeted out about a project that I had just I've been working on with with a co-worker of mine by the way to work with Microsoft so that's why this is under the Microsoft github label here and the project is WinRT rust so you might notice I'm on a Mac I am NOT a Windows program or at all despite where I work but I've been doing a lot of windows programming lately and it turns out that it's actually quite quite interesting and a lot of fun and this project has been quite a lot of fun to work on so what is the the windows runtime basically kind of the short of it is that it is a way for calling windows api's and the reason that this exists is Windows itself is very old as far as software goes it's it's really old going all the way back to some some api's that windows expose go all the way back to the 80s early 80s of that and if you try and count up all the different api's that windows exposes it turns out that it can be upwards in the millions of different api's that it has and so back in the 80s when they started working on this stuff the way that they decided to expose their API it was kind of as flat C style api's which is kind of the norm back then and and a very sensible way to expose thing and and the way frankly that Linux Princeton still does today and and that works pretty well especially if everybody's programming and C right but when you get to wanting to use other languages languages that might have kind of richer or different type systems or object systems from C then you might run into an issue where you're coding one way and then you come to to call into an API and it turns out that it's completely you know different flat C style API and so the windows runtime basically is a way of exposing these types of api's and a more I guess they would call an object-oriented sense but we've met to rust which is not really an auditory into the language at all and it feels quite at home there as well so it is it is object-oriented in the sense that you have kind of things that you would call objects and you can call methods on them that's basically what what object-oriented means in this sense there's not kind of this there's not really a sense of kind of deep rich hierarchies of where you inherit from from one class or another or something like that it's kind of a little bit simpler than that but getting ahead of myself this is just a way of exposing a lot of the windows API in a way that is not kind of a flat C style API surface so what can you use this project for it well you can basically use it to call a large swath of Windows API for doing anything that you would do on and kind of like a desktop operating system like you know creating apps so there's api's for doing a whole bunch of stuff like drawing to a window and all the way up to kind of modern things like doing machine learning and stuff like that and the cool thing about it is that all these APRs that are described in kind of language agnostic metadata and this project what it does is it reads that language agnostic metadata and generates a lot my screens about to turn off here make sure that doesn't happen all right it it generates rust code based on this language agnostic metadata it turns out that the code that it generates is relatively ish straightforward and so you can see an example of it using air like there's an XML document API for instance for parsing XML documents and so you can see this just this is all generating nobody wrote this XML document rust API it was all generated from language agnostic metadata and so you call XML document new and then a lot of these api's can fail because at the at this the the API boundary surface between the RUS code and what is most likely C++ because most of these api's are probably implemented in C++ there's there's possibility for failure there if if some things go wrong so you're gonna see a lot of question marks here but then you just call the methods like you normally would and rest like load XML and you can get the document element blah blah and so this should probably feel relatively kind of rustic to you but it's all generated from language agnostic metadata and and if you want to see what you can what you can do with this then here is a co-worker created a clone of minesweeper kind of a modern version of minesweeper and by and large this is implemented using winter TRS and so if we jump into the code in Maine here you can see here we're kind of importing a whole bunch of of Windows ap is like collections of numerics and composition for doing graphics and stuff like that and then there's a little bit of setup here that we don't have a good answer for how to kind of package this up into a into a nice you know at some point there will probably be a crate where it's just like you know initialize you know everything a window or whatever we don't have that yet because we don't want to really commit to that but definitely encourage the rest of the community to to take a look at that and come up with things and you know hopefully we can together come up with some some nice API for for just bootstrapping a Windows application and then it's using things like Linnet which if you're not familiar is an awesome an awesome crate for doing kind of cross-platform window instantiation and handling and stuff like that and so we fully take advantage of that which is great and you know this is all kind of win it stuff but if you go into the actual logic of of the whole thing like here minesweeper down RS it's kind of where the meat of the whole thing is you can see in here this is all calling these these rich window AP is for for setting brushes and offsets and visible blah blah so drawing to the screen here and this all these all these ap eyes were generated by this one or TRS from metadata that's just present on any Windows installation so whoo that was a long-winded explanation of what this is hopefully that made sense to everybody and why it's useful basically this is just another way of being able to create Windows applications one question that I often have gotten in the past couple of days since we announced this is how does this relate to the Wynn API RS so it's somewhat confusing that they're that Windows has been alone around for so long that it kind of has multiple API surfaces one of them is the win32 API surface and that's that kind of flat C style API that I was talking about before where you're just calling C style functions and that's kind of you know before eight years ago or whatever that's kind of how you built Windows applications before when our tea came out but that still exists like Windows can't get rid of anything it cares a lot about backwards compatibility so like it all still exists and the Wynn API thing is is in general a way of it's a hand created version of of that of the win32 API which can be useful if you need to call into the old API s and not every single API or every single thing that you would want to do in a Windows app is is available through a rich WinRT API and so you might have to you might have to use the old 132 API since Ted and so when API is is kind of a handcrafted version of that which is awesome and and kudos to the people who are keeping up with that it's it's it's really great I think the thing is if you can use when RT then that's what you should be using and if if you can't because you need some API that might not be available and when our teeth and then you use one API instead so hopefully that answers that and one way I think we might have one more question so every call can fail or doesn't have a notion of methods that can't fail as well I think the assumption that we have right now is that every single call can fail now whether that's actually true in practice is a very good question I think it's probably not true and practice but there's not really a good way to know whether a call will fail because ultimately the like we're going over I won't get into exactly how when our tea works but it's for those that know anything about windows it's it's basically a wrapper around comm and all of those comm calls have the potential to fail like they return a result that can either be true or false and we're basically or not true or false it can be either be like success or some any number of errors and we're basically just always propagating that up maybe in the future we can get smarter about about which ones can fail and which ones can't and then remove some of that error handling that for all intents and purposes will never fail by the way this this crate here is not on Chris at i/o right now because it's still kind of a pre-release we just wanted to put it out there to see what people think get some feedback on it there's been talk with there's an existing when our tea implementation that exists out there those and in various kind of state of being done and there was some talk with the developers there of potentially collaborating and then taking over the crate on crates that i/o since this seems to be further along than the melon is and of course has people who are paid to maintain it maintain it which is great all right so I'm assuming there's no more questions then we can get on what the activity is for today so for those who don't care about Windows at all bear with me just for one more second we're going to get out windows land really really quickly so if we go back to when RT RS here so this is really great for for windows api's if you have another question go ahead and just ask it I'll pick it up when you ask it this is really great for windows api's and kind of it's built into we have a way like there's a known location of this metadata that describes windows API is on any Windows 10 installation and so like by default it can just go out and like grab that and be smart about what you actually want to use and then you know generate the code for you on the fly but when RT is kind of in general is a way that you can build apps that don't interact with the Windows operating system and there's a lot of people out there that create api's that are kind of built on top of this windows r runtime kind of API surface and the reason they do that is because when our T gives you a way of exposing a language agnostic API surface so like we have rust that's calling into presumably C++ based API s and the windows and you know that are built into Windows but of course they could also be in net they could be in JavaScript just like they could be in theory in any language it just needs to be a language that that so you know exposes support for Windows RT which is not not that difficult to do and so we want to be able to have the ability to author api's maybe you know hopefully eventually in Windows there might be built in some rust api's as well who knows and so we want the ability to to give that opportunity but then we need some place to publish the metadata where that exists so we have an API and you know and dotnet can call that API as long as it gets a hold of the metadata that describes that API and C can call it in C++ and JavaScript and you know whatever as long as it has a Windows runtime invitation as and as long as it has a hold of the metadata that describes that API it can generate code that can call that API but where do you host the metadata for Windows you know built-in API so that's quite easy you posted somewhere on the operating system and in well-known location and you know whatever can just pull from it but if you're some random app or whatever you have to have a place or or even the EPA the Windows API wants to expose it in a way that's not kind of baked into the operating system because we want to have different versions and stuff like that and so where do you do that and the answer as of right now is in my opinion a little bit weird but kind of makes sense coming from kind of the Microsoft perspective that's as a nougat package and though for those that don't know nougat is is kind of a dotnet the.net equivalent of crates that i/o basically and nougat you know can of course package up c-sharp code or f-sharp code or whatever but it doesn't have to and some packages just bundle up this this metadata to describe these api's and so here we have when 2d uwp which is basically an api for doing 2d graphics rendering on windows and this this nuga package only really it includes metadata to describe that api and it might include some some dll's as well to kind of back up those api's that aren't they might not be built into the operating system but that's really all that they're exposing and so this is where that metadata that rich metadata lives and what we want to be able to do is to kind of fetch that metadata and generate the RUS code that is able to call the api's that that metadata describes and so the ideal would be to somewhere say I have you know a dependency on this metadata from this good package fetch that metadata for me move it into a well-known place and then when our TRS well know where to find it and will generate the rest code based on that and so the idea for today is basically to create a tool that that does that that and the idea that I have in my head is that there's a there's a place in and cargo tom'll that is that is unchecked it doesn't you know cargo won't won't complain if you write something there where you can basically put in anything you want to and we would have some kind of key that's like nougat dependencies or something like that and then we would have a cargo sub command where you say cargo nougat install or whatever and with nowhere to fetch it from you know nougat org download it unpack it to a to a specific location that then when our TRS would know about so that I can generate the rest code for you and so this is this is kind of a way of of allowing that metadata to live where it has has up until this point always lived it lives on on nougat but integrated almost seamlessly into the rest workflow where you just have to no cargo do get install instead of cargo install and it should all just kind of work now the obviously the ideal would be that we would just you know package up some of these on crates that aisle and stuff like that but that's there's a lot of these api's a lot and so to have somebody who's responsible for always saying oh there's a new version of one 2d uwp make sure that the crates that aisle version is up-to-date and stuff like that is a lot tasks for and so this is kind of a way of everybody wins that people who are in charge of these api's if they don't care about rust don't know about rust who whatever they don't have to their stuff just goes where it's always gone and our tools do the best to kind of work around that and pick up the metadata from where it lives naturally so a real quick question the import macro is used by the user the crate does that mean when I build rust doc I won't really see all the different API it's like for one API yes so from so when API is basically definitions for all the api's that exist for windows when RT RS is effectively a a code generator and so WinRT RS doesn't really know anything about API is that actually exists out there it just knows how to take metadata and generate code based on them the only thing it really knows about about existing API is it knows oh there's a place on on your Windows 10 installation where that metadata is described and so I know how to fetch that and generate the code for you but it doesn't know what that what that metadata will say at all it has zero knowledge of that and so you know how this will actually end up working in practice is that it is a big question lothar one day be a you know Windows 10 I don't know like collections is an API that exists Windows 10 collections you know version whatever on crates at i/o that's basically just take it using WinRT RS to read the metadata that that that API exposes and generate the rest code for it and then basically it's just like a cache on crates at i/o like that's completely possible we're not doing that right now because that's a that's a big commitment to have to like maintain hundreds or thousands of api's and crates that I oh that this would expose so we'll see but yes and in reality when our TRS doesn't actually contain any api's it's just a way of generating codes that met it that rich metadata describes hopefully that makes sense and it seems like it does okay okay we're 25 minutes in and I haven't done anything yet so really gotta get going what the plan is completely is we're going to create something a la carte go expand or if you know cargo edit or cargo add or some of these cargo subcommands it turns out cargo if you if you card will install cargo inspects band and then you type cargo space expand what cargo does at first if it if it sees a a sub command it doesn't recognize is it looks in your path for a cargo - that sub command and if it finds it will just run that and so if you have cargo expand on your path then cargo space expand will just execute cargo expand and so what we're going to do here you know that that's all that we care about fan we won't we won't use it today although it's an awesome tool you should check it out we will be following that same pattern where we'll probably do cargo nougat or something like that which unfortunately exists on crates at i/o so it's something we're gonna have to figure out at some point but we'll just we're not going to publish the crates at i/o today so it's fine we'll just create cargo - nougat and that will allow us to do cargo space nougat Wed just using the regular cargo as long as somebody has cargo and open it installed and we'll do cargo nougat and install and it will note where to look inside of the cargo dot Tamil file for the dependencies that you've that you've kind of listed out and it will will download them and and and do the right thing put them in the right place so what are we going to do we're going to be using struct opt today to do to do parsing of command line arguments the reason we pick struct oft is it's a pretty it's on top of clap which is kind of like the de-facto command parsing library instruct top is just like a nice derive macro on top of it that eventually will be brought into a clap in the next version whenever that comes out and and I know struck topped Oh fairly well I guess I mean used it before I should say I'll probably end up messing it up somehow but I've used it before so I'm familiar with it so we're gonna use it today to do the command line parsing and then I think what we'll end up doing is using Cerf as the HTTP library if you're familiar with Russ HTTP then you might be wondering why are we using Cerf versus using something like hyper and the reason is purely I don't know I just I use hyper and other projects that I do for work and I thought why not use Cerf because it seems nice and I hadn't actually been I haven't actually used Cerf in a kind of a real thing before I played around with it a little bit but I haven't actually really gotten to use it so I thought one and if we you know if we end up not wanting to use surf at some point switching it out for something else shouldn't be too terribly hard you could argue we should use something that's synchronous by default because we're probably not going to do asynchronous stuff here but that we can make these changes later on so that's the plan for today hopefully that makes a bit of sense and I think we're going to get started so if you have any questions about what we're doing and why we're doing it anything like that please let me know and we'll go ahead and get started so I'm going to start now by doing a new project so we can do Cargo new and what do we want to call it cargo new cargo - doing it alright and then I'm gonna go ahead and open this up and yeah it's kind of that's what I've been using lately I keep saying that I've been using it lately up and using it for like a year and a half now so I should probably stop saying that and just to make sure that everything looks good let's just go ahead and and and we get hello world awesome so we're ready to go here and I think that the first thing that we want to do is do a like create an install command so we'll we'll give away to where when we when the user runs this if they do carbon tetra negative space install it will run our install command so we'll run whatever we want it to and so I'm now thinking that struct opt is maybe a little bit of overkill but I'm just gonna use it because I think if once if this in gains more and more functionality it's nice tab we don't necessarily need it cool so we can do struck up 0 3 here and put it in here and then as you can see here this is kind of how you define your your options so you create a struct and it can be called whatever and they call it up here and then you define kind of the flags that go along with it and so if we just copy this and I think it uses the documentation as well to kind of create good documentation or great so like this we're gonna have ops I think we call this nougat here and then we'll have our derive here a utility for interacting nougat packages and unlike here where this is basically Flags so like for instance you have you know - see or whatever here for NB cars then we don't want to do that necessarily right now what we want to do instead is have a sub command and I think in examples they have a they have a sub command there's sub-command aliases where is it yeah so here is an example of a sub command where this is their top-level struct them and they have sub command so you annotate it as a sub command and call it whatever you want so we can do the same hitting here and we're gonna call this stall and all like that and then we have to create a struct called install as well let's do below and I think for now we don't need anything in there so this has described the command line arguments that we want to go ahead and take and so I think the next thing that we need to do if I remember correctly is go ahead and call this from arcs here so if we call from arcs here then we will then we'll get an A we'll get an OP that kind of describes our describes our our command and now I'm thinking this is this isn't how it works we don't want it to always be installed we we basically want sub command like this and then sub-command and then you know sub-command and this is where you know we say the various sub commands that we might have in turn out we only have one scope sub command called install but you know in the future maybe we have more and I think I still need to annotate this like this and so now our sub command here will tell us what kind of sub command do we have and of course right now we have we only have one but in the future we might have more so we can do let's just do this real quick where we print out the sub command that we get so off daaad so come in and hopefully what that will do is print out install when we go ahead and run it whoops so if I run cargo run - - install oh that's not good what's going on here we got some kind of linker error undefined symbols for x86 64 well this is strange yeah where we are running unstable this is very weird I've never seen this before so sorry for those who are hey what is going on and if we do because it struck top somehow we make our main here oh that is it is very strange I think if when I run into interesting things like this I like to just like blow away my target for I don't know because sometimes that helps and then just like hope that it works and we'll see yeah no idea why that happened if somebody figures out what I did there - cause that BAFF and let me know something charge yeah I don't know so we should probably bring this back now and see if we get the right thing oh we don't obstruct opt so use struct opt dropped and that hopefully brings everything in looks like it there we go great so I think Chad was asking how did I fix the issue the trusty old delete everything in your build folder or your target folder so of course everything if you see down here when you build something it will go into the target folder here and if you end up with something weird where especially some strange linker error I have seen this very rare occasions where something gets confused and usually the best thing to do is just clean the entire thing so what I did was RM minus RF target I could have just deleted the whole thing and that that seemed to fix it so I built it from from scratch and worked I don't know who knows who knows okay so we're good to go we know that when cargo install gets called what happens if we call like this we get our help Minea which is why we're using this thing which is nice and foo argument food was not expected so I tried cargo clean and it also didn't work so as far as I'm aware cargo clean is not exactly the same as just our MRF target folder and it does do something in addition to that but I just could I just it gave me a weird error and I I don't have time for this and I just deleted it so that's that seemed to work I don't know who knows about these things all right great where are we we are successfully running our thing where we actually get the install command so I guess what we can do now I say opt this is a little ridiculous but like opt sub command and it's install like this and of course that's the only thing we can get otherwise it would have aired and so in here like we can say like perform do install great so now we're going to do install and this needs command : : and do install doesn't exist so let's come down here and great do installed for now like this and let's think a little bit about what we want to do here so what does installing mean it means we want to read the presumably we're gonna try and find a carbon tamil file we want to parse it we want to look for a special key that we will have in there that is our nougat dependencies or whatever and then go one by one and download them all and and then you know then we'll talk about what we do after we've done a lot of it okay so the first thing that we need to do is find a cargo of a cargo Tamil file I think for now the heuristic we will use is a carbon atomic pile should be in the current working directory so wherever the person is running the command from we're gonna try and find a carbon a Tamil file there and if it's not there we're gonna blow up is that the best solution ever no but we're gonna do that from now because it's definitely the simplest thing to do so how do we do that let me go ahead and close on these things here the first thing to do is look at the standard docks and here and inside of FS I think is a good place to look here and I think what we want to do is read dirt here or we can just go ahead and try and read it that's probably the best now the question is do we want to so the thing I'm thinking in my head is do we want to use standard FS read ear so that the standard way of or standard FS read or whatever the standard library or do we want to use something like a sinc standard which would be the async choice for now because we're going to use surf and surf runs on top of facing standard Tokyo being another popular one so if you're if you want to use Hyper instead you might more lean towards using hyper Tokyo the question is do we want to use that we're going to bring in an asynchronous runtime no matter what and so and it's totally fine to do blocking outside of our asynchronous context and then when we need asynchrony do asynchronous programming but you know a good question it is why don't we just go ahead and you know do a sync the whole time but I did you know what we're not going to do that we're going to just use the regular standard library for now and then we'll me absolutely need when we absolutely need to do asynchronous which is what we're using serf then we'll we'll go ahead and create an asynchronous runtime and question from the chat aren't asynchronous files really slow like much slower than blocking even I'm not sure I know that async files basically are they're not really a like files aren't async most operating systems don't provide asynchronous of api's for for file handling and so but what most async runtimes end up doing is having kind of a built in thread pool for for doing these operations and does then end up being slow slower than it would otherwise I have no idea it doesn't really matter for our use case speed is not what we're going for here we're just running a simple thing as long as it's not incredibly slow we're really not trying to eke out the every last bit of performance here okay so let's go ahead and read our carbon on Tamil file that we're just going to assume is is where we need it to be and what we can do is just do cargo tamil like this and yeah let's go ahead and return the result here so yeah let's do this the i/o result which just hard codes the air to be an i/o error instead of you having to provide the air and we're going to return unit from this all right so now we can do our nice question mark here was complaining about when we have to everybody's favorite thing that all the blog posts have been about lately the explicit wrapping of our of our unit here okay so we're gonna read the the carbonyl Tamil file it's complaining about here we one two for now we'll just go ahead and unwrap we can clean that up in a little bit and say you know some nicer air handling for that but for now loaders crash if the corridor tamil file is not there and this gives us bytes here so so we need some way of taking these bytes and making sense of them so we want to read this carbonyl tamil and and have some kind of representation of it as a carbon on top or at least a Tamil file but I believe if I remember correctly that there is a crate out there for parsing cargo Tamil files yeah cargo Tamil that looks good cool documentation so this great defense trucks that can be deserialize with Saturday to load and inspect corona tamil metal data that's awesome cool I think that's what we want so yeah manifest is the top-level structure there you have here and yeah we can even pass in a slice here which is what we're going we're going to get back a vector but we can easily turn that into a slice and it will either give us a result of right now perfect so we want to add carbon little tamil zero eight to our file here cargo cargo 0.8 and no I was a car boat underscore Tamil yes underscore yes cool and so now what we can do here is use cargo cargo Tamil Manifest this gives us manifests that we want to go ahead and parse and we can say a man if that's got from slice and then we pass in our bites and don't think this will not is it going to dereference en law hopefully cargo check is taking a while what thing oh it's added a new dependency that's why but also streaming for me is it definitely slows my Machine down so see if there's any questions is that sufficient for extending the format ah that's a good question so we want to we're going to have a slightly extended version of parvata tamil because cargo Tamil and I think even like this cargo tamil manifest and see hopefully the manifest format awesome I like it when DuckDuckGo works the first time doesn't normally there is a here it's metadata so the metadata table inside of karbala tamil cargo will default warned about unused keys in the car tamil to insist in you know typos and stuff like that that's awesome but the package dot metadata table is completely ignored and so anything that you put inside a package that metadata just gets ignored by by cargo which is great because we can add stuff to it and so the question from chat is does this crate do the right thing and just give us how to know like a raw representation that we can deal with so it's met its package let's look here package and then it looked like metadata was it was yeah I think it is yeah so it looks like we can pass in anything that is a cargo Tamil file which hopefully yeah I think I think it will work so oh and there's a question from chat what gives the thumbs up yeah it's called crates I think where is it where is it crates yeah this one crates by its era whose car gives you the thumbs up if it's the latest version of a dependency and a little thumbs down it's not for a exclamation point if it doesn't exist at all cool great so it should be hopefully this should work then everything should work first of all we can propagate our error up from parsing here although actually it won't be a there's no conversion there well I don't feel like doing error handling right now we will error handling is something that people have asked about covering so we can do that a little bit later on because you know it's nice to talk about a different error handling strategies and crates so you can use a question am i using our LS for us to analyze our I'm using rest analyzer which for me has been much better than our LS was in terms of reliability so I'm quite enjoying it I think definitely recommend using rest analyze or a few if you can it somewhat falls apart an async code or at least it did the last time I was using it although now I think they have better support for for associated types which was kind of keeping it from from working well so we'll see I haven't used I haven't done any tasting program in the past couple weeks so and that support just came out there's a manifest from slice with metadata ripping from slice with him okay parses Caracara parses its packaged up metadata into a custom sorted compatible type package that's a good question if we want that do we want to so the whole thing with this here is that we want other people other people presumably might be using this and so we want them to be able to continue using it for their own thing and only we only care that our thing works as well I'm not sure let's let's try without the from slice both metadata and see what happens then maybe we may will we move back to it I don't know we'll see great so now we want to let's let's run this I think per and I think that manifest implements debug which means we can kind of look at it and we can run it on our own carbo Tamil for this project which is great for testing manifest let's run it ah takes a while I guess because we just just added a dependency great cool so this is some text should be able to do that instead and it's pretty printed so it's a little trick if you want to get pretty printed things that works quite well great so if we go into our cargo dot Tamil here and we say package meta data and I think if I remember correctly yeah so it could be like packaged up in a day the dot Android we're just coming to test it that it has like you know equals R so that should still work and we want a package diet metadata dot new get and I think I think camelcase is what is normally done for these if it's not let me know dependencies and then we want to have like let's go whoops wind so we want to do something like when to de UW P equals and I think I can do it like this and the latest version was 125 dot 0 okay and so if we run this that oh oh and my F looks like the crates thing does not do the right thing which is kind of funny could not parse as Tamil so that's why he's complaining maybe is it okay that works yeah this is the dot there maybe maybe it's the dot I'll say this feel like Chad is probably screaming at me right now and I'm not looking which is not a good idea there we go okay and then presumably this is a cool so we can look at our metadata then here have this exhaust or metadata so some okay and then it's a table and then Android in the good dependencies cool so this is probably not completely the right thing but we want to say new yet that's equals manifest metadata dot unwrapped for now and again we're just really not doing the right thing when it comes to air handling and then I don't know how to access a value from so we have we should have value here and then values so we're gonna pattern match on the value here I guess so let's call this metadata and we're going to say match metadata and if it is a table that's great and otherwise we're going to for now panic ain't no table cool and then we can do T here and get out the new gets new get tendencies this is obviously fairly nasty code here but that's fine for now this will we match here then this will be another and of course this will panic if if nougat dependencies isn't there at all and then we want to have tables not the right thing with gold value value and then we want to value table and this gives us our depth dips and otherwise we panic they're great and here let my depth is this definitely - oh I'm inside of my match here yes not do not want that and I sometimes I don't like when I rearrange this the whole thing see why do you do that to me great and now it's complaining that metadata doesn't do this doesn't it well let us see let's package sorry which package is also an option right dot package dot unwrap dot metadata looks like pancake met manifesto pancake all right let's see anything from chat here I guess yeah yeah that this this parsing is not exactly ideal this is super rough I'm just trying to get it to work as best as I can without thinking too hard about it so looks like we are we are moving the depths out from the table do we wanna from for now we should just be able to do that to get a reference to it instead so now Depp's is a reference into our metadata we can clone and at some point if we you just want to stop dealing with with that great cool so this is good we have I feel like we can take all this and move it to like it gets depth that takes a manifests and it will take a reference to a manifest and return a reference to a hash map of string and value I'm not mistaken of course standard collections hashmap since it's not by default there but it's a tunnel map of course of course it is - tamil map I hope I don't have to bring in another dependency just for that that would be unfortunate Allah where am I your package metadata Oh its value sorry in value and it's a map this is tunnel map map is there a way to just turn it into a hatch map no bummer bummer oh well and I don't believe car Gautama re-exports tamil so it looks like if we want to do this we are going to have to take another dependency I mean we're already we already have a dependency on that thing now we have to explicitly take a dependency which is fine so we want tamil 5/3 I guess and hopefully that just works out and now we can do use no use tamil map map this is no longer a hash map it is just a map oh yeah that's true so just for those that aren't we're we're we're calling manifests not packaged unwrapped and that will get a will take ownership of package but we can't take ownership because we only have a reference here we can do something like this where we turn it into a reference the whole time we're just kind of ugly but we're saying get the package but don't give me the package itself give me like don't give me an option of the package itself give me an option of a reference to that thing and so we're still going to always be pointing into this manifest and this this then works so this will give us our depths so then now we can do it oops dips get deps on our manifest like this and we can go ahead and print line looking let's do the same thing hopefully this this works or we can do depths here and hopefully we don't need to see that anymore hopefully we'll we'll see what we expect to see cool so now we see our our table here where we have our dependency oh that's interesting that it is that's how Tom works I guess you know so this one because it has a dot in it it needs to be a string like that yeah interesting I'm curious I guess floats or whatever don't work in Tom well that's why that doesn't work okay cool so now we have our dependencies you know I know I know I made it this whole hubbub about bringing in another dependency fee of this map but now I'm thinking we should just have a deck of tuples with the name of the thing and the version mm-hmm but maybe that's not what we want to have because eventually we want to support other configuration like being able to are going to assume for now that this is all coming from new Georg but maybe in the future will allow people to specify other noodle repositories and stuff like that I don't know we'll see so this is fine okay so we have we are things are going a bit slow sorry for that but we are doing pretty well we have parsed our carbon otoño file we've gotten the dependencies from it so now what we want to do is take those dependencies and download all of the nougat packages for those dependencies and so what I feel like we want to do is turn the these this map or whatever into like a vector of dependencies or whatever so we could have some kind of struct dependency here where we have a name and we have a version and I mean if we wanted to do something like like this I guess you know major minor super minor or whatever they're called but for now we'll just do a string that's easier mmm dependency and so let's do that I'm just gonna return back a Veck and this completely destroys the whole thing I did was getting a tunnel as a dependency now that's fine let us introduce derive debug on this so we can take a look at it so now we have these steps and what we want to do is say for hopefully this implements iterator over the key in value let's see what is K why is it complaining that's fine we can do here to do to get it to shut up oops so this will give us a key so we'll assume the key is keys a string and so that's good we'll assume that the key is a name let's just call that name and then value we want to match on value and if we only want to do something if it is a string so again sorry for the horrendous error handling right now but at least we know that where we need to handle errors we only expect something for a string and this will be our let's call it version you know what I like doing this value and then we'll match on the value and that gives us the version here and otherwise well penny and because we're not handling errors right now we're just crashing all the time so a string it needs to be a string maybe at some point we will allow numbers as well for the odd case that you only want to specify like a major number or something I don't know we'll see so this gets us the value and we are sorry the version and then we want to build our dependency where we say name is key and version is version and you know what we can do we can just go ahead and do depths dot it or map where we get the key and the value like this so we're switching this from an explicit floor up for loop to a map here and then we can call collect on it like this and we are getting because because we're only dealing with you know I'm just redoing all the code that I did before but it will we learn some stuff so we're going to take ownership over manifest so that we can like rip stuff out of it so we don't need any of this now we're taking ownership over it instead of taking a reference to it that's fine so that we can get stuff out of it so he here we can call depth so it er here will give us an iterator where iterates over and gives us references to the things inside of the collection what we would like to do instead is iterate and get own ownership over each thing in term in the collection and so a way to do that is to do into it or here that I would have expected that to not be that but we'll see and then version what is it complaining about here it's also a string I would expect these to be to not be references it's giving us references why is that because this is rep there we go so sorry with what I did let me explain real quick what that was all about before I had ref here and what that means is is that instead of getting ownership over depths here I was just getting a reference to it so basically a pointer into depth and so into it or here no matter what it can't give me ownership over the things because all that depth was was it was a reference to the thing and what I really wanted to say and I'll give me ownership owl at each of these steps I'm like taking ownership over the thing until Intuit are here will give me an iterator where I get the ownership over the key in the value which is exactly what I want and now that thing is complaining and collect is complaining because it's like I don't know what you want me to collect into and hopefully if we remove this and this it will now know so now collect knows it needs to collect into a vector because that's the return type all right so now we have a vector of dependencies debugs is not a thing this is why can't I can't move a tab index how do I get ownership of I think it's removed right probably I know those the weird map thing so remove yeah so indexing by default just gives you a reference thing so it doesn't doesn't like rip the thing out of the collection to hand to you it just says hey it's over there gives you a reference pointer to the thing in the collection say this is where it is it's in the collection over there and we don't want that we want to rip the thing out of the collection and take ownership over it and the way to do that is to call remove like this and what remove will will do is literally rip the thing out of will rip the thing out of the collection and of course now remove returns an option before the the indexing was if it wasn't there we just panicked for us now it's not doing that we explicitly have to handle that case and so that's why I had to wrap it in some and the last thing is we are trying to mutate T but we didn't say that it is mutable okay now let's run this thing and see what happens great now we have a vector of dependencies we get a name we get a version so that is wonderful I'm gonna stop real quick and pretend to drink water even though I don't have anything and I want to ask if anybody has any questions in chat or anything like that we went somewhat fast there I don't currently see any questions just one asks is anybody confused about what we're doing how we're doing it if there was a piece of code that you didn't understand that I wrote now is the time to ask otherwise I assume this was a hundred percent clear which I went really fast so and I can't imagine all of you have done to Russell before so I can it's totally understandable if somebody does not understand a piece of code feel free to ask otherwise we will keep going all right I'll assume we're good there just let me know if something occurs to you oh quick question so but it seems we have somebody new who has joined the stream what tool are we building I spent about 30 minutes going over that before so I can't give you the full answer but the the short of it is is for some strange reason what I need to do is download new get packages and I want a way of specifying those NuGet packages in my carbonyl tamil and it turns out that carbonyl tamil has a place where it can allow arbitrary data inside a package metadata and so we are parsing a carbonyl tamil and getting this information here and we're now going to download the dependencies from nougat org so why we're doing that is I'll you know chat if you want to help me out with that but that's that's kind of exactly what we're here great so we have our dependencies here what we want to do now is go ahead and and download them so we need to basically form URLs and and download our dependencies so I think what we'll do is something like download dependencies where we pass in our dependencies and notice I'm I'm just I mean if I were doing this not in front of people would I clean up a little bit more yeah because I'm wouldn't be talking so much and maybe I would have a little bit of a better idea of exactly where I want to go but it's fine to just kind of go down the the file and just keep adding stuff we we don't need to have it in a perfect structure right now we can clean that up later so our depths are going to be a vector of dependency and now what we're going to do is is download them so I guess the first thing we can do is turn our dependencies into a iterator of URLs because I think we have all the information from the dependency itself to turn it into a well-formed URL that we need so if we do imple dependency here URL and let's just have it returned as a string and this is going to return this the URL string that the dependency can be founded and so if we take when to the dot u WP as an example here here is the the raw download package if you just wanted to get a raw link to it and it seems like we just need to have this this you know prefixed or you are here in the name of the package and the version and that's it so that's pretty easy we just want to do format boom-boom-boom where the first one is going to be so of course I need self here self dot name and self dot version here and this will format a URL string for us so that's great that was pretty easy and then we can we can call like map or whatever on our you know we can do or url's and do Depp's dodd hitter map deb-deb URL like this and now we have a URLs iterator here so we can we get have an actual iterator to to the URLs that we that we want to to download right so now I guess the thing that we need to do is to go ahead and actually do the downloading and that's where surfing's going to come in that we looked at before so let's look at surf real quick and again we're using surf because I just wanted to give it a try this is what surf looks like so we're gonna want this we need a Singh standard as a dependency because that's what surf that's the asynchronous runtime and surf runs on top of and we're gonna we're gonna take advantage of this block on which basically takes an asynchronous block and and blocks the current thread until that that whole workflow finishes luckily though like the actual workflow that we're going to be doing will take advantage of of it being asynchronous and we'll be able to download all of our dependencies in parallel which is pretty cool so the first thing that we need to do is add a sync standard and yeah a sync standard as a dependency when oh well might be good but maybe I ain't up 1.5 okay in our mind 1.5 and surf is I think there's a 200 Alf one or two or something like that let's see that's probably gonna complain I should close this orbit maybe it's not published on crates IO oops yeah 1.03 should still work so one zero like that so there was a question was wouldn't a sink stander be brought in transitively so yes it's but it's brought in transitively when we bring in a sink surf surf itself depends on a sink standards but we need to explicitly call a sink standard and after we go back here we're gonna be calling it just like this and task comes from acing standard and in order to use it's not good enough and rest you don't if you use a dependency explicitly if you use like the actual API itself it's not good enough that some other thing has transitively depended on it you need to explicitly declare you that you depend on it hopefully that helps cool great so then we can just cut this here for now we can just go ahead and copy this here where we're going to and and in the spirit of just really not handling error and we are type thinking about error and only at all right now we're just going to wrap around this thing great and so in here what we want to do is in a somewhat similar to this but we want to call URLs dot map where we get the URL I can't do map unfortunately because we're going to call something a sink inside of it so we need to do a for loop here so for URL and URLs we want to do something like this and I'm thinking now we need some way of relating back the contents that we get to the dependency and so we might instead of just getting the URL here we might want to do something like this where we get the let's not do this let us do four depths and for depth and depths we're going to call dev URL like this this will get us the result here the response and let's let me go to the docs real quick and we want a response and there should be some way of getting body bytes and it seems like what we want is just the bytes here like this because we're going to have to we're going to have to unzip because of nougat piles at the end of they are zip files and so we really just want to have the bytes of that file there was a question in chat why can't we a sink inside of map because the closure that gets passed to map is not itself a is not asynchronous we could we could we could inside of call it inside of map but what we would do is be mapping over the thing with and then we'd get a an iterator of futures and then we would need some way of collecting all those features you know what let's do that this will be good this would be good to show people so we're gonna call map here depth and then because we need to do async stuff and here we're going to cut we're going to put this all into an async block like this and now it should be fine to call to call asynchronous things in here so this gives us a response then we want to call respond res body bytes dot a weight question mark we should be dot oh wait and then we can put that inside of of the thing here great so now we'll what we're getting back from here of course we can get rid of all this so what we're getting back from here is just an iterator of futures and futures and rust but when they're doing nothing when they're when you just create a future it doesn't do anything futures are innate there this is not like JavaScript or something like that where there's some background runtime that's constantly pushing things forward there's nothing ticking away here you have to basically submit your future in to the runtime in order for it to run and so we're creating a bunch of futures here but we're not actually submitting them back into the runtime for it to be for it to be actually run and so what we want to do is we want a way of saying I have a collection of futures and give me back all the results when on or give me back all the results when all of them are done and so if I can I'm trying to think what and Chatta saying isn't there futures all that's exactly what I'm trying to think of there is there is a function there there there definitely is a function that allows you to give it a collection of futures and it will wait on all the futures until until they are done I just cannot remember where this thing is we should look first look inside docs that are s async standard we're already depending on it and see if there is some kind and it might not be in here might be in futures the thing that we're looking for and I think it's this join although join is not exactly well we won't join join is if you have a kind of a a known number of if you have a known number of futures that you want to to await but we have an unknown number we don't know until run time how many how many things we're going to be downloading I wish I could remember this what is it this is it yeah exactly joinable yeah so here it is crazy future which represents a collection of the outputs of the future is good and you can see down here foo right here is just written as an asynchronous function so it returns a future and so what we call it Julian all on these futures here it will give us back it will will wait on all of them and then the only thing that I want to be 100% sure about is that this does not that this doesn't wait for one to finish before going on and trying to finish the next one the returned future will drive execution for all its underlying futures okay now I think this is this is fine this should work the end portion of part about it is that we need one extra it's not that we're we're already depending on futures transitively so it's fine but we have to have one more non-transitive dependency and join all might be available the future is kind of a collection of a collection of dependencies and so there might be a more efficient way of getting this but that's all for now that we can just do this futures join all and we can pass in our cool oh and of course we're gonna call a wait on this or otherwise it won't matter and you can see here it seems like it's futures futures future course futures future and this error that we're getting here is because we don't know what type of error that we expect I believe if you look at body bytes here for instance with type of error it's returning it is returning an i/o error interesting but this surf duck yet well sorry I just lost loves chat there because I went back on the wrong browser sorry about that you asked the question I apologize I you might need to repeat it let's see surf yet returns back a interesting seems like surf get doesn't return in air so I'm wondering why this is interesting it looks like it is returning an error for some reason this is not I'm not sure why that is so we need to just say what the what the error is I think the best thing to do is to say that and this is a way of saying what type of results that we're returning so figure out what the what the okay type is and the error type is going to be box I wish that that was I did it again got rid of chatting box stood error error we can see how far this takes us it's not unmatched angle break it up just because this and now we can see that it is an error and send and sync that's very ugly and I guess this is the same oops there's a we can go up here and say a type error is oops oh come on type error is all this stuff here so not to keep repeating it okay so it looks like it's not happy that we are borrowing DEP here so we want to move DEP in a sink move just means instead of trying to borrow here just like closures async blocks are very yeah they're they're a little magical and that they try and figure out the minimum a basic thing to do to get things to they want to do the least amount with ownership so they try to borrow if they really really try to borrow and if they can't borrow they'll return in error and move means no don't borrow and move this again so now we're moving in great and then so this is great we have iterated over our our depths we've mapped we're doing async inside of map here we're where we're getting the URL we're calling it we're downloading the result we're getting the bytes which might take a while and then we're returning the bytes and then we're getting all the the results here so so this is where russ analyzer is not quite perfect it doesn't know what results is here unfortunately but results here should be I believe vector of a vector of results of of vector of bytes basically and so I think what we can do here is just return this and interesting block on inside of a singing standard Docs that RSA sync standard block on black on should return the result of the futures that we're returning from inside of here so it should return you know and and okay here oh it's because of this so now we can say hey that's you you said up here that you're returning a empty duple you're returning unit here but I'm getting a vector of results here and so we should be explicit and say we want a vector let's just actually say we want a vector of vector what do we want here we want to return the dependency we will return a vector of dependency to the vector of you eights so this is basically saying give me a vector with with a tuple where each element is a 2-fold and the first element in that tuple is the dependency and the second element is a vector of you eights which will be the value that we've downloaded from nougat org which is totally not over returning or returning right now is a vector of results so basically what we want to do I think the first thing that we should do is if you want if you have a vector of results but you want a result of vectors you should be able to do collect where collect can be smart and say oh I know how to turn a know how to turn an iterator of results into a collection that's inside of a result so kind of like inverse the whole thing and we want to say this is a result of vector of u8 and there and one thing I like to do is write to do down here to case sometimes sometimes the areas that you're getting are too much because it's like you're doing a lot of things wrong to do can be nice to kind of say like past this point trust that I'm going to do the right thing let's see what it's saying and this is kind of a hard error message to read so a good thing to do and that situation is do cargo check so that I could see a little bit of a nicer error message so it doesn't know how to return can't can't build a result of vectors of you eights from an iterator of vectors of results of vectors of you eights that's the that's the issue here we've got we've got vectors and vectors and vectors that are are happening here what we want is a and it's saying now I am getting a see if I can get this to work real quick we might have to I think so basically what we want to do at some point it's just we want to we don't want to deal with these errors right now so I think the the easiest thing to do is just return this results here instead of instead of wrapping it up instead of saying that this is going to be a result here and that's actually how that actually works nice this this works because we have one less result to deal with I think whenever you're running into like types on types and types on types is it's nice to just break things up so that you you know how you're building a building things up here and so it bears repeating what does this what does this represent here and impact I don't know if if we do this what do we get we just why is it not Oh helps when you hit the right method yeah so this now if we if we don't do the it er dance we get a vector of results right so again just to just to step through the types real quick of what's happening results here is an iterator of where it's either a byte vector or an error and so when we call join all here results here is an I can just write it out is going to be a vector results of vector of you eights or error and so we've got vectors on vectors and vectors and stuff like that but this vector right here is bytes and we might have erred out getting those bytes and we're getting those bytes for many different things for each dependency that we have to get them for and so that's what's being returned down here is this vector of results and what we want instead is to say I don't want a vector of results what I want is a result of vectors so like can you please turn that inside out and it turns out that collect knows how to do that so collects can take a vector of results and turn it into a result of vectors and so you will notice that these are just this result in this vector are just swapped with one another and that's exactly what's happening here and then we're then we're in wrapping for now but that's fine I mean the last thing that that it will complain about here is that you know oh I you said you wanted to return a vector of two poles of the dependency and the bite and you're just returning back effector of the lights you know that's what we said up here we want the vector tuple dependencies and bytes so we need to I think the best thing to do here would be to turn this into Intuit earth here this from going from it or to into it or means that we will get our dependency now we'll get ownership of our dependency instead of getting a reference to our dependency so that's good and now we can return back our Depp along with our bytes and of course this is all that's that's wrong there and I'm just doing underscore to say figure it out because before we were saying hey you should find a vector of bytes here but it shouldn't it's a vector of these tuples so this underscore just says hey you figure it out oh okay this is by no means ideal right we're unwrapping all over the place and stuff like that that's that's fine for now we can definitely clean this up but what we have here which is great is we have a vector of our dependencies and like basically map together with the bytes that we've downloaded from from do get so let's even go ahead and run this and it's it's compiling because we've added some dependencies since the last time that we built so let me see if there's any questions here so can you explain why we need task block on yes so the reason we need task block on is because Cerf which is the library that we're using for HTTP only knows about async it only can work with async ap is but download dependencies here is a synchronous API when we call download dependencies we don't want it to return until we've downloaded every dependency why are we doing it that way well we because this is we just arbitrarily decided that that what one we picked an async HTTP library why we did that it's fashionable that's why and we get the ability to download our to easily download our dependencies in parallel with one another instead of having to set up like a thread pool or something like that to like spin off a bunch of threads to download on different threads we get that kind of relatively for free so that's good but because this is also this is the synchronous world but download dependencies is the synchronous world we we need to translate the aceing of this world into the synchronous world and that's what tasks belakang does it says given some asynchronous piece of work where things might be happening in parallel because everything is a scene for this inside of it block on this current thread until all that work is done so that's why we need times block on please let me know if if that helps and you know in a more a less helpful but more direct way to say it is we want to call dot await and you can't call dot away unless you have an asynchronous block and you can't run an asynchronous block inside of synchronous code unless you call block on so hopefully that helps and it looks like the chat was also mentioning about making main async as well we could do that but making main async there's a sink standard comes with a you know async main or whatever basically all it does is just generate code that calls test block on so it's the same thing to collect his way smarter yes it is doesn't this mean if something fails that you'll still load all the dependencies rather than failing fast yes yes we right now we will not feel fast which is probably a problem but mm yes we will we will try to download all the dependencies even if one of them failed only to later on then blow up because one failed not ideal but we can change that in the future so maybe I'm wrong but since you are not using tamil map you can delete the explicit dependency yes we can you can do that go here not curl lock and delete that cool good one ah yes there's a try join all maybe we should have used that instead but hey we at least got some good learning out of here about about collect so try join all instead of join all basically does the fail fast where if one of them fails it will just stop trying to get the rest of them which is definitely what we should do let's just change that real quick try join all and now this should be result of the vector like that and then we don't need this this whole shenanigans anymore good job chat cool so real quick it's 625 and evening my local time I planned on on finishing at 6:30 in the evening my local time so we're at the end here for me unfortunately I hope this was helpful weird by no means done what we have to do now of course is take all the bite that we've gotten and unzipped them and then we know and put them somewhere on disk in appropriate place and and do all that that jazz but at least we've gotten to a good place we've explored a lot of things we've explored a synchronous programming we've explored a CLI toolchain we've talked a lot about ownership and stuff like that I hope this was helpful to you please do let me know if you would like to see me conclude this on stream or if I should just go ahead and finish this on my own sometime but if you would like to see me finish this up on stream please let me know let me do my shameless plug real quick for my Twitter this is my Twitter twitter.com slash Ryan Levesque where i underscore Lauer please follow me if that's what you're into and that's where I not told my streams so that's pretty helpful or of course you can follow me on Twitch I hope that this was was good for you I'm we're gonna finish up I'll take any last remaining questions that you might have and then we will call it an evening for now so any more questions did you have to do anything special to get rust on Windows so I'm on I'm on a Mac right now that's it happens to be that what we're doing in this one in this exercise it doesn't matter what what operating system you're on and the future I probably will do some Windows specific witness specific programming rust works pretty well on Windows I use when I program on Windows with rust I usually do it in the native Windows environment with PowerShell or whatever using you know em SVC not and I don't use wsl for rust programming normally because you know if I want a UNIX thing Mac usually works fine for me unless I specifically want a Linux environment that I might might do it and it works quite well and if it doesn't then we should do something about it so let my let me know if you run into any issues there so we can see if we can get that trick somehow yeah great thank you so it looks like some people want to see how this ends so yeah in the future we'll probably go ahead and and stream it might be tomorrow because I really actually need to get this done so it might be tomorrow same ish time for another two hours or so and see if we can finish this up and yeah I will post this on youtube and appreciate everybody please let me know again if you have any feedback or anything like that of how I could improve this and otherwise yeah I really appreciate you coming out to watch so thanks thanks a bunch see you around

Channel: Ryan Levick

Views: 10,548

Rating: 4.939394 out of 5

Keywords:

Id: v1L91-rCiQs

Length: 115min 36sec (6936 seconds)

Published: Tue May 05 2020