Hello everyone, welcome back. It's been several weeks now of, or several months actually, where we've been working on implementing TCP and that's been a lot of fun, but I figured we want to sort of keep iterating through new topics as people get interested in them. And so if you recall, there's this website I use that I set up where, in fact, I live coded this website too. You can find it in an earlier stream. where people can vote for what things they would like to see me stream. In this particular instance, procedural macros was the thing that most people were excited about. It was a pretty neck and neck race between that and a second take of the open source contribution stream. So my guesses will end up doing that next. But for now, we're going to dive into procedural macros. So... For those of you who are not aware what procedural macros are, let's start just with what macros are. And to do that, I'm going to actually let's not do that. Let's start a new empty project here. Actually, how do I want to do this? Let's just start a vim terminal. File type, I can't type today. It's going to be a chat. File format. Fine. Let's do this instead. So while writing Rust code, you may have used macros before. Something like vec. So if I do this. So the exclamation mark here means that it's a macro. And what a macro does is essentially it is given the piece of code that you give it, and it returns some other code to run instead. In the case of vector, you could imagine the vec macro being something like the vec macro up here really being something like let's mute v is vec new. v.push1, v.push2, v.push3, v. So you could imagine that this is what the vec macro expands to. In reality, it's actually more optimized than this. You don't want to keep allocating, for example, as you grow it. So this is not actually what it expands to, but you can sort of imagine that this code ends up being turned into this code. And that is essentially what a macro is. Now, normally you can write macros using macro rules. So let's say that we wanted to define the vec macro. We could do something like... In this case... We probably want like eXper star comma. So this is an example of a normal vector, normal, I don't know, e. This is an example of how you would write a normal macro in Rust. So if you want to write it inline in the file. So you use this macro rules macro that is a macro that lets you write other macros. And you basically write pattern matching rules for the arguments to the macro. I think this might have to be parentheses. And then you write the body that that should expand to. In this case, the pattern is saying anything that's an expression separated by a comma, zero or more times. It expands into the following code. It's a block that generates this vec. And then this syntax is saying, write out the statement in between the parentheses as many times as they were in the input. So this is saying write out this line many times, and this macro we could use instead of the one from the standard library. You wouldn't want to, but this is sort of the way macros work in general. These are all very well in handy, but they're pretty restricted in what they can do. So you have to be able to write this kind of pattern. And sometimes you want fancier things. Like if you've ever used the Serti crate that lets you do serialization and deserialization, it allows you to do things like struct foo. It has a bar type use size. And you can do derive, serialize, deserialize. And what derive is, if we think about it, is really a way of taking some kind of code, annotating it in such a way that some program is going to run to generate the code needed for serialize and deserialize. And in fact, this turns out to be a macro. This is what we call a drive macro. So a drive macro is gonna take, it's gonna be given us input this business, and then it's gonna generate something like impulse serialized for foo, and then whatever stuff is necessary in here, right? There are also other type of macros you may have seen. So a macro like test. Right? So this is a macro that transforms this function into the appropriate format for a test. This is what we call an attribute macro. Usually you could also use these with parameters. So for example, if you've ever used a rocket to the sort of web framework, you could do stuff like route, get. slash, and now this is going to be generated as one of the routable parts of your application, where if you get a get request for slash, then this function would be evaluated. And so this is an attribute macro. All of these are the class of macros that we call procedural macros. So the idea of a procedural macro is that you write a program that's going to run at compile time, and it's going to be given essentially a stream of... tokens, so tokens in the sense are a bunch of sort of the building blocks of the Rust language. You're going to get a stream of them and you're going to produce a stream that's going to replace or add to the original stream. So someone asked here, how can I see the macro expansion? And there are actually many ways. What we're gonna be using today is, there's a crate called cargo-expand. I'll get back to this a little bit later, but basically there's this command that you can run on your own. So if you run cargo rustc and then with these arguments, which are nightly only, then the Rust compiler is gonna run and it's gonna basically print out with all the macros expanded. So here's one example where, There's a derived debug. And if you run cargo expand, you get the fully expanded, or you get one level of expansion for that code. Actually, you might get a fully expanded. These are sort of special. So that is one way to do it, and probably the one we'll end up using today. And this is going to be really handy for debugging procedural macros later. So procedural macros are, if we go back a little bit, they're effectively, so this is from the Rust reference, which actually might be less helpful than we want. But essentially, a procedural macro are mostly a conversion from token stream to token stream, right? There are some variations. So if you write a function. like macro, so that would be something like vec that we saw earlier, then it's just take a token stream and give the replacement stream. If you write a derived macro, then you're given the token stream of the thing to transform, and the output token stream is added to the same place as was, right? So if you're going to add an impl, for example, you don't want to replace the original struct, right? Because that struct still needs to be there, and the user has defined it, so you're not really going to be changing it, but you're going to be adding stuff. And then finally, attribute macros are things like route or tests that we saw before. And they're given the token stream of the attributes. So this would be the stuff that comes between the parentheses and the token stream of the thing that has been annotated. And they're supposed to generate a token stream that is, let's say here, the return token stream replaces an item with an arbitrary number of items. We're going to be talking about items a lot here. Items in Rust, in fact, this is the reference probably. Yeah, so items are basically anything. They're like modules, functions, type aliases, different types that are defined, implementations, traits, external blocks. Like think of them as basically anything with curly brackets. It's not quite that simple. But it can be simpler things too, like use statements are also items. And so procedural macros are basically a way of taking a stream of items and doing something with them and then producing a stream of items. We won't actually read through this document. I don't think that would be helpful for us. Instead, what we're going to do is we are going to follow the proc macro workshop. So this was essentially a workshop from the Rust Latim conference. And this is written by David, who is a really prolific Rust developer and has built many of the tools that we'll end up using today. So he's written... In particular, the syncrate, which is what we'll use to parse a token stream into something that we can modify. And the quote crate, which is the inverse. It takes essentially sort of an abstract syntax tree, like something that we're going to be modifying, and turns it into Rust code again. Basically, it turns it back into a token stream. And David wrote this as sort of a workshop guide with a bunch of exercises. for this conference. And so what I thought would do is basically try to work through some of the suggested projects here because they're basically built for people learning procedural macros. And I have not done them before. In fact, I haven't even written that many procedural macros. So this will be an interesting learning experience where I have less experience than what I usually do for these streams. Do input items have to be valid? Um, that's a good question. I believe the only requirement is that it lexes, but not necessarily that it parses. So all of the individual parts need to... Basically, you need to be able to generate a valid token stream from them. So you couldn't have an unterminated open double quotes, for example. But as long as you can parse it meaningfully, then you're given that stream of tokens. But what you produce, the token stream you produce, is what the compiler is going to be running on. And so that obviously has to be valid for the resulting program to compile. Yeah, but something like just an identifier, open curly brackets, and some comma-separated identifiers, which would not normally be valid rust is totally fine. In fact, the thing you give there is a valid pattern, I think. But that's sort of separate. OK, so we're going to be working through these. And in particular, we're going to start with essentially what this suggests we should start with, which is write derive builder. So for those of you who don't know what builders are, the builder pattern is essentially a way of constructing something that you're later going to build. So an example of this would be, and the example that's given here indeed, is imagine that you want to run some kind of command line utility from Rust. Well, When you do that, you're going to have to specify the name of the program. You're going to have to specify arguments to that program. You might want to specify where the program should run. You might want to specify whether the program should replace the current one or be started in a separate process. You might want to say where the input should come from, where the output should go. And all of these are arguments you may or may not want to set for any given execution. And so rather than have a constructor that takes like a million fields, you have a builder, which has defaults for basically most of the options. And then it has methods for overriding particular parts of that construct. So here, for example, you create a new builder, you dot executable returns a builder whose executable has been set to cargo. And so you can keep sort of layering on here, continuing to configure the command builder. So basically the command builder. you haven't run yet. And then eventually you call build and that's gonna actually produce the final type that you want to run or indeed run it depending a little bit on the setup. In this case, this is sort of trivial, but here the idea is that what this is gonna produce is a command, right? So our derived builder is gonna produce a command builder that has these methods. And then build is gonna give us one of these back with all the fields set. And so what we're going to start out with doing is we are going to just clone this, and that's where we're going to start. So we're going to clone this. Oops. Apparently, I just cannot type today. It seems less than ideal. All right. Great. So let's see what we have here. So for this, the skeleton is located under the builder directory. All right. So we're gonna cd it to builder. So this is essentially its own self-contained crate that we're going to be solving this in. The cargo.toml has already been set up basically correctly. So you'll notice that for lib, it's set to proc macro equals true. So this is what we want, right? We're writing a library that's going to be a procedural macro, and you need to communicate this to cargo so that it knows to essentially compile it in a special mode where it has access to those things. For example, you don't require a main in a procedural macro. What is also neat about this is this also comes with a bunch of tests that we can use. So if we look at the test directory, let's look at progress first. So this uses a crate called try build, which is handy for writing procedural macros. It basically tries to build the given source code. and then checks that the compiler output is what you expect. So if you expect it to compile, that's all well and fine, try build will do that, but you might expect it to fail with a particular error message. So you might wanna check that, for example, the compiler highlights the right part of the code if something doesn't work out. And we'll see some examples of that later. In particular, the first test we wanna pass is the parse test. Right? So it looks for a derived macro with the right name, right? So we're going to have derived builder. It doesn't require any specific code to be generated. So really all that matters is to see that we can parse it correctly. Why did the chat go away? Weird. Right, so the only goal here really is that we can parse the input, but we don't have to produce anything. In particular, what it's telling us to do here is use the Syncrate to parse. Well, let's take a look at the Syncrate. So the Syncrate, as it says, is a parser for Rust source code. And essentially what it does is it gives you a parse macro input. which lets you parse, well, a token stream into a syntax tree. And particularly, it generates this kind of derived input business. So let's take a look at, no, that's not what I want. Let's take a look at the docs. So down here, where is my parse macro input? Great. So that should give us a derived input. So in particular, what we want here is we want to parse the token stream we're given as a derived input. So derived input here is the input to a derived macro. And you'll see that what we get back is something that has attrs. So attrs are any other attributes specified on the struct that we're deriving for or the item we're... we're deriving for, the visibility of that struct, right, the struct might be public, it might be create only, whatever, ident, which is the name of the struct, generics, which are any generics on the struct itself, and data, which are the contents of whatever data type that we're deriving over. And so in our case, the test here is basically just a test that derive builder on command, produces code that compiles, which would include adding no code, which is indeed what we're going to do. So here, what we would expect is, given this, we would expect adders to be empty, because there are no other attributes on the struct. We would expect visibility to be pub. We would expect ident to be command. We would expect generics to be empty. And we would expect data to contain these fields. So if we look at what data is, we So data can either be a struct, an enum or a union. In our case, we're expecting it to be a struct. And so it's gonna have a struct token, which is just struct. It's gonna have a bunch of fields, which are gonna be these. And it's gonna have a semicolon token. It may or may not have a semicolon token. In this case, it does not. So the fields are gonna be named. So unnamed fields would be like a tuple struct. Then the fields do not have names, they're just numbered. So we're expecting to get fields named. And so here we're going to have a brace token, which is going to be this. And then we're going to have the named fields. So punctuated here just means that it's comma separated values. So this is going to be an iterator that gives us comma separated fields. Every field in here, again, has all of these. In particular, executable, for example, it has no attributes. It doesn't have any... pound square bracket above it. Its visibility should be private. Its ident should be sum executable. So it's an option because it might not have a name. Colon token is, well, a colon. And tie is going to be the type, which in this case is going to be string. So you can see that this, going all the way back up, this derived input basically gives us access to the entire thing that's been defined. So let's try passing the first test. So we're going to start out by uncommenting this, and then run cargo test. And see what that gives us. Apparently, I'm lying. What's the new rear? Oh, that's probably because I'm in the wrong directory. Here, cargo test. The other thing we're going to need here is for the builder, we're going to have to depend on the SYN crate, which is currently version 0.15. So let's see what that gives us. Okay, so great. So it ran zero one parse. It failed because the proc macro derived panicked in particular because we haven't defined any derived macro called builder. So we're gonna have to define one. Great, so going back to the source for this crate, right? Here, what we really hit here was we hit this not yet implemented, right? So this has already been set up for us so that it sort of defines the basic setup that we need for proc macros, which is we need external crate proc macros. So this is a special crate that the compiler exposes to us that gives us access to this token stream. type, which is what the compiler produces to macros and expect macros to produce. And then the proc macro, when you write a proc macro, you also have access to three attributes for functions. It's proc macro, proc macro derive, and then proc macro attr, which is to define the three different types of procedural macros that we talked about earlier. In this case, we want a derived macro, so we're going to write proc macro derive, and we're going to give it a name, which is the name that the user is going to put inside of the the derived statement on their struct or enum or whatever. This is gonna be a public function cause it's gonna be essentially called externally. And it has the signature that we expect, right? So the input is a token stream. So this is gonna be our struct and the output is a token stream that we're gonna produce. And in our case, what we want to do here is we want to parse, let's call this an AST. I don't actually need this anymore, I think. Because 2018 edition, maybe? Great. I guess we'll find out. So what we're gonna do is we're gonna run something along the lines of what was given in the example down here. This. This obviously means that we're gonna have to use parse macro input. So let AST is going to be equal to this. Notice that there's no result return type here. The reason for this is procedural macros are expected to always produce a token stream, and a panic is considered a failure to compile. And so you really just want to panic if something goes wrong. And what the compiler is going to do if you panic is, and we saw this back here, it's going to highlight a span that's contained in the error you produce in the panic, basically the span there was processing at the time. And then the message it's going to give you is the panic message from the procedural macros. Um, okay, so this is going to give us the AST and let's just print out the AST. Uh, and then what we want to return is an empty token stream. I think there's a empty on there. See what that gives us. Okay. So we can't do that. Um, Oh, do I actually need to, uh, that's a little weird, but okay. and we need to use drive input. Okay, so we're gonna have to look at the proc macro here. All right, so proc macro token stream. Oh, it's just new, which gives you empty. Okay, so instead of empty, it's gonna be new. And I think this implements display, maybe. Isn't there a nice way to... Debugging, I feel like you're supposed to, oh, you're able to print token streams. Okay, that's fine. I'm surprised though, this doesn't implement debug. It does. What are you on about? Why can't I do that? Derive input does not implement debug. Pretty sure it's supposed to implement debug. Send drive input cannot be formatted. Huh. How weird. Are we not building sin with derive? Which, oh, so it has a bunch, so sin has a bunch of features. I guess the real question is what features are on by default? Derive is enabled by default. Ah, extra traits. Okay, so back in cargo Toml, we're gonna do version is this and we're gonna say features, add this on and then in theory that should give us debug for these types. Oh, it might even say in the debug tips, the product macro workshop. See what it says. Um, yeah, so cargo expand, I already know about, um, yeah, see, we found it ourselves. Um, uh, we could add pretty printing, but you know, Great, okay, so here we now pass the first test because we're, basically because we removed the unimplemented, right? That's the real reason. Previously it was crashing, now it's no longer crashing, but that also means that we're parsing the input. So let's make this a little bit nicer for us. Let's use pound question mark. Right, so here we get to see the full parsed syntax tree for what we got. So let's see if it matches what we expected. So there are no attributes. Remember this is zero in parse. So it's this struct that we're comparing against. So there are no attributes, it's public, the ident is command, there are no generics. Data is a data struct, it's struct token is struct, its fields are named and the fields are executable. Um, which has... So types are specified with paths. In this case, string is just the final part of the path. There's no preceding path, which is why the path has no leading colon. And it only has one segment, and that is string. The next field is a vector of strings. So we're going to see, yeah, so comma field is called args. This is a, so this also only has one path segment. A vector is the ident of the type, and the type also takes arguments. So if you look up here, the arguments for string were none. The arguments for the vector is deep down there, string. So clearly our type is being parsed correctly. We don't use any of the fields. Like we don't actually do anything with any of this, but at least now we have the type. All right, so let's look at the second test. So we want to have the macro produce a struct for the builder state and a builder function that creates an empty instance of the builder. As a quick start, try generating the following code, but make sure the type name matches what's in the caller's input. Right, so here, we want a builder function on command that returns the builder, and we're gonna have to define a new struct called builder. So we wanted to generate this business. Great. Yeah, so if you look at what the test actually does, it expects there to be a builder method on command, which of course there isn't by default. And then it just sees that that returns the thing it can represent. The resources down here mentions the quote crate. We talked about this a little bit early on. Quote is basically a crate that lets you go the other direction. SYN lets you go from a token stream into the tree, and then quote lets you take a tree that you can specify in a convenient way and turn it into a token stream again, roughly. That's what we're going to use to go the other way. In particular, if we go back to our cargo toml, we're going to see that we're going to have here, we're gonna have quote version 06. It's to do any more. And so now we're also going to use quote, quote. Right. So what we want to do here is the quote macro lets us use this it lets us write, it's sort of like a templating language almost. It lets us basically write Rust code and then interpolate things in there. So for example, the name we wanted to use here, Is it just in ident? So we want to generate an impl for command, right? But we need to know the name command. So where does that come from? Well, we know the derive input has an ident that is the command, right? So we can just do this, which is gonna be the name. So we're gonna impl, so pound is the way you signify replacement of some variable. This is going to generate a builder function. It's going to return some type that we don't really know yet. And who knows what's going to go here. So we're going to need some kind of struct here that's going to be the builder. But we don't just want to call it builder. Because if we just called it builder, then you couldn't have derive builder for two different structs in the same file. Because they would generate the same builder type. So we actually need to generate a unique name for this. So how do we do that? Well, there are a couple of different ways. The easiest way is to generate a new ident. So if you look over here and look at... So an ident is sort of a name in Rust. Like a variable would have an ident, a type would be an ident. In our case, we want to create a new ident that's the concatenation of two existing ones or of some strings that we want to combine. In particular, what we want is, let's call it builder name. So it's going to be, we can just, we're just writing a normal Rust program, right? So we can just do this. Uh, and give in the, uh, name. Can I just do that? Or do I have to do something fancy? Uh, this implements display. Great. Um, so that's going to be the name. And the B identifier is going to be syn ident new. We're going to give in the B name. But ident takes a second argument. And the second argument is a span. So if you look at new, span here. So spans are basically parts of the original input file that we want to be able to refer to. So in this case, if something goes wrong with creating this name, we want the compiler to highlight the original name that caused us to build this string. So for example, imagine that the user wrote something like struct this, which is not a valid name for a struct. Then our builder that we're going to generate, let's make this simpler. our builder would generate this, right? Which is also not a valid struct name. And what we want is for the compiler, when it tries to compile this, to point at this being the error. And what that means is this identifier here, we want its span to be this. So that the compiler knows to point at that when it talks about this. Does that roughly make sense? So here, we're going to take the span of the original name and use that as the ident span for this. So now this can generate. By default. Right? So currently, it generates no fields, but that's fine. This test does not require us to add any fields. We might add some later, though. So let's see what this gives us. So it doesn't compile because do I need to do this in two steps probably? Expanded equals this. Expanded. So what quote gives you back is not quite a token stream. It returns some type that we can turn into a token stream, hence the into. Expected str, that's fine. Oh, I wish this took into str. Oh, well. All right, so let's see what this gives us. OK, so that compiles. But let's, oh, I guess we need to actually turn on the test as well. Great, so now we pass both tests. But let's take a look at what this actually generated. So if I run cargo expand, it's going to give me the expanded source code. Let's do it just for test. Oh, two, what's the syntax for this? Oh, I see, sure. So I guess we're looking at main and we're looking at builder O two. So let's do cargo expand, see what that generates for us. Oh, right, not there, here. All right, so this is what gets generated. Well, this looks pretty sensible, right? So this is the original struct, the one that had the derive on it. And we generate struct command builder. And we generate impl command. So this actually generates what we expect. Let's try to see what happens if we put something weird in here. Like if in our main, we try to do what we talked about earlier and make this 777. What now happens? Oh, expected identifier. Is there a way I can make this not work? It might not be. I wanted to see whether the spans aligned correctly, but I don't think there's an iOS way for us to test this just quickly. OK, so if you remember back to the test, it asks us to. Once we generate all of this to actually generate this struct. So that's what we'll do. We'll go back to our source. And instead of generating an empty one, we're going to generate this one. Great. And this they want to generate. Actually, let's do these. Great. So now let's just see that that actually works correctly. So we still pass the test. Test 0 passes, Test 2 passes. Time to move on to Test 3. call setters. So now we want to generate methods on the builder for setting the different values. So this is what we talked about before, right? When you have a builder, you want to be able to do dot arg and give one argument and have that be added to the list of arguments that are currently there, or like dot executable to set the executable, and so forth. So here, you'll see that it still has this derived builder. And it now with the builder gets back, it's expecting to be able to call these and have them do the appropriate thing. So they give an example here of the way this method should look like is fn and then the name. This should be the signature. And the method should return the builder itself so you can keep building with it. This is basically so that you get to chain calls for a builder, which is sometimes handy. And all it's really going to do is it's going to change the appropriate attribute and then return self. And of course, now in the very beginning, what we can do. is just generate each of these manually. I don't know what the later tests do, but it could be that eventually we want to generate these dynamically based on the fields, right? Because this macro is kind of useless if it requires these to be the four fields, right? We want it to be able to handle any set of fields. In fact, let's just sort of peek at what, optional field repeating field. Let's look at like seven. Okay, so these are sort of different. It might be that we just wanna do this, the straightforward way first and then expand to other fields later. Okay, so for the third test, we wanna generate a bunch of methods. In particular, we want to imple bident and then we want methods like this. Right? And we want, I guess we want them to be pub. We want args, I guess, what does the test test for? Executable args and then current error. And string and current error. should be a string. Right, so let's see what that does. Great, so we still pass all the tests. Now we're seeing this is becoming really verbose, right? Because we're repeating the fields from the original struct multiple times over. And this is really what we want our macro to do for us, right? We don't want to have to write these out because then the derive isn't particularly useful. And so we basically need at some point to write a thing that's gonna walk all the fields of the struct. So let's look at form. Call build. Okay, so we now want to generate the final build method that's gonna go from a command builder to a command. And this should, let's see, this method should require that every one of the fields has been explicitly set. It should return an error if a field is missing. The precise error type is not important. Consider using boxed in error, which you can construct using the input from string for boxed in error. Okay, sure. So now we're going to generate the build method as well. Uh, and this is of course not going to return a command. It's going to return name, right? Because in theory, at least we're generic over or not generic over, but we're abstracting over what the type of the word derived from is, um, and so here, um, it's going to be something like if let, if a self dot executable dot is none. Um, Then return. Why do we want to do that? Oh, great. So we can just do executable is not set to. And here again, we see this repetition that we don't really want that we'll eventually fix by instead just looping over the fields, which you remember we have in the parsed AST. Just checking is the chat still working, because it froze for a second there. And of course, otherwise here, if all the fields are, ooh, in fact, you can do this even nicer, but let's ignore that for now. So this is gonna be name, this executable is gonna be self executable. Ooh, in fact, here's a better way to do this. OK, or is not set. I think that should work. which is much nicer. And there, and args, instead of having all these ifs. See what that gives us. Recursion limit while expanding the macro stringify. Ooh. That's interesting. Does it say anything about recursion limits? No. Huh. Also, what is happening with the chat? Just checking that this is. Huh, weird. Oh. All right. Oh, I guess we get out of that. That seems fine. Oh, we need to actually use that. Yeah, that's fine. As long as the stream is still working. I didn't see anything either. Thanks, though. So here, this error is actually kind of interesting. So if you look up here, you see that it says error was not in scope back when we had this. And the reason for this is in generated macro code, you can't really rely on something being in scope. Like you can't rely on the user having written use error somewhere. And so you really need to use full paths for things or explicitly use them yourself. Otherwise, it just wouldn't work. It's still failing though. Cannot move out of borrowed content. Oh, I see. This is going to have to do like a slow. Because we only have a mutable reference to self here. And so we can't move out of these fields. We need to clone out the string. Great. Okay, so that passes. Let's see whether the next one. Oh, we did four. So five. Should be a freebie. Great. So five should just work. Great. Can't you use the trait at the impl level? Use the trait at the impl level. Yeah, so I mean, you could, in theory, use the trait here. My point is more that you can't rely on the user having imported the thing that you're naming. So whenever you're naming a type that's sort of external to you, you need to explicitly give out its path or explicitly use it. Um, okay. So six optional fields. Okay, here's what I want to do. Before we do this one, I want us to fix up this one. So instead of doing this one yet, what we're first gonna do is have this no longer be specific to command. I want us to actually iterate over the fields, right? This is, it doesn't explicitly say this in the instructions, but we can do whatever we want, we're learning. Okay, so let's think about this. Well, so quote, if you take a look at it, Has, where is it? Oh. Right, so it has the ability to do repetition. This is similar to the dollar parentheses star that we saw for the macro rules in the very beginning. And this lets us basically evaluate an expression multiple times, as many times as the variable that is used inside appears. In particular, as long as the variable is something that is iterable, it can do the appropriate thing. So what do we wanna do here? Well, What we want to do is we want to iterate over all the fields of the original thing and wrap the type in option. Right? So how's that going to work? Well, remember how we got the syntax tree way back up here, right? So really what we wanna do is we want to walk the fields of the data struct, and then we really just wanna replicate them in our own, but we wanna wrap the type in an option. So that's gonna be something like, I guess, let AST or let fields is gonna be, ast.data if let sumData struct is this else. unimplemented. So keep in mind that the user could, in theory, put a derive builder on an enum, for example, instead. And that is something that we currently just do not support. So we're going to assume that the derive is on a struct. And here you could imagine that we should give better error messages. I don't know whether that's a later exercise, but you could imagine if the user puts like derive builder on an enum, we really want to highlight the enum part and say only works for structs. All right, so the fields are going to be fields. And we really want to do... if let actually, it can be even better here. So the struct contains a data struct, which contains a fields, we don't care about the other things. Fields, we're expecting to be a sin I'm going back here. So where's the data? Should be under D. Data struct. So a data struct has a fields. And so this is going to be a syn fields. And we're expecting it to be named. Right? And that contains a fields named. Fields named contains a bunch of other fields, and the one we care about is ref named. So that's no joke. Right? And then. Let's take a look at what happens if here we put fields. Don't actually know what this will do. Actually, that's probably going to not work in all sorts of ways. But if we go back to here, let's see what this expands to now. Oh, I guess if let's, oh, did I do, oh, here. This should just say named. Let's do our cargo expat. Data structure is not, oh, this should say fields. Okay, so this now expands to command builder and notice that it now auto filled out all of the fields. All the fields are here. Um, there are twice because we wrote them out manually as well. Right? So we both have this hash fields and these, but what we really want here is when we're printing them out down here, we want to print them out, um, essentially the same as they were, but with an option around the type that they contain. Right. So named here is in fact an iterator. Um, if we look back at. here. So named is punctuated. And a punctuated if I remember correctly, influence into iterator. Does it also implement? Great. Yeah, it are. So really, what we want to do here is down here, we're going to do expansion of fields. Right. So let's see what this will do. In theory, I think this should generate basically the same thing. Maybe this way. Great. Yeah, so this now generates the same thing. And the reason for this is this pound open bracket star, what that means is call into iterator on the variable that's used inside, iterate over it, and for every element, run the thing inside with the variable set to the item you've got. And in this case, fields is the iterator over all of the fields. And so for each field in fields, we're printing out that field followed by a comma, which is the same thing that you would normally generate. Of course, this is not actually what we want to generate. Optionized is going to be fields.iter.map. And then this is going to be optionized. Right? Because really what we want to do with these, as I mentioned, is replace the type with the same type, but wrapped in option. So going back up to where we got the AST. So we want to leave all the other things, but we might want to set the visibility manually. In fact, maybe we just want to create our own field, come to think of it. That's a good question. We might want to do that. Because we don't want to inherit things like attributes on that field, the visibility of that field, or colon type. Like we don't want to inherit any other things. We just want the ident and the type. So let's manually construct a field instead. So we're going to map the fields that we get, we're going to map them to sin field. So let's look at what sin field gives us. So a sin field field. Oh, we can just construct one great. So we want to generate a field whose adders is an empty vector. whose visibility is a sin visibility. public. Actually, no, just inherited is fine. The ident should be the same as the one in the original field. The colon token is going to be Same as the original one, I guess. Seems fine. And the type, this is where we don't actually want this to be the original. But let's just see that it expands to the thing we expect. Do I need to clone this probably? Great, so this still generates the same thing, but now we have a starting place for actually modifying these fields as we desire. In particular, down here, we want to create sort of an option-wrapped type. So f.tie.clone. So this is the original type. and we're going to be modifying that type to wrap in an Option. If we look at something like Vec, what that really does is it's So Vec contains a T, and we want to generate Option that contains a T where the T is the original field type. So we basically want the same kind of wrapping as is applied to Vec. So looking at this So what we really need to generate is the same structure here. It's going to be a path. It's going to have a single path segment. The ident here is going to be option. And the arguments are going to be basically the same as for vec. But the inner args here is going to be a vector of the original type. So you'll notice here that here it starts with path, this. And that is the same as this path. Everything below sort of nests. So we're really just gonna nest that inside the option. So let's see how this is gonna look. Let's get part of the stream. This is based on David Tolney's PROC macro workshop. Let's see. So we're going to have to generate, this is like original type. And then our type is going to be a synpath. What is a synpath, though? Synpath, which contains, ooh, leading colon. That's interesting. Oh, I see. Sure. It does not have a leading colon. So it's going to be this leading colon is going to be none. segments is going to be Punctuated path segments. Okay, so how do I construct a punctuated? New, great. So let segments is gonna be syn punctuated. So punctuated here just means an iterator that's separated by some punctuating character. In this case, comma, colon for a path. punctuated, punctuated, new. All right. And then here we're going to segments.push. There's only going to be a single segment here, and that's going to be Option. That's the only type we actually want in here. So remembering back to Vec, the original thing is a... What's the difference between this path and this path, though? I want to see. Oh, it's a type, isn't it? Yeah. Interesting. So we actually need to generate something slightly higher, which is going to be a type path. which is going to include a path, a type path. So this is going to be a syn type path, which contains a Q self. I don't know what Q self is here. But it is none for the VEX, so it will be none for us. And then it's going to contain a path, and that's going to be this path. So for the punctuated one that we had, oh, man. Right, so we're generating, remember, we're generating the segment to the option type. So this is the, let's see if I can explain this better. So look at a type like this one. What is the type of args? Well, the type of args is a segmented path to a type. Logically, it's something like a, b, c, that. So the segments of this path are a, b, c, and option, including its parameters after. Given that we just want to generate option, we want a path with a single segment. And that segment is going to be option of the type that the user gave us. So hence, we create a new segment. We're going to push a value. So remember, a punctuated stream has delimiters as well. We don't want to push any delimiters. We only want to push the value. And in particular, the value we're going to push is a path segment. All right, so what is a path segment? Oh, here, great. Fine. So we're going to push a sin path segment. Path segment has an identifier. Now we're getting somewhere. And the identifier is going to be the identifier of the original type. Original type dot ident. Actually, is that even true? Original path dot ident. Yeah, great. Should force it to standard option option? Yeah, arguably, we could do that. I mean, it's pretty easy to do that, so we can do that. But I'm going to not do it right now. So let's do a to do use standard option option, not just option. So the reason for this is you could imagine that the user defines their own type named option, and our code is currently written would end up using that option type, which is not really what we want here. We want to use the one from standard option. And this is why, really, the macro should generate this path. And then the path segment also contains arguments. So the argument is going to be, wait, I'm lying here. This should be option. This should not be that. This should be syn ident new option. And what span should this have? I don't actually know how to give this a span. Oh, you're right. We can just use quote directly. That's a good point. That is far better. I totally agree with you. That is far better. I did not think of that. So this is just going to be, quote, good thinking. So this is going to be, quote, name is going to be f.tie. ident and tie is going to be f.tie.tie. Actually, I'm lying. It should be... ident, this should be tie. You're right, that is far easier. Because we can get rid of all of that and do this instead, and say name, colon. Now we can even do this pretty easily. I love quote, forgot how much I like quote. So let's see how that works. Great. That was far easier. You're totally right. Thanks. So now we see it actually generates all the fields for us. And now we don't need these manually added fields. They are now just noise. And you can imagine that we can do the same thing down here. So for the methods, it's going to be the same kind of thing. It's going to be quote, a hub FN name of the field, take mute self name, colon, chai mute self self dot name is some name self. And now this can just be methods. Let's see if that works. What does this expand to? Oh, did I lie? Probably lied. Yeah, we need to extract. So this now expands to great, so you get all the same methods. Right? Beautiful. Almost as though it was planned. And what else do we want? Well, in build, we also want to generate these. So let's generate them. Clone to take in build. So, okay, I'll address that when we get to it. It's a good question. Okay, so we're going to generate this build method, and it's going to have all these lines. And in particular, it's going to do something like name self.name.clone, okay, or. We're going to have to figure out what should go here. I think what we want is stringify name. But can I? Oh, that's a good question. Is there a way for me to join these strings, I wonder? Isn't there a concat? I forget this. concatenates literals. Great. So I want concat. This with is not set. And now this is going to be same thing again, build fields, comma separated. Okay, so one question was raised here in chat. Um, here cargo expand. Uh, what did I do? Two token is not, uh, this has to be a star and this doesn't need the tough. Okay, so this now generates the right build. And there's a question from chat, which is, instead of this being clone, should it be take? Right, so option is a way where you can take the value that's in the option and leave a none. That way you don't have to clone it. The reason we want this to be clone, not take, is because if it's take, you couldn't reuse the builder without resetting the fields. And that would be like, it would be fine. You could still use the Builder, but it would be unintuitive. Arguably here, Build should just take self. In fact, that's something we can fix. Build should just take self to sort of document the fact that it doesn't change the Builder. And now hopefully all the same tests will pass. Ooh, not so much. Why? Expected type string. Found result. Where? Interesting. Missing question mark in build. Oh, yeah, you're right. This should have a question mark. You are totally right. Good eye. Beautiful. OK, so now we pass all the tests, but we no longer have any of this repetition. I guess, actually, we need empty fields as well. Empty is going to be none. So this is going to be build empty, comma separated with star. OK, so at this point, what did I do? What did I do? Respect that identifier. I should remove this comma. This should not have that. Right, so now we pass all the tests without specializing to the particular fields that were being given. In theory, it now works across any fields that are given. Beautiful. All right, so now we can try our hands at the next test. So optional fields, let's look at this. Some fields may not always need to be specified. So currently, remember, we require that all of the fields are set, right? We do this like unwrap or business. And specifically, something that's an option T in the original struct shouldn't need to be set later on. Have your macro identify fields in the macro input whose type is option and make the corresponding builder method optional for the caller. In the test case below, current is optional and not passed to one of the builders. So currentDir is option. And so building one where you don't call currentDir should not be wrong. It should allow you to have that be set to none. Let's see, be aware that the Rust compiler performs name resolution only after macro expansion is finished completely. That means during the evaluation of a procedural macro, types do not exist yet, only tokens. In general, many different token representations may end up referring to the same type. For example, option and standard option option and vec option t as into iterator item are all different names for the same type. Conversely, a single token representation may end up referring to many different types in different places. For example, in the meaning of error, will depend on whether the surrounding scope is imported standard error error or standard IO error. As a consequence, it isn't possible in general for a macro to compare two token representations and tell whether they refer to the same type. In the context of the current test case, all of this means that there isn't some compiler representation of option that our macro can compare fields against to find out whether they refer to the eventual option type after name resolution. Instead, all we get to look at are the tokens for how the users describe the type in their code. By necessity, the macro will look for fields whose type is written literally as option and will not realize when the same type has been written in some different way. The syntax tree for types parsed from tokens is somewhat complicated. So this is what we've already dug into this, right? Which we're gonna get into in this question. So here's the nested, this is the same thing that we saw. And in particular, what we're really looking for is this business, right? We want to look for somewhere where the identifier for the type is option. Um, all right. So the only place where this really comes up is for build. So in build fields down here, we don't want this to fail for fields that are optional. Right? So we want to do something like, So the type is going to be and we're going to have to sort of basically dig down into the type hierarchy for this field. So this is going to be f dot tie, which is a type dot we're going to assume that it's a type of let soon type is f.tie. So p now is a path. And a path, we remember, have the fields qself and so a path contains a tie path, which has a field path, which has segments. which has to have at least one segment. So here, we can do something like if segments iter last unwrap, because we know there's at least one element there. So that will give us a path segment. Actually, let's do if p.path.segments.len is equal to 1. And the last ones, ident is equal to option, we probably won't be able to do the equality this way. But we'll see. So if that's the case, then we want this. Otherwise, we're going to do the standard thing, then we just want this. Right, so I think we always know that this is a path. No, that's not true. They could inline a type. But if it's a path and that path does not have any colons in it, basically, and I guess at this point, this can then be. zero. And if the identifier of that one path segment is option, then we know that they're just option types, so we don't need to do this unwrapping. There's a separate observation here, which is if it is an option, we don't need to store it internally as option option. So that's the other thing we might want to do, which is up here, we could do let's just do a convenience tie is option, which takes a tie. Actually, we can do this even. Right. Takes a field. And it's going to do this. It's going to tidy up our code a bunch. Excellent. Now you could imagine us optimizing this for like if they wrote standard option option, it would also work. But we're gonna ignore that for now. And now down here, we can say, if is option for fields, then this else this The reason I wanted to do that is because we want to do this for the fields as well, so that we don't end up storing an option option, which is just unnecessary. Because currently we're just unconditionally wrapping an option. So we want to do if tie is option. then leave off the option part. And similarly, for the methods, here, we want F ties option. Otherwise, this. then just set the field, right? So this will still be option because that was the underlying type, but now because the inner type does not contain an option, we're still good. Let's see what this generates. Do I really? So this is gonna be a syn fields. So if we look now at current there, oh, this example does not actually have that. So I guess let's take 06, which had an optional field. So, and then change main to have this instead. And now do a cargo expand. So now, currentDir is now stored just as an option. Notice it doesn't double capture the option. It just uses the option type from the original code. And currentDir just takes an option, sets self to be that, and currentDir does not unwrap. And so let's see if we now pass 6. Ooh, we fail six. Oh, they're expecting currentDir to not even take an option. That's interesting. Oh, that's very interesting. So specifically, if we look at 06, notice that when they do give current error, they're expecting this to just take a string argument, even though the underlying type is option string, which would mean that there is no way to unset current error, but I guess it makes sense for ergonomics. So this is gonna require us to do a little bit of trickery because we basically need to, remove the option, the wrapping option type, which is a little more annoying. So this is gonna be let unwrap option T, which is gonna take a type, Uh, SYNTYPE. It's going to give you a SYNTYPE. Uh-huh. Assert this. This is not, shouldn't be reachable because this should only be called if this is true. I guess we can instead just do assert tie as option. And I guess this should take a type, shouldn't it? So this is going to be type. This is going to be type. And this is going to be type. So we're going to assert that tie is indeed an option. And then in here, if you recall back to, do we have a convenient way to get at the expansion? Down here. So the actual type that's contained in an option is hidden down in this arguments under args. So we really want to extract the type that's in here. So how do we want to do that? Inner tie is going to be p. So p is the path, path, segments, 0. arguments. So what is the type of arguments? Path arguments. We're going to assume that that is this. Path arguments this. And otherwise, here, we can actually give a helpful error message, which is like, um, which is something along the lines of unimplement. It's really a panic. It's a panic saying, um, option did not can option type. was not option T. Right, because this means that it's taking some other arguments, or it has no arguments. So arguably, this should be a part of this assertion. So I guess and if we really want this, though, Actually, how about this just returns a reference to the inner type? That is much nicer. That way we don't need to write this multiple times. Yeah, let's do that instead. That's much nicer. Is some is some and is some. All right, so if it's angle bracketed, then this should give us this business. So let's look back here. That gives us one of these, which is, so this contains the full syntax tree, which is why it needs to have all these surrounding tokens and stuff as well. All we really care about is args. We don't care about the surroundings. So we're going to do inner tie. dot args. So those are going to be the generic arguments to that type. And then that's going to be another iterator over generic arguments. And we expect that to only be a single type. If I just punctuate it, does that give me a length? It does. Great. So if Length is equal to 1. We could also invert some of these. So if this return none, it makes the code a little less indented. So here we know that we have exactly one argument within there. If let there's a first or something for this as well, right? For punctuated first, great. So if let Actually, no, it's not even that, is it? Punctuated first gives me an option pair. Okay, so innerTie.args.first.unwrap. So we can unwrap because we've already checked that the length is one. And that gives us one of these pairs. And in particular, what do I want from the pair? I want value. If let, right. So what that is going to give us is one of these. And generic argument type. if t is equal to this inner type, then return some of that t. Great, so this should give us the inner option type, which should mean that, well, so here in theory, we could rewrap it in option. I don't think that's important. But it does mean that down here, if let sum inner tie is this, then this should take inner tie and set it to sum that. Where? 75. Ooh, did not like that. Oh, we can just make this a normal function. Takes one of these, and it returns one of those without a semical. type. 20. Oh, wait. Why is that not okay? Oh. I'm going to go back to the previous one. Fine, borrow checker, borrow value. That's fine. What does it expand to? That's what I'm more curious about. So now it still generates option string. That's fine. Current error now takes a string. So that is what we wanted. And this seems to generate the right thing. So why is it complaining? It is complaining that it expected a vec. Oh, are we extracting the vec as well? Yeah, so for args, notice that we're turning args into string as well, even though that is a vec, not an option. So really what we want here is if the segment is not 1 or if it's not option, that's what it really should be. Yeah, so now the only thing that is kept option is currentDir because it already was an option. All the other things retain their inner type. Although this is showing that our type extraction is working for Beck as well. And currentDir is a string. Beautiful. Now let's see if we pass the tests. It's the real question. Hey, look at that. Excellent. Okay, so we now pass the first six tests. Time for number seven. Yeah, so command here is based on, in the standard library, there's also a command builder, basically. But when you pass many arguments, you sometimes just want to add one argument. You don't want to replace all of them. So in particular, you want to do things like this. You want to give this as an arg and this as an arg. Currently, we can't really do that. In particular, what this is asking is let's add an attribute to fields that have fields that are iterable, basically, or that implement extend, and generate an appropriate method if they have this field attribute tag. So that this is really saying, add a builder method arg that when called extends this thing with the given argument. The generated code may assume the fields with this attribute have the type vec, and should use the word given in the string literal as the name for the corresponding builder method. Right, so this would be called arg, this would be called n. In order for the compiler to know that these builder attributes are associated with your macro, they must be declared at the entry point of the derived macro. Otherwise, the compiler will report them as unrecognized attributes and refuse to compile the caller's code. Okay, so this means that up here, we're going to have to say that this also takes attributes builder. Attributes builder. These are called inert attributes. They do not correspond to a macro invocation on their own. They're simply looked at by other macro invocations. Great. OK, so this means that for methods, we also want a different one. We want extend methods, which is going to be basically if So looking back at what we get back from a data struct, fields, fields named, punctuated field. So remember that fields have this attrs, which are other attributes on that field, and it's a vec of attributes. OK, so if field.atters, if not field.atters.isEmpty, Then we're just going to sort of print out f adders and just see what it is. I don't know if this will work. I just want to quote an empty thing just to retain the type. And then I want 07. That's a good word cargo expand gives us. Interesting. But how do I? Oh, right. I don't actually invoke it. Do I? Extend methods should go here. Right, so it did find some attributes, right? So there are two things in our original input that have this builder attribute on them. So we would expect to see two attributes that are non-empty and they're indeed both here. And what we want to look for is we want to look for ones where the path, which is the thing, so if you look at it here, the attribute has some like symbols and stuff, and then it has a path, which is the name of the attribute, and then it has a token stream, which is the tokens contained within that attribute. So in this case, the path is builder. So recall this is the pattern we're looking at. So the path is builder, and then the token stream inside is going to be each equals env. So if we look at here, we get a group delimited by parentheses. So those are the arguments. And the stream inside is there's an ident each. It's punctuated by an equal. Then there's a punctuation equals. And then there's a literal, which is env, the string env. So we have all the information that we need here. And so let's use it. So what do we wanna do here? Well, first of all, we wanna, we probably want to do, this has to be a filter map, right? Because we don't wanna emit any methods. Oops, that's not the right place, is it? Here. Let's space these out a little. So if it's empty, then this should return 9. Otherwise, we want to deal with the attributes. Well, first we want to check that it's actually a relevant attribute. So we're going to do something along the lines of for, then we might not need this, for attr in f.atters. we get back here, we're going to say none. And for each attribute, we want to look at if, we want to look at the path, adder.path, and the path has segments like before, right? dot len not equal to one. And actually, if it's equal to one, and add our path segments zero, which is a path segment dot ident is equal to builder, then we want to do something with it. All what do we want to do with it? Well, then we want to look at the token stream and look for what we expected to say each. We expected to have an equal sign, and then we expect the string that's going to be the name of the method. So here, we're going to do something like assert adder.txt. Let's see. Ooh. So the token stream is gonna be adder.tts, right? And we wanna walk the token stream to get, to check that these tokens are actually the right ones. So we wanna assert that tts.next, search equal. Actually, well, we want that to be a group. So what is a group? Well, I guess this probably re-exports token stream would be my guess, or it would be my hope. Token stream. Apparently not. Oh, maybe it's a re-export. No? Interesting. In that case, we'll go over here. So a token stream. Ah, right, so a token stream implements into iterator and it gives us token trees in a particular group, which is what we want. So if let Brock macro, oh, we can probably even parse this. What does Sin give me here? Uh, maybe sin has a convenient parser here. Parse. Parse a simplified tuple struct. Oh, it's just going to be a parse macro input. And then we're just going to have to figure out what to parse it as. In particular, we want to parse it as attribute. Maybe, let's see, derive input. Hmm. We're gonna get custom syntax. I'm like fairly sure there's gonna be a... I really want like inert, but it doesn't seem like that's there. Could be attribute. Yeah. But the real question is, how do I parse? Oh, I see. It wouldn't know what to parse it as is the real problem. Oh, examples. Attribute parse outer. The TTS field contains the rest of the Adder body token. Try parsing the tokens of an attribute into the structured. All right, so it's already parsed basically as much as it can. So we're expecting this then to be token tree group g is Adder.tts.next. Let's see that that actually gives us what we expect. This probably has to do like, is there a convenient way to clone this? If I have a token stream, how do I? Sure, that seems fine. I'm Twitter. Proc macro two, ah, so now we get to, so notice that it's referring to proc macro two. So proc macro two is a crate that's just a wrapper around the compiler provided proc macro crate that basically gives you backwards compatibility. It gives you features from nightly that aren't stable yet. And it gives you some, basically the ability to use this outside of just the procedural macro so that you can write tests or. Like basically do things when you're not in the context of running within the compiler. So these shouldn't really matter. Like it's fine for us to do this. And I don't think this has anything more useful on token stream. I think it's basically the same. Yeah. We probably need to depend on it, don't we? Macro 2 is 0.4. Oh, right. And this probably has to do something like some quote, just so that it can infer the type. All right, so what does this give us? All right, so it gives me the group. That's great. And now we want to look at the stream. So this is proc macro to group. And we want the stream from that. And then we want, what do we want to do with that stream? tokens is going to be stream.IntoIter. We want to assert equal tokens. Next is going to be a, I guess we want to do next. I don't know if this will even work, but we can try. That's really what we want to say, right? We expect that there's an eek. We expect that there's an equal. This won't actually type check, but, and then arg is tokens.next.unwrap. And then we want arg. What Rust plugin are you using? ALE, it's pretty nice. I'm a big fan of it. Wait, what do you mean group does not have stream? Oh, fine, fine, fine. G is going to be g.stream. Right. Token tree, you say. Can I do ooh, so I can probably then do quote each into that something that would fly. No. See, I just don't want to fully parse this and I don't think it should be necessary. But I guess, unwrap. So this really should be, how about now? Will that let me do this? No. I mean, it's not terribly surprising. I want this to be a ProcMacro2TokenTree. Actually, I just want to use ProcMacro2TokenTree just because I'm going to be repeating it a bunch. I want this to be an ident. Probably I'm not allowed to say that, am I? Expected item find reference. So let's see, what can I do with a, do I really need to like walk it? Cause that sounds really unfortunate. So I guess this would be panic unexpected this. I might be happier about this. What was the thing before? Thing before is a punct. Which we want to be equal to this. Expected each found this. Expected equals found that. See what this gives us. This should say p. I'm not allowed to compare punct to that. Okay, what can I compare punct to? Apparently nothing really, but ask car. I can compare. This does need to be mutable, that's true. Beautiful. All right, so now we get just the literals, which is what we wanted. Now we just wanna extract the literal from the literal here and that gives us... How do I get out? But how do I get out the value? Oh, it implements display, I suppose. But that might actually not matter. We don't actually need to get out the value. All we really need is now we have that. Let's see what this gives us. So if we now do FN. arg. If I do this, what does that give us? What does that expand to? expected identifier found arg. Yeah, so now it's basically this turns into each, right? Because the way the way that this is written is this is a string, and I want just the inner part. So the question is, how do we do that? Is there a way for me to turn a string into an identifier? Yes, there is. It's going to be ident. Oops. Arg is going to be ident new using arg and arg.span. No, this is sinident, because I want to create an identifier for this method. And I want it to be tied to the span of the original each part so that it's spelled wrong, you still get something reasonable. So this, I want to be this. And it needs a string. Rock macro derived panicked. Did it now. What? Oh, arg is not a valid identifier. I really, I can't print this? How do I get? How do I turn a string into an identifier? Let's see if there's something in 07 here. OK, that's fine. Inert attributes, the word inert indicates that these attributes do not correspond to macroindications. That's fine. So this token tree, when I have a literal, How can I extract the contents of that? Because display displays is going to display display it as a string. How do I parse that out? Interesting. interesting interesting interesting um sin can i parse an ident i wonder or a literal lit stir. Yeah. horse. Specifically, can I turn my literal into one of these? Is there a parse literal maybe? parse macro input. Is there a literal type lit? Alright, so how do I make one of these? Aha, great. So I can do sin lit. New from this. So that gives me a little that gives me a lit and then I should be able to do if let sin lit. Is this right? Actually, I can just do the same here. So if I get a syn lit strs, then that's great. Otherwise, panic. Expected identifier found. didn't really expect an identifier, but it expected string found whatever that is. And now right, but if I got a stir, then I'll presumably from a stir, I can actually get the string value. Great. So now I can do s.value and s.spend. Let's see how that works out. Expected literal found token tree. Oh, right. We need to let arg is match tokens dot next dot unwrap. We have to do the same thing here, except this should be literal. L, L, expected string found that, and then this is gonna match on the arg. So panics are bad practice, but when you're writing procedural macros, you are specifically supposed to panic. That is the way you signal errors to the compiler. Now I think that the next exercise is basically how do you give good compiler messages from procedural macros, but it is still all about panicking. We'll see that in a bit in the next exercise. It cannot be formatted with the default formatter. 197. Well, then how am I supposed to use it? Debug. All right, great. So here you see, we did in fact get a function with the name arg and the function with the name env, which are the things that were in here. Great, so we did actually get something semi-reasonable. Now, of course, what we really want this to be is we want this to contain the inner type. So I think here's something we probably want. We want this to take a wrapper to be a string. So this we want to extract from option. This we want to extract from option. However, here we want cyInnerType from vec. The specification said we were allowed to assume that this was a vec. unwrap. So this is going to take arg, I guess, arg of type inner tie. It's going to also take, I guess, mute self. And it's going to return mute self. And the body here is going to be if. self.name is none, then self.name is some vec arg. Else, I guess we can do it the other way around. If let sum ref mute args or values is self.name, then we just do values.push arg else this, and then return self. See how that expands too. expected lifetime parameter really fine. This string is not a part of the return type. So it's just tied this way. Now what does it expand to? You can't do anything without using unsafe? That's not true. There is no unsafety in this code. There is, in fact, no unsafety in most of the code I write. So that's just not true. OK, so what does this generate to? arg is if self.args is sum, then we push. Otherwise, yeah, so this is what the vec macro actually de-sugars to. Great. Does that pass though? It's the real question. This theme is called Groovebox. I recently switched to it. It's pretty neat. Pretty happy with it. Duplicate definitions with name env. Oh, that's sneaky. So this one, the name of the builder method and the name of the field are the same. So we actually don't want to generate a, for anything that has a builder, we specifically don't want to generate the standard one. It's a little awkward. Like the question is basically for args, should it be possible to specify all the arguments at once? It sort of has to be, right? Because the previous test assumed that you're able to just call args. So it's specifically if the name is the same as the field, then don't generate the normal one. So how is it possible for the past test to even work here? This test expects env to take a vector. This one expects Well, this one doesn't actually expect anything. Oh, that's super weird. I don't actually know what this wants me to generate. I think env still needs to take vec string, because otherwise the other code samples would no longer work. So let's look at five here, for example. This still calls args and still calls env, and it calls them both with vectors. So here, we still need to generate args, and we still need to generate env. Actually, that's not true. I guess using this could imply that you don't want the default one. Hmm. I don't actually know what the exercise wants us to do here. It sort of makes sense to generate both when you can. The question is when you can't. I think when you can't, you don't generate the vector version. Because the user has explicitly said they want this one. So That's going to be a little awkward for us because it means that if you generate an extended method and the name of the extended method is the same as the name of the field, then don't generate the normal method. Also, I don't think this interacts correctly with, if the user had to type those option Vec, this wouldn't work correctly. Although that doesn't actually come up here. How do we want to do this? Hmm. It's a very good question. Because we could do these same checks when generating the normal builder. But I think really what we want is just a single one. And then that mute made, I guess, avoid conflict as false. And then down here, we want to say. if arg is equal to f.ident, then avoid conflict is true. And then we'll just generate this. Yeah, I think we're going to have to do something like that. We're going to have to tidy it up more than this, though. Extended method takes an f, which is a field, and generates an option token stream. combination, I guess, of a bool and an option stream. So this would be, instead of this, this would say method equals this, and then return some, I don't know. And if arg is equal to f.ident, then it conflicts. Because now, methods, this can also be, this doesn't rely on anything outside. So this can go down here. Just tidy up our file a little bit. So really what we want to do here is conflicts and extend method is extended method. Hmm. So let's see. So if it conflicts, we don't want to generate this. Although, I mean, we can generate it regardless, right? So method is this. Or I guess set method. And then we can do here, match on that. If what we get back is none, then we just return the set method. If it's sum of true and extend method, then we give extend method, because we can't also include the set method, because the set method would have a conflicting name. And if we have false extend method, Then we produce quote of set method and extend method. See what that gives us. It gives us a whole bunch of errors, what it does. And then now this should just be methods. Oh, I see. Into. Expert is that. And then expert.into. It's a bunch of errors. Trade from that is not implemented. What? Something's very funky here. Extend, I guess this should really just say extend method. Extend method. Why is this getting weird? Token stream implements two tokens is not satisfied. I guess one question is, we don't really need to do the into down here. That's mostly just causing us pain. In fact, this isn't even returning a token stream currently because. I see, oh, I see what's going on. This is a proc macro two token stream. Uh, great. and fident. But arg here, so ident shouldn't, ident new just gives me a new, right? I don't need to unwrap it or anything. Yeah. And f.identify of a field. Oh, I see. Could the fields ident be? OK, I see. So this really should be f. And this should say name. Really, I can't compare two idents? Am I just confused about what? Oh. Yeah, this no longer needs that. No, this should be able to do that. Alright, so let's see what it now generates. Puffin args, puffin arg, puffin n, which is the single extend. And it does not generate the conflicting, the conflicting n that sets all of the method. Great. Okay, see what that does. failed something. Seven failed. And is not set. Oh, they want us to not even make those. When they have each set, it shouldn't be turned into an option either. Interesting. The example to consider here is if no env is ever given, should it fail or should it give you an empty vector? In all the previous examples, it gave you an empty vector. So it seems like seven is a little bit under-specified. Assume that fields of the type vec, in order for compiler to know, Okay, so I think this is assuming that if something has a builder tag, then it should no longer be an option. It should be an empty vec instead. Which I think means that what we want here is... is built, I guess. And that's going to be something like f.adders. Oh, that's all sorts of weird. Adder, any f. builder of which returns you an option. I guess, proc macro to group. Yep. So this is gonna be these, then return some G, otherwise give me none. So now this should be let G is builder of F. And this no longer needs a return. So that is a little nicer. And what this means is also that here, else if builder of f is sum, then we also just wanna do that. And here, what we want to generate is different as well, because it's no longer going to be an option. So instead, this is just going to take name. Right, so if vector is given in, it should just set the vector equal because the vector does not have a type wrapper. And I guess down here, if that, oops. And I guess these are the same. And down here as well, more builder of. F. And here, when we generate the initial builder, anything that is a builder, we want to generate as an empty vec. Actually, let's do new. Otherwise, we're going to generate a none. Let's see what that expands into. So I enter type. So now in the builder, args and env are both just vectors. They're not options anymore. When we create a new one, they're just created as vectors. When you call build, oh, AL does. When you call build, they're just cloned. They're not required to be sum. That's weird too, I guess. Sure. And for args, nope, args should take a vec string. So that is wrong. This should take the type. So args takes a vec string and just sets. and just set self.args to that. Arg takes a string and pushes it. And env, because they would conflict, is only the builder version, so it only has a version that takes string. All right, let's see what that gives us. We still failed the last one. SpecDestruct vec found option. Where? Oh, right. The code we generate for the extended method here should now always just do self.name.push, because it's now no longer an option. So notice how previously the code for adding just one of these things for an extendable attribute would do a sum match on self.theField. But in reality, because there are now no longer options, it should really just do the push. Let's see. Hey, OK, we passed 7. Now we're getting somewhere. So progress, compile fail. Okay, so this is an interesting attribute of try build. So try build. And try build also lets you write tests that are not supposed to compile, basically so that you can check the output of your compiler. So you want it to provide helpful error messages. And so this is probably going to just fail, but let's try to run it and see what happens. In fact, let's just speed this up a little by commenting these out for now and see what it gives us. I see. So here, let's look at what this, so this is someone asked earlier about, about the fact that we're writing unwrap in so many places, and isn't that bad practice? And it is bad practice in Rust in general. In the case of proc macros, panicking is the way you communicate to the compiler that something went wrong. Preferred way to report an error from a procedural macro is by including an invocation of the standard library's compile error macro in the code emitted by the procedural macros. Interesting. Interesting indeed. So here the problem is this is missing an H. So let's look at syn error. So here, oh, interesting. What does this even do? Interesting. OK, so we probably want to generate a new one of these. What's the difference between new spend? Some other spend with a par stream. Yeah, so I see. Well, that's a good question. Yeah, we don't actually have a parse stream, although it could totally be that we could. But at this point. Remember, what we get from the token stream is ident equals literal, as I don't think there's any point in parsing that directly with syn. So instead, let's just generate the message manually, which would be by doing... So this is up for the each, right? So here, instead of asserting that it's equal to each, it's like if I not equal to each, then well, certainly sin error, new. And the span is going to be the span of the eye. I And the message is going to be this. The real question is, what do we do with this error? See, that's what I'm trying to figure out. Because once you have one of these errors, what do I do? Render the error. Why does this return a token stream? Oh, I thought I mentioned that. So I use the AL, which is a Vim plugin that gives you highlighting of compiler errors inline, ALE. How do we make the, like, what does this method do? Oh, I see it generates. It generates a compile error statement. Hmm. Oh, I see what's going on. So really, it's not really that we want to panic. It's that we want to generate a syntax tree that has so there's a macro in Rust that's compile error that lets you emit a compiler at a particular point. Yeah. OK, so in theory, this should be able to return some false this to compile error. Maybe? What does this expand to? uh oh and then So eight, no, oh, eight. See what this expands to. Yeah. See, so here we now get expect, yeah, so now we get what we wanted and the highlighting because we take the span from the right place gives us in theory the right error message. So really what happened here is the, what we produced was when we tried to, basically when we tried to generate the method for this line, like when we're trying to generate the output what we instead generated was an invocation of the compile error macro, which caused it to produce this error up here. I don't know if that's what I wanted, but in theory it might be. So what do we see here? Oh, it wants us to highlight the entire thing. So we actually specifically highlighted the keyword that was wrong and it's wanting a wider highlight, um, which of course is, is relatively easy for us to provide. So instead of this being the I span and we're going to make it the, uh, G dot span. Oh, come on. Our builder off is too helpful. Oh, that's fine. We can just have this be f.addrs0.span. That's me just sort of bullshit guessing. But so what does? What do we have on field? Field adders attribute. Attribute. Huh. But how do I get the span of it? Why can't I get the span of this attribute? Spanned. Oh, really? All right. a look at that. What does it now highlight? Oh, what? Hmm. That doesn't seem right. That should not be the span of this attribute. So I mean, what it wants us to highlight is everything between the square brackets. And it seems like we can only really get the spans for this, or the span for the attribute is just this for some reason. So maybe what we need to do is generate, can I merge spans? Probably. Spend, spend, spend, spend, spend, spend, spend, spend, spend, spend, join. What does join actually do? That's why I can't type. Really? of a group, oh wait, spin. Give me back to sin. So on a span, if I join to what does that do? Join inner, what is inner? Oh, Docs are as rock. Where is the real proc macro here? Does this have a span? Great. What does span join do? Join. Okay, what does it do? Self.zero, thanks. Span contains a bridge client span. That's very unhelpful. I basically wanna know, I wanna know if I join two spans, What do I produce? Is it disjoint parts, or do they get combined? Because I think really what I want is, I want to sort of, what happens if I join this with the i.span, for example? This won't produce anything particularly useful, but. Oh, you need to opt in. That's annoying. The real oddity here is that the span of this attribute is not the span of the attribute. This is not a sensible span for that attribute. It might be that the easier part here is actually to do what builder of does. No, see, it really just should be of the adder, the whole adder. So if we look at what a sin attribute is, right, the sin attribute, I guess it includes the pound, and we really want the path and the Does this implement span? No. Oh, it does. OK. So what happens if I do adders 0 path span? So that highlights just the path. Right. And if I join it with f adders 0 TTS span, Oh, and what was the requirement for me to get join from PROC MACRO 2? Join. Semver except and not exposed by default. Can I create a new span though? Basically, how do I extend a span? Spacing token tree. This really seems like span is wrong for attribute. Style? Oh, what's the style, though? It doesn't implement spend, so it doesn't matter. Hmm. Let's look at this. Join spans. Okay. Join spans. Is that a method that I get? Use the token macro instead. Now it doesn't really help me either. Spend How do I make my own span? Hmm. I mean, we could just leave it the way it was, but it seems a little bit sort of dissatisfying. Oh, actually, maybe this is a bug here. Cause the, the token stream, like, the spandem limitation for attribute is that it joins the spans of the attributes token stream. Filter map. Oh. Oh, OK, it needs this for that to work. See here? It doesn't actually join anything unless it has this. So that's why it's broken, which suggests that we actually I think the span in the test case is the one from attribute parse meta. Sorry, from sin. So attribute parse meta. Oh, I see what you've done. OK, so what we did here previously is we sort of manually parsed the stream inside of the attributes. I see. And it just so happens that there's an attribute has a convenience method for parsing things that have a particular format. And so we instead use parseMeta. That gives us basically what we want here is name value, right? So name equals value pairs. I see. I see. So really what we can do here is something along the lines of meta is parse macro input. I guess that would be, can we do g as synmeta? Is that well-defined? And then e. not like that. Although into token stream. So that's something I can use here. Oh, that's not an expression, is it? Oh, sorry, this can just be this is just the attribute itself. So rather than do all of this business. This is gonna return an optional, I guess, sin attribute, and then return adder. And then that will give us this. And then we'll not do the join. Let's see whether it likes this. Expected token stream found. Right. 82 token stream. I guess the alternative would be that I could just call this. I guess the question is, what is this result? Oh, I see. So if I get one of these, no, that's not what I meant. Match g parse meta. So inner is going to be this. If it's inner, then I. If it's an error, then it's already determined what the parse error is. And so at that point, I can do this business, except the error is already constructed for me. Although, can I give it a new message? I see that's if it fails to parse. So this can still just be E and then we might need this for later. And this is not gonna be used. And now the question is, what is this business? So this gives me attribute parse meta, gives me a meta. And if, Let's send meta name value. Actually, let's do meta name value. Then we're going to do some stuff. actually, then we're just going to return and be if we get some other meta, then we're going to have to figure out what to do. Haven't decided that quite yet. Now, down here. I guess if meta dot ident not equal to each, then we're going to pull this trick again. Right, then the ident is wrong. So we're going to return false, and then basically the span of the meta with expectedEach. Great. The e-token we can ignore. And then this is just going to be meta.lit, which is now much easier to get out. That saves us a bunch of stuff. So this is going to be meta.lit. Excellent. And now the question is, what do we do if we get something, if someone does like builder foo and doesn't say foo equals? And in that case, we can also just construct an appropriate message, something along the lines of, this is still metaspan. And I guess that's probably fine. And now we also don't need minus 6. This is nice. And it doesn't like, right, this is to compile error. Expected proc marker two literal found. Oh great, we even get the literal. Even better. So now, okay, so it's still not quite right. The highlighting now highlights builder, which is still not what we wanted. And now it's using the, The YouTube stream is down for you. Well, that's kind of damning. Is anyone on YouTube been able to confirm whether or not you can see the stream? I guess I can check here too. YouTube as a whole is having a massive outage. Great. Well, I guess go to Twitch. I guess now there may be more people on Twitch. We'll just continue. I mean, people can always just catch up to the recorded video that's uploaded after. Um, okay. So we're still not getting quite the right span, uh, for I'm assuming we're hitting this message. Oh, it's cause we're already. I see. Uh, no, that's still, that's still not right. So I mean, we're presumably hitting this just to double check that I'm not going insane. I see. So it means at least you found one from the other. Expected builder. Oh, really? We're hitting this one? Why are we hitting that one? What kind of meta am I getting here? I'm getting a meta. I'm getting a meta list. Oh, I see. OK, fine, So this should be a list of fine. If nvs.len not equal to one, then we have another error, which is specifically, oops. Ah, why? Then I expected only this. And then, OK, so what's the type of the inner thing here? Nested meta. What else could be the nested? Like the thing that I get back from this, if I get a meta list, nested is a punctuated nested meta. Okay, so we're gonna do, we're gonna match on NVs first. And if it is a syn-nestedMeta, meta, syn-meta, nameValue, then we're going to return that. If it's any other kind of meta, Then we're going to return basically the same thing here. We're going to return this error in a lot of places. So arguably, we can make this nicer. What does this give me? Meta list, you say? No, .nested. Actually, that's not even. Why does a meta list have an ident? Oh. I see what's going on. The list is, oh. OK, so the list that it's naming here is if you look at this, if we look at this in isolation. So this is a list that contains, it's a list. And that list contains builder whatever. So that's the list. And then I guess builder, or is that further up here? Yeah. So it's a list. And the item of the list is builder. It nested inside of it, it contains a name value pairing. So that's really what the structure of this thing is. which means that we need to parse it as such. So we expect to get this back. We expect the list to have only one item, I think. In fact, metaList is not a list at all. So a list contains a metaList. And a meta list is not a list. Oh, I see nested is a list. I see. Oh, I see. So really, the reason this is coming across is you might actually have someone write the following. If I write a equals b comma c. So this is a list whose identifier is builder, where the first thing is a name value. And the second thing is a just a is a literal, well, it might be a meta word, actually. I see. So that's why that can be that way. So if nvs dot ident not equal to builder, then what does builder even do down here? So our builder of, so this should just be true. Because we call builder of, and builder of checks that it is, in fact, a builder. So this is fine. And then if the nested length is not equal to one, then rewrite this. And then if the first thing is not a name value, then we also show this. And that span should have the meta, which is not including builder. This span should have including builder. And this should have if nv. So that's a meta name value. If nv.ident not equal to each, then also return this error. So we're going to create this error a lot of times. I would like this to be nicer. Let error. It's going to take a span. And it's going to be this business. Great, so now this can be error and we have spam. That is so much nicer. This can be the same. And then nv. This is air meta span. This is air also meta span. And this no longer needs to happen because that's, oops, so that's already happening outside. Ooh, all right, let's see how that expands. Return expected option. No method name first. That is true because this is we're looking at the nested arguments to builder. That gives me a pair, and I want the value. In fact, this we need to unwrap. this, and then we're going to match args value. And this is yelling at me because what exactly? nvs.nested. It's because the punctuated iterator, when I call first, just gives me access to the first element, which I don't own, which means that for the this, is there a way for me to just take the first? pop. Sure. n vs nested. Pop unwrap because we know the length is one that gives me a pair. And then probably something like into value. Great into value. So now I own it. And now this needs to be mutable. All right, how about now? Great. And this gives me an error. And it's just builder. So we've gone full round trip on getting the span to be back to what it was. Now that said, I think the code is much nicer now because we're using this meta parsing. It's probably also more robust. Right. So. gotta be this one. I guess no, it's not that one. It's this one. So this we're saying should really have the span of should have the span of the end. Oh, is it because we pop it? So if I do do and yes, span. And then down here. Do this? Like does that help at all? No, so it's not the pop. Good old fashioned debugging. Yeah, okay, so it is failing there. It's just that this span contains just the ident. And I'm pretty sure that's because we're not including the feature that lets you join spans, and therefore, syn is also not able to join spans. So I think what we really want to do is enable. Oh. Hmm. Actually, let's take a look at the cargo tomel for this crate and see how we might get at. Yeah. Because specifically, I think what we want is we want the, from the proc macro to create, I think we want this feature. So the question is, how do we do that? We want to concurrently only only underline one token. Oh, I see. So I can't use this. I need to use new spend. And what does new spend do? Error. Sorry, for those not watching the chat, David, who wrote these assignments, is actually in chat and is pointing at all the things I'm doing wrong. Oh, I see. I see. Interesting. Well, I guess we'll do that. Because I feel like the other option would be to just join the spans manually. But I agree with you. It seems better to just do this instead. So in that case, I guess t. This would be t. This would give in. nvs, this would give in nvs. Although now it might be a problem that we pop it, come to think of it. This should give in meta. So I guess in these cases, we really just wanna give in the full token tree. That's gonna complain somewhere. Meta list, oh, I see. And so now this, I guess has to be one of these. Uh, it's sort of annoying, but so this has to be generic over T, which is sin two tokens. T T returns me. What exactly a, one of these. And now this should be possible. Right, yeah, so we can't use Error New because you can't join spans as a span, unless I guess if they're consecutive. But then you still need the nightly feature. And two tokens is somewhere else. It's in. Rock macro to maybe it's in quotes. Now what does it highlight? Hey, it highlights the right thing. Great. All right. Now let's get rid of this because no longer needed. Now do we pass the test? Okay, great. Good job team. All right. Let's check that all the other ones also still pass. Beautiful. Now, the final one. Oh, nine. Redefined prelude types. Does your macro still work if some of the standard library prelude items mean something different? Hmm. Yes, we talked a little bit about this earlier. And in fact, here, we get pretty close. We already use standard option, option. We would still fail on this assumption, I guess. And these would fail. So sum is also a part of the option type. So this needs to say option colon, colon, sum. Um, just like this does now, man, option option sum. Uh, what else do we have down here? Those are all fine. This would have to be where does vec live standard vec vec. This would be standard option, option, none. This would be standard. Where does result live? Result, result. Where does box live? Standard box to box. What else do we have? Mute self, these are all fine. What else is it? Try to override. Result box, option sum, none. Okay. This needs to be standard result, result, okay. There's another question of what happens if here, if they had a module called standard. And of course, this was very nice of you, David, to remove the other things that were option here. But in theory, you could totally imagine that they then use this option type on this struct. And remember how we have this assumption that the option type is just called option, which would totally break if that weren't the case. Is that the right one? progress that was the right one excellent did we pass all? we passed all the tests nice Um, now of course there's still more things that we could do with this. First of all, the code is a bit of a mess, um, because it's all written in one, but like we did the thing, take that proc macro workshop. Um, Although, why does my output give so many warnings? I want to get rid of these warnings. See, it gives me warnings from the test files. Can I just suppress that somehow? Is there a? I want this output. I want it to look literally like this, but I'm getting all these warnings. Field is never used. Hmm. Are these the generated fields? Oh, there's something the implementation doesn't test, but that we very clearly need to do, which is, oh, no, we already marketed this pub. This should probably also be pub. Maybe you override rust flags. Oh, I probably do. Yep. I see. So you're saying, well, I don't really want to remove that. I guess, arguably, it's unclear whether targetCpunative makes a difference for me. I guess maybe I should just remove that at some point. Well, all right, but we do pass all the tests. So yeah, we pass all the tests there. Right, so things we might want to fix about this. Well, first of all, the code is a bit of a mess. There's a lot of this things that look almost the same but not quite. For example, the first and last case here could probably be combined maybe even with this one if you made this optional somehow. Like you could just construct, you could set name to be quote this of name, for example, in this particular instance. So in theory, all these three could collapse into one. I'm also a little bit sad about this vec business for the methods. So for the extended method, the fact that we have to do, where's the check that we have? Yeah, here. Don't include the set method if the extended method ends up having the same name. But I feel like these are things that you're going to run into when you write real world macros regardless. In fact, how about we do some of that tuning? I don't think I want to start another one of the projects. One thing we might do is actually do a... If there's enough interest in this, we might do a second stream on... Yeah. on proc macros on some of the more advanced ones. So the suggestion here is basically do this one first and then do the other ones because the other ones have other things you need to deal with that are potentially harder or different. And so I kind of want to do the other ones too so we might do those in a subsequent stream if there's sufficient interest to cover more of proc macros. I think this is a good intro to like, here are all the bits and pieces you need to at least get started and get things to fit together. which is why I think what I want to do for the remainder of today is tidy up this code, at least to some extent. So the first is what we do about generating these methods that are almost the same. So what we really want to do here is, let's see, how are these different? So this is the set method. So the set method will, the only thing that differs is what we choose to use as the argument type and whether or not we use option. So let's try to structure this a little bit differently, which is. going to use arg type and value. So in this case, we want the arg type to be inner type. And we want the wrapper to be quote this. Right? In this case, we want the type to be type. And we want the assignment to be name. And in the last case, we want type to be type and name to be this. And then set method can now be equal to do without this. to be equal to arg type and value. Oh, they're different, aren't they? Ooh. Oh, right, this is our Type ident versus token stream. Oh, I see. Sure. Just to make them have the same type. Right. So that already makes me feel a lot better. And then let's also leave circleMets for ourselves. If the field is an option, don't require, wait, why does this need to take? Setting is an option, T. Setting should take just a T. But we then need to store it within a sum. If the field is a builder, it is a Vectee. It is a Vectee. So we take... the regular and the value in the builder is not wrapped in an option. So we shouldn't wrap the value in some And finally, otherwise, we take the type used by the target and we store it in an option in the builder in case it was never set. Yeah, I'm of two minds of what I want REST format to do with my comments. It would be nice if it broke comments, but at the same time, and not in this particular case, but in some cases, I really want comments to be... Uh, I don't want it to wrap my comments if I've intentionally wrapped them, which is hard to detect. I agree with you though, that it would be nice if it broke comments at 80, at a max width of 80 columns, right? So currently it just, it allows a line to run for as long as you have space for within your maximum column. But I never want text to run that wide. I want it to force an earlier break. Um, Excellent. So here, this is not a builder, so just use the regular set. Actually, let's not even do it that way. Let's say we need to take care not to include a builder. method with the same name as the set method. For example, consider this struct, deriveBuilder. I want my comments to be later. Comments should be more important in code. Struct command, env. vex string builder each equals env. It would not be OK to generate both env vex string and env string. for the field and end string for the builder. That's these two. Now I need to be parenthesized. To generate both. By here. I think here we're decently happy with where we're at. I mean, we could simplify this, but I'm okay with these being separate. I do sort of want some observations about what these are. Builder fields, fields. So these are the builder fields. These are the methods. These are, these are the build fields. So this is for when you call. Builder build empty. And then I really wish I didn't have to assign this to a variable, but I guess it's OK. So the other thing that would be neat here is if the proc macro also generated comments, it's unclear that it can generate particularly useful comments here. But the one thing it can do is refer back to the original builder using the new fancy... auto linking that Rustdoc supports. In particular, this could say, implement the API guidelines. No, predictability, flexibility, checklist, builder. Wait, is there more than one builder now? No, great. So this implements the builder pattern for. And then this is kind of cool. I don't know whether this will actually work. But in theory, it might. For like, we'll quote extrapolate things inside of Doc comments. I guess we're about to find out. Name. If I run cargo expand now, what do I get? No, it does not. Does not generate that. That's kind of unfortunate. Think you're meant to use doc equals. What do you mean? For what? Uh, Oh, I see. I see. I mean, I guess we can do that. That just seems less, it seems a little sad, but I guess, um, what was the, uh, but am I allowed to use macros here? I'm not actually sure whether I'm allowed to use a macro here. Right, like this definitely seems weird. Am I the only one who thinks this seems weird? I mean, I guess we could try it, but. produced unparsable tokens. Yeah, see, I don't think you can use macros in here. So I don't think that helps me at all. Let's look at quote and see what it says. Searching identifiers, method calls. interpolation. Huh? I mean, we could always generate it manually, but that seems unfortunate, too. Right? Like, what one way for us to do this is to just create a new token stream manually. Um, and then just extend it with the appropriate token trees. Um, uh, I guess, I guess one answer would be if this didn't need to do that, if like, is this okay? And that could very well be. Well, that produced something kind of like what we wanted, although that doesn't seem right. Like, is this? Yeah, it also didn't do expansion. It could be that we just need to generate this manually. So I guess one question would be, how do you even generate comments? I don't actually know what a comment would parse to. Although. Let's bring this back to what it was for a second. I think what we can do here is cheat a little by going up here and then doing give me the whole token stream. and then go to main, and then do like... Hello. And have nothing in there. Now let's see what this expands to. So now. Oh, that's interesting. So triple equals really does expand to pound doc. You're right. Well, that's useful. OK, so in theory, then, what we should be able to do is here's what I'm thinking. Like, I guess let doc is something along the lines of quote doc equals empty string. And then let's print out what exactly doc is. And then we just stick doc here. Does that make sense? And then we're just going to modify it in place. Oh, doc, please. All right, so that is great. And now we just modify this string to be the appropriate... I don't think doc compounds that way, but we can just generate it manually. So we do this. And then we do, can I even mutate a token stream? Actually, I probably can. In fact, what I can probably do instead is now that we have that, say token stream new. uh dot append and we're gonna append a i guess uh extend no that's not even append a uh token tree a token tree. Funct. Roth macro. New pound. spacing, you say? What is a spacing? Alone? Sure. Alone. All right, and then we want to generate a group. Okay, and what is a group? A group is a new delimiter, is bracket. Delimiter, bracket, and a token stream, which is this. So what's an inner? It's an ident and specifically an ident doc. OK, so how do I make an ident? Because now we're no longer using the din types. Can I just create like a new span? Is that a thing? I think, sure, that seems fine. False light. OK, so then we have that. And then what we want to append next is, a punk equals it's going to be alone. And then a literal has got to be a nicer way to do this because this is a pain. Literal. And literal. string. And that string is going to be this. Or that new line, new line. Uh, actually let's do this formats little nicer. This, this with names, and this is gonna produce that. And it's going to produce that. And this is going to expand to T S. I have no idea what this is gonna give us, but we're about to find out. So this can now go away. And now what will this expand to? Well, it's a syntax error somewhere. Did I mess up here maybe? Yeah. See what this gives us. Does it not take append? What do I have to do for a token stream? It implements extend. Really? So I need to do... fine. It's gonna be like, extend back of these. And this to extend back of this. I mean, I mostly just want to see whether this could even work. 34. Yeah. And why doesn't this do something useful? OK. Okay. Theory, that should be all valid. Quote two. Oh, I can't do that, can I? This has to be a proc macro two token stream. Oh, that's going to be annoying. This has to be a proc micro two. This has to be a two. This has to be a two. These have to be two. This has to be two. What about now? Well, I mean, I don't know if this produces the right thing. I'm not actually sure. It compiles. So this suggests that CargoExpand is actually doing the wrong thing here. Or not necessarily, but it doesn't realize that the new lines technically would need to be done the same thing with. But that's fine. So if I run cargo doc here, what would this give me? So I think this actually generates the right documentation. I mean, it's terrible. Yeah, command builder. Implements the builder pattern for command. Now, this shorthand URL will not actually work unless you're on nightly and stuff. But this is such pain for generating that documentation, though. It seems like quote just doesn't have a good way to deal with. generating comments, specifically with interpolation of things. I wonder whether Not particularly helpful. What about issues? Omit. Yeah, it doesn't really say. Maybe this is worth raising as a thing. But hey, I mean, it works. I don't know whether we want to keep it there, but you know. This ugly business generates a doc comment for... builder that points to the original type. And this experience tells me that, oh, I see. Oh, I see. You just format the message. I see. OK, sure. So here's. suggested alternative method of doing it doc is format is this. I think this is what you mean, right? So this, and then none of this, and then doc is equal to doc. And you're saying that should work. Yeah, I agree, that's nicer. It's a little weird, the indirection is a little weird, but yeah, that is much nicer. Great. I don't think for the functions, it doesn't really make sense to add any doc comments. I know that some people like to have comments and just say what the method do, but these methods, it would not be useful to have comments. Builder maybe, but unclear. What else? This is now a decent amount nicer. What we could do here, just for our own sanity, is list here is builderEachEqual is of elements inside. The last element was, and then I guess here, nvs nested 0. Ah, no, sorry. That's not true. It's this. List here is dot dot. in and then this is hopefully leach equals food. And yes, nested 0. This is just sort of the thought process that we went through when writing this code. And it's useful to document what each point is so the next time we get to this code, we don't have to do it all again. So this is shouldn't be necessary. This too was either just a literal. or an identifier, or I guess was not key equals value. That's really what that comes down to. This is a, what would that even be? What other metas are there? Let's save ourselves some future trouble. No, this. Inside of there was either just an identifier or So that would be something like this. That would be if it's a word or a key value mapping, which would be builder equals foo, neither of which are OK. Of course, this kind of syntax might actually be something we want to let the user use. But this is more to document what each of these cases are. This can go in here. This is a nested one because it's not supposed to be used outside. Builder of we use other places. We do even. Yeah. OK, I think I'm decently happy there. Implement derive builder. Um... Okay, I think that's where we're going to end today then. I think it's a good place to stop. What I'll do is... I think what I'll do is I'll do sort of an informal poll, probably on Twitter, although also message me if you have particular thoughts, of whether people want to see more procedural macros or whether we should consider that topic now covered and then move on to whatever the next thing would be, which currently looks like this one, although there are always like a bunch of shakeups in the votes after any given. after we move on from one stream. Also keep in mind that I've added a bunch of new topics here, so feel free to vote for more things that you think are interesting. So... Okay, I will upload the video, start a poll for whether people wanna see more PROC macros. If so, what we'll do is basically work through one or, I mean, in theory, these will be faster now that we know what we're doing maybe, but work through one or two of the other projects from here from the same workshop. And otherwise, if people don't wanna see any more of it, then we'll do whatever the next voted topic is. Great, thanks for coming out. I hope this was interesting. I certainly learned a bunch. And I'll see you next stream, which will probably be in three weeks from now, maybe. Little unclear. I'll announce it as usual. So thanks for watching. Have a great day.
