Crust of Rust: Declarative Macros

welcome back it's another crust of rust mini session I I really like that name it's a good name so we did the previous one on lifetimes and in in particular sort of lifetime annotations and when you need lifetime annotations and when you don't and to sort of continue this trend of focusing on a particular use case and trying to write like real code that needs this particular pattern I wanted to take a look at declarative macros so nothing no like proc macros anything like that I have a video on that but that is sort of a more advanced concept whereas here because crust of Russ is sort of intended to be like let's deal with like one piece and just like do something with that piece and hopefully something that's understandable to a broader range of Russ developers even if you are a little bit newer to the language like this is something that is really useful to know about and so we're gonna be talking about macro rules and just before I jump in I want to sort of point out you can if you like these videos you can follow me on Twitter or YouTube or twitch or wherever and I'll put in particularly on Twitter post whenever I'm about to make new videos especially for crusts of rust I'll make sure to like and down some well in advance so that if you want to watch it then you can just jump on jump in I will also upload all of these to YouTube afterwards so I have a YouTube channel where I upload basically all the videos I do both the longer sort of rust live coding streams that are a little bit more advanced but also these shorter crusts of rust sort of isolated episodes and the macro we're gonna implement today is perhaps unsurprisingly also from the standard library you'll remember last time we did this sort of stir split thing which was turned out to be something that was already in the standard library and this was sort of by design because the standard library exemplifies a lot of real use cases and some replicating things that are there is almost inherently going to be something that real people need and I want to look at the vet macro so you may have seen this either in the form of like declaring a macro declaring a vector using the sort of vector literal syntax but you might also seen this version where you give some element that is clone and then you say how many of those who want and what this resolves to is a vector that has all those elements great so what we're gonna do is basically write this macro so you'll see that the this deserve the general syntax of the macro rules which is you say macro rules and this is how you define a new macro and the next thing is gonna be its name so this is how its invoked followed by exclamation point and then you give a bracket and then you give a bunch of patterns so the patterns for macros is there basically there are arguments to the macro but the arguments to the macro are much looser than arguments to a function for example here instead of having like a variable followed by a type what you have is sort of a syntax pattern were you anything that's variable anything that is sort of provided by the user is gonna be one of these dollar variables that has a type and the type here is not a rust type it is a sort of syntax type so this can be something like it can be an identifier or it can be an expression it can be an item it can be a block it can also be something like at the name of a type or the path to a type or the path to a module we won't dive too much into all the different types here what I want to teach you more about is like how do you think about macros what do they expand to and sort of internally how do they work what do they do and how can I write useful macros if there's something that I want to express and well we'll get to a bunch of this in the process of implementing vector I highly recommend that if you're going to be writing vectors take a look at the little book of rust macros this is just it's a fantastic little gem of all of the weird little like oddities of macros it explains a lot of the syntax that explains of the sort of quirks of it but it also goes into a lot of really handy patterns so if you see section four section four basically goes through if you want to do something that does something like this here's a pattern you can use and we'll will actually see some of that in our implementation of that before we dive in are there any questions about like what we're gonna cover or the vector macro macro in particular there's two do that first dark theme for documentation yes I can nice um if you just search for the little Booker of rust macros it should be the first results and keep that open over here all right so we're gonna do the same thing we did last time which is we're gonna do cargo new lib and we're gonna call it Vic Mackey because why not and we want our just a dedicated library for this in part because it's gonna be easier to develop in isolation you'll see this a little bit later but also just because it's good to get into the habit of making libraries so let me just construct a couple of terminals here for that and dive into source live I'm gonna get rid of these and so we want macro rules you want a Veck and notice here that I'm giving curly brackets for this macros are a little weird and rust in that they have optional limiters not in the sense you can omit them but in that the caller can use any of multiple types of de limiters um so if we write Vectis and let's say we just have an empty and empty pattern this is sort of the empty macro then someone can choose to invoke it either as Beck like this or Veck like this or Vic like this all of these are valid in vacations you'll notice that like this on its own does not work and to see why let's try to run cargo check on this it fine yeah so this is fine it expands to nothing and nothing is allowed to be in these locations and usually the Russ format actually changed these the limiters because it knows about the name that it knows that then if you have a mic it has basically hard-coded the name of the vet micro to say always use square brackets for this you'll see this if we define something else like if we give it a different name which we might want to so we don't conflict with a standard library version huh I wonder why that curly brackets do not need this then you see now it did not do that replacement because it doesn't know that this does a Veck macro we've declared should probably also be formatted with square brackets these are just optional and you in the current macro rule syntax you don't have a way to say you must use a particular delimiter yeah so we've got a question about like sets and maps and stuff and we'll get to that a little bit later in the stream so when you write a vector really what you're writing is a bunch of patterns over the input syntax and one thing you need to know here is that the patterns are patterns over rust syntax trees they're not argument lists so while you can do like arg1 is a type arg2 is an expression and r3 is a path right you can do this and notice now it says like we're missing tokens in macro arguments you can totally express the arguments this way but you don't have to so in macros you're allowed to introduce whatever you want a syntax now this might not be yeah let's do this I'm gonna get rid of these because they're no longer that useful so with this macro I can give whatever syntax pattern I want here as long as that pattern is valid rust it basically means that you need to write a pattern where the input is syntactically valid rust program but that is the only restriction apart from that the input syntax can be whatever you want right like if I were to try to write something that matches what I wrote above one example of that would be maybe you 32 to X dot foo semicolon standard path right so you'll notice this compiles just fine and this is not valid rust syntax if you if you try to compile this program it the compiler would go no but it is syntactically valid it can be parsed and so all that matters for the input to a macro is that it can be parsed and then what matters for compilation is whether the output is valid rust so the input has to be syntactically valid the output has to be valid rust because the output is what actually gets compiled basically you can think of every macro invocation as being replaced by whatever this input was when you're working with macros there's a really handy crate it's basically a cargo command called cargo expand so you can install this with cargo install cargo expand and then you can invoke it as cargo expand and what it does is it takes as input the source for the crate you're currently in and then it expands all the macros with their definitions in this case you'll notice if I run cargo expand the output here is like it has the preload this is what rust injects into every rust program but it doesn't have anything about our macro and you might think this is weird but because we have this here but it's because the replacement is empty we could try to make it not empty right so we could we could for example here say let's say it's good rid of this pattern and say that it's gonna be ident so ident as an identifier and now we could say type arg 2 is equal to Arg 1 now the snowland going to need a semicolon and this that needs to be identifier so like also you 32 and you'll see that if we now run cargo expand this type alias actually gets included in the output because that's what the macro expands to so any questions about like macro patterns before we move on this we haven't actually written anything that's specific to vet this is more talking about what macro rules even does when you say syntactically valid you mean it is valid rust grammar or just valid rust tokens it has to be valid grammar so this is why when I tried to use a single arrow here it told me that I'm not allowed to have a type token followed by arrow and this is basically because the rust grammar does not allow that particular pattern and there might be good reasons for this right it might be that that introduces some kind of parsing ambiguity and that's why that particular syntax is not allowed whereas with a fat arrow it is so you'll see the the output is actually pretty helpful it's saying aloud here are the following identifiers and so you can choose what you want here and in fact here as might actually make a lot of sense right and you see the moment I changed it to ass this invocation is no longer valid because it doesn't match what the macro pattern says it was expecting and you could think that like if if this was a macro that was named like type def then this is a pretty reasonable syntax for it to use another thing you should be aware of for macros is that identifier so you define inside the macro or sort of exists in a separate universe from everything outside the macro so for example if I if I did not take an identifier here let's do let's say that there's sort of a default version of this where this is just gonna be also you 32 so there's one version that does this I may have to do that yeah that's a bad example let's do this so this in and of itself works and if we run cargo expand you see that it expands to type also you're 32 is equal to you 32 but if I write something out here there's gonna be another u 32 is equal to also u 32 maybe this does not apply for types actually let me see if I can come up with a better example let's have a foo function here which calls our a vac with let's have this instead just say let X is equal to 42 so this is not gonna take any arguments so I'm gonna use this macro and then I'm gonna say like X plus 1 if I try to compile this cargo check you'll see it says cannot find value X in the scope right even though this macro defines X and the reason for this is the identifier Xin in macro world are just completely distinct from the variables outside of macro world so if I up here said let X let mute X is 42 and the macro tried to say X plus equals 1 this also would not work because the macro does not it doesn't is not even able to name X in that scope and this is where identifier is come in so we can take like an X which is an ident and then use that here and then think of this as like passing the name into this other universe so now where the X that's here is this X the name out here and so this lets us cross that boundary but you can't otherwise cross that boundary without explicitly opting into it and this is one of the ways in which ross macros are hygienic they're not allowed to access things outside of their own scope identifier some particular great all right hygiene is not actually going to matter for us in this particular instance but let's do sort of test-driven development here again right we want empty vac is going to be the most obvious test we want where let X is gonna be this and then we want to assert that I guess assert that X is empty and we this is trivially true for the vector of the comes from the stand library we want to be able to use our own vector here so let's start with no patterns right what happens if there are no patterns well it's now telling us that our macro rules is invalid right it's it's basically saying I was expecting to see a pattern here but there is no pattern arguably this error message is really bad and it should tell us missing pattern instead what it's really telling us is that it's missing a token in macro arguments and the reason for this is macro rules itself is a macro and it is expecting the between the open and curly open and close curly bracket there's going to be patterns so let's define one right so the for the empty vector this should be really straightforward this should just be Veck new right we also here want to macro use and macro export this macro export because we want anyone who uses our library to be able to call this macro macros by default or think of this sort of like pub for a macro without this the macro will not be callable from any other library that depends on our crate and if we were in cargo cargo test here you see that this test passes and if we were in cargo expand it should be pretty obvious why that is cargo span test whoo that's a good question can I make it expand my tests without Yunus tests it's a very good question maybe not maybe if I do tests yeah that's annoying fine let's just do this thing expand that's also fine so you'll see that this function gets expanded to just Veck new which is what we expected right that's how we define our macro great very straightforward any questions about why this example works so this is obviously a simple macro example it takes no arguments and just produces a static value but it's useful just to see that this does the right thing what about ownership in the case of the dollar X example yes in the dollar x example we had before when you pass in the identifier when you pass something to a macro when you pass on identifiers to a macro you're just passing sort of access to the name you're not passing ownership it's not like calling a function you're not moving that thing you're just giving the name away to the macro and then what matters for the purposes of ownership is what does the macro the code that the macro expands to do with that name it might move it but it might also not try carb expand Lib tests I don't think that's gonna do it oh nice that does work great good call yeah so this this expense what we expected and it seems like this is relatively straightforward for people all right so this case is straight forward let's now look at something that's a little bit more complicated right so what we want here is one that has not an empty back but like single so this is gonna be 42 you want to assert that it's not empty and we want to search in fact we want to search that the length is 1 and we want to assert that X 0 is 42 and then we immediately see that it's basically telling us this macro was not expecting anything and that's because we don't have a pattern that covers that so let's add something like that so we're gonna add it's gonna take an e let's just call it this is just a variable name right I guess we can call that element if we want to be fancy and it's gonna be of type expression so expression is anything that is an expression in rust and this is almost everything in the language that you can write inside of a function think of it as anything that you could terminate with a semicolon roughly and what is this gonna expand to well these I guess mute and we're obviously not allowed to use the back micro in here that would be cheating we're gonna say these is vector U and then we're gonna do these dot push element and then we're gonna expand it to be right so we can start a new vector we push the one element we got and this is just going to replace whatever the expression in here was so this is going to expand to V s dot push 42 and then we're gonna give back yes but you see that it's complaining here right if we run cargo check Carlo tests it says macro expansion ignores token vs and any following the use of avec is likely invalid in expression context it says only about let expressions in this position or experimental it's clearly not very happy and the reason for this is that what we've told Russ to do here is we've told it that this expression expands to these three statements and that's not something you can do in rust it wouldn't you don't even have a way to write what this expands to in the language let's see what cargo expand gives us might not even give us anything reasonable yeah you see here it expands to just Veck new and then nothing more but it includes like the left here this is clearly not okay right like you can't have equals let mute vs that makes no sense really what we meant to do here was we meant for this to be a block and inside that block we can declare variables and write other expressions and then the final value in the block is what gets returned and so you this is why in many macros you'll see this sort of double curly bracket and as before because the outer curly bracket is what the macro rules syntax require us us to do it just say this is what this is the chunk of things that something expands to and the reason it has to do that is because you might want to write you might want one call to a macro to expand too many items for example like multiple functions or modules in which case you don't want it to be a block so you don't want it to default to produce a block but you need to the the delimiter to be able to say all of the stuff in here in this particular case we do want it to expand to a block and that's what this additional curly bracket is okay so let's take questions on that before we move on so this roughly make sense why we need this extra extra curly bracket I can use a macro call inside of the macro you can there's some sort of weird rules around there but in general you can [Music] how is macro rules the macro rules macro implemented so the macro rules macro is as far as I'm aware not a normal macro it is basically a procedural macro sort of and and you can sort of see this immediately right we don't have a way with the macro rules syntax to have this representation we don't have a way to say that we want an identifier followed by curly brackets because all of the all of the macros we define using the syntax turn into something that you have to call with the limiters immediately so we couldn't even write this syntax using macro rules itself and you need to drop to something like proc macros should the no argument version of the macro used double braces or doesn't it matter so the this version actually doesn't need the double brace and the reason for that is this evaluates to a valid expression you can think of it this way without the curly bracket what's between here is not a valid expression it is in fact a sequence of expressions and that's why you couldn't use it in expression context basically where the RUS compiler is expecting an expression you can't put all these three things but you can put this in some sense you can think of it as the we're expecting the output type of this macro to be an expression because we're expecting it to be used in expression context like the caller is gonna place it where an expression would normally go and this conforms to that this is an expression but these three lines are not one expression and so therefore would not work in that context whereas with a block this becomes an expression our macros v2 still coming yeah so there's a there's a propose a proportional Wow a proposal for a new version of macro rules and I think the idea is that it's just going to be called macro or macros that is gonna allow you to it's gonna be a more powerful version with slightly different guarantees about things like hygiene and that it gonna work a little bit better with the module system and with you statements and stuff than the existing macro rules but they can't really change macro rules because a bunch of code relies on it so it's gonna be a new a new macro basically for defining macros declarative macros as these are called how does the square brackets work in the instantiation they're not referenced in the macro definition yeah so this is what we got out earlier that when you define macro rules the delimiter is just freely chosen by the user between square brackets round brackets and curly brackets and you can't as the definer of the macro you can't impose one or the other on the caller the caller gets to choose and all of them are valid and have the same meaning the right-hand side of the macro can be written with parentheses yeah so you can actually do this here to the fact that you can choose which delimiter to use here you can do the same thing here if I recall correctly yeah so you can replace actually rust format it's gonna undo this change for me but it's valid to use round parentheses here and I think it's even valid to use square parenthesis basically any choice you can use here you can also use for this outer block but basically every macro I've ever seen uses the curly brackets for this and as you as you may also have seen if I save this file rust format turns it back into curly brackets can you VAX it to the custom compile time stuff like X is some file or something like with proc macros proc macros are well proc macros you basically write a rust program where as macro rules does not let you do that it's entirely declarative all you can do is basically do substitution between one one vassan tactically valid rust syntax tree to a valid rust program or rust syntax tree can reflection be used to return an expression I'm not sure what you mean by that but macros do not have access to things like type information that were well declarative macros don't they really are just substitution if you want more access to like introspection I think even proc macros don't really give you that but certainly not in declarative macros can proc micro take identifies before the argument block no so this is what we mentioned earlier that with with declarative macros like this hoser can proc macros yeah with proc macros you can basically write you yourself define the input syntax and you're not constrained to the same the same syntax requirements that macro rules has the limiter ambiguity doesn't seem rusty it's a little weird I'm not sure whether the like macro v2 stuff is gonna have that alright I think that's most of the questions let's keep going can you quickly describe the difference between the types of macros so declarative macros and what we see here you give a pattern for the input and you give a substitution proc macros are programs that take rust syntax streams as input or token streams as input and produce a different token stream to replace it with so they are much more expressive but also more complicated to write and then there are a couple of other things too like with proc macros you can do things like add attributes like this or things like derives and so that's another way in which proc macros are special which is stuff something you can't do with declarative macros all right so let's now go go ahead and try something real fancy we're gonna go with double so now we have 42 and 43 and we're gonna assert that the length is 2 and that both of them end up so here again we get an error saying well no rule expected comma like you're not allowed to have a comma here so the I can't resolve this call essentially and the reason of course is because we've we've said in our pattern that we're only gonna accept one expression and 42 comma 43 is not an expression it is multiple expressions mmm it's at least not one expression so you can imagine the like we could go this route oh that's not what I meant to you we could go this route right and say it's gonna be ye 1 and then we're gonna do e 2 and this is gonna push II 1 and then push e 2 right and this will work if we try to run the cargo test it's fine but obviously this is not gonna scale we want our vector to work the our vet macro or a victim a CRO to work for an arbitrary number of elements and produce an appropriate vector and so this is why the patterns index allows for repetition so if you surround some part of your pattern with dollar and then parentheses then you can say right after the closing parenthesis you can give a delimiter and then either star or + or question mark I think you can also put a question mark like there but I forget and what this means is I want well plus means one or more this is sort of you can think of regular expressions here where plus means one or more star being zero or more repetitions of whatever is inside the parenthesis separated by this token so this is saying one or more comma separated things that make sense so this will match 1 comma 2 it will also match 1 comma 2 comma 3 it will match one because one is still a one or more longer list separated by commas but it will not support like one semicolon for we could have this list B semicolon separate instead if we did this now this would be a semicolon separated list of one or more expressions but in this case this is the format we want right Liz this is now valid syntax and then inside the replacement part of your macro you can use the same syntax so dollar parenthesis to give repetitions corresponding to the pattern so here I can say and and this also expands to syntax so inside here I can say V s dot push element and then semicolon separated they do that stupidly maybe ooh that syntax highlighting is just straight-up wrong I wonder why why is snot fine so what this is saying is repeat what is inside these parentheses the same number of times as the pattern that had element in it so this is actually fairly complicated the the mapping between these pattern inputs and outputs because you can imagine that you have multiple of these right like semicolon and then like X expert that is also like some comma plus and rust needs to know am i repeating this many times or this many times and the answer here is it looks for the pattern that has this variable in it that's how many times it's going to be repeated and so here I'm saying why I want to repeat vs dot push of element for each element that was in this pattern and similarly here I get to say sort of start to say I won't expand it that many times so if I now run in cargo tests this works just fine and if I run cargo expand you'll see that indeed and double X the the let X expands to a block this is the the double bracket the curly bracket we add Vecna and then pushed forward to push 43 and then returns me yes all right so that was a bunch of new syntax I threw at you so let's talk through it you can generate the macro with another macro rules you actually have to be pretty careful with having macro rules generate macro rules because the compiler sometimes gets confused and we might actually see an example of that later in the stream yeah you can L you can use any single rust token here so like for example this can be else I think not for expression but if this was like type for example this item yeah so this would be an else separated list of items it's just any single rust token is what you would use here what's the meaning of the Star Online 8 well oh this star um this star is really just to say that this is a repetition arguably it should be a plus to match the plus we add up here but it's really just to say this is a repetition and you can use the repeater in more than one place so I can do this and if we now look at the expand you'll see that it did the expansion multiple times right this this is still totally fine and be its because each time it encounters this pattern of dollar parenthesis inside of the expansion it it looks for which pattern my matching against and then pulls out the variables every time what happens you use both variables in a single repetition that I think is just an error like if we do this and then I do like let this then if I now run cargo check cargo a test all right I guess I have to give this well also gonna we're gonna deal with this as well so you'll see that this says meta variable X repeats zero times but element repeats one time and so it goes I don't know what you want me to do here because these are just different um I think maybe it allows it if they're the same like if I did 42 and like foo here and 42 here and like foo comma bar then I think it will actually allow the expansion so it's just the only requirement is that if you use variables for multiple different repeating patterns they have to repeat the same number of times there are some questions here about beckwith capacity we are getting we're gonna get there how are you defined something like the format macro the format macro is not macro rules macro rules is is somewhat limited in what you can define it you can get away with a lot in macro rules but like for some more elaborate things you just need to drop to a proc macro Plus this one or more star to 0 or more that's right any idea where that macro language got its inspiration from it feels a little bit magic I'm not sure actually where this sort of dollar parentheses came from it's a little bit reminiscent of regular expressions right so a parenthesis is a grouping and regular expressions and then plus is one or more of the previous pattern which if the parentheses is everything inside the parentheses where the like dollar came from and where the comma or the separator came from I'm not sure right so let's now do another test so single and double both work that's great let's not do triple and everything that would be annoying but one thing we want to do is like trailing so for example if I define a really long set of things here right so one two three four five six seven I wonder how long this has to be before Russ format goes this should probably be on multiple lines fine but now how about now great so once you get two really long things like this where things wrap you very often just like want to be able to have a trailing comma like this should be okay especially think of this is like if you have a list of things these are the numbers are here pretty large if I do this might be a better example if this was like static stir right then this should be okay I don't really want to have to special case the last thing and can't have it have a trailing comma I really want to allow this but the current pattern doesn't allow that because the current pattern says it has to be comma separated but there are no if there's a trailing comma it doesn't fit in the pattern because that would extend it would expect that it's a comma separated list of things so there should be an expression following that comma and the way the way you get around this there are a couple of ways you could just add a comma here which makes it required you could also put the comma here this is saying I want it to be separated by nothing but each pattern should be followed by a comma but then you require a trailing comma which is not what we want either so the way you actually do this is you add this pattern which is kind of stupid right so this is saying this is our normal pattern and then following the normal pattern we want to allow any number of commas in some sense you could think of like question mark is really other thing here right of zero or one and I think that's also legal great it's just my highlighter doesn't pick it up so this is saying 0 or 1 of this pattern and notice we're not using this pattern in the expansion at all we're just saying that it's allowed to be there in the pattern and the question mark is 0 or 1 so 0 is also ok so it's fine for there to not be a trailing comma do you actually need the comma before the plus or is it just indicating the user should use commas as separators you do need it if you don't have this here then now it's it's expressing its expecting a sequence of expressions that are not separated by anything so this would make the invocation this like notice there's no separate or apart from just whitespace so you could think of whitespace as being the default separator but we specifically want the separator to be comma it's not like the user chooses the separator it is the separate there is no separator can you match a specific number like you can with reg X brace I don't think so I don't think this is full reg X I think the only things you have a question mark + + star but the you like look it up in the macro book yeah it might be but I'm not aware of it can you check repetitions lengths to bring more intuitive error with two repetitions that are not the same length the compiler gives you decent errors here well one thing you'll find with declarative macros is that you sort of get what you get like you can't really give nice error messages you can try to tweak your macros so the error message just happened to be nice but if you really want to like provide a really powerful macro where things can go wrong in subtle ways you really want a proc macro where you get better control over what went wrong and and what code you omit and what errors are emitted what are the general benefits for defining your own macros it's just really handy right like one thing I use it a lot for is things like I want to generate here's here's a maybe trivial example but imagine that I have some trait foo and doesn't have any methods or it has methods that make sense for any number type in fact how might we do this has max value and it does like X value yes this and this trade can obviously be implemented for like all the numeric types and so I could write imple max value for you 32 FN max value the turn sell you 32 max right and then I copy paste this a bunch of times and I do the same for I 32 and then I do the same for you 64 and then I do the same for right like I can totally do this and just have lots of them but macro rules is a really handy way to just do this quickly right so I do a macro rules and I can just call it in pull and it takes a it takes a T which is a type and it generates this I'll explain this in a second and now I can just write like in pull you 32 in pull is probably not a good name here max in pol I 32 you 32 I 64 youth 64 etc this doesn't work because that's interesting oh right this is also stupidity right so now I don't have to write out that implementation multiple times because they all basically have the exact same pattern and instead I can just define the pattern and then repeat that pattern for multiple times so that's an example of where where you would use macro rules and it's not for performance just because you have a bunch of repeated patterns and you just want to express the pattern rather than then like imagine that I wanted to change something in this for every implementation for a numeric type well if I wanted to do that and I had a copy of the implementation for every type that would be really annoying to do yeah derive logic is written with proc macros what determines of a macro is called with parentheses a square bracket you choose as the caller every time okay so let's get back to our macro this now works the trailing comma just works so this seems pretty great right do we now have a macro that just sort of does the right thing but let's do this the other way around first but the other pattern that we want to support is one that takes an element a semicolon and then a count right and there are a couple of ways we can do this right so the the simplest one is like 4 in 0 2 count vs dot push it's 2 element let's see how that works so this is the semantics of having like a thing like Veck things semicolon count is that you get a vector that has that many of that thing and this like this works if I do clone to and I say 42 2 then this test will probably pass just fine um unfortunately there's a couple of things wrong with this the first of these is remember that the when the macro expands it basically does substitution right it takes this expression places it there and that's fine if that expression is like a constant but it's not fine ism' if this is a more complicated expression so imagine for example that we have non-literal let's imagine that this is some 42 and this is going to be Y dot tape unwrapped so this is basically the same thing as before right we have a thing here and we want to of it and that thing is gonna be what was inside this some so it should just work but you'll see that this actually panics doesn't work and the unwrap fails and the reason for that should be clear if we do our cargo expand because that test expands to for each in this loop do y dot take and obviously only the first take is gonna succeed the second time we take the value there is none and so this will just not work and so this is something you need to be aware of when you do macro expansions is it that it really does substitution in our particular case what we want to do here right is really something like let X is element and then we want to do X dot clone and really what we want to say here instead that way the expression that gets passed only is only evaluated a single time and then we remember its result and then we clone it for each time we push and the this will in fact work right that was also a lot so let's do questions on that let's see our macros in rust the answer of people who want object-oriented programming like inheritance no I don't think you want macros for object orientation at all if you find macro rules readable no so macro rules once they go beyond sort of simple ones I think these are decently readable um I think once you get to the more complicated ones they can become a bit of an eyesore and you really want to move to a proc macro um the biggest downside of a proc macro is that it it's a relatively heavyweight thing because now you need to be able to sort of parse and interpret well interpret not really parse rust token streams and so you needed a dependency on like syn and maybe quote and and proc macros add an additional compilation step so it makes using users of your macro now need spend a lot more cycles on compiling your thing as opposed to macro rules which are relatively lightweight let's see can you have a test where count is not a valid expression yeah so if we do a test that like invalid count where this is like I guess foo right let's make it a little simpler so this will not compile because it will say it expected an integer found a string and the way this actually works behind the scenes is that rust when compiling this any error that gets generated by the macro like in this case there's an error that is generated for this part of the macro which is like expected integer found string any error that gets produced for the code that the macro generates gets assigned to in the output the output error messages gets assigned to the corresponding place in the macro input right so the error is about count and so rust figures out that the count came from here and then points that error at that location in the output which is why this output error she ends up being pretty nice it points to the count and says expected integer found string even though this error really comes from the location much further up here up here that's where that error originated and yeah the other problem with the previous approach where we had element here is that if element was pretty expensive it would end up being called end times whereas now it only gets called once and then you clone well a clone : and literal like 42 clone be optimized out by the compiler yeah usually so 42 is an integer it's an i-32 usually usually which implements copy and the compiler it's smart enough to recognize that for for if it knows that the type is copy then it will just copy which in the case of something that's as small as an integer a copy is just like a register register move usually can use this newly introduced syntax to say compact arbitrarily nested for-loops I'm not sure I follow but usually like macro rules are pretty restricted in what you can write although you can write patterns here the like match a for loop if you wanted to great alright so let's move on a little bit so there are a couple of downsides to the approach we're currently taking I'm gonna remove this because it doesn't compile although there so I'll show you a little trick actually which is in dock tests so rust doesn't have a way to say that a unit test should not compile but there's a crate call like compile fail I think it's compile fail though lets you write tests that are not supposed to compile but one trick you can pull here is this doc tests can so if you do this well I guess we'll do allow a dead code and now do cargo T then you see that it actually runs that test when it runs the documentation tests and it checks that it does in fact not compile and so if we now change this to something that did compile like 42 my missing something where in this what happens oh right let's do what do we call this victim AK yeah so now this compiles and if I mark this as compile fail then that test is no longer going to pass because it's supposed to fail which means that I can now make this foo to make sure that that actually does not compile yeah it's pretty cool it's like a cheap way to get compiled fail tests there are a couple of reasons why this is you you may not want to rely too much on this pattern such as if it compiles for dear if it doesn't compile for a different reason than you wanted it not to compile this won't quite do the right thing but it's a heavy pattern all right so are we done like now our macro does the same things in the standard library and in fact let's switch over and see what Veck does right so vex supports the pattern element expression semicolon and expression okay we got that covered right that is this guy it supports any in fact so this is how they express trailing commas is either X expression comma star or X expression in with a comma inside the pattern star and that is also a valid way to express this they are basically the same the advantage of us doing it this way is that we can express it with a single pattern of the rather than multiple and knows that they don't have this version of the pattern they get rid of this entirely and we can do the same thing by making the star instead of plus to say that this is like zero or more repetitions of elements one reason than this is a little annoying is because we're gonna get this warning right that the vs that we declare in the case where this list is empty does not need to be mutable and so we actually want to in that case we would have to allow put an allow on this just in the case where the input list is empty and we never push and that works fine so this just reduces the number of patterns and it's gonna make the documentation easier to read right because now the documentation is only going to show the two patterns that we care about oh yeah try build is another good crate to know for compile fail tests should panic is different so should panic it must still compile it's just that when you run it panics that's different from compiled failed macro export is not always required it's just that without macro export you wouldn't be able to call this macro outside of this crate think of it as pub all right great so there are a couple of things that are unfortunate about this and the first of these is that we're calling new everywhere and then we're doing a bunch of pushes and this is sad because imagine that I gave you I I called a Veck macro and I give it like a thousand and twenty four elements then if you create an empty vector and then call push a thousand twenty four times you're gonna have to do a bunch of reallocations of the vector right the I forget what the default capacity of a vector is but it's probably somewhere around sixteen you after you've you push on the first element and then it has to allocate a vector holding sixteen elements then it pushes fifteen more then when it pushes the seventeenth element it has to realloc copy all the elements into a vector that's twice as large to the 32 and then it pushes another 15 elements yeah another 15 elements and then when it pushes the 33rd element it has to resize the vector again and it keeps doing this over and over even though if we know that there are thousand 24 things why don't we just allocate for a thousand 24 that seems much more obvious and of course this is much easier to do for the case where the user gives in the count because we know what the count is right so here we can really just do with capacity count although here we got to be aware of the same thing we looked at earlier which is now we're gonna evaluate the count expression twice which is probably not what the user expected so we do you let count is count and then we use count here and count here okay so now we have a much more efficient implementation of this sort of I want this many iterations we can still do better though which is push is a little stupid because every time you do a push we've gotten rid of the allocation part but it still has to do like a pointer increment and this sounds silly but if we're like creating a vector of a thousand 24 elements almost like in a busy loop then every instruction counts and there's no reason to have to do the bounds check for every push right and the allocation check for every push when we know that that's not necessary and there is already a way for us to get around this which is we can do vs not extend and then we can do like the standard error repeat element take count so this is saying the standard itter repeat is a really handy iterator method which is just it yields clones of the element you give for as long as you take from the iterator and then dot take is a method on iterator that says only taking this many things and so now extend knows that it's gonna keep adding to this vector in theory we could do even better if this implemented exact size iterator but we're gonna not deal with that today cuz it's a slightly more complicated topic this because repeat plus count repeat plus take I think does not implement exact size and you can think of here as if you try to extend a vector with something that you know the size of the iterator in advance you can just sort of pre do all the bounds checking in advance and you don't need to do it during the iteration but this is pretty decent for a starting point all right so questions about that change first do you think the dogs would be more readable with just that one pattern or with 2/3 I think fewer patterns are better unless the patterns get extremely complicated as a result do you forgive our cities ten on the first allocation yeah that makes sense the argument still holds yeah so standard eater repeat has a bound that the type of the element implements clone because the compiler optimized a series of pushes using a with capacity call it could I don't think it will because that's a much more sophisticated optimization right that that requires the compiler to know the semantics effect that it needs to know that there's some relationship between you and push and with capacity there's a much more sophisticated operation that required special treatment of that so I don't think it'll do that it would do that if we use Veck from iterator and managed to produce an iterator the implemented exact size iterator but that's a topic for a different day are you sure oh yes so the macro this is an important point actually the macro doesn't have any trait bounds right the macro doesn't say that element the type of element needs to implement clone macros in general can't express that instead what's gonna happen is if you try to use something that isn't clone then the compiler is still gonna generate this code and then just as if you had written this code with the given element in this position the compiler would complain that that element is not that type does not implement clone just so the macro expansion will generate that error and will point at element because the compiler generated an error here which is this variable so I mean we can try it right if I do a non clone and let's say I have a Y that is a I do a struct foo and I say Y is a foo and I want to construct a Veck of Foos that is of length two if I do cargo check it's fine it'll say up here that the trait foo implements clone is not satisfied it'll point me at foo and it says it even points you at required by this bound and standard it a repeat and what's really happening behind it is expands the macro the macro generates this this compile error for where we create the standard error repeat and then that error gets sort of bubbled up by the macro expansion to point to where the type that didn't implement clone came from there pretty sophisticated machinery but the error output you get is pretty nice and then you get a bunch of related errors like there's no method take because repeat doesn't implement iterator and other errors like we can't print foo because it doesn't implement debug so hopefully that was illustrative capacity from plain pushing goes zero one two four eight yeah so it actually does purely power of two expansion so that actually means that it will do ten allocations before it even gets to our thousand twenty-four to be really inefficient is that hygienic what happens if the caller defined a mod STD in a mod it err yeah so this is one of the things you have to be careful about with whenever you define macros is that the caller might have modules that override you here like STD might refer to a module in the caller scope that is not the standard library there are a couple of ways you can work around this this is one ways in which macros aren't entirely hygienic you'll see things like crate so crate refers to the crate where the macro was defined always no matter how the caller brought that into scope and then you can also do colon colon which is this is a root level path so STD must be a crate although they could still pull tricks like rename a crate but if someone renames the crate STD then they deserve the problems they get may arguably it should be this why not make resize oh yeah Beck resize would also work here we do Veck resize element count and also work in theory 13 did I do those backwards great yeah does another way to do it and this is probably even more efficient because as you observe this doesn't have to do the bounce checking the reason why the standard library deals with the trailing comma as it does is because it disallow this invocation with comma only yeah that's interesting so with our syntax above this is legal right just putting a single comma in there with the version the standard library uses that is not legal so that's like an interesting observation and the translation between the two is actually pretty straightforward right if we did this and then we said this should turn into create a vac right ooh recursion limit really that seems false oh I might have to define the other one firsts the order might actually matter here there we go yeah basically if there's if there's not a trailing common so the issue we're seeing is that this ended up giving us infinite recursion and the reason for that is if you have an expression that doesn't have a trailing comma then or rather if you have something like this word there are a bunch of elements then it's gonna keep invoking this rule it's not even true I don't know why it has to be in that order but let's not worry too much about that all right now that seems fine great alright so now we have something that works here obviously this with capacity trick is really nice we can even make this new now if we wanted to because the resize is going to take care of making this a single allocation so why don't we do this make this count because now it's only used once and it'll still work fine great so now we have a relatively simple expansion that's still efficient for our repeated example we still have this problem up here right of this really should be with capacity all right so let's try to do that with capacity but what do we put here what is the count so there isn't really a weight at least currently in macro syntax to say how many items are there here but this is where we can get tricky with macros and this is actually a pattern that is covered in the in the little book book of Ross macros which is how to count things and it turns out there are many ways to count things that are really sneaky so before I go to that because that's sort of its own trick let's see if there are questions about any of the things we've covered so far this is going to be its own like self-contained bit as someone wants us to have a clone bound earlier so I mean you could do something that you could do something like X is element and then FN like test C clone takes a reference to C does nothing with it and then we're gonna do test X right so this is one way to sort of test in your macro that something implements implements clone if you wanted to but generally when you write macros you just sort of write them the way the code would be written and the hopefully the macro rules error propagation should take care of propagating any bounds that the user needs to know about you can't really document them in the in the pattern for the macro usually if there's something that's non obvious you would put it in the documentation for the macro right so you can write here documentation here that will appear on the macro itself can you access data from the calling scope from within the macro no not generally is something we mentioned earlier in the stream okay so what do we do here how do we get the count you have a version of the macro the defaults to evaluate element many times if element doesn't implement clone yeah I mean you could you could totally have one that that that does the element that does like vs not push element to reevaluate expression this many times I think that's generally not what you would want like clone is probably just better like if you can reevaluate the expression that many times then why isn't the thing you returned clone if that makes sense but it doesn't have to be all right if there aren't other questions about that let's try to figure out how we can produce a value here like how can we do let count is something in such a way yes let's make this the new fancy to do macro in such a way that we can do with capacity I can't how can we do this well the first thing we're gonna need is this obviously sort of has to be a macro right like there's no function you can give a very attic number of elements - really so what we're gonna do here is probably invoke a macro of some kind so let's try to do this in let me try to give you like a stupid version first that we can then try to improve upon I'm gonna propose to you that we do this hang with me here for a second all right so what do you think this will do see the trick here right like this creates an array that has all the elements in it and we call length from that array that's gonna give us the number of things in element now this particular expansion doesn't actually work because we're gonna be consuming element twice we're gonna be evaluating every expression multiple times and we've already talked about how that's not okay right so this clearly won't work but is there some variant of this that we could use well what if for each element rather than put element here we put something else because we don't actually care about that the things that get put inside of this array all we care about is that it has the right length so we're gonna do is actually define a sort of private variant of this macro and you'll see why I define this as a new pattern rather than a nested macro in a second there are there are a couple of other ways to do this but we can do it I want to do as a pattern for now and we can move it out eventually so I'm gonna use ampersand in this pattern just to sort of indicate that you shouldn't be calling this because no one's gonna be writing like that followed by ampersand count and then something else it's clearly meant for internal use and there were some ways to improve this later on and it is going to take element it's going to be expression and we don't know what it's gonna expand to yet right but it's sort of gonna be something like this if it could be that that'd be great but that doesn't actually work and then here we're gonna do crate a vac of element right so I make sense like this is gonna be count so this is gonna invoke the aiviq macro itself but it's gonna end up hitting the pattern that's down here and I guess I put a let's make this a semicolon so that's gonna end up expanding to whatever this expands to but you still have this problem this is gonna repeat element multiple times so the trick we're gonna pull is you can define a macro so what what happens actually if we just like made this unit that'd be great right if that just worked but what happens if we do this well the compiler tells us we attempted to repeat an expression containing no syntax mayor variables matched as repeating at this depth what does that mean well remember how when we talked about these special patterns we said that rust figures out how many times to repeat this particular block by looking at which variables we use inside of it so here we're using element and therefore we look at the pattern that in surely took element and look at how many times that repeats well this pattern or this expression rather this repetition doesn't name any variable so rust doesn't know how many times to repeat and one way to exemplify this is imagine that this was really the definition right how many times should this unit be repeated this many times or this many times rust does not know and this is why it requires us to use the variable in there but we just said we don't want to use the variable so can we do any better well it turns out we can so let's define another variant here which is going to be substitute and substitute is not gonna take a repetition it is just going to take an expression and then what's it gonna return is unit so notice that this takes this does actually use its argument but it takes one right and then it just returns unit and now we're gonna expand this with ABEC itself with substitution does this make sense right so now rust knows how many times to repeat this because we're using element in there so it knows to look at this pattern but then when it expands this macro for each element that macro ends up using just unit it does not actually put this expression anywhere which means that that expression won't be called multiple times which is what our worry was and so now we end up with num this many units inside of this and then we want the length and that is a totally valid expression and it doesn't actually use the expressions from element anywhere and so when this expands to is then just the length of some really long slice which is in fact even known at compile time right let's see if this actually works no rules expected the token yeah it's because we probably need these right so now it says no rules expected this token that should be substance so now it's saying cannot infer type of lettin so this is kind of interesting we have this this ends up producing an array but it the type does not give the length of the array and so it can't actually infer which Ling method are we talking about and so we need to do bu this is where things get like this is where sometimes you see really nasty hacks in macros so we're actually gonna do this sorry this needs to be a I'll explain this syntax in a second this is definitely an ugly hack but it will work yeah okay so what this is doing is saying take this array take a reference to it and then call the implementation of Len four slices of units so notice slice not array and call it and because arrays implement a CF slice we're allowed to call any method that exists on a slice on the array by just you calling the ass ref trade so this ends up calling slice Len the the slice method Len on this array turned into a slice this is like mind blowing weird expansion so let's actually look at what this expands to so let's look at our double in the very early days so our double function if you recall from where we define it right double is just a vector with two elements let's see what that expands to so it expensive with capacity the length trick and then array with two units in it right so this whole expression is actually evaluatable at compile time by the compiler because this is because I hate that it jumps down like that because where's my double here because this is a two element array the compiler knows that at compile time it also knows that a two element arrays length is evaluatable at compile time to be 2 so this will actually be a compile time with capacity to even if it wasn't though this will still a value it to the right expression 2 which is the real length and so this will not allocate anymore all right so I just true through a really fancy trick at you let's see um I also want to point out there are actually multiple different ways to pull this particular counting trick this is the recommended one because it works for a raise of any length but there are some other ones that are pretty nasty so one Road Oh gross it is gross across it it is really gross but it works it does the right thing this is turning to be nasty it is but it is pretty sneaky and and specifically what this gets really nice is that unit is a zero size type so this doesn't actually take any size on the stack you're not we're not actually using any memory for this is entirely a computation that happens without there without that in fact you could get really fancy here and use like size of there's some tricks around size of that but then you end up allocating it it's in fact here let me pull this up in the macro book if you look down here at counting so you'll see this is sort of the replacement trick that we used and there are a bunch of things like just do zero plus multiple replacements with one right see it ends up expanding to zero plus one plus one plus one plus one and it turns out you will crash the compiler if you do enough of these you can do recursion you can do like a bunch of sneaky things like this by counting the number of tokens and you can like batch them but ultimately like you really just want to do slight length this has been tested up to ten thousand tokens but you can't do this in rust 1.2 but apart from that oh it cannot be used to produce a constant expression yeah so okay so this produces a non-constant expression that's too bad there are also versions to do that like this thing you can extract the enum counter to get the number of items so actually I lied then this is not the version we use is not constant but it is sort of the nice one this one is come this one does work at compile time but don't do it alright let's see yeah when it gets too nasty like you could write this as a procedural macro right you shouldn't need to pull this trick but it is cool that it's possible and one of the reasons I wanted to include this trick is because it tells you a lot about how macros work even though you might not necessarily want to do this yourself there's not a predefined count macro no not currently and part of it is because the it doesn't know what you want to count right you want to count the number of tokens do you want to count the number of like the number of characters in the syntax tree do you want to count the number of the number of items right and so it's sort of hard you need like generic macros in some sense to be able to say what should the type of the thing that we're counting be oh sweet ok great so Lynn is Const for slices great so that then what I said was true is there some way to test verify that the length is known as compile time yeah we can even do that just to finish up here so we can do thank you let me do this yep that count is Const yep so it is in fact constant great why not use int like zeroes for vet capacity you don't want to use a zero because zero then it's gonna keep reallocating do the counting sub patterns show up in the docs yeah they do which is a little sad I forget whether you can do doc hidden here but the other way to get around this is sort of what I was gonna say wrapping up is you can do this move these guys out make this macro export but also doc hidden and then here have this instead be avec be count I missed it somewhere great down here so now in fact if we try to do this see if this works right it showed up on the wrong monitor because of course it did ooh that's not at all what I'm into yoo-hoo sorry about the light but you see this only shows one macro and it only has the things that we expect so this is one way in which you can you can hide those sort of nasty patterns now yeah at this point you don't need the prefixes either why does it matter if we use the parentheses when the length is known at compile time so if we didn't use zero size types here then this would still be stuck allocated it might be optimized out by LVM but you would still need this array the array we construct here to be to be somewhere in order to be counted whereas given that it's a zero size type it doesn't even need to be stored anywhere even though like the compiler can recognize that this array is no longer used at runtime and optimize it away but by making it zero size we sort of guaranteed that it won't be appear in memory anywhere and also note that like this isn't this context is really just for our own sake it's not actually used for anything you could do like C and then make this be C and that would also work just fine nice all right I think that's everything I wanted to talk about macros we now have an equivalent of the standard library Veck macro in an a relatively performant version at that that we implement our selves so it's pretty cool we're still missing a bunch of documentation and stuff of course but I feel like that's sort of out of the scope of these crusts of rust episodes the thing I didn't quite get to you which I consider doing but I don't think it's terribly important is you could imagine extending this to instead of this defining a vector it defines like a hash map or a b-tree map or a set of some kind where you just change this to instead of being elements be like key arrow value and then you repeat that with inserts instead of pushes so that's something you can do and there's a crate called map lit that basically does that and I highly recommend you try it out to sort of as practice for yourself I'm gonna end the explanation there but I'm gonna take some last minute questions if there are any and then we'll wrap up for this time let's see no it looks like people are generally following if you're looking for an exercise like I see some people are wondering whether like how can I practice this try expanding or extending what we have here to work for a hashmap for example because it should be a fairly straightforward change to the syntax but it it will require you to like check your understanding of these concepts can you show us the standard library version oh that's a good question the standard library source version who knows what that does uses from LM so we could have done that too this is oh I see what they do this is a little weird I don't know why they choose to do it that way so this is gonna create a box slice so it creates a it basically creates an array that is gonna and they use the Box keyword to make that array beyond the heap and then there's a conversion from a box slice to avec but I feel like that will end up having to move Veck is two words and a pointer and box is one word in a pointer so I'm not sure why this conversion is free oh right because the the words you got on the stack okay so the answer to this is basically the representation of a vector right is you have on the stack you have the length the capacity and a pointer to the data which is on the heap right and a box is just a pointer to something that's on the heap well what this does is create an array of all the elements and the box keyword says do that on the heap this is something that I think you need a nightly yeah you need this nightly feature to get it to work and what that gives you is a boxed array right so this is an array on the heap and then it uses here it's saying treat that as a box slice instead of a boxed array and then call into that and the reason for this is if you have a sequence of things on the heap like you have that pointer you can trivially construct a vector right because you just construct something that has that length that capacity you should sort of count them and that pointer and it will just work and this allows them to get away with not doing all the tricks that we did that make stuff like u8 Veck be optimized to memset maybe yeah although we don't need to touch the stack either nice alright I think we're gonna wrap everything up there I hope that was useful thanks everyone for watching and I will catch you on whatever the next crust of rust is I'll make sure to announce it on Twitter and if you have ideas for additional sort of small concepts thing things do you think have relatively self-contained real code we could write then reach out and let me know sweet enjoy everyone I'll see you next time

Channel: Jon Gjengset

Views: 28,661

Rating: 4.9809299 out of 5

Keywords: rust, live-coding, macros, macro_rules, declarative

Id: q6paRBbLgNw

Length: 96min 11sec (5771 seconds)

Published: Wed Apr 29 2020

Reddit Comments

[u/ztlpn]

Thanks for the stream, great as always! I've kind of avoided macros in the past due to the fear that they are not exactly the pinnacle of elegance and at least now I know where the nastiness lies! The last part reminded me of the dirty tricks that you sometimes have to do in generic C++ code 😅

A couple of suggestions for future stream topics:

• Error handling. I've seen this mentioned on twitter so this is my +1 vote. There is a lot of different opinions on how to do proper error handling floating around and it would be interesting to see your take on the best practices. I guess writing a small library and a small utility using it and getting to the point where errors are nice is hard but doable in 90 minutes.
• Benchmarking Rust code. This is a fairly niche topic, but I still want to suggest it because Rust is a performance-oriented language and performance means benchmarking, right? There is no section on benchmarking in the Rust book and the Bencher struct in std is nightly-only and kind of abandoned. So it would be nice to see a howto stream on benchmarking.

Also, please, please don't completely abandon your long-form stream format! I get that shorter streams are easier both for you and the audience but a chance to dive deep into a problem and watch you make substantial progress on it is something unique and awesome.

[u/kodemizer]

Awesome video.

That macro counting hack is NASTY though.

[u/dandxy89]

Thanks Jon. Watching your videos is what got be interested in Rust in the first place.

[u/nickgerace]

Great stream today Jon!

[u/DKN0B0]

Great stream as always. Glad you also showed another use case (at 44:10), that definitely gave some good inspiration for where macro_rules can be useful.

[u/elibenporat]

Just had a chance to watch this. This one was even better than the first! I'm loving this format; you are comprehensive, without being repetitive and build things up step by step.

If you are planning more of these, I'd love to see one on Iterators. This is clearly a large topic, but for me, the biggest gaps are how to make a type iterable and how to add my own "itertools" type of functionality.

[u/jwbuiter]

What about using this for the first rule?

($($element:expr),*) =>{{
    let vs:Box<[_]> = Box::new([$($element),*]);
    vs.into_vec()
}};

This way, you avoid both reallocation and the counting hack