C# Source Generators in Action, by Andrey Dyatlov

[Music] i'm andre i'm working as a software engineer at jet brains mostly working on c-sharp features for a sharper and wider lately i've been working on nullable reference types and sewer generators support and resharper and i've been very interested in meta programming for quite some years now and because of that when source generators became available i was very curious where do they fit among existing tools what are they pros and cons and when i should use them so in this session i'll briefly introduce search generators to you there will be live demo sessions in the first one i'll focus on quickly building a simple yet useful source generator from scratch and after that i'll compare search generators with other meta programming tools we have available today and in the second demo i'll focus on some advanced scenarios and best practices when using source generators but first let's discuss why we need source generators and what problems are going to be solved by them and first and foremost it's generating boilerplate code so if you have some standard implementation of quality members or two string methods in lots of types in your solution instead of coding them yourself you can simply delegate this work to a source generator and in this case you won't even have to update them every time you add remove some members to the container type of course they can also be used to do type generation from scammers to do some object mapping code to produce code for object materialization from some service and because we can generate boilerplate code easily now we can also have some performance optimizations for example we can get rid of reflection in serialization you can register types for dependency injection without reflection i'll get back to it in a bit later and first let's start with a simple example that is likely familiar to most people it's an interface called i notify property changed it has a single event notifying its listeners that some property of an object has changed and perhaps it needs to be updated on the user interface some other logic should take place and because it's just a single event it's really easy to implement in any type for example a car model with a speed property but now we have a problem because this auto property doesn't fire any notifications about its changes so instead i'll have to write it as regular property with the banking fields and etc i'll fire the notification and that's already a lot of code and what seven words every time i'll need to change something for example the name of this property and it's type i'll have to update almost every single line this code and it's a lot of places what we can do here we can only leave back in fields in our model and generate notification properties with the source generator because they all follow exactly the same pattern and i'll show you how it looks like these and without source generators so you might notice that there is a lot less code with source generators and this code won't grow very fast when you add new properties for example a model name for this car or its number of doors so when at some point you'll add some business logic for example an acceleration method that will increase the speed of this car by 10 percent you'll still be able to see what this type is about what members are available there and you won't have to scroll multiple screens of boilerplate so what are source generators as i mentioned before it's a new meta programming tool from microsoft basically you are going to write the types that will be a part of the compilation process it can access a model of your code and generate new files based on it so rossum begins a compilation of your project it reaches some source generation step of compilation it transfers control to your type along with everything it knows about the target project and based on this information you can generate new source code that will be added as an input to compilation and it will resume as if these files was there forever so let's get to the first demo and i have a website called source gen dev where you can upload a bunch of examples provided by microsoft and one of them is actually implementing the notify property changed interface and here we have the actual source code for our program there is an example of your model type there is a bunch of backing fields and an auto notify attribute there is a program that simply subscribes to the property changed event and changed a couple of properties to verify that everything works and we are getting the notifications and we can check what was added to this program by source generator its automatic attribute and the partial declaration of our example viewmodel type so it's essentially a part of this type that implements the notify property changed interface and adds a property changed event and the properties set files and notifications and on the top right you can see the actual source code for this generator but it might seem that there is a lot of code and in order to work with this you have to be familiar with this rosslyn api how the compiler builds its syntax trees what types are available because if you check it for the first time you'll see a lot of types you probably don't know such as i syntax receiver attribute data type constant you'll see some save cast to see sharp compilations sharp parse options add syntax trees sharp syntax repars text source text from and what if you don't know all these types it might seem that you have to be familiar with russian api to start working with them and in my first demo i want to show you that everything you need to know is about three methods total and everything else will be easy to find this code completion and derived type search so here i have a car model type these three biking fields i've demonstrated on one of the previous slides an acceleration method and a lot of boilerplate implementing the property changed notifications for these backing fields and this program simply subscribes to its property changed event and fires a couple of modifications to verify that i'm getting notifications for model name changes for number of doors property changed for speed property change and let's remove all this boilerplate with the help from source generators i'll need a new project where i'll declare my generator it has to be a new.net standard 2.0 library uh because this assembly will be loaded in the visual studio process you cannot use anything fancy such as dotnet standard core five and you have it has to be dotnet standard 2.0 don't forget that and i'll need a couple of packages in order to work with the compiler api i won't focus on this now but you'll be able to find a source code for this demo uh on my github and you'll also be able to check these packages i mean any example focusing on source generators file source generator i need to declare a type that will implement i source generator interface and it has to be annotated with a generator attribute the interface itself contains just two methods initialized they won't need it there yet i'll get back to it in the second demo and execute where the magic happens and here you have access to context compilation which is basically everything the compiler knows about the target project for example you can access any types that are available in the target project please get typed by metadata name for example here i am instructing system component model i notified property changed interface since i'm going to implement it in a couple of types so let's notify interface and you have access to compilation syntax trees essentially one synthetic is one file in the target project so for each file as with nh3 you can access its root and you can check the ascendant nodes themselves and since i'm going to work with type declarations i'm only interested in class declarations index elements and here you might want to stop me and ask okay how we should know that this type even exists if we are not familiar with how roslin works and there are two methods you can use first you can check what descendant nodes and self return to you and it's a collection of syntax node element you can check derived types from it and these are every syntax element available in c sharp language usually they have quite descriptive names so you won't have any problems if you search for example for record and there is a record declaration syntax if you want to extend records only if you want to do something with interfaces there is an interface declaration syntax this class there is a bunch of types of the first one is class declaration syntax which is exactly what we need but what if you don't know how to name some method for example a method called how would you name this element if you cannot guess the name of the syntax element you need you have another website that can help you it's called sharp plot you can switch it to syntaxtree mode and type any codes there you are interested in for example console writeline and you'll be able to see that console.writeline is an invocation expression is a simple member access expression qualifier and an empty argument list so if you are not sure what syntax elements are produced by the compiler you can simply put your code on sharpclub and check the generated syntax string so let's get back to our example and syntax elements is essentially what is written in the code editor so there is a class keyboard there is some name identifier and what we need is semantics what members are available in this type what interfaces does it implement what attributes are there and to move from syntax to semantics you can use compilation get semantic model from this syntax string it's available for that and you can move to semantics with a semantic model get the clark symbol in this interval declaration and these are the three methods you really need to know semantic model get the card symbol compilation get semantic model and compilation get typed by metadata name so everything else you'll be able to find this code completion for example since i'm working with types here i'm interested in type symbols and i'm interested only in types that already implement my notify program changed interface so there is an interfaces property and i can check that one of them is my notify property changed interface i can put them in a hash set and this will be the types i'm going to extend so let's name some target types and for each such a type i'm going to provide declarations that will implement this interface so target types for each i'm going to provide source code generate property changed for this target type you don't have to provide any uh compiler structures you need to provide a simple string as is called in the editor so here i'm going to use string interpolation uh i'll define using system component model here to use the notify proper changed interface and the namespace that will be the same as a companion namespace of my partial type i'm going to extend there will be a partial class that is the same name as the time i'm extending so it's essentially a part of the same type and since i'm going to implement the notify property changed interface i can also move as a property changed event itself in the generated part so let's simply copy it it's this to do because we are working with simple strings and i'll need to provide properties that will fire these notifications so generate properties for this target type and here i don't know yet how many properties there will be so i'll need a string builder and i'll need to check what members are available in the target type code completion is your friend here you can type members and there's a great members method i'm interested in fields only so i can type field and there is an i field symbol which is exactly what i need and i won't rely on any fancy attribute based configuration here so instead i rely on simple naming convention if something is name it visa field suffix i'm going to provide properties that will fire notification when i'll update its values so let's check that the name of this backing field and is i'm breaking field suffix and for each such a field i'm going to generate a property with the name that will be the same as baking field name except for the suffix science so without the breaking field uh suffix that's our c sharp right feature and now i have to provide the actual property implementation apparently my visual studio is lagging don't worry and builder append line and again i can use one of the properties i declare in my actual program to use it as a template for what i'm going to provide simply paste it here escape the curry braces and update it because it's not always a string property i need to use field symbol type as return type for this property i need the name i've just calculated i'll return back in field from its getter i'll update my field sample etc and i'll find the notification for the property name so i have to return the string builder and now i have left is to add this source code to my actual target project this context add source i'll need to provide a name for this file for example the same as the target type name is notify csufix and provide this source code now let's add this and generator to our main program and i'll also have to tell the compilers that it's not a regular reference i'm not going to use types provided by it and it's actually an analyzer so it will do something with my uh code its output item type is analyzer and if i did everything correct i didn't make any typos i can now simply remove all this boilerplate from my type mark it as partial and run this program again everything should work let's check it yep everything works everything compiles i'm still getting notifications whenever i change any properties for example the model name the number of doors the speed property and the only problem you might have noticed is that my code editor looks a bit broken there is a lot of red code and that happens because rosslyn which is the underlying engine that powers visual studio will only update generator references once you open the solution and when i open this solution there was no source generator so i have to close it and open it again uh if you are not using resharper you'll have to uh restart your id here don't worry you don't have to do this every time you change something and your code editor need to restart the generator this is only required if you change the generator itself and if this is a problem for you for example if you want to develop some code and get immediate feedback on every time you change something in generator you can switch to jetbrains rider which supports hot reload for surge generators and now everything works so let me show you that you don't have to restart your id every time you change something so you can add a new double property weight of this car braking field and as soon as you add this property you can see it in code completion you will be able to get notifications for it it will simply work as soon as you've added the breaking field and now you might have a couple of questions about the generated code because right now there is some code in the car model types that we don't see and how we can check it and check that everything works properly how we can debug it if there is some problem and these errors are usually quite hard for meta programming tools because you not always have the source code available and since visual studio 16.9 you can simply find the reference to this generator in your solution explorer and there will be a list of files generated from this generator but in large projects it might not be convenient because there will be a lot of files it will not be grouped so instead i suggest you to work with this generated code as if it was your own code so i can navigate to the car model type and i'll see that there are two parts of it one in my main program in and one in the generator folder i can navigate to this type there will be the full properties i've declared i will be able to check the source code for them and i'll be able to set a breakpoint here and simply debug this code without any problems the next time i'll run this application let's check it and yep i can check what value i've assigned to the baking field and i can debug this code as my own and another interesting question is okay how can we test the generation process itself and for this you can ask the compiler to compile some arbitrary source and run some generators over it i have a project prepared for this in advance you can check exactly how it works on my github after the stock server will be linked so uh i simply provide some source code i'll create compilation from it um the code is a bit cumbersome so let's not focus on it i checked on my github if you want to test your source generators and then i can run my generator over this code and check that the returned compilation has some syntax trees that include my notifycs file and i can check its source code i will be able to check that my index backing field has resulted in the new properties that fires the notifications and the regular fields that doesn't follow my name in convention uh wasn't processed at all and of course in this case you'll be able to set a breakpoint in the generator and hit it when you run this test but what if something is wrong in production but not in our test code and because our tests are fine they don't know what the problem is how can we debug it and for this you can add a line debugger i'll launch to your generator and the next time you compile your project it will be prompted to attach a debugger example you can choose the same visual studio instance and let it debug its own build process so let's try it and see how it's different from our test run example uh you'll be able to see that in a real project there is not one file but three and only one of them is my program cs and two others are auto generated files with some assembly white attributes don't forget to remove this line after you finish debugging the problem because as i mentioned before austin will only update generator references once you open the solution so if this line is in the compiled test simply is the moment you open your solution you'll be prompted to attach debugger every time you change something in the code editor which is quite a lot so don't forget to remove this line and fix the behavior and test and before we'll uh go to the next section i want to mention a couple of things i've overlooked for the sake of simplicity in this generator and you might want to handle them in the actual generator for example here i've only used class declaration syntax there's a usa sharp 9 feature called records you might want to support them as well for this you also have to update this keyboard to record you might want to check that container namespace is not the global namespace for the assembly in which case it will be empty and this code won't compile you might want to check that this class is nested in some other type in which case you'll have to declare a container declaration as well and you might also want to check whether the interface is implemented directly or is inherited from some base type that's the difference between interfaces and all interfaces the same with breaking fields you might want to check whether they are in your own type or then created from some base type these checks are relatively straightforward so i don't want to waste time for this because i believe that you don't have any problems doing them so uh let's get back to our demo and i hope that i've demonstrated that in just about 15 minutes you can declare a generator that will save thousands of lines of boilerplate in an actual source code for example in real world projects there might be a project space dozens of types implementing notify proper changes for dozens of properties and now uh you might notice that implementing this interface isn't actually all that new and before i used 4d to implement exactly the same interface without all this boilerplate so should we use supers generators because they are new shining provided by microsoft or do they have some unique advantages over existing tools for example text templates highly in kyle generator and i am going to compare them to io living which is probably the most common way to extending some functionality so uh the main difference here is that search generators can only add new files and i'll even rewrite this byte code after the assembly is compiled if you haven't worked this volume it looks like this you simply implement the interface and some type declares a property change property event and declare some auto properties and after this code is compiled these properties are transformed to a regular properties with backing fields that will fire some notifications from these sectors so it works like magic and i want to highlight a few key differences with source generators here for example in order to alter some byte code you first need to compile it and for that you have to declare the property changed event because without it the project won't compile it don't have any byte code so this iodine it might be hard to declare something that is required for compilation for example some members that you need to implement from interfaces but on the other hand these source generators you won't be able to declare regular order properties there because the only option available for source generators is to add some new sources to your solution so if you already have an auto property told it doesn't fire any notifications search generators cannot change that instead you'll have to rely on baking fields and either some naming conventions or hp configurations to tell the generator how to generate a property from this backing field and one other interesting difference is that whatever you add from a source generator will be visible at compile time usually it's quite beneficial because uh you don't care as if you've declared a member yourself of as it was generated you can simply use it but for property change that might be a bit cumbersome because you have baking fields and code completion you can accidentally assign some new value to it instead of the property and miss notification so back to pros and cons of generator server iodine actually source generators are super in many aspects for example compiler will check as a string you provide to it instead of writing some byte codes that might result in various program exceptions if you do it wrong you have no problems navigating through the generated sources and debugging it something that might be very hard this island because you don't have actual source code and the only disadvantages of search generators are they were partial to extend any type which is probably not a big deal and second they cannot change existing code whereas eileen can remove a change anything for example to optimize your code and this brings us the question whether i'll win is obsolete now or not but as i mentioned before alvin can alter existing methods and this is often used to add some functional aspects to your code for example two spoiler exceptions to do some login injection to drop trace messages for example this body you can draw a line in a log every time you create a disposable object and every time it reaches finalization queue so with these logs you'll be able to find where you've created the numbers that you forgot to dispose and according to the official documentation of source generators all these tasks uh code writing tasks they include optimization eyeliding call cite your writing and login injection and while they have many valuable use cases they do not fit into the idea of search generation and this is a bit disappointing so let me show you that actually you still you are still able to provide some extension points for source generators for example for login injection and by this we usually understand that you want to lock something at the beginning of a method for example the values of the all arguments and at the end of a message log its execution type as there were any returned values let's review any exceptions for example this post sharp it looks like this you simply add a log attribute to some method and after the compilation there are about 75 lines of code it's a big enterprise framework that boils down to the following first let's load the parameter values at the beginning of a method then under try statement execute the actual source code that was there before the transformation and after that isolog said the message has ended successfully or that there was some exceptions so can we do this with source generators example here there is some service that might be part of some bank service for example to get clients accounts and get the balance and in objective programming if you want some extension point you simply add another layer of obstruction so let's introduce an interface with this service and now i can generate a decorator for this interface from a generator for example i can implement the same interface in approximate implementation i'll accept the actual underline business object in my constructor and then implement every single method these decorators that will add all the required login and then simply dedicate the control to the underlying object with this technique i'll be able to add whatever i need before and after the call and i'll only have problems changing something deep is in the method which is probably not needed very often and i'll also have some problems too when i won't be able to extract something in an interface for example static members so private methods for this for example you can try to use uh abstract classes and extend them from generator and with this i believe that i living will become a niche tool because most modifications can be easily implemented with generators there will be a slight implementation difference eye link is difficult to debug and test but at the same time implementation details won't be visible during compilation so for property changed i would probably stick with 4d because i cannot imagine a situation where i'd like to debug this code but for example if i'm going to implement some equality members i'm going to use search generators because i might be curious how let's say collections are compared and some extensions are actually almost impossible to replace with generators for example this kodi 4d you can add null checks for every method this is something you probably don't want to do with decorators you are able to add configurability to every call in your program and since this call might be located deep is in the method you won't have an extension point for search generators there for example this post sharp you can add deadlock detection by dropping a line in a log every time you work with some synchronization primitive and unfortunately for these tasks i living is still the tool to go and otherwise is not actually the only way to generate some code we also have runtime generation which is usually done with a type named i'll generator if you haven't used it directly you might have used expression compiles that uses it under the hood and it is often too used to add some functional aspects that require a lot of boilerplate i've personally used aisle generator type to provide some login decorators and encryption over some messaging interfaces and if everything you need to extend is available at compile time you can simply replace the aisle generator these uh source generators it will work the same if something you need depends on runtime arguments for example if you want to provide methods that will process some regular expression and it might be provided by your customers from the user interface and you don't know them until runtime source generators are entirely a compile time tool so in this case you'll still have to go to the source as a i'll generate type and one other tool i want to mention is reflection it might be a bit surprising because reflection doesn't generate any code does it but actually we often use reflection to write less code for example for serialization whatever framework you are going to use mute sub json system text json date contract serializer xml serializer in order to service an object you need to know what fields and properties does it have and this will have to check this reflection but actually we could have provided a serialization method for every type in our project and updated every time we had to remove some fields or properties in it usually don't do this because it's a lot of code and a lot of maintenance but if we can generate the serialization methods automatically from a source generator it won't be a problem and we won't have to use reflection and there is actually a project that does exactly this it's called json searching and it adds a serialization methods to json to types in your project and according to its maintainers measurements the first startup time and the serialization time is a few orders of magnitude faster than using system text json so if everything you need is available at compile time another example is dependency injection container building which is often done by registering every single type in an assembly that is found with reflection as they implemented interfaces that are again found with reflection you can simply provide uh some customized code from the source generator to either serialize your objects or to build a dependency injection container and if something depends on runtime arguments for example runtime object inspection or if you want to work around some compiler limitations such as access to private members uh reflection is still the tool you'll use so this aisle generator and reflection everything depends on the way you use them check whether it makes sense for your particular scenario to replace them with source generated code i'll viewing i believe will become niche but they still have the unique pros and the only tools that has likely gone is text templates because just as search generators they only create new files they have their own syntax without direct support in most ids and for example roster our compiler is already using source generators to provide uh syntax types from say xml declarations something where a text template would be applicable you can check pull requests that use the source generators there and the only advantages text templates have they can create any files and not limited to just c-sharp and you have much more fine-grained control on when text templates are executed because source generators are going to be triggered by the compiler every time something has changed in the code editor and this might be too often for some cpuins expensive tasks and in the next section i'm going to explain how to deal with some common problems for example how to handle dependencies in your generators what if you want to use multiple generators how they will fit together how to let your colleagues know about some generated members and how to make a generator configurable what to do if you need some additional data that is not a part of your c-sharp project how to let your customers know about some problem and finally how to make a generator faster and probably the first problem you might have is you don't want to declare your own generator for some common tasks so there are some good news generators can be added as a regular nugget package and charity also generator is as easy as is any russian analyzer but if you are going to write a generator you are going to share with the community don't forget that you are writing code for someone else's project for example don't forget to check the targets projects language version because there's a common misconception people tend to think that source generator size c sharp 9 feature actually you can use them in these earlier language versions as long as you are using the modern compiler so these people will probably not be happy if you add some uh pattern matching or covariant return types and just say solutions and actually source generators aren't having a c-sharp language fusion you can declare generators that will write some visual basic code as well and don't forget that your code has some dependencies for example if you're going to provide serialization members you might want to use some common libraries such as newtons of json for let's say arrays of primitives and don't forget to take a public dependency on this package so everyone who will reference your generator will also get this assembly and won't have to wonder uh what they need to reference in order for the generated code to compile and another interesting question is what to do if you have multiple generators so you can declare an order of the execution what to do if you want to use something that will be from another generator in a different one and the short answer is that generators are completely and independent uh everything the compiler passes to your generator is immutable so you cannot affect any other generator in any way and every file from all the generators will be added all together to the resulting compilation so when any of the generators is executing there is nothing that will be added by other generators yet so if you need to add something that relies on another generator you can simply use it within your source code it won't be a problem because when this code will be compiled everything from the other generators that will be already added and if you want to extend something from different generators for example to add login injection to some interface and send add caching to it uh the fun approach you can use is decorators so simply provide one decorator from one interface another from another generator and you'll be able to combine them in the actual source code and one final advice here if i will go to the second demo is use a new partial keyboard it got extended in c sharp nine before that we could only use partial for private members that won't return anything so if you don't provide an actual implementation say invocation can be emitted and now we can use them for public methods doesn't matter whether they return something cannot and because we'll have to return something this invocation cannot be skipped so what you'll get there is essentially an abstract method that requires an implementation within the very same type so in this case i'll have to provide the second part of this type and provide an actual implementation for this method it might seem a bit strange why would i need to declare something only to immediately implement it but as you've probably guessed by topic of this talk as the second part might be source generated in which case the original declaration serves two purposes first of all there is now a place where you can put an attribute to control what is generated and second a few years later either your colleagues won't have to wonder what methods are available in these types they immediately spot this declaration and will be able to use it to navigate to the actual source code and let's get to the second demo for a few more examples of what you can do with source generators uh here i have login injection implemented with the search generator there are some services that uh demonstrate how it might work in some bank application i can calculate balance for my clients i can check what accounts they have and i have an extension list this login provided by a generator that simply wraps an interface in a decorator that will add some login for it as i mentioned before these decorators so let's check how it works i'll need to compile this project first uh the generator is operating with a lock attribute so i simply annotate any interface i want to extend and process it from a generator and here i request uh how many uh money there are on some accounts for one of the clients john doe uh it also checks what accounts they have there are two accounts and then 568 milliseconds i want to check what's the performance problem here i calculate that they owe me 5 000 us dollars and for the second client chain though there is actually an exception that is thrown and i now know where it happens is which arguments i can't about this problem so that's login injection and here's the narrator that does it it's on purpose very similar to what i've built in the first demo it simply checks every single file in the target project and check every type declaration in it after that it checks what types are annotated with my log attribute by checking say attribute classes and if it's an interface i'll simply provide this login method and decorator so this code is not really interesting but let's discuss how we can improve this generator and checking every single file in the target project is not very good because there are large projects for example resharper solution has about 650 projects it compiles about five minutes on my machine so if i add a generator that will work another five minutes it will already be somewhat bad if i let 10 generators i'll have to wait uh about one hour for this to compile and i'll never agree to wait such a time so checking every single file is not what we want and instead you can subscribe to compiler notifications about syntax elements that are there in the standard project is the initialize method in its context there is a single method a register for syntax notifications that you can use to provide a syntax receiver linux receiver has a type that should implement an i-syntax research interface and this interface contains a single method on visit syntax node so when the compiler first checks uh what is there in my project to provide you this compilation object it also checks every syntax syntax element in the target project and it will find notifications for every generator about the syntax elements it found so i can check that syntax node is actually a type declaration syntax and i can check that it has some attributes so i might need to probably access it in my generator some attributes and i can store them for later for example i can add this declaration to a hash set store type declaration syntax elements and this run will be shared among all generators so the compiler will only check every file in the target project once and notify every generator about it so here instead of checking every single file myself i can instead access context syntax receiver cast it to my syntax receiver type i've just declared and i'll be able to work with an already populated hash set of every type declaration in the target project that has some attributes instead of checking them and finding them by looking through every single file on the starlit project and it will work exactly the same and it might be tempting to check that it's the log attributes i annotated this and it might seem that you have everything you need from this index element so attribute list we can select many attributes in it and you can try to check whether any of them has a name that equals to our log attribute and in some scenarios it might work but i want to warn you against it because in real projects log attribute might not always be written like this like the written in the full form log attribute it might be written with some namespace and it actually can be written with an alias because c sharp supports them so you cannot check that something is named log you need semantic model for this and it is not available there yet so simply store every single type that has some attributes and then check them in your actual execute method and move this semantic model get declared symbol method as early as possible because working with syntax is not always convenient and another problem with source generators is they are triggered very often by the compiler because it doesn't know whether some changes in the code editor require you to restart your generator and produce something different so if i type console right line this generator has just been triggered 60 times so that's three times for every symbol i've typed in an editor which is a lot and instead uh you might want to check whether someone is still interested in what you are going to generate this context cancellation token because maybe your customer has already typed something else and there will be another run of this generator and you can check this this throw with constellation requested and this method is actually very interesting because it shows an exception and it's probably not a problem if and nobody is interested in what we are going to generate but what if there is an actual exception in your search generator so what if it runs in let's say no reference exception or in any other kind of exception you you as a customer be able to debug what's going wrong and guess what the problem is let's check it and when i compile this project i'm definitely going to get some errors about us login implementations not been there and the only error i have telling is that typo namespace name login input default which is supposed to be generated from my source generator is not there and as a customer i might want to reverse that i've annotated everything properly as i've configured the generator whether it was run at all i forgot to add some output item type analyzer for example or if there is some other problem and my generators are simply not executed and it's very inconvenient so you actually have a warning telling you what's wrong sir it's telling you that generator login proxy generator failed to generate source it will not contribute to the output and compilation errors may of course result which we have served and this warning is easily lost among compilation errors it for triggers so i have no idea why the compiler team has decided that the mixing of warning is enough here and i recommend you to lift it to an error with ms built warning accessors note that regular warning says errors won't work here you'll need them as build prefix and now the next time you compile this project you'll get a very clear error telling you exactly what is wrong with your generator and you'll know that it was triggered and it's configured properly but as there is some error in it and you might want to contain the generated browser or debug this generator yourself now i have an error telling exactly what is wrong and speaking about misconfiguration of generators here i simply ignore every log attribute that is not placed on an interface and not generate anything this might be quite inconvenient for your customers if you are using this generator for the first time so for example they might accidentally place this lock attribute on the actual implementation types instead let's check what we'll do our generator in this case and i've moved says the log attributes let me recompile this project and check whatever's i'll get and whether i'll be able to understand what's going wrong here and yeah the only error i'm getting is telling me is that the type of name space login input default is not found again i have no idea why the generator has not generated anything and this is a problem that you as a generator ulcer might can solve you can check at some diagnostics from your generators but essentially generators are called analyzers you can provide a diagnostic id and category such as slope generator you can tell your customers what is wrong for example lock must be applied to an interface and you can tell the compiler whether it's a boring coin error and last but not least you can provide a location where the problem has happened every syntax element has a method called get location that you can use to guide your customers to the exact place where the problem has happened and now if i compile this project again i'll see a few more errors telling exactly what is wrong and how to fix this problem i can simply double click on this server telling us that log attribute is misused and i'll see of where the problem is i'll see how to fix it let's fix it again and run our generator and by the way there is another interesting problem this is a lock attribute it is not clear of where your customers are going to get it right now i have to provide my generator with an instruction telling my customers please don't forget to use this lock attribute declare in in your uh code and use it to configure my generator which is quite inconvenient and it might seem that we can simply uh move this attribute to our generator as well after all that generating code for this project so low message is this log attribute and i can simply add this source file head source log cs and everything should work let's try it and unfortunately uh the only error you'll get there again will tell you that the login input default is not found let's check that it actually happens and there are no puzzle errors so the log attribute is resolved correctly and you might have no idea why it doesn't work and as i mentioned before every file from every generator including your own will only be added after all the generators has finished their work so until there there is no like log attribute in the target project so when you ask us target project to provide you this log attribute it is not there yet and it's quite inconvenient so in the future the compiler team wants to provide you with some api to add some static sources for example some attributes you might need in the initialize method but for now we have a method called compilation at syntax trees where we can add some syntax trees to the target project and for this we'll first need to parse them just sharp synthesis parts text this block attribute and i also need to provide the sharp parts options to tell the compiler uh what language version should use for example they will be the same for every file in the target project and they are available from your context so this will be our log syntax stream that i'll add to the compilation and as i mentioned before every method you are going to call on the compilation objectives what you'll get is a new compilation object that will differ from the original one by this one attribute and the scope of this compilation is limited to your generator so you won't be able to affect any other generator or resulting project you'll simply be able to use the log attribute that you just added don't forget to add it to the actual source code of the target project as well and now everything will finally work and you'll be able to pack the attributes you need to configure your generator as part of it let's check that everything works i should get the login attributes working now i've had hit in the console window and i'm still getting everything and there is one other thing you might want to configure in this generator and the problem is that this is some sensitive information i print the names of my customers they contact the mails how much money they owe me so at least in production you might want to hide this information behind some encryption and this generator might not do this yet and you also might not want to do this in debug because uh yo developers might have hard time deciphering every log when they try to develop your application for this you can provide the generator fees and access to some ms build properties with the compiler visible property include the property named log encryption and you'll be able to set it for example a directory build prox file for your solution let's do this and i'll set it to true only in release mode so encryption room and now i can access this property from my generator and change something based on it i can use context analyzer config options global options access this property this is a build property prefix i'll get a full encryption string i'll be able to parse it for boolean value i'll get whether someone wants me to encrypt this log and i won't write any real encryption in just five minutes so let's simply i use this variable entry and change something so we'll see that something has changed in release mode for example i can use system convert to base64 button uh utf-8 get bytes from my original message i'll press this parameter from natural method now let's switch this to release mode and you'll see that uh we now cannot read the slope without some tool uh because base64 encryption is not real encryption algorithm it simply transforms some strings so don't use it in production for some sensitive data so let's simply check that something has changed in release mode and in real production you might want to configure an actual encryption key for your generator and this again might be a bit tricky because you don't want it to be as part of your generator yep you cannot treat the slog anymore in production and if you want to provide encryption key you might not want to see it as a part of your generator because it might be searched by the generator and you might not want to see the encryption key in your repository because a lot of people might have access to it and for this you can provide an additional file for a generator that will only be available let's say on the build server and you can provide some file let's say of encryption key that you'll be able to read from a generator and uses an encryption key this context additional files you'll be able to find first files that has both key extension and use its text as an encryption key and with this you can gain access to something that is not part of the target compilation so it is not there but you'll be able to use this file as an encryption key and last but not least i must warn you about some security problems because not every generator you are going to find um might be good for you and you are not i from it doing something like this youtube send sources to my email and i'll pack every single file in your project in the timeline to hack gmail.com here i've commented the actual cmtp send and i'll leave a line knock knock now all these sources are belong to us to let you know that all your sources have been stolen by a generator and you might wonder what's the big deal here we have russian analyzers for a few years by now they can install our source code it hasn't been a problem yet we can run some third party code and it might execute something for example the console rate line as i did here as part of this sort code library we referenced so how generators are different from this and one interesting distinction is that you don't need to uh actually call anything so let me remove everything from my main method and i'll still be able to print these two lines not not now and they will be executed as a part of your application these its rights it might be a web server running these administrative privileges or there might be some database connection strings nearby these databases some sensitive information or it can be an application you deployed to your customers and i might use it as a backdoor to them any other year i would have hard time stressing enough how problematic this could be if you followed news earlier this year you might have heard about solarwinds bridge and some hikers are getting access to a build process of the company and injecting some code in say product so it can be turned into a bugger for the customers among them there are some very interesting companies for example some of the united states government structures and what source generators essentially are they are a part of your build process that will inject some code into your program so it's essentially already backdoor if you haven't checked what uh is the source code of your source generator here i've used the model initializer to simply run some code even before your application is fully loaded it's in user sharpness feature and unfortunately one other difference with a third party library is that uh surfboard libraries code is static until you update them and with the source generator you might check that the generated files don't contain anything problematic but you cannot guarantee that the second run of this generator on a build server will be the same so it might try to check whether you're running in release mode whether it's running on a build server and change some code injected in your product and to be safe from it you can ask the compiler to emit every file generated by a source generator on the hard drive for example in the generated folder these any compiler generated files tool and this is an option i recommend you to set on your build server so you can check what is generated during an actual product build that you are going to ship to your customers so let's check that this folder is there also you also need to check that these files aren't temporary please because generator might spawn a process that will try to clean up it and you'll see a folder with this generator and some suspicious file called hex cs so don't overlook it in production and i hope that this won't scare you off source generators because this is essentially the risk we have um is every third party code so if you are going to use them uh leave it says 8785 or into an error rupees i must build warnings as errors if you are going to write source generators don't forget to check context cancellation token and use i syntax receiver to make your generator faster produce diagnostics to tell your customers what's wrong with your generator and pack every attribute you need as a part of your generator and to wrap it up a server generators let you create boilerplate code easily which makes possible to do some performance optimizations at startup for example by eliminating reflection and they solved a lot of typical meta programming problems such as navigation to the generated code and debugging test coverage for code generation process and finally they're easy to use because write something as a string is something we do every day in the code editor it's not as difficult as writing some byte code and because of this many many programming tools become niche but since generators can only add new files still with us and if you want to use some examples of source generators available today there is a very useful list of github projects containing them there is an svg2c sharp source generator you might want to check that transforms svg files into types returning a sky picture there is a json sort surgeon i mentioned before that provides serialization to json without reflection there is a generator provided by microsoft inserted to be involved generator that adds inserted two methods and all required types to your project so check it out and if you now want to write your own source generator check search generators cookbook from microsoft there are a few examples that i explained in details if you're interested in something i've built during this session for example how to test your generator you can check these examples on my github account how you can try to go to search and dev and use some examples there to play with them and i'm looking forward to what you are going to build with source generators but don't forget that this great power comes great responsibility every link you'll be able to find on github used by this qr code and thank you for watching [Music] all right let's add ourselves back yeah there we are thank you andre for a fantastic talk it's a it's a really deep topic and i think you managed to squeeze five hours of amazing content on source generators into one hour so thank you for that absolutely definitely definitely uh we do have some questions that came in so let's uh let's see if we can uh we can get them answered um sure first question that came in was really about the tooling that you've been using so um i think you mentioned sourcegen.dev at the end in the resources are there any other tools that you recommend to see and inspect syntax trees and uh yeah get familiar with what you can do with them well sharp block bio is a website i've used to check both rostline generates for some arbitrary code and that's where you can check uh how some syntax elements are named which could be very useful if uh something will change in the for example c sharp 10 version so if some code is processed differently you'll be able to check how the new element is called and you'll be able to update your generator according to it and that's what i usually use when i want to work with the compiler so nice and really fancy there all right that makes sense um then a couple of more practical questions i guess um that that came in from a couple of people um are there scenarios where you do not recommend using source generators uh first i would definitely recommend to you to check the source generator's code if you're going to add something to your project because of some security risks and if you cannot afford to do the code audit or at least check what is generated that's probably not something you want to add to your project and personally i don't like search generators for the property change notifications because i'll see my banking fields and i will never want to debunk it it might be a bit cumbersome to use if i want to extend every single method in my project for example is now checks because i cannot move for example private message to interfaces to provide decorators so for some tasks i would definitely still prefer to use iodine yeah cool that makes sense um yeah you mentioned seeing the code and so on so i guess that ties in nicely into the next question that we got how easy is it for unit test generators do you generate plumbing for those or are there libraries that people can use or i haven't seen any libraries but since i've written a couple of generators myself i simply added a single line of code that compiles some arbitrary source and runs generators over it so it's a single method i don't think you really need the library for that and you can check it on my github and add it to your unit test projects and after that it's a simple compile some source code and run generators over it check what it returned so right i can i can add there what i really like is verify by simon crop that is really cool because you can grab the generated code and then compare it with a verified uh snapshot of what has been generated before that's really cool cool all right we we have a couple more um matthias i think i think you were creating a list of questions as well yeah i i also have one uh that's more a bit more far from from the topic i mean slightly a bit uh philosophical um are you still looking forward to have like proper aop support in c sharp i mean uh source generators have a bit of limitation making making the original class partial and stuff like that but do you think with um with options that we have like post sharp and foley you mentioned both something like proper integration for aop is still interesting i believe it definitely is still interesting because your generators have the limitations i mentioned a couple before for example it will be really hard to change something within the method in the middle of a method or change something that requires you to provide an extension point for a private method and because of this i would definitely still use aop but i don't believe that something like this is going to be provided as a part of c-sharp compiler now because they have their own tool and every other scenario is already covered by existing eye leading tools such as 4d and post chart yeah it still has its use but i don't think it will be integrated into this language right now yeah i guess it is pretty niche i mean there are uh usually you just have a handful of um how to say very specific use cases for aop at least that's what i remember uh when i when i studied about them um one thing that i remember do we still have a bit of time yeah yeah go ahead one thing i remember and i'm sorry maybe you mentioned it but i had to run for the door uh really quick in the meantime um one thing that i discovered is uh usually you're limited by this partial declaration of of methods and types right what i found also works pretty nice and it was more by accident is that you can derive from the original type and just make it hide uh the other one so do you know what i what i mean yeah i understand what you mean it seems a bit hecky to me because uh imagine that a few years later your colleagues will find this and they'll try to navigate to some method and they'll have no idea how to look for something you've generated so i would not do this but i've been trying with an idea of working around source generators limitations for example if you actually want to provide some extension points within a method let's say to add loading every time you work with some synchronization primitives another interesting approach you can use you can simply generate a copy of the original type and simply put all the required strings there and use this copy instead of the original type so the generator will basically see everything that is declared in your project you can simply provide the copy and augment it the way you want as long as you are using the generated type it will not have any problems with extension points all right all right yeah i have a couple more questions that are that seem to be coming in so that's uh that's always nice if you have questions do feel free to ask them in the chat and we'll see if we can uh if we can ask them to end andrei uh one of the questions we got was on compilation time you mentioned that source generators essentially run individually and run as part of the compilation of course uh do you notice any any any any noticeable slowdown in compile times or something uh well as i mentioned before i've actually measured say i run on average three times for every symbol you type in the code editor so if you don't see any uh critical legs and code editors and probably source generators are working quite fast and you can elevate this problem by building out already this context cancellation token you can also uh use a syntax receiver i mentioned before to make the exploration type liner across all generators and quite frankly there is now a lot of things going on in your code editor every time you change something there is a roster analyzer that has to provide you with code completion that has to know everything you've changed which members are available to highlight something there is a resharper analyzing that does similar things so if your generator doesn't do any let's say web requests and updating gigabytes of data it's not usually a problem if it is one approach you can use is to wait some time before doing some heavy loading check whether the cancellation is triggered or you can switch to text templates because with them you'll have a much more clear uh and fine-grained control over when you do some heavy loading andrei thank you again thank you and we'll see you around you

Channel: JetBrainsTV

Views: 4,664

Keywords: JetBrains, software development, developer tools, programming, developer

Id: 052xutD86uI

Length: 71min 56sec (4316 seconds)

Published: Fri May 14 2021