ElixirConf 2021 - José Valim - Elixir 1.13

[Music] [Applause] [Music] so yeah uh today i will be talking about uh elixir 1.13 exclusively exclusively but if we have time for questions i'm not sure i'll be glad to answer questions about anything uh elixir right so why elixir 1.13 uh so we are on galaxy 1.12 next one's going to be 3.13 and it's coming out this november so we were used to release every january in july but we shifted to every may and november so it can align with the otp releases that come in may so it's going to come um next month all right and we almost hit uh uh a thousand one hundred contributors so if you want to be there a thousand and a hundred you can visit the issues tracker and you can probably make it in time and here like i usually like to add some statistics about uh hex pm uh it started a long time ago when we had like 500 packages and now and like uh thousands of downloads but now you know we are almost 14 000 packages almost 7 billion dollar loads so those numbers keep on growing right and elixir 1.13 is uh it's a very different release for us for the maintainers because most of the time when when there is a new elixir version it's usually a mixture between like new apis for users for you the developers right for other developers and improvements to the tooling right there is usually a mixture of both and uh elixir 1.13 is different because like the huge majority of it is focused on the tooling okay so uh and we are going to see what are those improvements so what this means is that uh most of the stuff that i'm going to talk about in the stock you're not going to use it directly most likely somebody is going to use that to build a better tooling or somebody is going to build a package that they are going to leverage for that package so um so it's really focused on on the the tooling and more the low level aspects of the language right so uh this also means that this talk i'm a couple times i'm going to go a little bit uh low level but we'll try i'll try to balance it out and and try to get everybody back okay so what i'm going to talk about today we're going to talk about semantic recompilation we're going to talk about mix xref we're going to talk about code fragments and code formatting all right so let's get started semantic recompilation so every time before we have to talk about elixir compounding code there is a very uh brief uh introduction that we need to make okay and we're going to go back in time a little bit we're going to go back to elixir uh 110. and in lxc110 so imagine that you have this module a right and you you're calling b inside the module body you're you're importing b and inside the function you're also calling b okay if you're calling something inside the module body but not inside the function we say that those things they are compile time dependencies okay and what this means is every time b changes because a has a compiled dependency because we are calling b in the module body a has a compile-time dependency on b so every time the file that that has b changes it means that a has to recompile and that's why sometimes you're working on your on your project and you change a file and elixir is like hey i am recompiling uh 20 files okay so um so this is caused by by those compile time dependencies is when we call modules outside of a function okay and alex 0.11 which i think was about a year ago did one very big improvement in here where we changed imports to be export dependencies okay and and uh the reason why this was important is because sometimes we have like compile time dependencies that have other dependencies so we have like this huge graph and as i said that's why for example if you look here uh hex the the package manager that we use it's a open source application so i got hacks on elixir 1.10 and i got the user file and user is a file that is usually used throughout our application right it's a it's very common for us to depend on user so with electron.10 if i change that file only that file and compiled okay it would say i have to compile 90 other files okay exactly because we had this compile time dependency and then this compile time dependency would have other dependencies so you know it's it spreads around then we end up having to compile a bunch of stuff and with this change in lx7.11 without doing without doing any change to hacks okay we just upgraded extra version doing the same command now compile 16 files only okay so this this is an improvement we did in elixir 1.11 okay that improved the lot compilation and it's really related to how elixir is tracking everything all right so throughout the years you know it's not only elixir not 11 throughout the releases we have always been proving compilation we're always listening to feedback and improve what we can but today if you are in if you go to your elixir applications today there is still one big issue even after those improvements to talk about there is one big issue and the issue is forward compilations there is a big a big amount of scenarios there are many scenarios where uh changing a file causes your whole project to be compiled so every time you change a mix access file so for example add a new dependency or a change of configuration every time you change the mix lock so you up to data dependency remove a dependency every time you change any uh you change the config access or any other file imported for air config xs we are going to recompile everything okay and if you have our link files in that project every time you change one of them right it's going to recompile everything so depending on your application and how you're doing things you're going to be seeing four compilations frequently which is quite annoying and the worst part is that all of these is end time based all this depends on the time the file was changed on the operating system okay and the issue is that sometimes simply like uh switching branches right um and rebasing for example is going to trigger those forward compilations okay so it's even nothing changed at the end it's going to trigger compilation because we think that the thing changed on disk so what we did in alex 0.13 in the next release was to solve this like considerably okay to the point that now uh the only i'm going to show the the end game where we got to and then i'm going to explain how we got there okay so um now in electrode 13 we are going to trigger for compilation if you change our mix access file but if and only if you configure the elixir compiler so if you change how the elixir compiler runs if you don't change how the elixir compiler runs you can touch you can change the mix access file in any way you want okay and it's not going to do it for compilation anymore and we are going to do it for compilation as well if you change your config excess but if and only if you are configuring your own application okay if you you're configuring any other application it's no longer going to trigger for a compilation so moving trial 0.13 is going to reduce the amount of compilations as you work as you switch branches considerably okay so so how does this work the whole idea here is that we have semantic recompilations okay so um and what we mean by this is that no more m time instead of saying hey has this file uh was this file the modification time changed in disk we are actually going to try and understand what is in those files and do the changes necessary based on those changes so for example a very simple change that uh kind of kick-started all of this was a contribution by mark andrea which is um before your elixir files in lib the dot ex files we would check the m times so if then time change would recompile them but now we also check their content as well so this is a very easy and kind of silly semantic check but it can already improve a lot so uh if if you change the file if the time the modification time changing disk but the content is the same we proceed to compile the contents the the digest of the contents and if the contents are the same we're like okay i don't have to recompile this right um similarly for the config access file right if you configure a dependency now instead of recompiling everything only the files calling set dependency recompile so for example if you if you compile sorry if you if we change the configuration for phoenix for example only the files that are calling something from phoenix is going to recompile and sure in the phoenix application you're still going to have a lot of files that are uh calling phoenix but for example everything in your context in your domains are most likely not calling phoenix and that doesn't have to be recompiled and if you have a small dependency right it's nothing is is not going to recompile anything or just one or two files okay and again we are only going to trigger for a compilation if you configure our own app it's a solver problem we are just not there yet right so maybe not even this is going to trigger for a compilation in the future but right now it does similar with mix xs if we change the mix access instead of recompile everything we just change the files that are reading the project configuration right we are only going to recompile everything if you change the compiler options same thing with the mix lock if you add remove or up data dependency only files called inside dependency at compilation time recompile okay jose this is nice i think this is going to be great or i hope that you think so does it mean that you need to know all of these no you don't need to know any of this right you just upgrade to alex 1.13 one comes out and hopefully you will be uh a more productive developer all right and there is one disclaimer here for which is uh those changes they are changing to how mix operates your project so there may be bugs okay so i would really appreciate for those who can to go out to the elixir github and try out uh the master version you can also the website also tells you how you can use elixir from source and really try it out in your project and see if anything goes wrong how would be something going wrong here is so as you're working uh your project goes into a state where it doesn't compile and then you have like to remove the underscore build directory right so if things go weird and like oh i have to remove build for things to work then it may it means that you may have found a bug okay so i would really appreciate uh if people would try this and and uh give feedback some people have already tried this i tweeted about this some time ago some people have tried and the the feedback so far has been uh overwhelmingly positive in terms of workflows so uh i would really you know appreciate more people try and give more feedback because i think this is a very nice change all right it's still on the top of a compilation one of the things that we have improved uh on 1.13 is mix xref and mix xref it's a task like mix compile mix run mix paycheck server it's a text a test that comes with elixir and it tells you information about your files and how they depend on each other right so if you go back to the example right uh of the module tracking example where you have a and a uh is calling thing zombie if you invoke mix xref passing the graph option it's going to say hey lib a depends only b at compilation time right and this is kind of easy right you can say hey i can look at this file here and i can see that it depends on b right no problem but what happens when for example you get your phoenix router right and for some reason the phoenix router it's not supposed to have compile time dependency on most things but for some reason it has a compile time dependency on something and your router is like 400 lines of code how do you figure out where that compile time dependency is coming from right yeah you know so um you have to kind of know or maybe the compile time dependency is coming from a macro right so you need to know how the macro expands so and you would need to know those rules what is the compile time dependency x part dependency so that's not very easy right so in alexander 13 we added mix xref trace and what it does that you give it a file and it's going to tell exactly where all of our compile export and runtime dependencies are right it lists everything so now if you want to say hey there is you know there is a compile time dependency that is trigging my that is making my application recompile a bunch of stuff now you have this command the trace command that is going to allow you to filter things then you can pass the label option say hey give me only the compile time dependencies okay so again i think it's going to be really helpful for um for developers developers working on large applications and we also added this was another contribution for uh mark andre who did a a bunch of great contributions related to a compilation the mixer f uh mix xref we also added the few of both flag and what the few above flag does is that it's going to raise if uh if we print anything so for example imagine that you that you refactor your application so no longer have cycles in your code okay you removed all the cycles in your code you spent like a day of work fixing everything removing the dependencies and then like okay this is great and then you don't do anything and then one month later you're like hey i have 10 new cycles in my code right so with the variable flag you can say hey you know i want to check if there are cycles and i want it to fail if there are more than zero cycles right if i have any cycles so now you can put this in your ci and you can have your rci failing uh for stuff like this okay so again a huge thank you to mike andre he was also in the thinking elixir podcast talking about those things so i recommend checking out that episode too um all right so uh let's get out of compilation and let's talk about code okay so the let me check the feedback here uh just to see if i still with me if i should slow down okay everything's good all right so cold fragments so uh in order to this is a new feature this is a new functionality that we are adding to elixir and i i will explain the the at a high level what is the rationale for it and what are the issues that we have today and how we are addressing them okay so imagine that you are in your editor okay and then you're like okay i want to define a module right so you define the module and then like okay i want to define a function uh so called add inside the math module and then you type a plus and okay i typed a plus but now i need auto completion okay you say i need auto completion here right now right and um the issue with this code and the issue that your editor has to solve is that this code is incomplete code okay this is incomplete code like if you try to compile this code before you type what you want to type while you're asking the suggestion right the compiler is going to say hey this is invalid this is a syntax error but your editor needs to deal with this anyway right your editor needs to handle the fact that this is incomplete code and show you a suggestion and and use as much contextual information variable names whatever you can have about this right your attorneys needs to be able to figure this out okay so um what we are and throughout the years we we have this functionality right and so for example if you go to the vs code for example or anything that is using the elixir language server protocol they implement those things those features are there right and uh marlo sariva who works uh with me at dashbit he is he started the elixir sense package that adds those features but the thing is that in order to implement this uh you know in a way that is efficient in a way that it doesn't have to handle a lot of weird corner cases we need to make this part of the language all right because integrated with the elixir tokenizer the elixir parts or the internal to chain of the language if we integrate those things then it's much easier to tackle those problems right so code fragment is a new module in elixir to work within complete code and uh and one of the reasons why we're adding this tradictor as i said elixir is the best place right to because you already have all the two chains to understand the code it's the best place to add this but it's also because um at the beginning of this year we announced a project uh called live book which is a collaborative and interactive uh code notebook and elixir so you can document stuff you can write code you can execute the code if you haven't tried out a live book yet i really recommend you to give to to to give it a try and have some fun with it and what we want to add to live book is to add this uh editor-like experience right where you say hey uh i type uh innum.m and then if you ask for autocompletion as you can see here we want to list everything that is there and this functionality that i'm showing here it was implemented with code.fragment right so we're already starting to to to use this right and it's kind of funny because um well i'll tell the funny and adults let me let me show more examples so here's an example of us using code fragment to show our completion here's another example uh of using the code fragment module to show some uh mouse over context right so if you mouse over a map so if you mouse over over a dot over the m over the a or over the b we want to show the documentation for that thing right you just need to mouse it over so again you are putting the cursor somewhere and that code may be incomplete and we need to figure out where the cursor is what we are trying to call and bring the documentation for that right so those are the challenges being solved uh by by code fragment and and the and let me tell the anecdote is that the initial version of code fragment it actually came from ix because if we use the alex the interactive shell in elixir ix we already had auto completion right you can type the module name or the beginning of the module name and press tab you can type the function name all this auto completion stuff was already there so what we did is that we extract this out into the code fragment module and now we we we are using it and because we extracted extract this functionality as we are improving code fragment now based on needs of like live book we are getting new functionality in iex2 so now for example in electron.13 we have instruct autocompletion in ix so if you type the percentage sign and you hit tab it's going to list all these trucks that are available you can um we also get cg seizure autocompletion right so if you type tilde and then you hit tab it's going to list all the cgs available for you and then after you put the student name the sigil letter you can hit tab again and we are going to show exactly how you can close you can start that cjo so again those are very small features but you know as we add all those those conveniences together i think uh it really helps the developer experience right and the fun and thought was that uh so i i implemented this uh for ix right uh and then marloes uh from dashbit he implemented it in electricity that is being used by editors and now jonathan he was working a live book and he was implementing those things for live books so we're going to implement this thing for the third time and then we're like okay it's enough it's uh we should probably put this somewhere and we realize that using elixir parties uh is going to be the most efficient and productive way of tackling this all right so again code fragment the odds that you're going to call a function in the code fragment module directly is extremely low but i hope uh as those changes start going up to the tooling you're going to be using them every day all right and the last thing i want to talk about the last big feature is um code formatting okay so if we were at the event i would ask how many people are using the code for matter but we are not i cannot see a quick raise of hands but i would be curious on that so the code for matter was introduced in elixir 1.6 and the reason for that is that we want to ensure a consistent code style in the community okay and the code for matter is minimal configuration by design and the code for matter expects files and strings as inputs okay and when we launch the code for matter okay uh it it was very integrated again this idea that it has integrated with how the language parses and so on so it was using a lot of internal apis to elixir which makes sense it's a project it's a module inside elixir so it can use elixir internal apis and with time we we wanted to address this so in every new release since 1.6 we are making all this internal functionality uh that is used by the formatter we make it public slowly okay little by little and in elixir 1.13 we have reached a very big milestone because as i listed here at the bottom uh the code formatter can it can only format files and strings okay so you cannot give it an elixir sd you cannot say hey i i got this file i changed this file and now i wanted to format it for me i changed like the st of this file and now i'm ready to format it for me you you couldn't do that in alex 1013 we have two new functions we have a couple more but the two main ones are quoted to algebra encoded to algebra with comments which again the odds that are going to call this directly are very low but what this means is that now the code for matter supports the strings files and asts inputs okay this was a contribution from uh lucas and roman and um and what it allows us to do is exactly that you can now say hey i i you can like automate changes to source code because you can say hey i want to read that file i want to parse it i want to change it and i want to format it back and lucas has started this project called sorcerer that has many conveniences to automate that right so for example somebody wants to implement a mix that's ad thing that automatically adds a dependency to your mix access i don't have to change it manually now we have the tooling to start doing this stuff okay and it's still in the uh code formatter area uh we have one very nice addition is that mix format now supports for matter plugins and what those plugins can do is that they can format custom file extensions and they can format custom sigils so for example imagine that in your application you create this mcgill for markdown okay and you are using a markdown and dot md and dot markdown files and imagine that you want to format your markdown and for the same reason we want to format your elixir code uh paul um in the lightning talks you talk about he talked about the surface formatter as well right so imagine that we have all those files in your application that you want to format and and you don't want to say hey mix format for elixir and then you want to have another command for formatting marketer files another command performance html right so uh we don't want to do this that's why we have formatted plugins now and now we can say hey i have a mixed machine formatter it implements the mixed task format behavior and it's going to change the code that is inside it's going to format code inside the m sigil it's going to change the code with the md and markdown extensions so you tell the features that the plugin provides and now you write the format logic and uh and now you have this integrated mixed format experience and using a plugin is very easy you just say hey i want to use the mixmacron format plugin and you now include the the the extensions that that plugin can format in your list of inputs and you're ready to go right and i'm really excited about this because as i get i as paul talked about the surface format and as chris mccord talked about uh the new higgs templates now we can have all those things integrated so now we can run mix format and it's going to format your elixir code it's going to format your hcgs with fixed templates it's going to format your dot hicks extensions so this is going to enable a bunch of really again tooling right uh tooling tooling and tooling right i'm almost uh channeling my inner steve ballmer here and talk about tooling but yeah it's all about better tooling proven tool and so on and of course uh we have general improvements and so just to the very time improvements that we added here and there so for example we now after almost 10 years we have a power operator right so for um power mathematical operation as part of the language and part of kernel we added things convenience things for uh validating two words saying like hey the keyword has those keys and those keys they must be there they're required you can now uh conveniences for mapping and filtering a map and getting a map back and also new additions like task completed and task ignore like sometimes for example there is an api that is expecting a task but you already have the task result so you can just say test completed to create a task that is already completed so you can give it to that without starting something separate to process it and we also have now task ignore and what task ignore does is that for example imagine that you want to start a task to do something and you want to wait five seconds and after those five seconds you're like you know what that task can finish i'm no longer interested in the result it can finish on its own right you can't do that today right in elixir what you can do today is that you can wait for five seconds and you can say hey i want i'm going to crash or you can say hey i want to shut down the task but you can say you know what let leave it running but i don't send me the result anymore i i don't care i'm no longer interested and now you can do that with task ignore precisely okay so that was it uh uh for uh what is coming in elixir 1.13 and i want to take the next uh five minutes to talk about what is next for the alexa language and if you see me giving a talk in the last two years this is going to be the exactly same stuff right nothing has changed right so what is next for elixir so new versions will continue to come every six months right so elixir is a stable uh no plans for a major release so those six months again is they are all improvements like we just seen this in this presentation we are continuing to improve uh what we can things that people uh bring it up and so on so no plans for a major release there are no plans for an actor to oh at the moment right and we do have a general direction right so one of the general directions is tooling that i've talked a lot about today and the other direction is compiler assertions and um i've talked this a couple times i'm going to give a quick refresh i usually i like to say that elixir code is very assertive right we usually have pattern matching we have guards so a lot of the times we know what are the types that we are working with and i think the compiler can use this information to find bugs earlier for us during compilation and we already had we already added improvements to elixir in this direction in the last releases and we'll continue to do so right so that's a general direction we don't have like a end goal in mind like hey we want to get this it's just something that we are chipping away and exploring um you know as things pop up and as we have ideas and so on generally speaking we have a mostly empty backlog so if you go to the issue tracker today um we are going to see that a lot of the issues in there is mostly about catching up with erlang otp right so one of the things that we also want to do is that every time there's a newer lego tp release we want to make sure that we are leveraging the the greatest and and the best that uh there is in it right and this brings me to the next point which is the reason why we can afford right to be stable right to have a mostly empty backlog is thanks to the ecosystem right we build on top of our lingodb and really the the bulk of the work right the herculean effort is happening on our lane otp right so all the performance improvements that we see making the vm more efficient uh the new features related to distribution to the process model it's all happening there okay and as an example right the just-in-time compiler that came with erlango tp24 is amazing uh for me everything is easily 33 percent faster but often more um so it's just fantastic right and the other reason uh where elixir can be at this space where it is right now is because of you because of the community right we enable the community to build on top of elixir so elixir is an extensible language everybody can get elixir and bring it to new domains to new areas and again one example that still blows my mind even if i was involved with it is the nx the numerical elixir project where now we have a subset of elixir that runs on the gpu right and this is is amazing the fact that we could do this without doing a single change to the language i think uh is a good testament for the language being accessible language that is going to allow us to to bring us to new areas new places and continue improving those areas right and in and um to continue this trend of tank in the ecosystem uh i would like also to plug in the airline ecosystem foundation the alien consistent foundation has been doing a great work in improving both areas that we talked that we talked about both on angle tp and the community so just some examples um we were just talking about the jit right and their inconsistent foundation uh it sponsored some of the work so the jit can um digit can work on arm 64 machines like apple m1 okay so this was work sponsored by the ecosystem foundation the ecosystem foundation also sponsored thomas to work on float to his training conversion which something that you would think it's in straightforward convert a float to a string but it actually isn't it has like literally decades of research into it if you want to get a float and get a a streamer at the smallest possible stream representation of that float and over the last years we have advancement in the research and thomas brought this uh to their language machine and this leverage and most apps they're going to leverage this right because well if you're sending things to the database uh you you need to convert a flowchoice string so you can send that to the database if you're using json if you are printing a float on an html page it's there all the time right and this is work now that is their language to be sponsored by an inconsistent foundation too uh and for example there are a lot of working groups in there the machine learning working group everybody's there talking about annex and everything that is happening in this machine learning area so i recommend uh you to join the language system foundation if you have not yet you can be um you can be a non-paying member but you can also join as a voting member and uh participate actively in the future of the foundation and i would also advise companies right to join as sponsors a a company cannot be part of the foundation as an entity that votes but you empower the foundation to continue doing work and improving the ecosystem for sponsorships alright so that's it so thanks everybody uh for your time and your attention and see you around

Channel: ElixirConf

Views: 4,808

Keywords:

Id: ydjx2kKHzrM

Length: 35min 56sec (2156 seconds)

Published: Fri Oct 22 2021