RustConf 2021 - Project Update: Lang Team by Niko Matsakis

so hello my name is nicolas matsakis and i am the co-lead of the language design team along with josh triplett i'm also a member of the rest compiler team working on rust for for a little while i started at mozilla research but i recently joined aws in january of this year and i'm gonna be telling you about where i think rust is and where i think we should be going so let's get started so 2021 was quite a year uh we went from last year a project that was primarily supported by one company mozilla to a project that has its own foundation with you know a current count six platinum sponsors including many of the biggest names in computing so it's not bad but if that's not enough many of those companies have formed teams employing people to work on rust the language right uh pretty cool so we've got a lot more people being paid to work on rust this year than ever before and including myself and one of those teams and if that is not enough to convince you the 2021 is some kind of landmark year for rust what about this rust is under consideration for inclusion in the linux kernel so that's pretty cool i mean given all of that i have to ask you is this it did we like make it is that what happened here uh well that's an interesting question i think to answer it we have to go back a little bit and say what was the ergol you know what was the purpose of rust when we started out um it wasn't just to get in the linux kernel although i'm not sure i thought that would ever happen but it was something different it was to take the power of c and c plus plus right and put it in a package that has that great feeling of javascript ruby and python the easy to use feeling and and get the best of both worlds you know have our cake and eat it too i used to say all the time uh and you can kind of put that in this like graph form right that's i see these kind of graphs a lot i think they're good so you got c plus plus on the on the one side really strong performance but not always so easy to use and then you've got on the other side javascript ruby and python which are really easy to use you can ramp up really quickly and get people started and there's lots of people who've been hired to work on your project but the performance while sometimes great other times especially in like critical code that's isn't what you need right so the premise of rust is kind of bam there we are top right corner all the performance all the ease of use all the power all the time and when things are going well you know there are times i think we actually do achieve that which is really cool so i recently saw a blog post that i think is a good example of this comes from tenable there's a link down there and you see that they had a logging service and they rewrote it from javascript into rust saw a 95 reduction in memory usage 75 reduction in cpu usage and maybe you're not so surprised because you're thinking well okay sure if you rewrite javascript into a language like c plus or bust it's going to get faster but you know not only that they did it all in a single sprint in two weeks that's pretty cool and it works right um so that's kind of what we're going for that you should be able to get this high performance thing stuff that was always possible but took a lot of engineering and maintenance effort and do it with the same skill set that you use to develop javascript and so forth right so well that is what we're going for um this is kind of how it should feel right i think we there's some sentences that come to mind when i think about rust like we want it to be performant you often hear people kind of say wow i wrote this rust program and it ran really fast right out of the box um well at least they remember to use cargo run release they say that other times they might complain and then find out that indeed it does run fast um or they might say if that after they've been working in rust for a while they've kind of found this curious thing that they've noticed that once they get it took a while sometimes to get things to compile but once they do they tend to work you know and if there is a bug of course there are bugs but they're not usually a result of like surprising less semantics but more like somewhere in your logic you made an error and what that means in practical terms is you can take a big program do it kind of ambitious refactoring get it to compile and then find that it works the first time or you could take some loop and run it in parallel and be confident that the compiler is going to detect any places where you might be using the same data from two threads in compatible ways and so your code just works right and what all that adds up to the performance the reliable is that rust is really meant to be empowering right we're trying to take complex things that used to be hard to do and make them easy to do and so you can still do them um and you know we want to make kind of wizardry stuff possible like you want to use that nifty kernel api that everyone's talking about you can do it in rust and you won't have to deal with 100 sig folds because it's wrapped up in a nice safe interface do you want to run code on an embedded device with no operating system go for it you've got a whole bunch of uh people in the embedded working group developing all kinds of cool abstractions to make that easy for you to do right that's what rest is all about but it's not always like that right uh if if what we're going for is the ease of use of javascript performance of c plus plus we're not quite there maybe more like here you know we've got the great performance i think that we nailed hands down uh and sometimes things are really easy right you're on the right path like in that terrible blog post but using rest especially early on still means spending a lot of time fighting the borrow checker and learning the tricks of the trade kind of how to work around what that error message means and how error messages can be really great but sometimes they're really confusing or what the right solution is to that particular situation and i think you just can't claim that on the whole rust is as easy to use as javascript ruby or python it's just not true you can see that in the data all right so here's some results these are survey results from 2019 i don't have ready access to a chart from 2020 the most recent survey but i'm sure the results are comparable and so we asked people how long it took until they felt productive using rust which is kind of related to ease of use right how long till they felt productive you can see in this chart that there's a good chunk of people who get productive pretty quickly this is sort of the tenantable case less than a month that's great um and that the only reason that's even possible trust me is all the hard work that we've put into the design of the language of course but also all the stuff around it the error messages the documentation cargo crates like without all that work this number would be a lot a lot smaller the number of people who get productive this quickly unfortunately you can see that for a lot of people getting productive and rest is measured not in weeks but in months um and there's a pretty decent chunk of people over here on the left that say they've never felt productive at all now when i read that i thought well i bet they're all beginners right i bet they're just using rust for for a week or two wrong that's not quite right so when i thought that we cross-referenced the data with well how long have those people who don't feel productive been using rust and you can see that it's all over the map right some of them they've only been using in a few months so yeah they're kind of beginners that's the largest stream but for some people they've been using west more than for more than three years and still don't feel productive right now it's difficult to interpret survey data that might not be the language that might not mean they don't know how to use rus they might just mean like the libraries they need are not there or the ide tooling isn't up to what they expect that's what one person told me uh and that might be true but at the same time i think these these numbers give pause right room for concern if we want people to feel as productive and rest as they do in python and i do we've got to do better the fact is russ really rocks well once you learn it but getting that done can be pretty difficult so i recently heard this best phrase ever which describes the situation rust is the language where you get the hangover first it's perfect in so many levels i mean first the obvious one getting used to rust it takes time right and once you're past that hangover that you get up front and you get kind of the drinking period that normally comes first you get that then like you have a new pair of wings right you can build and maintain ambitious projects and you start using it not just for ambitious ones i hear a lot of people saying uh at first i used russ to write this you know powerful thing of doing the work but then i had this old script that i wrote and i needed to modify it and it was kind of annoying and i wanted a library so i decided to just rewrite it in rust then i could use cargo the code's about the same length but now it runs faster than it did before and i'm able to come back and modify it again and again without having to be afraid so this is exactly why that metaphor is awesome because it goes the other way right uh many other languages or you know in practice what often happens right is it feels really great at first you're writing a lot of code the code's running really easily but then the hangover comes when you realize that there's these subtle bugs that don't show up until after it's in production or you have to go back and rework it and you can't find all the places you have to make a change so all things considered i'd rather have to hang over first i'd just like to see what we can do to make it a lot shorter or maybe have no hangover ball that would be even better so i think this whole setup you know is exactly why rust keeps winning that most loved question on stack overflow is because what that question asks is if you're using rust do you want to keep using it people who are using rust once they've kind of gotten past that hangover and they're into it they're in that sweet period that comes later so they do want to keep using rest but you know that's a really cool number we love to cite but if you look at the most popular one what languages you use the most plus doesn't do nearly as well now granted we come into five percent of respondents number 19 which is way better than i ever thought we were going to do when we started working on this project right but still i think west deserves to be a lot higher in that list i think there's a whole lot of projects that would benefit the people with from from being written in rust they would go faster they would use less resources and they'd be easier to maintain and people would like it but they're not doing that today so that comes back to my question are we there yet no we're not there yet but we have come a long way right 2015 that's when we did our 1.0 release that was basically the hey we showed rust cam work what had seemed like a pie in the sky research effort was being used to ship production systems that was really cool now 2021 we're showing rust adoption has grown to the point where it is used in many critical systems that underlie the internet uh and in all kinds of places that you are almost certainly using today right and we're seeing more and more rust developers getting hired to work on the project i think we can expect that trend to continue so we're getting out of this trap of uh kind of relatively few developers working on it which which i think is going to be great and it's going to lead to more and more development and help us grow faster and sustainably but if the next goal you know is to achieve the widespread usage that i said i think russ deserves i don't think we're there yet um and i think there's space for us to grow and there's a lot of places where we'd be a good fit if people could just learn it so how do we get there how do we get resting to widespread usage well you do the rust evangelism strike force because that works great no i think you know going out and hectoring people into using rust isn't probably going to do the trick um i think we need a different approach so what about this what if we said a goal for ourselves instead of taking six months which is a common number right here for people to feel productive and rest what if we could make it so that they felt productive in six weeks or even less you know yeah that would be pretty cool right i mean it's i'm not sure how we're gonna do it but it would be pretty cool well i think it would be more than that uh i think it would be transformative and i think we can do it but we have to really up our game right we have to approach it very deliberately and there's gonna be a couple of things we have to do so one of them is two that listed here to start so we're gonna have to be focused and we're gonna have to be really creative so when i say focused what i mean is we've gotta keep our eye on the ball on the goal of making rest easier to use shorter time to productivity without sacrificing all the things that make russ great right if we just end up turning rust into another language that that no nobody wins so i think the way we should approach this is we need to identify the ways that we want rust to feel i've given three here there are some i think there are more we can say but we want rest for example to be reliable we want that if it compiles it works we want it to be performing we want to be empowering and if we keep our eye on those goals and when we're working on a design we sort of make sure that it feels all those ways then i think we're gonna find we're doing better and if we're it's not that's where the creativity part comes in right um we're trying to get a delicate balance of exposing a lot of complexity in a really simple way and it's really hard and we have to think hard and we have to try a lot of different approaches before we actually are able to do it um so you know we need to both be focused on what we need to do and be creative in the ways we go about doing it okay so what else well we need to think broadly by which i mean we have to look at the experience of using west holistically look at all the things users are trying to do and figure out how to make sure that all the pieces they need to do those things are there even though a lot of those things aren't going to be under our direct control not the language proper or the standard library right so actually we can give a good example if we go back to that blog post i started out with the one from tenable where they reduce their resource usage if you look at their blog you're going to find that before that post came another post one where they were evaluating rust and deciding whether to use it it's a nice read it's called building a microservice with rust and one of the things that i liked about it was they went through what are all the pieces of the puzzle they need to build a typical microservice right so here they have a clear goal they're going to build this service and they need an http api code to work with the database code to read messages from the queue a docker file and so on and you can see how they found most of those pieces and some of them they had to kind of cobble together with some scripts and other stuff and that gives you a clue well maybe that's a missing piece right that's something that if you didn't know what to do you might get stuff and if we want to push productivity down to a few weeks we need to make sure that all those pieces actually are there and are documented and that includes like kind of glue code like or libraries like i just talked about but also things like documentation ide integration and debugging support so learning ownership and borrowing now that's always going to take some time but the rest of that stuff we can definitely solve um we should do so all right so we're going to think broadly now what else are we going to do well we need to be bold we've got to be willing to take some risks so if you want new results and i do you've got to do new things right here's a list of things we've never done mostly for good reasons i will say and i think though we're going to have to revisit and take a careful look at our hard lines and our fixed assumptions of things that we know are a bad idea and try to try to reevaluate them in terms of those experiences are we really is it really preserving the experience to have this fixed rule or can we change the rule in a way that makes some other parts of the experience we want better without say hurting the performance part of the experience the reliable part and i don't know if these specific things these specific questions are the right ones right i can't tell you exactly which uh assumptions we have to re-evaluate but i can tell you for sure that if we want to get the goal of being productive in six weeks um if we want to make rust and widespread mainstream language would i do you have to be willing to re-examine those basic assumptions and make sure that they still hold so on the topic of revisiting re-examining basic assumptions let me turn a little bit and talk about the rush 2021 edition you may have heard we have one this year yes and additions are really cool because they give us this awesome tool that lets us remake decisions that we made and tweak them if we think we might like to do it differently without breaking all the code that already exists so how does that work the idea of an addition is simple every crate declares what rust edition it's going to use so in west 2015 that's kind of the 1.0 release rush 2018 and soon to be rush 2021 and no matter which edition you're using your crate interoperates with all the other editions and all the other crates right additions are an internal implementation decision they don't affect your avi or your they know they don't affect your api they also don't affect your api right and even better if you'd like to upgrade from an older upgrade older edition to a newer one we have tooling that will do it automatically for you right so this is not a split the ecosystem situation this is just get access to newer and better rust and what we use additions for is to make targeted improvements that make rust easier to use and it's often the kind of stuff that you probably wouldn't even realize it changed it's just that there were corner cases that made it impossible for us to to improve rust in a way we wanted to unless we were willing to tweak the rules so here's an example of something like that today in rust when you have closures they always capture an entire local variable so these two closures um here will both capture the tuple variable which means that they are going to get in an error because you can't have two owners of the same variable at one time but in 2012 2021 this code actually works and that's because the compiler observes that they're accessing different fields of the tuple and so it'll give tuple.0 over to this first closure and tuple.1 over to the second culture right and now that's great and once i saw this pattern i started to realize how often you know it comes up that i'm doing little work arounds for this limitation i know those little workarounds won't be necessary um but making this change did require us to use an addition because it can affect when destructors run when memory gets free other subtle things so i want to just give a big shout out to the rsc2229 project group for their hard work on this they were really persistent and they saw it through and it's going to be awesome now here's another example this is not a subtle change that you'd never notice but it's a pretty cool feature that uh doesn't really break much stuff and mostly just works except for the occasional corner case which is that we used to allow or we now we used to require you to put variables outside of the placeholder like this when printing and soon not yet but we will allow you to just put them in line this is actually going to work on all the additions the problem that we had to use in addition for is that when in certain places of like panicking uh accepted straight cert certain rather dubious things used to be accepted in work around panicking and we had to to change those semantics a little um in additions let us do that so that's really great and shout out tomorrow for making that happen use it epic effort and i don't have time to go through all this stuff in the edition but let me just call out a bunch of people who contributed um you all are fantastic and thank you so much edition's going to be awesome so all right we're ready to be focused creative bold what do we do happens that this very question was facing us earlier this year earlier this year as we thought about how to approach async rest so you may recall that in 2019 we shipped the async await mvp invest this was a huge achievement the design was really cool um and the implementation had gotten done and we were ready it had taken many years but we were ready to see it come into practice and in the time since i think we've been validated for all the effort we put into it there's more and more uses of async west and we've had a lot of time to gain up quite a lot of experience and we now you know it's time to look past that mvp and see what do we really want asyncrust to be what are the what is the how can we make asynchronous team reliable efficient performance um what does the full feature look like and the problem is there's such a huge range of possibilities right and so many variations to explore so many ways it could be so how do we get everybody on the same page with what it should be so i raised this question with shane miller who works on the aws rust team and she shared with me some of the approaches she'd used to do similar drives within amazon and elsewhere in the past and i thought it was pretty cool so the idea was basically you start out with an in-depth examination of the status quo what's it like to use async quest today and get detailed and you find out what problems people want to solve today what problems they are solving and the challenges that they face in doing so and these challenges are often not in the area as you think it's a really simple example when i discuss asymptotes with working programmers oftentimes the first thing that they talk about is what ide should i use or how well does the debugger work or not work right how much raw assembly do i have to deal with when i'm debugging that stuff is totally independent from async rest but it's still a blocker to adoption right it's what stops them from feeling productive so it's totally relevant anyway once you get a good handle on the status quo you start to think about the shiny future the idea is to go back through those status quo stories of what people were trying to do and what problems they hint and reimagine how could it be right and that brings you to first you start out with what you want it to be like and you go backward and think what are the steps if i had a feature that did something like this i could have that happen how could i get a feature that goes somewhere like that let me go backwards further um until you have a path right maybe a shaky and unreliable path but a path that's going to take you into a really awesome experience so we did this for async rust and uh we just we were trying to figure out how can we approach this in an open source way right so tyler mandy and i led the async foundations working group and what we did was we we did this collaborative uh effort uh so it wasn't just just one person drafting it but rather we went out and we consulted sorry keep my slide there we consulted the community sort of uh pretty broadly for ideas about the status quo about the shiny future when we ran sessions where we had people come in and uh like join us with a hack md a shared hackmd and we would write stories together that worked in all the experiences of the people in the call sometimes we went and talked to companies uh or in private conversations to get a feeling for how they were using it the things that they made you know in building their own software so and out of all that we kind of uh developed this picture of what the shiny future and status quo look like right and i want to talk to you a little bit about that um so we we started out with some of the basic experiences right in terms of thinking about the shiny future of how asyncrust should feel in a year or two and one thing is it should be consistent right you should be able to basically if you know synchronous rest learning asynchronous should be as simple as adding some async awaits and taking advantage of a bunch of new capabilities the smaller we can keep the gap between them the easier it will be to move from one to the other now in practice what we found is that's not always true right so we've there has been a lot of effort to keep them analogous but some things are more different than it seems like they should be so take for example this is this is an iterator trait this is not async it's standard rust synchronous rust it requests the next item and um takes ownership of it and if you look at the async version that's currently in nightly you see it looks pretty different it's got a different name it's called stream and its method doesn't isn't called next it's called poll next and it uses this thing called a pin and what it does is it basically asks is the next item ready and if so gets it and otherwise says no it's not ready right which is the underlying mechanism that you use to implement futures so it's kind of exposing this this abstraction up if you want to implement this trait you have to understand futures all the way down um now what we'd like to get to and we think we will get to is this async iterator trait which looks pretty much exactly like the synchronous iterator trait except it has the keyword async right well that fulfills our experience that we're looking for and getting there is going to require implementing a bunch of interesting stuff much of which is already underway and has long been desired like type laa's impetus generic associated types but the end result is going to be this really simple experience which is great and here's another example this one has to do with looking beyond the boundaries of the language at all the things that people need for the full life cycle of what they're trying to do so one thing we found when talking to people using asyncrest is that there's a real gap in tooling that emerges when you shift from synchronous to asynchronous rest with synchronous rust most of the things you're doing like processes and threads and they're well understood to the operating system so you can use kind of c based tooling which has its flaws but uh you know things like perf and so on to analyze performance and it works pretty well but with asyncrust the user space scheduler has a whole lot of concepts like tasks and channels that aren't understood by the runtime and it's really hard to visualize them and that makes it hard to debug problems it also makes it hard to do things like gain get live metrics of your services that are in production which a lot of people like to do so fortunately there's a lot of effort within the ecosystem itself to address this need right so tokyo for example eliza wiseman is building this really cool thing called tokyo console it's kind of giving you a top-like view of all your asynchronous tasks and it's going to get extended in various ways there's all kinds of plans to detect and warn about common pitfalls and replay events and so on and so i think you know seeing developments like this is going to work to make the using async rust from start to finish a productive enjoyable experience and another interesting thing about it um well i'll get to that interesting thing in a second so there's some more things in the vision doc that i don't have time to go into you should definitely check it out if you'd like to read it so we've got for example apis to go across runtimes and spawning tasks and so on it's going to be pretty cool future so that that other thing i wanted to mention i it's kind of a general point whenever i talk about making less easier to use i think a kind of warning flag goes off in people's minds that they think that this is going to come at the expense of the power and flexibility that they came to us for and i think as i said earlier if we succeed in this goal by turning rust into a different language that it no longer achieves the goals it's set out to do the great power of c plus plus if we fail you know then we haven't hit that upper right corner that we're shooting for and um but what i found in practice is that this tension between easy to use and exposing power it's very real but it's also very surmountable and when you do it you get a kind of better for everyone feeling um and work on that goes to benefit one group usually winds up benefiting all the groups if you do it right but the console project is actually an interesting example of this because it started out as an effort to help people who already had services kind of going into production or who are debugging their really complex asynchronous so they already know asyncrest and they need help debugging it right it's like a late stage feature but what we found when we uh well i didn't do these demos didn't say we but what i've heard from people who were doing demos is that just seeing the set of of tasks help people to understand the model better so it becomes also a teaching tool right it was targeting late stage but it's useful for early stage and definitely i think anyone who enjoys the ease of use that russ already offers knows that focusing on early stage also benefits late stage all right i would also point out i guess just that working on async itself benefits synchronous right it's another example it's not about early and late but about working on one area of rust often requires fundamental work that improves and extends all of rust so what's next um i think if we can achieve these goals and we will what we're going to do is find that we aren't just making systems programming accessible to more people which is what we've often talked about but we're even changing kind of what systems programming is in a sense or what systems programming languages are used for at least right so instead of being the go-to language for critical systems rust is going to be the go-to language for anything running in the cloud embedded devices and all kinds of things things where reliability counts right and if we do that what would that be like it'd be pretty great but we've always found we're going to have more people that's going to bring us more good ideas because people have good ideas having more good ideas is going to make rust better it's going to lead to more people which is going to be a nice little virtuous cycle right if we can get on that train we're going to find rust accelerating and improving faster and more than ever before so thank you so much uh this is the end of my talk as i said i'm really excited about what's coming up for rust in the next few years i hope i've gotten you excited and i hope you enjoy the rest of rust conf all right ciao

Channel: Rust

Views: 9,827

Keywords: rust-lang, rust, rustlang

Id: ylOpCXI2EMM

Length: 33min 4sec (1984 seconds)

Published: Wed Sep 15 2021