Data Oriented GUI in Rust by Raph Levien - Bay Area Rust Meetup

next up we have Rafe Rafe works at Google on fuchsia and has written stuff like zai and a whole bunch of cool things and is talking about GOI and rust alright let's get plugged in here yes I'll hit present great fantastic okay great so I'm gonna be talking about kind of an exploration like kind of asking this question of like can you reasonably write GUI code in rust like where the challenges where's it hard and so just kind of as an outline of this talk so first of all I should say that this this is my 20% a kind of personal project this is not an official Google product and any opinions expressed are also not official Google policy right so I'm the there's kind of a lot of material here so I'm going to just start talking a little bit about you know what makes GUI programming different or special compared with just regular types or programming that you might do and what are the specific things that I'm trying to you know get into and especially like one of the themes is performance this is kind of the the niche that I feel like one of the things that we don't have as much as we'd like in kind of GUI toolkits is performance and I feel like this is an opportunity to do something that's that goes a little bit beyond a little bit better than what's out there now and then there's one specific piece of this so you know flutter is a very interesting kind of new GUI toolkit and I think there's a lot to learn in there a lot of good pieces one of the pieces that I think is most interesting and can be like generalized to other systems is the layout protocol so I'm going to kind of that's kind of going to be the the case study of like how do we do the flutter layout protocol in a smooth way in rust so doing it in just object-oriented techniques if you just try and write your object-oriented code in rust it's not going to go well and an alternative to that is a data oriented approach and I'm gonna go into quite a bit more detail about that and then kind of the the the latter part of the talk really is a workshop in like how do you use specific techniques to write rust programs and a kind of a more natural intuitive way so what am I trying to do here so I'm trying this is all really motivated there's an experimental Windows front-end for Z editor so there's a the Mac front-end there's a good gtk front-end that exists there's the fuchsia front-end which we're also working on but there's no nothing that's really good for a window so it's like okay let's let's go in and try and do that and there's features that I want so I don't want it to look like a cross-platform app I want it to look like a Windows app I want performance performance performance so there's actually a lot of stuff in here the latency energy usage start of time smooth scrolling these are all actually kind of difficult problems I'm not going to go into a lot of detail on these tonight that would be another another talk but the the consequence of wanting to solve these performance problems is I can't really use existing abstractions very well I am pushed towards using win API calls and be able to control what happens at a very low level of the system and then another goal of the work is that I want this thing to be very easily usable from rest so if I have a GUI program then it would be great if I could just type cargo run and there's no like complicated like management of you know non rust dependencies or anything like that it should just build an executable that will be a GUI app so doing a GUI toolkit is a fairly ambitious project and in order to make it practical as a sort of a side project that means that there's a lot of problems that I have to explicitly choose not to solve and so here are some of the things that are out of scope of this work I am NOT doing cross-platform GUI toolkit and again cross-platform would really make this work a lot harder because I'd have to find like you really run into this problem of lowest common denominator of really not being able to address like maybe the best way of doing let's say swap chain presentation right so I'm explicitly putting that out of scope and I'm trying to learn what I can from doing a GUI toolkit on one specific platform which is in this case Windows I'm not trying to solve GUI for all possible apps I'm really like just making GUI for zai editor one of the really interesting like directions of GUI toolkits especially in the last maybe two three four years are these reactive api's where it looks like a functional program those are cool you I am not doing that I am doing a kind of a traditional widget tree and then it might be interesting future work of can you layer a reactive interface on top of a widget tree and I think that would be really interesting work for somebody not me to do and then also you know just kind of emphasize like at the moment this is experimental and this is this is kind of a toy toolkit on not doing internationalization I'm not doing accessibility I'd like to have a structure that makes those things possible like you know I don't like for it to become really real those problems need to be solved but I'm not doing that today so what are just I wanted to go through like some of the alternatives that I considered like what you know like you know I've kind of motivated a little bit why I want to do it myself but there are other ways that you could do this you could take an existing toolkit like GT K or QT those are kind of two of the main ones that might run on Windows and you know they're not fully satisfying so definitely my concern about performance basically means there's nothing out there that really performs exactly the way that I want it to and then certainly another big challenge of using these toolkits is that when you're programming to them the API doesn't feel like rest you're kind of in a different world where you've got objects that you're manipulating and you know there's an there's an interface that you're aware of and then they also don't really necessarily look like Windows native I mean QT actually is pretty good and you can make gtk look good but it's still like now there's two layers of impedance mismatch I'm programming to something that's not rust native and then it's displaying to the end-user as something that's not fully windows native immediate mode GUI is something that I looked at there's a couple of different ways to do immediate mode GUI there's a rust bindings for deer in GUI or you could just write it it's pretty simple code and I think that's an interesting thing to explore it wasn't exactly what I wanted to do because it's really difficult to like propagate small Delta's through an immediate immediate mode GUI and that's really the whole thing about zai then when you press a key everything that kind of flows through this pipeline is a little bit of a delta and you just only want to do the work for that Delta you don't want to like repaint the whole world and generally in an immediate mode GUI you end up repainting the whole world there are some existing rest frameworks that I looked at there's piston which you know really piston is also at I think at this point kind of experimental speculative and the features that it has are just not quite what I need for text editing and when I was experimenting with it I was also finding the performance in its current state was not what I wanted so I felt like this was an opportunity to try different things and there's also realm which is I think an example that is actually addressing a higher level of the stack the current implementation of realm is a wrapper on top of GTRs providing this kind of reactive interface so that that wasn't really gonna solve my problems either so let's look a little bit more deeply and like how do you program a GUI how do you program you know you want to display a window you want to display widgets in that window and have behaviors and so on and so forth so you you have these components and so you've got like a lot of components and they're interacting with each other so they're they're they're kind of they're coupled the components themselves are very stateful so that kind of pushes you in a particular direction and then the components themselves are dynamic like you might be you may press a button and it causes a tab to pop up or something like that so you've kind of got this combination of stateful coupled dynamic things which a lot of people might look at that and say this is this is a hard place for rust to be so in the traditional object-oriented world each one of these components is an object and then those objects live in a inheritance hierarchy so in the case of GUI programming you've got like maybe something called view which is the base of this hierarchy and then things derive from that and you might have it might be a very rich hierarchy because might say okay one type of component is a container that contains other components and then all of your containers like your rows and columns and you know scrollviews and stuff like that derived from that so you've got this very inheritance based like hierarchy and then you certainly in a traditional GUI you don't have to do this but in a traditional GUI you generally put those the objects themselves at runtime in a tree hierarchy it can containment hierarchy which is different than the inheritance hierarchy so there's kind of two hierarchy concepts and then again in a traditional object-oriented world you've got references all over the place like all of the containers they don't you could say they owned their children but there's not really a distinction between you know own reference or any other kind of reference it's just a reference and then the same for your parent you have a reference up to your parent and again in the traditional object-oriented world they didn't really come from a place of thinking about you know whether something is mutable or not you know what you know kind of controlling access to mutable state so you kind of have this model where all of the objects that are making up your GUI are available to be called by any other object that has a reference to it at any time and then that might yeah very likely to cause a mutation or something like that and you can actually do kind of crazy things like let's say that you're traversing this hierarchy and you need to paint and then in the middle of the paint call you say all right I'm going to insert something I'm going to you I'm actually going to mutate my my widget graph at the time that I'm painting and if you're in C++ well you just caused iterator invalidation and that you know that causes things to blow up in a very pretty way if you're in Java then it just makes it just means that you know something is gonna happen it's probably not going to be undefined behavior but it might very easily like violate kind of assumptions or invariants and there's nothing that really prevents you from doing these strange things there's no enforcement of kind of expectations and you know I worked on Android you toolkit for several years and I was very impressed how creative developers could be in coming up with ways to explore that space so can you program using this traditional I mean we have a lot of knowledge we know how to build gooeys in this kind of traditional object-oriented it really kind of grew up hand-in-hand with the evolution of small talk in other languages can we just take those patterns and program them in rust and the answer the short answer is yes but you just don't want to do that so you've got a lot of things that rust does not support well so the first is inheritance and you know rust does not do inheritance it doesn't have like a class and a you know kind of the traditional inheritance model you can fake it there's actually tricks that you can use like using do RAF I won't go into too much detail because this is really a talk about you know it's what you want to do rather than what you don't want to do I think a bigger problem in an inheritance is you end up when you when you have this graph structure when you've got a widget and it's got a parent and it's got children how do you represent those references you need to have those references because you're going to be doing things like propagating like you might if you have an invalidation you'll be bubbling that up the tree and then there's other things you're doing that are going down the tree and so you need those references you need to be able to traverse that graph structure how do you represent that and the kind of escape hatch that rust gives you in this case is interior mutability so this allows you to say there's a ref cell and you can have many yeah that's a so there's ref cell which allows you to get a mutable access to that and then there's a reference count which is either going to be a weak reference or a strong reference and the reason you do this is you say okay I basically own my children that's a strong reference but when I have a reference to my parent if that was also a strong reference I'd have a cycle and that would actually mean that I have a memory leak so the using a weak reference for that breaks that cycle of weights the memory leak so you can do this I've actually written a fair amount of code using this model but it is not fun it is pretty painful so you get a lot of extra noise in the program where you're constantly like borrowing these explicit references the tights get you know very get a lot of angle brackets you know in a row which is not great to look at and it's also not fun at runtime because you know you need to borrow in order to actually be able to do anything with these objects and if your borrow is too long and you do something while you're still holding the borrow then you get a panic or in the case of multi-threaded programming you get a you know I've kind of written this in the single threaded case that's actually another problem it kind of forces you to choose am i in a single threaded world or in a multi-threaded world in a multi-threaded world you'd get a deadlock which is also not a great experience so I think we want to look at alternatives like don't just write object-oriented code and rust what how else might you structure a system and so there's from the kind of game world you know mostly C++ programmers there's been a lot of really interesting work in this space and I think the general term which I've adopted for the work that I've done is data oriented programming I'm saying it's not object you know you're not factoring everything into objects but you're looking at the data and you're trying to say how do I program using this data and a specific thing that's been getting a lot of attention lately there was really inspiration for this exploration what is this thing called entity component system architecture so to kind of break that down an entity is a reference to a component so it's usually implemented just as a number you know you size it's a it's effectively an index into a vector that's holding the components and then sometimes it's a little bit more sophisticated it can have a generation so that you know if you deleted and you've got a stale reference that you can actually detect that rather than just accessing the wrong component so the next piece in a decomposed system is component and that's your state so these these components have to be stateful they have to represent like you know did the user hover their mouse over it was I just clicked you know that state and in the energy component system you really represent state in the simplest possible way of saying you know each component has some state it's just bytes and then it's up to the rest of the system to decide how to interpret that state and then the behavior is entirely encoded in these systems so there's one system for each type of behavior and so if you have like a button that might have several behaviors so you might have a lot of systems that that implement that behavior and so kind of the way that you can think of the system running is that in a kind of a continuous loop like so you're kind of always updating these systems you run the system for all of the components that might be interacting and then in a simple implementation you might just that might just be a brute-force loop over everything in the world sometimes you might do a more sophisticated filtering so that you say okay the mouse is here it's only going to interact with this and that's more complicated but you you can kind of have this model that there's just kind of this continuous you're always invoking the system's so this is really getting a lot of attention in the gaming world implementations in C++ people are interested in it because it really solves a lot of problems of both efficient implementation these things tend to be a lot faster and that they're kind of the coupling that you get allows you to change things more flexibly people are finding so there is a rest crate this is not what I'm doing there but it is an interesting exploration and I kind of encourage other people to play with this and experiment with like a pure entity component system and there are crates out there that will help you do this like specs and one of the other things that specs does this interesting is that allows you to run your systems in parallel and exploit multi threads which is a really interesting direction so as I said I'm not doing a pure entity component system architecture I'm doing something that was inspired by it but I think the right way you want to think about this is that it's a hybrid it is using concepts from data Orion programming and it's doing other things that are actually pretty traditional pretty familiar to anybody who's built one of these GUI toolkits or use one of these GUI toolkits so I'm gonna kind of walk through the actual implementation the actual struct that implements UI state in this case and the first thing that you'll notice is that I've got a structure of arrays architecture and an object-oriented world all of the state for a widget is in that widget so you've got you know you've got the widget specific behavior you've got the references to children a parent you've got other things like geometry you know all of that is kind of encapsulated in one object and then you have a whole bunch of those and kind of one of the central principles or one of the big tools from the data oriented programming world is you don't do that you do structure of arrays so that you have all of the related data in you know in its own structure and then you have an array so you've got one instance of that structure for every component in the system so I've got that so I've got these VEX that are blue that are all kind of parallel so the the same ID which is just a you size that's used as a handle to reference one of these components is an index into any one of these arrays so let's go into the what arrays do I have what state am i representing so the first one kind of the central one is widgets as a vector of box of dying widget so dine widget is a trait I'll go into a little bit more detail but just think of that as having your behavior so that you know it tells how to paint how to layout how to handle mouse input and then each type of widget has its own imple of widget trait and of course it's dine widget because you know I want to be all up-to-date and with it um actually the code is still box widget but you know future work so those are the that's the that's the behavior that is specific to the widget to that to that particular widget so then I have the graph structure which I mentioned I kind of gave the example of what does that look like in the kind of interior made ability world in this world it's pretty simple you have these arrays that just hold the graph structure and one of them is the just a vector of children for each node and then each node also has an ID that just references its parent so there's no complicated ownership or borrowing you're not fighting the borrow checker that's really a theme that if you're trying to program object-oriented programming in rust you will be fighting the borrow checker and in this kind of data oriented world you're not doing that so then I also have other state per widget I have geometry which is pretty much just the size location and size of the widget and then I have these listeners which I will be talking about in considerably more detail later so this this decision to represent like the core widget state in this box this vac of box of Dyan widget this is kind of the central architectural decision that everything kind of flows from the rest of the design really flows from that choice so you know when you want to reference a widget you just have the index into the end of this vac which is you size you've got the widget trait which has this widget specific behavior that is only the widget specific state though so if the other state like geometry and the tree and stuff like that that's actually held by the the system and then you've also got this kind of principle that when you're traversing when you're actually doing the work of the GUI yeah you're you're going through this Veck and you're borrowing a single widget and you've got a single mutable reference to that widget and only one of them at a time so that that is a little bit of a challenge like if you would just need to interact now like if I press this button and it causes this you know other thing to change how to express that interaction mr. e which I will get back to but before I do I want to go into this detail like one of the really challenging things that any GUI toolkit has to deal with is layout and the in in a kind of any modern GUI toolkit you want to do layout in this responsive way so that if for example you're dynamically resizing a window there's all kinds of things that might want to happen as a result of that like for example you might want to go from a one pane to a two pane layout and so that concept of like a break which is not continuous is a test of how expressive is your layout system there's like a lot of interest in things like cassowary which is a sort of a linear constraint solver and it's kind of difficult to express those things it's easy to express kind of a smooth you know like as it resizes you have more space in proportional you know proportionally and given two different widgets but it's very difficult to do things like that those breaks or if you have like wrapping if you've got like text or if you've got like a series of buttons or something like that and then you expand and might go from needing two lines to one so you want to be able to express those you know patterns that that kind of responsiveness to size and flutter has a really interesting way of doing this which is which allows you to express those and is really really really efficient in his Ennis simple programming and kind of like satisfying all those constraints at the same time is fairly impressive so the way that it works is that you go through the tree and you pass a constraint down and then you traverse down into the tree and then what flows back up is just sighs this is how big how wide and how tall each widget needs to be and the constraint there's a little bit more complexity to it with you know inherent size but the the the guts of it is is just what's on this slide that the constrain is just a minimum and a maximum for width and height and so the size that comes back up has to be somewhere between minimum and maximum and that's basically all there is to it so then you do a one pass traversal of the widget tree so you when you do this traversal when you're laying out your system if you only touch each widget once and that is a huge principle like if you look at Android you know Android has relative layout it's not really recommended for exactly this reason but if you have nested relative layout in Android that's an exponential increase in the number of times that you're gonna touch you know that you're gonna call a layout method so the difference between linear and exponential and kind of matters so it would be nice if you could do like a really simple traversal like for example when you're painting you can just say you know just go through the tree and paint all of your widgets for doing layout using this flutter protocol is not quite that simple because the order of the traversal is dynamic and I'll go into that a little bit more detail in the next slide but but first of all if you're interested in this and also other details about how to build the guts of a widget system of a you know GUI toolkit I really recommend this video fletchers rendering pipeline from a couple of years ago at Google i/o Adam Barth and it is very dense it has a lot of good stuff in it including a lot more detail about this layout protocol so why is the traversal length why is the order of traversal dynamic so if we go into a little bit more detail into a row or a column that it roughly looks like this that you've got children that are both non flexible and flexible link if you think about a bar that has like buttons and then maybe like a text field and maybe your buttons are not flexible but your text field is flexible so the algorithm basically says do your non flexible children first go to each of your non flexible children don't give them a bound let them tell you any what they want then you get those widths and then you just compute it well how much space do I have left over like I've got this width the non flexible children of these widths this width is left over and then you go and you proportion those out to your flexible children so it depends which children are flexible and which children are not flexible and that might be dynamic so you that needs to control the traversal that you do your non flexible children first so the other thing I think to notice here is that like this is kind of the way the code looks in flutter and you do these recursive calls and you can immediately see that in this model where you've got like a Veck of widgets and then you take a mutable reference from that vac to just the widget that you you know you're calling layout on and then if that widget wants to do a recursive call on some other widget it needs to get a mutable reference which it's going to borrow from the immutable reference of the BEC of widgets and you're stuck you can't do this how do we fix that so the in the data oriented world you could there's tricks and like I think maybe maybe trick like would be better to say this is a pattern right and so you see this pattern in things like iterator and in future where you have a computation and you don't want to do the entire computation in one gulp you don't want to say I'm gonna call you and then you're gonna do your work and then when you return your computation is done you want to break it out so that there's some way where you can say I'm gonna call you and then you're gonna do some of the work and then you're gonna return something to me and then I'm gonna call you again and that is going to resume the work that was suspended and by kind of these repeated calls we're gonna get the job done so to kind of adapt that idea or that pattern to the flutter a layout protocol what we're gonna do is we're gonna have this this is pretty much the signature of this method in the code where we're gonna we're gonna call the method with a mute self so that it can you know it can access all of its own state and do anything at once with it including mutate we're gonna pass in the constraints which are just same as same as the flutter code we're gonna tell it what it's children are and notice that that's a that's a non mutable borrow you don't want to be able to you know so that's encoded in the type alright and then what we're gonna do is that we're going to pass this option size which is basically the result of the last of the the last widget that's the child right so when this widget returns it's going to either return the size and say I'm done here's my size we're laid out now or it's gonna say don't know yet but please request to the layout of this child and then call me again when you get the answer to that and and then you're passing down or you're passing to the child what are my constraints for that child and so there's this kind of flow of data so it's really of a flow of data rather than a flow of control instead of a recursive call down the call stack I think I have a picture of that yeah so in the kind of object-oriented world you're kind of going deeper and deeper down the call stack with control flow and in this kind of data oriented world you've got this data flow where your system is just repeatedly calling layout on one object at a time in your in your you know box of a vector of widgets and then you've got this data flow that the you know when you request oh please layout my child then you've got the data that comes back up from that call into the next call and that implements this layout protocol without fighting the Boro checker at all there's no there's no interior immutability there is no anything that's that's kind of difficult or painful to program so an another way of looking at this actually that's kind of interesting is that if you have like a really simple recursive program like you know if you think about factorial as a recursive program and you don't want your call sex to be deep there's kind of this transformation of how do you make that into an iterative program and that really is pretty similar I think that's the same idea that you could maybe think of that not just as a special one-off hack but as a general transformation that you can do to get your algorithms into this structure so so that's kind of like when I'm talking about like programming techniques for how do you implement these patterns in rust that's the first the second I'm going to talk about is state splitting and so state splitting or there's just what I'm calling it and so when you think about like different parts of the system that you're you have different needs of what needs to be mutable and what needs to be immutable so when you're doing layout which we just went into and some detail you need a mutable borrow of the widgets because you want them to be able to do whatever access to their own data you need to mutable e borrow the geometry because you need to store it somewhere and then you need to traverse the graph and that should be an immutable borrow because you know you want to be able to say like for child in you know this widgets children and while you're in the process of traversing that that needs to be an immutable borrow you do not want anything mutating that at the same time there's something else you might be doing where you're gonna have a different you know different pattern of what needs to be mutable what needs to not be mutable for example if you're doing something like you know you press a button and it causes a panel to pop up well popping up the panel means now you need to be able to mutate the widget graph so you know depending on exactly what you're doing you have a different pattern of which pieces of that state need to be mutable and which pieces of that state need not to be mutable and the way this is implemented is actually kind of simple that and this is one of the reasons why it has to be structure of arrays rather than array of structures because structure of arrays allows you to borrow each one of those arrays with a different reference with different mutability so when you're doing layout then you borrow widgets and geometry with mutable references and you borrow the graph with an immutable reference and for the other recursions other traversals you have different patterns there and rust helps you do this it makes it really easy if what you're taking that reference from is self so if you've got self than you can kind of take references from the pieces self from the fields of the self struct whereas if for example you were handed just an in mute reference to the thing then it's a lot harder to take the to do that state splitting it you get you know your compiler will will complain so this is just a pattern where you're not fighting the borrowed check are not fighting the compiler that you take those references and then as you do the traversal like in this case you've got these kind of three main references as you do the traversal you traverse those down so the other thing like for example if you look at this slide that one of the fields that I didn't talk about yet is this layout context and so that's another important piece of the state splitting concept that that context lets you do things and in the case of layout it doesn't let you do a whole lot maybe it allows you to query the geometry of other widgets in the tree so layout context is not a very powerful context but there are other contexts where you want to be able to do more things and so the idea is that that context has the that kind of encapsulates the mutable reference that let you do the actions that are desired at that point and that is that is writing like what can you do like can i mutate the graph can i query you know can i query a geometry of a child can i send an event those are methods that are on the context so you're writing into the type of these methods what can you do and what can't you do and so that's a static compile time enforcement of the invariants you know for example can't mutate the widget graph while painting the context that's passed into the paint method does not have a mutate widget graph method on it so you don't get to have as much fun at runtime so the the last part of the design really focuses around this listener concept and again so we kind of talked about like you know how do you break the state down into these components how do you do these things like you know pain and layout but you get these interactions you these are coupled like the you know different pieces of the world are different are interacting with other pieces of the world and you know one of the things you wouldn't be able to do like if you press a button is just call you know the update state method on you know like a text field or something like that so the way you you need to encode these interactions you need to be able to express these interactions and the way I do it in my code base is that your context for these handler methods basically lets you put events in a queue and so your queueing events that are going to be sent to generally other widgets in the tree and so the way I do this so you've got this you've got this coupling right you've got this dynamic coupling like what's an event and so I'm gonna reach into the toolbox of loose coupling tools in rest and one of the ones that you see not that often is any and like I don't want to overly constrain like you might have an extremely simple event like if you've got a button then an event might be just like you know set it to pressed or not pressed right so these the actual state content of that might be a pool but then I might have a very complicated widget like a text field and it might have some very advanced structure that's saying exactly what do I want to do to update the text field in this text widget so box any pretty much lets me use whatever structure I want and I just put in a box and then I put it into that cue and then when the handlers return so the handlers are invoked when you have some kind of input like a mouse or key or whatever and so you run the handlers and you give them a context which lets them put the events into these cues and when that's all done then you run the cues and you know running the cue is basically just calling the dispatch into the listener for each of those widgets so you kind of have this it's almost like a bipartite graph where you've got the widget which expresses really kind of view and behavior to input and then you've got the listener which is an object that lets you encode I think application state so maybe in a you know kind of traditional lingo you might call it and then the listener is given a context that lets it do a lot of stuff so it can interact with widgets that interaction widgets a synchronous so you can do both sending commands and querying and getting data out of the widgets and then you can do other things like mutate the widget graph you know and kind of express this sort of complex interactions in your app so again like the idea here from a like maybe what is a general lesson that you can do in building these systems is that you you've got these tools for loose coupling both in the data and in the control so you're not doing control flow you're not calling methods you're sending data across and then you've got this this box any which is like this kind of very interesting way of getting much looser coupling from the different components of the system so that's kind of you know the the structure that I wanted to you know just kind of report on the explorations that I'm doing and very happy to take questions the code is here as I win and you can I think let me see if I can take a chance here and just just run it a little bit so there's I'm not gonna compile it it would take about half a minute to compile that I did it earlier that just to show there's no interesting dependencies so there's a calculator app and you know it's not that interesting I've done almost no work on like the styling so I you know I'm not even showing like highlights as you mouse over yet but one of the things is really cool is sorry about that is this dynamic resizing and I don't know if if you see it on these monitors but there's like a real 60 frame a second and then it's like sticking to the it's sticking to the resize cursor when you do that which is it turns out is really difficult in Windows it's kind of surprising how hard it is to do that so you know I researched that so you didn't have to and then this is also this is very experimental but this is the editor itself and you know it works it has very basic functionality you know you can select and stuff like that it's still very much work in progress but I just wanted to show that you know that this is actually kind of in the process of becoming hopefully a real a real app so now I'd like to open it up for questions okay I have so many questions so I'll just ask him I also have Manish how many I can ask why one I have to choose so you said you were going for two like the native look and feel yes how do you kind of married is this system that you've built with like the the windows API is that like actually like due to standard drawing and like the components and things like that right that's a good question so in the case of things like drawing the menus which is in the file dialog which are really important it's pretty straightforward I just call them I just you know they're just win API methods that are available and I can I call those and those are not tightly integrated with this kind of widget like display mechanism so one of the things that I am not doing is embedding the windows common controls I'm kind of taking control over the way that the application surface is being drawn so in that case it's just like taking care to draw the application surface in a way that's consistent with the windows style so it seems like the the API you presented for the widget seemed very naturally expressible as a kuru teen like you explicitly reified the state as this layout result but I guess I was wondering if you had any plan I don't know if it's actually in the nightly z8 or not but like you actually have any plans to try and use like the explicit car routine API because also like I only saw like that most recent posts or whatever I don't know for example if like having a pin self would like fit with your borrowing model well yeah I guess some peers if you have any thoughts on usage of co-routines to try and model your widget painting function its it's a really interesting question so like I think painting actually is not hard painting paint painting is just straightforward code but layout is slightly difficult and you know I thought about that when I looked at it this idea like how do I transform like what I would express is sort of a call where I want to like you know kind of express this as straight line code the simple fact of the matter is that it's not that much worse the way it is just doing it the simple way then it would be if you implement it you kind of have to do the same things you have to you have to kind of keep track of where you are in the iteration so I think it's an interesting question but I wasn't thinking to myself wow this code would be so much clearer and nicer if I had access to co-routines thank you yeah so I guess how do you deal with errors if that makes sense so like because the system is so highly decoupled a lot of the time like if I get an exception in my Java program because I'm doing something silly like trying to do IO and the UI thread and like try and grab an image from github and it fails or whatever like I'm doing that somewhere else in this system so how do you track that failure all the way through to where it actually turns into a hard error in the like paint method or whatever that's a really good question at the moment I don't have any real error handling in the code I I did promise that it would be a toy UI and that's definitely one of the things that distinguishes toy from production systems um but I think it is a good question it's potentially kind of complicated because like you know something is happening because it got sent an event and if if the error is actually inherent in the event it got sent then you have this thing of like how do you tribute how do you know where that came from right and I don't have an answer for that that's actually a very fundamental thing like anytime you build these asynchronous pipelines you have to deal with that and when you have that Paul call stack you know at least you have that context and then I'm saying well I'm getting rid of that tall call stack but I don't know I mean so far like as I say I'm kind of trying to do a very specific thing where you know I'm really just trying to build an editor and I think I'll probably end up with basically a lot of these methods like these methods return like you saw that kind of mean um that's either like in the case of layout it's like sighs or request child and it could easily be sized request child or error it could just be a result of that and I'll probably just do that and then have kind of patterns where if the error is inherent in the input then I wrap that in the result but it's yeah it's a good question and be interesting to see how that actually works out when it's kind of more baked this might be too big of a question but like the you know the big user interface and rust is servo and so I was wondering I'm not sure if you can briefly compare and contrast like the the kind of techniques that servos doing for parallel layout and styling and layout to this approach yeah yeah I am I probably should have put servo in that kind of alternatives you know a slide and I'm not deeply familiar with you know they're kind of advanced technologies the servo is doing there's definitely a feeling of a lot of what servo this is my interpretation a lot of the kind of very fancy stuff in servo is to deal with web content which has a lot of sort of nasty unpredictable aspects to it like you might have a ton of overdraw or it might have kind of complex clipping or something like that that requires this parallel layout to work what I'm doing you know once I get access to that really low-level I might be doing sophisticated things in the text plane like I might be doing you know really fine grained calculation of deltas but nothing is really complicated so I'm finding that like if I just call the right if I like break it down so I'm doing the minimal amount of work and just calling the Wynn API things to draw that I'm getting what I need and so it feels like maybe that's one thing of like less complexity you brought up parallelism and that was one of my prepared slides which is that in their existing code base everything is single threaded but it's really interesting when you look at these you know when you look at the design you you see opportunities like maybe I could call the layout of my children in parallel they'd be but the widget you know the in the widget trait that it has to represent to do layout it's just an emu itself it doesn't need to have like this like entire world mutable state if you make that layout context really really tiny then you can call all of them in parallel so I think that there is actually a lot of potential to do that kind of thing in your slides you had type ID equals you size yeah um is that how it isn't your production code reduce do you do something to make that a little bit that's narrow prone at the moment it is just that and I think I'll see how that goes in the ECS world you usually put some kind of Junction or some kind of extra thing to to make it you know so that you're much more likely to catch like it because you there's nothing that enforces that that ID has to be valid and so you know like having some additional runtime checking useful so far I haven't needed it but you have you hated any bugs well I mean there's always bugs but related to the know I have an idea nixing I have not hit ID bugs yet no do you compact the vector of objects when you remove an object do you remove objects right now the code doesn't have object removal but that is one of the things that needs to happen I mean you you're constantly you know this will be a long-running session where you will be doing things like popping up you know dialogues and so on and so forth so that needs to happen um I think it's very likely that you will not need to compact that vector I don't you know it's because basically if you think about it the size of that vector really represents the high-water mark of the number of objects that are active at one time and that seems pretty manageable that seems pretty small there's a small amount of additional work that needs to happen to make removal work reliably and like one of the things I kind of left out is that you want to have a method call on a container that says on child removed essentially and then you want to be able you want to give that container the option if it has any state that is specific to that child you you know the guarantee is like nothing is valid after on child removed you know for that child and so it should forget anything having to do with that child's ID does that answer yes thanks other questions have you given thought so like you're avoiding inheritance which is phenomenal and you've got this even a single widget trait traits like look a lot like static mix-ins mhm but there's not much of an affordance for like specializing these things and this is new described had to give much thought to how you would do that with this kind of system how I would do like you talked about like having on child removed yeah but you presumably have elements of the system that you don't want to force all widgets to implement I'm seeing see what you're saying it right now I have a very simple thing where pretty much all of the methods on on widget have a default implementation which is to do nothing and that's working pretty well for me I actually think it's unlikely that I'll run into a case where I want to do anything more fancy or complex but you know you can imagine like you know it's kind of a function well how many methods do I expect having on a widget right now it's like a half a dozen by the time I'm done I expect it'll be probably two dozen maybe and so if it was going to get what you don't expect people to have to implement right exactly most of which would be have default implementations most of them would be kind of specialized and then you know you're gonna have a lot more for doing things like accessibility and so on and so forth but you know I don't I don't think that you know I think that's just kind of at the threshold like if you look at mature widget systems it's not uncommon for there to be a hundred methods or more on a you know view base and then you might really look at well how can I organize this so that I don't have like this huge V table for every object but I don't think I'm quite where I need that yet as just a small follow-up I looked at a few of like the EC SS for like doing game programming and there is exploration in like doing more generic things so you could have a table of like all your buttons that like also has another table that like you know this is all the widgets and then yeah so you could like dynamically grow and and rather than like having you know one baked meta widget you could have like thousands of these other things that may or may not implement your interfaces yeah and again I really like I I'm kind of excited about the possibility like I kind of would love to see somebody explore a more pure more data oriented like like strictly non dynamic dispatcher approach to this problem and then compare and learn like which is actually a more natural exploration of the of the patterns expression of the patterns so from the sound of it you are and you're not really using the win32 components you're not really that coupled to two windows what okay yeah so this is something that I've been thinking about a little bit and you know they're the reason why I chose Windows aside from the fact that we need Windows client for zayed etre is that Windows has this nice property that what you need is provided by the system and that's the 2d graphics abstraction and especially the text depth abstraction making something that can just draw rectangles and strokes and stuff like that is not a very complex API but an API that you know handles text is complex and so when I was writing kind of directly for Windows there's a solution to this which is you just use direct 2d and you just use direct write and then your text rendering font selection emoji all of that works just like you know just like the platform if you're doing something else like a lot of these you know servo is no exception that if you're doing servo a web render then you need like some other technology there to do the especially the text um so the way that I would like to see this happen is I would like to say this is kind of a wish list really that I would love to see a 2d graphics abstraction that has methods for draw a rectangle you know clip to this path whatever you know this has kind of been known since the days of PostScript what these api's look like and i think that there's a real opportunity in the rust ecosystem to build this thing so that then it can be implemented with like a bunch of backends and of course you'd have like you know direct 2d is one of the backends and you'd have almost certainly like web render as a back-end and you can imagine like maybe you know half a dozen things that would that would be useful like maybe you know just direct pdf's if you're if you're if that's what you're trying to do so yeah the code is structured in layers and so the goal is that there's this lowest layer which I calls I use I win shell which is hopefully where all of the unsafe code lives which is the kind of you know like interacting with the win32 message loop and message pump and you know all that kind of stuff and then there's this I win UI layer which is where which was really what I'm talking about today and then the layer on top of that is I win itself it's the editor app which then the idea is like there could be other apps that use this toolkit maybe maybe not but the layering is is to encourage that so the idea is you kind of make like right now as I win shell is a very thin wrapper it's like in most cases it's just safe wrappers around win32 when API functions and you could just totally imagine you know having a cross-platform implementation of that layer that the layers above would would be able to use so it's something I'm thinking about but I'm kind of wanting to like get it in shape on Windows first because when you do cross-platform you have to make some kind of tough decisions about you know what am i what am I maybe giving up by abstracting it and like am I giving up performance you know am i giving up some kind of expressive expressivity and by doing the windows native one first then you can you can actually have that as a basis of comparison and say okay here's the cross-platform version does it perform the same you can do benchmarks etc I'm sorry one other question so I'm you you mentioned the one of the impedance for the impedance for this was being high-performance right you want a high-performance GUI toolkit so I'm curious what the current performance bottleneck is in running the entire layout and painting engine as well as generally what the experience of performance debugging is figuring out what these bottlenecks are and if you have any experience another GUI GUI toolkits how that contrasts so performance and GUI toolkits is a really difficult deep question and especially it's on a lot of different axes and a lot of times people will focus on just one like the ability to do smooth 60 frames a second updating and like that's probably not where most of the pain is probably most of the pain is in things like startup you know time and like transition from one activity to another kind of thing and in those things you get all of the cost and things like inflation of your widget tree and so on and so forth but even but it's also true that just scrolling is is performance intensive and then you get into so they're scrolling which just requires traversing you're getting your painting done really fast which like I could talk all night about this I'm trying to think how do it how do i distill this into a way that's useful then there's this business which is on the screen now which is minimal invalidation and minimal invalidation means that if you press a key and the key inserts a character that you do not repaint the entire screen from top to bottom that you only address the specific area that got updated or in my case I'm doing Lion granularity because trying to do it finer than that just just isn't worth it so there's this principle of you know like one of the principles of UI performance is do as little work as possible so you can really look at the system from the outside to saying there's some input and there's some output which was like drawing calls and then there's all this stuff in the middle and if there's a huge amount of stuff going on in the middle or if there is a lot more drawing calls than you actually needed to update the state to its new appearance then that is performance you know potential to improve the performance so this is definitely an in minimal in validation this is something that GUI toolkits used to do because if you were repenting the screen it would visibly take longer than one-one vsync so it's kind of falling out of favor a little bit but I think it's still really interesting and important to do if for no other reason than to reduce battery consumption power consumption because you know just doing this tiny amount of work is gonna consume less GPU been with us stuff like that so that's a real goal and not that I've identified as a real problem I mean if you put one of these other editors I won't name them the ones that are implemented using web toolkits underneath then you know you see is this tremendous amount of work that's happening just in response to like pressing a key so then you know like this goes I'm not gonna go into all the details but like you know I thought through like what do you need in this data oriented world to do minimal invalidation like how do you keep track of that state how do you you know how do you do all these processes so that when you're done you're only drawing you know updating some small region and and I think that's like definitely one of the one of the opportunities that especially if you look at the kind of like immediate mode and some of the other things that are maybe a little bit more like game engine you eyes that they tend not to to give you that opportunity because you would you be so kind as to go back to your cross-platform bonus slide sure based on my memory and astute observer might comment that these two like top level bullet points are basically like the beginning motivations in the flutter rendering pipeline talk you linked to if I remember correctly like I'm not sure that I may have a different one yeah that might be because the flutter I mean flutter if you actually look at flutter flutter is using skia as the kind of provider at these extra actually it's more complicated than that because it's using skier for the 2d abstraction and then for text layout it's if there's a very interesting story here where that used to be an adapted version of blinks text layout engine which was very much web you know based and they finally you know ripped that out and replaced it with something else called Lib txt which I think is very interesting because it really is a focus like how do you solve like the modern text layout problem which means text layout with emoji and you know solve these problems like how do you enumerate the fonts on your system so that when you actually choose a phone it's making roughly the same choice that the system font rendering would have done so yeah that's like the layer that's definitely like layers below what you're gonna see in the flutter flutter talk right so um I must be remember you'd ever talk but mmm this the ski and live text things what I'm thinking of know is it is is your opinion that that's like the main body of work because it seems like that's aside from the the the bits in dart that seems like the the largest chunk of flutter right is like just getting boxes onto the screen so is that actually like could you run win yours I win gooey on top of those abstractions conceivably and conceive conceivably conceivably absolutely I have looked at it and said the thing kind of one of the observations is so skia is a very large heavyweight dependency in a lot of different ways for example if you get cloned skia and its dependencies I forget exactly it's like some hundreds of megabytes and it will take a long time to compile and so I feel like when I say cargo run and that happens in like some reasonable amount of time I feel like taking on a dependency to skier kind of degrades that that dimension so if you are on Windows then I think you really want to be using the windows native drawing you know API is because those are good on a different platform skia might be a really good choice and so that's really why I'm specifically saying I want this to be an abstraction that gives you those choices and then you know I think skia obviously works on pretty much every modern mobile and desktop platform that you might care about so that would be a good thing to do if you had that then it would kind of fill the matrix where at least you had a workable solution even if it wasn't ideal in all respects of which you know kind of compiled time size of dependency is one that it's kind of not good so it's so your is what you're saying then that you think choosing to abstract across the platforms at that layer in the stack is like not appropriate for a text editor or I mean I'm trying to parse that last you it would do a very good job of filling the matrix but on Windows you want to be doing this yeah I'm not sure what you're asking why didn't you use yeah I guess why didn't I use git yeah aside from the cargo run he's not the only reason oh yeah I think it's a pretty compelling reason yeah and the other the other reason not to use ski is this ski is API a C++ and there is extra work it's not impossible but there is extra work that needs to be done to make that kind of smoothly interoperate with rust and that work is not in really good shape like there's experient mean like a servo used to use git server used to have a skier binding and then that's being replaced by web render and honestly like I don't want to I need to be a little bit careful because so many of these people are my colleagues and stuff like that but let me just phrase it this way that I actually and web render dependency to be kind of a really interesting direction to go with this abstraction as well right thank you sorry for the confusion question I know maybe one or two questions alright well thanks everybody for coming thanks all for coming as always we've still got quite a bit of time so feel free to hang around if you're interested in speaking at one of the future meetups let me know and like I said the next meetup will probably be in Stanford there will be updates on the meetup page probably in the middle of July right thank you

Channel: Rust

Views: 62,757

Keywords: rust, rustlang, gui, programming, xi editor, bay, area, san francisco, entity component system, data oriented

Id: 4YTfxresvS8

Length: 65min 59sec (3959 seconds)

Published: Tue Jul 03 2018