[Linux.conf.au 2013] - The real story behind Wayland and X

okay the first talk for today is by Daniel Stone and it's a real story behind Wayland and X Daniel currently works at kelabra where he's been working full-time on Weiland for the past year or so having recently overhauled the input subsystem he's also written a couple of graphics drivers in his time and has recently moved from Melbourne to Finland and more recently the UK so it's nice to welcome him back here so thank you very much for coming to talk to us today Daniel turn this on Rowan thank you also hi yeah as you can probably guess I'm here to talk about Wayland annex pretty much what they actually are and since my experience has been that no one has any idea everything you read on the internet about it will be wrong it's ridiculous although there are the occasional gems which I'd recommend strongly but this I think is pretty much the state of play which I'd like to change so ideally I'd like everyone in this room at the end of it even if you don't agree with me that's fine but just stop posting stuff that's blatantly wrong and lwn comments something really bought of it sorry LW anything there's actually a cool bro side coming up you'll see anyway so why am i qualified to say this more than everyone else on the internet who says I've got nails on the Internet's wrong well it's born between these two ex releases I have fixed bugs older than myself sadly and I got tricked into packaging x4 86 at some stage then came to LCA in Adelaide when I was very bright eyed and enthusiastic I got plied with free drinks until I agreed to work on keyboard support you were drinking Malibu and coke so the climate that was um and at the time was all wandering promised annex was going to be amazing and worked on build system stuff for a while built the first modular extra dog server I think I first got it to build in a car and Quebec or something sorry no I was driving back with clay at about 130 miles an hour I wasn't driving then I traded Melbourne summer for finished winter dropped about 67 degrees in a day I was pretty happy about that especially because I've never really seen snow I saw a fair bit of it and it was to work on consumer devices for Nokia internet tablets and phones all the also internet tablet OS my mo Miko Miko Harmattan pick a name they're all good platforms there are a few and that was a very exciting time because I went from writing build systems to actually putting stuff in people's hands and at the time it's 7 years old but the UI hasn't aged a bit well that's terrible looks even worse in high-res yeah we're pretty easily impressed and everything was done in software on the worse time you've seen with the exception of the Raspberry Pi we had ggk 2.6 which was still pretty seedy eh at the time and three level sub menus were cool every single application had a file menu and edit menu everything that wasn't anything like tablets are today it was the desktop really slow but we had a very nice screen we had high DPI long before Apple and we made some errors but honestly that was not my fault yeah so while I was there I got us to make Bluetooth keyboards work so as often this fine little and if I'm in pop hot-plug and you know you can plug in a keyboard and it turns up and plug in a mouse and it turns up and I had a device inhale support which turned into I had a D boss and health support and then we removed a boss and health support and everything was alright again but in the intervening time that happened and yeah good times and the iPhone say what you will about it completely changed consumer devices in terms of what people were willing to accept at least I thought it did turns out some people accepted early Android but you're not just friends around oh yeah more broadly speaking it meant that I went from keyboards to looking you know finding like you know when you press this the screen flickers white for a second so you know that's not on people I'm going to hover so basically spent the next few years not doing much more than finding little bits where the screen would flash white or you know you'd see a kind of half-completed frame or something and working out how to fix them and generally the answer to fix them was break the eggs back subtly I felt really bad about it until I looked at the source for everyone else's acts overs and everyone else broke this back in the same ways and I did this for a little while it did take its toll on my once gorgeous features so at 18 I was very excited to go overseas for the first time it's been 12 CA and this is after a few years of bashing on stuff via smiles long disappeared that was two weeks before I finished working on expert and that was also long hair in the rain to be fair but what can you do you can't actually see the rosy cheeks on that one that's ridiculous actually look 12 oh yay now the Rose it's all back over on something as a quick aside the guy who invented Weyland Christian hoax burg was also an ex developer so sorry let me know when it smudged so yeah before we get into the whole you're just being ex because you don't understand X and how good it is we've worked on X for the last 10 years we have the facial features to show for it so but self-indulgence isn't fun so let's go on to X because I assume you're here to hear about that rather than me um originally back in a day so we're not talking about current ex running fancy crazy desktop environments or in the case of this laptop I borrowed xfce back in the day when I were butter lad in Bendigo someone had a really good idea mechanism not policy will be a will create this amazing framework that will let anything do anything once and it will be great and apparently people are still libertarians in America today they have not learned and they're not X developers apparently it was a sorry and you know maybe they work and they left before I got horrible it was pretty simple do you know draw a rectangle make it white draw some text on that white little rectangle maybe draw an image below that call what could possibly go wrong you had a window the window manager could put a little frame around it maybe with the title maybe with a close button if you're lucky maybe with a little stipple pun because that was the fashion at the time your boy earth could be dashed good night your board is gonna be dashed oh they could be tired and we had a root weave we had one button at one three button Mouse it was Network transparent hello to everyone commenting on LW and that was a fact at the time so that's pretty easy life was good you could still achieve quite pleasing visual effects that was cutting-edge at the time because x11 shall inspired by my birth added color but everyone grew up some people grew up hardware grew up suddenly you might have a graphics tablet you might have a couple of mice you might have a keyboard which had multiple bells and could you know take messages to show and show to you on an LCD screen and loads of other things which no keyboard had but you might someone at s3 I did once and and we had multiple GPUs and they almost vaguely work dish then we were drawing more complicated things to show to the user we had accelerated 3d people wanted to watch videos and they needed to scale them from about yay big to your actual screen because this was the mid 90s we had some very tasteful very minimal but still graphically demanding themes because again this was the mid 90s where minimalism was key and speaking of minimalism everyone decided they wanted to do everything with their windows we wanted to move them around to multiple desktops we wanted them to be weird types we wanted to use Excise because isn't it cool to have your window be two circles brilliant ellipses I don't care yeah basically terrible everything got ridiculously complicated and this is a quote from a ex Culebra Matassa T developer and if you think that writing a window manager is really easy this man worked on Matassa T and spiral a spoiler alert no one's ever compared Whalen to nihilism ilysm give it time only not so minimum they added loads of stuff 25 extensions I think more we Bend a few because they were terrible even at the time thousands of pages of specs came out I trouble see him yeah you didn't write any specs and while they were busy extending anything they never touch to the core because x3 86 were not the ex consortium the time yeah they were not the ex consortium no one was the ex consortium they weren't doing anything so the core protocol stayed the same and the core server code changed the same the way we achieved all these great shiny new things was to wrap the server and more and more and more and more and more layers we worked around stuff why would you ever fix anything when you can work around it why would you ever fix an OS when you could work around the fact you don't really have an OS because you can run next server which would run video bios's might do it in real mode might do it through an x86 emulator it might run up codes the x86 emulator didn't support might suspend your laptop for you might bash your i/o BOTS my Rheem up all your PCI devices yeah that one should probably be somewhere over here in memory probably be fine no one else wants it and to do that in assembly it was a binary interpreter for elf cough and a dot out across loads of architectures except the ones where it wasn't it was an entire OS basically it was a terrible OS basically not just basically it was a terrible OS who remembers generating a config file please generate me a config file please use this conflict file why still don't know so everything was awful is the summary and that is the last 10 years of xtalks where we've all been saying everything is awful but at least we had some great visual effects to show for it this was the age of possibility I found no that I think that was the worst one ever if you search for enlightenment 13 screenshots search for enlightenment 13 screenshots and Google Images unfortunately most of them are gone cuz themes dog is gone no steampunk knows that but more time passed I became a not a miner anymore and got a passport and we got this wonderful thing over here with a very good domain name first thing we did is say we should let people be able to build it high bead Oh so we modular as the entire thing and suddenly it was all what I told and it was really easy to build but that was quite a lot to build we accidentally split X into 345 I said look that up that's not exaggeration it is kind of a convenient number now look basically sorry but I think it was still in that way mostly though we went the other way we've been stuff we found a lot of stuff we didn't like so we came in with I do remember finding 1.1 million as a pic of this before but that was on my blog which was on a freebsd 4.3 machine which finally died last year I think and that's now lost to the ether so we did that we got rid of over 300,000 of what I originally thought was 500,000 lines of code I personally Bend a hundred thousand which I'm still pretty proud of Ajax beat me by deleting the print server which was pretty amazing it was exactly like ax except where was exactly not like X at all and then you could find out the LPR output by looking at a window property and someone added X print support to GL x gears that's when we knew it had to go but sadly there was only it depends I think it was in a uni so we probably had an amazing laser so you're probably hundred or so FPS faster than what we could do at the time spoofer and yeah sadly at some point there was no more so we started sorry yeah X oh yeah no they're going to head we haven't run out of subtly we're still working on it but in the meantime we visit ourselves with adult work we added stuff again we added a new drawing mod model it's not really a drawing model I'll get back to that later good luck more or less a drawing model to replace the draw some rectangles and some text we added three inputs ducks hi yes nope one of them isn't used anymore brilliant but we still have all four need to interoperate they're more codependent than independent and generally it's all a bit of a nightmare there are exactly three of us who know how it all hangs together moving on we added more display management extensions which again kind of yeah almost independent people still use core x11 fewer in two before you would have heard Dave tell you about housing oil is horrible random 1.0 let you change resolutions which was amazing 1.1 let you change refresh rates 1.2 which was actually brilliant it gave you multiple outputs they could be hot dog genius and now Dave's just done one point four which I almost understand we added more buffer transfer models because the original model was what you are sending all your commands interleaved with that huge chunks of data and brilliant so no one uses that anymore we added the first direct rendering infrastructure no one uses that anymore yeah this isn't during extra dog days got added during the passage of time before yeah even before I was using mark to trade mp3s and vendor goes buying papers so everyone uses the shared memory transport or the second version of the direct rendering infrastructure which is all very nice quick detailer I'll do lwn commenters again take note if you haven't been paying attention to anything please at least listen to this everyone uses shared memory indirect DRI - they don't work over the Internet ex isn't network transparent anymore its network capable but it's not Network transparent so stop saying that because it's rubbish thank you basically we changed everything about how Xbox used to be this nice shiny model you saw before with all the cool little widgets then we wanted cool themes and so rather than teach X about gradients which someone did but it was a disaster we just drew them all client-side and said to the server here's a massive picture don't touch it please you'll only make it worse phones got way more complex so we just dealt with them client-side mostly and we did there was a project to give clients enough information to you know work out how big a certain font was going to be when you drew this and so and so forth but you ended up sending so much data that it was cheaper for the server to send the entire font over to the client and have the client work it out so what we do every button or no not every button every input field every text box used to be its own window in X which is why no one understands the function that validates the window tree I tried to remove 90% of it but then someone wanted me to write a benchmark and I lost interest because it would probably break some motif up from 1995 but realtalk 'it's said all right we don't need this one window that's all it is this is my this is my content everything inside it I'll deal with and then when we wanted to do things like expose again we could teach the X overall about how to do really complex effects or we could just save the window manager you probably know what you're doing more than me anyway and just draw the entire thing tell me what to chuck on the screen I'll take care of that you can deal with everything else so you might be wondering and it is a valid question at this point what the Xserve is actually being left with to do if you search google images for dramatic pause up to a.m. you get some pretty weird stuff some that's fairly online there are no ponies and rainbows spoiler alert that's the other doesn't do very much because the clients draw everything locally I tell the ex over here's what I've drawn just put it up on the window please no more server tells the window manager here's what someone else is drawn maybe put it somewhere we'll see when your manager thinks about it decides what it wants to draw decides where it should be drawn if you're in expose maybe it's up here you're in a pager maybe it's nowhere nice then the episode just does what it's told you put one image up on the screen basically we made a second-deck server because the first one was great by second I mean 105 because Linux is all about choice I did not count the number of window managers and when I say Linux I mean UNIX because choice extends to Carl's as well so we have a name for what the ik server does IPC terrible terrible terrible IPC why is it really bad I don't have a slide saying that it's not int respectable but it's not introspective all it's just a bunch of strings or maybe numbers or maybe pics maps that you just kind of bang around into some windows there's a couple of specs for them but they're incomprehensible and there's a lot of it when I start up Giada today empty text field all I do start you at an empty document 130 times it asks for the name of atoms mostly for window management extensions and their blocking because ex Lib is terrible and their 130 times round trips around to the x over 34 times at round trips around to the x over to get the value of the property so that means it's probably about 75 properties that's tried to find the name of and hasn't worked out it changes 116 properties I tried to find out what they were but I lost interest the one thing I did profile is during the startup I measured how long to see the giada main loop was blocked on the X server for so just really simple cumulative sum it's actually a bit more than that because I'm terrible at maths I only lasted a semester and a half at uni before I went off to Finland and so it didn't occur to me until now that the mats is broken and that's a bit more but 25 will do let's be generous sometimes however because the Xserve is doing loads and loads of stuff which you don't want it to do it spends 1.4 seconds it's about to give up on running it but try to for a third or a fourth time and on yeah this time it took one and a half seconds not really sure why ltte would let me find out I guess but okay I've got a lot of tabs in Chrome but the minimum it took for chrome to start up load all my tabs the minimum amount of time it's been blocking waiting for a reply from the Xserve er 500 milliseconds that's only the blocking and again my math is broken so at least fun and so why do we have these huge variable agencies because of because by trying to bang a rectangle into a square hole so things like this doesn't obviously affect startup but things like when you're very size the server will say oh this window is resized there's a whole little strip around here that doesn't have anything in it draw something in it for them maybe a bit of white maybe a bit of grain graze always popular yeah all drawn usually one way or another you end up waiting for the GPU to finish what it's been doing so that's a you know nice little arbitrary store could be really quick could take a second who knows the point is that it's massively massively variable and so all your IPC might be blocked on the server doing something really stupid and it's really stupid because no one wants the server to do it the client knows what's supposed to be in the window Giada it's still yep some of them is real work it's an example of many even you can't say otherwise yet as soon as the service done that the first thing the client is going to do is say gray is not particularly useful I'm going to draw something over it so the server has massive arbitrary variable agency because it's doing the wrong thing that no one wants it to do yes so we put our minds to a think what would an ideal future look like bear with me clients draw what they want to locally I installed the server what they've drawn the server decides what to draw and wet all we've done is been a really terrible middleman that's four slides Weyland to get here on earth at figures old I think was about a hundred so again lwn commenters if you find yourself saying it X is the UNIX way what one thing is X doing and what is it doing well yes it's terrible at everything it touches which is everything and all I've got the Internet peanut galleries attention those who do not understand UNIX are condemned to quote Henry Spencer anyway calm blue ocean everything's our own friends dr. Weyland Weyland is actually tractable I've gone through x2 try and cram it into time I'm still a little bit over time Weyland you can actually describe to people X I still haven't been able to even with the aid of these amazing slides but the core concept behind Weyland is that every frame is perfect they are not sacred Rob we'll get to why later so yeah a frame here's set of pixels that you should show coherently together at any one time not with any others it's your apps content it knows what it wants to draw that's a frame so doubling back to our original explanation it's a medium I say draw some stuff here and over there and over there and over there and over there and it has no idea if you're trying to draw one key here in frame notes together or maybe you're trying to draw a progress bar or things that should be divided and since there's no boundary between inaudible be displayed at fairly random times and not separated you will see bad things those are actually all spaces took ages di2 which was pretty much the last thing Christian their Wayland author did Forex almost fixes this it does more or less give you the concept of a frame and a coherent presentation of a frame which is all we want the server to do okay not really there's a lot of parts it still can't fix because their core X protocol we can't touch because it's all about the choice to run terrible motif ups we learned on the other hand all we have is frames that's all we know that's a conscious choice display this cool it might not whatever that's it it's all it does so what's perfect means that we don't want the chunks of gray and then swiftly followed up by text we don't want flickering when you resize flash flash flash flash horrible we don't want stuff to tear when you get incomplete drawing and you decide to scan at exactly in the middle of it we don't ever want that to happen because we want to catch up to 2007 so in 2013 it makes us look amateur and it's really embarrassing when that happens Weyland is also descriptive not prescriptive by way of example when you launch a pop-up window in x11 you don't launch a pop-up window and say give me all the input put this window exactly here and nowhere else screensaver give me all the input put this window exactly here which is the entire screen and nowhere else so what happens we can't use our volume keys during a pop-up can't use our volume keys during a screensaver which is great when you resume your laptop and there's music blaring out of the speakers and you just bang the lid shot and deal with it later the screensaver won't trigger when you have a menu open we've tried to fix it we sat down we Specht out something that would let us sort this out once and for all we decided it was actually impossible it would break the ex model too badly and again it's 2013 and that is so embarrassing and why are we putting up with it why are we defending you in Wayland we say this window here's it's a pop-up window this is the clicker came from you know where to put it you know what to do with it you know if I click outside the window should go away sewed it up everything is up to the compositor which means drawing means it could put your pop-up window on Mars if it wants to might be really hard to implement what if there's a buggy compositor which puts my pop-up window on Mars and I'm slightly out of order possible to do don't encourage it don't run bad compositors that's fine window managers are about that complex as well as for the screen saver it's a part of the compositor because one of the other genius ideas we came up with was we probably shouldn't encourage you to type your password into random clients the session doesn't know anything about probably a bad idea security notes tech note Weiland is also event-driven again by way of example when you want to listen for input devices in X you say tell me about the new input devices tell me about the current input devices lock on the server wait till the reply shows up because XF is horrible deal with the reply then whenever you get a notification deal with the notification in a different format so you have to write two functions good Weiland again we've been Dahl those pointless middle steps you register for device notifications as soon as that happens we send your event saying it looks a lot like YouTube it sends your event saying these are all the current devices in the same format as the ones you'll get later saying these are all new devices all the same code handling it and you have to go out of your way to not make stuff hop okay and to not make a dynamic admittedly in the sample clients we ship with the sample compositor we've actually done that but I wouldn't encourage it and I really need to fix it anyway we've got proper object lifetimes that may sound really dull but as anyone seen this before x11 error something it'll tell you some numbers which I won't translate for you because again X is brewing and it's pretty horrible but everyone has seen it because any object in X can just go away if you're dealing with a mouse if you're trying to say tell me how many buttons you have and what they're called because my should be able to have a 32-bit number of buttons someone might have unplugged the mouse and the Xserve will send you back an error which is fatal by default and you have to try really hard to ignore it's only in X lip it's still hard with x cb though and it's still hard to relate that back to what you want it's you have to go out of your way a long way out of your way to do the right thing no toolkit does the right thing on Wayland all your objects are client-side for every input device every client gets its own new object and the client destroys the object when it wants so you don't get any of this I don't have time for this this is really boring but basically there's another hole and class of errors it eliminates blah blah blah yeah basically if you're running a browser and all the four parts of your browser support different versions of a given extension and the server only gets one thing of this client supports this version of this extension so it's a complete crapshoot as to what actually happens and what version the server thinks the client has so it makes it really hard to introduce properly backwards-compatible behavior because what if a different part of the client that's not aware of the new version gets an event it doesn't understand in Wayland everyone who wants to listen on an interface they listen on an interface say this is the version I support tell me about the version you support and they're all separate everyone gets exactly what they want totally Goldilocks everything's brilliant and now you're convinced because I'm a brilliant salesman and that's an md5 nice what can you do you can use something called Western although you don't have to take a terrible phone picture in my life although the lights actually taken out all the chrome noise which is cool there's a lot of blue that Western is our reference compositor for Wayland it's got a plug-in system so you can implement we have a basic desktop shell with a panel installed which I can't show you because my laptop doesn't work with the projector and it's 2013 but that's because I forgot to bring the cable and so it's all fun and yeah it has the desktop shell it's got a tablet shell if you want to implement a tiling window manager you can write a plug-in to Western it does that it's all an external shell that way lund handles a few in the framework and it still supports x11 clients we spin up an X server and the X clients think they're just talking to a regular x over we just subtly bang them into the way one session and it hangs together surprisingly well it supports all your hardware broadly speaking take note of FB dev and Pixma and software rendering it doesn't require GL nothing in Wayland requires GL lwn commenters hi it supports hardware overlays for video a lot better than X does the overlays will give you much better visual quality especially when you're scaling than your 3d call will and Western uses them to the hill if you like C++ something about qml don't know I don't like C++ Marsha the engine behind genome shell has a pretty out-of-date port which makes it a hybrid X 11 and Wayland compositor similar to Western genome she'll was running on it I tried to bring the port up-to-date run out of time but in the grand glorious future it'll be great and someone demoed it once in July so it's ready for the enterprising well you talk it's gtk clutter QT all the ports are in upstream get the clutter port is now in a released version as well gstreamer supports video needs a bit of work there's a bunch of video stuff we need to sort out but it's it's not a core problem but we are working on it and now I'm Steve Jobs so what I was saying before about how X is really terrible variably Latian type AC but extremely chatty it's really needy and also really bad so everything you do will bottleneck through X and it'll do it repeatedly because you'll go from the client to the x over to the window manager to the server back to the client guess what the worst case for this is that it's really horrible this is actually X's worst case is remote display I'm not joking again little thought experiment maybe if we killed all these really long round trips across a connection it takes ages because we're in Australia in the speed of light is still iffy we run a compositor locally the client talks to the local compositor all it's round trips are basically free and then the local compositor deals with through remote control stop because Optus is still horrible and ISPs still have bandwidth cuts maybe let's compress the images or maybe you're on a mobile either way that sounds like a good idea from the 70s I think and if we did all that 2x we'd have VNC and it would be better it would actually be better I'm not making this up again it's not theatric it's a fact so when I say we I mean Christians been experimenting with this for Wayland there is a prototype in his branch in his tree for Western and Wayland which gives you a remote display based on this it looks a lot like VNC but then again X today with a modern toolkit that isn't motif I don't know why I said both of those looks a lot like the NC just a awful version of VNC so we think it's going to be better at remoting the next it actually can't be I'm with up do we have any questions so when can we start using it to show when I'm going to start using it the touchpad driver isn't very good at the moment it doesn't handle stuff like scrolling tap to click an acceleration very well I'll start using it when the port to genome shell is brought up to date again which I was meant to do and when the touchpad driver is on par with X which I was meant to do any day now basically yeah four times so can we have circular windows in Wayland you can have circular windows in Waylon but it looks horrible do you remember trillion on windows Sonique terrible you can you have the freedom to make poor choices so you're working progress remoting is it action time of user interface to rootless or do you have to open the full and desktop to the remote machine and you have the choice third question being is that rootless or is it routed so route had been so the traditional fancy where you get a huge chunk of your desktop can route less where you get here's another window you have the choice Whalen's all about choice I can't um we're just going to take one last question I believe we had someone down here how does things go with multiple GPUs other things go with multiple GPUs that's a client problem we're Sam and Molly are there alright I'm Jeeter cane cluttery his fault told him to sort it up hidden in all seriousness we've made as much provision in the protocol as we can for dealing with that we tell the clients which outputs they're on the GL implementation has a way to associate the outputs in the GPS if it wants but it's a really really hard problem punting it to clients isn't necessarily fair but given they do all the rendering that's about the only thing we can do okay well thank you very much for a very interesting talk Daniel and some fantastic slides you

Channel: HackersOnBoard

Views: 31,743

Rating: 4.8767505 out of 5

Keywords: Linux.conf.au, The real story behind Wayland and X, Wayland, XWayland, Windows, display, OS X (Operating System), GNU/Linux (Operating System)

Id: cQoQE_HDG8g

Length: 45min 35sec (2735 seconds)

Published: Wed Mar 27 2013