Deep Thoughts Engineering Speaker Series: John Carmack

welcome to the inaugural deep thoughts engineering speaker series my name is Kent Beck I help engineers educate themselves at Facebook in various ways and just a second about the series the one of the things I noticed in teaching engineers I do a lot of one-on-one teaching is young engineers don't seem to understand just how deeply really good engineers care about what they do so the the theme of the series is to bring people who really care about their work to talk about what they really care about the most and that's it's been a theme in my career I mean it borders on obsession and it's on the wrong side of the border but that's where really extraordinary results come from so the first lesson for me in setting up this series was asked for exactly what you want you can compromise later I thought well okay if we want to get the best engineers at Facebook to talk who's number one on that list is Carmack like it's not even not even a question and so I just emailed him and he said yes go figure so that was you know sometimes not having a sense of shame is a problem but sometimes it comes in handy so thank you very much for making the trek up from from the wilds of Dallas to meet us I always like introductions that are short number one and this will be fairly short but I also like introductions that are that are personal and the lesson that I learned from John is I saw a post that you made at one point where you you described you'd spend two months working on the perfect now I'm gonna forget the details but the perfect implementation of I don't know sparks flying off a tracer bullet or something like that and there was a there was a really good way to implement that that was perfectly clean and you just couldn't get a performant implementation of it and so you just said well but there's this stupid trick so let me just do this stupid trick and it worked just fine it looked look just fine do you remember this ringing any bells okay so the lesson of that for me is that that even if you're really really smart sometimes the best way to be really really smart is to figure out why you don't have to be really really smart so I want to thank you for for that lesson and I've tried to keep that in mind for the rest of my career and now without any further ado I would like to introduce you to John Carmack okay so I I got the invitation and I do kind of beat myself up for not being more engaged with the broader Facebook engineering community here I do kind of like my little Hermitage down in Dallas but I you know there's more that I should be doing to try to kind of be a part of the community here so I mean I I thought about it and I wrote down at least a dozen things that I could go and talk for hours about that would be interesting topics to hopefully educational to some degree but what I settled on somewhat topically was this notion of ideas and engineering and this was kind of triggered by before oculus connect I once before last Michael Abrash had sent me an email asking if it was okay to tell one of the stories of the original quake development about how I had gone through all of these different paths for basically visibility calling to get the speed up to a certain way before settling on this one potentially visible set IDM and he did a nice presentation at connect with this the basic idea being that research what I've you know what he's the head of here at oculus is not about these lightbulb brilliant ideas that just strike you out of them you know out of the blue but it's a whole lot of work that leads to the ideas and I quibbled a little bit and said that there there definitely are lightbulb ideas where you can be sitting there working and something does come to you in a flash that may or may not be a great idea but the core of the lesson is absolutely true that you get the ideas by being down in the mud working on the problems that it's the hard work that leads you to the insights that are hopefully the great ideas and we have in our culture the broader culture here this sense that everybody loves the idea of the lone inventor that that has the brilliant idea that then they patent it they they get rich afterwards or it's stolen by some faceless evil company and they're exploited and these memes run through our culture in different ways and you know over the years as someone with a little bit of notoriety as a game developer or a rocket scientist or whatever I I get emails from a lot of people that are often you know earnest people that believe that they've gotten magical idea and usually it's approached by saying I've got a great idea I you to sign an NDA so I can tell you you know my my great idea and you know and then maybe you can work on it with me yeah I they're usually people that you know think okay this idea is magical I can't build it but we need to find somebody that can implement this idea and then we'll all be hugely successful now sometimes I can talk these people down from their you know their patent application and their NDA requests and I can have I you know I can look over the stuff I mean I gave advice for free I'm happy to look like I'm happy to hear your ideas and tell you what I think about them and invariably within these cases we'll go through the stuff and it's either something that's you know obvious to someone that's kind of a part of the industry that they're talking about or it's something that has a fatal flaw or it's just kind of incoherent or it doesn't really address things in different ways but I still try to encourage most of these people then you know if you're thinking about ideas this may be the first time they felt that they had a really great idea and often they're you know they're young not always but I am you know I tried to let them know that it's great to have ideas but you need to have lots of them because lots of your ideas won't work out very well and the way to go about it of course is to to try to build the things try to build the prototypes and and often it means you know really knocking your ideas down that it turns out that you know you're your great idea is probably not that great you may need to have a dozen more ideas before you land on something that you know that really is great there's a there's a quote by Robert Heinlein the odd science fiction author and he said that an idea is worth exactly one bottle of scotch where for a writer you know someone who's even more than software in many cases where ideas are the stuff that they build it's still a matter of execution is much more important than that sort of original core idea and there was another Heinlein story about how one of his friends I was a struggling writer was really down and he just highlights said over a whole page full of ideas like wonderful ideas that could each be turned into some great science fiction story with just they you know we hope that this gets you unstuck that you can follow something off from this and I've taken that tack through most of my career trying to say that you know the ideas really aren't the important thing it's the execution where a good project is a result of hundreds and hundreds or even thousands of good decisions that are made along the way that you know you could start with a great idea and yet the implementations not good it's you know it's gonna fail that good implementation is where you need to focus your efforts on because the ideas they do they come and they go and it's it hasn't mattered that much now I us into a lot of the way I think about things this influences in many ways my views on software patents in some of these other areas where you know I don't think that trying to nail down this high value on a specific idea when the idea is this tiny fraction of the product is a particularly good thing but still there's this sense that clearly ideas mean something there's plenty of cases where you have competently executed projects that failed to leave any dent on the world even though they were done with all the right decisions but there was something missing there of maybe there was grand ideas that that were necessary to make the big impacts so I've actually been toying with this idea about ideas recently and this is you know what you called sort of a cocktail party theory something that you wouldn't necessarily defend to the death but it's worth thinking about a little bit and that's that ascribing low value to ideas may be helpful in getting more ideas where I see a lot of these people that feel that they've gotten this idea and it's so valuable and they latch on to it and they've you know they think that this is something that's going to be great they won't necessarily look at it as critically as they might want to as they probably should or they will be jealously guarding of it or different behaviors that I see you know I see in other people and in fact I've seen in my earlier days I can remember having early ideas when I was at the age of many of these people that email me and say I am you know it's like oh this would be this is gonna be a great idea like I can remember on the PC 80 thinking I could hijack the DMA controller to blitz screens and I'm like oh this will be great this will be an enormous speed up it'll be asynchronous and it'll be great and I never could get it to work and I was bummed out for days afterwards I was like oh this brilliant idea that I was all kind of hyped up on I just wasn't gonna work out and so I try to when I've got like a timely thing about this was I had already decided on this topic I am you know a few weeks ago and last week I got yet another one of those emails coming in from a young guy that said I've invented the holodeck you know I have I you know I've come up with this invention and I filed a patent for it and he sent it to me and Michael Abrash and Palmer you know Palmer is probably getting a lot of these things now also and I you know I had to tell him it's like well first of all I'm sorry but I can't read your patent application I am but you know I try to be encouraging still I tried to say it's like alright well if you need to start prototyping things I like I should have mentioned that the reason oculus is here today is because Palmer not had a bright idea but built a whole bunch of prototypes that eventually were physical artifacts that could make their way to my hands and could wind up building the things that kind of got us where we are here today so this sense that the idea wasn't the important thing but you know going iterating on it billing it over and over so I've been thinking kind of through that lens though is that well am i ascribing perhaps too little value to some of these things but doing it as a psychological hack where I'm tricking myself into not wasting time or spending time in ways that I you know that are not going to be as productive as if I just say well ideas come ideas go and I like another after I had mentioned to Michael a brash about the original I am my comments on there I said a couple more emails and just like well here's the idea that I had this week here's the idea that I had right now and now I look at them and saying well this is kind of interesting this would be awesome if it works out but it probably won't I almost look at these things now it's like I've been through the idea train enough times now where you look at something and think wow this will be glorious if it works you know and I get the idea hi at beginning but done it enough times now you can say well it probably won't and it can almost be the puzzle game then to find out how to bust my own idea where I for instance just a couple weeks ago I was discussing with the Microsoft Mo yang developers about Minecraft you know ways to optimize Minecraft for VR and different things and this kind of tiny little topic came up where this is a very very small idea I'm thinking that okay if we optimize sort of everything's broken up into all of these blocks the world has broken up here and there chunks and they're rendered you know they're sorted in the order that you render them but maybe there would be a way that we could optimize the ordering of the indexes that make up all of these individual blocks so that we could guarantee a certain draw order irrespective of where you're at this would be a minor benefit it would make certain the scenes go you know a few percent faster and I started thinking about there are some cases like there are some classic cases with occlusion and what's interesting is these thoughts go all the way back to like the times of quake thinking about draw order and the fact that they can come up in topics that are happening you know today 20-something years later is interesting there's something to be said there for ideas that even didn't pan out in the old days sometimes come back and can make any impact reality is like that but one of the classic issues with computer graphics is drawing order either you're drawing back-to-front and painters order or nowadays on modern hardware you're trying to draw front to back so that you don't wind up drawing things once you've already filled them in so in some cases for simple geometries there are I there are orders statically that you can assign to things where if you had a surface like this and you had back face culling you could say that if you wanted to draw front to back you could always draw this one this one this one and this one because nothing can ever occlude those they would be the topmost thing then you would have to draw these two because if you were over here looking at it this one could include this one and then at the final part you could draw those in stage three so for some things there are static orders this is that you know this is a small thing that I that I looked at back in the days of doom and quake in the early days of graphics technology now there are lots of surfaces lots of orders of geometry then don't work out like that like a you can make a set of three things that go you know that overlap in a way where there's no static order that you can draw then draw the triangles in a way that will guarantee that they get I they getting drawn in the right order you have to introduce splits so you know being aware of both of these cases I was at least thinking it's like okay well minecraft has this interesting characteristic that everything is diced into blocks it's all just cubes so is it possible to apply a static ordering since you know that nothing would ever have to be split like this because they can't cross over that would it be possible to make a static ordering this is one of those things that this would be clever it would be no additional memory I it would give us some speed-up in some of the worst cases when you have lots of occlusion and I'm thinking oh this is a great idea aren't I clever this will be fun and I can start thinking about the implementation but I I look at it like okay I start thinking through how would I build this going kind of from the outside in until eventually I started drawing the pictures of the blocks and figuring out what busts the idea so you have blocks like this with a solid wall here solid wall there a little wall there and there you have a sequence where from over here this one has to include this one from there that one has to include that one but from here this one has to include that one so the idea is busted it doesn't actually work out seemed like a great idea but there's a clear failure case and it doesn't work so this is just the the type of little idea that I I ride the idea high at the beginning I'm like oh this is a great idea this has all these wonderful characteristics but it's a little puzzle to figure out how to bust the idea and I so the author Nassim Taleb wrote the book black swan fooled by randomness and is more recent one was anti fragile and he espouses the kind of interesting theory that I that I recognize from the way a lot of the way I look at a lot of things and the idea is that your best system is one where you can take great gains from the highs from the winning solutions but you are not damaged much by the lows and that's the way I approach ideas now where I get an idea I get the lightbulb or the little moment of enlightenment there and joy this is great this is like the best time to be you know doing what I'm doing having a brilliant idea thinking it's gonna be great but I am already kind of slightly smiling to myself it's like it's probably not gonna work out but it's sure fun while I'm having the idea and then hunting the way that it winds up not working out is I it's gone about is an effective challenge where that's the best use of the time where if you're not trying to bust your own idea you swerve away from the things that may wind up punching holes in it I mean you see this in all sorts of things about how you know people when they're they're invested and attached in ion to certain ideas they will look at the things where it looks like the ideas working out but not so much on the things that may punch holes in it or the things that that may challenge it in different ways so now I try to look at these things like okay get the joy out of it but then really try hard to punch holes in my own idea here and that becomes almost an enjoyable task there and I can pat myself on the back when I break my own ideas I'm very happy with showing clearly that this doesn't work but these wind up coming from you know I could trace they even on the broken idea the depths of these things that went back to graphics work twenty-five years ago and that's down to again the part about being down there in the mud working with the idea and that's how you get these moments of enlightenment it's working through you know farming the ideas and the things that go into it I have this argument or this discussion lecture even with some creative people where I would have some designers that I would talk with that would be in kind of this you know this funk about their they're not feeling inspired they're not I you know they're not getting their creative spark that they're looking for their muse isn't with them in some way and I will be brashley telling them it's like no you can farm your creativity by just working through you know exposing yourself to lots of things always looking at them through the lens of the problem that you're working on and plowing through not staring at a blank wall which I think our culture somewhat inappropriately does give this vision of the genius that is just staring they're doing the the thought experiments staring at a blank office wall and that's how you get your enormous insights and while some that does happen you know works for Einstein in some ways but I think that it's far more productive to be just trawling through all sorts of stimulus for the different things that you're looking on looking at it through the lens of your current problem I mean sometimes it pays off and I sometimes like I was recently reading the book the annotated Turing I have a pet Souls book on kind of going through Turing's old papers and there was talk about it started going through something about how there's a mapping from the continuum to all points in an in dimensional space and I got hugely excited because there's all this stuff that I'm working on with our interested in with light fields and some of these other areas that are these high dimensional spaces that are very sparse like oh and some new mapping for this would be a wonderful thing this would be you know great and once i deciphered the mathematical language unfortunately was like oh i already knew that this is kind of just the bit spreading trick where if you have a you know a sequence of bits going off in some way you can just turn that into a two-dimensional space by taking like every second bit here making coordinates out of it like that and you could do that bring any in dimensional space you can spread them out but still it was one of those things where I was reading a book completely unrelated to anything that I was I you know that I was working on and if I hadn't already known that that would have been a good insight that's something that's useful for a lot of cash blocking things and different stuff like that but I had kind of run across on my own but that sense of being out there searching for not so much searching specifically for something but continuously exposing myself to all of these other things and seeing what the current problem that I'm looking at I applies to it and I find myself more and more nowadays seeing this serendipity that seems almost you know almost weird in some ways that I can be learning something in one place and then reading something completely unrelated and finding some reference to it or some tie between that and that's I know that's been very interesting but the problems of dying like different things one of them one of the significant problems with virtual reality and this was the first idea that I forwarded to Michael right after he had sent me this I'm you know the original email there one of the things that you've probably heard people talk about in VR is that we need 16 K by 16 K displays to kind of really get retina class resolution for reality iein and historically you you look at that say clearly while we could actually build that with like way for scale integration of silicon micro displays I that would be kind of an interesting thing we couldn't render that in any reasonable system today and really that's not the direction that things will eventually get it's not nearly as bad as that seems because the eye only has that resolution in a very very small area so called the fovea and several of us have done research and prototyping where instead of rendering the entire screen at a super high resolution you render the screen at a low resolution and then you render a smaller inset area at a higher resolution this works in is useful even on our systems today because our optics are not great across the entire field of view their clearest in the center and they're blurry or as you go out so there's some argument that doing this sort of rendering today is been a good idea and this is one of the reasons we've been pushing for the multi-view extension which is a way for us to kind of push all the draw calls for a scene down and then have the driver and the hardware reissue those two make a second stereo eye view and ideally than a second foveal insert and another foveal answer for the other eye so this is something we've been kind of pushing on for you know for a while but the the magic screen that we want and there's work being done on this now would be something that could track your eye and then move this region around to exactly where it needed to go that would be something where you could deliver hopefully this retina class resolution by having that in only this tiny little area no screen like that exists now but we have people that are working on technology that may get us this maybe sooner rather than later so there's still the question of how you wind up delivering all those bits to it and rendering it a few of us have built systems like this where you render kind of two separate views but we talked about the fovea as if it's this fixed little region like here's the fovea region of your eye and it occupies this many degrees of Arc but it's really not like that at all if you look at the Wikipedia graph picture for it you wind up having a something that looks kind of like this for the resolution with oviya where lots of resolution right here and then a smooth curve going down with like one little gap where you've got your blind spot where then the optic nerve kind of goes in and looking at this it occurs to me it's like well you know the shape of that curve that looks a little bit like I am like a projective like a projective texture mapping curve like you've got to divide in there somewhere and I start thinking about this again where if you had this two-step thing then you're looking at saying we have one image at this resolution and then one image at that resolution and you can sort of take the integral of the difference between there to see how non-optimal that would be for what you're trying to do it's still a whole lot better than picking one average sort of across the whole thing especially if you want to get a lot in there but it's not really what you'd like to have so I start thinking about how else could we address to build some rendering like this and one of the tragic things about the conventional graphics pipeline is the wider you make the feel the view the worst it puts the pixels in where if you have a field of view something like 80 degrees or so each pixel on here it's not too far off between the angular resolution what you get as it goes over here what's in the center but the wider you make it the worse it gets if you go and you render 120 degree field of view something where it's like that at that point the pixels over here at the edge you actually have much more resolution on the outside which is exactly where we don't want it it's doing the inverse of what we'd like to have which is what pushes us to saying it's like well we'll do the whole broad thing at some low resolution and then we'll do a tighter inset but started thinking that I projected geometry might be able to help to help us with this and this is something that I I think a lot of people may take some heart from this but a lot of the math the heavy math in projective geometry and uh you know a lot of the higher in math that goes on it took me a long time there were many many years a decade when I was considered this graphics guru genius when I really couldn't do from scratch a lot of them you know a lot of the mathematics that underpins a lot of that but slowly eventually with a couple decades of experience most of it did eventually sink in on me to to be able to do to formulate some of these things more directly and what I started thinking about was well what if you had if you have a single view that projects instead of projecting normal onto the surface you're projecting kind of off centered there which you could think about this like well it's almost like you had the whole screen there but you're only taking the the end of it where it's stretching out in the wrong direction that we want they're making it the right direction and then doing the entire way so you're looking down sort of at the corner of something it would be rendering once that way and once that way that winds up giving you the direction of the compression of the angles going in the right directions it's taking what's doing the opposite of what we want they're turning it around and making it do what we want there so that winds up getting something that that kind of curves up gives you an increasing curve they're not an enormous difference you know this here goes is maybe a factor of two off from what you'd like to have but this makes it a factor of two in the right direction which makes it a factor of four or better than what we would have from our conventional rendering and a factor of four is there's nothing to sneeze at um but then you'd say well what if you wanted this is this goes down by like a factor of a hundred in the total case from the very outside to the very the very top it's it's much larger well maybe you could start stretching this out what happens if you I if you stretch it out to something extreme like this you can build any amount of angle that you want out of that so this seemed I like an interesting idea this is one of those ideas that I has direct impact on what we're doing I may have some real value to getting this this big thing the retina class a Virtual Reality Display I mean the hardware people have to figure out the magic to make it happen and focus but this might be the software side of things listen so I'm it seems like a good enough idea I start I'm so I take the steps to try to bust it I go ahead and I write they you know I write some make some checker boards and render the images there start putting some images on it and it looks really pretty promising looking at the corner of a cube I get tripped up a little bit because the three-dimensional aspect of this is not the same as the two-dimensional we're looking here it's clear how everything breaks down but looking in three dimensions you've got a degree of anisotropy as you look down into the corner of a cube but still it looks pretty good stretching it out starts to show some issues where the pixels then get very distended and there's a degree of anisotropy where the core center of it where your fovea would be looking is extremely high pixel density and could get up to very much what you'd like to see like that but on the outside you've got these very stretched out pixels and I don't know whether that's good or bad or a problem or not that's something that might not be a problem because that's inherently what you're not looking at that's something that's only happening at the edges if they're well filtered maybe that's a good enough idea but even if it only works out where you don't just end it you just leave it at the cube there are some GPUs have hardware support already for distributing primitives across cube mostly for rendering environment maps so if you say well we're gonna take the three sides of a cube here and we're going to push one stream down it can split them all up that seems almost optimal maybe that's only a factor of four better than what we're doing now but factor of four is a pretty wonderful thing in a lot of cases so this is an idea which I'm at least cautious I am about maybe a great idea this may be something that you know that has some real significant wind to it and I I noticed that I feel good when I try to attack the idea early I see this where ideas that I have that I don't try to engage and grapple with quickly I find myself falling into more of a pet idea side of things where I have this problem a lot now with I'm not actively building rockets but I still think about aerospace stuff I still think about all this stuff when I'm trying to go to sleep some night having rocket systems dreams whatever I am but that's problematic because I don't have the shop running where I can go and build and test those things right now so I wind up getting ideas and they sit there and they turn around in my head for awhile and without testing them I get a little bit attached to them they're like my my pet ideas in some ways and I think that I fall into some of the same traps that I you know the people that wind up emailing me about the things wind up thinking for instance I'm oh one thing just every time I look up at ductwork I see spiral welded ductwork and this is done by you take a strip of metal and you instead of forming it into a cylinder and welding it in different ways you have a machine that winds it and makes a continuous weld and now that's an exciting thing from like from a rocket engineering standpoint because normally when you make a cylinder you have if it's a long cylinder you've got welds like this and then you have one weld that goes down the side now the problem is is that in a cylinder you've got twice as much stress going this way as you have going this way and welding always impacts the strength of things so you have a supercritical well going down there so you get companies like SpaceX that do very nice friction-stir welding to try to make the best possible things but if you spiral weld it then all of the welds are at a small angle like this which is you've got twice as much strength going this way so even if the weld D rates by 40 percent you still have a really good solution and plus it's really cheap which is why they make ducts out of those so I wind up thinking about all these ideas about how okay we should just let's take some marriage in steel and roll this through here I get that worked out and then you've got a great system that can be built inexpensively or perhaps even on-site you can build enormous things like this maybe you can spiral well to saturn v tank on site where you're going to launch it or on a barge or whatever you need to do with that maybe it's even possible to to gently adjust the angles to make tapers to be able to make you know actual cone sections from there to make a one-piece monopropellant based I you know launch vehicle tank and this is an idea that I've actually been kicking around for a while and I'm probably gonna be years before I wind up getting around to testing this and it's so that's not as valuable of an idea in many cases because I haven't been able to test it I probably won't I have a fondness for this idea and I get reminded of it every time I wind up looking at exposed infrastructure but then there's other things like I am you know the way a rocket engine works is I you have a combustion chamber you know you make hot gas up here and it expands mostly through and accelerates and converts then you're the thermal energy into kinetic energy coming out and there's all these issues about how I you want a large expansion to accelerate it but you can't do that inside the atmosphere again an unrelated thing comes up I read some statistics about muzzle velocity is a battleship guns and there's an interesting aspect about that where they can fire at several times the speed of sound even though the charge that's firing a you know the shell is less than half of the full munition weight there and you run the numbers from the traditional rocket equation and they're going faster than then they should be if it was a rocket with that amount of propellant of that ISP and it's for an interesting reason a when you have a gun you have a whole bunch of propellant in here with a rocket engine propellant comes in and it turns into gas kind of immediately it comes in turns into gas here the gas it combust and accelerates with the gun you wind up having something where it starts burning in different cases but instead of burning at the end like if you flipped a show around and lit the end of it it would be a gas generator rather like a rocket but if you're actually pushing it out the interesting thing is that the burning propellant here is not only pushing the payload but it's pushing the other propellant so the propellant gets pushed and then it starts burning and you wind up being able to achieve in some cases a higher velocity than you can with a traditional rocket so I spend this time thinking about well how can we make a rocket engine that might have gun barrel type dynamics and I ideally think about this dowel and I've got sort of a test plan in mind but it'll be years before I wind up kind of getting around to spinning something up to him to try this again I am but then there are ideas that are at this middle ground where light fields are an interesting technology this is again something from recent months I so we have in virtual reality we have panoramic videos panoramic photos where you basically you can take the two kinds that we have right now in production are mono scopic panoramic photos which is where you sample the environment around you as if theoretically it was from a single point source realistically there multiple lenses and they have to deal with stitching and warping things around but it's a sampling of the environment from one source and those are you know easy to make fairly reliably but you don't have any ability to have depth perception between your eyes or to kind of move around between it at all I am the next step up from that are the stereo panoramas and I am and videos which are an enormous hack I am there they deliver value for us right now but it's really a mess how their how they're defined and how they're created that takes lots more cameras that are somewhat offsets rather than all focusing into the center and you stitch together something that looks kind of right for your two eyes and can deliver stereo when you're looking at things but the things further away from the center artists right till you look over at them it's an intermediate step but where we all really want to be going is something close to do Whitefield technology and the idea behind light fields is that if you have iron it's like our normal systems here you just have rays coming off from one point sampling an environment around you if instead you had some bounding region around you and from each of these regions you had light rays coming off in all the different directions you could in theory then synthesize any view inside here so you could say well I'm gonna view up here I want to make a view this way I'll take some of the race from here and some of the rays from here and this this actually works you can have if you have enough rays to be able to build this stuff from I am you can synthesize moving around in an area and that's pretty magical I think that this is what we want from our VR snapshots where you know you can duck down you could look around a corner you can get all of this but a raw light field takes an enormous amount of data you know it's gigs and gigs and gigs of data and you're trading off resolution at all times because I in the extreme case you want to have what winds up being one rendering sort of for every millimeter you could take the box that you want to view in if you want a one meter box you in most raw case you want to step every millimeter or two across there and make these thousands and thousands of renders at the full resolution that you want to be using and this is interesting for you know for demos there are some things that have been done in these relatively blurry images the step that people take generally to start making this more practical is to turn it is to not just have raised with directions but include depth so you've taken image here and you've got a picture but you've also got a depth for it and that lets you do much smarter things as light as far as saying well I want a sample array here I'm gonna hit it at that depth and I can say well that's not exactly the one that I want I probably want something over here in the simplest case if you were inside a beach ball then you could have one panoramic photo with one depth and if it didn't have I if it didn't have surfaces that changed a lot in their view as you moved around that would be all the information you needed but in most cases we have a clue to this you know if I'm from here I need to be able to know that what's behind that occlusion over there so you have surfaces that you have views that are rendered in more k-i from more points of view the depths are tricky to get if it's synthetic if it's just a ray traced image then you've got the information immediately but if it's a real-world capture then you start having to apply these computer vision and computational photography things to take your multiple cameras extract depths from them add them to your structure here so that you can start rear end during them but they still take an enormous amount of data you still wind up I if you want to have significant occlusion and like complex structure there you may need dozens of images at these full resolutions that you want that wind up being circles or Hemi cubes or or something along all these things to the point where it's still stressful on even modern high-end systems so the idea that you know I started thinking about on this and this came as a result of you know getting o toys demos talking with them about their stuff and then you know I I toss the ideas back their their way see what everybody thinks about it but the trick when you do a projection from here if you have an image and say it's got a room and you've got some object inside there so you've got big death discontinuities here I'm when you wind up sampling a point on there from array that's not exactly where this camera came from you have to wind up making an adjustment to the sample that to the position where you get because you could say this would be exactly the right point if I was exactly at this camera point if I'm offset from this then the points actually going to be over here a little bit and you can tell make a calculation based on the depth now interestingly this is exactly the same system for doing what we call depth correct asynchronous or death correct time warping where we have this system that we apply to to the virtual reality renders where you render your scene and then at the very last moment before we show it on the screen we sample the sensors again and we distort that scene in a way that lets it be closer to where you actually are rather than where you were when you started on it normally we only do this for attitude which is a simple projection but if you have all the depth you can do this reprojection so that even if you're moving just side-to-side you can have it do some of this stuff you'll wind up with problems at the silhouette edges but it's interesting that this is a technique that I had been working on years ago just for them I am for the time war pre projection but it's also directly applicable for all of these light fields so the I'm one of the problems with that is that if you start it you hit this point on the depth if it actually says well you should be over here if that's on the same surface then that's really a good it's a good approximation but if it's moved off of that surface and it's on something different then that's actually going to be pretty wrong and if you look closely at a lot of the if you get a chance to look at light field renderings you'll often see this on cases where I guess technically we should call these luma graphs if they've got depth depth augmented information with them but there's a lot of confusion about terminology and everybody's making a light-filled camera now if they've got more than one lens just that's the buzzword to put it but I am you'll see that in general these things look remarkably good when you have eye surfaces without a lot of occlusion you can look at it and it can be great you can duck down move around look at it from different angles and see changes in specular highlights and everything but there are usually problems there at the edges of things like if we had one a light field here as I'm looking around that occluder there there would be issues where you would see ghosts and double images and some of these other other artifacts for it so my thinking for this is that if you just sample from you you've got a grid of light-filled images of images captures here and you can sample between the four of them there you've got the four closest images and then you can find the four closest trays on each of those and you can do this I just straight filtering through all of those but you wind up with ghost images my my first thought on this was that well we can actually tell from the initial point and then the final point there we've got a depth from that we can tell whether that was close or wildly off for what we were initially looking for so instead of just blindly blending all of those samples back together we should go ahead and look at them and throw out the ones that have clearly gone to another surface so you wind up just just sampling from the ones that are the closest there to get rid of most of the ghosting but then as I started thinking about the samples that we're looking at here the ones that we're getting you're in between four samples there and you've got four samples that the Rays gonna be between there's gonna be sixteen samples at your disposal but what you really want to do is find three of them that bound the sample that you want where you want to be able to say that I have a sample from here a sample from here a sample from here and then here's where I am and just do the interpolation between those because you may have other samples that are out here and then some of them that are way over here because they actually went to the wrong surface if you can identify the samples there there's an interesting idea that I that occurred to me that we may be able to save three-quarters of our pixel samples here by saying that we have say we have the four images that are the nearest ones for what we're doing here they're all slightly different I similar images they're very close and you look at that so there's an enormous amount of waste where it looks like the same wall on each of these it is slightly different they're slightly different positions so they have slightly different rendering artifacts for them but it's I you know they're there's a lot of commonality now these compressed quite well what you wind up doing if you want to compress these or for transport is you wind up saying you have 64 images or something on here you just wind a spiral out video encode it because it does very much look like these linear moves moving around and it encodes quite well but you need them all decompressed to actually render it them so the idea that I'm excited about that I have not yet proven they're disproving here is that if instead of making these from exactly the same central render points if we offset them by a pixel each way so that you have a dither interleave of the samples so if you had I know it's a sample from like this one might have samples from those angles and then the next one over exaggerated scale might have samples from that angle so that they're sampling different sets of angles now that would mean that the image there would be aliased you couldn't sample it directly because if stepping over information that you would need for that for the direct referencing but if you gather the potentially 16 samples and it may only need to be done with potentially four samples even you've gathered the samples from those different ones throw out the ones that have these kind of off kilter depth values and then you find which ones bound the surface there I think that that can be done where you can get a 4x space savings and again for X for X is actually usually the number where things start mattering and I mentioned that on the resolution side of things and I used to say that about I mean a lot of computer speed stuff where 4x is when you know your computer got a whole lot faster I that would be I you know that would be a great thing and I need to find the time to go and test this it's you know it's great when you wind up having more ideas than you get an opportunity to to explore so of course I send some of these things over to Oh toy and I hope they get to go explore them because I'm happy if it if it just works out and it's possible that this could even be extended more ways where if you can tile by or if you can interleave on a 2 by 2 factor maybe you can interleave on a 3 or a 4 by 4 factor maybe there's a 16 x factor that you could get that would mean more sampling it would trade a more expensive fragment shader to go and gather all of these and discard them and and do those tests there but it's a potentially larger impact there if you go too far you wind up getting to the point where some of the changes in visual effects from different angles that you want to get from things like light fields you might not get because you would have something all the way over here contributing samples that go over to it but I'm suspecting it may be good for at least this this kind of factor of four I which is exciting and maybe it's good for even more another path of an idea in the last month that has that is actually working out extremely well is again on the Minecraft project one of the things that we've learned in virtual reality is the simulator sickness that most people associate with VR there's a lot of different types of discomfort that are issues in VR you know there's I strain from focus I strain convergence I'm mismatched but what most people associated with is the kind of sick to your stomach queasy feeling of simulator sickness which is mostly a fact a matter of when your brain is seeing an image set that it does so that it accepts well enough and it's accelerating in some way that's different from what your vestibular system is telling you that mismatch causes the discomfort so you know if you've got something curving up on a roller coaster and virtual reality but your body's staying still your body is saying it's like my eyes in my vestibular system are not matching you know you probably ate poison you should throw up and that's nice you know that's the kind of general theory behind a lot of this so you know in VR the thing to avoid is acceleration if you assume that your user is seated or standing roughly in place anytime you're accelerating that's when the body is going to complain linear motions are generally not a problem you know you can be kind of moving along rocketing along even at speed in a straight direction and if there's not too much stuff like cluttering up your view that's usually not a problem but if you slowly accelerate up to a specific point like certain space game launchers that I'm you know kind of rocket you down an area that's particularly bad news and some people just like I close my eyes for that point because it's going to you uncomfortable now there's an interesting aspect about this where if you make changes indiscreet enough chalks then it winds up not being uncomfortable so one of the obvious things that especially affects the PC side of virtual reality is when you're turning that's an angular acceleration you know you may be staying motionless but the angular acceleration is really bad news and even people that you know I am far less prone to simulator sickness than not than most people and my theory about that is because I wear multifocal glasses the world is always swimming a little bit so I get it's trained me to be better at virtual reality but even they're just sitting there and spinning you know stick turning I call it where you're going based on a continuous input is pretty bad and you don't want to do that for very long and some people will be almost instantly sick from it but it's been found that if you if you break that up and you only do it at say ten Hertz where it's no longer smooth so it's this choppy stutter turn it winds up not being uncomfortable now it doesn't look great because it's stuttering that way but people can functionally play through things and you know and wind up not being sick now there's an interesting trade-off there because I it definitely gets you out of that sense of your brains like I'm buying this this is great I'm in this virtual reality but stutter stutter stutter you know that's okay I did quite buy that but at least I'm not sick but it's a trade-off that you can make but it's always better to try to remove them the kind of accelerations as much as possible I mean I advocate for my gear VR playing in a swivel chair because you can just turn around like that wherever you want and that's the worst offender kind of solved in a more direct way but minecraft is one of these games that is almost the worst possible case for simulator sickness because not only is it rotating around as a major part of it but it's bounding up and down parabolic arcs are not your friend in virtual reality which is a shame one of our very best games as a jetpack a game Omega agent and a lot of people really can't play it because it's lots of arcing around like that even with some remediating steps that they've taken but so trying to address the issues in Minecraft has been an interesting challenge obviously I say well just don't I you know just don't turn with a stick turn in reality whenever you can and if you need to turn with the stick well we can do the stutter turning that's been proven out in a few different places and that that largely works but there's all of these other little things in there the bounding up it's so much about hopping up blocks falling down blocks and that's just bad it's all of these linear and all of these accelerations and there's all these little things with minecarts and other stuff that do pull you around and even starting and stopping like I've tried to argue that games you should be instantaneous start none of this accelerate up and friction ramp down but when it's something that again it's a game that you give me play multiplayer you don't want the VR people behaving in a different way and that becomes a sensitive issue from AI you know especially when it winds up being competitive so the idea occurred to me and it's like well what if we know stuttering I basically solves the comfort problem I what if I just identified any of the accelerations that are going to cause a problem whether it's a slow start a friction stop a minecart turn or falling off and I just stuttered it to ten Hertz this would be it's like oK we've identified the root of the problem and here's a solution for it I you know maybe this solves everything you know implement it and it's terrible you know it's completely a jittering all I mean it just looks it doesn't make people sick but it's not fun to play because it's jerking all the time it doesn't feel good doesn't feel polished I am so it's kind of like well there's something to that thing about identifying there's all these things that could cause accelerations and maybe there's a better way to approach it so what I I what I came to after that as the ideas like okay I'm logging the information say you've got AI you know a path that goes up like this and if you if you smoothly go through that that's a set of accelerations it's going to be uncomfortable but if you were able to go ahead and linearize that and you just made it a linear ramp from there to there that's no longer uncomfortable there's one there's an impulsive change to another slope and another impulsive change when you get off of it so I I logged the positions positions and velocities the same it builds off of the work that I was doing with the the kind of stutter elimination and I kept a little bit of a backlog and I called it lag and linearise where it lags it a few frames back and then turns it into linear steps and it would do a few kind of smarter things about identifying exactly where when you made a transition to jumping up or falling off it could push it there and this worked reasonably well there was I it was a tunable parameter where you could say I at the start minecraft works on 20 Hertz tick so everything went in multiples of that so if you double that and said okay now it's essentially a 10 Hertz linear step but everything gets interpolated between that helps a little bit then you go to 3 or 4 steps and then it's clearly doing this linearization thing the the jumps start looking a little bit more like triangles I have you know not being parabolic but getting more comfortable but at the expense of of being adding latency to the gameplay and of course latency is something that I crusade against in almost all other cases it's strange for me to be saying well we're gonna add late and see here we're gonna spend latency to try to make it more comfortable and this had this was having some good benefits it was something it's hard to test him it's hard to evaluate I'm you know like how much less comfortable or uncomfortable are you getting from this you have the tests that run up and down the mountains over and over and say it's like well do I feel more or less sick it's hard to get back to a baseline and going through the different cases there but I would go and play for an hour at a time and say I this feels better than what it was before but I wish there was a more and more scientific way to make an exact statement about it but some other people were complaining about them the addition of latency my thought was that we'd make this a slider where we'd say it's and you have to be careful how you word things like this because you don't want it to be something it's like you know more comfortable because why would anybody not want to be comfortable you know you have to it's the the trade-off between responsiveness and comfort and I thought we could have a slider that goes from like one to four from there one is just as it is normally and four would be the more linearized that you could about the most that you could tolerate before really hating the latency and we leave it up to users but after beating on the problem a little bit more and trying some of these tricks about like well identifying exactly when you just start to jump so I could start the linearization there but I started looking more closely at the actual data of what I what all these transitions were and it should have been obvious again at the beginning but there's a 20 Hertz tick and everything is deterministic a jump always has the same set of I of changes where it was I think 9 something like went up to 99 frames to go all the way to the top and land on like the next the next one up or 12 if you wind up coming all the way down to the ground so there were basically three important cases that happened here there was jumping up landing on the next block there was jumping up and landing on the same level and then there was walking off and falling down to the level below and it turned out that there are I a small table of like 20 values that I would take from position and velocity offsets so I could tell what I wanted when you jump from here up to there is not this parabolic arc I wanted this perfect linear arc that just take your linear line that just takes you directly up there and then from here one that just takes you straight down now these were twelve ticks this is far more than I would ever try to put in with a lag and linearize approach you know that would be you're just far too much I am Lag to be comfortably playable but if I could just make this transition from saying alright I know that if you are at this offset from a tile then you were at the beginning of a jump and I could map all of these two points along the linearized line and this works magically this is one of these I I amazing this is one of the idea really turns out right and it's clearly that this came from this was three steps along the way where you know initial ideas and struggles working through then the different things and then winding up here with something where now you're running around and even if you've got Auto jump on you bump into the next block and if slide you straight up like it's on a ramp you go over there and you slide down and this has been spectacular so I'm still arguing about whether I need to keep thee I keep the Lagon linearize and all these there's still all these other things there's the slow starts and the friction stops this kills dead the problem of single block step ups and steps down and it works really great but there are still enough other things where the lag and linearize may still wind up having some I you know some utility for that as well like there's another I'm another aspect of comfort with especially with gear VR in navigation games the problem is that we only have a sense that where your head mount is the tracker that we've got is on the headset so we know where you're looking we have to assume that you want to move in the direction that you're looking so I you know when you're going forward you're going in the direction of the head now now that causes a snake walking where if you're looking around while you're walking there instead of walking straight looking around kind of like you want on a linear path that would be comfortable you're you're weaving as you go through it and that's not comfortable and the more you look around with your head the more that happen so Laggan linearize helps that a little bit where instead of instead of taking a you know a path like this you wind up taking here you're more of a linearized path i but it's still probably not really what you want so the other thing that's being I at least experimented with as a direction there is that maybe we need to have some offsets between allowing you to look around for some amount of time before changing your actual direction of body movement now what I'd really like to have argue for this a regular basis if we had a controller that also had the IMU inside it so that we could at least track up yalll between it then you could clearly say well the controller determines what direction you're going when you turn in your swivel chair the controller changes your body direction your turn direction and you could turn your head and if completely independently from that I am you know maybe we get that at some point I don't think it's in that the cards right now so hacking something around for like anti-snake walking maybe as an aspect of lag and linearized maybe something different I but it was still this was and this is all again mostly stuff that's happened in the last month or two different ideas that come on they track kind of continuously about all the different things that are going there's stuff that I'd love to get back to with some of the vision research work for inside out tracking where there's that hazard again there where if I get something that I think is a good idea and I don't get to beat on it soon enough it starts perhaps solidifying into a pet idea in some way where I maybe won't be as harsh on it when I do finally get around to working on it so I worry about that I just wish again that there was more time to be doing all the different things that you know that I want to be investigating and you know then there's even things that when I go even further afield and start thinking about you know Fusion Energi or other things like that then I get ideas for that that I'd like to be trying but probably are not going to be again happening in the real near future but the big takeaway that I want to leave about this is by looking at the ideas they're about taking the joy in the in the having of the idea and cultivating that and making that one of the things that you really look for but challenging them and not feeling it all bad when they get knocked down because usually the process winds up leading to at least a better insight about things or a better way of looking at things or techniques that lead to better techniques that solve the problems even better so I we have any time for any questions or anything this [Applause]

Channel: John Carmack

Views: 63,795

Rating: 4.9643283 out of 5

Keywords:

Id: dSCBCk4xVa0

Length: 62min 23sec (3743 seconds)

Published: Mon Feb 24 2020

Reddit Comments

[u/qznc_bot2]

There is a discussion on Hacker News, but feel free to comment here as well.