Virtual Reality Engineer Explains One Concept in 5 Levels of Difficulty | WIRED

hi I'm John Carmack I'm the chief technology officer for oculus I work on virtual reality I've been challenged today to talk about one concept at five levels of increasing complexity so we're gonna be talking about reality and virtual reality what the technology allows us to do today what it may allow us to do in the future and whether that should even be our goal to approximate reality so do you know what virtual reality is yes it's simple it's vert it's like a video game except it feels like you're into video games that's actually a really good description the idea is that if you've got a system here that can make you see whatever we want you to see then we can make you believe that you're anywhere else like on top of a mountain or in a dungeon or under the ocean or in Minecraft yeah or in Minecraft when you look at an TV on the wall there he's showing like a picture of a mountain or something how can you tell that it's not just a window and there's something else behind it because it always doesn't look quite right if you have a static picture of a person on a screen and you move around like this it's not really changing and it's interesting those are things that we have to figure out the ways around it to fool you in virtual reality we need to figure out when you look at something in reality how can you tell whether it's real or not have you been to a 3d movie where you put on the little glasses and um yeah so what they do the trick for that is if you ever add a theater and you take up the glasses and you look at it you'll see it's blurry where there's actually two pictures that they're showing at the same time and what those little glasses do is they like one I see one picture and the other I see a different picture so then your eyes can say oh it looks like I'm seeing right through the screen or something as floating out in front of it in the VR headsets what we do is there's actually either two screens or one screen split in half so that it draws a different picture a completely different picture for each eye and we make sure that each eye can only see the picture it intended to and that's what can make things feel like they've got this real depth to them that it's something that you could reach out and touch and it doesn't feel like a flat TV screen [Music] try and look I look over there and concentrate on his face over there can you see me waving my hand without turning your eyes no all right so at some point you can probably see it right now all right so without moving your eyes this is kind of hard tell me how many fingers I'm holding up there's barely tell hmm and you see that even in like if you hold out your hand and you focus on your hand then your foot would be blurry because there's that difference there and you can change between that like you can then focus on your foot and your hand gets blurry because your eyes can see different amounts of detail in different places so we're hoping in the future that hardware can be like that where we can make a display that puts lots of detail right where you're looking and every time you look someplace else it moves the detail over to there so we don't need a render a hundred times as much as we've got right now figuring out where you're looking is a pretty hard problem what we try to go about this is by taking a camera and looking at people's eyes and then try to figure out is the eye looking over here or up here and we're working hard on stuff like this right now so we hopefully can have a virtual reality that's as detailed and realistic feeling as the reality that we've actually got around us but it's gonna be a long time before we get to I you know where we can really fool people so do you have a basic sense of what latency is yeah so my understanding of what latency is is it's it's basically the time delay between the rendering at different points so it's basically a delay and it happens in all parts of the system monitors can be a big one like you know consumer televisions can often have 50 milliseconds or more of latency just in the TV part and then you've got the processing in the computer and all of these add up to the total latency the latency is in my opinion the most important part of the AR because if you have that offset your body is no longer immersed and you get you gain that motion sickness which can pull a lot of people out of the experience games that feel really good they've got that sense of it happens instantly when you're you know when you're doing something it really is a testament to this kind of technology and the how it's developing and how over time it's gonna just be you're gonna be able to pack more pixel density in this displays and it's gonna get a lot more immersive like 30 years ago you had desktop PCs which were you spent whatever on that but there was always this idea well you could spend a million dollars and buy a supercomputer and it's gonna be a lot faster and that's not really true today for scalar processing when you just do one thing after another okay a high-end overclocked cooled gaming PC is about the fastest thing in the world it is within a very small Delta yeah for some things you'll get some power from you IBM power system that might be a little bit faster but not a whole lot so if I'm looking at this and saying we need to be five times faster you know what do you do you can't just say make each thing faster you're bait as a developer you're making a trade-off I can put my effort into making this more efficient or making it more fun and more fun usually wins out for very good reasons so there's some good judgment and trickery that goes into the design of things you can always design a game that will just not work well I am I mean in the old days games had to be so precisely designed and nowadays you've got a ton more freedom but you really can't just the hardware old games are running off and they fit and that's all the days you could have yeah any crazy idea you know you could probably make a pretty good video game which is a wonderful wonderful thing yeah a lot of freedom but VR makes you have to give up a little bit of that freedom you don't know not do so many crazy things so that you could wind up having it be as responsive and high-quality as it needs to be mm-hmm that's very interesting you know an interesting topic is what are the limits to what we can do with virtual reality where I'm pretty pleased with what we have today what we can show people and say virtual reality it's cool people get an amazing response from it but we're still clearly a very very long ways from reality that kind of notes back to realism in our history and how realism was response to romanticism and realism was meant to capture the mundane everyday lives of you know individuals and not idealize any of their activities or any way and I think that that's really important for virtual Rhianna I think it's kind of like a rite of passage for any kind of our technology to go through mostly in VR we talk about the display and optics the visual side of things but we should at least take off the other senses and haptics is an interesting thing about virtual reality really doesn't have that aspect of touching things you can move your hands around you could do everything but it's a disconnected experience because you know you don't have you know the actual solidity there and I am pessimistic about progress and haptics technology about almost all other areas I'm an optimist I'm excited about what's coming up but I don't have any brilliant vision about how we're going to revolutionize haptics they make it feel like we're touching the things in the virtual world so I've you know tried the demos where there's like VRLA there's one that has waves like audio waves I believe that come up and then you can put your hands through that and feel the way is whenever you're supposed to be feeling bubbles or any kind of force note or something those are pretty interesting I've seen some pretty interesting things that you could do with audios you can cut down a lot of the storage I guess in the power that you would need in order to power huge scene you can just you know mimic the sounds of those scenes actually being there and then not actually building out for example the professor at USC would have the sound of a train drive by without a very actually rendering the sound and you'd feel like you're deeply immersed in this world without having to have such an expensive scene built around you so I think those are pretty you know significant and that is one potential quality improvement that's still on the horizon is when we do spatialization we use the HRT of the head relative transfer function to make it sound like it's in different places but usually we just use this one kind of generic here is your average human HRT F function and it's possible that of course if you are right in the average then it's perfect for you but there's always people off to the extremes that it does do a very good job at and there may be better ways to allow people to sample their own perfect H RTF which can improve the audio experiences a lot it all comes down to all these trade-offs you know with display and with resolution it's one of those things where if people have one bad experience they kind of out rule everything else is really difficult to build trust again with people who haven't done beer before but it's easy to break all that trust whether they do it was a huge concern about that at oculus and the term internally that went around was poisoning the well they were very very concerned I mean for a long time there was a fight about whether gear VR should even be done because the worry was that if we let a product go out like year VR that didn't have those things that if somebody saw it and he was bad it made them sick I made their eyes hurt then they'd be like I'm never gonna try VR again I tried at that time and it was you know it was terrible and there was legitimate arguments about whether it was even a good idea to do that and it turned out that yes it's obviously better to have all of those things but you can still do something that's valuable for the user without it it's weird being at the beginning of a medium like this right I'm very excited to see how filmmakers tackle kind of creating content in those things especially if there are experiences of traditional medium mostly today I've been talking a lot about what can we do what's possible we think might be possible in the next couple years but really at the professional level it's more the question of wisdom of what should we be doing I mean that's what I think we're trying to figure out is from artists and storage or storytelling perspective kind of what are the things that will make this meaningfully different from what we're used to like a television or a wall and we've been finding a lot of things that aspects of virtual reality that very much do that in my opinion things that allow you to feel presence first and foremost where you get lost you have to remind yourself this isn't actually happening and things that ultimately allow you to embody other characters things that you can actually change your own self-perception and play with neuroplasticity and teach yourself things that are bizarre and unique as an engineer of course I love quantifiable things I like saying here's my 18 millisecond motion to photon here's my angular resolution that I'm improving I'm doing the color space right but you can look at you know not too far back where you say we have DVDs at this amazing resolution but more people want to watch YouTube videos at really bad early internet video speeds where there are things that if you deliver a value to people then these objective quantities may not be the most important thing and well I'm certainly pushing as hard as we can on lots of these things that make the experience better in potentially every way or maybe just for videos or the different things I don't think that it's necessary I've commented that I think usually my favorite titles on mobile they're fully synthetic are ones that don't even try they just go and do like light mapped flat shaded and I think it's a lovely aesthetic I think that you don't wind up fighting all of the aliasing while you get some other titles and I we're gonna be high tech with our you know our specular bump maps with roughness and you've got aliasing everywhere and you can't hold framerate and it's all problematic while some of these that are clearly you know very synthetic worlds where I it's nothing but these cartoony flat shaded things with lighting but they look and they feel good and you can buy that you're in that place and we want to know what's around that monolith over there we did a project called life of us which is exactly that mindset where let's embrace low poly aesthetic and just simple vertex shading and we ended up you know realizing you can embody these two various creatures and transform yourself and when you do that with co-presence of another creature another human it makes for a totally magical journey and you don't even think for a second you actually dismiss the whole idea of photorealism and embrace that reality for it is I think it actually helps put you at ease a little bit the end goal of reality of course in computer graphics people have chased photorealistic form you know for a long time and basically we've we've achieved it photo realism if you're willing to throw enough you know discrete path trace trees and things you can generate photorealistic views you understand the light really well of course it still takes a half hour per frame like it always has or more to render the different things so it's an understood problem and given infinite computing power I'm we could be doing that in virtual reality however a point that I've made to two people in recent years is that we are running out of Moore's Law I am I mean maybe we'll see some wonderful breakthrough in you know quantum structures or whatever or bandwidth I can but yeah but if we just wind wind up following the path that we're on we're gonna get double and quadruple but we're not gonna get 50 times more powerful than we are right now we will run into atomic limits on our fabrication so given that I'm trying to tell people that start buying back into optimization start buying back into thinking a little bit more creatively because you can't just wait it's not going to get to that point where it really is fixed to to those highest degrees just by waiting for computing to advance if we want this to be something used by a billion people then we need it to be lighter cheaper more comfortable and there's constantly novel iux innovations like Google Earth whether that is the kind of elimination of your periphery as you zoom in and move and also giving you the user action to decide where you're going and they're looking there's a constantly seeing people coming up with ways to kind of break from the paradigms of actual reality and then introduce very well you know there's tons of opportunities for the synthetic case where you want to be able to have your synthetic fantasy world where everybody is a creature that's created by the inviter and simulated reasonably yeah but of course we've got been you know we still don't do people well simulated that's a hard problem we've been beating our heads against it for a long time I do think we're making progress and I wouldn't bet on that being solved in 10 years but maybe 20 years because it is gonna take a lot of AI it's gonna take a lot of machine learning where it's not gonna be a matter of us dissecting all the micro expressions that people do it's gonna be let's take every YouTube video ever made and run it through some enormous learner that's going to figure out how to make people look realistic absolutely there are these crucial thresholds where you pass you know technological hurdle and all of a sudden that unlocks a whole world of creative potential but I think to your point very much we need to solve the actual human and social challenges and and turn those into opportunities to figure out how this technology fits into our lives I'm still believe that the magics out there and you haven't found it yet so somebody is going to you know happen upon the formula I feel like I've felt the little pockets of math there's that oh you can imagine the utility by creating a world or you can imagine the power of the story that would be told you in this context and a lot of it I think is just picking those putting them together in a meaningful way and then crafting something that's that's really bigger than intentional but I've now been joining with real people in virtual reality and you I think we also have different levels of connection like the audio side is the leaps and bounds above in terms of the nuance of personality in humanity so yeah when I hear people laughing and joking and and you know really enjoying something soon we'll get into the word macro gesture land or I can wave and say you know but when we get into micro gestures and I'm actually getting a sense of facial reactions and other things I think then we'll have a really incredibly rewarding time spending spending time with each other so v4 right now is pretty amazing when you look at it it's things that you haven't seen but we are just getting started the next five years both technologically and creatively are going to really take this medium someplace that you've never imagined [Music]

Channel: WIRED

Views: 981,239

Rating: 4.893322 out of 5

Keywords: oculus, vr, john carmack, virtual reality, explaining vr, how does vr work, how does virtual reality work, realism, realism in vr, realism in virtual reality, 5 levels, 5 levels of difficulty, levels of difficulty, john carmack oculus, oculus vr, explaining, explanation, within, aaron koblin, five levels, vr gaming, virtual reality gaming, video game, vr discussion, oculus rift, virtual, engineer, one concept, explainer, vr explained, wired

Id: akveRNY6Ulw

Length: 15min 52sec (952 seconds)

Published: Mon Oct 16 2017

Reddit Comments

[u/[deleted]]

well, that kid totally lost him after about 30seconds ... That's just Carmack being Carmack, no 2ways about it

[u/knoodrake]

To a 12yo girl, what comes to mind as first thing to explain about VR ? Foveated rendering of course !

[u/Onikaze]

I would like to see the longer uncut conversations. do they ever post those?

[u/Saerain]

It was funny how the generic college student had a more relevant grasp than the grad student of "interactive media and games."

[u/vehementi]

With the older students the video editor did an excellent job of conveying just how much Carmack was probably thinking "shut the fuck up" when the shot cut abruptly cuts away from the students trying to sound intelligent and impress him

[u/Le_9k_Redditor]

No way that kid is 5

[u/ParadiseDecay]

John if your reading this we still have your imp! ;)

[u/my_name_is_memorable]

Here’s how I would have handled it.

Kid: “it’s like a video game, but it Feels like you’re IN the video game!”

Me: “exactly! So why does it feel like being in the video game? Why is it better than a TV?”

Kid: “uhhh....”

Me: “well in VR, you can walk around things and see all sides of them. What happens if you walk around your TV. Do you see behind the person in the movie? Of course not. You see the back of your TV”

Kid: “of course!”

Me: “now, there’s more stuff we do to make you feel ‘inside the game’ “

... and I’d take it from there

[u/mykebm]

Just about to post this, great video