The Largest Unsolved Problem in VR.

in 1995 K Reef starred in a dystopian science fiction film all about virtual reality and you may be thinking the Matrix but that came later the first VR film for Reeves was actually Johnny pneumonic a film that ended up being a box office flop and is subjectively pretty terrible but there's one scene that I have always absolutely loved in order to hack into a database Johnny Dawns a full VR rig to navigate through cyberspace breaking through security and even threatening to crash another user crash from here man and I love this scene so much because it's one of the first ever 3D representations of what a virtual reality user interface could possibly look like gloves a headset gestures typing this film shows just how Wild the possibilities are and now 30 years later almost none of this is science fiction we have the internet gloves hand tracking VR headsets but the user interfaces of these devices still look like this and to this day these interfaces are universally the most criticized elements of modern virtual reality and so I decided to embark on a journey to explore the wild world of VR interfaces why are they designed the way they are why do they not look like this why are they so frustrating and what can we do better but in order to talk about VR design this deeply you need to see the world from the perspective of a designer and so that's where we're going to start with the philosophical foundations of design you don't need a degree to understand design you just need that perspective shift and a few key terms part one foundations of design 50 years ago Don Norman released two books that have practically become known as the Bible of design the Design of Everyday Things and emotional design and while they talk of many things the core messages can be shrunk down into a few main principles with a Golden Rule and the first of those Concepts is probably the most popular one good design is invisible try and think of the last time you had a great user interface or design experience now try and think of all the times you've had a bad experience it's much easier to point out when something feels wrong or bad than it is to pinpoint why something feels good and I think we're all familiar with the bad examples from accidentally suggestive signs to washing machines needing a key chart to operate the worst one for me are doors I am notoriously terrible with doors of all types and I can list bad examples all day long but I know I can't do the same with good examples and that's the Bittersweet nature of good design it's invisible the next set of Concepts may sound a little more complicated but it's actually really simple when you break it down there are basically three all-encompassing principles that design abides by it's a way to deconstruct why things don't work and to construct things that do first up are affordances and affordance is the sum of all possible interactions that a human can have with an object the affordance of a door for example is the mechanical ability to swing 180° a door is AFF fored opening and closing but how do you know which way the door swings is it a push or pull or sliding door and this is where the second principle comes in signifiers a signifier communicates to people how they can interact with the affordances a flat panel on a door signifies it's a push door and in VR for example an object that is interactable May glow or highlight when you hover over it that object's affordance is that it can be moved and interacted with the sign fire is the glow letting you know and the final of these principles is feedback the return of information to the user a haptic buzz when you select something a button's color changing the click of a key press on a keyboard and when all of these things come together you're able to systematically figure out how a person can interact with an object when where and in which way that interaction can happen and of course a way to let people know that this interaction did happen and I know this may all seem extremely obvious but there's a reason why these basic principles are taught the way they are now when things work or don't work as intended it's easier to single out exactly why like a coffee mug with a hole too small to put your fingers into or a game asking you to press a lettered button that you can't see both are examples of bad or missing affordances and this practice of applying these Concepts not only leads to better designs but can also literally save lives and prevent catastrophe it was the first step in a nuclear nightmare in 1979 the 3M Island nuclear plant shut down after a partial meltdown after the investigation it was discovered that a single bad signifier almost led to complete nuclear disaster on one of the control panels a light was used to show whether a valve was open or closed if the valve is open the light is off and once a button is pressed the light goes on and the valve closes the problem occurred when an operator pressed the button to close the valve as they do every day the light switched on as it should but the valve itself was actually stuck open the light was signifying not whether the valve was actually open or closed like you'd expect but instead the light reacted only to the button press itself having no connection to the valve that the button controls disaster could have been averted by simply signifying the right affordances and giving the right feedback part to the design of today's things I find it pretty spectacular that you could travel back in time 17 years buy an iPhone and you'll be greeted with almost the exact same interface you are today and if you were to travel 40 years in time you could grab a Macintosh off a store shelf and be met with the same exact affordances signifiers and feedback that we have today files folders windows and a desktop these are all skoric symbols which is another very important term for VR scoris is the practice of using digital design elements that mimic a physical object the computer isn't actually full of folders and your smartphone doesn't need to make a shutter noise but these are all good design practices that take something complex or unfamiliar and uses symbols and signifiers to break down their affordances into something recognizable and intuitive they're all essentially just analogies connections that enable your brain to build what's called a conceptual model which I think is one of the coolest things that the human brain does automatically you see a thing and without any direction you can intuitively make a model in your mind of how something works or at least should work as a simple example you don't need an instruction manual to operate a pair of scissors there's only one place for your fingers and it's not the pointy end and it's designed to do one thing a pair of scissors is a tool that cuts but what is a pair of scissors in a piece of software there are no handles or cutting edges but it's still a tool that cuts and as human culture evolves we design ever more complex things to be intuitive by relating them to things of the past shutter sounds desktops documents folders trash bins scissors and when you look at it the last 50 years of design and user interfaces are really built upon centuries of physical things that we can all relate to without thinking and that leads me to a really big question why have these interfaces stuck around for half a century why are we held back by physical constructs in the digital world why don't we have Johnny demonic style cyberspace exploration is it a lack of innovation is it just because this is what everyone every one's used to and no one dare change the formula or is it the inverse perhaps these Technologies smartphones home PCS the internet have only proliferated as far into the human condition as they have because of the interfaces because of good design I mean computers existed before the smartphones and feature phones and pdas existed before the iPhone but the explosion point the moment where everything accelerated smartphone Wars cheaper devices and worldwide adoption didn't happen until a design good enough allowed humans to interface with technology in a way that makes sense and I'd argue that the answer to that big question is 100% because it's just what everyone's used to but it's also 100% because of incredible design that we have ever been able to become used to it at all part three pu art design Paradox there have certainly been some interesting shakeups and ideas thrown out recently especially with artificial intelligence and things like the Humane AI pin but they're as far away from being viable today as VR was in 2012 the technology is interesting but the human interface has a long way to go a story that may sound very familiar to VR itself virtual reality is an inherently interesting technology that I don't think we figured out how to interface with yet and Johnny neonic is such an interesting case point because it displays the concept that anything is possible in this medium you can do anything from floating panels to skoric spacial ual trash bins to delete files to navigating the internet like you're flying through resz but that's almost overwhelming and I think that's vr's first big design problem VR itself is difficult to describe it's hard to put it into words and we've all heard the terms before there's unlimited possibilities you can be anywhere anyone do anything infinite creativity but Infinity is an impossible concept to grasp we are incapable of it as humans and so instead we resort to the hypothetical experience with the cliche imagine you're on a beach watching the sunset and within a blink you're on the edge of a black hole in your Virtual Office but you're actually there that's what VR is but then you actually get a headset and get into VR and you quickly realize that it's nothing like that the promises of VR are so tall essentially godhood of your own personal reality but that's not the reality and the marketing doesn't reflect that and if that's what you were expecting you better have a really great imagination or you're sorely and deeply disappointed maybe even lied to and so instead of thinking about VR in the impossibility of infinite places let's think for a moment about virtual reality strictly from a design perspective in the physical world design is extremely limited to Mechanical affordances a door can only open in the way a door was mechanically designed to do so but the physical world is also heavily contextualized through signifiers and detail a sign can be read at a glance details can be immediately interpreted because we have a very solid conceptual model of reality and physics we know from a very young age how the world works even if we don't know the math behind it which is also why computers and smartphones base everything on pedomorphic symbols and analogies we don't think like a computer we think like humans and humans live in reality alongside these devices and those devices have their own physical constraints an iPhone is AFF fored touch on the entire screen there is no difference in functionality between these pixels and those pixels virtual reality on the other hand is legitimately fundamentally different from everything else there is no mechanical constraint at all there are no defined laws of physics affordance is an unlimited resource anything anywhere can be used for any purpose but the use of signifiers are deeply impacted signs are hard to read detail is limited to rendering power and the big drawback is that the conceptual models we built from existing in physical reality and interacting with modern technology are prone to failure from a philosophical point of view everything within virtual reality is just a skor ISM of a different concept A Door In VR is not a physical door obviously it's a symbol of a passageway between one place to another a button is not a button and a pair of scissors is not a pair of scissors but instead it's a tool that cuts VR design at its core is the art of connection and analogy using symbols and representation to do things that are not physically possible and you may think this is just game design how is VR design any different well even in the most immersive of flat firstperson game design you are always just an outside observer in virtual reality you're instead an inhabitant which is its own double-edged sword I think one of the most frustrating and confusing Parts is when a skoric object that we have a conceptual model for in the physical world doesn't line up with the virtual one and we're left with the same frustrating problems that we deal with in the physical world but now in the virtual I can't open the damn door and I think the core real problem here is that modern VR interfaces make people feel stupid it's frustrating and even after tens of thousands of hours in VR I have to constantly ask myself is this bad or am I just bad but this Paradox of skill issue places us on the singular Golden Rule of design the rule that trumps all others you can throw out affordances signifiers feedback conceptual models they're all just abstractions to help guide a design to follow the Golden Rule and it's that it is never the human's fault we automatically blame ourselves for bad design it's our nature it has to be us and while feeling stupid we form a negative relationship with that design we intuitively resist it because it makes us feel bad and the Golden Rule exists to remind designers that the purpose of design is to build things that communicate with humans good design is good communication and it makes me think of all the people that have walked away from a VR experience definitely feeling a variety of emotions some of it awe and wonder but I guarantee that every single one of those experiences is also filled with quite a few negative emotions and as we talked about earlier the negatives weigh twice as much as the positive so what can we do about all this part four the perfect ER operating system if there's one lesson Don Norman and my mother instilled in me it's to never complain about something unless I have an alternative and in this case I do I figured the best way to put all of this into practice is to conceptualize my dream interface one that takes what we have today shaves off as many frustrating pain points as possible and allows people to enjoy VR because they're not thinking about VR they're just doing it I have five core VR operating system philosophies or principles that I think every OS should go by and the first is a virtual operating system should respect the user's space both virtual and physical it should serve as an anchor or a bridge between worlds but never is something that gets in the way or takes attention without permission the base core functions of any operating system should be a wristwatch or a hand menu that is always easily glanceable with basic information but is never obtrusive unless there's intent I can check the time reply to a message change quick settings all without leaving the application that I'm in and I actually find it especially disrespectful how meta treats panels in VR they stick to your face obscure Real World objects I mean I can't even see the table I'm marking in mixed reality because a panel follows me like a stalker everywhere I go it's a nightmare to imagine ad popups in VR or panels that you can't get away from so rule one respect people's space two all core OS and menu level systems should be designed for hand tracking first and controllers second this may sound controversial but hand tracking when implemented well can be significantly simpler than controller navigation the first instinct when designing a UI for a controller is to hide functions behind buttons and in VR you can't even see those buttons with a hands first design philosophy you don't have the option to hide functions behind buttons and on top of that it is way easier to add controller support to a hand Focus design than it is to ever add a hand tracking support with gestures mapped to 12 different buttons and if there's one thing that the Vision Pro taught me it's that I do not want to use controllers unless I have to in those cases such as games we'll grab the controllers and get the slicing but for every moment up until that I don't want to have to hold things in my hands three standardization the first 15 to 20 minutes of every single VR game is the exact same you hop into the game and are immediately bombarded with the 20 questions of VR smooth turn or snap turn teleport or smooth Locomotion viget or no viget now almost every VR user I've ever met already knows their preferences and this is generally the same among most VR games so why isn't this figured out when you first set up your headset in the initial setup and then just set as a global standard for every game imposing this standard on developers not only increases people's Comfort within VR but it also makes those crucial first 30 minutes of an experience so much better the application can hook you rather than asking you a bunch of questions but there's more on standardization I was pretty blown away when I first started developing on the Vision Pro Apple has built an entire standard look and feel for every user interface for every app using Swift UI now the developers can go around that if they want to but user feedback shows that this standardized UI implementation from operating system to application allows people to spend more time actually using the applications rather than menu diving and being frustrated and I strongly believe that meta or valve or whichever big company needs to introduce an OS themed style guide or user interface template just like how every other platform does do out there four predictability of confirmation make navigation easy satisfying and predictable right now almost every VR application uses raycasts or laser pointers from the controller but they have some pretty massive flaws depth perception conflicts shaky hands weird controller tracking glitches that result in misclicks and I think the most interesting is What's called the Heisenberg effect of spatial interaction it's a phenomenon where you're pointing at something with the controller and in order to interact with that thing you pull trigger this causes a natural response where the wrist slightly pitches vertically so you end up not actually hitting your target now this is correctable after the fact but it's rarely done by developers and when it is done everyone has their own way of doing it meaning every application's laser just feels a whole lot different and this could fall under standardization but I believe the problem is not with standard lasers it's with predictability even in meta's core operating system if I hover over a selection with hand tracking and pinch the Target location is moved because of the gesture which is really frustrating and it's very unpredictable now apple on the other hand with the Vision Pro uses an entirely different method for selection and Confirmation instead of a laser shooting from your controller the laser shoots from your eyes you look at something and then pinch to interact with it but that's not perfect either in fact the longer I use Vision Pro the more frustrating it is when the eye tracking doesn't work perfectly for example a menu with two options save this video recording or delete it there's an internal Panic unlike anything I've ever experienced before don't look at it don't look at it don't look at it for tens of thousands of years humans have used their eyes for information gathering not for decision making and I think we need a more predictable way of confirmation whether that's reserving decision-making to a separate swipe gesture standardizing raycast lurp and Heisenberg confirmation or introducing an entirely new control method like head gaze and gesture interaction taking the same core idea of Apple's eye tracking system but instead of using your eyes for selection you use an invisible laser shot from your headset and I think with the correct signifiers and backend systems you'd be able to interact with most things in VR without the downsides of the alternative and five defining a VR conceptual model right now I can't really tell you how the quest Os or steam VR flows it's just a collection of menus that I navigate through because kind of have to I have no model for how these things are connected and when I do at some point form some model it ends up breaking because of an update that just rearranges the same thing and this is the most exciting part of this entire guideline say what you want about the Apple Vision Pro it's cool in some ways it's very flawed in others but there's one single thing that I think it does perfectly let me break it down I want you to think of a desktop computer not today's computer but instead one from 1977 7 with no graphical user interface you have a single application with code exposed and the interaction method is quite literally a command line think of this as the first dimension of human computer interaction a line now extend that line to the second dimension add a graphical user interface a place to put multiple applications side by side all fixed on a 2d plane it's what we have today this is your desktop which holds all of your open applications of which there are a few main types which widgets windows and some applications that full screen and take your entire desktop this is a game or an immersive app this is the second dimension of the computer human interface and this is exactly what the Vision Pro Nails Apple has what they call a shared space and within that shared space there are distinctly three types of applications flat 2D panels that exist in 3D volumes which are 3D applications that just take up a certain userdefined volume and of course you have your full screen games your immersive applications that take over your entire shared space and when you break it down you realize that the shared space is just a desktop pulled out to the third dimension and it functions exactly that way it's not a loose analogy it's the same exact thing and I think this is the biggest breakthrough that VR software has ever had a strong conceptual model that is familiar in all the ways that we already interact with technology but it amplifies them and if you've ever operated a desktop computer you'll intuitively know almost exactly how this works and again I don't think it's the eye tracking and I don't think it's the lack of controllers I think it's an intuitive clear model that you don't need to think about because you've already been using it for your entire life part five simul insir ratio [Applause] in so it may seem a little ridiculous and perhaps unnecessary to go into the detail I did in this video but I felt absolutely compelled to share some of the concepts that I learned while on this journey and it's the kind of video that I wished existed but I also went this deep because the world of design is legitimately so fascinating and I think it's a really powerful feeling to leave the role of the everyone else and to see everything from just a slightly different perspective and while terms like signifiers or affordances seem like such a mechanical way to think about the world the funny part is that all of these design terms are really just philosophical Concepts that try to explain and guide people to make things that people love and as we build more and more complex things we're adding little tiny bits to Humanity's Collective intuition and Beyond the technical jumbo I found designed to be a beautiful poem of analogies that are rooted deeply in our physical world and these analogies allow us to navigate anything and anywhere that is as long as the design is good enough to guide our intuition and speaking of intuition throughout all of human history intuition has actually been this mystical thing it's like magic and I'm not surprised it's immediate knowledge to know without knowing and in Plato's Republic to be touched by intuition is to be touched by the soul of Eternity to feel intuition about something is almost like filling a cup from The Well of all of Mankind's knowledge and to drink from it and if you take a second to think about it how do you really know something that you don't know there are a lot of factors to it environment designs books but there's something else out there our gut and that's what I find interesting about design and that's what I find especially interesting about design in virtual reality it's a kick-ass technology that we legitimately haven't figured out yet no matter what anybody says no matter the confidence in my words nobody really knows anything and the things we build today to please our intuition are the analogies and schor isms that will be used tomorrow and it's not the alien science fiction of Johnny neonic and it's not the alien nature of today's interfaces it's instead going to be built off of what Humanity already knows and that is exactly why I find VR so damn cool but we can do better don't be afraid to try new things but definitely also refer to those things that have been done that are good it's it is the wild west but we are Paving roads and putting down foundations for the future [Music]

Channel: ThrillSeeker

Views: 477,407

Keywords: VR, Virtual Reality, AR, Apple Vision Pro, Quest 2, Quest 3, VR headset, Best VR, VR games, VR design, Paradox, Thrillseeker, Tuesday Newsday, VR news, Oculus Quest, Meta Quest, Valve index, Philosophy, Vision Pro, VRChat, Game Design, Norman Doors, Best VR headset, VR UI, UI UX, User interface, iOS 18, Apple, iPhone

Id: Fhlw88_Beu4

Length: 25min 43sec (1543 seconds)

Published: Sun Jun 02 2024