Best practices to design AR applications (Google I/O '18)

Video Statistics and Information

Video
Captions Word Cloud
Reddit Comments
Captions
[MUSIC PLAYING] ALESHA UNPINGCO: We're here to talk to you today about designing AR applications. So Google has a long history in designing for AR. We've been doing mobile AR for the past four years and working on other augmented reality projects before that. What's changed most recently is that mobile AR is really starting to take off, and Google's AR core makes it possible for anybody to create quality AR content. So I'll some fun facts about AR core. The first thing is that we released 1.0 at the end of February, and that made AR content available to more than 100 million devices. And we're already seeing rapid growth with more than 300 apps already available in the Google Play Store. So you're here because you want to learn how to design for AR. And what we've found is that once you understand your users and the type of experience you're trying to create, design principles for getting started fall into five different categories, which we call the pillars of AR design. First, you want to understand your user's environment. Where will the users be experiencing your app? Think about the surfaces that are available and how your app can adapt to different environmental constraints. Then you want to consider the user's movement and how much space the user will need in order to experience your app. And when it comes to initialization and onboarding, you want to make the onboarding process as clear as possible so that users understand exactly what to do in this entirely new medium. When it comes to object interactions, design natural object interactions that convey the affordances and also the feedback so that users understand how these digital objects fit in the context of your real physical space. And when you're thinking about user interfaces, balance the on-screen UI with volumetric interface design so that you're able to create an experience that is meaningful and usable for your users. So we have some examples that showcase the different guidelines within each of these pillars. And the thing that we want to point out is that this framework can help anybody get started with AR content creation. So throughout our talk, we're going to show you some different demos that you'll be able to play with very soon through an app we're launching on the Google Play Store called AR Core Elements. And many of the core interaction patterns you'll see in this talk are already available in [INAUDIBLE] form or will be available for Unity later this summer. ALEX FAABORG: All right. So let's start out talking about the user's environment-- the first pillar of AR design. So ARCore is a relatively new technology. So just to begin, let's talk about what ARCore can actually do. So it does lots of things. The first thing that everyone's familiar with is surface plane detection, where it can understand surfaces, tables, floors, those types of things. It can also do walls. Vertical surfaces. And then, of course, what's better than horizontal and vertical surfaces? It can also do angled surfaces with oriented points. Being able to place objects on any angle. ARCore does light estimation. This is really important for having objects sort of look realistic in a scene. And there's also some other fun things you can do with that that we'll get into. And announced at I/O yesterday, Cloud Anchors, which is available now for ARCore both on Android and also on iOS. It lets you do multi-player experiences in ARCore with two people viewing the same thing. And also announced yesterday, augmented images. The ability to recognize an image, but then also not just recognize it, but use that image to get 3D post data off of it so you know where it is in space. So that's the current set of ARCore capabilities, and this is, of course, growing over time. And what we found is the more your app integrates with the user's environment, really, the more magical your app is going to feel. So let's look through a few examples here. First, surface plane detection. A lot of AR apps currently only use one surface. Say you're playing a game-- a game board will appear on a surface. What we found is there's really no reason that you have to use one surface. You could use all of the detected surfaces. And at moments when you have your game interacting with multiple surfaces, those can be really kind of like breakout moments in your game where it feels very magical. Even something like if, say, you're playing a physics based game and you destroy your opponents castle or something and the bricks fall onto the floor. At that moment when you see the items on the floor, it can be really quite stunning. All right. Light estimation. This is critical for making objects look realistic in the scene. Here's an example that we're working on to sort of test out some different techniques. Here we have three fake plants and one real plant, although I think it's actually a real fake plant. And we have unlit, which is the most basic lighting you can do. Really not very realistic. Then dynamic and a combination of dynamic and baked. And what's great with dynamic is you get real time shadows. And then when you combine dynamic with baked, you start to see things like soft shadowing where the leaves of the plant are actually a little bit darker and it's picking that up. You can sort of see an example of how this looks with some movement. And lighting, there's definitely a lot of innovation that's going to be occurring in this space as we try to get more and more realistic lighting. You can see where we are right now. And especially as the scene gets darker, you start to see that the unlit object just doesn't perform as well. So it's really important that you're using the real-time lighting APIs that are in ARCore. The other thing you can do is you can actually use lighting as a trigger to change something. So here in this example when you turn the light switch off in the room, the city actually glows and it responds to that change. And these types of moments can really feel great in magical freezers where-- imagine you're playing a sort of a city simulation game and it's having these kind of significant, meaningful changes based off the environmental light. All right. Oriented points. This is actually a very new feature. So we don't have a whole lot of examples here, but here is one of the more basic ones. I filmed this when I was out skiing. And here I'm just attaching androids to the side of a tree. And you can see that as I place them, they stick to exactly that point on the tree, at the angle of where the branches were. All right. Cloud anchors. Announced yesterday. Here's an example of that being used. Here both players see the same game board in exactly the same place, and they can play that game together. And this is really tremendously fun. You can actually try it out in the Sandbox if you want to stop by later today. And again, this works with both Android and iOS. Augmented images. There's a lot of different ways you can use this. One demo that we have in the Sandbox that you can go check out is actually an art exhibit that was built using augmented images. [MUSIC PLAYING] So we're really excited about what you can do with augmented images. And there's really lots of possibilities from artwork to even something like just having a toy sort of come to life. You know, the surface of a product box where you can see sort of 3D models of what you're about to play with. All right. So now that we've gone over some of the basics on the core capabilities of ARCore, let's talk about how you'd actually start to design an app for AR. So one of the first things you're thinking is, OK, where do I actually start? You know, blank page and ready to start having brainstorming and new ideas for AR. And one of the things that I'd want you to first focus on is AR exists outside of the phone, so your design work should really exist outside of the phone as well. So something I've found with a lot of people who've done tremendous amounts of mobile design is they tend to be very attached to sort of the phone frame and sort of flows of screens. They've been doing that for so long. One of the first things you need to do when you're starting to think about AR is to actually put away all of those, you know, 2D UI stencils. And don't really think about the phone at all. Instead, what you want to do is you want to sketch the actual environment that the user's in. So you should be sketching living rooms, and tables, and outdoor spaces. And then as you sketch the user's environment, then you start to sketch the various objects that they're going to be interacting with in that environment. In many ways, you can sort of think of AR as having a lot of the same challenges as responsive design for the web in terms of different window sizes, but it's even more complicated because now you have responsive design for 3D spaces that are the user's actual living room. So you want to sketch the user for scale to get a sense of how you're going to start crafting this experience. The user could be very large relative to the AR objects or very small. And then you want to start thinking about how that user is going to move around in that environment. ALESHA UNPINGCO: And that brings us to user movement. So now that we understand how to design for the environment, let's think about how to design for user movement. And as Alex mentioned, it's completely OK to design beyond the bounds of the screen. And what we've found is that in many ways, this can make the experience feel more delightful and even more immersive. Because when you have an object that begins on-screen and also extends beyond the boundaries of the phone [INAUDIBLE] port, it can make the user feel like the object is really there. And beyond that, it can also motivate the user to organically begin moving the phone around their environment so that they can appreciate the full scale of these digital objects in their physical space. And that brings us to our next observation. Because users are more familiar with 2D mobile applications that don't typically require user movement as a form of interaction, it can be very challenging to help convey to users that they're able to move around. So many users don't because it just doesn't feel natural based on how you use 2D apps in the past. But what we realized is that characters, animations, or objects that convey visual interest on-screen and then move off-screen can be a natural way to motivate users to move. So here we have a bird and it appears in the middle of the screen. And when it flies off-screen, it's replaced with a marker that moves around and slides along the edge to help users understand the bird's location in relation to the user. Another major thing that you want to think about is that whenever you have an experience that requires a user to move, you also want to think about how much space a user needs. So we found that experiences fall into three different sizes. There's table scale, there's room scale, and there's also world scale. And when it comes to table scale, what we found is that your experience is able to scale to the smallest of surfaces so that many users are able to enjoy your experience. And with room scale, it expands the impact of AR so that content will start to feel life sized and you're able to do a lot more with the space that's available. And world scale has no limits. It allows users to appreciate AR in whatever area they see fit. And this is an area we're particularly excited about because what it means for procedurally generated content in world scale. So no matter what size your experience ends up being, just remember to set the right expectation for users so they have an understanding of how much space they will need. Because it can be a very frustrating part of the experience if the user is playing a game and in the middle of the game they realize they don't have enough space to enjoy it. And when it comes to how much movement your experience requires, there's no one size fits all solution. It really depends on the experience that you're trying to create. For example, if you have a game that requires user movement as a core part of interaction, that can be a very delightful experience. You can use proximity or distance to trigger different actions so that as a user gets closer to this frog, it can leap behind the mushroom, or maybe the mushroom can disappear. And that can be really cool to see. However, if you have a utility app where the core purpose of the app is to help users understand very complex data and information, then requiring users to move might be a really bad experience. Because what it means is that users who have different movement or environment limitations won't be able to get the complete app experience. So allowing users to manipulate the object to rotate it, to move it around in a space that's more appropriate will ensure that all users have easy access to the data that they seek. ALEX FAABORG: Because AR is relatively new, the actual process for users to flow from 2D parts of your app into 3D can be, at times, a bit awkward. Amber is just starting to create some sort of standards around that. So we'll talk about initialising into AR. One of the first things you can do is you can leverage standard View in AR material icon so users, when they see that, they know that when they hit this icon they're going to be going into AR. You can use this in all the normal places that icons appear. Like floating action button or on top of cards as the indicator that you can actually view this object in 3D in your environment. One of the next things you'll see if you've been playing with lots of AR apps is something that you might not initially understand. I want to talk about the concept of how understanding depth actually requires some movement. So you'll see these types of animations where I was trying to get the user to move their phone around. So why is that actually happening? Basically, we perceive depth because we have two eyes, but we actually get a lot of our depth information by actually sort of moving our head around and being in the scene. And for the case of ARCore, most current phones on the market only have a single camera on the back, so the device only has one eye. And if it hasn't moved yet, it doesn't necessarily know what's going on. So this is the first thing the phone sees. It's going to say, all right, well, that's interesting, but I don't totally have a sense of where these objects are yet. And once you just move just a little bit, then it becomes clear. As soon as you bring in a little bit of movement, then you have enough of that data and different angles on the scene that it can start to build up a model of what it's seeing. So that's why we have these animations at the start of the app to try to get that movement, to try to get ARCore to have enough information to recognize the scene. Next thing you want to think about is deciding if users are able to easily move the objects after they've been placed or if these are really more permanent objects. And there's, again, no right answer here. So more persistent objects might be like a game board or something that itself takes input, but we want to recommend that you use standard icons to set expectations for users so they know as they're placing that object if that object is going to move around later on as they swipe on it. So some examples of that. Let's say you are placing, like, a city game. And here as you're swiping on the city, you're actually going to be interacting with the game itself. So we recommend using an anchor icon for these more persistent object placements. And you still want to enable the user to move the game board later, perhaps through a menu screen or some type of re-anchoring flow. So set expectations ahead of time that the city actually is going to be sort of stuck to the ground there for a while as you interact with the game. Versus something like, say, you're shopping for furniture and you're just placing a chair on the scene. Here, the chair itself has an interactive, so you can actually map swipe gestures onto the chair and just easily move it around. So using the plus icon to kind of set expectations ahead of time that you're not really committing to exactly where you're placing this object. All right. So now that we're talking about objects interactions, there's actually quite a bit of details there. ALESHA UNPINGCO: So now that we understand how to onboard users, let's start thinking about how users can interact with objects in their space. One of the things that we challenge you to think about as designers and developers in the community is thinking about how to problems solve for user behavior, even when it's unintentional. So one of the things we recommend is giving users feedback on object collisions. And this solves a huge problem that we see in mobile AR where a user will be moving the device around and once the device collides with an object in AR, that object might disappear and the user has no feedback in terms of how to fix it. So what we recommend is we recommend providing feedback in the form of camera filters or special effects that helps users understand when object collision is not an intended interaction. And this tends to work really well. The other thing that you want to think about is how to give users the right type of feedback on object placement. And it's really important in this case to think of each stage of the user journey, even as it relates to surface feedback. So surface feedback in AR is very important because it helps users understand how ARCore understands the environment. It gives the users a sense of the surfaces that are available, the range of the surfaces that are available. So we recommend including feedback on the surfaces when the user is placing objects in the scene. The other thing that we recommend is maintaining the height of the tallest surface as a user drags an object from one surface to another. And once an object is suspended in the air, make sure that you're always communicating visual feedback on the dropdown point. That's way it's very clear to the user at all times where the object is going to land. And once an object is placed into the scene, we also recommend providing feedback in the form of visual feedback on the surface, or even on the object itself, just to communicate the object's entry into the physical environment. So now that we know how to play with objects in your scene, let's think about how an object might get there. We recommend using gallery interfaces in order to communicate to users how they can take objects that live on-screen and drag it out into their real world. So here you see we have a gallery strip at the bottom bar. And as a user selects an object, they're able to drag it onto their space. And not only that, we're able to support both selection states and also very familiar gestures that allow users to manipulate the objects. So you can use pitch to scale, twist to rotate, and even drag to move. And you see many examples in our talk of how dragging objects is a very common and expected behavior. But another alternative for object selection and object movement is through a reticle. So reticle selection is also very effective in that it allows users to manipulate objects in their scene without covering too much of the user's view. So we have an example here where reticle selection is being used to select a rock. And that's triggered via the Action button in the bottom right. But what it allows users to do is that it allows users to see the many surfaces that are available. And as you can imagine, if a user is selecting an object with their finger and dragging it across the screen, you don't have as much screen real estate to see all of the surfaces that the user might want to place the object on. So reticle selection is very, very impactful here. The other thing that you get with reticle selection are raycasts. So raycasts are very effective in helping the user get a sense of the virtual weights applied to each of these objects. So here we have another example where the user is able to pick up a feather. And once the father is picked up, you'll notice that the raycast has very little movement and very little bends on it. And for the most part, it remains straight. However, when the user picks up the rock, you're able to see a more dramatic bend applied to the raycast. That signifies the larger amount of mass, the heavier weight of this object in relation to the feather. ALEX FAABORG: All right. So let's move on to the final pillar, which is volumetric interface design. I think one of the first things you want to consider here is that the phone is the user's viewport. They're actually using the phone to look out into the scene and see the application. And because of that, you don't actually want to place a lot of 2D UI on the screen. That's actually going to obscure the user's view onto your application. So showing a sort of quick example. It's obviously a lot nicer to have a limited set of controls. As soon as you start to clutter the screen, it really gets in the way of the user's ability to enjoy the AR app. And the sort of counter-intuitive thing that we've even found is that users are so focused on the app out in the world that often designers will place a control on a screen level because they want to draw attention to that control, but it's actually having the opposite effect. The users are actually more focused out in the scene, so they'll actually just miss controls that are drawn on the surface. Just kind of tune those out. So really, you want to be very mindful of when you're making decisions on if you're going to put a control up on the screen versus out into the scene itself. For not just obscuring the view, but also for discoverability of them finding that. And that's not to say that you should never put a control up onto the screen, but you want to be considering a few different metrics on it. So our recommendations is that you only really leverage on-screen surface UI for things like controls that have a very high frequency of use or controls that require very fast access. So like a camera shutter button is kind of the perfect example of something that hits both criteria, where in a camera you're obviously taking lots of pictures and also you want to take pictures very quickly. But imagine if you're playing a game and there is some ability to, like, fire or something. That would be a good candidate for an onscreen control because you're both hitting that button a lot and also you need to get to that button very quickly. So we talked about using the View in AR icon to get people into the experience and transition from 2D into AR, but also want to be very careful about the opposite. Of when users are now in AR and they're actually transitioning back to a 2D experience. And one thing we found is that if the user is not initiating that action to go back into a 2D experience, it can actually be pretty obnoxious because they're so focused out in the scene. So the user's viewing the application, and then suddenly a 2D UI shows up and blocks the entire viewport. That can be pretty annoying. So depending on even if the user is exiting, or they're customizing an item in the scene, or whatever the use case is, you want that flow back to 2D screen level UI that's covering most of the screen to be something that the user's actively doing, and not something that happens by surprise. So a common thing with mobile application design is you want to maintain touch targets that are about the size of the user's finger. For 3D, this is, of course, a bit harder because the object could be any distance away from the user. So quick example of some things you can do here. Here we have two tennis balls. And when you tap on the tennis ball, confetti fires out of it because in AR you can do whatever you want. And we're showing the touch target size with the dotted line. So one of these tennis balls is actually maintaining a reasonable touch target size as it gets farther away, whereas the other one is not. It's just mapping to the virtual size of the object. And, of course, it's a lot easier to interact with the one that is maintaining a large target size. We've also found for interfaces where you're manipulating objects, if you're not doing tricks to kind of maintain target size, you often get these problems where you swipe an object very far away, and then it's actually hard to bring the object back because it's now such a small target. So you have to actually walk over to the object to get it, which is a little bit frustrating. Maybe you could say it's very immersive, but either way, it's nicer to be able to actually bring the objects back as well. So on the whole, you want to be thinking about what controls are going to be on the screen versus what controls are going to be out in the scene. And kind of a mantra that the team has had is to say, you know, scene over screen. Obviously, we talked about sort of battery cases of when you'd want to put something on a screen level, but I found that it's many people's initial reaction to design everything for the screen level because that's the type of design work we've been doing for 2D applications, but you really want to start thinking more about volumetric UI and having your UI out into the scene itself. So give a quick example of this. This is actually one of the demos that ships with scene form. It's a solar system simulator. Loads of fun. Also, it's missing a planet right now. We did fix that for the public release, in case you notice that in the video. But imagine now you need to design the UI for this. So a lot of people would initially think, oh, I'll have a gear menu up in the corner. That will throw something up on the screen. You know, the problem there is then you're not going to be able to sort of be as immersed in the simulation itself as you're interacting with it. So an alternative way of doing that is actually leverage these objects on the scene itself. So as you're tapping on planets, you'll get feedback on what planet that is, which is nice for educational use cases. And then in this particular demo, when you tap on the sun, that's how you start to control the entire solar system. So here the user is tapping on the sun, and that brings up a panel. This is actually an Android view. So in scene form, you can map just standard Android views into AR. And here you have controls like changing the orbit speed or the rotational speed of the planets themselves. And it's really nice to be able to interact with these objects in the scene, and not to have that sort of sudden loss of being able to see things and being sort of taken out of the experience. And that kind of brings me to my final point, which is this idea of AR presence. So we'd actually seen this coming up in user research studies where people would be looking through a phone and then they would kind of look outside of the phone to see that something was placed correctly. And then, of course, we're recording and so they laugh. And they're like, oh, yeah. Right. Of course. I can only see it through the phone. And we always laughed when we saw this happen. And then I was testing out an app. It was these sort of plastic interlocking bricks and I had instructions of what I was building, and I was playing it for a long time. And at one moment I looked over to see the instruction book, and it wasn't there. And I had the reaction that you normally have if, like, an object just disappears in real life. And, of course, then immediately I'm like, oh, that's silly. Yeah. It's AR. But I was so immersed in the experience and the application, and I'd been playing it for so long that I was no longer kind of mentally tracking what was real and what was virtual. And I was just sort of buying that the experience was happening. And you're going to start to have this experience as well as you're interacting with these applications. And I would say that's the moment when your application is performing really, really well because it means that the user is just completely immersed and engrossed in the application. So if you ever have these moments where people are looking at a vase through their phone and then they look down and it disappears and they react, that's good. That means the app is performing great. ALESHA UNPINGCO: All right. So we've gone through the five pillars of AR design, which, again, include understanding the user's environment, planning for users' movement, onboarding users by initialising smoothly, designing natural object interactions, and balancing on-screen and volumetric interface design. And again, this framework, we believe, will help anybody get started with creating apps that everybody can enjoy. So we have a quick video for you. Some amazing content that many designers and developers like yourselves from the community have created. We hope you enjoy. [MUSIC PLAYING] We're very happy to share with you everything that we had today, and we look forward to seeing what you create. Please fill out our survey and check out our resources online. Thank you. [MUSIC PLAYING]
Info
Channel: Google Developers
Views: 48,362
Rating: 4.9605026 out of 5
Keywords: type: Conference Talk (Full production);, pr_pr: Google I/O, purpose: Educate
Id: bNJJCREZgVM
Channel Id: undefined
Length: 33min 59sec (2039 seconds)
Published: Wed May 09 2018
Related Videos
Note
Please note that this website is currently a work in progress! Lots of interesting data and statistics to come.