Unity at GDC - Building an ARCore app: Learnings and Challenges

[Music] wait okay now from the beginning hello everyone hi welcome to building an AR core app learnings and challenges just to get a sense of everyone's experience of air core who has just started developing with air core in the last month or two or is looking to start oh wow awesome awesome anyone else beyond that there's like a few months of experience cool cool great awesome so today's talk is not going to be about what is in a Airport instead we are going to focus on some areas to consider when building air core apps and possible ways to address them the idea is you can go through tutorials on your own but this is kind of like some real-life information that is more difficult to find with a Google search so a bit about myself I am a software engineer at Google and I work on air core initiatives before this I was the team lead for the Google VR and tango unity SDKs and engine integrations I also worked on cardboard camera Android and iOS and I competed at the Olympics in the figure skating slightly unrelated note thank you in my current work building air core apps on unity is a very large part of my work and today I'm going to talk about some challenges and learnings from a almost a year of prototyping with air core today we're going to start off with an overview of air core concepts and then we'll dive straight into the key issues this has to do with surface detection planes air interactions - DUI and most importantly performance okay it's four so for our overview let's this is a screenshot from the a are stickers app it demonstrates motion tracking so with motion tracking what this means is that your phone knows where it is in the world like position and rotation you can do a lot of really interesting things with this like object interactions based on distance so for these little guys when you walk closer they will interact differently they'll animate differently and what's even better is the more you track the more better understanding your phone games of the world so this brings us to anchors anchors are like poses except the poses are temporal and they don't last forever anchors is basic it's literally anchoring a virtual object to a place in the world so what this means is as aircore gets a better understanding of the world your anchor updates with it for one example is let's say when you start up your core it thinks the floor is about five centimeters higher than it should be if you place an anchor on the floor over time that will also go down five centimeters as air core understands that the floor is actually five centimeters lower than where it initially thought it was so that is what anchors does and planes so Airport understands horizontal planes this is anything from a table to a floor and it knows like the boundary of the Florida text with X like horizontal X and Z extents planes can merge with others so this means if you detect a portion of the floor and in a separate portion over time now as it detects more and more of the floor these will all merge into one and we'll come back to this later on why this is really important it's important to note that air core is good at tracking textured surfaces much like the human eye these are features or easy identifiable things that you can pick out at a distance so things like white balls or glossy surfaces they are core will have a hard time identifying those the same way that your eye would have a difficulty picking a point on a white wall and next is also lighting estimation a our core can detect the lighting information of an environment and it gives back a value that represents the average intensity of the ambient light this lets you render your objects with really realistic lighting so in this example we have used Unity lighting lighting estimation aware shaders so it takes that average intensity value and multiplies the color by that value so when the room darkens the lion also darkens it also lets us detect when the average value changes so we can have lighting aware interactions like this scaredy lion and finally we have points in the air core this is we call them a point cloud these are features that again your eye can pick out at a distance there are normal points and what we call oriented points when there is a cluster of points together we can estimate the normal than its perpendicular to the surface this allows us to place objects on surfaces at an arbitrary slope so not exactly a non horizontal plane but close enough okay let's dive right into learnings and challenges I want to note that the answers I'm going to cover are not the only true way to do things as you'll see there's no one-size-fits-all solution and it often depends on your use case that what you're trying to do surface detection so when we talked about planes we did we did cover that when air course starts up it needs some time to find the floor or find a horizontal surface some challenges this presents is that your users are new to AR they may not know that surfaces need to be detected or that you need to detect a surface first before placing objects they may not trust the scale or placement of objects so one approach we can take is tutorials for example in the AR Snickers app they show a tutorial that's basically a phone moving in a circle they show it just once and it goes away it's very minimal but it tells us how to find a surface and why it's important so this is the onboarding when the user needs it approach because in studies we've discovered that users don't read tutorials much like the rest of us detection takes some time so while this is going on how do you make it look good do you allow initial object placement without a floor what about when you lose tracking or what about when you do is planes so the addresses each of those in order one option is to yes allow initially placing objects even when you haven't detected a plane this is the approach that a are stickers uses and when a floor is the test so they've set some arbitrary floor height below your eye level and then when you actually do find a floor it enters the object right to the floor when you lose tracking what you can do is so one possible approach to taking care of loss of tracking is have like know where your floor values are while tracking is running keep the fake floor so that when tracking goes away your objects are still on the fake plane your objects do not necessarily need to be anchored to the real plane so that you have the optimal user even when tracking goes wrong for some reason and here so this is a screenshot from the tabletop app this was a launch app for Air chord developer preview 1.0 we had it as a press demo and I was part of the two-person team that worked on this we decided to go with spinning dots and this kind of a plane visualizer to show people that a surface was being detected and we decided not to allow placement before before a plane was detected this is just a choice so we had a very different decision and approached and here are stickers so again this is one of those it depends approaches and yeah visual indicators which we we talked about so one one pitfall with showing tutorials is if you may not want to show it all the time you may board a user but you also want them to realize that plane detection is important so when placing objects what you can do is show the part of the plane that's right underneath two objects what we chose to do in this app was show the plane whenever they're scanning the surface whatare stickers chose to do was not show the plane at all and just let people place so this results in a few different user experiences for one the user may not be aware that planes are there but that maybe that's what you're going for or maybe you want something super robust and you're thinking from a debugging perspective where I always want to know where my planes are so it depends on user education and place how to play testing to know what your users expect so to summarize our learnings we have to we want tutorial visuals for movement and optimal areas to scan so that they know not to scan glossy surfaces or white walls you may consider starting with a default floor plane and potentially having loading indicators like a spinning circle or some kind of a loading polygon and this takes us to planes so air core provides us with what they call trackables these can be points or planes and when you get all de trackables for a session they really mean all of the trackables so this resulted in some challenges all trackables remember when we talked about how some planes can be subsumed or merged with others well the little plane fragments then how they have this flag that's called subsumed by and it becomes non null when they are substituted but they're also returned in the list of trackables that air core gives you so what you want to do is filter out for the good and valid planes there we found that there are three key things to check for which is subsumed by is not null mesh very well subsumed by as null so obscene by being null means that it has never been taken over by another plane and it has its boundary vertex has a vertex count that is not zero and it's X instead extents are not zero we'd learned the hard way that if you're visualizing planes you might be visualizing invalid planes this can result in things like a plane floating in mid-air for some reason and it took us a while to figure out that we can't just take all the planes that aircore gives us and use it blindly having multiple planes so this results in you might think yeah of course I want all the planes you know of course I want the table and the floor I want to place objects everywhere one complication of that is dragging an object between things so if you drag it object from the table to the floor do you just have a jump sharply do you have it fall from the table to the floor what about if you go to opposite way what looks good what does the user expect it may not you may not want users to drag things between floors and tables all the time imagine if you had an art installation and you had a huge AR sculpture you wouldn't want a user to drag that from the floor to a table where it would be way too big and probably not the appropriate context for appreciation so it depends on your use case and it depends on if you have the resources like the developer resources to invest into making that interaction work what we chose to do for tabletop was to allow only placing on a single plane so we went with the lowest plane be detected and we stopped it at that this simplified a lot of things and we were able to ship the quality app in a short amount of time parallax effects so in this video notice that the burger originally appears to be on tree stump but it is actually placed on the floor behind this is so one way to mitigate this is to put contextual visual indicators for example the feet its feet and its shadow show that something isn't quite right when you're looking at it on trees um another example which we'll see a little bit later is to have show how far off it is from the actual floor so having visual indicators goes a long way to show to mitigate these kind of parallax effects so to summarize our learnings filter for valid planes think about what multi-plane drag and multiple objects if it applies to your use case and what your development costs are and add context of where it indicator visuals to mitigate parallax effects and this takes us to a are interactions so placing objects sounds like yeah I'll just place objects but this sound this is actually a pretty complex mechanism there's kind of a couple ways of selection one is dragging from a list like a are stickers does or you tap on the button and it places an object in front of you there are several consequences to the choice you make for example if you drag and drop this hides a large part of your screen underneath your finger and it might be difficult to place things on in the world or to place things where you want them to go on the other hand when you tap you need to show a target visual to tell people where they go into place this means that you can only aim like a bull's-eye where if your center of your phone and say look I know you're going to place something here one complication with tapping is that you don't know ahead of time what asset the user is going to select this means that you cannot put a sized footprint to show them how big their to show them a preview of what their object will look like in the world whereas that is something that you can do with drag-and-drop so this is so if you're choosing how to select assets this is again it depends on the aesthetics of your app and what your use case will be and do you allow placing of only one asset or many at a time this sounds simple but again it is a complex issue if you have many assets should they collide what about their physics should you place objects on top of one another do you should they fall and tumble realistically what if you try to drag an object from the floor to the top of another object what if the user wants to find an already placed object so these there are no concrete answers to these questions I did see once I saw a really cool app that was for furniture shopping but you would place objects on top of each other you could put a hurricane in it you could have a laser gun and shoot things around but it was all fun but it wasn't appropriate for furniture shopping so again it depends for a fun app like air stickers I think right now they can't you cannot place an asset on top of another one you can elevate them but they're not actually placed on top of each other but they collide realistically like when they're beside each other so that is one reasonable choice that you could make manipulation dragging rotating scaling and elevation so in here you can see that we can scale objects we can elevate them but there are cases when this is something you want to restrict and you don't want to let the users scale for example furniture shopping or an art installation you want to preserve the exact size of the object if the user was shopping for a couch and they scaled it up already scaled it down that is not a realistic representation of whether the couch fits in their space or what they will get in the mail no not a meal but well what they get from Ikea but then the complexity of this allowing scaling is that you want to communicate somehow when did one day try to scale that we intentionally do not want you to scale because this is a true-to-life representation some approaches we could take is a bounding box or to show dimensions of like this is X meters by Y meters or to just I don't know flash for it and have sirens it's really up to you fine user actions that make sense so there are a few ways you could think of for rotating assets you could have two fingers where you spin them around or you could have one finger where you swipe it in a circle but consider that the latter when you spin it around with one finger that could be confusing with moving objects and dragging them around in scenes so pick things that will feel intuitive to the user and reuse existing interactions so that you can take advantage of existing user education and for elevation you may want to allow users elevate objects again like much like our installation and furniture shopping use cases it depends but if you do allow elevation you may want to show that it's actually up off the floor and not far away not dragged far away onto the plane so here you can imagine that without that visual - in that vertical visual people would think that it's actually further back on the floor instead of straight up from where it is and finally for dissection planes and oriented points are all good surfaces for placement but do you really want to allow people people to place objects on both planes and oriented points what about dragging between them how does the user know what they're placing on imagine if a user placed an object on an oriented point that was at a slope and they tried to drag it around this wouldn't works so well as well as it does for planes you could have like a fake Horizonte cloaked plane and have them drag it around on that but it would it might be confusing so one way you could approach this is have a different kind of visual / placement type or maybe disallow placing an oriented points at all or you could allow placement on oriented points only for a particular kind of asset for example you could let people put little figurines on or into points but not larger objects like houses and allow those to be placed only on planes so to summarize our learnings asset selection and placement mechanisms really depend on your use case and the aesthetics are going after the same applies to object orientation and handle planes and oriented point placement distinctly with the appropriate visual indicators UI blindness okay so we've done some user studies on our prototypes and what we found is that users get blind - - DUI when they are in a are one possible reason is because their focus is past the camera and into the real world so they're so they look past a lot of the 2d controls for example compare the AR stickers to DUI to the Android camera - DUI on the far right there are a lot of controls they on the camera UI that are missing and that you would miss anyway in the AR stickers app such as the hamburger menu icon HDR flash not that these controls wouldn't be great but we would need to message them in this different way one example of how we are blind right now is can somebody help me pick out a Tootsie not the assets but a a icon on the AR stickers app that is more geared toward a utility function the hint is it's not in the bottom it's not the shutter button and it's not one of the asset buttons yes good good yes the garbage can I missed seeing the garbage can and it's it's very easy to miss it's a 2d UI it should be you would think that oh it's a garbage can right in front of you but no it's this is one of the things that can be missed with UI blindness so one approach you can take is change or 2d UI design approaches everything should be absolutely critical to the AR experience know just in case buttons and have collapsible UI components so this asset menu at the top for ear stickers that can go away with a swipe and it can come back to maximize screen space and many users really like using portrait because they're so engaged and so invested in the experience they just want more of a freshest screen space so having landscape mode is very very important to air experiences and adjust your UI accordingly it's not shown here but it is okay to disable some key UI function to the UI functionality in favor of screen space when you're in landscape mode so to summarize our learnings adjust to D UI design approaches and be super minimal use only things that are critical to your AR experience and maximize screen space in landscape just really really go for our precious screen space real estate ok and finally the thing that makes or breaks our app is performance um one challenge you may run into is over time you see a general slowdown without a clear cause you go through your app you try to refactor things but there is very little gain one some ways you can combat this is look at memory management random memory access patterns or cache misses or fragmentation leads to this general slowdown consider data oriented programming as opposed to object-oriented programming favorite composition over inheritance I'm not going to go way deep into details of data oriented programming unity has a really great talk coming up on Friday you may want to attend that one briefly here's a one example of object-oriented programming versus data oriented programming in object-oriented programming you would focus on the procedural calls for example if you had a bunch of err objects each object is a model behavior you create a new instance when you place it and then on update for each of these objects check if the user is near me and place some kind of an animation if they are in data oriented programming your focus is on the data that drives the app their type their memory layout how they are used so this right off the bat enables object pooling and efficient reuse so in our a bunch of a our object example with data oriented programming we would have an error object manager you have an object pool with a max number of objects you could allocate and on the object managers update method you check if each of these objects should play the animation so instead of having for each object go into update and do this thing you have in this one model behavior of the object manager do this thing for all of my objects some really great side effects is that you minimize the number of monobehaviour xand per frame let's see take a look at the 10,000 updates unity blog post which goes into some details and numbers the huge difference that it makes by eliminating update calls from your game and profile constantly you never know when any change like a big feature change or a bug fix could introduce additional memory footprint into your app so profile do a lot of profiling profile on every milestone use the unity profiler tools they go really long way in helping you figure out what is leaking memory you may notice that performance decreases as objects are added this is partly because anchors are quite costly so be very cost efficient with your anchors consider limiting the number of objects you can place in the scene when you move an object destroy the old anchor you can do this now of aircore 1.0 which you couldn't before you can destroy anchors so destroy your old anchor and create a new one where the object has been moved to if you have an object that moves around constantly one great example is someone brought up to me today was having a fish swimming around consider having the an anchor around a central point of where the object is moving and just keeping that one anchor and when you delete objects always delete your old anchors something this goes a little back into user interaction something you need to consider is that users can lose or forget objects they have placed so think about cleaning up or detecting when those objects have been lost and are floating around uselessly in your scene and and cleaning them up ok this is a huge one scene changes so it depends on what kind of a phone you have like a pixel versus pixel 2 or Android and versus Android oh but scene changes can take a while so if I have two AR Cora scenes and I switch between them that's easily five seconds lost in the transition that is huge so when also consider that when aircore we start this connection stale state might stick around so what you want to do is reduce your scene changes and costume them so for example in my if I had to aircore scenes just get rid of that scene change entirely put them all into the same scene have state specific prefabs and just switch between those prefabs make on enable and on disable your friends start is only called once and destroy on destroy may not ever be called it's not guaranteed group your air core and your 2d scenes together group them in this into the same flow so that people don't have to go between 2d and air for too many times that being said you can consider having a 2d landing page the 4-day are seeing starts up what this does is when someone goes into the 2d scene you can actually secretly start up the air core connection behind and have it try like actually it will actually doing plane detection so that when people switch from the landing page to a arc or voila the transition is super quick or it appears to be super quick one risk of this last approach is that if the user is covering the lens you might hi of garbage data but you still earn a lot in reducing startup time and getting a snappy - DUI should you go native or unity I've done both they're both feasible they both looks like in tabletop what we did was we actually had the air core experience is all done in unity and then we had a really clean interface that went into Java and all of the 2d you are was in fluent in Java this works this is snappy but there are some drawbacks one of which being you need to have well you need to have a very crispy defined interface with Java and if you're more comfortable with unity canvases and that layout you probably want to stick to unity it's not really the Android way of building UI is very very very similar to how unity works just in a different language but it's totally the same paradigm with canvases and the layout groups you can achieve the same material you are look as Android it uses the same principles the drawback with using unity components is that you have to be very conscious to use again data oriented programming and whereas with Java you can be a little more lacs with your memory management and stuff because it's it's native and it yeah you don't it doesn't take exactly the same memory footprint as unity 2d you on so it really depends on the skill sets you have at hand and it depends on your end goal whether you want to go native or unity for your to D UI my personal feel opinion is that they're both they both have strengths and you can achieve the same end goal with either Android or unity or iOS so to summarize our learnings always be profiling use general engineering best practices and consider data oriented programming to maximize performance clean up your anchors clean up your anchors reduce your scene changes or consolidate them or costume them and pick whatever to D UI approach you're familiar with just be careful again to be conscious of performance and keep profiling what's next okay so today we've covered a lot of ground we've seen how seemingly simple concepts like surface detection planes and object interaction are actually much more complex in practice we've explored some challenges that we've come across in some ways we can address them I hope that today's tour gave you an insight into topics to consider for your next aircraft but what I really hope to see is not only what awesome air experiences you can come up with but also what other challenges and solutions you will find in your journey we are just getting started with air core development and I can't wait to see what we can all build together thank you [Applause] does anyone have any questions yeah um when you when you said that you were doing some of your UI in the java end as opposed to in unity did you get accurate profiling out of unity while you were doing that or did you do that in a different you know Android studio or something yes so this goes into you know how unity has the in editor profiling versus on our device I think we didn't do comprehensive profiling of both UI and the air core experience we did mostly mostly focus on air core experience but when we did both it didn't I recall now that it didn't have much difference like take it ripping away the UI and leaving it and didn't really have much impact on the overall app Thanks yep Thanks that was great talk thank you very much as lots of information it's troubling to catch up here together thank you you mentioned that to know if a plane is valid they were like four things that it's so subsume is different than no subsume by is no vertex count difference Iran something else is different than zero okay so subsumed by is now the number of vertices on the boundary polygon is not zero and extent X and extent said are both also not 0 so extent X and said if you think of their coordinate systems as Zed pointing for well in OpenGL okay in OpenGL negative said said pointing this way why this way and X this way then you have your horizontal plane like this so your x extent this way and your Z extent this way are both not zero okay and you need to check all those to know about plane is valid like one of them for example is o because each one tells you like they can die in different forms so I guess okay so it's no it's you need to check all of them or you can just to get by by checking this one it's a good question I we at least we found that the subsumed by and a number of vertices like you can have one without the other you can have something that's not subsumed by but has zero vertices so that's invalid as four vertices and extent X and said I know I haven't come across a case where one is zero and the other is not but if you like you it's possible that you can get away with just checking one so it's really up to you thanks then just a quick follow up question you mentioned like you sang the praises of data oriented programming for performance reasons I've actually done similar and it seems that when you do object oriented is much easier to end up with a unity project that non coders can manipulate and touch because there's like each object has its own set of properties and behavior and such so do you have to kind of like pick and choose which areas to make that trade-off on or is there a way to do data oriented programming but still don't lose the like access to non-programmers to the project that's a great and that's a great perspective I think you can either pick and choose which areas to make data oriented and which not however as like just a gut feeling is as that sprawl of the object-oriented takes over sometimes it might be difficult later on to refactor everything to make it more efficient this also goes back into it goes it goes into a slightly tangent of asset pipeline like how do you make sure that you can still work with your artists should you maybe have a debug scene with an object-oriented throw away script that they can iterate on and then later you port this whole thing and make it data oriented so several different development approaches you can take cool thank you so much everyone for commenting and I hope [Applause] [Music]

Channel: Unity

Views: 17,585

Keywords: Unity3d, Unity, Unity Technologies, Games, Game Development, Game Dev, Game Engine

Id: yS2OtBcpDLA

Length: 40min 19sec (2419 seconds)

Published: Wed Mar 28 2018