Architect Multi-User 3D, AR, and VR Apps with AWS - AWS Online Tech Talks

welcome everyone to this presentation on creating multi-user training apps in AR VR and web 3d my name is Chris Schulz I'm a specialist Solutions Architect with AWS and my focus is on augmented reality virtual reality and real time 3d applications of AWS services I think there should be a fun and interesting topic so I'm anxious to get started and share with you some insights I have based on a project that I've worked on recently so this presentation is really based on a presentation I gave at our recent AWS reinvent conference I've modified it and expanded it slightly to fit the webinar forum so in the reinvent version it was driven by a lot of audience participation and open dialogue obviously that's a little harder to accommodate in a webinar so this version will be focused a little bit more on the presentation side but we'll still have plenty of time for you to ask questions at the end so I realize many of you maybe have been enticed by this presentation for a number of different reasons so the description of the presentation covered a lot of ground so it mentioned augmented reality and virtual reality applied to training it also specifically mentioned an AWS service called Amazon Sumerian which is our service for creating immersive 3d content and it talked about architecting real-time multi-user applications so each of these topics could be a whole session onto itself I'm going to cover a little bit about each one of them and dive deep into each one of them but if there's something I don't cover about one of the topics you're most interested in I will be providing some suggested follow-up resources for you to use afterwards as well as my contact information if you have additional questions after this webinar so let's dive in we'll start by talking a little bit about Amazon's Sumerian in case you aren't already familiar with that that service so Sumerian is our tool for creating immersive 3d web experiences that can be accessed in any web browser you can think of Sumerian is a toolset that lets you build and deploy these experiences very easily and these experiences can be consumed in really three different ways one is as a interactive web 3d experience that works through Chrome or Firefox on desktop or tablet or mobile device it can also be experienced through a virtual reality headset or as augmented reality which superimposes the interactive content on top of the real world when you're viewing on your smartphone with Sumerian you can target any one of these modes of consuming the experience or you can build experiences that target all three and that's what I'm going to be showing you in the example that I'll be sharing in just a bit if we look at the general content production pipeline Sumerian has a lot of features and benefits that really make it stand out so we wanted to make sure that Sumerian made it easy to create this type of 3d content so literally you can create Sumerian experiences with just a few minutes of work the editor and the API and everything that makes these experiences work is all browser-based so you can actually you don't have to install any tools locally you just access access them through your regular web browser and within that editor that runs in your browser we provide both a sort of visual metaphor for adding interactivity as well as full programmatic access through JavaScript so whether you're a program or a non programmer you'll be able to create the ideas you want to create in Sumerian another thing we do to make creation easy is we give you a library of starter assets and templates to work with so you don't have to always create things from scratch and a big part of the Cimmerian story is really integration as you might imagine with any of our AWS services so augmented and virtual reality applications really benefit from a variety of different services being integrated with them in sumerians specifically we have special integrations for things like Lex our chat bot service which are also offers natural language processing or poly our text-to-speech service or IOT for creating 3d visualizations of IOT data but really any of the AWS services that are available through the regular AWS JavaScript S key SDK can be used in Sumerian and that's really opens up the possibility for even integrating third-party JavaScript libraries because as all of Sumerian just runs in the browser using regular web technologies so you can incorporate third-party JavaScript libraries easily to integrate with any service that you want to use and then the final piece of the value of Sumerian is that it makes a delivery much easier than some of the other options for creating this type of AR and VR content so it's multi-platform just like any other web application you don't have to compile specific builds or specific devices which is a huge win and deployment can be as easy as just clicking on a publish button to share your work with the world globally across our content delivery network so those are some of the values and benefits that Sumerian offers but really to understand it it's best to see it in action and that's what we're gonna really focus the rest of this presentation on so in my role as a specialist solutions architect especially for emerging technologies like AR and VR a lot of what I do is help customers not only understand how the technology itself works but understand what is made possible by these technologies so I undertook a while ago a prototyping effort to demonstrate the value that AR and VR could apply to training scenarios err and VR are really powerful tools when it comes to training because there's a lot of studies that have shown the physicality and the spatial nature of AR and VR really increase retention of information particularly if you're training about a physical topic so I set out with the goal in building this prototype that I wanted to demonstrate to customers the art of the possible when it comes to applying AR VR and web 3d to training and the vision specifically for this this demo was to leverage these web-based 3d AR and VR technologies to empower a domain expert of some sort so this could be in an enterprise it could be in an educational institution someone who has some expertise they want to share give them the power to be able to lead live training sessions remotely with users or learners across the world very easily and additionally I wanted to make sure it also offered a low effort mechanism to persist that knowledge so that it became easily accessible as on-demand training after the fact there were a couple constraints I put on myself for this effort one was that I limited it to a team of one just myself and I time box the effort to two weeks now the reason I put these constraints on myself was I wanted to a show our customers just how much could be accomplished in a small amount of time with just one individual in using these technologies to bring an idea like this to life and the other thing that timeboxing gave me is it forced me to make certain design decisions quickly and sort of gave me a yardstick by which to measure those decisions could would they help me achieve my goal quicker in the end so let's take a look at the finished product so as you might imagine in AR VR and web 3d experience has a lot of moving parts so trying to demo it live over a webinar can be a little challenging so what I've done is I've put together a video showing the whole experience working I'll play that video narrate over it and potentially pause it here and there to emphasize a few points so let's start with the view that a student sees if they're logging in from an 8 from their smartphone to view the AR experience so everything is protected by a user login once the student signs in they can then place the virtual object that they're being trained on in their physical space so here we see an industrial air compressor it's um been miniaturized to fit their space but they have full access to the virtual machine now we'll look at what a student logging in from their desktop or their tablet would see similar login process but this time the piece of equipment is in a virtual setting the student still has complete control over their viewing angle and what part of the machine they're looking at at any time so this is fully interactive throughout the live training session look at the instructor logging in will move the augmented reality view out the way for now so the instructor is actually in virtual reality and if you look on the left at the student view you'll notice that the instructor is represented as little Avatar and all the instructors movements are mimicked by that avatar so where the instructors looking where the instructor is positioned relative to the machine in VR some of the built-in features of Sumerian are it gives you controllers that you can use to interact with the VR world around you it tracks your position in space so all of that functionality is just built into Sumerian you don't have to do anything custom to get that that functionality another bit of functionality that's built in is the ability to move yourself greater distances in VR using a mechanism we call teleporting so you point on the floor hold the trigger on the controller and it will zip you right to that location that's a pretty common interaction model for navigating in VR and we've provided it for you so you don't have to build it yourself then I also added a custom implementation for VR which lets me use the thumbstick to rotate my body by 45 degrees now in VR obviously you can turn your body physically and it will respond but if you are perhaps in a seated situation or in a small room sometimes it's convenient to be able to have that sort of manipulation so that's basic navigation in this application now let's actually start the training session so in VR we can have some a lot of freedom to use spaciality in the way we design our user interface what I've provided is a little tool palette that automatically reveals itself when the instructor rotates their risk and then conceals itself when they turn their their hand back so this palette could have any number of tools I've only implemented two just to sort of show a couple examples of what you might do and the way you interact with those tools is you grab them by intersecting the blue sphere on the controller and pull the trigger on the side of the controller to grab that tool and pick it up so we've picked up the laser pointer this is one way the instructor can draw users attention or students attention as they're communicating information so if you look at the student view on the left you see it's completely in sync with what the instructor is pointing at that laser pointer is what I call ephemeral data so it's not recorded anywhere it's only viewable in the live during the live stream of this training but we want I wanted to be able to do some persistent data as well and that's where these info pins come in so if I grab an info pin I can press a button on my controller and it starts listening to my voice and transcribing what I'm saying into text that text then becomes a part of that info pin so it can be accessed at any time later so once I've recorded my message or my annotation to be a part of that in I just let go of the pin and it stays wherever I put it if you look at the student side you see the pin appears when I release release it into the world and disappears when I pick it back up again so let's go ahead and place a few pins on this machine just to sort of show the whole process the instructor would go through during the lesson there are going to be certain parts of this machine because it's like 15 feet tall in the the instructors view that the instructor can't reach so this is virtual reality we have complete control over over the the world so I've provided a way to scale down that machine to make it easier to access with a couple buttons on my controller so I just scale it down to a miniature size that's about waist-high and now I can easily reach every part of that machine to place the pins that I need the place and then when I'm done I can scale it back up you'll notice the pins sort of stay a consistent scale while the machine itself changes its scale and in the students side the scale doesn't change the student is always the one in control of their view of the experience so that's the basics of the interaction but there's one Easter egg I put in here that I'll show you just to show how you can get creative with navigation jetpack engaged so this is a feature I called jetpack mode which you can enable to allow you to actually levitate yourself to parts of the machine that you can't get to you you'll notice that I made a design accommodation here where while you're moving your peripheral vision gets blurred and restricted this is actually something that you need to pay attention to when designing VR experiences is how to keep your users from feeling queasy when they're navigating VR teleportation is one really good solved removing users in VR when they're Cui's without getting queasy that tunnel vision is another another way so here's the that a our view again since we didn't see it live this is just to show that all the same features you saw working in the student view on desktop are also working in the AR mode so the laser pointer the tracking of V of the domain experts the placing of the pins all of that can be viewed from the air experience as well and this view shows that those pins are actually interactive so when the student touches one of the pins it reveals that text that was dictated by the instructor so this could be viewed during the live stream or it could be viewed as an on-demand module afterwards you could also imagine that that text could be automatically translated to other languages using an Amazon service or that that same metadata could be applied to an application for field service technicians for example where they have the actual piece of equipment in front of them but those annotations get overlaid on top of the real piece of equipment to help them operate or repair maintain that piece of equipment so let's take a look at how this was built and walk through some of the architecture so there were five primary services that I used to create this application of course the first and the most visibly obvious is Sumerian so Sumerian provided the 3d development tools that I needed to build the experience as well as a way to deliver that VR content over the web to end-users I also used AWS app sync which provides a way to synchronize data across multiple connected clients now that data synchronization can be asynchronous or it can be real-time and so I used it for sort of the real-time multi-user features in this application it's also used to help make the persistent data so those info pins for example possible the actual place that that metadata is being stored is in DynamoDB so that's where our content data storage happens and for user login or identity management I'm using Amazon kognito as the service to help power that and then AWS amplify isn't a service per se it's more of a tool belt it's it's a set of utilities for developers that reduces some of the heavy lifting you normally have to do when building into point web applications and it provided an easy way for me to get my web hosting up and running and it also provided all of that really great-looking user interface for my user login so let's look at how those pieces connect together so starting with the users when a user logs into the application they're first going to access the URL by a cloud front which is our content distribution network Amazon Cognito is what's powering that user authentication piece and once the users authorize they can see the full front-end application which is hosted on Amazon s3 I built the front-end as a react j/s application it's not a very complex front-end because most of the functionality is actually built in in Sumerian but things like the user interface are delivered as react.js components and then of course my sumerian scene is embedded within that application while the the live training session is happening Sumerian is talking to appsync which is handling all of the coordination between all the students and the instructor while they're connected and then app sync is persisting some of its data like those info pins to our dynamo DB database within appsync itself there is you define your API through something called a graph QL schema and I'll talk a little bit more in detail about what graph QL is in a moment and you also use something called resolvers which simply tell Apps Inc how and where to store the data finally as I mentioned amplify provided a toolset that I could use to really coordinate the all of this infrastructure and connecting all these components together and it was super helpful in helping me meet that two-week time frame so there were a number of design decisions and challenges that I encountered along the way with a project like this this is the point when I delivered this as a talk at reinvent where we really opened it up for a lot of open discussion what I've done here or our our session today is taken some of the the topics and ideas that came up during the reinvent session and chosen those to list here and we'll focus on just the first four to dive a little bit more deeply in how these pieces were implemented and how they came into play so we'll start with the user login so when building this application I was faced with a design choice and architecture choice to make about how to support user login so obviously one choice I could have made would be to roll my own to implement my own custom user login which would have meant implementing a service on the back end to manage users and to do authentication a database to store user information and then the user interface on the front to make it all work or as I chose to do I could use amplify combined with Cognito to give me that functionality in a much more easy way to implement so to give you an idea of what easy means amplify when you're using amplify one of the tools it provides is a command-line interface for adding features and functionality to your web application so I'm going to give you sort of a high-level view of what I did to add user login and user management functionality to my application first to add user logins support to my web app I just needed to execute this command line which is amplify update auth so this adds the configuration and the user interface components that are needed to support authentication you I executed amplify push which takes care of setting up all my back-end resources for me automatically so I didn't have to go in and configure anything manually in my JavaScript code for my application itself I just had to change that first line to that second line to enable authentication now I chose to do this app as react app but amplify also supports view and angular and a few other front-end application frameworks so you have a lot of options there and then finally to share the application with the world I just use amplify publish which takes care of building my application and distributing it to all the right places so that it can be served up globally so it really was this easy there's a few more details in here that I've glossed over but you can find all of those details in this Sumerian tutorial so if you're interested in using amplify with Sumerian or adding user login functionality to a Sumerian experience this is a great place to start this tutorial the next thing I want to touch on a little bit is building for reuse so I do a lot of these demo applications and prototypes and one thing that's important for me is that they not be one-off examples but they that they actually result in some sort of reusable elements that can help accelerate our customers when they're building their own projects and so this project in particular resulted in a lot of reusable components and that's because Sumerian actually has a great model for adding interactivity which lends itself really well to designing in a reusable fashion so within the Sumerian editor any object you see in the virtual world is something we call an entity so an entity just has a 3d position or rotation it's it's got a point in space that it lives and most entities are visible and you can make any of those entities interactive by adding scripts to them the great thing about the way this is implemented in Sumerian is you drag and drop those scripts onto the entity itself and you can combine multiple scripts so a single entity might have many scripts attached to it so this really encourages you to build small focused single purpose scripts which turns out is a great approach for creating reusable reusable functionality so what you see here is a screen shot of that info peon and the scripts that I had attached to it you're not seeing the actual script code here what you're seeing are the public sort of parameters that each script exposes to be able to customize that scripts behavior here's what an actual script looks like so this is one of those scripts on the info pin it's not a very long script but it's also not a trivial piece of code so I don't want to have to write it multiple times so it's really great that I could write at once and then reuse it across many different entities here's just a list of some of the reusable scripts that I ended up creating as a part of this project that I've now been able to reuse on other projects I think these scripts are really useful in a variety of different contexts so we're working to create an open source repository where we can share these scripts and others with customers and also where the community at large can contribute new scripts for Cimmerian that isn't launched yet but to be informed of when it does launch I suggest signing up for our public slack channel and you'll find me on slack as a WS - Chris or you can always email me directly and I'll keep you informed of when those scripts launch as well so let's look at one of the challenges I needed to solve as a part of this application so it turns out that multi-user applications especially ones that need to sort of reproduce real time movement can be a little bit tricky to implement and game developers have have struggled with this and solved this over the years it's not a trivial problem and the the challenge here is that the updates that describe where any object is in the world those are only happening at about one frame a second or two updates a second but the application itself is probably running at 30 or 60 frames a second or in the case of VR maybe 90 frames a second so if we only update the position of our elements when a new update comes in it can look very jerky so I'll show this video which kind of demonstrates the before and after of the problem I was trying to solve so this is the same application that you saw but you'll this is doing no compensation for those low frequency updates so it looks very jerky in almost stop-motion the the movement of the avatar you'll see that the laser pointer is also kind of hard to follow because it jumps from location to location and isn't moving fluidly this is because the update cycle updates are only coming in about once or twice a second so I had to come up with a strategy that would make it look much smoother as if we're getting updates 30 times a second or 60 times a second so this is the end result of the solution I came up with it's very fluid very easy to follow from an end-user perspective you don't necessarily realize that the updates are coming in at a very slow fashion so how did I implement that well it's it's easiest to illustrate this through a little bit of a sort of illustration series here so imagine the white point on the grid here is the starting point of my avatar that little head representing the instructor in the scene and then I get a position update from the server that says the avatar has now moved to this new location in the 3d world instead of immediately moving my virtual object to that new location what I do is I move it some fraction of the way in between those two points I'm going to show it as moving half way here just for simplicity but it's actually a much smaller fraction in the actual implementation so then that's the first frame update on the the client side so it's a rendering update which is about thirty or sixty times a second so so we then another frame update happens and we still haven't gotten a new position update so we do this apply the same algorithm we move halfway toward our goal then another frame update happens and the position still hasn't changed so we move another half way toward our goal so we're getting closer and closer to our goal but never quite reaching it so it's sort of doing this smooth smooth motion between the two points now let's say we do get a position update it's maybe a second later than the original position update and now a frame update happens and we keep following that same patter and keep moving halfway toward our goal so with this approach where you're always getting a little closer to your goal the illusion is that we're getting much more frequent positional updates than we actually are resource I would point you to to implement your own version of this is a part of our API Docs that describe the tween move action now this this is a piece of code that won't necessarily do exactly what I'm describing but it'll be the the most important component you would use to implement an approach like I've shown and if you have additional questions about how I implemented that please feel free to ask okay we'll move on to the final challenge that I wanted to talk through and that was learning graph QL so as I mentioned when using app sync to control our real-time data synchronization app sync uses graph QL as the way to define the API and the subscriptions and events that happen to make make all that work graph QL was something that was new to me when I started this project and I found it a little challenging at first to know where to start what my first instinct was was to look at the app sync documentation itself and looked for clues there but it turns out the app sync documentation sort of assumes you already know the basics of graph QL and just talks about the pieces of of how you apply it in app sync what I ended up finding to be an excellent learning resource was the graph QL org website itself they've got a lot of great information and they even have an interactive tutorial that you can do in browser that teaches you how to write and use graph QL and they break it down with very simple examples so this is a great place a great place to start to get an idea of what graph QL actually looks like I wanted to show you the graph QL I created for this particular application so I've already talked about entities in sumerian being sort of the core building block of these visual elements so in graph QL you create data types that describe any information that you want to share between your connected clients so obviously one of the most important data types in Sumerian is an entity so I described an entity with just five properties here so as I mentioned previously every entity has a position in space so that's represented by this translation property it has a rotation which tells what direction it's pointing so that's represented by this rotation property and every entity also has a unique ID which is what this instance name is the other two properties here parent names so entities can be nested inside of other entities so the parent name just tells which which other entity this entity may be nested under and then entity type was a custom property I created just to describe for example whether this entity represented one of those info pins or whether it represented the instructors VR avatar or some other element in our sumerian scene so we did with just these five pieces of information were able to describe to all the connected clients any changes that are occurring in the application a little bit about graph QL here so it's a very declarative kind of language for defining your api's what this is saying is that instance name is a string and the exclamation point indicates that it's a required so whenever you get an entity object from Apps Inc it will always have an instance name that with parent name which is also a string but has no exclamation points so that indicates it's an optional value that may or may not be there when you get an entity reference and then of course as you might guess from the translation and rotation properties the square bracket indicates that it's a an array of float values an array of numbers we don't have to talk much about the laser pointer it was just sort of a specialized version of entity so really there were only two data types I had to define for this experience and for many sumerian experiences you would only need to define this one entity datatype so in addition to your datatypes you also describe any queries that can be run against your data in my case that was just a single query it was this get entities query which returns an array of entities so this is used when my scene first starts up to know which for example how many info pins there are already in the scene and where they should be displayed again it's very declarative you don't see implementation details you just see a description of what the query is called and what it returns the other thing we can define is mutations which are things that change our data so this is very familiar to you know most any type of application we have a way to add new entities remove entities or update entities so very simple the the indications here explain what data is required to make that mutation so to add an entity we need all five properties to be provided and what it returns is a copy of that entity to remove an entity we only need the instance name for example and finally subscriptions so this is really where that real-time piece comes in so your graph QL schema can define these named subscriptions you can think of them as events or messages that connected clients can subscribe to to know when a change happens so for example I have an add entity added subscription which will always return an entity and the trigger for that is any time the identity mutation is called so if my let's say my instructor version of the application releases one of those he grabs one of those info pins and releases it into the world it calls the add entity mutation and then all of the connected clients receive this entity added subscription message so very easy again it's all declarative and then finally you just bundle all this up into a schema so you say that my query is represented by that previous query block I shared mutation by the mutation block and subscription by the subscription block when you put it all together it looks like this so it's very succinct it's not a lot of code any of the logic about how this is actually implemented is done by those resolvers I mentioned and in apps Inc we provide a bunch of pre-built resolvers which you can choose to for example persist this to a dynamo DB database if you're interested in more specific details about implementing multi-user functionality in Sumerian we have a tutorial focused specifically on that so it uses a lot of the same technologies I've talked about so definitely check that out and at this time I like to open it up for any of your questions I've also included my contact information on here feel free to reach out to me again slack is a great way to get a hold of me I also would encourage you to check out our YouTube channel which has a lot of training and tutorial videos you can follow to get started in Sumerian or to learn about specific specific topics very easily and you'll also see us appear quite often on the

Channel: AWS Online Tech Talks

Views: 2,764

Rating: 5 out of 5

Keywords: AR, AWS, AWS AppSync, Amazon Sumerian, Amazon Web Services, Cloud Computing, VR, Webinar, augmented reality, virtual reality

Id: dtE0yBySFrM

Length: 36min 53sec (2213 seconds)

Published: Mon Jan 27 2020