Intro to Apple’s Augmented Reality Tools: Reality Converter and Reality Composer - iOS Conf SG 2021

[Music] hi everyone i'm roxanna ios developer by day and era enthusiast by night and today i will show you how to use reality converter and reality composer which are apple's own augmented reality tools you can find me on instagram or twitter so feel free to reach out and continue the conversation there ar provides so many opportunities from previewing furniture into our home to experiencing newly launched cars this technology can improve our experience especially in the current of the in the context of the current pandemic where we do so many things from the comfort of our home and the big question is are apple air glasses coming soon maybe there are a lot of rumors um and there's this great roundup by mac rumors you can check out um but what we know for sure is that apple is investing a lot into this technology and building ios ar experiences is as easy as it can be and before we dive into how to use reality converter and reality composer i'd like to go over some key fundamentals we should know before beforehand first we can talk about ar quick look if you haven't heard about this technology it basically provides a very powerful experience with user interactions like moving and scaling of an object people occlusion sharing the model or all supported out of the box so you can either put this into an app in just a few lines of code or you can actually just share it directly into your devices and open it you don't need to have an app necessarily you can uh open those files and it will open the ar experience this air quick look supports two input formats usdz we will talk a little bit more about this in a bit and then also reality which is the file you will get from reality composer if you want to test out some models that are using air quick look you can go on this gallery that apple put together and you can just try out some things there and see see how it looks another great thing about ar quick look is that is supported also in websites of course because when you go into this website you will be able to see it there so it's not just for ios and ipad os apps and of course it's not it's we should mention that there are some limitations with eric but it's great to get you started and it's great in a lot of cases so it's worth looking into it i mentioned usdz as a file format that is supported by ar quick look and the technology behind it is called universal scene description what is really interesting about this file format is that it was developed by pixar it's not developed by apple so as you might know pacer has a lot of experience with 3d content a lot of animations animation movies and basically they try to do something that has a focus on speed scalability and collaboration and of course the usdz is a distribution format for the this usd and you can use it with air quickly when it comes to reality converter it's it's a great tool you will see you will use this one to convert your existing 3d objects into usdz it's still in beta version but so far i think it works quite nice i haven't had issues with it and it's very easy to use you just do a drag and drop of your models and it converts them to usdc of course it needs to be one of the format that is supposed supported like opg gmtf and you have some different options that i'll go over that you can do inside this tool and the way you used to convert usdz well your existing models to use usdz before was with uh the python tools that were you can still download those and you can still use them um you would use a terminal to convert your models uh validate them inspect them and do different things with your models but now we have this one which is just much more nicer to use and um it has more functionality in many ways this one is only available on mac os and just to give you like a very high level understanding of how this works you would have um your 3d content that comes with the geometry so for example fbx or obg and then you also have some textures and you import those into reality converter it does it it's magic and it just exports it to usdz so you can then use this one in for example reality composer which we will do in a bit reality composer is also a tool by apple and it's really really good especially if you don't have experience with 3d content creation tools like blender or similar software and basically you use this one to prototype and produce content for your area experiences you can then integrate those into xcode or even use them directly with ar quick look i think this is really great i you know because when you start using if you want if you're a developer and you want to get into augmented reality it can be a little bit intimidating if you don't have 3d experience it was the same for me and having this tool it was actually a little bit more straightforward and it kind of allowed me to start playing around with this technology and and understand a little bit more when it comes to creating your experiences it even has like a built-in air library you have um you can for example link do live linking from your mac to an ipad so you can actually test out your experience in ar you wouldn't be able to do it from your mac os you need to have it in the device so you would be able to do that perhaps even make some adjustments send it back it's a very nice flow when you want to create all these experiences it's available in the app store and you can download it for ios ipad os and mac os okay so if you want to use reality converter first off you can download it from the apple developer downloads page uh here you can see the latest version and the release notes and before we can actually test reality converter i would like to go and find a model and the website that i usually like to use is sketch web because they have quite a nice selection of 3d models so if you're looking for some models to just start experimenting this is a really good place actually about half a year ago i believe they introduced the option to download directly in usdz um so theoretically you wouldn't even need to use reality converter but of course for this um demo we will work out with different formats and here when you press on the download 3d model button you'll be able to see what file formats we have over here so let's go ahead and download the original format fbx and try to see how that goes just click on download and i'll move it on over here and i'll open reality converter what i need to do now is unzip this file that i just downloaded and drag and drop the full the whole folder first off if you don't have it yet you might need to install the fbx python sdk uh if you don't have it yet you will get like a prompt saying you need to download this if they even give you a link where you can download it from so no issue over there just make sure to have the latest version installed and because i already had that it worked but you can see that we actually it didn't work as expected uh the model that i've seen on sketchfab was much more colorful so it seems like some of the materials are missing something is off but here i can see there's there are two warnings so if i open that up i can see some suggestions of what i can do and kind of to make this work so at this point if you are the creator of creator of the model you might want to go and fix those things in whatever tool you use like maybe blender or talk with the designer that made this uh but in our case we will not do that i will just um actually what i like to do when i run into those kind of issues is use the gltf version because they've had great success with using gltf so i remembered that in sketchfab there was an option to download the gftf so i'll try to to do that i'll download it over here and again move it into my folder and i'll keep this one for now and i will unzip this new folder uh this is the new folder which has a gltf and again what i will do i'll just drag and drop the full folder and you see now it looks how we expected it to look over here you'll be able to see the previous one as well you can have like a list of multiple models if you want to work at them in the same time but in our case this looks perfect it looks like how i expected it to look so i will continue with this and i will use this one instead to convert it to usdz i can switch between those over here and i can give you a tour of what you can find in reality converter of course uh first thing you have this um 3d viewport which you can kind of navigate your way around the model you can zoom in and out and just move around and for because of this if you end up being lost in this viewport we have this button over here which says frame which will just bring you back to your model in this tab of model you can see different models you have and after you finish all the work you need to do with them you're actually able to export them all in the same time but because this one has some warnings and it didn't work out well i will just remove it and just keep this one that was successfully converted on the other side we have a button to share this one is very useful if you want to test out even from this point the model how it looks on your device so for example you can air drop it to your device and see how it looks um you can even put it into nodes or message or mail they all support air quick look so you can just test it out directly then we have the tab of environment where you can see some pre-loaded ibls from image based lighting so you can kind of switch between different environment conditions and see how your model will look you can also just click over here to show it or hide it if you want to keep the grid only and or if you want to see the environment as well it's up to you you can also show and hide the grid when you you you see the grid it's actually quite useful because over here in the top left you're able to see how big is one side of the small square over here in our case it's 100 meters and how big is um the side of the big square which is kind of this one in our case 500 meters also from the environment tab you can change between the exposure so you can change the exposure of your model if if that's what you need then we have the materials tab which is um the place where you can actually switch between your materials and attach uh base color metallic roughness normal everything you need over here i don't see how often you will want to change those in here but you have them here as a possibility and of course if you make some changes you can press on the reset button and finally in the properties tab uh here you can add the copyright information if you want to do that and i think here is actually the most important feature from reality converter and that is the base units and that is very important because different uh digital content creation tools have uh export models in different base units so perhaps you would get a model exported from a tool in meters another one in centimeters so here you can actually adjust it to make sure that it is how it was intended uh so basically if you would end up seeing your model is maybe 100 times bigger or 100 times smaller than you would expect maybe the base units are wrong so you can come here and change it and i will do the same for this one because 500 meters seems quite big so i'll just change it to centimeters and you see that the grid here that the sizing has changed so now you would see that um kind of now we will have the model cover maybe two of those big grids so it will be um you know two times the the big square of 500 centimeters so this is what you need and what you can do uh with reality converter and the final thing you can do is just export it so in our case just export we don't have multiple scenes so it's enough to click on export and i will call it my island perhaps i'll move it over here so now it was exposed exported successfully to usdz reality composer is actually embedded into xcode so you don't need to download it if you already have xcode you just need to go into the menu and pick um open developer tools and select reality composer from there um and this is how it will look when you open it up you can choose an anchor you can see here we have five uh types of anchors horizontal when you want to have an experience on a tabletop or on the floor we have vertical for example if you want to make some air experiences where you have wall decorations we can have image as an anchor so any ar content you want to place will be attached to that image this could for example be great if you want to do like an ar business card um and kind of show some things when you detect your business card um we also have face filters of course very well known uh nowadays you can do that from here as well and another one that it's called object it's not very well known but basically you have the this app you can use from apple to scan a real life object and you would use that as an anchor so perhaps in a museum you would use this to scan a statue and then detect that statue and start an ar experience from it but in this um demo we will go and use horizontal which is pretty um well used anchor type and you can actually here select if you want to have some template content or not i'll just leave it disabled now but you can enable it if you want so with this one picked i'll say choose and it will start off this scene for me and it's actually okay if you select it horizontally and you want to change it later you can do so from here so that's not a big issue and if we start to look around um you can see we have the scenes tab or you can see multiple scenes you can create for example if you want to have more complex experiences you would want to have different scenes in our case we'll just use one again you have the viewport the 3d viewport that you can navigate around and of course this will make more sense when you have some uh 3d content but as in a reality converter you have this frame option so if you need to just have your model framed again um without you having to move around the viewport you can do so from here the snap option i will show you in a bit what is used for maybe more interested is interesting is when you click on the plus button over here it will open up the ar library this is provided by apple so you have some basic models that you can use in your experiences you can change them modify them as you want and use them and you can see here we have some basic shapes uh different kind of activities even food nature places you have many things you can try out and maybe directly use in your experience or maybe just use them as a placeholder until you have your own models imported and what is really nice is that each of those actually have their own set of properties because they were defined in that way and i'll show you what i mean by that so let's just add a few shapes into this scene added a cube i'll add like a spring um i'm thinking maybe to add a cupcake um there's also you can have like furniture pieces um and they just get added into your scene where there's space um there's also an interesting one called frame just search it i'll just pick this one as well and now let's just go through them and see what you can do with each one of them for example if i pick up the cube over here first of all you will start seeing those uh kind of cones with the error arrows uh and different colors and they all represent um an axis so for example x y and z if i select this one and i can start moving it up and down um on the x on the y axis if i pick this one i can move my object on the z axis and if i pick the red one it will be on the x axis as you can see when i select one of those i can also also get this circle which i can use to either [Music] change the scale of the object on that axis or even kind of rotate it and this is what happens uh when you can see it goes from 15 degrees and 15 degrees and that's because i have snap activated over here if i disable that one i can just go like degree by degree so it's up to you if you what kind of accuracy you want when you're making those modifications and now uh if you want you can look into this properties tab and here you also have um options to change the transform the things that we did now uh from the ui you can change them from here directly as well xyz the rotation the scale you can do it from here too then we have this um tab which is called look and it gives us some some customization option for example i can pick what kind of material i want this object to look like so for example i can pick gold change the width and height and you can see those are kind of some of those options are some are options that only make sense for a cube they won't maybe uh make sense for the bench over here we have the depth and even the the radius of this so this was kind of like specific parameters i could change for my cube over here same you have for a spring you can change different things over here the revolutions you can customize it very well um for the cupcake it's quite interesting i'll just zoom in a bit you have the style so you can have realistic or stylized or iconic and if we move on to the frame this is also very nice to use because let's say over here you have to configure it and pick your image so let's say i'll just choose to import an image i want to put into this frame i have this image so your frame will actually adapt and kind of the size and the the form of it according to the image you uploaded and then you have some options here like the style of it the size the mount uh different options over here you can make it you can actually make a lot of interesting experiences with with those kind of options and things you already have in the ar library and uh also let's just look at this couch selected you can also switch between realistic and stylized now i'll just remove all those test items and i'll add one to show you a little bit about the physics and how you can activate that on your objects so let's say if i go and pick activities and find there should be like uh yeah let's pick up the baseball and i can use the frame scene now so take this nice back to our object when i select this one and i say that it participates in physics i will change the motion type to dynamic and here you can pick what kind of material you want your motion to mimic so in our case we wanted to kind of behave like a rubber material and the collision shape if you're not sure you can also in our cases a sphere is pretty straightforward but if you're not sure you can pick automatic so that's fine and the way we can test this one i'll just make that this is kind of floating maybe 1.5 meters above the ground above the zero axis and what happens is if i play my scene over here the the baseball the basketball will actually fall and kind of have this bounce effect because it has it participates in physics and it has like a rubber uh material so we behave like this so this is pretty much what you can do from the properties tab depends a lot on what kind of model you have but kind of like a nice overview and now we can actually go and import the model we just converted so that will be my island in usdz and you can see it was quite big uh and probably i could have changed it from meters to millimeters but over here i'll just go and scale it down a little bit and i'll also make sure that it's center on the axis again i'll kind of frame it i'm a little bit closer i can zoom in and this is how it looks right now in reality composer and now that we had a look at the what you can do in uh in this uh tool there's one more thing we call behaviors and this is actually when it gets even more interesting and you can customize your experiences with behaviors and over here on the left corner uh you have the plus button which will open up a menu here you have some predefined behaviors from apple like tap and flip tap and play sound and so on and you can also add your own custom behavior let's uh just test one that is already here so let's maybe pick tap and flip and you can see it has a warning sign and that's because every time you add a behavior you need to specify for which object that behavior is so in our case we need to go over here and say the affected choose the affected object i'll just click on it and say done that's fine and the same will happen with the action um sequence over here we have an option of emphasize i'll click on this one on the model click done and there are some customizations here from apple i'll just leave it like that um and if you want to preview this action you have this preview button over here so i can just press it i'll just zoom out a bit so we can see what is going on uh so you can preview this specific act action you can also preview the whole action sequence in our case we just have one but you can do it from here as well let's say we want to add another action so we can have actually a sequence i'll just keep it emphasized as well pick the object make sure but i'll say i wanted to um instead of flip like the first one i want it to spin and then style a basic style you can just experiment and see which does what and now i can preview this one so now if i press play over here it will actually play the whole action sequence and the actions will go from left to right and they will go one after the other if you want them to happen in the same time all you need to do is select one of the actions and drag and drop it on top of the other one so now you can see they are connected which means that if i play the action sequence now they will happen in the same time over here you can also select this option to loop so your action sequence will just go on and on and on and if you want to preview your whole experience you can play um of the button over here but now you actually need to trigger that action that behavior so we know that the trigger is a tab so i'll do that and it will trigger my behavior now let's remove this one and add our own custom behaviors so i'll select custom and now i have to pick the trigger i want to have a trigger that will let's say scene start but you can see that you have tab you have scene start proximity to camera collide and notification will use scene start you don't need to use an object that is effective for this one because it's just kind of like a general trigger and then for the action sequence uh you can see here what options you have but i want to play ambience now i have to pick the no actually this one i have to pick the audio that will play and you also have a library of different audios you can use so let's say i'll use this forest floor this should be fine so now when i start and i play my scene because it is triggered by scene start it will just go and start the the ambience sound now i want to add another one you can add as many behaviors as you want i'll do again a custom one and i'll say approximately the camera i'll pick the object and one meter let's say two meters you can select the distance where you that will trigger this proximity to camera um and the action sequence i want it to be emphasized the object is already um chosen and i'll say let's say float perhaps 10 seconds in a basic style it's always good to go here and rename your behaviors so just to keep everything organized um so let's go ahead and play again this will start the ambient sound and when i get closer to the camera it will start floating and you see that the actually the ambient sound um kind of finished so i will just need to make sure that i put a loop here so it just goes on and on um but yeah this is pretty much like some fundamentals you can do in reality composer it's now it's all up to you and go and use your creativity to do some interesting things um and kind of to to now to use this experience you can go and export it but also just to mention that you should go into preferences and make sure that usdz export is enabled if you want to export it to usdz and also from here you can change the unit so maybe if you want to switch to imperial unit you can do that from here i will just actually export it into um reality file because i want to use it for ar quick look on my device usdz uh it's kind of like more general format you can export even better if you want to use it on web applications you cannot use reality files on web so i'll go and export the whole project give it a name and the last thing i did was to airdrop this reality file into my device so i can test it out i just saved it into the files on my device and open it up so you can see here how the experience looked when i got closer it started floating exactly how we set up the behavior to be thank you very much for joining this talk on reality converter and reality composer i hope it inspired you to create some cool ar experiences

Channel: iOS Conf SG

Views: 1,198

Keywords: ios, singapore, apple, iphone, ipad, iosconfsg

Id: K9ditjo7nLs

Length: 34min 56sec (2096 seconds)

Published: Wed Jan 27 2021