3DoF to 6DoF, Photogrammetry in the Metaverse | Masterclass by Eric Hanson

If you are an immersive creator with primary  practices in 360 Virtual Tour panorama   photography, 360 or VR180 Videography  and want to learn the next level of   immersive storytelling INSIDE or OUTSIDE  of a VR headset, this video is for you.  Photogrammetry, I consider it as  the building block of the FUTURE   of Metaverse. The Metaverse that you can not  tell which is faked CG or real reality anymore.  In order to harness this kind of power for you,   we invited the TOP Photogrammetry expert - Eric  Hanson - to teach us how to do photogrammetry   professionally for Hollywood level VFX, for  in VR headset 360 Virtual Tour, VR experience,   and for Augmented reality. Using custom build  camera rigs - like what you see here - the 3   SONY mirrorless for interior HDR photogrammetry  in a low light scenario and the 5 SONY setup   for a rapid exterior photogrammetry where  you need to race with the SUN to get the best   result without shadow overcasting issue. Let's jump right in!  hi my name is Eric Hanson i was invited by  hugh to come speak about photogrammetry today   photogrammetry is a kind of a big technical  word but it's actually the next big media   form for creative use and it's something that  i became involved with probably eight or ten   years ago and about five or six years ago it  literally changed the path of my career and   this is uh all i really want to work in today my  original background is in visual effects i worked   on films like the fifth element day after tomorrow  a bunch of big tentpole films here in la working   for all the major studios and facilities and so  my background is that of an environment artist   um i don't do character work i do the worlds that  characters live in and from design to execution   of these different worlds now i am on faculty  at the usc school of cinematic arts leading   uh cinematic vr curriculum and then i have my  own practice on the side called blue planet vr   so what i would like to do today is just kind of  talk a bit about uh what makes photogrammetry so   special and why it's the next big revolution  i believe and how you can practice it it's   actually become very accessible compared to  where we were say even eight or ten years ago so photogrammetry again it's a technical term  but really it just means capturing the real   world in incredible fidelity and resolution  for me as a visual effects artist it obviously   allowed me to rather than kind of paint things  fictitiously as matte paintings or 3d environments   but actually base it on the  real world so it has obvious   implications for extending your practice and  visual effects above and beyond that though   there's this whole creative side that is truly  untapped and that's something that i've worked so   hard in the last number of years just to refine  the acuity of the real world as i call it the   the uh the veracity of reality right and but  above and beyond that we can now take reality   and begin to manipulate it and warp it and do  use it for creative ends and that's something   i'm just now beginning on so it's got immense  potential creatively or for practical purposes   obviously it's a great tool for mapping and kind  of looking at the geospatial world but again i   think you'll be excited to see what creative  potential that it has as we explore this topic you can also look at photogrammetry as being  an extension of virtual tours panoramic imagery   which i've been involved with for decades is  essential use technology for that practice   360 video that i've created a lot of work with  as well is fundamental technology for filmmaking   and virtual reality but photogrammetry can extend  those practices in new and more powerful ways so   uh we'll introduce the concept of three  degree of freedom and sixth degree of freedom   3DoF as we call it is something that you  would do with panoramic imaging or 360 video   and again it's a fundamental tool for telling  stories or providing again spatial tours   of locations it's a can be very strong  uh especially with a proper narrative   but when we shift over to sixth degree of freedom  by having a full three uh three-dimensional model   or capture of your environment you get a stronger  neural kind of compelling sense of presence   and it's this remarkable sense of presence  that i think is the most powerful thing   about photogrammetry uh the first time uh  we took one of our photogrammetric models   brought it into an early vive headset with unity  i just had chills down my spine and was you know i   just felt like wow this is a turning point for me  in my career and uh i still feel that way working   with it six years later it's just a fundamentally  terribly exciting and powerful tool to use so one thing about photogrammetry today  which is rather remarkable we're at a unique   point in time where the technology has been  around in different forms for a while but   uh say five six maybe seven years ago a new  evolution of algorithm and basically gpu   computing and so forth is bring the con the heavy  computation involved to something that's very   accessible and what that's done uh is it's open  doors to again extending your practice in new ways   the other thing is it's been a grand  conjunction of technology as i see it   from gpu coming to the fore these new algorithms  uh drone technology and vr headset tech   and all this kind of coalesces into what  can be a very powerful experience for you   the other thing about photogrammetry is it's  now become used in ar you can see plenty of   examples on your phone where you can bring  in real world objects into your environment   you can also go to a site like sketchfab  or nera and see plenty of photogrammetry   very heavy models that are brought in for  study this could be used for education and   academic purposes as an example and then it could  uh be used to extend your particular skill base in   the kind of film and media work that you do and uh  again visual effects in my uh domain it's been a   huge tool and of course that now extends into the  world of virtual production with led walls which   is changing the whole face of film production  now and if you work in a real-time package like   unreal you're familiar with quixil mega scans  which again is photogrammetry based assets   and again that's a whole new kind of palette of  materials and models you can work with to create   incredibly imaginative worlds and that again  feeds in directly to virtual production so   the the range of where photogrammetry is  affecting different practices is very broad okay so another amazing thing about photogrammetry  today again speaking of uh you know the ease of   entry into the practice is that it really just  takes something here like a cell phone you'd be   amazed how many people do a remarkable work with  your phone and of course it's always nice to know   if you don't have your professional  kit you can at least capture something   on the road wherever you might be with a cell but  if you're serious about the field at all you'll   want to use professional dslrs behind me is a rig  let me turn this way uh five cameras and this is a   custom rig that i've developed to help me speed up  my captures of particular types of scenes and this   is based on sony mirrorless a7 series cameras,  to be honest a lot of photogrammetry is handheld   and the use of a pole that we'll talk  about being able to extend that with   drones or multi-camera rigs like this only kind  of increases the range of what you can produce okay so what we're going to do here  is look at a custom rig an example of   a custom rig for photogrammetry and it's  useful to develop your own rig oftentimes   for certain circumstances about a particular shoot   this i designed for a shoot the gobi desert  at a place called the miguel caves these are   kind of cavern-like spaces interior spaces that  have a lot of detailed buddhist artwork and   this is under special permission with the chinese  government we knew we were under a time constraint   we also knew there would be very little light and  we wanted to work rapidly and also in very high   resolution so for that uh for that uh particular  use i designed a rig with a very tall tripod this   actually this whole rig is compacted down this  will actually go much higher and with three   cameras so we can look at the top of the boom this  is using a nodal ninja carbon pole to extend and i   have it much much lower i could take this up you  know 10 12 feet higher with the top camera the   thing to note here is that top camera is facing  downward is panned down and you might notice all   three cameras are in portrait mode and the reason  for that is i extend my overlap between the three   cameras by placing them in portrait the top camera  again is shooting the ground surface the lower   camera is shooting up at the ceiling which had a  lot of artwork in this particular case and then   the center camera of course is shooting across the  room on the artwork on the uh the opposite wall   and again in the course we'll talk more about how  the methodology of an interior is done but again   this was designed to make kind of shorter work of  what we had to do also being on a tripod allowed   us to shoot an hdr stack for the dim lighting and  the high dynamic range of the shooting condition   and on top of that these all have wireless  rf shutters and then i have a cluster of   ganged shutters here and the reason you might  ask why don't you just shutter all with one   is that i want the authority or the uh what should  i say the uh the feedback of hearing each camera   separately that it got the proper stack so uh  this can be uh basically the control so i would   have this in one hand and again this is on a video  dolly uh wheel platform and uh so what i would do   is take a shot move it one foot take another shot  and work my way around the perimeter of the room   using this shutter so this gave me a really nice  overlap of full ceiling to full floor uh from one   move so in one revolution of shooting with this  rig i pretty much had the the space captured   one revolution around the space right so and  again i'll show diagrams in the course it'll uh   describe this a bit better um so this again i  wouldn't necessarily use this in an exterior   situation unless it called for it but anytime i  shoot interiors especially man-made interiors this   will be very useful especially having a flat floor  that i can can roll it around on what i'll show   upcoming will be another rig that's handheld  with five cameras and that's another way that   maybe i could work on uneven ground maybe more in  a natural condition with multiple cameras again   i could have shot one camera with this shoot and  just simply put it on a pole as i'll show down   low looking up holding it maybe a bit overhead and  again getting on a pole but again having all three   at once on a rigid platform support allows me to  work very rapidly so that helped and even then   this took hours and hours of capture so but  i was able to get one of the spaces done   in under a day whereas if i shot with one  this would be a two or three day operation   but that's because again i was working extremely  uh carefully to make sure that we had every   capture made because like i say as great a camera  as sony is on occasion the hdr uh stacking will   skip a frame so again that's why i split this out  and make every every shutter separate from the   other so i hear the three shutters being made if  i did them all together i wouldn't have had that okay well aside from looking like some kind of  strange religious exorcism device uh this is   actually a five camera photogrammetry rig that i  came up with concocted to help me shoot outdoor   locations again increasing the speed of capture  the um what i'm doing again is shooting in   portrait mode so i get heavier overlap between  three cameras this one's sky facing this one   direct and this one aiming down a bit more to  the ground and then side lateral cameras to give   me some oblique angles from the side and again  typically this would be shot with one camera by   hand and it's perfectly fine and you would you  would kind of recreate these positions manually   uh but again the reason for this is speed of  capture i can move much quicker now the as i   put this together i realized the weight of this  was a bit formidable so i put this rather geeky   little platform little belt device with an arm  so it's kind of a very poor man's steadicam   and but again takes the the weight onto my  hips for this rig so i can kind of angle it at   different angles as i walk and capture and again  this is placed together with shutters ganged in   this case i would probably put all these on a  single shutter i would slave them to a single   and i wouldn't worry about the hdr because this  would be hard for me to do an hdr capture here   unless i had sony a4 uh i'm sorry a7r4s where the  stack is fast enough and captured a handhold but   these cameras being different types i would gang  it to a single exposure and a single shutter but   but again it's a great method very quickly being  able to scan about a a an area so if i go here   i would take a shot maybe angle down rotate  about bring up at a higher angle and rotate   that way i'd get a nice sweep of a different  angles heights elevations and bearing angles   on the subject the other one i showed the three  camera rig on the tall mast on the tall pole   again is designed for interiors primarily this  rig i would use more on exteriors more natural   landscapes when i would want to move a little  quicker because again speed especially outside   outdoors like today on a sunny day  we're racing the movement of the sun   as we capture it's always a big challenge  in photogrammetry is trying to minimize   the passage of the sun throughout your capture  otherwise you're going to get stair-step shadows   and it'll still solve fine but you'll get kind  of an odd shadow characteristic so this is a   way i can go five times quicker and capture and  have less movement of shadow throughout the shoot so the use of drones in photogrametry is another  as i mentioned it's it's technology has come along   too and uh it's kind of an uh inseparable for  using handheld photography and the reason for that   is we can get the context of a site so rather than  go very small and just say a patch of ground we   can begin to capture a larger sphere of whether  it's an urban fabric or a natural landscape   but the drone allows us to get even a more  extended range i do a lot of work with gis   data which is geographic information systems it's  basically terrain data and that's another way you   can kind of extend it but again with uh fairly  inexpensive drones you can get a very nice way   to extend your capture a little bit further and of  course this is used heavily in the area of mapping   of which there's many photogrammetry based  mapping programs and mission planning software for   your drone that you may be familiar with other  kind of non-mapping packages like reality capture   have actually added that functionality in so  the uh again everything is kind of i guess   the point in a lot of this technology  is it's all overlapping and converging   and giving you just an immense amount of power  so we'll cover these techniques in the course okay so hugh and i are developing  a upcoming course in photogrammetry   and part of this will be for the new user so what  we'll do is we'll talk about easy entry points   for the field both technically and uh you know  practice wise um so the tools again one reason   we're doing this now is that the tools are very  accessible compared to what how they were years   ago i'm a 3d artist i have been my whole life  but these days you don't necessarily need to be   a veteran 3d artist to practice this we will cover  some of the open-source and more accessible tools   for instance reality capture just was purchased  by epic and is now an easy handshake into unreal   and the barrier of cost and accessibility has  come much further down so we'll also look at   an open-source package called mesh room which is a  another easy way to experiment with photogrammetry   and kind of understand how to produce it so in  this series we'll be covering both the application   of photogrammetry as well as the practice and  then the shooting methodology and then the post-production okay well what i'd like to do next  is actually show you some outtakes of blue planet   VR this is an application that I developed over  the last number of years i released it on steam   last year during the pandemic and just about a  month ago in may i release it on the quest after   a lot of optimization and additional development  so what i'm going to do is put the headset on   we'll record this and you'll be able to see  what i'm seeing so here we are in blue planet   so let's start by going into fallen roof  and i just touch the sphere and it'll load and what we see now is i'm in  the desert southwest of the u.s   we're in southern utah and we're in a very kind of  what should i say sensitive region of archaeology   and native american culture and as  i just slowly pan my head around   i'm looking out through the spectacular  canyon in what's called bear's ears   and as i slowly turn trying not to make you  nauseous i can see an orientation table and   if we look down there's a little platform you can  stand on and then you get a basic description of   what this site is but then we have a little bit  of additional information if i touch this map   we can see where bears ears is located in  the state of utah and if i look at this   route icon i can see a small model and this  is indicative of the area that you can explore   in the scene so here we're standing at this  lower end of that little uh cascade of rocks   and we can climb up to get in the higher reach  where we can see some scale figures so again   giving you a preview of what you can explore the  question mark is what i call the deeper story   and these are generally environmental  issues or kind of sensitivity issues and   ways that you can help protect these sites and  some background information on where you're   standing and then we can go back to information  or we can click our trigger finger and dismiss it   now that i've dismissed it i can look at my  hand controller and i can see i'm going to use   the a and the x button to teleport and this  is just a way for me to kind of move around   a little easier so as i hold down one of these  buttons i get a laser pointer where it goes red   is not available for my travel but green is so  i put in these small little markers to help you   so um so i hold and release and  then i move myself to that marker   and now we're kind of moving our way up much like  if i was climbing up this slope in the real world   so now we're almost under the ceiling let's go up  to the top and now we're entering the main site   the fallen roof these pieces of stone fell  apparently within the last hundred years and   has revealed the spectacular pattern of weather  weathering up on the ceiling but these are what   are called grainy areas these were storage  for food for the ancient puebloan culture   about 800 years ago so this is a archaeological  site that with proper permitting you can visit   if i go over here this is an informational kiosk  that says look overhead for hand pictographs   and i can see here three hands which are rock  art of the original inhabitants but i can look   out on the site let's move out to the edge here  right about there and i can see these floating   glass spheres and this is another beauty of  photogrammetry is i can get novel points of view   that i wouldn't be able to necessarily without  a drone so what i can do is just simply teleport   onto that sphere i'm now kind of standing above it  it gives me an alternate point of view of the site   and let's go a little further and now i've got  a lot of air below me and i can kind of turn and   look at the site from an aerial perspective  that you could really only get from a drone   um and again that's nothing that i shot from this  vantage point it's just recreating the environment   you can see it in different ways  so again i'm going to return home   and i'll just show you one more  and uh let's look at uh borobudur and there's a fair amount of cultural heritage  sites uh shown in blue planet this is one of   them this is a uh the world's largest Buddhist  temple in Indonesia this describes it a bit   we can see where it is on the island of java this  is the little panel that we're going to explore   and here's a cross-section of this describes  the actual architecture of the temple   and the information let's click that off one thing  we can do is highlight text descriptions of each   sculpture and i can understand maybe get a  little bit more insight into what it is i'm   looking at here but i'll leave that on and i'll  traverse down a bit and then this says entire   temple model i'll click that and here i have a  three-dimensional model a Borobudur and we can   see where we're standing here in the red region  okay i'm gonna teleport beyond that and let's   look at that now not from photogrammetry but from  aerial drone panoramas so if we touch this sphere now we're seeing the temple shot from my  drone up above and looking in and we can   see sculpted panels is where we just were and  this was actually shot on Chinese new year so   this is a tremendous amount of visitors on  the site but an amazing location you can   look out and see the volcanoes in the distance of  Indonesia and of course the summit of the temple   is one of the most poetic locations  let's go there and look at the dawn view and this is a popular thing to do at the site  and exquisite lighting and exquisite surround   so again this is a 360 panorama but this is  a good example of three doff this was not   possible to capture in photogrammetry at that time  so uh but a 360 panorama could be so again these   two can kind of live together in ways to help  tell a broader story so i'll return back home and i will show you one last one and  that is a hang gliding experience   so let's look at Iceland canyon and this was uh from a um specially  chartered uh helicopter flight in iceland   and now we can uh again kind of see  where this is this is near the famed uh   ice lagoon and again there's the model of what  we captured aerially and let me dismiss that   and you can see we're standing on quite a  precipice here and uh but as i look at this now i   have two new controls press name and press to fly  so if i hold this down both trigger fingers down   i'm gonna run right off of this cliff and launch  my flight so i'm going to press and hold and i   get an icon of the glider and as i look back now  you can see i'm actually in flight and as i tilt   this icon i can actually ascend straight up or  i can dive down or maybe i can fly up the canyon   and this is really just an aid for you to see  your orientation but you don't even really   have to look at it you can just tilt your  hand controllers to get the flight dynamic   that you're after so this whole canyon is a  complex of different waterfalls and little slot   canyons and you can take your time and explore  all this with this interface so let's go back home and i could keep taking you  through so many of these but um   again uh this is just an example of  what i'm doing with photogrammetry   which is a way to share the feeling present at  these uh remarkable locations and again uh i think   the interesting thing about photogrammetry  i'm actually gonna take the headset off   is that you'll find uh ways to integrate it into  your own work and that's the the really i think uh   powerful and inspiring thing about photogrammetry  is just the it's the world of what-if   is what i always say about vr and so i  hope you'll join me in our upcoming course all right well i'd like to close here by talking  about this course that we'll be offering we're   just now putting together and it's going to cover  a myriad of topics it'll be a few hours in length   it'll cover the application the use of  photogrammetry from 360 filmmaking to vr to   visual effects and then we go into shooting  technique both in interior exterior object   materials and we'll do case studies for each  one of those and we'll do some field work   and show you how that all is done uh we'll  review hardware and software software we're   going to be using in the course will involve some  photogrammetry software such as reality capture   we'll also show mushroom uh we'll talk about  color grading we'll talk about hdr tone mapping um   we'll use uh sns pro for that um and then we'll  also of course then go into 3d applications both   at the high level with maya and at a simple level  with mesh mixer and show ways for you to do the   work even if you're not a seasoned 3d artist and  then we'll go through the entire post-production   process and then we'll talk about the  final models the assets that you'll get   and how to use those how to integrate them into  your 360 footage or how to bring them into a   real-time engine like unity or unreal and lastly  how to deploy them to a vr headset and so we'll   show you kind of the a to z you know soup to nuts  uh production pipeline uh all the way through and   again i'm going to the focus is to make  it broad and to keep it to where you know   whatever your background is you'll be able to  think about ways to integrate so see you there you

Channel: Hugh Hou

Views: 41,613

Keywords: photogrammetry, photogrammetry tutorial, photogrammetry masterclass, hugh hou, creatorup, eric hason, reality capture, meshroom, photogrammetry camera rig, photogrammetry camera, photoscan, virtual reality, metaverse, augmented reality

Id: sfv5kwngq-Y

Length: 30min 9sec (1809 seconds)

Published: Fri Sep 10 2021