Real Human to Metahuman Photogrammetry Workflow

hi everybody my name is nick justician i teach virtual production at drexel university and in this video we're going to take a walkthrough of the process of getting to hear a fully articulated mend a human starting from here you know an actual human being in front of a camera so the process we're going to use is a technique called photogrametry and i'll be using the capturing reality software reality capture and reality capture is owned by epic and it's a very inexpensive approach for processing photos and creating 3d models from those of course if we can photogrammetry model ahead then we can use the resulting model in unreal engine with the metahuman identity plug-in to generate a a really nice likeness in uh metahuman creator so let's start with the photography session itself in general with photogrammetry you're going to want to take a lot of photos and so it helps to be able to browse them i'm using a software tool called xn view mp this is uh open i'm not sure if it's open source but it is free software so i'll include a link in the description for that if you'd like and uh in this session we shot about 100 photos going all around my head so i'm sitting still with his blank face as possible um this was done quite a few years ago actually um at the time we were using address off metashape instead um but the uh the overall principle is is the same today and so thankfully i have these photos and can still uh create new 3d models with it so one of the things to note about this session is that um it is being done in the studio so you can shoot these photos with really any imaging device right so you can use a cell phone if you want and you can use natural light but for this shoot we really wanted to optimize this as much as possible so we are using a slr camera these photos are 24 megapixel each 6000 by 4000 i think we're shooting with a lens set to about 55 millimeters and um in this studio setup we actually have uh four different lights set up so here in the front view you can see there's two highlights in my eyes and so there's two lights in front of me and then on these off angles you can see that there's a pair of matching lights uh behind me and all of these lights are strobes so they fire a single bright light as a flash of light uh they are uh the light only lasts for a split second like a thousandth of a second or even less and as a result the photos have virtually zero motion blur whatsoever even with a handheld camera and so that's one of the big advantages of using a strobe you can see here in the background i mean the room looks dark it was actually fully lit it's just that the camera's exposure settings are matched to the strobes and as a result the natural light really barely registers on the camera and so strobes can be really helpful for doing photogrammetry of people because you're going to have to walk around a lot to get a full-sized human photographed enough times again i think we shot around 100 photos here and so the the strobe duration actually eliminates motion blur so i think the shutter speed here is like 200 of a second maybe a 250th of a second and that's really just a as fast as you can go and still be in sync with the strobes and uh we also used a uh narrow f-stop on these and i can't remember if somewhere in here we have that actual data i think it was like f11 um here's exif data here and let's see if we got all right there we are so exposure time uh 250th of a second f11 yeah and so there you go there's some of that extra information about uh these photos and so uh another thing we're doing here and this is a little bit of a more specialized uh set of equipment uh but you it's readily available for most photography stores and we're what we're doing here is using cross polarization so on every one of these lights is a sheet of polarizing plastic or gel j g e l gel so let's just say that this gel on this light is polarized vertically um all three i'm sorry all four of the lights have the same gel on it they're all polarized in exactly the same direction and then the lens taking the photo also has a polarizing filter on it however it's polarized in exactly a perpendicular opposite direction so if this is polarized vertically the lens polarizing filter is polarized horizontally and what this helps do is knock down the specular reflections so you'll notice that there's very little shine on my face in these photos maybe some extreme uh angles are giving us a bit of highlights and of course you know the gels are not 100 perfect i mean we're not using a calibrated optical setup but you can see that there's very little reflection or shine on my face for the most part so this really helps with both the texture being able to grab the texture for human skin but also for photogrammetry because it won't get confused by lots of different changes in highlights in my face as the camera goes around me so i'll do another video later on that shows a little bit more about this in fact these days i'm typically shooting with a ring flash so instead of using multiple strobes in the room uh single flash it's actually wrapped around the lens of the camera and that's cross polarized with the lens opening and um that gives us you know a nice little uh cross polarization very diffuse look all right so the only other thing of course to point out is that the photogrammetry series is done by taking photos from several different angles and so the the photographer walks all the way around me all the way around the back comes around the other side and then we offset vertically as well so uh to help enhance the the three dimensionality that's being captured the photographer started out at one level and then crouched down lower for a second orbit all the way around and then we did a few close-ups of my eye just to be able to get a good look at my uh eye textures so uh so that's the photograph session itself so the next step is to go into a software tool that converts these photos into a 3d model so let's take a look at the reality capture software so um this is epic software for doing photogrammetry and it's a really simple process in fact i'm going to use mostly default settings so just starting the app and i'm going to go ahead and bring in new inputs which each input is just a photograph so i click on the new inputs here's my directory of photos and i'll start from the first photo just hold the shift key and select to the last photo of my head now there's a few photos in here where either there was a strobe misfire or the camera wasn't quite pointed in the right way so i'm just going to control select those so this one there was no strobe another one without the strobe here the camera is pointed down in my hands and i think there might be one more that's just not particularly useful um maybe i'll just eliminate this one here and also this one where the light in my head really kind of overlap and so that um let's just eliminate those two all right so uh for the most part the rest of the the photos should be pretty viable so i'll click open on that and then we should get all of those photos in here and so we should be ready to uh process those so we just click on this three view yep there we go so here we go we have all of our photos visible here um i'm just toggling between single view and three view et cetera so here's the photo that's selected for example and this will be my 3d model once we get to process it so i'm just going to go ahead and align images and i'm just actually going to run this full process here so we'll uh let reality capture do its thing and i'll just uh fast forward so you can see how this processes okay so the process is finished it took about just under nine minutes to complete on my computer of course your results are going to vary depending on the speed of your computer the number of photos you used the quality of the photos the resolution etc one real important thing to highlight by the way is that never ever ever ever ever should you crop or reframe the photos that you captured from the camera the calculations here are absolutely critically based on the original relationship between the pixels at the center of the lens etc and so any cropping is going to completely ruin this calculation so you can you know convert from raw format to jpeg or maybe even make subtle color adjustments and i wouldn't necessarily be recommending color adjustments but super important is that you do not crop or reframe at all the photos should be the original framing straight out of the camera so with that let's uh refine our 3d model and get it ready for unreal i'm just going to switch to uh the oneplus one layout here uh right mouse button lets me orbit a little bit middle mouse button doesn't really do anything left mouse button sliding i can do with some more sliding with two mouse buttons okay right mouse button all right so um you know it's a less than perfect model we've got uh there must be something out here that's uh been solved for the the top of the head's a little awkward you know the the collar is going to be a distraction really i want to get this down to just the head that we're going to use for the modeling in unreal with the metahuman plugin so what i can do is go over to here and i go with my view menu let's go to tools and choose lasso and so this will let me select different geometries so i'm just going to left click and drag and select anything and everything that might be back here and then i'm going to filter the selection so that'll remove any geometry that was inside that lasso that i don't need so now i'm going to do the same thing here at the top of the head i think i'll push forward with my mouse wheel and just kind of trace along here and select as much of this as possible i think i got it right right mouse click let's go ahead and filter that selection there's a couple spiky bumps up there not critical that we eliminate all of this stuff it seems that the plugin is smart enough to recognize some things this is two mouse buttons select this geometry and filter so you can see that um again when i'm in lasso mode the left mouse click uses the the lasso itself and so if i want to uh kind of pan and move around my scene i can hold both mouse buttons down to do that right mouse button still orbits still some spikiness up here and i'm just going to go ahead and lasso around that and filter that out so another thing i can do to eliminate some of the excess is to adjust this reconstruction region that's around the model so i'm just going to go ahead and set the reconstruction region and it gives me some controls to alter that and each one of these little controls here at the faces let me compress this and i really just need this around the head and give a little extra space around there there we go so that's fine so that'll be our reconstruction region and let's do a set the ground plane and this is going to let me rotate the model so i want the nose facing forward the best view to do this would be switch from perspective so i click on the view option go to top and now the top of the screen in this top view is the front so this way i'm going to rotate so my nose is facing forward i can get in here and move left and right a bit and let's take a look at a side view and this is uh pretty good so from top and sides we're looking in the correct way and again i could choose front and i should be looking at my face wonderful and let's go back to perspective mode here and uh let's go back to our tools and choose the lasso again and we want to get rid of i'm just right mouse clicking to orbit uh two mouse buttons to move i want to get rid of anything that's like below my neck and even some of the shadow and all that so again lasso is selected so i'm just going to drag select around all here and then click filter selection okay so there we go we have a pretty complete head with very limited excess geometry looks like there's something down here just select that filter and we should be pretty good at this point so with this i'll export this model as an fbx so i'll just go to the capturing reality logo and choose export i'm using the autodesk fbx format and then finally give it a name i'll call this demo head vo1 and out it goes i'm going to keep all the defaults and say okay as a quick side note my license for reality capture is set up for unlimited processing so i can process as many photos as i like at whatever resolution and i don't have any uh you know per image charge related to my exports if you are using a price per image license then you'll be charged at the time of actually using the data for export and such so this particular example again i have in this case like 80 let's just say 86 images and they're 24 megapixel each so the cost of this model you can calculate it in advance uh there is a capturingreality.com and and price per image or ppi credit calculator so um you can go here and i'm going to calculate based on the number of images in this case it's 86 and the megapixel per image is 24 and so this model would have cost about a dollar 30 in us dollars so it's just something to be aware of it's very nominal cost given the resolution and quality of the output but it's something to factor in and it's something that you can check in advance before you actually have to pay or anything like that okay export is done i'll close on that took about two and a half minutes and now we're ready to use this model in unreal engine so let's go over to unreal five of course you'll need that metahuman plug-in installed i'll have a link for the unreal engine demo of that it's a six minute tutorial it's super clear so uh no reason to rehash that so the plugins already installed for the purposes of this i'm just going to create a new folder to store this demo so i'll call this demo and then mh id for metahuman id and i'll go in here and then this is where i'll import that fbx model so i'll just right click and choose import to game and look for that demo fbx it should be in here demo head right here and open that so this will bring in that 3d model we'll just click on import with all the defaults okay imports done took about 30 seconds these are just warnings and they're not errors so i'll let them go um one thing that i like to do not absolutely critical at all but i like to get rid of the specular highlighting in the 3d model so i go into the material just double click the resulting material and hold the s key down and tap the mouse once the left mouse button once gives us a scalar parameter default to zero and feed that into specular so now there's not going to be any specular values at all so we shouldn't get any shine in the material here in unreal so we can close that and say yes i'm going to save that change and what else are we going to do here we want to make sure while we're at it we might as well do a save all and yes save all these new assets that were created and now we'll put the metahuman plugin to use as soon as the save is finished we'll just do a right click in our content browser and we'll go to metahuman since the plugin is installed and we're going to choose this blue icon metahuman identity that starts a new identity i'll just call this demo and uh mhid hopefully that is okay even though it's the same name as the folder and double-click to open up that window do an automatic login and then what we're going to do immediately is just click components from mesh that's all we need to do here i'll just type in the word demo it'll filter down to the assets and that we have and uh here's the demo head so select that and that brings that 3d model into our plugin while we're here i'm just going to go ahead to body and choose the medium male i actually have more weight than this here but you know that'll do and then i'll click on neutral pose and we'll want to set up our neutral pose according to the tutorial it works best with unlit i've got my mesh selected i'll tap f to fit alt click and drag here to face the uh camera per the tutorial we'll change this field of view to 20 so we get a less wide angle distortion zoom out with my mouse wheel here try to fill out the frame we want to make sure we don't get too too close or too far away we want to try and maximize the use of our frame so there's a little space at the top little space at the bottom everything's blank and we want to try and make sure that we're looking as straight on as possible and i'll just go with this and we'll go ahead and promote the frame for tracking and track this frame with that tracked we're ready to solve for our metahuman identity because again i already selected our body type so we'll go ahead and solve and that only took a few seconds so in the a view we have the neutral pose in the b view we have the template mesh so we can look at that template mesh maybe hold l and rotate the lighting and that's looking pretty good so we're ready to upload this now when this gets uploaded to metahuman creator it's going to retain the name of the asset that we created here so i'm just going to click and save the asset this saves all the work that we've done here locally in the unreal project and then we'll use the metahuman to mesh the metahuman button here to send this up to the metahuman creator okay that took less than a minute and it says that metahuman is now available let's say okay and take a look okay so here in meta human creator is the results of our scan so we will of course need to edit this to to make it a usable human being so i'll go ahead to edit selected and while i'm at it i'm just going to go ahead and add a t-shirt and i like to wear black shirts so i'll make it a black shirt and put on a pair of jeans and some sneakers we'll make those black too and we can hit custom mesh to take a look at what's going on with the face i also like to make sure if i can try and get the skin tones eyebrows and hair as close as possible before playing around with the sculpting very much so skin i'll go ahead and assign and we'll make an adjustment here there we go bring a general skin tone in here that should be pretty close texture as well it actually dramatically affects uh the look of the face so you can see for example um like the lips do subtly change shape depending on which of these uh textures are selected so this is not a continuous uh blending uh it looks like a slider but in fact it's more like a selector when we're just selecting from a variety of different available textures i think i'm going to go in here and find something that's pretty close to that'll work all right so also like to put in some hair so go for head hair even though i don't have very much uh the tough thing is that you know there's an option here that's uh buzz cut and actually probably closer to being what my hairline is but i usually wear my hair a little longer so i'm gonna go with this admittedly this is more hair than i have but uh we're doing okay with that so i'll just use that um let's see here looking around what anything else oh um since we're here i'm just gonna go ahead and also add some salt and pepper to my hair and maybe darken up that gray a little bit there we are and uh there we go i mean so uh another thing that we can look at is um adjusting the eyebrows and i think i'll go with this and again add some salt and pepper to the eyebrows increase the roughness a little bit okay so there we go it's actually a little bit of a kind of looks like a younger version of me uh one other thing is the eyes will make a big difference so i'll go ahead and select the eyes and rather than a preset i'm just going to go ahead and [Music] choose an iris here i'm actually a mix of kind of a blue green and a light blue and let's get in a little bit closer let's see what we can do to blend these colors a little bit maybe more desaturated and gray maybe a little more green something along these lines so there we go um now we can still do some more of our meta-human adjustments so in the custom mesh panel right now we are set up for all regions of this being based on the scan so what's happening under the hood here is there are the various if i could blend you could see all these various uh metahumans that uh form up uh each one of these is like a zygote that um helps define the overall shape of a head and um there is going on here a blend of those plus then a deformation that's applying the uh any offsets from the uh the blend of metahuman zygotes versus the uh 3d scan so if we want we can say all right let's bring the top of the head a little bit closer to i'll just just go back to the zygote so that none of those spikiness on the top of that scan has anything to uh drive that head and again we can just see if there's any subtlety i'll actually stick with the scan for this another thing we can do is we can enable editing where we can actually go through more adjustments of this so i'm going to enable editing and it's good to duplicate and unlock so what this will do is make a copy of that original output from the plugin and in fact i'm going to go ahead and rename this and just call this vo1 and uh let's see what are we gonna do here so i've enabled editing so i can go into the blend mode and instead of blending here i'm just gonna go right to sculpt and this gives our metahuman sculpting tools control over what we're doing and so you know i could maybe make this nose a little narrower maybe pull in this these cheeks and uh jaw a little bit you know kind of uh more idealize myself for perhaps but you get the idea is that now you we can adjust the overall shape and uh get this looking the way we'd like and that's it at this point uh this is a fully articulated metahuman i can hit two to go to full body and uh this can be exported to unreal it could also be uh exported into maya for further animations and modifications so that is the overall process to start from a real live living human head and use photogrammetry to create a metahuman hope this helps until next time have fun

Channel: Pixel Prof

Views: 58,799

Keywords: metahuman, scan, photogrammetry, polarization, UE5, Unreal Engine, Virtual Production

Id: 8JVWGb9kBng

Length: 28min 44sec (1724 seconds)

Published: Sun Jun 12 2022