Turn AI Images into 3D Animated Characters: Tutorial

hello welcome back to another prompt news video  today I'm going to be showing you how to get   your AI character which you might have created in  mid-journey stable diffusion or any text to image   generator and take this character and make them  into a game ready character that you can control   the facial expressions using just an iPhone  hopefully in this video I'm introducing you   guys to technology or software you do not know  exists and when you've watched the entire video   you'll know a few pieces of cool software that are  absolutely free that you can go and download now   and start tinkering with I have a written format  of this tutorial on the promptmuse.com website   because sometimes it's easier to look at that for  references than scrubbing through a video before   we start the tutorial I've just put together  a quick visual representation of the tutorial   before we go ahead into it and also this will show  you the software that you will need to in store   I'm using my redhead.c kptpt model to create the  consistent images of my character if you do not   have a trained model you can use this tutorial up  here to train your own or you can use my one for   free which is down in the description we're then  going to take those images into blender and use a   plugin called face Builder which turns images  into 3D mesh with correct topology as well as   automatically unwrap texture map of these images  from stable diffusion then going to be turning   that mesh into a meta human character and then  using it in Unreal Engine I'll also be showing you   the ACT we'll be using to control our character by  just using our facial movements because I'm using   a plug-in that when you use reference images it  actually snaps the mesh to the reference images   which is really cool which means I don't actually  have to do any 3D modeling at all but I do need   consistent headshots of this character so I need  a front headshot a side headshot and a perspective   headshot which is quite hard to do with prompting  what I did was take a photo of myself on the   trusty iPad I then just took the background out  with holding down and clicking on it that's a new   Apple feature and it removes the background for  you automatically I then just sent those files to   myself via email or Google drive or whatever and  got them on my computer and then went to the image   to image tab over here and then uploaded my photo  one by one in the image to image Tab and then I   wrote in my prompt that I wanted a beautiful zwx  person so beautiful usually gives me a symmetrical   face which I want and I want her to be in a photo  shoot so I want the lighting to be diffused I then   also said I wanted her hair up because I wanted up  out of her face down in the negative I made sure   that her mouth was closed I don't want any teeth  showing in the image because I'm going to be using   this as a UV W texture on my 3D character and yet  the teeth are going to be 3D modeled anyway then   I set my sampling to a hundred used Euler a my  input was 5 12 by 512 so my output is 512 by 5 12.   and then set my CFG scale to around about 19 so  it's adhering to the prompt and then my denoising   strength to about 0.5 to 0.6 and this means if  you set it to zero the photo of me the input file   would not change at all so putting it about 0.6  means it looks more or less like my character I   don't want it to look like me whatsoever I'm using  my face just so it can reference my pose and my   facial position I then created a seed on the first  run and if I like that seed I then hit the ReUse   button over here and use that seed for all my  other images which created the consistency though   this took about 10 minutes all in all to create  several headshots and that's it we have finished   in stable diffusion and we can now crack on with  the modeling phase and to model this character I'm   going to be using blender download link is in my  description and as I say it's free for you to use   so let's head over to blender so now we're going  to take those reference images and turn them into   a head mesh and to do this I'm using a plugin  called face Builder it's not free but you can   download a free trial to try it out for yourself  once that's downloaded we just want to clear up   our scene by selecting and deleting any elements  that are there and head over to this small arrow   on the right hand side of your screen if you click  that you'll come down and you should see a face   Builder tab here give that a click and we want to  create our new head model look you can see that's   a pretty generic male head model which is no good  for our project so we're going to get our images   in and turn that into our character so to do this  come down to views and then click on ADD images   navigate to the images that you just created from  stable diffusion and just select all those images   and click on open images and you can see down  here we've got all our perspectives here so if you   click on the image this is where the magic happens  for instance for our front profile image you can   see if you click on a line face the neural network  of the AI workout where the faces according to the   mesh and creates you a 3D model of those images  this is pretty cool but we're going to have to   use every single profile just to make sure it fits  to our character another thing this doesn't work   perfectly so you might have to go in for instance  and left click to place a pin and then pull the   mesh into position where you want it it so you  can do this in absolutely every viewport and it's   important just to get those very small details  correct if you find you've placed a pin in the   wrong place you simply right click on the pin and  that deletes it to come out of that profile view   just press escape on your keyboard and you can  now see our male head mesh has transformed into a   female head shape and that's from one perspective  so we're just going to go into another perspective   I'm going to repeat that process again I'm going  to click on the Align face button and that will   snap our mesh to our reference image do not  add too many pins but make sure that the detail   and refinement is there so I'm just going to go  through that very quickly and go through and pin   all the points that I want to for my character  so once you're happy with your mesh just come   out and we are now going to create the texture  to add the uvw texture to this character just   come to the textures panel and you can set the  resolution here I'm doing 4096 pixels by 4096   so that's quite a big uvw mat and I'm going to  be using the UV map butterfly so that's going   to flatten the face which makes it easier later on  to come in and add details in Photoshop I'm going   to Simply click on create texture and I'm just  going to take the images from the front profile   and both of the side profiles the reason why I  don't take it from the perspective is because of   overlap issues within the texture so these three  profiles are going to be absolutely fine to create   the texture from so we now have a texture map  for our character now you'll probably see here   there's a few black spaces and this is where we  haven't got the perspective for that view that   does not matter because we're going to be taking  this character into something called a meta human   and we're going to be using their skin textures  and then combining this later on so now we just   want to export that texture as a PNG because  we're going to be using that later so find the   export button under textures give that a click and  they're just going to save mine on my desktop here   so you can see it there we also want to save the  mesh because we're going to be turning that into   our meta human to just go up to file and to export  and Dot fbx and then just give it a name and over   where it says path I'm just going to select that  to copy and then where there's a little image to   the right of it with a box just give that a  click and you'll see that the texture is now   copied into that file we then go to export fbx  and again just save that anywhere we are now done   in blender and we are going to move into Unreal  Engine so see you in a second open up your epic   games launcher you're going to need to make an  epics games account as well again absolutely free   once you have launched the launcher go to launch  so this is where all your unreal projects live   and we're going to set up a blank document and to  do this if you go to film video and live events   from the left hand panel and then click on blank  you then want to give your project a name I've   actually Switched Off starter content and Ray  tracing because that will speed the load times up   I'm going to click on create so this is what your  screen will look like when you've loaded up your   first unreal project what we want to do is install  some meta human plugins so if you go over to   settings over here and click that and come down to  plugins and then from here you have a search bar   and we're just simply going to type in metahuman  you will then see the meta human new experimental   plugin we just want to check the box and then we  will get a message saying do you want to install   it and yes we do we will then have to restart your  project and if you click on restart now that will   do everything for you and restart without you  having to touch anything which is fantastic   and that will open up our project with our plugin  now installed so the project opens again with our   plugin folder which we now want to close down if  you come over to the bottom left hand corner of   your screen you'll see something called content  draw so this is where all our files internal of   our project will live so this is where we want to  import our mesh that we just created in blender   come over to your content drawer and anywhere in  this Blank Space right click and go up this pop-up   menu to import to game just navigate to where  you saved that dot fbx that we just exported   out to blender and click on open you will now be  presented with these import options you actually   have to do anything here so just click import  all and that will import your mesh if you get   any errors do not worry just press clear and move  on so now you can see the elements from our SBX   file have successfully loaded in we need to make  a meta human identity solver and to do just right   click anywhere on the content drawer where there's  a space and come up here until you see metahumans   so if you installed that plugin correctly you will  have this and then come across from metahuman and   you want to select meta human identity and you  want to give this meta human identity a name this   name is going to be the name of your meta human  so I'm going to go her red head cut one so once   you name that file we would then want to double  left click on that file this is our meta human   identity solver this is where it's going to create  the blend shapes for our animations automatically   what we want to do is come up to components from  mesh and click on that drop down and we can see   our mesh here so yours will be named differently  to mine but just click on that mesh and it will   be loaded in you want to position the screen  so we're looking face onto the character now   this can be quite tricky with the controls you're  going to have to use a combination of the middle   Mouse button and the right Mouse button to  get her in view we then want to come over   and just make sure neutral pose is selected so  the next thing we want to do now this has become   available click on promote frame we also want  to right click on this Frame zero down here and   click on lock camera so it just ensures that we  don't move our position now the next step is to   click on track active frame and this is going to  pinpoint all those positions of the eyelids Mouse   Etc now you can see our trackers have now been  placed on our face they're not in the correct   place you can move them but that looks good to  me you can look at each elevation of the face   and make sure that it's okay I'm now going to go  to the penultimate step which is the meta human   identity solve and click on that so now that is  done we want to come over to body so just come   down here and select a body that's most suitable  for your or human but you've selected the body   come up to a mesh to metahuman and that's going  to transfer this model to our meta human Creator   it might ask at this point to sign in so do that  so you will now get a message saying your meta   human is now available in the Creator and grid so  we just click OK and we don't need this project   anymore so if you come out of there and then go to  window and then come down and to Quick thoughts or   Bridge has loaded up come to the left hand side  panel and underneath meta human presets you want   to click on my meta humans this will load up  all the metahumans that you have created so   this is where the identity solver that we've just  completed has exported our model to so we're just   going to click on that and it will open up an  additional option here and we want to load it   into the meta human Creator by clicking that  blue button alternatively you can access your   meta humans by going into this browser address  and will take you to the same place once the   meta human creator has loaded up you might see it  a bit laggy or slow this is solely dependent on   your internet connection and how many people are  hitting their servers also one thing to note you   only get an hour each time creating this and  I think they've done this to limit how many   people are on their servers if I click on this  tile here this is the character that we've just   saved from Unreal Engine using the meta identity  so if we click on the character and then click on   edit selected I'm going to click stop on the idle  animation because that will just slow everything   down we want to add some makeup skin and hair  and make her look all fancy first thing we need   to do is go to custom mesh up here you can give  an influence over the mesh how much you want it   to be generic metahuman or how much you want it  to be like your mesh I can see I've got an error   here with my lip it's not symmetrical and it's  sort of cold up so we're going to go in and fix   that and I'm going to select the mouth section  here and down here we've got region influence so   it's just going to influence that tiny bit and not  the area around the mouth because that is true to   my character so I'm going to bring the region of  influence down and you can see that has fixed my   error once you have finished you need to click on  the enabling editing if we click on duplicate and   unlock this will save a character before you've  locked it in and so you can come back and edit   this if you need to so if you come down to skin  we can now assign her a skin color and we can go   into all these options makeup teeth now just  a point here this is very limited to what you   can do the styles that you've got if you've got  particular freckles or moles or makeup on your   character and you can't do it within this do not  worry because we have got that uvw map still that   we can apply to our character in unreal engine  in the meta human Creator you can sculpt your   character and refine their features further if  you see this floating panel up here this is your   sculpting panel if you click the middle Arrow icon  you simply just click on one of the dots with the   left Mouse button and drag and you can see you  can change the look of your character this is   fairly limited so it's best to get the base mesh  as close as you can to your character before you   bring it into metahuman but if there's something  that you can clearly see that you need to fix you   can come in from any perspective and refine your  3D character further so I just sped up the process   of me adding hair and eyebrows because it's so  simple you don't actually need a tutorial on how   to do that a program that you may find helpful  that is absolutely free is called Pure ref and   it allows you to bring in your reference images  and drag them over the top of your character with   pure ref you can then overlay your reference image  over the top of your meta human and then you can   right click and go to mode and put transparent  over Mouse and that means that you can now lock   it in and then go into sculpt mode and then  change the point to exactly where they are on   your reference image so you might find that useful  links to that program and again it's absolutely   free is in my description below so we have finally  finished all the modeling that we need to do to   our meta human so now we need to bring her into  Unreal Engine and you could start a brand new   project and drop her in there but the lighting  is not going to be very good and I suppose you   probably want your meta human to look good to  do that you can just download a project that's   already created with preset lighting just open  up your epic games launcher and at the top you   should see Marketplace if you click on Marketplace  and in the search products just type in metahuman   caps why not and then you'll see a meta-human's  lighting and you click on that and you can just   download that so what this is is a bundle of meta  human Creator lighting presets so this plugin was   created by the award-winning cinematographer Craig  Fraser so they have gifted us these presets for   absolutely free which is very cool once that has  downloaded launch your Unreal Engine and you'll   see it in your project file so once you've  launched Unreal Engine you'll see that meta   human lighting preset there so we're just gonna  open that when you open up your project or your   lighting preset the first preset will look like  this and to access all the additional presets   just go down into your content drawer remember  this is where all your assets are contained and   if you look down you've got all contents and  then within a Maps you've got all the different   lighting presets here so I'm going to go for the  lighting preset Moonlight and that gives like a   nice dark mysterious Moonlight lighting setup  now all we are missing is our meta human all we   need to do is go up to the top horizontal toolbar  and click on Windows and then come down and we're   using that quixel bridge again and from the left  hand menu click on metahumans and then underneath   metahuman presets you'll see my metahumans this  is my redhead YouTube Tut character here and if   we click on her little slate here you can see  that we have an additional options here to the   right of the screen we can change the resolution  of the character we want to download so if your   computer's not particularly fast you might want  to look at medium quality or low quality if you   ever want to go back you can either do that  through your browser or start it up from here   and that will take you to the same place through  the browser and then you can edit her further and   it will automatically save back into here so the  full circle is done to get this character into   Unreal Engine now we need to download her onto our  PC so you can either click the big green download   button here or you can see this little arrow on  her tile and click that and that will start the   download process this takes about an hour for me  my internet connection is so bad and my computer   is not the fastest so there we go so she's going  to download now once this character has finished   downloading locally to your machine you'll get a  blue arrow up here so you just click that it will   then import that into your scene so once that's  finished in porting if you come back down to your   content drawer remember that's where we keep all  our files and then go from content and then to   metahumans expand that and you should be able to  see your character there so open up her folder   you'll see your character there in a 3D preview  simply hold down and drag your character onto the   scene you can place your character anywhere what  you need to do is just make sure your character   is selected and come over to her details option  here and right at the top you've got transform   so you just need to put zero zero sure and Bam  she's magically in the middle of your scene once   you've got your character in we're gonna go up  to this movie board here click on that and add   level sequence I'm gonna give that level sequence  a name I'm just going to keep mine actually as new   level sequence and click save and then you get  this little sequencer bar down here this is like   your animation timeline but what we want to do is  just create a camera so click on that camera icon   and that will create a camera for your character  we can just move this camera around so we can see   our character in view from here you can actually  control your camera and zoom in and zoom out I'm   going to zoom in quite a bit there you can then  play around with the camera settings just to get   the details you want so this is something I like  to do especially if your character is out of focus   I go over here in the details panel for the camera  and find Focus settings and in Focus settings come   down to see the draw your debug focus is just on  top of that purple thing and check that box there   and then above it manual focus distance just go  back and forth until you see this purple plane   this purple plane helps you set the focal depth  so once you're happy with that and it's not so   blurry just uncheck that there and there we have  our character nicely set up ready for animating   so I'm just going to quickly add some texture  details to my character to make her look like   my original image in the content drawer I'm just  going to navigate down to metahumans and then go   to my lady me lady which is redhead YouTube  tutorial and click on that and then it's very   simple you've got your face and then you've got  all the different face options for this character   and the reason why there are so many faces because  there's different maps for wrinkles and different   Expressions so if you're going to change one  just ensure that you do all the changes on   all these Maps here so to get it out of here so  you can edit it in Photoshop just right click on   the map and go to asset actions in the menu and  click on export I'm just going to do mine in onto   the desktop load up Photoshop or whatever photo  editing Suite you actually have I've just loaded   that map in that was on my desktop which is a  target file so now you can draw on this image   I'd recommend creating different layers so you can  duplicate it and add it to all the other images as   you can see she hasn't got eyebrows and I want  to fix that because my character actually has   quite a defined eyebrows that I could not get in  metahumans we can then use those elements on our   character from our Unreal Engine so if we come  here we can then zoom in and let's still some   bits I'm gonna steal these eyebrows here because  they're quite good I'm just gonna take those off   that character I'm just going for speed not  accuracy to save the target file back out   just go up to file I need to click on save a copy  and then just save over the top of that file that   we pulled out of Unreal Engine consistently drag  and drop that Target file in and it will update   the character image there so now we are on to the  fun bit we're going to connect our face using our   iPhone and connect it directly to our meta human  so we can control our meta human just using our   face so what we need to do first is go into the  Apple App Store on our phone and you can see this   live link face here if you search live link phase  that will come up and click and install that once   that is installed come over to the app and open  the app and you should see something like this   to connect this app to our meta human we need to  be on the same IP so if you go up to the top left   hand corner you will see a cook if you give that  a click we want to click on the first option which   is live link if you come down to targets you just  want to add a Target there so you're going to need   to know your IP address and if you don't know it  do not worry to find your IP address all you need   to do is come down to your taskbar where it says  search down here on your computer and type in CMD   and this is to open up your command prompt panel  now you want to type into this ipconfig and hit   return on the keyboard and now it will display  your IP address so you want to look at ipv4   address if you come here so now on your iPhone  you just want to copy that ipv4 address into   your phone just by typing it in will do nothing  technical here and then once it's all typed in   you want to then click on ADD and now it has  added your IP address and that's pretty much all   you need to do so on the top where it says live  if that is not green make sure you click on it   and turn it to Green also we want to recalibrate  our face which means we want the AI to be able to   measure our face to make sure that it is set up  correctly for our metahuman to do this you just   need to have a neutral straight forward look on  your face and look into the lens of the camera   when you're ready just click that big red record  button and it will count down and take your photo   if you just click save that has recalibrated your  face so now we want to ensure that we have all the   correct components or the plugins installed  in unreal just to make sure these things all   connect together so if you go on to the top right  hand corner in unreal you'll see an option that   says settings and come down to plugins and then  we're just going to type in the search bar Apple   and please install the Apple AR kit so you just  need to check the check box next to it and the   Apple AR kit face support and then once those  are checked we also want to install live link   so type in Live Link in the search bar and  just make sure that all these plugins here have just ensure that all these plugins here are  installed and you know that because they have   a check button it might ask you to restart your  project down here once you've checked those box   which you want to do so please do that and with  all that done we can now just click on our meta   human and then come down here and under live link  you will now see all your settings so we want to   go into AI kit face subject and under drop down  menu you should if this has all worked correctly   see your iPhone we want to use the AR kit face  so give that box a check and then on the live   link body I add that and I will say use Livelink  body even though we're not um and we are now ready   to pick up our phone and see if this hasn't worked  as you can see it seems to work there's a tiny bit   of a lag between the data but if you're finding  that your Wi-Fi connection is too slow what you   can do is buy an Apple to ethernet adapter these  are very very cheap on Amazon a couple of pounds   and then buy an ethernet cable if you don't have  one and connect it directly to your computer I   would love to see what you guys create using this  tutorials if you could tag me at prompt Muse in   any and all socials I would love to see what you  guys are creating and I will be picking a winner   out of the lot and putting them on my social  media and giving you Amazon vouchers to say a   massive thank you for doing this tutorial again  thank you so much and that will do it bye-bye

Channel: Prompt Muse

Views: 337,944

Keywords: #aiactor #ai #midjourney #stablediffusion #aianimation #unrealengine

Id: FKoy7bncHLs

Length: 28min 58sec (1738 seconds)

Published: Fri Jan 13 2023

Reddit Comments

[u/Hybridx21]

This is by Prompt Muse. Not me.

[u/shmoculus]

Incredible, thanks for sharing