Taking AI animation to the NEXT level

hello and welcome back to another prompt news  tutorial today I'm going to be showing you how   to create a fully animated AI character the beauty  of this technique is that you don't have to act   out the scenes before in fact you don't need to  look like your actor or your actress this method   could potentially be used in the movie industry  if not just the concept artwork it could be used   for an entire movie itself for this tutorial I'm  going to be training a realistic data set for my   puppet so I'm going to be using myself so you  can judge the likeness at the end for yourself   I gather up loads of images of myself I then train  it with Koya Laura and then once I've done that I   use character creator to create a puppet and you  do not need to know any 3D there is no learning   curve to this software you're basically using  sliders and dragging and dropping assets onto   your 3D character it is super simple to use  obviously you can go down the blender root   and The Meta human root if you want to but I  just warn you there is a learning curve so it's   up to you what you want to do but my workflow  is through character creator this means I can   then switch out to any character using the data  set so once I've got those rendered files of my   puppet I can then switch out those data sets from  one character to another and it's really easy and   efficient way of working so let's dive in and  get on with the tutorial step one preparing   your training images I've prepared various images  of myself it's some images that are more like you   rather than photographs with a filter on because  trust me it works 10 times better if you do that   and you will get their likeness a lot better it's  important that you choose your head your body from   Full bodies in there some close-ups and far away  you need to ensure that your hairstyle is quite   different in each image that you're taking from  different profiles that your backgrounds change   that your outfits change if you pick too many  head shots into your training data you end up   with things like this because the AI doesn't  know what the body is so it gets confused so   you need to put some body shots in there I'm using  a website called burme which is absolutely free to   use and it crops your images down to the size you  want so I'm going to be using 512 by 512 because   that's pretty much my resolution you can go up to  seven six eight five seven six eight but remember   that's going to use more vram and take longer to  train so once I've done that I want to rename all   my files so click on rename and just put X in  the box and save as zip and that will save all   your files in numerical order so one dot png2.png  because that's how you want it head over to Google   Drive and simply save your zip file there Step 2  training your data set I'm using the link off coil   Laura dream Booth we are going to make sure that  we're logged into our Google account here and then   we're going to press connect here so once you've  done both of those we can start the training   simply come down to Step One all we need to do is  check the Mount Drive button and run that cell and   this is going to Mount Your Google Drive so I'm  going to click connect to Google Drive it's going   to give you some warnings about connecting to an  un Google offered notebook now that's running once   that's completed you'll get a green check next  to it so this downloads all the dependencies into   your file structure over here now remember this is  remote this isn't on your PC so once you've come   out of here you'll lose everything but this is  where we connect the Google Drive so we can pull   files across from our Google Drive and then once  we've finished we can take the train model and   Export that to our Google Drive so we're just  going to come down here the 1.2 open special   fire Explorer ignore that that's when we don't  need to do that the default setting here is for   anything version free which is more of an anime  style model and if you click here there's a few   more preloaded links in there if I click on stable  diffusion 1.5 that's good for me because I'm doing   a realistic character I just want to chip in and  say the stable diffusion 1.5 base model is a good   all-rounder training file to use obviously you  can use whatever you want but if you're starting   off this is a good one to learn with you can  see in this bit below you can actually load   in your custom model which means you can go to  the hugging face link for example a good model   to use is realistic version too so you can get the  hugging face link for your model and place it in   there and run that cell but we're not doing that  for this tutorial so I'm just going to leave the   space underneath which is stable diffusion 2 model  blanks we're not going to be using that and then   just ignore 2.2 we're not going to be running our  own custom model so 2.3 download available they   sometimes you might notice when you switch models  in automatic one-on-one or whatever program you're   using that the images are really desaturated  and lost their colors that's usually down to   the veil not being detected or being corrupted  so we are going to load in the stable diffusion   1.5 V which is already there so it's a stable  diffusion phase and that's going to again just   download it all into our file structure and then  we're just going to hit on run on there and then   we come down to 3.1 locating train data directory  so this is going to create some file path to where   our train underscore data file is and that's where  all our input data set images so my face images   will be going into that folder and don't worry you  don't have to do anything it does it automatically   for you so I'm going to hit run on that cell once  again and it says your train data directory so if   we go and have a look so expand Allure down here  by the way if you don't have this open it's this   folder here to go into here go to Laura and you've  got your regularization data and your train data   do not drag and drop your images into here wait  we're going to be doing that automatically on to   3.2 unzip data set so this ZIP file underscore URL  so this is why we zipped our file up and put it   onto our Google drive because we're now going to  grab it so if you go to drive this is essentially   your Google Drive my drive and then I'm going to  come down and find my zip file which is a hundred   underscore prompt Muse images dot zip I'm going  to click on the three dots there and copy path   and then I'm just going to paste in the top zip  file underscore URL I'm going to leave the unzip   underscore to blank I'm just going to close this  hierarchy here so we can see what we're doing and   you can see there it's extracted all my files from  that zip file into that new folder so we're going   to come down to 3.3 which is the image scraper  again I'm not going to use this this is based on   more or less anime databases so what it does is  scrape regularization images which I don't want   to do because I'm not using anime so I'm going to  ignore this but if you are using anime you know   you can do it here nor 3.3 data clearing this  is to do with the cell above it you're scraping   all these images you might not be aware what they  actually are there will probably be some in there   um but you know hopefully no don't do that and the  convert transparency images well it says what it   does so if the image has a transparent background  that's very hard for machine learning so you want   to convert that and also random colors as well  so you check that if you're doing the anime and   scraping the images which we're not going to  do okay so let's go down to 4.2 which is data   annotation we're going to be using blip captioning  which is tagging images with a description this is   used for realistic imagery the one below it which  is the waifu diffusion is used more for anime so   we're just going to be using this blip captioning  over here so I'm not going to change any of these   settings I'm going to leave them as default and  run that cell what that will be doing is reading   the input images that I put into the Google  collab it's then going to be describing what it   sees in the images everything it moves out of the  description is what it's going to train upon so   it's going to describe my microphone the fact that  I'm wearing a necklace or potentially a brown top   this means it's not going to train upon the those  things which makes it so much easier to create a   way more flexible model when I'm generating the  images later on so I'm not stuck in this room with   this necklace and a brown top on so to show you  what I mean we're just going to show you the files   it created so if you come into your files Laura  and then expand on train data you can see it's   generated these caption files here so if we just  pick any image here 13 and you can see I've got a   microphone and a necklace so it should pick up on  those attributes so I'm going to click the caption   that goes along with that image and yeah it said  a woman sitting in a chair holding a microphone so   exactly that now I can actually add on to this and  add necklace if I didn't want it to train on me   wearing a necklace but I kind of like my necklace  and yeah it'd be good to have that in the model   as well so you can edit these further if you want  to but for this tutorial I'm not going to do that   I'm just going to leave it as is so I'm just going  to close those images there and close that window   ignore the way through diffusion tag as I said  that's for anime and I'm going to ignore the   custom caption tag this creates you a text file  caption which again I'm going to ignore that now   we're on to training model so in 5.1 model config  so if you've used stable diffusion version 2 to   train your model you need to check these two boxes  here I haven't I've used stable diffusion 1.5   so I'm going to leave those unchecked under the  project name give your project name as something   that you will remember so I'm going to just call  mine prompt tutorial and then underneath it you've   got pre-trained model name all so I need to  change this to my stable diffusion trained   model we downloaded all these dependencies in the  first cells this would have made you a pre-trained   underscore folder so if you just expand that  and then within this it's your safe tenses   model so if you go with the three dots click on  it copy path and simply just paste that in there   so we have the vae so the v file which controls  the color in your images so we also need to grab   that and that would have installed during the  first cell as well so that will be in the Fey   folder so repeat that process copy that path and  just simply paste it in there now this is where   it's going to be saving your finalized models  and I say models because it's going to create   multiple versions but we'll get to that later  once you've closed this Google notebook this   will all go all these files will disappear make  sure you check output to drive and that will save   it to your Google Drive and just run that cell so  you can see here the output path is content drive   my drive Laura output so there'll be a folder  on your Google Drive called Laura and it will   be in the output file we're getting to the most  important settings here so we want to keep the   train repeats to 10. we've got the instance token  I'm just going to keep mine at mksks now you will   see like random names sometimes like SKS this is  because it it's not a name that stable diffusion   Associates with something so it's not going to  call it up so by associating MK SKS with my model   it knows it's calling up my model my image I'm  going to keep that as is if you're not sure just   keep it as mksks Styles so we are not training  a style we are training a woman or you can put   person I actually prefer to put person you camp  at women it's up to your own discretion if you   want to do that resolution we're doing 512 by 512  because we have input images that are 5 12 by 512   if you're doing 768 put 768 here just change it up  the slider just leaving all these settings here as   default and I'm just going to run that cell so  we come down to 5.3 lore and optimization config   but you really really need to experiment with  the settings yourself to see if you can get a   better result because obviously you are trading  different images than I am but however I will   give you my settings because I have done a lot of  tests come down to the convolution dim which I'm   going to be setting quite low at eight and the  convolution alpha I'm going to be setting at one   and then we come down to network dim which I'm  going to set at 16 and then I'm going to change   the network Alpha to eight so these settings  actually have huge influence on your model I   used to do the settings at one two eight by one  two eight but I've played around and I quite   like these settings for my realistic models what  settings might work for me might not work for you   guys because of different trading sets different  resolutions and all that but I digress okay I'm   gonna get leave the optimizer config as addon W8  bit so the first learning rate I'm going to change   to five E hyphen four so the text encoder learning  rate I'm going to change to one e hyphen four and   then the learning rate scheduler I'm going to  change to cozine with restarts the warm up steps   I'm going to do 0.0 305 and to be honest I'm quite  happy with that so this is probably going to be a   learning rate of about 950 steps but we'll see  once we hit run so we're going to run that cell   and then we're going to go to 5.4 training config  I'm going to leave low RAM on I'm going to enable   sample prompt I'm going to leave the sampler  as DDM noise offset I'm going to leave at zero   sometimes I go 0.01 I'm going to be creating  10 epochs here which will save a file at every   learning stage which means I can test the files  out in my web UI at the end to see if it's either   undercooked or overcooked or just about right I  like to do about 10 because it gives me a nice   diverse range to pull from the train batch sizes  now you can go quite low you can go to one I'm   probably gonna go to two and see how it goes from  there so the batch sizes is how many files it's   training together if I'm training six it's going  to be a lot quicker than it will be for two if   I went to one I would probably completely run out  of ram so if you do have a ram if you try sticking   to six or higher but if you don't have any Ram  issues whatsoever you can train on anything as   low as one here the mixed and save Precision  I'm both leaving those at fp16 and my epochs   save every Epoch so that's 10 epochs I should have  at the end I'm saving the model as a save tenses   model and I'm leaving this all as default here so  that's pretty simple so I'm going to run that cell and now we come to our final cell you'll be glad  to hear all you need to do then just run that   cell and leave everything as default and let the  training commence this might take probably about   30 to 40 minutes if I wanted it to be done quicker  I would actually increase the batch size hopefully   all this makes sense I wanted to describe what I'm  doing as I do it so you have at least a sort of   understanding of what's going on which hopefully  again will allow you to make changes to suit your   training data once the training is complete  you do not have to do any of the remaining   cells in the notebook your files will now be saved  automatically into your Google Drive so head over   to your Google Drive you will have a Laura file  in there an output file and in there lives your   Laura files and remember I said it would save a  file at every training step and we said 10 in this   demonstration so it's giving us 10 files here as  you're probably aware I use automatic 111 on run   pod and the link for the instructions are all here  so if you don't want to use automatic 111 locally   on your computer and you don't want to have to set  up a run pod like I've got the developer of this   Koya Laura notebook has just come out with a brand  spanking new automatic 111 notebook with control   net one and the brand new control net number two  as well as the ability to use your newly trained   Laura files you can use the caligorastro collab UI  and it's basically automatic 111 to generate your   images so I just want to throw that in there as  an additional option now grab these files download   them and load them into your stable diffusion  model Laura file just whack them all in there   step 3 creating our puppet now underneath the  generate button you'll see this sun icon here   give that a click and then this will open up this  panel along here select Laura and you should see   your Laura files now in there now if you don't  you can always click the refresh button you can   test out all these files here by clicking in on  each file that will then load its tag up into the   prompt like this so you just test them all out  just use one at a time you can also control the   weights of these Laura files as well by adjusting  this figure so I'm just going to load in my Laura   file with a prompt as well as a negative prompt  and just run it through and see what it looks   like so I'm quite happy with the overall state  of my Laura file it does look like me so I'm   just going to create an image of me bald I'm  going to be taking this bold image of myself   and dragging and dropping that into the headshot  plugin which then generates me a 3D model of my   face I can go in and sculpt this further if I  want to but I'm just using this as a puppet or   a guide for the AI it doesn't have to look great  this kind of has to look sort of similar to me   once I'm happy with the face and the body shape I  then add some hair again it's super easy I'm just   dragging and dropping from a library now once I've  finished with my character I'm now going to export   it to iclone so these programs work together in a  pipeline so iclone is more of an animation program   so this is where I'm going to be adding the facial  animation as well as the body Idol I use a plugin   called motion live I just activate motion live and  I've downloaded the motion Live app on my phone   here all links are below in the description it's  super easy to use all you're doing is recording   your facial movements and that's being applied to  your 3D model in real time so I've just dragged   and dropped a lighting situation in here and got  some Physics on the hair and I'm pretty much ready   to render come over to the render panel which is  here and I have selected PNG sequence I actually   went 768 by 768 in the end try and be divisible by  16 if you can and we come down here and I'm just   doing 300 to 500 frames and I'm going to export  those as PNG so we're going to jump back into   automatic 111 and I'm going to go through the  settings I'm using I'm using image to image and   I've loaded in my Laura here remember the buttons  over here and then I've just added a simple prompt   because I don't want the prompt to fight the  original image that's quite important so you   want to keep the prompt as brief as possible do  not put too many details that are not in the image   that you want to create then we have the negative  prompt which is as important as this prompt up   here so you can copy my one or put it down in the  description below so this is our rendered image   added in here so the sampling method I have set  to DPN plus plus sde you can use whatever you   want I've had good results with Euler a I've had  good results with k m Carrabba's sampling steps   I'm keeping relatively low for this width and  height I'm going for seven six eight by seven six   eight the same as my input denoising strength I'm  keeping low so I'm keeping the denoising strength   at 11. actually also you'll notice my CFG scale is  down six as well again we want to pull from this   image as much as we can and apply the Laura over  the top without the prompt compromising things too   much so I've already set my seed because I've  tested this out already when you do this just   render a minus one seed until you get something  you like and then lock it in with the ReUse button   so on control net I've enabled head and I  have the model enabled as well I haven't   changed anything from the default settings  here I've got a secondary model in as well   which I've enabled canny and enabled the canny  model again I haven't changed any of the default   settings so let's render and see what it creates  us that looks like me like on a really good day   and it's following the mouth movements as well  which we need for lip syncing I have seen so much   AI generation where the mouth is just a blurry  mess using my workflow you can now get precise   mouth lip syncing this is why I made the model on  me so you can see that it is working okay so we're   now going to back to render these frames which  will give us a frame by frame animation and to   do that head over to batch and put your input so  where your 3D files are sitting on your drive and   then your output where you want them to render to  and then hit generate and congratulations we have   completed our animation all that is rendering I  want to show you something really cool and this   is why I like this workflow I literally can come  over and switch my checkpoint file and run that   again and look at that it's created a semi-anime  Style version with a click of a button so you can   literally change the render type but by using your  diffusion checkpoint I think this is really cool   and this is a great starting point from where  things are going to go from here now we've got   control net 2 which has just come out which  I'm not using in this video so that's going   to take things to a whole new another level so  I've simply just thrown those frames into after   effects from automatic 111. if you have DaVinci  Resolve use the dirt removal and the D flicker   times two in there because that will give you  really nice move results to your animation so   I'm going to hit spacebar so you can have a look  at the animation I've slowed it down so you can   see the frames are blending are so well into each  other it's probably not very good animation I did   there but you can see the technique I switched  the checkpoint model over I didn't change any   of the settings I'd probably go back in and  change some settings to make the anime version   smoother but that shows you how quickly you can  flick essentially the AI renderer to change and   then that took about two minutes to render those  anime frames so I would love to see what you guys   make with this workflow so please tag me in  your social media so I can have a look so I   love looking through all the creative projects  everybody's doing with AI at the moment I put   all my AI creative projects and workflows all  on my social media as well on Instagram I'm   pump Muse on Twitter and prompt Muse and on  Discord there is a link below so I'm really   really excited to see what you guys create with  this this tutorial will be written up step by   step on the prompt News website as well and we  have a fantastic Weekly Newsletter that surrounds   the latest and greatest in AI Tech so please  join that on the prompt News website I thank   you for watching this video I really appreciate  it if you subscribe to this Channel and give it   a thumbs up I do have a buy me a coffee link  down in the description and that is just to   buy me a coffee I didn't put any content behind  any paywalls I like to make sure that all my   contents free and accessible to everybody and  having that allows me to continue to do that so   I thank you very much this video has been days  in the making because I have had to test out   so much stuff and variations to get this to work  so I know you guys appreciate that please share   this video on your social media and at me as well  when you do it because I like to see that that's   amazing and I can say thank you to you as well  so yeah I think that will do it for today bye

Channel: Prompt Muse

Views: 120,644

Keywords: #aiactors #3danimation #LoRatraining #Lifelikecharacer

Id: HUPcr5njxkM

Length: 24min 40sec (1480 seconds)

Published: Fri Apr 14 2023