OpenPose to Blender Facial Motion Capture Tutorial

did you ever wish that you have your own artificial intelligence that would look at your face and turn it into data like this then if you had all that data you could bring it into blender and put it on a 3d character like this Wow well what is open pose and how do I install it here we go so you can Google open pose and take a look at their website it has a lot of really good information it's open source and it's free for non-commercial use and its precompiled too so you go to the download page if you click around it has a download link that you can download these four different versions I didn't need the unity versions and there is a GPU and a CPU version and my GPU on this computer isn't powerful enough to actually run open pose so I downloaded the CPU version but I think if you have a more modern Nvidia video card you can use the GPU version and I've done that onto the computers and it works a little faster than the CPU version so let you decide you click download it once you download that you unpack the zip and it'll create a directory called open pose clear 3d CPU and you can then open up instructions text and forget these lines I added those lines later but basically if you double-click on models get models it'll download everything via batch file so go into the models directory and I've already downloaded all the models so there's a bunch of junk in here but there's a get models dot bat and if you double-click on this file it will open a DOS window and it'll download and install all the models that you want to use to use open notes now that we've installed it we're going to go ahead and run an actual video through it so I've created a folder and I've placed it in the open flow pose clear 3d CPU output folder and I created a folder called capture test and I put my mpeg-4 video of a quick facial capture I did with my cellphone and what we're going to do is we're going to run this video into open pose and create all of the data for the facial motion capture so if you look at instructions desk text in the root of the FLIR 3d CPU folder if you open this guy up I've been keeping the command lines for every one that I've done so far as a test and every time I do a new one I copy and paste in here and for this video I created this ahead of time but I used these previous ones to do it basically what I did was I said this is the switch that tells it which video to do and I said go to output capture test video capture test mp4 these are relative paths from the root of the FLIR 3dd CPU folder so you can see I'm an output capture test and I put that output capture test right here and then the name of the file right here and then I say write JSON to output but I can put into here output capture test and it'll put the JSON into this folder now and then I use the face switch if you want to track the hands you can put the hand switch in here as well and paste that into there and it'll do the face and the hands but I have no hands in this so I'm not going to do that and this tells it where to write the video output and I want the output to go to output capture test and it's going to create a capture test avi file that it'll create right alongside right here so if I take this line I copy it and then I go to the command line and I go to the open pose folder then I can right click and paste this line in and then when I press ENTER it'll start open pose it takes a minute to draw the first frame but what it's doing is it's analyzing the first frame in the file and it's finding all the points of my face and plotting them to the screen and figuring out where all the points if it's recognition are so in a second my face will pop up right here and there's the first frame and if I minimize this and go back to the other page you'll see that it has created the AVI file and it's creating the JSON files all alongside of it right here so we'll have all of the files slowly showing up as it analyzes the file so I'm going to pause the video here and let it think for a long while and come back in a little bit left right pop down blink blink twist twist hello this is a test this is my left ear this is my left ear this is my right ear eyebrows up down so now that we've captured our facial motions to JSON files using open pose we can go and try and get them imported into blender to do that I've created a Python script and I've posted it here on github you can go and grab it on the github page for the script there's a long description of how to modify the file to make it work for your rig you can download the Python file and open it in the text editor inside a blender or you can just click on the file select it from github copy it then when you're inside a blender open a new file you can go to scripting and create a new text file and paste it in you now have a script that you can modify to work with your rig and when you're done with the modifications you can press the Run button to execute the script and apply the JSON files to your rig so now that we've finished with the open pose capture we have our directory with our two video files one of the input file and one the output file and all of the JSON files that were generated by open pose now I've taken my characters blend file and saved it into this folder the script doesn't have a browse for files dialog box it just looks in the same folder that your blend files saved into so either move the JSON files to where your blend file is or save your blend file into the JSON folder like I have here so now that we have the script modified and the way we want it we can go ahead and just execute it but when I'm running Python I like to actually see what I'm doing in the console to see if there's any errors so I turned the console on and I leave it on in the corner of the screen so I can see what's going on so I'm gonna go ahead and execute the script and you can see it's running up through all the frames there's about 900 frames and it goes actually quite quickly so that's what's nice about modifying the file seeing if it works and then trying it again and running it over and over again so I go back to the beginning and I hit play there's a little bit of a head twitch at the beginning but it looks like it worked well so let's render this out and see what it looks like left right pop down blink blink twist twist hello this is a test this is my left ear this is my left ear this is my right ear eyebrows up down [Music] [Laughter]

Channel: Checkered Bug

Views: 68,156

Keywords:

Id: EUR6vsE0k6E

Length: 9min 28sec (568 seconds)

Published: Sat Apr 25 2020