How to Make Midjourney Consistent Characters Talk Using Runway & Pika Lip Sync (AI Video Tutorial)

mid Journey allows you to create consistent characters and both Runway and Pika allow you to take an image combine it with an MP3 and create a lip sync movie it's not quite perfect yet but it's still pretty cool when you start factoring in things like multi-motion brush you can really be more intentional about the types of AI movies that you're creating so it's really exciting let's take a look at how it's done now I want this to be a complete tutorial so I'm going to go through the process of how you generate the consistent characters in mid journey and there's some chapters in the description so if you already know how to do that you can just jump ahead to the paa or Runway parts of this as well all right so today we're going to generate two consistent characters in mid Journey using character reference images we're going to have those characters change locations and cinematic styles using style reference images then we'll put our two characters in the same scene using inpainting We'll add some camera motion and then we'll get everyone talking with lip sync using both pika and Runway at the end I'll show you the movie we made okay I did a time travel movie with consistent characters using Gen 2 and wave to lip about 8 months ago and it's a really good way to quickly put a character in a variety of locations so we're going to do that again I'm going to do everything in Discord because currently everyone has access to Discord and not everyone has access to the alpha all right so the first thing we need to do is create our characters so we'll go to imagine I'm going to paste my prompt in there I'm not too conern concerned about um the aspect ratio now I just want to focus on the face I want to focus on the character so you just go 1 2 3 4 1 2 3 4 depending on which one you wanted to upscale so let's say we like this third one here we could just click upscale here it will generate a higher resolution image of that character and then we'll click here rightclick copy the link so once I have that link I'm going to go and type a new prompt in so I'm going to keep the easy and just say a medieval queen and then you go dash dash CF so this is the command that says okay this next URL that I put in here is going to be the character so we're going to paste that in there and it's going to be really long and we're not going to worry about that right now because it's going to shorten it in a second um and all I'm doing now is just putting one of these in so I can get the URL I'm going to get in a second I'll show you what I mean this URL here after the CF it has the MJ run and the this weird code I'm going to save this for my project I have this little character list here so I'm not using all of these characters but I am using some of them these are the ones that I thought like I may want to come back to them um at some point so I have like different characters that I like I have different styles that I like um and I can just keep these nice and handy in a Google doc I also have my script here too so I can keep track of what character one and what character 2 say that's going to come in later all right so this is the character that I generated that I I like a lot more so I'm going to use her for the time traveler and then I started by trying to put her in different scenarios so at first I kept it really generic like I would say an Interstellar astronaut and I would get images like this which is it looks like her but this is not really um anything that like she's part of the actual Ro rocket ship here so you still have to prompt you still have to do pretty good prompts um you'll get little things like this where it turns their heads around her head is backwards but it's her still tried some other characters I thought this is a good character for the guy who keeps getting it wrong and then I created some different styles so I would just type in Imagine and I would come up with different scenarios that I like so this is like a medieval castle I'm really just looking for textures and colors there are no people in these shots they're just sort of like the Ambiance the vibe that I'm going for and you can see here's an example of the character that I made in that style so to combine a character in style we're first going to write our prompt and then we'll add an aspect ratio if we don't want it to be square so I want it to be 16 by 9 then we'll add the character reference which is the URL for that character and I'm referencing this character and then we'll go srf which is the style reference and that's this URL so it's it's going to take this prompt put this character in it and give it this style and make it this aspect ratio let's see how it does and then I'm going to do another one that takes place in Egypt also with the flip phone same aspect ratio same character but now we're changing the style up so now we're going to be in Egypt so I'm changing the prompt to have some language about Egypt and then I'm adding this other Egyptian style so now you can see you know they're similar enough like it's not exactly the same person but like it's pretty pretty close um and the nice thing is that we can recognize her so we can recognize her in this Egyptian um atmosphere as opposed to these other like these these look similar like if you saw all of these they look the same there might be some slight differences between them but it's more or less the same character now one thing you might be noticing is that the outfits that she's wearing are very similar to this outfit this sort of like scoop neck kind of thing that she's wearing um and you might not want that so there's two ways to do this so one is that you could upscale so let's let's upscale this one and then we'll vary the region so we'll just highlight a section here and then we'll say a flowing gown worn in ancient Egypt with jewelry and ornamentation there's probably a better way to do that but that's what I'm going to do so now there's some different styles that kind of match the background or the the Ambiance a little bit more I'm not going to use any of these in particular but I just want to show you how you can change the outfits because it really does stick pretty closely to the character reference costume whatever they're wearing as the character reference it seems to bring that along with it at least in my experience now the other thing you can do is is at the character reference level so this is the character I created but I didn't like this shirt so I decided to vary the region on it like this and then just change the prompt in here to different things until I found one that I liked so you know we get these different types of suits and sweaters and things like that so I just kind of played around until I found that I liked and eventually I was like I think this one's pretty good so I'll go here and go copy link I'll create a new image just to get the character reference URL there might be an easier way to do this but this is how I do it so I'll just say a time traveler in a jungle I'm not going to do any other kind of reference I just basically want to get a short URL for this one so this is all I need and I'm going to go back to my little spreadsheet you don't have to do the spreadsheet but I think it's helpful and so instead of this one I'm going to go in insert image insert image into cell link paste it in there so I've got my two characters here now I've got some styles that I've already created this is the time machine control room then there's different places that they're going to go one thing I've noticed is that it seems to keep the framing of the character so a lot of times you'll you'll get more close-ups because this is the character reference that we have so one thing that you can do is you can zoom out so when you find one that you like just click the zoom out 2x so I've got these four and I'm going to upscale one of those and because I've upscaled it now I can pan down I can click this pan down button and it'll it'll fill in this area down here and so now we have these um you know more of like a torso shot um I tried to play around with like giving him a pot belly didn't really work out but essentially this is a way of giving you a shot that looks similar to this like where you have more of their body in the front frame but it's the same exact person it's literally the same exact inpainted shot that's a tip because you can do multiple character reference images so you could do this portrait shot and you could also do you know a full body shot even if you wanted to so I mentioned we're going to take two characters and put them in the same shot as far as I know you can only do one character reference per image you can have multiple images in that character reference but it can only be one character so you'll get a shot that looks something like this like it's her talking to her clone almost or her sister who looks very much like her so this is actually good now that we have this shot um of the two characters talking to each other we can go in here to very region and then we can just use this little tool to kind of paint out this first one I'm going to try and get close so I can still get this sense that there's like a head here and that there's elbows on a table and then what I'm going to do is take um I'm going to take another character that I created for this one and put that character in here so I'm just going to say a person at a table in a Paris Cafe and I'm going to replace this character reference with this other one and you can see it starting to paint that other character in there there we go and there's little things you know about each of them so some are better than others but still we've taken two different characters that we've generated and inpainted one of them into a scene we could have this as our establishing shot and then cut back and forth between them what a great use of painting and consistent characters I'm learning so much today I love it all right so now let's go to Pika we're going to start on the explore page we're going to click image or video I added an image here and now we're going to click lip sync and that's going to bring us over to this page okay all of these are voices that come from 11 Labs so you can if you may be familiar with some of these or you've heard them before um I don't love any of them for the characters that I have but you would choose the character that you want and then for me I have this character style script so I can just copy some of the script from here paste it in here and click generate voice so if we don't like that voice we'll click the back button load a different voice in there and then pick a different voice so once we have this so you see we've got a 5-second clip here we're going to attach and continue but uhoh look what happens we can only choose 3 seconds here the other way to do that is to use an MP3 but we're still even with the MP3 we're still stuck to that 3 seconds so the way we can get around that is by attaching a video now there's two ways that we can do the video one is to attach the image generate it and then extend it so I did that here with I've got a 3se second clip and then I turned it into I added 4 seconds to it so now it's 7 seconds but you can see it really I mean this is the problem this is why everyone is going bonkers around Sora because even though it's a consistent character that we're bringing in she doesn't remain consistent in in the paa generation so the way around that is to make our own little video of her so what I did is I went into cap cut I took the still image and cap cut brought it in I added this effect to kind of give it a little bit of a wobble and then I did a zooming effect to kind of slowly zoom in on her um so I'm not worried that she's not talking yet cuz I'm going to take that video bring it in here got my video there I'm going to click lip sync and then I'm going to add my audio and then attach and continue and now you see I've got a full 6 seconds here that's a nice little cap cut hack for generating longer lip sync videos with temporal consistency and pea so the other thing is like if you create a video with Pika um and they have the speech and you want to add 4 seconds you can't lip sync onto the 4 second so you're you're basically stuck with the one you have you can resync it so you can change it that way but it's changing this first one so you can't add 4 seconds of dialogue to a liip sync video in the same way you can extend other pika videos this time instead of creating a video in cap cut I brought the image in a stable video and did this orbit camera move the camera motion and the lip sync combined really helped sell it as a real video I think camera motion and lip sync are a good match all right let's take one of our mid-journey images and bring it into Runway so we're going to start on the runway homepage click Start generating or we're going to go to gen two we're going to upload our file here and then what I'm going to do is go to the motion brush option and I'll just paint the background here I'll choose the autod detect this is one way to do it we can do a camera control in a second too but this is one way to add motion is if we highlight this castle and then have it move to the right and then let's just add some um ambient noise to her hair let's um brush in her hand and have it move up so it's kind of moving closer to her face and then let's paint her body and have it move down so kind of like you know actually let's just have her shoulder move down um and that way she'll kind of have her hand go up and her shoulder go down and her hair will have some ambient noise let's see how this one turns out um so we're not adding any lip sync here what we're doing is taking our image and we're adding some animation then we're going to take that video and lip sync to that video all right so this is our time travel guy who uh presses the wrong button when he's not supposed to and I love that I can actually um I can I can generate an image where he's pointing at a button and now with motion brush I can just kind of paint his hand in there and have it move that direction so he's actually pressing the button and then I can just paint his head and have it like tilt up a little bit then I'll paint in these dials and give him some ambient noise so hopefully they'll glow or maybe Spin and we'll give that a try so when those are done I can go back to the homepage and click on generative audio and then I'm going to upload my audio files this is an MP3 that I generated in 11 Labs so I'm going to upload that one since I generated the motion in Gen 2 I can click browse um but if I had done it in stable video or something else I could click upload all right so that's the export that I have from Gen 2 I'm going to bring that in it's going to detect a face and um you can see his hand move I I don't love this one so I'm going to redo it um but just to show you how we would lip sync um we've got the audio up there there's the video and then we generate it so I'm probably going to redo this one because her face really changes a little too much so she is a consistent character in the image but um over the course of those four seconds she loses all that Clarity and I think that's because there's too much motion going on so what I'm going to do is instead of using the motion brush I'm going to use a camera movement all right so there was a bunch of stuff in here where I motion brushed in I'm going to cancel all of that out I'm going to still do her hand moving up um and I'll still add some ambient noise to her hair and then what I'm going to do is go over here to the camera control so that's the motion brush so that's adding some motion specifically to her but now I want to affect the overall uh image so I'm going to do is go to the camera control and I'll do a pan and I like this little thing on the side that lets you kind of go left and right you can see how your change here is going to affect the final output so I really like this feature all right so once I have that one looking the way I want it to I can sync it up with the MP3 from 11 labs and it'll look and sound something like this we used motion brush camera control and lip sync for this one so here's an image here's some style references here's what that looks like when you combine the character and the style reference here's adding some motion to that image here's the voice that we're going to choose and then here's everything Al together it's a lot of steps but it gives you so much creative control and if you found this tutorial helpful please be sure to subscribe all right so here's the time travel movie we made made with mid Journey consistent characters talking with Pika and Runway lip sync who are you what are you doing in time dispatch oh uh I'm the new time travel dispatcher have you ever operated a time machine before um well I completed the training fine can you send me back 20 years let's see the rewind button is you've sent me too far in the past this is like 50 years ago I think I had caps lock on let me try again again nope this is like ancient Egyptian times sorry let me make a quick adjustment not even close can you just bring me back I'll show you how to do it ah I got it I need to hit the transport button no ah no wait no I wasn't supposed to hit the transport button was i f sounds good for dinner brontosaurus or woolly [Music] mammoth

Channel: AI Video School

Views: 3,792

Keywords:

Id: nL9UowfT0PE

Length: 17min 38sec (1058 seconds)

Published: Mon Apr 01 2024