Wild AI Video Workflow with Viggle, Kaiber, Leonardo, Midjourney, Gen-2, and MORE!

hey everyone today I've got a pretty great AI film making workflow to share with you and I think it has a lot of interesting Potential from pre-production all the way to generating short films it's a bit of a kit bash but the results are really promising as we can see in examples from Friends of the channel on Kenny Harry and Jeff synthesized of course I'm putting my own spin on it so we'll be digging through everything that I learned hopefully saving you some time should you decide to try this workflow out okay let's dive in so the inspiration for this work flow oddly comes from the 2016 Gareth Edwards film Rogue one which as I think about it is appropriate considering that it does have historical significance as featuring the first major film with a fully deep faked character people definitely have feelings about that film and yes it can be a bit of a mess I personally have always liked it but you can clearly tell that there were production and script issues but there was one really interesting story that came out in a 2017 interview with editor Colin ghoul where he talked about a featurelength story reel that he ripped together before the script was even finished there was no screenplay there was just a story breakdown at that point scene by scene he got me to rip hundreds of movies and basically make Rogue one using other films so that they could work out how much dialogue they actually needed in the film by making the whole film that way I used a lot of Star Wars films but also hundreds of other films too it gave us a good idea of the timing I've always wondered what that real looked like I mean it is probably completely incomprehensible to anyone that wasn't involved in the production but the story has always stuck around in my head and it did get me thinking could we do something similar using our current AI tools but take things one step further creating sort of a hybrid storyboard animatic animation so taking that idea and applying it to the famous are you not entertained scene from Gladiator by way of John Carter from Mars another movie that's kind of a mess but I find entertaining and just a dash of Warhammer 40K gives us this is this not why you are [Music] here so that was all wholly AI generated or at least augmented so let's break it down to see uh how it's done what works and what doesn't work so the first step was obviously clipping out our reference footage and then from there taking it over to vigle doai which we looked at previously vigle just rolled out a 2.0 update and it yeah it looks pretty good now if you missed the full like walkthrough tutorial on vigle uh you can check it out it is linked down below though you can probably follow along with this very easily uh yes 2.0 still does all of the dancing stuff although Martin Harland did figure out a pretty great use case for some of those dance moves creating this T800 T1000 dance off and I mean just this is something that I did not know I needed in my life so rolling over to Mid journey to create our model for our main character who's dressed like one step shy of Shan connory and zardas uh if you've never seen Shan connory in that film I'm sorry that image is now burned into your mind I've lived with it for a very long time you can obviously use whatever image generator that you want I think the important detail here is to have a 9916 format and make sure that you're getting head to toe so use prompts like full body head to feet from here we're back over to vigle and by using the command for/ miix you'll be prompted for an image source a video source and then options for background and fine tune we'll talk about those in one second uh for our image source obviously it will be our character shot uh for our video Source I'm using this which is actually not the are you not entertained shot but rather the is this not why you're here shot options for background are white or green you'll want to definitely have it on a green screen background and then just make sure your fine-tune is turned on running that gets us this which is obviously a bit stuttery and kind of a mess don't worry we're going to clean all of this up one thing I've definitely noticed is that vigle doesn't really respond well to camera movement which is why in a lot of like the you know dancing videos that we see they're always locked down shots two other quick points namely about our God Emperor shot at the end uh doing the thumbs up thumbs down but I did choose to kind of leave it ambiguous because it's a 15-second short and I'm still thinking narratively it's a cliffhanger initially trying out this m Journey image and using that as a reference for Walkin Phoenix's arm raay yielded some results like this where he has kind of like this invisible forearm or as another note when we had this close-up shot in which the top of his head is cropped in frame uh you know when you run it through vigle you end up with a result like this which to be honest I can't get mad at vigle for you know it took what we gave it just reminds me of the Rota scene from Hannibal I mean between this and the Sean connory image from earlier this video is taking some weird turns the solution to all of this was by utilizing Leonardo a uh grabbing a screenshot of Wen Phoenix doing the hand raise uh and bringing that in using that as an image to image reference keeping the image strength actually rather low at 0.3 and utilizing the prompt God Emperor King Warhammer 40K arena in the background and running that we ended up with this now I think the trick in bringing this back over to vigle was by using an image reference with the hand already raised vigle was able to figure out like essentially what we were aiming for that said he is definitely still on the shaky side looks like an emperor who has had way too much coffee today so the solve to all of our shaky footage is actually by bringing everything over to kyber doai and using their new motion 3.0 feature so I have been pretty bullish on kyber namely because I think it's probably one of the most unique AI video generators around and when you factor in the new motion 3.0 model which should be released like literally any day now hopefully by the time that you watch this uh you end up with a generator that does some pretty interesting unique stuff so in this case we're going to take our vigle output bring it into kyber um giving it the prompt male cyber Punk Warrior Dwayne Johnson The Rock can you smell what he's cooking uh I only did that because I feel that within kyber when you name an actor it's not necessarily going to look like the actor but it's going to end up creating some level of consistency with that character's face uh in this case I did leather armor no shirt and yelling and then I did that in the style of it's was one of the presets the 3D rendering style from here we just make sure the new 3.0 model is turned on and then on the evolved slider just turn it down to zero if you crank it up you kind of get that Shifty warpy look that kyar is kind of known for uh but with 3.0 if we turn it down to zero you're not going to get as much of that ended up pretty happy with our output so here is where things get pretty interesting because now we're going to start moving into our background comp so the way that I thought that things ended up working best was to take a screenshot once again of one of our initial frames of the reference video bring it back over to leonardo.da said there's some smudging going on right here not a huge deal considering that essentially that's where our character is going to go now I didn't want just a static background I wanted some movement and some life happening uh you know as you would in the actual film uh so I took the back plate and brought it over to Gen 2 where I did give it the very just simple command of a move to the right now things do go into like a very very soft depth of field thing here but that's okay cuz we actually kind of want that in this case um so now taking this and then running our back plates through kyber as well just to give everything kind of a unified cohesive look now if you're wondering why we're taking the extra step of that Gen 2 output running it through kyber on its own as opposed to taking the Gen 2 output marrying it with our kyber character and then just baking it in kyber together uh the reason for that is that extra layer of stylization on your initial kyber character once again occurs and you're just going to end up with a lot more morphing and inconsistencies from here we take both our kyber background and our character background and bring them into a video editor uh here I'm using Premiere you can use really any of them uh Da Vinci you'll be able to do this as well and da Vinci is free if you don't have one already um so on the top layer here we have our character and beneath him uh on this layer we have our background um so just sliding those two together and then using a chroma key remover in Premiere it's called Ultra key we just take our little eye dropper and drop onto this color and uh boom now they're more or less comped together now a few tricks here uh you can play around with the choke soften and contrast on your characters uh in your chroma key remover I did find that if you put a gossan blur onto your character like just really really subtle in this case it's actually just set to two um it just kind of helps sell the character you know kind of comped into that background a little color correction never hurts as well Da Vinci users I mean you have a whole Suite of tools available to you there and the other plus side is that again because these two are comped together we can actually move our character and reframe them however we want as well and as a final touch just to make things feel a little more cinematic I did add in uh black bars at the top and the bottom so it's like faux letter box so for our crowd chanting I ended up using the totally free site audio genen to generate that audio so uh using the prompt crowd chanting in an arena we got this I mean yeah that's fine it's just background sound uh for the dialogue I did the thing that I think that everybody probably initially thinks of uh I went over to 11 Labs but surprisingly ended up having a lot of trouble uh I initially tried speech to speech and taking the clip of Russell Crow saying is this not why you're here uh yielded some pretty terrible results is there here is here yeah so yeah a bit of a mess and I wasn't necessarily overall thrilled with any of the straight text to speech versions of it either uh so I actually just ended up finding an alternate again another free source called typ cast I ended up using uh the Frankenstein model and we ended up getting this is this not why you are here sounds a little bit more like the Hulk but whatever it's it works for you know our demo purposes here finally rinding out for this soundtrack I probably could have just used sunno which their V3 free model by the way is now open to everyone even if you're on the free tier but I just ended up going into Ableton and put together a quick like 20 second Cu uh just basically using Loops so overall is this method perfect no it is not is it watchable as a full feature film it's debatable but I do think that it works pretty well for a short film but if you're working on pre-production on say a giant Star Wars movie or even if you're just a small Indie filmmaker I think that this method is a lot more useful and productive than say just you know ripping a bunch of movies together my head has been really spinning with incorporating other tools into this kit bashing technique and I do have some more workflow videos coming up very soon so if you're interested in that and you have not had the chance to yet I do invite you to hit the like And subscribe button I thank you for watching my name is [Music] Tim

Channel: Theoretically Media

Views: 12,746

Keywords: ai movie, ai animation, ai video generator, text to video, ai filmmaking tools, ai filmmaking, ai video, ai video editing, viggle ai, kaiber ai, kaiber ai tutorial, adobe premiere pro, midjourney, runway gen 2, leonardo ai tutorial, AI Kitbash

Id: Thl38Lv8yoE

Length: 11min 58sec (718 seconds)

Published: Tue Apr 02 2024