Filmmakers Test Sora + AI Actor Swapping

some filmmakers got a hold of Sora and the results are absolutely astounding there's a major update to a free AI video generator and Firefly beat mid Journey with a brand new feature this is your AI film news of the week now up to this point we've mainly seen Tech demos from open AI with just a single shot but not really a story well that all changed this week when some filmmakers were given access to Sora so you'll find a link below this video there are some really fun film examples my personal favorite one is this film called Airhead which was created by shy kids it's a really interesting concept and uh it really showcases a lot of the dynamic movements using Sora now another incredible example is the music video concept created by Paul trillo Paul was actually kind enough to send us the raw file so the video that you're watching right now was directly from him now there are plenty of other Demos in here that you absolutely must check out they're very very impressive so the question now is what did these filmmakers actually think of the experience of using Sora well Paul trillo was kind enough to share some of his insights in LinkedIn and one really interesting note that he pointed out about his specific project is is that most of the project was one prompt with a stitched video at the very end which is really incredible because traditionally you would have thought that that would have taken a lot of different shots that were basically spliced together and he did say that it does take lots of tweaking and rerolling which is a completely normal part of working with artificial intelligence however he did go on to say that control in Nuance is always best achieved with cameras actors and traditional bfx so it seems like our initial impressions of Sora have been confirmed by filmmakers we can't wait to test this tool and begin playing around with it ourselves and while we're talking about open AI I should note that they have actually been meeting with Studios out in Hollywood to Showcase their tool we were actually interviewed by the LA Times about not only Sora but the larger evolution of AI in Hollywood that interview should be live around the time of this web show now controlling your characters is essential if you want them to be expressive and convey the story that you're trying to project into the world well that's not always easy to do with artificial intelligence but there's one tool that just came out that is trying to solve that problem it's called vigle which is one of my favorite product names up to this point and basically they are branding themselves as a tool that allows you to drag and drop and swap characters from videos now we saw some pretty interesting examples of this tool in action from Able art and we saw another great example from Max inhorn I actually want to show you the power of this tool in action so let's get started to use the tool all you have to do is go to the link below this video and you will be redirected to their Discord Channel so all you have to do is go to one of the animate channels here and you'll do forward slash and we will do mix now comes the fun part so what I have right now is this video of this woman taking a walk next to this big building and she's holding a camera so what you need to do is drag and drop your video footage into the video section here then you need to get an image of the character that you want to replace inside of the video for our example I have this image of this woman now a full body shot is typically better you want to make sure that none of the extremities are cut off and you can see basically most of the character so we'll basically take that image and drag and drop it into the image section so now we also need to select a background we're going to do green for green screen so that we can easily key it out and I always like turning on fine-tuning and when you're ready go ahead and click enter so after just about a minute we have this video footage here which basically is our character and its motion tracked and it's not perfect especially because the uh woman in the original video was holding a camera so her hand is kind of being a little jittery and the Fidelity isn't super high resolution but it really is incredible considering all we did is really upload an image and it swapped the character but now we need to composite this scene together the problem is if we simply drag and drop green screen footage over the original footage you can see that there's actually the original character in the background which is not great so we actually need to rotoscope the underlying character out of this video now of course there is an AI tool that can help us with this so I'm going to use runways in painting tool it blows my mind every single time I use the tool all you have to do is log in to Runway go to edit videos and select in painting now all you have to do is simply drag and drop your original footage into the timeline here and we'll adjust our brush size and literally all you have to do is brush out the character just do this really quickly it's amazing how quick this tool works just like that okay the character is now selected deselect and again in real time the character is now taken out we can preview this and you can see that it did a pretty dang good job so now all you have to do is Click export you can select the resolution in the file format if you're editing you typically want to use a prores format because it is an editing codec as opposed to MP4 which is more of a delivery codec for Distributing online and you can select uh 4K or even an image sequence uh which is typically standard for visual effects pipines so when you're ready all you have to do is Click export so the export should only take a few minutes and now we have this video footage that has the character removed of course you could go in and fine-tune this if you needed to but we'll just do a very quick example so you could really use any editing software that you want I'll use Adobe After Effects you could use Premiere you could use daci resolve you could use Fusion it's entirely up to you you could probably even use cap cut if you wanted to and what I'm going to do is I'm going to bring in our video plate that has the character removed and then I'll bring in our green screen footage so we'll just drag this and drop it into our project panel here and I will drop this original blank footage here so character is removed and then we will drag and drop the green screen footage over the background so now you need to go to your effects and presets browser or whatever uh tool you're using you can just use the king tool there and for After Effects we'll use key light 1.2 and we will will grab the green color here so you can see that we have selected this character here and we could go in and refine you can see the edges are a little fuzzy here so we'll go to screen mat and we can turn up the clip black just a little bit and we can shrink it just a little bit ever so slightly and that looks pretty good so obviously the compositing of this character is not ideal she seems a lot brighter uh than the environment it's not bad but it's not ideal either so I'm going to show you just a really quick tip this is not industry standard it's just a really quick way that you can uh composite your characters from Green Screen into your background footage I'm going to go to trone and drag the trone effect you could use a tent effect if you uh don't have trone in your video editing application that you're using and I'm just going to select colors that are indicative of the highlights midtones and shadows so we'll grab the bright part of our background for the highlights we'll grab the dark part for our shadows and then we'll select a middleof the road color for the color grading I'll select kind of one of this orange kind of gray panel here uh that should be about a 50% gray and now we will change the blend with original and we'll just dial this in as it makes sense maybe down to we'll say about 82% here just dialing it in uh as it makes sense again you could finesse this for a long time if you wanted to and let's go ahead and check out our result here so there you go we have this character walking around this environment now is this perfect absolutely not but is it really easy and is there storytelling potential here yeah for sure and finally if we wanted to get rid of the vigle text in the bottom right corner that's no problem we can just select the pin tool and cut it out just like that and we'll hit M for mask go to subtract and there you go so now we have our character and there's no watermark and she is in this environment now I wanted to test out vigle in a few other use cases so let's take a look at those really quick so I have this second example here so we have basically this video footage of this guy in the woods this is just stock footage that I got off of art list and he's just like looking around and he's also like walking around without shoes on which is not safe I think that's exactly how you get tetanus and basically I want to add in this alien character I want to swap this alien character for the other character well I used vigle and this was the result so you can see that it's not great he kind of dances a lot now the input image only had the front of the character so I guess it is kind of impressive that it did that but whenever we cut to the front view you can see that it really is distorting the face and it didn't do a great job with adding in the depth now it's still amazing but I would not say that this is Silver Screen quality just yet again there are other competitors out there like wond Studio that are trying to be a version of this tool where you simply drag and drop a character and you can swap them inside of a video and wonder studio is also really interesting because it gives you the tracking data that you can take into to a 3D application I want to show you a few other examples that I tested with vigle here so the first is this video of this guy dancing it's an AI demo right we have to have someone dancing and I wanted to replace him with this image of this woman and the result is this it's okay it's not perfect I mean a lot of the movement is pretty interesting but it has that Distortion that is present in a lot of AI 3D models we also have this video of this woman dancing here you can see that it's kind of challenging because you can't really see a definitive outline of where her legs are and I want to replace her with this person right here and the result is this and you can see that the person is just really warped and distorted and it looks kind of strange we also have this really interesting Michael Bay style orbiting shot and I want to replace the woman in the shot with this guy here and the result is this which is really creepy the guy looks really off you can see that when it's a front view it kind of uh becomes in focus and uh a lot sharper but when it's the side view wo that is uh not looking great and finally we have this video footage of this guy just making a really creepy kiss to the camera and I was curious does this work with a 3D animated character so I wanted to replace him with this character and uh yeah this is just straight up nightmare fuel I uh think I'm going to see this character in my dreams tonight we also announced this last week our big NAB party in Vegas the party is going to be on April 15th at 8:00 p.m. there are only a few tickets left to enter the AI prompting tournament the winner will of course win in Apple Vision Pro but there are plenty of general admission tickets if you want to hang out and meet fellow AI filmmakers we're talking with the venue to actually have ai generated cocktails at the party so it's going to be a fun time and while we're talking about events I should note that there is an unofficial curious Refuge Meetup happening in Brooklyn on April 2nd if you're in the area and would like to meet fellow filmmakers I definitely want to encourage you to make it out to that event we actually have to keep increasing the capacity cuz so many people are going to be there the team at hyper just updated their AI video generator to create clips that are 4 seconds long and again this tool is completely free and it's creating some of the most realistic AI videos that are available to everyone let me show you how to use it all you have to do is go to the hyper website and go ahead and sign in now all you have to do is upload your image I have this image of just this like futuristic rocket engine and I'll just drag and drop it into our scene here and you can prompt in and say what you want to see we'll say rocket engines and you can click on these settings here by default it's going to go to two go ahead and click on four and click create so I have an example from earlier I'll go ahead and play this back you can see that it looks pretty cool now not loving the light leak that's kind of Tak over the entire frame but it does look pretty darn cool and I actually took these video clips into topaz video and upscaled them into 4K and created a little montage video that you are watching right now and I got to say some of the results that I'm seeing from hyper look better than Runway and pabs so I definitely recommend testing out this tool for yourself I think you'll find it to be helpful in your AI video pipelines Adobe also came out with the ability for you to add in structure reference to images whenever you upload them to Adobe Firefly it's just like using an image reference inside of mid Journey but the results are actually really interesting inside of firefly now to use the tool all you have to do is go to the Firefly website and click on the get Adobe Firefly free button and you can click the generate button here now to use structure reference basically you need to upload an image I have this image of this adorable looking bird character and we'll just drag and drop him into the upload settings here and you can dial in the strength if you want it to be more or less like the underlying image I want it to be more like the underlying image because I really want it to inherent the composition and style of the image that we uploaded and we can also go in here and add in a style so I actually want this to be in a watercolor style so I have this watercolor image as a reference so I'll just grab this image and drop it into our reference section great and now we can prompt in a watercolor painting of a bird and go ahead and click try prompt okay let's check out our results here and I got to say they look pretty darn good Adobe Firefly did an amazing job with the style reference but I want to show you this in action over on Mid Journey so we can compare and contrast so to do the same thing on Mid Journey all you have to do is go over to the mid Journey website and we will click this plus button here and we will select our bird as our image reference and we will select the watercolor painting of the boy as the style reference again if you want to use a style reference you have to click on the little paperclip icon so now we have the bird as the image reference and the boy as the style reference and I will say a watercolor painting of a bird and you do have the ability to go in and adjust the weight and I want to more accurately try to replicate what we got from Adobe Firefly so I'm going to say-- IW for image weight and I'll say image weight of 2.5 it has a maximum image weight of three so 2.5 should be pretty darn close to the original composition and we can also go in and add style weight style weight can be anywhere from zero to 1,000 I think for this one I'm going to say a style weight of SW 5500 so uh the default is 100 so this is going to have much more stylization than what is normal inside of mid journey and we'll go ahead and click enter okay let's take a look at our results here so we have this image of the bird this image this image and this image and I would say that these are much more like an oil painting than a watercolor painting So it seems like in our initial test here Adobe Firefly is actually doing a better job for style transfers so if you need to change the style of one image to another style Adobe Firefly might be a better tool to use I should also note that enrollment for our AI film making and AI advertising course is now open this is going to be an incredible session we have many new lessons heading your way including how to do virtual photo shoots and and how to do character Replacements it's going to be a lot of fun if you're interested you can click the link below this video to learn more we also just interviewed maton at pabs over on the podcast where he shared his experience with transitioning from being a literal rock star to becoming an AI filmmaker it's a great conversation I highly recommend checking it out we also came across this interesting prompt breakdown from Nicholas Newbert he is one of our favorite AI art directors and he was exploring this idea of basically a 1977 inspired DVD screenshot and the photography looks really fantastic it kind of has this uh dramatic cinematic shot on film Vibe you'll find a link to the prompt breakdown below this video we love checking out awesome examples of prompts and experiments from around the industry so if you ever work on a good experiment and are proud of the results results be sure to tag Us online that is @ curious refuge on most social media platforms the character replacement tool that we talked about earlier vigle is available to all people through their Discord but if you want to run a similar tool locally on your own machine there's actually a new comfy UI node called champ that basically allows you to do a very similar thing there's some really interesting breakdowns of the tool in in action and if you want to download the code for yourself you'll find a link below this video there's also been a lot of AI misinformation recently that I think it's important to address if you're coming across Twitter or LinkedIn and you see a shocking AI headline chances are it's either not true or it's not telling the full story we actually came across this post this last week that basically said a video is 100 % Ai and in reality it's actually like haen where the lips are replaced so I just want to throw that out there if you're coming across news definitely try to validate it because there is a lot of misinformation about there not only with people creating AI videos but also people with sharing AI news some researchers at MIT came out with the ability to create highquality images 30 times faster using stable diffusion and Dolly 3 now the reason why this is important we already have the ability to create realtime images but the quality of those real-time images is not amazing but if you could have really highquality realtime AI images generated eventually you're looking at really high quality real-time AI video and that is going to open up an entirely new way of telling stories we also came across a report this last week where they asked 70,000 large language model users to vote on their favorite large language model and Claude 3 Opus actually has a slight advantage over chat GPT 4 now does that mean that you should never use chat gbt 4 or that you even need to cancel your subscription and move over to Claude 3 not necessarily but if you are a screenwriter and you need to take a full script and upload it to your language model so that you can ask it questions Claud 3 Opus actually is is the best tool that you can use at this point but there have been many rumors about chat gbt 5 coming in the very near future so we wouldn't be surprised if we saw that tool very soon mid Journey also announced this last week that they kind of hit a hiccup in their AI video exploration process they said that uh they're not happy with the quality from mid Journey video so there's actually a 50% chance that mid Journey 3D could come out before mid Journey video you can find a link to the office hours recap below this video we're obviously super interested to see examples of mid Journey video whenever it comes out but with Incredible Sora video releases I'm sure mid journey is having to bump the brakes and make sure that the quality is as good as possible and that brings us to our AI films of the week the first film that I want to shout out is a music video concept called escape the music video is is really cool and I really appreciate some of the particle simulations inside of the video it really comes across as a high-end music video project and I think they did a fantastic job with it the next project that I want to highlight is called but a dream by Dustin Hollywood it has some really high-end visualization some of the best AI generated videos that I've seen up to this point there's great examples of smoke water and abstract simulations that are simulating classic V effect shots it really is incredible and I think he embodied this dreamlike aesthetic in a very convincing way and finally our most impressive film project of the week comes from Winston Mayo who created a prince of Egypt liveaction film it's basically the Moses Story and there's a trailer that you can watch online but what was so interesting is Winston Mayo actually spent 94 days creating a feature film of this project the shots look really good you can tell that he has a storytelling background and he even gave us a shout out for helping him along his education journey in learning more about artificial intelligence so fantastic work with your accomplishment I can't believe you were able to put together an entire feature film in 94 days thank you so much for watching this week's episode of AI film news of course if you want AI film news s directly to your inbox you can subscribe over on Curious Refuge you'll find a link below this video also if you want to join us for our April session absolutely no pressure but we would love to have you inside of the program we just received a few messages just this last week of even more students that have landed jobs from the training inside of our program and of course hit that subscribe button if you want to stay up toate on the latest AI film news thank you so much for watching we'll see you next time

Channel: Curious Refuge

Views: 54,102

Keywords: SORA, openai, sora news, sora, ai videos, image to video, ai movies, text to video ai free, haiper, midjourney, midjourney v5 tutorial, midjourney multiple characters, ai generated art, adobe firefly, firefly, viggle, sora ai, openaisora, sora release, paul trillo, curious refuge, ai films, ai short film, hyper realistic ai video, ai trailer, ai, artificial intelligence, ai film news, chat gpt, caleb ward ai, claude 3, generative ai, midjourney video, gpt 4, ai news

Id: hq5SCThpAww

Length: 25min 1sec (1501 seconds)

Published: Fri Mar 29 2024