Adobe's NEW AI Video Features are AMAZING! (Let’s Recreate Them)

Microsoft released a brand new tool that is going to change entertainment forever Adobe also announced their brand new AI video editing features directly in Premiere Pro but the problem is they're not going to be available until the end of the year luckily for you inside of this video I'll show you how to reverse engineer each one of their brand new AI video features using today's technology meta also released a brand new creative AI assistant that is completely free and we hosted a party for 450 people in the most iconic Esports arena in the world and created a new genre of entertainment so it's been a big week this is your AI film news of the week I want to kick things off by showing you a brand new tool that Microsoft released that has huge implications not only for entertainment but also Society at large it's called Vasa 1 it's very similar to emo that we demoed just a few weeks ago basically you upload an image and an audio clip and Vasa 1 will turn it into a realistic video now the thing that makes Vasa 1 really unique is the fact that you actually have the ability to control and change the direction of eye movement it also has the ability to project the same movement on different types of characters which has huge implications not only for the future of entertainment but also advertising soon in the future you will have different types of media depending on who is actually consuming the content it also has the ability to create animations that are nonhuman but in a very realistic way so for example we have the Mona Lisa singing The Paparazzi song it looks uh pretty darn expressive and realistic we also have this animation of this woman dressed in a Victorian outfit and it's kind of in a digital painting style it seems pretty darn lifelike we also can create visuals in a Pixar 3D aesthetic and you can see that as the character moves his head around it looks really really good and finally we have this video of this man who was a pencil drawing and now he has come to life now I should note that you also have the ability to control the 3D Dynamics using this tool so it creates a 3D representation of the character from a single image so whenever you get into the animation process using this tool you'll have much more control over the orientation of your character it doesn't actually have to be completely facing the camera like some other AI tools they actually will have that 3D movement which is essential for telling compelling stories now all of these announcements are incredible and definitely would change the way that we create animation in the future but but it gets even better the wild part about using this tool is the fact that it actually renders in real time so you can upload an image and upload an audio clip and it will play back that footage in real time now that has huge implications not only for the future of liveaction entertainment but also for other subsections of society for example having the ability to talk with an AI therapist and it actually seem seems like a real person is talking with you or for example I can't imagine that this technology is going to be used to create relationships with AI avatars and education in the future where AI tools and avatars like this may be presenting information like I'm presenting to you right now now unfortunately this tool is not available to us just yet Microsoft is being very intentional about the way in which they release this tool but it really does seem that this underlying technology when paired with traditional animation or film making pipelines is going to dramatically change the way that we create stories in the very near future in honor of NAB 2024 the team at Adobe announced some brand new AI video tools that are coming to Premiere Pro they're very important so let's hop in now the first thing that they announced is video in painting basically in inside of Premiere Pro you'll have the ability to create a mask and then prompt in what you want to see the cool thing is you can select whichever video model you want in their example they used the Adobe Firefly video example to transform a briefcase from having a few diamonds in it to having a lot of diamonds in it now I think this is pretty cool but unfortunately you don't have access to these features inside of Premiere Pro until the end of the year but I'm going to show you how to pull this off using tools that are available to you right now so for our example we have this shot of a briefcase and let's say I want to change the contents of this briefcase all you have to do is go over to pika's website that's Pika doart and you can click on the image or video button We'll add in our video of our briefcase here and bring it inside of p laabs and we will go ahead and click on the modify region button now you just simply have to select the region that you're looking to change so I want to change most of the briefcase so I'll select most of the briefcase there and then I'll say a potato in a briefcase and go ahead and click generate here's my generation from P Labs you can see we have these beautiful potatoes in the briefcase which are really better than diamonds and you can see that it actually did a pretty good job Adobe also showcased a demo of their video where they basically changed a watch face using their tool I actually thought it looked pretty good the lighting wasn't perfect but it really didn't do that bad of a job but again you can do the same thing inside of pabs for example we have this clip of this man wearing a watch we'll go ahead and grab that video and drop it into pabs and go to modify region and you can just simply select the region that you want to change I just want to change the watch face here so I'll make the region about there and I'll just say a fancy watch and go ahead and click generate so after a few seconds we have this video of this watch here now I should note that I don't think the Fidelity of the result from Pik Labs is quite as good as the one from Adobe Firefly video but if you're in a pinch and you need to do some video in painting Pika is the way to go now I want to show you one last example here so inside of the Adobe demo they had this man wearing a suit and now he's wearing a tie okay that's interesting I actually think we can do the exact same thing so we have this video of this man who is in the desert and just living his best life I think he should be wearing a tie so let's go ahead and make the change all we have to do is drag and drop him into pabs and we will go to modify region we'll move that region to make it right over where the tie should be like that and we will say a business tie and go ahead and click generate and after a few seconds we have the video of the man and he is now wearing a tie Adobe also announced a brand new object removal tool that also pairs with a Content aware feature that allows it to detect objects that you can remove from your footage it's basically like automatic rotoscoping it looks really really powerful I love the demo that they show basically they select an object they remove it and it removes it from the wall now the example isn't 100 % perfect it does seem like the the beveling on the brick does get a little soft where the object was removed but especially at a quick glance unless you were really looking for that problem I don't think it would be really noticeable for the average viewer now you don't have to wait until this tool is available you can actually do the exact same thing inside of Runway just go to the runway website go to log in and go to edit videos then select in painting then all you have to do is drag and drop your footage into the timeline here I have this video here of this person walking in a garage so simply grab your footage and drop it inside of Runway and then once you've imported it you can simply grab the footage and then drop it in the middle here to add it to your timeline and now we can adjust our brush size and just select things that we want to get rid of so I want to get rid of this sign over here click and let go and then after just a few seconds it gets rid of it which is so wild to me we also can select the arrows on the ground here deselect and then it just gets rid of it and then you can play it back and the wild thing is the preview is basically in real time and if there's any weird Parts you can see there towards the end there's kind of some weird Parts again you can just paint out frame here it'll think about it and then it removes it just like that it does an amazing job it's almost in complete real time and it looks really good once you're ready to export simply go to the export button you can export it in up to 4K in prores image sequence or MP4 it's completely up to you and now side by side we can watch both of these videos you can see in the original we have our sign we have the arrow on the ground and in the runway version we have removed our objects it was super easy to pull this off Adobe also announced generative expand basically having the ability to extend the length of a clip inside Premiere Pro you using artificial intelligence it's pretty amazing they have some examples over on their website basically you have the ability to not only expand your video clip but also select the AI model that you want to use to expand the actual clip itself now I thought their demo was cool but I wanted to see how the liveaction footage integrated with the AI video at the end did it make it feel like a cohesive clip and so I put them together and this was the results so this is the original clip and you can see that when it cuts to the AI video it actually becomes really static and it has this slow push in it's really interesting in fact it kind of feels like pabs whenever you upload an image to pabs it usually almost always does that kind of slow zoom in effect uh so I actually don't think that in this example it did a very good job but you can actually extend the length of a clip using currently available AI tools let me show you how how to do it so I have this video clip here of this woman looking at the camera nothing too fancy just a pretty normal clip here now all I'm going to do is go one frame back from the very last frame and go ahead and click on the export frame button inside of premere pro this works in any video editing application so you don't necessarily have to use premere Pro you could use dibenci resolve or anything else that you like to use and go ahead and save the image so this is our image result now let's hop over to Runway when you're inside of Runway just go down to image to video and simply drag and drop your image now of course you can go in and prompt the scene if you want to see a specific movement I'm not going to do any prompting here and I'm actually going to change the overall movement to about three and then go ahead and click generate so this is the clip that we have from Runway and I should note that when you compare and contrast this to the original shot it actually looks pretty darn realistic so let's go ahead and extend our clip so let's hop back over to Adobe Premiere and all we have to do is drag and drop the video footage into our timeline so all you have to do is drag and drop your footage from Runway to the end of the original video and if we play this back we have our original video and then it cuts to the runway footage now you can go in and customize the color grade to help it really really match but for a really quick fix if you need to extend the length of a clip this seems like your best option until the Premiere Pro feature drops later this year now adobi had a few other examples of this generative extend model one was this shot of this car that basically was driving and then they extended the clip and what I really didn't like about this example is they paused it for a second and then played it because I was like does that actually create a smooth movement or does it look weird and so I took out the middle fluff to create a synced video clip and here here is the result so you can see it plays and it's actually pretty smooth it actually did inherit some of the camera movement and I wasn't expecting that now is this perfect of course not especially if you begin to zoom in on some of the signs inside of that garage you can see they just turn into mashed potatoes there so it's not perfect I don't know if this tool is going to be Silver Screen ready by the time it launches but for some online Concepts or if you're really in a pinch it does seem like this tool could potentially be useful now this tool does look like it may have some issues for example in the Adobe demo they have this video of this man walking down a hallway and I actually thought it was a little weird I thought it was a video editing mistake because they actually darkened the frame that was extended and not the image at the bottom and they made it seem like the image at the bottom was the extended scene and not the video above it felt like it was out of order but then I removed that middle stutter and tried to create a video that played back seamlessly and I discovered that they may be trying to cover up something here because if you zoom in on the briefcase at the exact moment where it switches to the generative expand video The Briefcase completely changes you can see it went from having a shadow and light on one side to just being like a gray Square his legs basically change the orientation and uh it just kind of looks much more different so I'm not sure if that was a video editing mistake or if they're actually trying to use a dark filter to cover up the issue there but it does seem like this tool is far from perfect and the final announcement inside of this Premiere Pro update is the addition of generative AI video directly in Premiere Pro this is going to save so much time in the creative process having the ability to quickly create a shot or a detail shot to help you tell your overall story is going to be a very powerful tool now the really interesting thing is they're opening up their API to allow other tech companies to have access so for example they can generate Sora videos directly in Premiere Pro or if you want to use Runway or Pika you can generate those directly in Premiere again this is really just opening up the opportunity to have those AI models in Premiere Pro this announcement does not guarantee that we actually will have Sora inside of Premiere now they have a quick example inside of their video where they basically showcased the possibility of generating a SORA of video and then editing it with liveaction footage and it was kind of weird because when I saw the liveaction footage and then I saw the Sora video it felt like it lived in a very different world and then they also showcased a Runway video that also felt like it lived in a different world so I went ahead and edited all of those videos together to see if it made it feel like the film was cohesive and here's the result we can see the man looks out and then it cuts to the wide shot and the wide shot felt like a video game cinematic and then the shot of the person driving didn't look super realistic it's also very dark and kind of hard to tell what is going on so generally speaking I would say that I'm a little skeptical about this workflow because you don't currently have the ability to add in stylization so again all of these features are not currently available in most cases I would say you are much better off using third-party tools and your own creativity to create cohesive films at this time I'd also like to extend a huge thank you to the 450 people who attended our AI creativity tournament in Las Vegas it was absolutely amazing to see the competitors competing at the hyperx arena and I'd like to extend a huge congratulations to Jagger for winning the Apple Vision Pro but more specifically the custom crochet trophy from Curious Refuge a big thanks to Tim at theoretically media and Dave Clark for agreeing to be judges for the competition we would love to do more competitions like this in the near future and there's also been some chatter about developing an AI creative Esports league so we will give you more information as that develops and I'd also like to thank everyone who traveled from all around the world to attend some of our panels at NAB it was really fun to share our vision for a democratized creative future with Incredible people from the industry meta also released their chat GPT competitor this week completely for free and and I got to say it's really impressive now it is integrated inside of the meta Suite so inside of Instagram and Facebook and WhatsApp but it's also in a desktop application that you can use just go to news and Google search so the results are really really up to date so for example we can say what's the news today and go ahead and click enter and in just a few seconds it will list out the news from today they also have the ability to generate real-time images using the tool just go to the imagine with meta AI button and basically if you just type in the word imagine and then start to type it will create images in real time so we'll say a guy in Las Vegas at the Luxor playing a video game so there we go we have a guy uh basically at a slot machine at the Luxor and if you want to get images that are even better go ahead and click enter and it will actually use their AI image model to produce images that are even better and so there you go we have this guy playing video games outside of the Arena now is this as good as mid Journey no it is not but the the fact that this is available to everyone who uses meta's tools is really interesting it's not as Niche it actually is much more mainstream now now you may be wondering how creative this tool is well it's actually about as creative as chat gbt 3.5 not as good as four at this point so basically we can ask it anything we want so I'll say create 10 movie ideas and go ahead and click enter and you can see that in real time it's creating these ideas so it's really really fast there's almost no latency when it comes to most text based questions I was really curious to see how this compared to chat GPT so I went to chat gbt and asked it to create 10 ideas as well and it gave me a few ideas and then I basically copied and pasted both meta's ideas and chat gpt's ideas to a document and inside of this document I took meta's ideas and I just changed the name to Manny and I took chat gbt's ideas and just said Greg's ideas and then I took the document and uploaded it to chat gbt so when I asked chat gbt to tell me which ideas were more derivative it actually said that the chat gbt ideas were worse than the meta ideas now if you actually look at that document and I'll share a link below this video you can see that really in most cases the chat gbt ideas are much more creative so I will say that as a professional creative assistant chat GPT is your best option but for free options meta is the best out there the other cool thing about using meta's AI tool is the fact that you have the ability to have a multimodal experience so you can have conversations and it can generate artwork so for example we can say create an image for idea number seven and go ahead and click generate and you can see that it's beginning to visualize ideas for that film you know kind of a grungy uh film aesthetic looks okay it's not hyper realistic but it is cool to Envision the idea of having a creative partner to go back and forth with and you'll also see that anytime you generate an image you also have the ability to animate it so we'll go ahead and click the animate button on this video and you can see that it animated the images here and it animated all of them you know the movements they're okay it's not great this isn't uh Runway quality videos at this point but you know they do look pretty interesting some researchers this last week also came out with the ability to turn any video into a video game it's really interesting basically it will convert any video into a Nerf scanned environment and then you have the ability to play a few different games based on the scan basically you upload a video it scans the environment and turns it into a video game now they have a few different examples of video games so for example you can be a character that runs around and kicks a ball or maybe you want to knock objects over there's also a racing game and another game where it's basically like Mario Kart and you have to collect coins they also have a game where basically it converts the video into a 3D scan and allows you to interact with the objects inside of the room so it's pretty cool because you can go in and move objects around and create a little claw game in a virtual environment and that brings us to our AI films of the week our first film this week is called Amy and basically it is a 1950s TV show style aesthetic that uses Dr Seuss rhyming to tell the story of a robot and how it looked to have a heart it's super weird I love the aesthetic but I should also note that it's a little sad so you have been warned our next film is called the fliff atre by the real robot basic basically it is an AI documentary that follows the flif atrus in their life I think that the Creator did a good job at working around some of the limitations with AI and they basically created this puffball animal that can be replicated again and again our final film of the week comes from pzf AI that basically put together a very realistic AI project and many of the shots look pretty darn convincing the sound design really helps to sell but it feels like a liveaction film now there are some static shots but generally speaking I'd say they did a really good job thank you so much for watching this episode of AI film news again thank you so much to everyone who made it out to the Vegas event it was absolutely incredible of course you can always subscribe here on YouTube to get the latest AI tutorials and news directly here on the platform and you can subscribe over on fous refuge.com to get AI film news delivered directly to your inbox thank you so much for watching this week's episode of course you can check out our website if you want to learn more about the future of film making you're an awesome member of this community we'll see you in the next one

Channel: Curious Refuge

Views: 31,487

Keywords: ai film news, the best ai films, meta, meta ai, chatgpt, adobe, adobe ai, ai art generator, microsoft ai, runwayml, runway tutorial, ai video generator, ai, ai art, artificial intelligence, premiere pro, adobe video and motion, text in premiere pro, curious refuge, ai film, star wars ai, ai short film, ai generated short film, ai video inpainting, ai scene extension, caleb ward, ai videos, shelby ward, ai news, text to video, ai film gallery, runway, ai filmmaking course

Id: akBCj8vqkq8

Length: 24min 46sec (1486 seconds)

Published: Mon Apr 22 2024