ComfyUI + Mask + AnimateDiff - Emulate Runway's Motion Brush

welcome to House of dim I'm going to show you my compy UI workflow that tries to imitate what uh Runway is doing with motion brush it's uh work in progress but I'm I'm pretty satisfied with this so far somebody out there's going to be able to improve on this and uh I hope you do and I hope you share it all right let's go let's go the workflow real quick we got our checkpoint loader load Vie if you need it you can replace that here if don't need it this one does keep it empty lighten it your uh positive negative prompts your seed Cas sampler this is for the resize I tried to do it after the last step but I was getting a ton of weird noise so if I put it before it's fine um somebody smart to make it right figure out why that is this is your uh frame count do a 16 for testing the uh anime diff loader you can apply emotion Laura it doesn't have the same effect as you think it might but you can toy with it and we will mess with a little bit too this is where we create our mask and apply the mask and the prompts for the mask to the uh final output right here I have the uh my ad frames to smooth it out let's get started okay let's try my prompt here we missing mountains randomize que it up yep that'll work so we're going to mask this whole section here so that's saved and I I named it a mask template go load it open in The Mask editor do this this to move Save ceue It Up R our Q it's there it's real subtle but it's there and then for uh this is one I like where we uh we add some frames to it kind of smooth it out and we also bump it up to 24 frames per second you can kind of see it kind of flowing there you can do really goofy stuff let's do a tital wave title waves lightning yeah so that's what we got out of that kind of weird but somebody more creative than me can do something neat with that all we're going to load one up I used earlier think it this one I already have my mask in there let's C it there we go little fire I could try something clear this out and we're going to try to let's just get all of this together maybe we'll get like a little bit ction in there the do noise all the way up see if we get better results yeah that's okay replace that let's try something crazy like a lava ring out of fireplace smoke particles my a different motion model let's do that it's kind of cool not really pouring out of it though try something different let's try a toy boat can type in here toy boat in the ocean tilt shift yeah that might be cool never we're going to use that one load that one up mask it let's just get these waves moving here waves crouching not weaves waves scatching pretty cool all right let's see what this smot all right Let's do let's that this mask cover everything but the boat and see what happens kind of cool but the that top up there looks funky I probably shouldn't have done that one yeah let's try something different do the Galaxy in a bottle in Galaxy let's try it set our MK to the inside of the bottle now we can put whatever we want in there let's do Galaxy first Galaxy spinning yeah that's kind of cool there it is smooth we can do uh ocean waves crashing yeah I dig it pretty neat so you can do motion lore on this but I don't know that's going to have much effect let's try it just adds a little bit a little bit some different so that's it that's my take on getting the motion brush similar to Runway again I'll put uh the workflow in the description if anybody uh comes up with some improvements please post them so I can make this better but I'm I'm liking it so far

Channel: House of Dim

Views: 1,744

Keywords: comfyui, stable diffusion, runway motion brush, ai art, animateddiff, tutorial

Id: l7g58GrJIdE

Length: 6min 51sec (411 seconds)

Published: Fri Dec 15 2023