ComfyUI, AnimateDiff, LCM, ControlNet - Human Masking (Workflow by toyxyz)

hello hello hello hello hello am I live I feel live it's hard to say if I'm live hello how's everybody doing tonight think we're live looks good as I'm streaming you get the chat up on the other window here there we go how's everybody doing tonight we're here again in comfy UI doing comfyi stuff uh last night we rebuilt coffee vector's workflow for um masking out faces and then um uh well not masking up faces but uh you know replacing replacing people uh using control net and yes we're using line art control net uh normal control net and open pose control net to and IP adapters to take input footage like this uh impose whoever's in this picture on it and come up with something terrifying like this that's a particularly bad example but I think it's funny couple of Mr Beans in the car I have the wrong mug tonight I'm using my wife's mug so she's the best ant ever not me I'm only kind of a good uncle all right uh so yeah that's the workflow we did last night we had to rebuild it uh it's available on my Discord if you want it uh if you want to play with it it's there uh but tonight I want to go over another workflow by toy XYZ uh let me see where it is is it this one no is it this one yes okay this workflow is very interesting this one is uh it's for like masking out a person in a video replacing them and then leaving the um leaving the background as it was so uh we got our input footage here of uh this fella dancing and we got Mr Bean as our load image and and to our IP adapters and then we're feeding that into all the stuff and and it is coming out with this lovely thing now obviously um there's always problems with open pose when people turn around so yeah I was kind of expecting it to lose Co consistency when when the dancer spins but uh it just is what it is um these are still early days with all this stuff so um you know this is more for fun than you know professional music videos but uh I don't know you can be very creative with this stuff so uh let's dig in um I think I think the way to go about this one is probably to rebuild it I'll just pull it up on the other screen and then uh I'll rebuild the the whole workflow slowly um I think that's probably going to be the smartest way yeah then we can go over everything as it as it's sort of uh as it's used all right uh so let me start a new tab of comfy move this one over here and clear all right we have an empty space so we are going to start as always with the K sampler as it is the heart of our whole situation so we have our K sampler this is the thing that does the diffusion it does the deed it does the magic out of this we're going to go into a vae decoder so we need to decode the latent out uh we'll do a preview image to get all the images so we can see what they look like and then um pop a couple video combines on the end of this like we always do video combine at 12 frames per second uh file name prefix 12 FPS anime diff helps my brain MP4 yeah so those are the basic settings let's uh do a second one copy paste contrl c contrl v plug that in there and set that to 24 frames um same deal 24 frames but this one we're going to um no you know what let's save a 24 frame one and let's save a film interpret interpolated one not that one we want to open up film vfi which is available in the frame interpolation uh frame interpolation available in this um comfyi frame interpolation group uh very cool uh gives you all kinds of fun fun stuff for frame interpolation I saw new node there I haven't played with yet so maybe we'll try that one out too okay so for this uh this one we're going to interpolate it two times at 12 frames per second in and what that'll do is um basically give you like a a smoother version of this video half as speed but you know twice as smooth all right so that's the back end of this so let's go over to the front end of this uh I'm trying to think of the easiest way to start so this um workflow uses imp painting which is uh kind of confusing um so we'll go through this slow so let's add the vae for imp painting because what we need to encode stuff for imp painting we need to tell it what it needs to be imp painting we'll be imp painting The Mask or the person and leaving the rest alone and that's how this works this is the background video and this is the new stuff that we're going to be masking there was stuff we're going to be replacing uh so let's uh add a load our checkpoint uh we'll just use deliberate or something easy peasy let's move this way over here because yeah the pixels that come into our uh vae encoder will come from our video that we're actually loading um so let's load a video so this is um sorry you can see that load video this is VHS load video and it's available in the VHS node tack you load that up you uploaded a video uh we can go back to the dancing person sure we've got our dancing fella let's set our frame rate to 24 let's set the size to uh okay so this is one way to control resolution in your um inputs which is not a bad idea this Force size thing um what these mean these question marks it means uh it's going to resize it so that it's either 512 wide by whatever tall or whatever wide by 512 tall so if you want to make sure that it stays 512 height on the on the there you want this one and going to resize it so that it's whatever wide like in this case it'll be like 768 x 512 or 820 or something by 512 um and that's sort of like as low as we can go while maintaining the low uh you basically you don't want to generate any lower than 512 uh on either side with stable diffusion especially stable diffusion 1.5 so that way we'll make sure that we are Whatever video we put in is small enough to render fast but it's also uh big enough that it won't look like crap when it comes time to do it so this load video it goes into the uh vae thing for the background and now we're going to generate a mask with this uh this video now this is the um yeah okay this is a little a little much but let's uh let's go through it slow you're going to need the impact pack and KJ nodes for comfy I impact pack and KJ nodes those are the only two node packs you'll need outside of our normal uh packs so those are available in the manager or later if you go download this from my Discord afterwards you can just uh click install missing nodes when it's done so yeah we're going to open up the image batch we want an image batch to image list image batch to image image list okay so this is changing a batch of images to a list of images and a list of images is an array that um is required for certain nodes in comfi so if you've dumped a batch of images into a node and it starts complaining that it needs them in a list you need this image batch to image list node okay and now we need to load up some uh segmentation detection stuff this is in the impact pack okay ultr litics we need the ultr litic detector provider and we need the Sam loader uh bbox we want segment person YOLO 8 m seg and for this one Sam vitb6 yep Auto path yep okay so those are the settings for those two okay and then all this stuff is going to go into a node called Simple detector I think I think it's Sam no ebox detector no we oh oh impact simple detector segs that's the one all right so we plug our bbox detector into the bbox detector widget thingamajig s m into the s m and the Sam into the M Sam images into the images so hopefully that makes sense bbox to bbox segm to segm Sam to Sam image to image uh batbox threshold was 0.5 03 10 0.5 I'm just double- checking my settings here these look good those are just default settings so this is going to create our segments and that's what we're going to use to determine what's the person and what's everything else um so uh let's go with uh now we need to turn this into a mask so segs to mask is it com yeah segs to combine mask now we have a mask we're going to grow this mask with blur that's in the KJ nodes one and we're going to set this to 10 have 10 pixels around it zero tapered Corners are on flip input is off use Cuda is on set a blur radius of five pixels and sigma one one one yeah so those are the settings for the grow mask with blur all right and then we have our mask and the mask needs to go into mask list to masks to mask mask batch and then this mask goes into here so that is a very confusing way to say this video here is providing the background this stuff here is going to cut the person out and then it's going to tell it that this is the mask and this is the background we need that because we're about to change the mask but leave the background alone okay so uh let's add the in paint processor as well in paint processor okay so uh the image goes into okay we're going to save the in paint for the in paint node and that comes out of oh this comes out of the main video okay we're going to save this for control net time but we need this for the imp painting control net uh but we'll keep this down here for now and I'll pull it up when we need it later so that's um yep that's that six to mask try and keep this tight nice and tidy nice and tidy all right uh yep what else uh okay so now we're going to um deter in all the control net stuff the pre-process the control net stuff that we need to pre-process uh we're going to take the open pose from The Mask we're going to uh convert that to like a thresholded image and then we're going to um add a blur around it so that we're giving oursel enough room around the body and we're using control nut to inpaint the seams so that we're transitioning gently from the pixels on the image to the thing so it'll look less like a a like a cutout um and more like it's actually blended into the footage uh yeah so let's add the uh let's add the mask first so in order to do this we need to color to mask from this so let's add a color to mask image color to mask is that it is it image color to mask no it's color to mask there we go image in okay so this colored mask is going to be white so we want 255 255 and 255 with a threshold of zero and we need a second color and we're going to combine them so we've got copy and paste we got a color the mask and then another color The Mask plug this image sequence into here and so this one is going to be grayish so it's 100 100 100 100 and you want the threshold around 168 which is like around half a little more than half uh and now we need to composite these two masks so we need a mask composite we a destination and a source whoops top one destination bottom One Source XY Z and we want to use the blend mode ad all right and we're going to grow this with blur grow mask with blur just like the other one and let's check these settings we want 40 for the 40 pixels around the edges tapered Corners are on flip input is off use Cuda is on we're going to do a blur radius of 15 pixels a sigma of 1.7 and the rest stay on one all right those are the settings for that grow mask okay and then we're going to take the mask and convert mask to image why is that a two Mask 2 image not segs image there we go mask to image and then the image we can preview that preview that very cool all right so that is our initial mask we also want uh let's just move that over there and we also want to add another one of these convert mask to image nodes and we want to look at the output of this one yeah and then you'll have another preview image node plug that in there okay so this will be the preview from the first m we made this would be the preview from the one we just made and uh let's also do a DW pose estimation this is going to do the open pose stuff uh this might work it might not it's actually been uh breaking a lot in the last couple days uh not just me but other people too um we'll add a preview for that and we'll pop it up here and so now this will be the DW pose this will be the first mask we generated and this will be the last mask we generated using the open pose information I believe uh yeah let me double check that yeah Uno do Trace okay cool uh let's go this will be a lot less confusing once it runs I think uh so our imp painting is doing its thing the Imp painting needs a mask and the mask comes from mask list to masks this one okay our imp painting processor is doing its thing we're going to need to plug that into control net up here in a bit H we got our black and white mask we got uh our DW pose estimation and then our previews for the two masks beautiful uh what else we need we need to set up the rest of the diffusion okay so that all that crap won't do anything yet because it's um it's all for processing the footage and putting the footage into different places uh so here we'll build the rest of the noty noty stuff up here so we got our uh checkpoint uh I'm going to unplug that from oh it's not even plugged into model yet okay out of our checkpoint we need to go into our uh IP adapters so we want an apply IP adapter node there it is uh we're just going to apply the uh regular one yeah uh the image uh we'll do a load image this is where we can do uh Mr Bean or whatever we want I think I have Mr Bean here somewhere Mr Bean there you are all right we got our Mr Bean it's important to always use Mr Bean I'm just kidding I just like to prep image for clip Vision okay we use this prep image for clip Vision node um because then we don't have to crop or scale or do any fun stuff with our image we can drop any image we want in here and this will take care of uh making sure that it's cropped wherever you ask it to uh and is got different kinds of um interpolation if you're really nerdy about it but it's just a quick way to pre-process your images before they go into thing and it saves you a lot of time uh the other nodes we need for this are clip Vision uh so we need uh the sd5 clip Vision link is on my Discord uh so if you just Google IP adapters plus um The Comfy UI thing all the links are at the top of that we also need IP adapter loader so that's the basic setup for a um yeah that right there is a basic setup for a uh IP adapters rig yeah my checkpoint goes in okay so yeah you load whatever image you want in there the uh AI will look at it using these uh clip Vision models and the IP adapter model and it will uh add that to your prompt so it like um yeah it's really sick all right so this model goes in here and now out of this model uh we need to go into animate diff so let's add animate diff loader no animate diff loader with context we need this next OKAY model goes into model context options drag out of that and let go and you want this animate diff uniform context options node and that's going to give you your context options which you're just leaving alone if you for the for the beginning that's fine uh and then we're going to add add a load Laura Laura loader and this is where we load the LCM Laura corresponding to uh the version of stable diffusion that we're using so if using stable diffusion 1.5 checkpoints you need to use the sd15 LCM if you want to use sdxl you can use the sdxl one no problemo and you can add this model sampling discrete node and change it to l CM I don't know if this is actually necessary but I've been doing it and it feels good and then this finally goes into the K sampler so our model has done some gymnastics here but it starts in our checkpoint loader here right find a good way to lay this stuff out starts in our checkpoint loader goes into the IP adapter into animate diff loads the LCM Laura and now we can render at like six steps instead of 20 which is awesome because it takes way less time I think it's running ads right now so I'll wait a few seconds before tun in the next thing good old YouTube and so the next thing you might be wondering is why haven't we touched the clip stuff yet um we'll get there relax geez calm down let's color these nodes before we go any further so that our brains don't explode uh image nodes are blue uh I do loaders as brown uh animate diff stuff and um clip and all that other stuff control net I always make yellow uh load Laura that's purple or sorry that's brown uh model sampling discreet is a has a do it to Fusion so I guess I'll make it I'll make it green cuz it's settings green and apply IP adapter is it does with images but it has more to do with diffusion so let's do purple all right case sampler diffusion purple vae encoder vaes are red vae decoder vaes are red image stuff is all blue all right uh this stuff is all image based so let's make it [Music] blue y'all are all blue too colors blue and DW POS estimation is yellow because it is control net related load video is blue because it is uh uh the stuff is all [Music] images yeah you let's go with images blue blue blue blue blue uh in paint processor you're actually Oh no you're blue too because you deal with images cool okay that's a lot less awful all right next [Music] um uh let's do clip okay so what does our clip go to well first our clip goes to a clip skip we need a clip set last layer set that to1 or two if you want this is the um clip Skip it's yellow because it's clip related all right after the clip skip we actually need to encode some clips those will be green and they will be red because they are [Music] our positive and really there we go positive and negative prompts what the what is going on today positive all right cool positive and negative prompts positive and negative prompts uh normally would just go straight into the K sampler but tonight we'll be using some control net we need two control nets for this uh we need the in painting control net and the open pose control net all right so let's build them NSFW nude into the negative because we're on YouTube and let's do Mr Bean as the prompt imagine that all right uh and uh okay so first things first let's add um the two apply control net nodes we need uh you want the apply control net Advanced nodes for this so we're going to loop our our Clips through negative goes through the negative or positive goes through the positive negative goes through the negative this don't do what I'm doing here do do it right and negative to negative okay so positive goes to positive positive positive negative goes negative negative negative negative scoom all right so that's our two control Nets uh we need to load loaders for the control Nets so we need the Advanced loaders with time step that one yep so for in painting we need time step because we need to tell it to only do it when it's in painting not when it's uh not when it's doing other stuff so this is a little confusing but we need to add a scaled soft control net weight you need to leave it at 0.825 flip weight's false time step goes into time step uh this goes into this this is fine we add a control net here we're going to add uh another control net loader this one doesn't need you don't need to use this but uh whatever um and this one we're going to use open pose so we've got our imp painting and our open pose you don't have these control Nets just grab them online Google control net sd15 and only download them from hugging face you know or civi TI but hugging face is safest okay control net is control netting I think uh VA encoding is going well everything's plugged in h hunky funky Dory you don't have a latent image yet you're going to need that from the vae encoder Boop so our VA encoder goes into the lat image of the case sampler now uh we need images for our control net to understand what it's doing so uh where do you come from image so I want the image from the maint processor into the uh uh apply control net so remember that in paint processor we made three months ago down here yeah let's pull that up here and this images comes from this and then for the final we actually want to pull from straight from our uh video input so this goes all the way up here all righto okay that's the entire workflow uh I think we'll find out if we get an error when I go to run it but just make this stuff all yellow all right okay uh you want you want a 12 one as well sure why not ain't gonna hurt nobody in all right cool that much room I think just move you all over okay let's set our frame load cap to just 16 frames just to see if this thing even works and we'll set every tell it to select every other frame so it'll be a little faster it'll run it at double time right uh I forgot two things I forgot to plug the clip through the Lura loader boop boop yeah and then I forgot to send this what is you oh yeah clip Vision all right why you over there all right this vae needs to go into our decoder as well you see that other error we had it's cuz I didn't plug the uh vae from our checkpoint into the decoder okay uh why are you still running a prompt oh you're doing open pose okay cool all right let's run that again okay our mask might be a little out to lunch here I'm not sure what's going on there but we'll look at it in a minute actually might be all right we shall soon see right what did I do animate diff oh I've got the wrong anime diff model loaded uh you should load an anime diff model that is related that is works with the uh platform Ian I'm using stable diffusion 1.5 so um yeah that's why that didn't work holy crap I think it's working I don't mean to sound so amazed but last night was uh not this quick [Music] okay I guess I could do this is less confusing all right cool um TI open pose you are a group title blur mask threshold mask video goes there uh well nailed it all right uh let's have a look at all the settings and make sure everything's not insane uh did I miss anything let me check these compare them up all right uh 40 Z true true blue radius 15 1.7 yeah you're good [Music] good 10 0.5 03 10 0.5 0.7 10 0 true true five one z z yep yep uh ah yes lower the IP adapter weight a bit just trying to make sure my settings are the same as my other my other one L CM Laura yep yep yep POS base multier yep one0 one one0 one let's try that again VI might have went a little too hard or the IP adapters might have went a little too hard on that uh looks like the open pose though maybe wasn't uh oh you know what I think I screwed up I think this I know why this open pose out goes into the open pose in I think I sent the actual video in there that's not good so let's run that again uh can you invert it so you do the back background and not the dancer yes probably um not quite sure how off top of my head because we're using um open POs to do the person so [Music] yeah yeah I sent uh video straight into open pose which won't work we prepress we pre-processed it we might as well use it now it should stick to the body like it makes sense this is creepy too just to imp paint a crazy character into the area with no understanding of what the body's doing oh I even have the I have it in the wrong um using the wrong settings for the thing too okay here we go this is working uh okay I need to set this to six steps and I need to set it to LCM uh duh let's run that again this should be a lot faster oh CFG is also way too high and we set it to two for LCM two or three I think that was with a super high CFG so now this is with a lower CFG this should be a lot less over overcooked he looks like Saddam Hussein now Mr Bean there we go Mr Bean dancing on the beach we did it we freaking did it all right let's add a group for these let's add a group for the [Music] defusion Imp painting decoding [Music] review don't actually need this because this doesn't work if you save yeah it's just the image yeah it doesn't save the batch this is useless all right cool uh this will be diffusion and group okay title diffusion title output a group title this is open control control net 2 and this is control net one uh imp paint free processor you can actually go in here because you're kind of part of this kind of part of this sure all right uh clip edit title pront edit title animate diff threshold mask open Poe BL maskk info video uh you are loaders IP adapters damn all right cool it's not as scary with the groups let's give some space between them here so we can yeah yeah a little space so you can see the connection cool all right we are fully connected uh I guess kind of like is outside of the whatever it can live here I won't get too OCD about this okay uh diffusion output control net one two prompt animate checkpoint IP adapters threshold mask and the blur mask input video open pose processing control net yes not so scary right not so scary okay uh let's see what it does with other footage uh I think I have the footage of the ladies driving in the vehicle uh we can try that yeah Scoopy whoop all right uh let's do 16 frames and see if we can get Mr Bean driving in his car now the really cool thing about this is um we're rendering at the input footage uh resolution so if we want we can actually add um an image up scaler on the end of this uh to render higher with the same input footage uh or just you know uh disable the um the force size or set a custom width and height and then you can just set the resolution of your uh whatever you want to render um this thing here with the force size will basically it'll squish the image down so if the image is bigger and you want to run a higher res animation then you you just want to set Force size to a lower or the same resolution easy peasy these masks uh I wonder if this 40 is too high for the expansion mask here I'm not quite sure I also don't know where this goes after it goes here it might have screwed something up grow mask with blur goes to oh yeah grow Master blur goes to mask list of masks oh did I plug the wrong one in no I have to look at the original thing I might have screwed up the this whole part cuz this doesn't go anywhere I don't know what we're doing with this blur mask yeah I have to look at the uh the the tutorial again I think I screwed something up it doesn't matter it seems to be working there we go Mr Beans driving down the road everything's fine totally normal so if we back off on the uh uh open pose control net a little bit it might give the uh AI some room to wiggle around and and make them move a little more we'll see but I mean look at this iteration time six steps right like it's so fast once you've generated the U once you've generated the open pose like footage and stuff it so fast oh good it just made his eyes terrifying I love how there's two of them okay let's try a different input [Music] video uh we can try a different uh input image as well what happens if we make me ride a motorcycle 40y old hipster riding a motorcycle some fraking nerd riding a motorcycle uh I'm only doing 16 frame runs because they the DW pose stuff right now is actually the biggest uh the biggest um bottleneck on this like there's something wrong with it it's only using CPU so it's like uh crazy slow crazy slow for [Music] wow that's a lot better than the ones I was getting earlier huh let's get that control net back up to one let run that one again I mean it looks nothing like me but it's a human person should I drag the input video over so you can see it's pretty good with the control net or the open pose let's do more frames uh let's do 96 frames where do we do that on here 96 instead of 16 give it a chance to jump out of context window and see what happens uh this is going to take a little while while it re generates all the DW poses so um get up grab a drink uh stretch your legs uh or don't I'm not your mom I'll be back but I'm going to do all that stuff I'll be right back oh yeah I got a drink all right is this thing done still DW posing all right all right what are you guys working on tonight don't everybody talk it once let's turn my camera back on soon I'm just uh rolling a naughty stick things are happening whoa not sure what it's doing why do it see two people oh we're still on the motorcycle footage right thought we were still on the dancing footage train going to change the background of the video I mean uh I think it would just be a matter of reversing this process yeah I'm telling it to only imp paint around the edges as opposed to inside the mask we can try and figure it out after this should be somewhat possible [Music] all right let's kill this animation before I forget to do it later byebye all right render render render we're doing 96 frames now going to take a little longer but we'll get 96 frames instead of 16 also maybe I should save this workflow while it's working masking animate if LCM control oh and I'll uh oh sorry let me move this over here yeah and then we'll save color yellow checkpoint color [Music] blue color blue color blue color [Music] blue color yellow yellow diffusion is purple purple and output is blue hey all right so H you know before I upload this to my Discord I think I'm going to figure out this little bit here and what I'm supposed to be doing with this blur um I guess I could just update it later CU it does work whatever nice it's doing a pretty good job over the context Windows motorcycles are changing into different motorcycles I think I switched to shorts at some point yep what it's pretty cool okay uh let's look at the mask so this if this is what we're masking we want to actually just uh imp paint the opposite of this so like I wonder if uh yeah invert mask invert mask Inver mask yeah uh uh so this will probably break the uh open pose stuff but I think if I'm right yeah so now this should just imp paint stuff around the person so you just really want to use this invert mask thing here I'm not sure what this is going to do though this should be pretty interesting actually I should probably do one with a no open POS access yeah yeah for I think the thing to do here would be make a background masking version and then use the depth mapping on the background we can try that uh I'll save this as a uh I'll save it as it was originally and then we'll Branch off and try and uh turn it into something that only affects the backgrounds it should be Zen enough if I'm right this should only imp paint the background and not the the the subject um but the prompt is also somebody riding a motor motorcycle so I don't know if it's just going to like I don't know what it's going to do still prompted on the subject so who knows yeah there you go it sucks because it's not doing anything on the background but it's definitely working super trippy so uh in that case let's remove um let's remove this mask and plug these guys back in master mask Master mask okay so run this make sure this works this should be normal yeah okay close oh uh cancel we're going to save this one just to have it and now we're going to oh look we got a mask inverted super sick you don't even need that invert node and then instead of open pose here we're going to use um depth and uh apply that at one and we're going to Zoe map it and that's going to go into the depth control net yeah yeah see we're going to make you yellow all right uh that now that should depth map everything around and that should give us something to play with so let's prompt for the background not the person and we don't use IP adapters or if we do we use it to define a background of some kind uh no we're just going to turn it off let's just prompt a new background uh riding fast through riding fast on a Country Road Middle of Europe let's try that so I screwed up or did I oh no we're good we good we good this one's still doing nothing madella Road all right now we're depth mapping the background so this will be our mask you know it's all right there probably some stuff we can do to fix that up oh hang on hang on hang on hang on hang on hang on hang on no no we [Music] need so we need uh this might be fine this might be fine I was thinking we would need to actually mask this out as the threshold uh but maybe not maybe having um having it is fine we'll see R rer render what here we go final step so this should be taking the depth map information in from the original video and using that to control net the background and then it hopefully will only do that on the background and leave the motorcycle person motorcycl maybe or maybe it breaks let's find out oh that's cool that's really trippy that's really trippy original footage that's really neat you could go even deeper do you could you could add more [Music] um um you could add more uh more control Nets I think if you got the line Ard in there as well yep all right let's try that out um yeah it's already got Zoe so should just have to do line art that just looks cool as hell on its own all right then we'll save this one as the uh background replacer and I'll drop them both on the Discord [Music] tonight uh no sorry you can it's just incredibly slow and a lot of stuff doesn't work right uh you can run it on the cloud though like on runp pod and stuff but um you know cost money yeah you're [Music] basically [Music] one minute I think with linard it'll be a lot more detailed um but it'll be probably closer to the um closer to the original oh sure I have a I5 uh 12th gen I think uh with 3090 and about like 32 gigs of vram or 32 gigs of RAM 24 gigs of vram uh right now this process is using about 14 gigabyt of vram so your mileage may vary I'm also streaming on Twitch I'm doing some other crap I got Discord open uh you might be able to get it under 12 but yeah this stuff's around 10 10 the minimum for animate diff and then once you start adding control net and IP adapters and all that other crap vram usage starts to increase and also resolution and all the other things what's up Bo man po a man I used to love that comic the way he would like shoot and then just sort of fall or drag himself into the location so this has line art as well so yeah the lines are definitely it's more rotoscopy it's cool a little halo around the people and then the other people flick but it's pretty interesting let me try that with the uh you know the uh where's my ladies driving yeah let's try that yeah I mean with the mask I guess we don't really need to mask out I tell itjust to to do the line art and stuff on the white areas because it's kind of better to have all the uh control net on the on the original frames for the depth and line art um because um The Mask will cover up anything this should be cool don't quite know what we're going to get but uh also I should open a new tab and look at my other work worklow and see if I can figure out let's go back to the tutorial toy XYZ uh is that new highres fix I got to play with too oh cool somebody made a that's neat somebody made a uh hdri thing for um anime diff very cool uh the [Music] overose what did I do wrong with The Masks here so my mask goes KJ knows Mas list to [Music] masks weird mask image MTB node why is that different oh where do you go oh so this mask is supposed to go into mask optional in the apply Advanced control net node but I don't see that option interesting I'm using the wrong ass control net thing okay we'll play with that on the other one that's interesting I see what I did wrong uh yeah uh yeah somebody made a because it's animating the hdri you could load that up in blender and then like you've got a Trippy world that's you know backgrounds that are animated we're doing blender animations pretty clever the underwater one is sick I got to make a bunch of these These are cool as hell I have to figure how to do that too that'll have to be the next one probably figure that out on stream actually got to be a way to draw a mask in right text what if I want to draw my own mask right in comfy okay uh maybe there is no way to draw yet and comfy oh that's cool so trippy to have like real people driving through uh animy diff land it's cool how it's totally replaced all the stuff hey Serge what's up bud all right well that's dope so let's save that as the background mask anime diff LCM control net we got our background mask let's get our human masking back up because I wanted to look at what we were doing with this broken ass stuff over here okay let me have another look at this image here okay so I'm going back through uh Toy xyz's workflow here and I'm trying to figure out what I've done wrong what I've done wrong so images come from the right place let's get get you back over here so I don't go crazy yes thank you okay bloop bloop color mask mask composite yes mask go to grow mask with blur yes that grow mask with blur goes out to ma convert mask to image yes but it's also not working so let's try mask to image I don't know why he's using an MTB node or if that matters that's the image see do I have the MTB nodes installed no I do not okay that's fine uh fine uh but it needs to plug into the mask optional of this one Advance control not yet not you so much no you so let's go positive to positive negative to negative same deal here control net comes from here the image comes from um open pose comes from the open pose image which is here bloop and then the mask comes from here mask optional yes yes okay uh but uh let me look at the other thing make sure it's plugged in right what we got tics just double checking all my settings here yeah that's all good let's go down uh pose estimation oh wait a minute I know why it's being weird I I think this comes from here I think this comes from here whoa sir get back into your spot okay okay does that make sense let's see what that does I need a video of someone yeah I need this uh let's do the first uh 32 frames so it doesn't take five [Music] ever okay uh we do the DW POS animation do estimation it feeds into this blur mask we get this blur mask we use that for the uh open pose interesting is that really how it works well we'll see and then yeah I think everything else is set up right we will soon see that looks corrector more corrector nice okay ah okay it's CU we're only getting the body after a little while all right that's fine I think that that did what it was supposed to do so let's see twerk twerk twerk all day long twerk twerk twerk while I sing this song uh I don't actually know what the prompt is 40-year-old hipster riding a motorcycle with my face as the input so uh this should be pretty weird wonder if I'll be dancing with the motorcycle here won't pop it's driving me crazy just needs to [Laughter] pop that's good oh why oh why why why I love it but why okay let's try um 40y old Miami dancing uh dancing yeah let's try that again yeah celsium uh six steps it's fast as hell once it gets into the sampler man look at that sixs is [Music] crazy I'm thinking what's plugged into the K the open pose though the actual open pose plus the [Music] mask I not need the mask might not need the mask we see something's off something's a little off open pose h [Music] it's just being really weird swear I was getting better better results without that mask [Music] yeah that's just being weird as hell did I not feed at the open pose yeah I did huh I swear I was getting better results yesterday wonder what I screwed up let's try uh Joanie [Music] baby this threshold goes Oh weird hold on vae encode for imp painting this mask comes from mask list of masks yes why does this come from two places it goes to two places imp pay preprocessor yes also the vae encoder yes yes uh this comes from chromas blur right yeah okay so that's correct or is it yeah it is I feel like that's not going to work at all h [Music] this doesn't make sense I don't know why it isn't better wonder why wonder why wonder why oh with those line art works really good let's try I any footage where people are like running around and doing stuff try this dancing dancing let's do the other way 512 by X all right let's get 32 frames of the jazzer size here and see if we can turn it into me it should be able to pull this skeleton I think with minimal problems because this ain't it whatever this hor horrible monstrosity is no clear for you maybe sets My Brain on Fire curly noodles set my brain on fire ah here we go I think this is more along the lines of what it's supposed to be doing you know if you have good input footage yeah because it it's going to imp paint everything inside this right do do do [Music] hey consumption what's up going on buddy we're making Grandma dance right now with my face hopefully oh hell yeah so there it is I mean I mean it's beautiful I I just don't I don't know if we could even do better than this I'm going to do even more frames how many frames do you think this could even be let's do the whole shab bigly Bango I should have checked how many frames this is that's pretty sick I mean with the with the background removal too like I mean I know the background is just white but it's awesome it's awesome all that from a freaking photo no idea how many frames is going to be I figure it can't be more than like 120 it's at three times speed all right well that all makes sense now so control net one is the open pose right no it's in painting yeah the mask comes from here you plugged in okay okay uh let trying to clean this up open pose is yellow you are blue blow you are purple you are brown you are yellow prompt is [Music] green control net is yellow yellow diffusion is is purple outputs are blue cool all right and DW still going LOL yeah I mean uh regular open pose is faster right now but it's not as um in depth is there a way to see number of frames throwing up video not really uh you can do it with FFM Peg and there's I don't know I've been thinking about building a little python thing to do it but I'm also lazy but it'll tell you once it's done pre-processing if you got it in a preview image box it'll tell you how many frames the batch is which is 104 in this case so it's going to do 104 frames uh once it gets through the animation stage here we're doing our uh simple detector which is going to do the segmentation pull the person out you can use this to do thresholding uh if you want to threshold an image between the subject and the background you can use this threshold mask uh any of these node groups that you want to save and use again later you can select them all right click outside and then you can say save selected as template so then you can say um human segmentation so then later in another thing I can go to node templates to go here human segmentation and I've got my thing for doing it and I just feed it the image from whatever I want to feed it and feed it out to wherever it goes it's one of the things that makes comfy I so powerful it's just you know you can modularize all these little things just drop them in when you want them and go they'll use greens for the loaders it's a little easier on the eyes yeah probably don't need to set one uh after it's done loading if you look at your image preview it'll tell you the amount in the batch so I know this is 104 frames because of this and also once it starts rendering it'll tell you up here it says I got too many frames so I'm using context window of 16 104 so it's telling me that there's 104 frames in this video um if you want to do it beforehand you can uh load it up in like a video editor and switch it to frame mode and see how many frames you want cut how many frames you want out uh in this case uh I know that I'm skipping every uh like I'm I'm only grabbing every third frame so this animation is actually 104 time 3 uh long but I'm only pulling 104 frames because I'm skipping two frames every frame if you want to render a certain amount of frames from a video all you do is edit the frame load cap to whatever you want so you just say I want to run 16 frames I want to run 32 Fram it only grab the first whatever of that and then you can use this skip first frames here to go to a part in the videos so say I want to skip ahead to 69 frames and then render 32 frames from the 69th frame that's how you would do that so you can just render out little sections for testing and then you can render out the whole thing when you're done tonight we've been mostly rendering 32 frame segments so that we can uh see what happens over a context change and it's fast enough it doesn't take this long to render every time there we go not bad did I drag it yeah I dragged it there we go pretty cool right on well I'm going to clean up this and I will post it on uh uh I will post it on my Discord as well as the background removal version um if you want to get those you can get there they're all on Pur at XYZ uh if you like these streams you want to support me I've got a patreon you can slide over there patreon.com Pur um supporters get access to a special channel on Discord and um uh but for right now all the stuff is free and and and open to everybody all the packs and all the uh workflows so uh just join the Discord you get all that stuff for free the patreon is just for people who want to put put some money in the tip jar which I really appreciate helps keep this stuff going um thank you so much everyone uh I uh will probably be back online Tuesday night uh I'm going to work on that LCM live webcam stuff uh tomorrow and uh if I get it working I'll stream about it and how to set it up but I have a feeling it's going to take me a day or two to get that uh really lined up so I'll work on that and then uh yeah we'll do a video on that live webcam and oh am I muted is my thing muted should be coming through you guys hear me hello words hello I muted hello I just did a whole long spiel okay cool yeah so patreon.com Pur if you want to help out uh Discord and yeah I'm going to work on the LCM thing uh I want to set up the webcam mode and I want to set up the live painting mode uh so we'll figure out how to get that going and then um uh yeah I'll either stream about it tomorrow night if I get it all set up and if I don't get it set up on uh I'll stream about it on Tuesday after I get it all set up uh but basically yeah I I I think I think the path to that is uh CR and a and a plug-in and then uh running the LCM through uh comi in server mode so uh that'll be hard for me to set up so it'll be probably hard for you to set up too but we'll figure it out together and I'll show you how to do it once I know and you'll be able to uh do live webcam LCM stuff and live drawing uh LCM stuff which is all the rage right now I mean it's really cool it's really cool um so yeah uh have a great night everybody thanks for hanging out and I appreciate you all we will see you tomorrow or Thursday deuces

Channel: Purz

Views: 1,611

Keywords:

Id: vzSTi8Y0TVI

Length: 121min 40sec (7300 seconds)

Published: Mon Nov 20 2023