A VFX artist’s perspective on generating AI assisted images with ComfyUI

how you guys doing so today I'm going to go through a video on uh primarily uh the use of AI generated images in a visual facts pipeline uh this video will be more catered towards people in that industry in that background uh but also people just jumping into AI generated IM imagery uh will probably at least get some insight on how to better control the elements in the image and also just some of the the work involved on the on the visual effects side and uh 3D tools and 2D compositing tools so my intention with this video is probably going to be a multi-part thing where U I'll go through the entire steps of producing a finalized shot um a visual effect shot using AI generated uh diffusion models this will be like a precursor to a lot of the more things that happen on the AI side uh and so what I'll do is I'll I'll jump into 3D and kind of like the the basis of like how this thing was created how was rigged and modeled and how the how the scene was set up and layout and and all that stuff the passes um some intermediate steps in uh compositing uh not really compositing in a sense of how is done in the visual effects environment but more intermedia step from getting elements from 3D that are more aligned for use with uh AI generate content and get those into a tool such as comy UI which is using uh lower bit depth images of 8bit so the workflows I'm going through are help mostly helping with art Direction uh currently like if you're trying to do this stuff with just text prompts you're never going to be able to describe for instance something like this kind of robot thing crushing all these cars and stuff so you really need to rely on EX external elements to go into the graph and use that as a way of driving everything so I'm not going to cover animation yet because well frankly there's no workflow that exists right now to be able to do that but I feel like in a bit of time couple months probably less than a year we're probably going to be able to see models that can handle that and what I'll do is come back to this whole scene and and work on it again right now I have the start of uh a workflow for animation but it's not it's not working yet and I I'll show a bit of uh the details of that later in the video so let's jump into Maya and 3D and start from the beginning so inside of Maya here's the scene so we have a hexapod robot and it is walking along these cars so if we go through the camera here's the uh camera this is what gets rendered let's press play and there's our shot so this 120 frames and a little bit of camera shake and yeah the uh the actual robot here is actually quite interesting how it's rigged up so I'll actually jump into that and kind of show you how all the stuff has been set up so I had to 3D model and rig this uh Spider the 3D modeling was done effectively by Kid bashing so I bought a I have a number of uh these Advanced looking technology kind of kits and then that was a combination of Kit bashing a bunch of that stuff with some nerve surfaces so like these these uh smooth looking things um same with the back here that was all just nerve surfaces and then yeah combining all that stuff into polygon geometry and and then finally I threw it all into zbrush and uh did a I just basically combined everything and then decimated the results so as you can see here like if I go inside and let me turn on two side of lighting you see that the inside of this thing is all it's not like a ton of intersected geometry and this just saves a lot on the poly count so I went from uh pretty much a 10 to 20x reduction in polygon geometry and that just keeps everything nice and fast in real time so for the hexapod rig the legs I did the legs in their own scene and then that way I could reference them in to the hexapod body and then that way if I make any changes to the leg rig I don't have to do it multiple times the ik set up for a single leg it's not trivial cuz the main problem is uh each of these joints has to move in one axis in order to for it to work so the main things to get that working was this joint here it's effectively constrained to this ball right here right there and it's kind of pointing towards there and that ball is also the uh pull Vector for the ik system and then that thing is kind of it's kind of moving along this is not like a perfect solution cuz if I do go above it does flip the hexapod rig itself is fairly straightforward except for the way the body moves so you notice that when I move the legs the body not only translates but also rotates and you get some interesting uh complicated movements by just moving the legs around so you don't necessarily have to animate the body so much but yeah you can and yeah this is a good example of like like how all the legs work all the little individual ik all the joints and how they're constrained and stuff and yeah and then all the uh the extra motion you get by one of these legs on the body control I have a parent group here and you'll see here that I have in purple on the rotations I have expression drivve in that and then in blue I have a point constraint the point constraint is driving that Group by uh Point constraints on these controls so you can see here it's all weighted uh between them all so that way as I move them around uh it's going to it's going to like average the position of uh all those points the expression in the expression editor right there uh this is basically just doing linear regression and it's taking all of those those controls right there and the location here is you need like a an origin for this whole thing to work or else it's going to like it's going to have this like swimming effect through it's not going to look right um and then yeah yeah so taking all those those points go into an array Vector array and yeah and then here we have the loop so this is just adding a bunch of this stuff up and then here's the actual linear regression formula and then this is the output back to those two channels on the rotations so I am by no means an animator I've rigged hundreds of assets for films and uh I think I've only had to animate one or two maybe Assets in a film but so here's my take on how I came up with the animation for this thing so the C only has one asset to be animated which is the spider and that's what the animation looks like 120 frames so the way this works uh what I found so this is a a specific walking gate for a hexapod called a uh 12-step I believe it's called a ripple 12ep Ripple and that can be done by first making a tri tripod gate which is three legs pick up at the same time and then they go down and then the next the opposite three legs go up and then what we can do to move the tripod which is a very simple looking gate to something that looks more organic like this is to offset the leg animations so the uh legs in the the the first leg of the three that one you shift all the animation forward a bunch and then the last leg you shift it backwards and then that way you're you're offsetting those three legs of when they actuate and that gives you this kind of motion other than that to add some weight to make a look like it's kind of heavy uh some things on the uh animation curves themselves so here's the animation for a given leg and on the Z which is the most important uh it starts off flat so it kind of ramps up its motion because it's heavy and it's you know can't move instantly and then when it hits the ground it's you know it's going rapidly and it just stops and the uh Center Point it was all shifted uh from the center and then waited over to the end because it should take longer to lift itself than going back down like going back down it's going to be going with gravity and that gives you this nice motion where like it it lifts up quickly and then it falls down or it lifts up slowly and then it falls down quickly to um give it more of a more organic kind of look to it after figuring out the walking gate I just pretty much just offset all the legs so they're actually touching something onto the ground whether it be the ground itself or one of the cars I was thinking of uh trying to incorporate animating the cars themselves so when the leg hits it it would like press down on the car but I I decided not to do that keep it simple and see if I can get the the AI generated video to do that which I wasn't able to the whole point I wanted to do was to make as simple a scene as possible to convey what I wanted so effectively like a previous animation and then that way try and get the AI system to see if it could do you know the hard part so what that meant really was basically just work with geometry and no texturing anything uh no Dynamics no none of that stuff uh the lighting was kept basically really simple just like a overcast day just like an ambient occlusion and kind of look to everything what I found out was I don't even really need to render a beauty pass cuz uh with this stuff uh the sketch pass which is like a tune Shader and the zdepth is pretty much all you really need to give the AI uh diffusion model something to work with so on the lighting side uh I got a number of passes the primary ones are your beauty your your mates and uh tune Shader and let's just see what that looks like so there's the so one of the things you want for the system is you want all the edges of things so I use a tun Shader and that gives me this kind of look uh the silouettes are a little brighter than the uh the details move in a bit something like that so you see like all the little kind of details I wanted them to be a part of this stuff to have an effect but I wanted them to be a lesser effect I wanted like the Silhouettes of objects to be the most important factors cuz like if that's a car I want that car to be a car and not kind of something else so on the color side if we go over here and we render that it's basically gray shaded and it's got a sky BLX so you it's kind of like an overcast day and that gives you all those kind of details and that's how that looks and then the the last one was uh and also on this color pass I have aob so I need to have Zep obviously so I render that uh aov on that mask on that pass and then to isolate objects so I have the city in blue some cars in green and then the robot in red and that gives me that the uh what I found the oh come on Maya so Maya crashing as we all know happens and so far at least I found with my super limited use of Arnold is it's very buggy and crashes all the time and what I think is probably a better solution is to instead of rendering stuff like I'm doing uh Ed of Maya with Arnold or another renderer U bring all this stuff into Unreal Engine and do layout there and then add in like Photo realistic assets from their uh free asset store and and kind of build it up that way that way you can get something that looks a little better than a gray Shad model and then you have you know you can have more our direction that way like if you need a puddle to look to be here or a bunch of debris or a tree in the background or or whatever that's far easier to do in Unreal Engine than it is to do in something like Maya not only that but rendering an Unreal Engine it's real time or near real time and in something like Maya and Arnold you're looking at like 10 minutes a half an hour to do the same thing and this whole system does have a lot of computations on the AI side and I think it's better to be spending more time on that side letting it do that stuff than uh spending time rendering elements to get to that side so before you can get to comy youri and do all that fun stuff you have to convert all your exrs into something else like pngs or MP4s uh Comfort UI it only reads uh 8bit images or video and yeah so you got to convert your exrs so stuff like zdepth unfortunately you can't bring in zth correctly with like actual values of pixel depth you have to compress all that stuff into an srgb image and so instance on here here so what I'm doing is bringing in and also another thing on the zdepth for com UI uh closer objects need to be white and then further objects needs to be you know darker off in the distance so you have to you have to invert it that way so converting the exr down to srgb I use a grade node and then I I adjust the black and white points and the gamma to get it into like the range of an srgb and I'm sure there's a better way if you're a new compositor tell me how like what's the best way to do this cuz as you see I'm I'm not I'm not even in Nuke I'm in N uh Natron um but yeah so like the main thing I found with uh trying to get good zde into comi is it really depends on what object you're going for so for instance you see that I clip the like the blackest point is like at the distance of just past this car if I had the Black Point to be like way down in the distance like if I had another zero here down there and I haven't even reached the end of this uh Street you'll notice that the robot is completely wided out and in srgb there's there's basically no detail uh there so and I and I know that when I'm working on this robot I want like accurate depths of all that geometry so I brought it in more and you'll find that depending on what asset or element you're working on comping in comp UI you want to have different depths uh Z depths for those things that are particular to those objects initially when I started out I produced uh image sequences of PNG files and I read them in that way but I think the better way is to use video files with comy UI I think it'll save on space uh memory space and uh I think it's just an easier way to work that way and the best way I found to get in that way is to um to use uh h265 high efficiency and um and lossless so if I go to the Advance see lossless but the main thing is to change uh to RGB and that way uh it like if you're doing a single unpacked uh image like this one it's it's not as important but if I'm if I'm packing in for instance the uh edge with the Z so like if if this is on channel R and then I had like lighting or or something else on on G and then and then uh the depth on on my last Channel and I did it all together if I don't encode in RGB they're going to uh like if I'm going to use like yuv they're going to bleed into each other and then you're going to have issues where like the color is going to show uh edges like the edges are going to bleed into color and all that so uh you just want to be cognizant of of what you're writing out and the good thing is like the when you're doing it losses this way these files are not very big they're like 50 megabytes or less and it's not like you can do like throw in like a 2K or 4K image com UI and be able to do anything with it cuz it's just too high resolution um these images uh as you see here it's it's like 720p kind of resolution so here's an example of one of those high efficiency MP4 brought back in so the RGB all nice and multicolored but then on the r channel on red we got the edges green we got lighting and then blue we got the depth and you'll notice that none of them bleed into each other which is very very important so definitely use RGB when you're encoding this stuff so finally inside of comfy UI and comfy UI is where all the magic happens and what comfyi is is a open source tool that uh is node based and it's designed for working with uh AI based diffusion models particularly generally what you're doing is you're bringing in pixel images and then you're working with Laten space images and a lat image from my understanding uh you can think of it as if the diffusion model that we're running is an encyclopedia then a a Laten image is like a description of the image like if you heard of like the term like a a picture is worth a thousand words it's kind of the same idea so with enough uh descriptors of an image in Laden space you can then describe what it's going to look like uh Through The Eyes of one of these AI diffusion models so uh this is the workflow that I've so far found success with and I don't think it's the most optimized I think there's ways to improve this uh but I'll go through the steps involved so first over here you bring in your uh your Source media and in this case I'm bringing in video files and because I only want to work on one frame of this sequence I'm setting my load cap to be one if I set the load cap to zero or a specified amount like 16 frames it's going to batch process 16 images at the same time through this whole network and if you're just working with like a single image you could just be running in a loading in a PNG file that represents these same files uh but yeah it depends on what you're doing um so the way I found going through this and let's just look over here so this is the CG render that goes in this is the like the RGB color pass uh first step is to build a base that I find with uh prompts so we go over here we're uh so some of these prompts in Brackets like this in com UI let's say I want to emphasize as overcast a uh or overcast I can emphasize this or deemphasize it by using control and bringing it up and down and that'll emphasize that it's overcast or or or not so if I if I wanted it if I knew that I was going to be wet and I want more puddles I would just had wet and that would you know bring in more puddles on the ground you can actually see that so if I run this so here it is right now going through the K sampler and so this is the original image not wet and then you can see here how it's building up a wet image of that and now we have wet ground and the next layer so I'm doing it in different passes so so this section right here is doing nothing but the robot I have the mask over here I'm taking the mask of of red and I'm masking out the Laten image that way and working on here and then this final one down here I'm increasing the resolution of everything and you'll notice over here uh so noce I added the word wet and this is just finishing up and we can see down here it's adding in it's making the whole scene wet more wet puddles and stuff it's adding more atmosphere in the in the distance too um once this decoding the image there we go so from dry to wet like if you had to do this in CG on a still image this would be a ton of work but this was like 1 minute or so so in uh you know with AI which is really cool um some things uh let's see here so this image here this was the original image from CG and this one is double the size and the way you up reses images in comi is uh well there's a number of ways the the best way I found um uh in terms of the context of what I'm doing here is first uh upscaling the image using uh like a super resolution model so this one this model here is called forx Ultra sharp and what this will do is it'll take an input image and it'll make it four times bigger by operis that image uh four times so if it's like a 1K image it's now going to be a 4K image and you don't want to rely on this on your own uh it'll give pretty good results but you can always do better uh in the example here what I wanted to do is upress two times and I don't have a two times upress model so what I do is I upress it four times and then I drop it back by 50% to give me two times and then over here I'm going to resample it to then produce the uh uh the final image this image here and this image here you'll notice that they're not the same they're similar but they're not the same like if you notice this car this car side is is facing us over here now that car is kind of rotated and also now there's a car the same color as the robot down there and there wasn't over here many many things the name has Chang and the reason why that happens is the D noise Factor so if I had this D noise set to zero it's well it's not going to do anything but if I had this noise set to one it's it's going to forget everything that came in and it's going to try and make a completely new image but depending on where you have it in the middle uh if you're not really caring so much about changing the image but you want to look uh similar but you know very sharp and high resolution uh 50% works well if you're more concerned about features in an image like like let's say this car for instance you want this car and you'll notice it's not the greatest looking car quality-wise you want that car you can keep that car in its same orientation by dropping this noise Factor down to something like 3 3.5 or3 to 3.5 and then that car will will be there so I if I put in3 and run this the image is uh the image is going to be uh more similar to this image uh but it'll have it'll have a little bit less uh detail in terms of sharpness to things see that once it finishes so there's it is so this this new image here now the car is facing us we still got a new car there maybe that was a car in Laten space not sure but it's it's came out looking like that now um yeah so that's something to think about um but it does it some things don't look as great like you'll see the background it's kind of more sketch sketchy looking whereas if uh if I dropped it back to 0.5 I think it gave a more natural looking result um a lot of nodes for this for instance like this case sampler uh bringing stuff in they look like this uh this is some basic stuff about comi uh if you want to control stuff like Steps um that are parameters here you can drop them as uh change them to inputs and then you can you can drive them that way so you'll notice here that I have a bunch of input values that I'm driving all of these K Samplers at and you'll see like 50% it's now cleaner looking in the background more more photo realistic um some things that I think uh could be improved on this that I'm trying to figure out still but I I haven't yet is uh right now I'm so like I have a bunch of masks I have the robot the cars and stuff if I had if I had a scene where I wanted a high amount of direction of the art the way it looks I might have a whole bunch of these things I might have like like 9 12 masks and I might I might want to be controlling everything like I might want to control just this car here maybe just parts of the robot like I just want to control the color of those things maybe this thing this wall here maybe I need to have a billboard there and I need to control the text of what showing up there uh you can do all that stuff through masks um but I think uh what you could do as an improvement I haven't figured this out but right now I'm I'm doing them one at a time and running a case sampler as a base and then I'm building up one case sampler per element that I want to run but I think you can chain them all together uh as conditioning and run run like lots of conditionings for different elements in an image and run 1K sampler for that so like you could go from a base image such as this to an image with all of the scriptors that you need to control and then do your final op resolution um you can control like these conditioning noes there is uh there's stuff like conditioning concatenation um I think this could probably do but I haven't been able to get to work and that is basically taking the descriptors of like these prompts with your control nodes your uh control Nets and describing that I think you can tie it all together that way let me know if you know how to do it I'd really like to know in terms of controlling the image that is mostly done I found through your zp and your edges and this is happening through what are called control net models and they're over here this one here is uh zdepth and I'm taking in uh the image here it's really it's really noodly it's hard to see what everything is but I'm taking this image over here this uh video and running through here with a strength of one and a start and end percentage zero and one and over here this is the edges coming in and I had the the strength a little bit less I do want I find that zdepth is something it's always good I find to have it really high uh especially with something that it doesn't really know what it is like I know this model has never seen like a robot kind of thing like this before so it's going to try and come up with something different all the time and you really have to constraint it down using control Nets to exactly what you want it to do the way you go about actually art directing what you want something to look like is through the prompts so you generally going to be working with a predominantly a positive and then you're always going to want a negative but I I find you're always working with the the positive prompt you generally if you're trying to do something that's photo real you generally want to put in negative prompts for things that are like drawings or cartoon or anime there's you can probably add quite a lot more uh but yeah most of the stuff you're working with the prompts here and uh so let's say I want to change this image a bit uh let's get rid of bosson Dynamics let's put Terminator robot like the ter minator uh from cinematic frame from the movie Terminator Terminator robot and let's change it down here as well so we'll get rid of uh Boston Dynamics Terminator and we'll make it blue let's get rid of spot as well and we'll run through this image and see how much it's changed so the base image image is going to be from the movie Terminator so it should look vastly different um the road this is like very much like the road like it's postapocalyptic thing uh so now it's the model knows what kind of the Terminator kind of looks are uh you'll notice like the robot now it looks like a something from like a T1000 uh over here on uh so over here we have the Terminator blue Terminator now we got over here so different look to everything and then down here over here I'm uh I don't have a prompt what I'm doing is concatenating both of these prompts together and using that so we'll let this finish out the uh overall shape of things doesn't really change a whole lot and that is due to the fact that the Z depth is really restricting what can change on the object but like minor things change but like if I drop [Music] the Zep down from one to Let's say7 and which one is this one let's drop P for these ones down as well okay good enough let me cancel this let's get rid of that and let's change it to a sunny day and see how much that affects the image so right off the bat you'll notice we don't have have overcast diffused lighting anymore it's now trying to throw in some you know Sunny contrasting also we got a sky we got sky and clouds sunny clarish day and you'll notice also the robot because we dropped the Z depth and edge control waiting down the look of everything is is less round or at least on that one over here it's it's similar and everything about the image like the cars and stuff that'll that all is very uh the same the the look of that kind of stuff is mostly determined by the seed U but yeah let's play around with the seed and see what happens with that so the uh base image here if I change it by one 27 and 28 it's going to change a lot of the look of these things so basically the seed is changing the noise coming in and the noise represents 80% of what the image is so you see how much that changed I'll cancel it and do it again so again everything kind of changes the the weird thing is like the lighting is not accurate to this robot uh also notice that one just came like made a like a cool little wall right there like the robot shadow it's not real a lot of a lot of details in the image I find with this they don't make a lot of sense but they're they're plausible let's not make it blue anymore actually if we make it a gray Terminator robots are gray right this case sampler the D noise is set to one so what that means is the robot here it's not taking any information from the one over here it's completely changing and then when it's samples up it's going to I'm taking uh Doo of5 so it's going to be it's basically just op rizing this thing so there it is that's uh that's pretty much all I have to say so far with this um as you can see like to do to do a lot of these changes in these images like if you're doing it conventionally with visual facts with like 3D renders and compositing to make that big of a change with this image it would take an enormous amount of work like every single department down to modeling like modeling rigging uh all of that stuff the lighting department rendering compositing probably is going to have to change a little bit not so much but like so much of the the image uh so much of the work changes in a in a whole pipeline whereas if you're generating still images like this like it's it's 30 seconds on a on a GPU which is pretty crazy you notice see this one uh which is interesting because I'm making a Terminator robot like the the models they're trained on so many images and like it knows the Terminator movie so what did it do it stuck a Terminator in there so that's that's pretty cool if I make the robot instead of a Terminator like uh from Transformers um let's go from Transformers let's try that out yeah so changing quite a lot it's adding a lot of different features so like it knows it knows what the Transformers movie looks like and it's probably going to add like Transformers hands and and weird stuff like that in the image which is pretty cool yeah uh I've been working on uh how to go from from these still images uh to video and unfortunately I've had no success with that uh I have a different model down here that I've been trying to do experiments with to uh get from this quality to uh like like a a sequence uh that is like temporally consistent with no like jumping and stuff and so far I've had no success with that um the main things I'm running into let me control control B is what you want to do to uh enable and disable noes the main issue I'm having is when you're doing video uh it's really difficult I found to encode motion so like this camera this is only 16 frames I think yeah 16 frames uh with the experiments I'm doing I I found that like yeah it's it's near impossible to encode a proper camera motion with this I think uh eventually we're probably going to see uh some changes to this stuff to be able tocode motion because going from something like this to moving with like for instance if I rendered out a image with motion vectors for every pixel uh in a in a sequence which is very easy to do in 3D software I could encode not only the camera but the like every single asset and everything every single thing in CG encoded motion could be done and with Laten consistency we could get like like uh like Final Shots through AI like this which should be pretty incredible but so far it's not it's not a thing right now um but overall like if you're doing uh concept work uh this is a really good way to do it because you could set set up a system through Maya or whatever software and use nuke and and automate entire process of generating these images so you're spending like absolutely zero time in uh in that and and running in and like you could have a 3D model like this and then tumble it and see how it looks from a different side and render that out and it would yeah it's very cool and the ability to just quickly change like humongous like having a big impact on the the image by just changing a propt is is a really nice thing like going from an overcast wet Day to a sunny dry day and having uh sharp Shadows on the ground like that through like not having to render anything is pretty cool if you're interested in doing this stuff some further things for you to look at uh with this is IP adapter uh I have one down here somewhere [Music] right here so uh a lot of stuff you'll want to look up how to actually use them but the just IP adapter will take an image the look of something like this and then and then apply it to your source and what it's doing is it's changing the model going in and producing a new model which which has these kind of uh uh image effects on it or the look it uh the look of this stuff there's many many many models out there uh the model that I'm using is Juggernaut sdxl Juggernaut um sdxl is stable diffusion and a lot of these are diffusion models stable diffusion model uh XL is a particular one that's been trained at a 1K Square the previous one um SD 1.5 uh that was trained on 52 pixel images and uh the XL the 1K the models you're getting out of that they they're a lot they have a lot more information they're and they produce a lot better photo realistic images um you can extend the model by another thing called aora and what that is is you're taking a base model and then you're you're adding more trained data for a particular thing onto it so if you like a base model how it's looking but let's say let's say you're going to build an actual I don't know um let's say you're making a movie about uh another parts of the Caribbean movie and you want you want a model trained on like old ships and stuff you could you could get tons of images of of like pirate ships and that kind of stuff and train a aora on that and then the model now has a much better understanding of pirate ships and and and that could be for like you any anything you're trying to do and uh there's tons of videos on YouTube for how to how to do stuff like training uh base models and luras and stuff um yeah the whole system I found to be uh give you it gives you a lot of freedom to to work on a lot of stuff uh and it's very much this is very much like compositing I find like if you're a compositor uh I think you'll find that moving to a system like this with nodes you're going to find it very easy the only things you really got to think about is uh these Laten images you're no longer working with pixels well you are sometimes like you do have rudimentary features of compositing and layering images and you generally want to be doing that with your Source images and and channels like if if you're unpacking an RGB thing for getting a certain Channel you can do stuff like this like that with this this system uh but eventually you're you're working in Laten space and you're running things through these case Samplers which is like feeding it into the model and generating an image and decoding it out to an image so that's all I got uh let me know uh if I had any mistakes in this there was a lot of information and I'm very new to all this stuff and there's probably a bunch of Corrections and another thing is if you are quite experienced with uh um comi and you know how to chain multiple conditions together um tell me how to do it or set up a workspace it' be great and and I'll link to it because it'd be really good to be able to take like a an image base image like this and be able to control many many things at once and have them all running through a single K sampler and then I'll present it at the end instead of running through like 10K Samplers it'd be a great way to go through that all right I'll talk to you guys later

Channel: Bryan Howard

Views: 6,685

Keywords:

Id: lFE8yI4i0Yw

Length: 47min 15sec (2835 seconds)

Published: Wed Dec 20 2023