Blender + Nuke | A.I Enhanced Digital Matte Painting Workflow

in this video we're talking about new AI techniques and we're merging traditional VFX workflows with those techniques we're going to go through a multi-step process to show how the digital mat painting was done from rough blockout and rough sculpt all the way to lighting temporary and lowres texture projection and then how we bring this into Ai and use upscale techniques to enhance the look and actually get this detail much faster than doing it manually then we'll bake it down and project it on a remes Geo in Nuke now you might be asking why do we want to do this approach what is the benefit the benefit is to not have to make every single asset in a very cost intensive way right so quick SL can solve some of this problem or if you do photob bashing you know you could take different models together and put them together but you might not have full control over the composition so this is actually a middle ground between photo bashing and creating your own assets all right guys so if you haven't seen how this works this is Adobe substance modeler and it's VR modeling software um essentially how it works is you can move around you can sculpt things uh in real time and everything's really fast um I did import this into centimeters not meters because if you do meters it does get a little bit slow so you want to find a good resolution for the mesh so when you import an obj uh just play around with the the scale there depending on the size your scene but essentially you can draw draw stuff around you can subtract stuff I won't do a whole like breakdown of the software but essentially this is what we're doing um now I'm not going to sculpt all of the detail the idea is that this workflow is supposed to be faster and the reason it is faster is because of uh this so uh in this tool here we have a clay and stuff like that but there's one called stamps and if you go to the stamps they have some default ones called rocks so you can grab one of these and basically just have some pretty decent rocks to start with now remember we're going to use AI to enhance this so the reason we're doing this is is because we want to catch light onto a base mesh it doesn't have to be perfect mesh we're not sculpting a rock super super detailed like if we were going to do it uh to make a full-on CG asset what we're doing is we're just giving enough information so that the imageo image AI is going to have light information to work with so essentially we can take these stamps and just kind of like place them and that's basically what I did um some sculptor probably look at this and say like this is like the wrong way to do it but the end result I think speaks for itself which uh after using the AI to kind of enhance the the uh forms there so essentially you can go around here um play around and this is why the Bas mesh is useful because you can kind of use it as a reference guide uh you can play around through scale here uh you can go in there and erase the edges and do like a proper sculpt um if you want to so this is just uh also just really fun to play with if you like doing stuff like this so you can smooth it out and different things like that so you want to go around place all the Rocks so I'll just show you the one that I did uh in a second here but that's the base idea so this is is the scene that I did uh I spent a few more minutes on it maybe 10 20 minutes just slapping it together again we're not trying to create a super detailed sculpture where we go in and we carve every single crack cuz if you look at a real Cliff there's an immense amount of detail actually there so we don't want to spend all the time we're trying to create Assets in in an efficient way so this is kind of like if you were in real life slapping a piece of clay together really really quick that's basically what this is but the reason we're doing this versus having a flat geometry which in older DMP workflows and if there's not a lot of Parallax is still totally a legit workflow um you can do it that way but the reason we're doing this is because the AI will see these different facing angles and the light we're going to light in in a blender so it'll cast nicer Shadows so even if we if we remesh this later on which we will do which will reduce some of this Parallax in Geometry we're still going to get a higher quality result by just doing a little bit more effort and the idea with image to image I think that a lot of people might not get right away unless you play around with AI quite a lot is the closer you can get the result to what you want meaning the lighting and the textures the closer it is the better result you're going to get if you just give it a flat surface with nothing to cast Shadows or no no texture on the surface it's not going to give you the result you want so you still have to do some work here and that's why I think it's a interesting tool in the workflow so we'll go from this and then we'll go on to the next step so this is the mesh after bringing it in from modeler so you can export it you can also REM it on the way out uh and you can remesh in blender as well if there's too many faces and things like that so those are things to keep in mind modifier for example modifier remesh uh if you have that problem you don't want a huge poly count because you want to bring this into nuke after to project onto and that's just something to keep in mind so this is going to give us enough lighting information I'm using Eevee right now that's why you see it flickering as I'm rotating but if I switch to Cycles we'll get a little bit better bounce Lighting in the cracks and things like that as well so that's the base model enough to get shadows and highlights um the next thing we want want to do essentially is give it a base material so we've already given it a base lighting I've already done like a light setup I don't think this was the exact light setup I put I think I had moved these lights around a bit more played with the Shadows um and also uh go to Cycles I don't want to do it while I'm recording because it'll probably stutter my microphone um as the GPU kind of kicks up there so essentially you want to find a good lighting good bounce lighting um but then we want to project uh Rock texture so that's going to give the AI a bit more to work with because if you give it this maybe they'll think it's snow or something like that cuz it's just blue it's flat there's nothing there to tell it that this is Rock under blue hour lighting so if you project a basic material and it doesn't have to look good it just has to be there so this is literally just a rock texture I think I got it off blender kit which is like a plugin that you can get assets there's a free tier and there's a paid tier I think it's like 14 bucks a month or something but you basically get a bunch of textures that you can search through it's a bit similar to quickle um so kind of like that but just projecting it from view so essentially what you want to do because this thing doesn't have UVS you basically just go into edit mode uh you can hit a and then you can hit U and then say project from view that's one way to do it um I think this material the way that it comes is actually using World coordinates and mapping it not even using the UVS the specific one from that asset store but if you don't have it just project from view any texture it's going to work now we don't care that there's a bunch of repeating material uh repeating materials here like tiling you know if you were doing this like the right way you'd have to spend a lot of time painting this material to get the scale to look realistic following the cracks and all of those things we probably have to bring it into substance painter or something like that so we're avoiding all of those steps by doing this we're just giving enough detail that hey this is clearly Rock and this is going to work with the AI so next I'm going to show you how to get this image out and get it into either Crea AI which is what we're using or we're going to use a magnific AI which is an alternative both work really well so so they do a little bit different things uh different offer there so we're going to show both all right so this is cre AI it's essentially stable diffusion image to image except with their new real-time capability so essentially you could like draw something here uh and I have the prompt like snake and then I could just increase the AI strength and then it's going to start to look more and more like a snake but sort of conforming to the drawing that we're working with so that's the idea with image to image um essentially you know push it too far you're going to get something totally random and pretty much might as well is like mid Journey or just stable Fusion thing but image image is where I think this is the most useful for visual effects artist so I'm going to remove this and undo uh and I actually you can bring in images here so you can do screen to image so you can actually stream from like blender or nuke or something but you can actually bring in the image so I brought in the Square render that I told you guys to render out it's yeah basically Square render in Cycles I'm going to scale it up here now again we push it too far it's going to go crazy I have snake as the prompt so it's going to do some really weird stuff but if I say rock Cliff side detailed uh cracks stuff like that and again way too far here but if we bring it down we really really bring it all the way down we start to get some results that could be interesting so you'll see that it's a little bit blurry initially and that's okay cuz we're going to use the AI upscaler which will enhance all the detail and kind of run this process again but the reason I like to generate first occasionally is to just see if we can get different results it changes a little bit more than the upscaler the upscaler just kind of add details but this the generate kind of changes the details so it depends what you want if you want to use the generate workflow or just go straight to upscaling but essentially we have the blurry version we can hit the seat a few times and we'll get different types of things we can see the result really really quickly and as long as the shape isn't changing completely this is something we could project so again once you're done with it you have one you like you can say send to enhance and upscale and it will go to this tab here and we can add a lot of detail this to this uh alternatively like I said you could just take this image directly into the upscaler so I'm going to show uh kind of both there this is the result after bringing it in so we have this blurry one that was there and then we have the upscale and we start to get a lot more detail in here uh for me this was a little bit too streaky so there was kind of these streaks that kept appearing in the Kaa upscale so both upscales have their own sort of strengths and weaknesses um I think Kaa is uh more creative in the sense that you can generate different shapes and things like that but I prefer magnific upscaler currently um so you know both are being developed uh kind of Hardcore right now so that's the upscale but let's go into magnific and see the difference so we go there uh this is the before so I just brought the render straight in didn't even generate uh that middle step process and then just upscale from there and for me this was a much more realistic result so especially around like this region um kind of where the light's going to be hitting and we actually are going to see that detail and it's really respecting the geometry too if you look at where the light and shadows are and you look at the results that it's giving us I mean this is like an incredible detail and that's all the the crack detail I was explaining earlier we didn't have to sculpt any of that so that's pretty cool we could improve this area a little bit down here um if you wanted to be picky about it and kind of blend this further so you could generate additional ones and then just blend them together so that's something that I actually did in the comp I I generated maybe two or three that I liked and then just keyx them together so um magnific it's not free but uh you just drop your image in and then you have some different sliders it's really really simple creativity will modify more HDR if you push it really far it'll almost become too detailed so generally I would just kind of give a few random uh values of these three uh and generate like 10 images and then just pick the best three and then you can use that in the map painting so that's how I did it uh and then we'll jump into how to reproject this and finalize this in Nuke so this is after bringing it into nuke I have the first version I have another version that I kind of cranked up the HDR level on magnific which gives it a little bit of a wetter look which kind of I was going for in the shot to look kind of uh sort of damp and wet so this looks slightly more reflective like there's things I can key out there to to give it that look so I kind of just ke mixed them together did a few different grades here so that's what I did and go to the project 3D we're projecting from our projection camera this is brought in from blender it's an Olympic export and we're projecting onto an Olympic export from blender onto geometry that we sculpted so now we have all the Parallax and it's going to feel 3D it's not going to feel like we're stuck at a 2D image in there that doesn't fit the scene so the lighting fits the textures fit everything's working and that's how we can do it so final shot camera is going to the camera so we have an animated camera and we have a static camera for the projection and um essentially after that this is the result I've done some grading and things like that to make it actually fit the scene so you can see there's definitely some stuff going on here to make the colors work but we are also stenciling out the extra detail we don't need the floor and the door and things like that we want to keep the CG that's in there so if you look at what's going on before that and we let it cash for a second this DMP is essentially just being slapped over the top of rcg scene to make it better so that is like a DMP enhancement so that's what we can see we have like the before it's not working that great and then we have after and that's looking a lot better and then we can obviously do all our hazing and things like that to uh get a much better result as well to make everything blend so there's a lot more steps involved there uh for the creative stuff but for the DMP that's about it also if it interests you we've just released some free VFX smoke assets like the ones you saw in this shot so one's for free uh the other ones are paid if you want a few more of them but they're there if you want it and additionally uh there's going to be a blender nuke workshop for this specific two shots so if you want more workflows like this or just seeing the entire process of grading and uh sort of the artistic thinking a longer process probably two to three hours of of a workshop that'll be there and essentially I think what's cool about this as well is it comes with a lot of assets so it comes with all the smoke assets instead of getting it separately so it's basically almost free compared to the price of the assets and then you'll also get some lens floor patterns we're going to talk about the first shot as well how to comp lasers how to do different things like that so there's a lot more there for the people who want it and it's basically a bit of a lower price course for those who just want to have a few hours of extra content and maybe you're more interested in doing your own indie films or just seeing more of these experimental workflows as well as seeing just more artistic and and look development workflow so that's there for the people who want it and that's about it thanks guys

Channel: Compositing Academy

Views: 12,963

Keywords:

Id: XG-5tchFBQM

Length: 13min 36sec (816 seconds)

Published: Sat Dec 23 2023