Creating a 3D Hole using Nuke + Photoshop A.I (Firefly) Tutorial

hey guys welcome to part two of this uh series here and this time we're going to use nuke camera projections to project AI generated images to create a 3D DMP effect with this hole in the ground now these are pretty standard camera workflows or camera projection workflows rather in Nuke so even if you're kind of new to camera projections or DMP this will be useful for you I just think that the added element of generative AI on top of this and generating potentially from different angles to fix the stretching that happens with camera projections is an interesting evolution of this workflow and so that's kind of what this was the purpose of this doing this project for me was to see if this is a viable workflow and I do think it is so first I'm going to show you what I did from the first camera projection I rendered out a stale image from nuke using tvi scale first to reformat it to be twice the size it's a little bit better than just a reformat notes if you type in tvi scale you'll find that and I just want to start with a higher res so that the generative AI would have something better to work with and potentially give high-res textures that we can downscale later and so this was the bass just the grass kind of clean plated from our one of our previous videos and if you're not familiar with the Photoshop workflow of how to use generative AI I check out the last video I just posted which kind of shows the way that I think this is going to be the most useful so I'm not going to recover those topics that I already did so just to show the layers though in case anyone wants to see this was the first layer giving it some prompts I think what was the prompt here it was deep hole in the ground artillery hole in the ground grass hanging on the edge of the whole chunks of dirt and it gave this weird rock in the center but this side was actually pretty good and so I just continued generating here circle out that rock and then try it again and that's giving some pretty decent results right off the bat but I did want to make it bigger and so employing the techniques I mentioned in the last video I kind of Clone stamped roughly down and you can see the edge is all soft and it's not very well integrated but that's not really what we're worried about because we just need to hint at the eye what we're trying to do and so if you generate on top of that we can clean up that edge and it does all the work for us to get that nice grass and stuff hanging over the edge and so I think that was the remainder of the layers I think I adjusted a little bit further and that was the first pass now if I take that image that I projected in Nuke I projected it on the lighter scan the flattened one that I did in the previous video it could also just be a flat card pretty much was just a flat card in terms of the shape of it but if you project that out from the beginning frame where I painted it and put it on that's our hole in 3D and if I play you'll see that we have a projection now you see immediately the problems that are associated with projections we see it starts to stretch towards the end and stuff like that but it's tracking well and we have everything in the in a good starting place and so this is what I wanted to experiment with was is to go further in the frame range and generate again except from a different angle to see if we can fix this stretching effect that is normally happening and that's a normal workflow for DMP anyway to project images from different angles but if we can generate it it should save us some time so the next thing I did was take that stretch projection frame and I rendered it out and I extended it with another generative AI to see will be down below in the hole because we know the hole is not flat on the surface so we want to see everything underneath and then we're going to patch it again over the top so this is the back wall and the ground is what we're trying to create and so the first result was weird it looked like an egg yolk or something so I changed it I also wanted to see more on the wall of the grass to see if we could get more layers of Parallax because if we just see the back wall it's going to feel a bit 2D so part of the trick of this I wanted to get like the edge wall that will overlap on the back wall so if you get at least three layers it's going to feel a bit more 3D so that's where understanding Parallax is important but just a bit more painting here and just generative fills so you can see the process of just some different layers here try to get something better on the bottom and I also even took some areas that I thought were good like over here I just cut this out and flipped it and then put this here and generate around it and again it's the same technique as the last video when it has something on the edge to touch it's going to give results that blend nicely with that edge and so uh just keep doing that over and over I kept wanting to generate these giant sticks for some reason so that was something that it kept having a problem with but it's not a perfect tool yet but it is a tool that you can certainly use so a little bit of manual concept clone stamping here and then just finishing it up at the end and that was the back wall and the floor so this is another thing that we're going to project in new so again this is looking at the stretch projection and then this is bringing in the new image with the generated from a new angle and then I just did a bit of Corrections on this layer adjusting the perspective a bit to match better some small comp brightness adjustments and things like that and cut it out with the Roto so we give it an alpha and we pre-multiply that and essentially after that we want to project it onto not just a flat cylinder but we want to create something that will give the illusion or enough Breaking Up of that Parallax so we read it as a rough surface so what I took was a cylinder and I lined up with that lidar Geo so I could see the other projection if I load the other projection here and so that's what this looks like and then I just took a Some Noise patterns and stuff like that with a displaced Geo to get the broken up Edge and lined it up with where the back wall would be and so we get the side walls and we get the broken up feeling and at the merge yo with another cylinder for the bottom so that would be the ground plane of the bottom of the hole and so that's what that starts to look like and another card with a displaced Geo just to give the wall a bit closer to the camera so we feel more parallaxes if we can get some things closer and some things further we're going to feel that distance a bit more and so if we merge that over that's what we get and now you notice we're seeing underground right now so after this we're going to add back the top of the grass so if we add that layer back over that's what that looks like so we're starting to cover up back on the top area and I'll go through that now with this top layer so the top layer is actually being branched off from the original projection so we have the original projection we have the back wall and the bottom being merged in I also did a little bit of sidewall Geo there and so that's the thing that we're seeing but here I've branched off the pipe here to create a new sort of patch that we're going to create and how we do this is we actually just because this is the way that this generated was the grass was over this something dark it's pretty easy to create an alpha we can just key it out with the Luma gear so we get this Alpha here and I solidified it a little bit and if we pre-multiply that's going to be the patch that we put over the top again on that flat top ground plane that we already have so if I mask this out this is what we're slapping over the top and I did add a little bit more around so I took a branch off here gave it a solid Alpha and I masked that just to give some more grass and stuff like that around we wouldn't necessarily have to project all this area but this is just fasting I'm just putting a patch over so I put this over the top like this so these are the two things that we're going to project onto a ground plane one important thing to note here is I did a little bit of a on premaled Roto paint and a pre-malt and that's going to fix some dark edges on the grass if I compare let's go to frame I think it's one actually we could just do this Frame let's just check this Frame and I'll just disable that correction and enable so that's just a simple Edge fix because we had that semi-transparent grass over the darkness we're going to have dark edges so that's how you would fix that it's actually similar to what I teach in my Keen class with the edge painting and Edge fixes a very common workflow to just extend the edges so on pre-mult we paint but only in the RGB and then we pre-melt again and then just some more corrections to fix some stretching a little bit sharpening because sometimes the AI will generate some softer and some sharper details and you need to blend those things together and then we have something like this we project it out and that's going to move with our image now one extra thing I did here and this is it pretty cool to know and useful is towards the end of the sequence this started to stretch a little bit you can see it's horizontally stretched it's the same thing we were seeing before remember when you're looking at the projection it looks a bit stretched when we go further and further and so one way you can actually fix that is to not just project onto a flat surface because we know that maybe some of this grass is hanging over the edge and so what I did here to prevent some of that stretching was essentially just take a card and displace it with a ramp so if you take a black and white ramp you're saying that we want to bend part of the card and not bend the other part and so what that does is we can ignore this part that's straight up but if we look just at the end and here what we're actually doing is just trying to get a bit of a bend and that's what I'm projecting the grass onto so rather than projecting on a flat card on the surface we're just getting a little bit of that band and that's going to change the perspective of the projection as time goes on and that's going to actually help prevent that stretching so there's different ways to do that like I mentioned in the clean planning tutorial posted two videos ago you can do eye transforms just different techniques you can do to prevent or distort things in different ways when you're dealing with projections and that's one way we can do it and so now it merges over the top and we have something that's covering up those areas and that's working pretty well now the rest of these little projections and patches you see in the script here are just corrective patches dissolves and stuff like that and that's the techniques I talked about in the clean plating tutorial two videos ago so if you're new to clean plating projections all those things I recommend checking out nuke 202 because that really dives into the 3D system and getting comfortable with those things and so that's Linked In the description as well but that's about it for this tutorial I hope you you guys found this useful and you can start to apply this in other creative ways I think there are a lot of other creative ways I've been experimenting with this on different projects and I've seen a lot of interesting things that we can do with this if you start to think outside the box a little bit and so this is just one test I've done but I do think that this scales into a lot of different scenarios that are very interesting so if you liked it press the like comment tell me what you think about this workflow as well if you've experimented with this at all and that's about it

Channel: Compositing Academy

Views: 9,797

Keywords: nuke, tutorial, photoshop, ai, firefly, projections, camera, generative, 3d projections, camera projections

Id: 8QEGlRX-kH4

Length: 10min 13sec (613 seconds)

Published: Sat Jul 01 2023