PDG Automation Pipeline: Make More Assets | Rob Stauffer | GDC 2019

[Music] okay hello welcome welcome to the hive so my name is Rob I work out of the Santa Monica office for side effects we're going to talk about using Houdini to make assets to make variations on assets make a lot of assets and we're also going to cover a little bit some new technology that we came out with called PD G so start off our gonna talk a little bit about procedural ism how many people here already use Houdini and helmet yeah so most of you do okay and built it so you know with traditional methods of modeling it's usually destructive right your model something if you pull points around you can sculpt it but then you have to say about that version if somebody comes back to you and wants to make some changes then you have to kind of start all over again if you're lucky enough that you saved out an earlier version of the scene or of the model then like great I can go back to that but it's it's really it's it's not very flexible so as you know it's with Houdini it is procedural it's not procedural in the way of say something like no man's sky or minecraft is but it is a way to you know play you know to be a component of prototyping and content creation and level building it's about you know giving the artists tools that they need to describe a creative process that they use to make that content it's not just automation I think sometimes especially for people new or reluctant to adopt Houdini and think about procedural ISM I think it's going to be just a lot of repetition and seeing the same thing over and over again but it's not that it's it's you can you know one of the bonuses is you can create you know random results but then you can work on that and give it kind of your own flair and it's not about moving the traditional tools either I mean it could still work however you want but you can use Houdini to make tools that will help you enhance your content so as we know that in this example here this is just a pipe tool it seems to be the first thing that everybody wants to create when they are trying to come up with an asset for a game engine and he's just that's that single curve that you're you draw and then the the digital asset is doing the rest of the work for you it's figuring out where the corners should be where all the flanges are you can control the width and and and such of each little component and you're only having to modify the four parameters at the top there at the top of the HDA so the way it all works is it all starts with Houdini where everything that you create becomes a node that includes when you're modeling in the viewport you know each time you you drop down something you it's creating a node so that you have that history and those nodes are wired together together to build a more complicated network and that network is essentially giving you a recipe for whatever it is that you're creating and that network that network could be processing geometry that came from somewhere else or creating geometry from scratch inside of Houdini and that recipe you can just keep using it over and over and over again to get different results and give them variations so we're not really talking about procedural modeling today so I'm not going to spend too much time on this but we're gonna be working on sort of a highway New Jersey barrier so I just want to give you sort of a peek at the modeling process for this barrier and I also want to point out that you know I am in the viewport here so for people that want to model in Houdini that think Oh everything's a node and it's a network and it's not what I'm used to we've worked really hard over the last several versions to start to provide better modeling tools in the viewport so that you can to give people that are modelers sort of a toolset that they're used to so here I'm just you know I have some pieces that I modeled I have some other pieces I took from somewhere else like these tubes that I'm just going to use two boolean out the shape and start to form the shape of this barrier and add a little bit of beveling so you know Here I am doing it in the viewport but or in the network view but you can also do all these things you have access to in the viewport so as I adjust the beveling a little bit more I'm also going to increase the resolution a bit here because I want to for what I'm gonna do I'm gonna add some noise to this to crunch it up and so I'm gonna need a lot more resolution on the geometry so now that we have that barrier that we're going to use as our starting point we want to add some detail to it and then what I'm gonna do is we're going to take a look at using wedging to create lots of variations on that detail so if we take a closer look here so here's that barrier with a little bit of some crunch to it some details that I added I created a an HD a that is basically bullying out the tips so that you get some damage there and then I have two socks below that that are we call them mountaintops in Houdini which is essentially point noise so I'm just adding some noise to the barrier and then I'm bullying out those shapes to get what you see here and you see as I move the sliders around especially on the damage tool it's sort of scattering the points on the top in a different direction so that we the chunks are changing so we've always had wedging inside of Houdini in our render operators but now we've introduced something called PDG which stands for a procedural dependency graph and it really opens up lots of possibilities and also makes this wedging a little bit easier and a little bit easier to track and so we'll get a peek at that in a moment so here it is inside of what is called tops now that's PDG it's tops which is tasks operators so you see I have three wedged nodes that I've set up here and so the first one is the first base offset for the first noise parameter and I have ten ten wedges of that and then below that I have four wedges of some noise height and then below that I'm wedging the chunks that are being taken out and so as I process that wedges it's almost instant this is not sped up and as I click on the dots it's showing me all that information all those wedge values that were applied to those different parameters and what's nice about that in the old wedge you couldn't do that the old way we did it before you kind of had to either write out that information to a variable and maybe bake the geometry and try to dig that out but now you can visually take a look and click on all those and all those lines that it's creating for you is also showing you all the dependencies so if you ever one point when its dirty only one of those wedges you could and it would generate or regenerate anything below it that it has to so here we see this is the look at the first wedge operator that's the one that has the ten and you see they're pointing to offset X and then it's also the path so this is very similar to how we used to do in the old web drop as well where you you know as you know with Houdini all those objects or all those operators is very much like a directory tree and so you just navigate to the parameter that you want to change and you put that in and so that's what I did here here's the secondary noise height wedge so again that's one parameter I'm changing the height and then the last one in this particular case I'm doing two parameters and this is the chunks of the that is being taken them out of the barrier so now once those two wedges are complete like I said we click on the work items and now I'm going to expand my setup a little bit well let's go back don't look like it's playing play it's not playing okay so what I did here is now you know before I just was creating the wedges so all I was doing was taking those values and storing them somewhere and I was exporting them to the environment when I'd click on the dots but I want to do something with that information and so with this setup I built something where I take that noise and I use it as a mask and then I build some textures from that that I bake out so now I add that baked manche or textures node underneath that which is just a wrap render operator and now it is also all connected to all those work items before so you see it's generated all the work items underneath and so now we can see a bigger set up of that so here we see it cooking so now the next thing I want to do is I take those baked out textures and I run them through a cop Network compositing network inside of Houdini and again all of these as I click on those work items in the middle you see that it's showing me all of the dependency so even if like one of the frames just failed for some unknown reason you only have it'll you'll get that visual feedback it'll be a red dot and then you dirty that work item and it's going to re-cook everything that is affected by it and you don't have to do any of the tracking of you know tracking it down that is sped up by the way it doesn't go that fast here's just another look at the cop network that I built so I want to I wanted to show this just because first of all there may be some new users that don't realize we have a compositing component to Houdini and also just to you know just really highlight how nice it is to work in all these different areas to create like this is another way to help you create these variations so everything works together nicely in that Houdini way so now the process has really you know I've really built up this network a little bit more so once those compositing nodes are done I'm then going to take those and I'm going to render an image and it's just going to be a little image so I get the feel for what the barrier looks like on the left hand side that's another new view for us which is called the task graph table and that's showing you all the tasks it shows you the index or the work item number it shows you how long it's taking all of these work items or also can be color-coded so that as things are you know if there's a bottleneck somewhere or it'll show you visually what processes are taken longer than others based on the color of the work items in here for this it's that's running pretty smoothly so everything's still green so what i'm doing here though as i've wait for each of those compositing networks to finish i bring them in and then i render the image of each one using those textures that I just baked out and so the way that works is that we have a variable called wedge ID or sorry wedge index or wedge number and so if you feed that in as just as you would like a frame value you can write out your wedges that way so per wedge it's using that wedge number and then you write out your geometry or any of your your images that you want to render out you just include that number and that'll help you sort of separate them and then at the end we actually have a node for image magic so that we can create a sort of contact sheet of that and so that's what that looks like so there's all 80 barriers that it might be hard to see from there but they are all different in some way so now this next example this is sort of the same thing now in this case I'm using an HD a for a blade or I mean for a sword and what's nice about PDG and tops is that we have what's called an HD a processor and so that what that means in my top Network tops by the way or PDG is also available as a standalone piece of software so which basically you open it up and you only see that you don't have access to Oh shops or anything like that but so you can you can manage networks that like in this example you could take an HD a if you have an HD a the process is some geometry you use an HD a processor to run that HD a without loading Houdini or opening Houdini at all so it's just a handy thing that to kind of have this little pipeline in your back pocket so so in this case we're no longer pointing this is another new thing with the new wedging and before we had to you would navigate to that spot in your network but now you can actually just set those attributes when you name the attribute whatever you want in this case its grip length or blade length and now that attribute is being exported into your environment so you use it just with an accent text just like you would some vex variables in whatever it is that you're changing so you see here on the left hand side where the green is that I have those variables that were set by the wedge in my asset so that it's controlling what the sword looks like as opposed to before where you had to explicitly set that path now you do have the option to do both it's up to you however you want to do it so here's just a look there are 3,200 different swords here because that's um why not and so here's just a quick look at some of those variations and the nice thing too with wedging every parameter in Houdini is wedgie bull so even if you you've added a ramp parameter you can wedge the you know the indicators for which part of the ramp it where it is the position and the value all of that stuff is legible so in this particular case that shape of the blade is being controlled by a ramp like that and I can I can wedge all those parameters if I want to do here's one more example it's all about variations today so here is another OTL or another HDA that really just makes a brick wall and so it's fairly simple setup you can change the number of bricks change the height change the width you can change the spacing however way you want just to come up with some sort of generic brick wall and then from that we're going to set up some wedges on this so that the goal here is what we're going to try and do is we're gonna get some variations on the wall and then you can use those variations to drive an RB D sim and then that RB d sim can drive a smoke sim for the dust or whatever when it when it gets hit so there you see it immediately generates those work items for me and it's important to note under up top you'll see in a second it says export to environment which is what enables you to see those as you click on those work items as well as override so what that override is doing is allowing you to see the changes in the viewport so put down another wedge node here and the reason why so another important difference I think is in the old wedge drop you could have as many parameters you want on one wedge drop and you were able to individually specify the number of iterations per wedge so in one parameter you could say I want five variations of this the next one could be ten variations you can do it all in one rock in tops we have to do that differently and we use the wedge index parameter to make sure that you that's how it lines up each wedge so like similar to that similar to the barrier example if I had you know ten five and two then I would get you know the appropriate amount of wedges from that all right so there you see in the viewport you're seeing some of those the results of our wedging efforts so from here I can render out and generate a contact sheet for those so you can say alright now I've done this and I want to see what it looks like and I want to pick the one I like the best and again all being tracked by work items now when you're doing wedging and you have all those individual work items if there's ever a part where you need to gather things together we use what's called you could wait for all so then it basically brings it back down to a frame based situation versus using the wedge index and the nice - all of these things can be generated before you actually kick off the job so you'll see all of your white work items you click on each one and you see what all those dependencies are before you even start the processing so now that we have our wedging setup for that we're going to do an RB D sim so we just take the sphere it's pretty straightforward RBD sim and there's our top Network again and this is going to kick off the simulation and you see there as you can click on those wedges that the bricks are changing so it's running so it's taking in all of those different variations you made on the wall and it's going to run based on each one and so you see as it kicked off that little green block in the middle is just pertaining to that wedge so those are the frames that were generated there this is just a quick little glance at you know if populating in the in your file browser you'll see that each one of those under it says are beady variations and then has a zero before the frame number so that is that's where I'm using the wedge index to identify what you know which which sim that came from which which wedge number and here's a much bigger you know and again another broader network so this is where we're getting that final result we have the variations on the geometry we run it through an RB DCM and then we take that our BD sim and we use it as a source for our fluid sim or our smoke sim and when we're done we can generate again a contact sheet using imagemagick so that you can sort of have one-stop shop to see what your variations look like and you can use that for anything if you're making con you know pyro if you're going to bake out pyro to be used as a texture sheet you can do all your variations as well first and see those parameters and say oh you pick the fire that you want and then use those values to generate the final images and all of this works with HQ on the left that's what it used to look like now you know this is what it looks like inside of tops it works with HQ it works with other schedules so sorry other schedulers as well such as deadlines or tractor there's a lot of other nodes included in PDG that really help you talk there's plenty of Python nodes too to kind of expand your your reach that you can write custom stuff for yourself perforce nodes things like renaming files like you could really string together a pipeline from this which is kind of nice and then using HQ that means all of this all this parallelization that you're taking advantage of with wedging or whatever it is that you're doing if you have HQ or any sort of a compute form you can distribute that out to multiple machines this is just a quick example of you know you can roll your own nodes if you want to I worked with David Larsen over at a logarithmic on this a little bit he built some substance nodes for me and and also Adobe sent us a Photoshop node to make a contact sheet so it just runs so as this finishes it kicks off a Photoshop action to make a contact sheet so from that and but the rest of the process I used substance designer instead of the using new texture and the original one the textures were ones that I made inside of Houdini but then we also were able to incorporate substance with the notes from David to add that variation to their textures that I used to do to them baked out the render and that's it [Applause] sure so dirty just really basically means it can be dirty for a variety of reasons most of the time it's because it needs to be reprocessed like you've decided that this data is old for whatever reason it could be something as simple as maybe you just had a bad frame or the geometry was bad when it was written out but it completed successfully right so you could dirty it manually by hand if you have frames that fail so dirtying is basically saying if you can imagine you know what the dots look like you know and say this one I know I need to redo for whatever reason I'm gonna mark it as dirty then PDG knows what all of the dependencies for that work item are so it will dirty all of those automatically and then you just cook the output of that and so it'll it'll redo everything and it clean it's clean exactly it's like data laundering the substance be ignored is it using the substance engine it's using yeah yeah it's talking directly to substance it's using the substance automation toolkit these will not so what will it's not going to these will not be shipping with Houdini what probably will happen in the not-so-distant future if you guys are familiar with or bolt will probably have a tops will add a top section most likely to or bolt that then people from the community as well as you know them can contribute there will put them there for people to download if they want to use them this one for the most part is it hasn't changed it's just we have you know we have the idea of schedulers in top so everything needs a scheduler so if you're not using HQ there's also by default there's a local scheduler which is just pointing to your local machine so the idea is just that with if you choose to use HQ or deadline that will manage the jobs all those work items will be sent to HQ that way so it's not that much different than how it worked before yes but PD G is not the scheduler it's just yeah it's just it's just putting it out there so yeah when when the done jobs are done in HQ it tells PDG so as they complete you get that feedback that the jobs have completed is that what you're asking well you while you're submitting it so you're not tracking the job in PDG it's just the way you would run a sim on PDG you would submit it and it goes to HQ and you wait until it's done and you get that data back so the difference is is that you'll be able to track as those jobs finish on the farm but once it goes to HQ HQ is dealing with it nothing is happening on houdini end anymore you're just waiting for those jobs to finish that I would have to ask Ken I'm not sure what they're doing right it's it's coming yeah I don't have a date for it yeah but it's coming yeah and a specific example for the game aside generations for the game generation sighs like ello DS yes yeah so I mean the barrier there so when I was baking that out that was a game rez object eventually so I'm using our baking tools to write out those textures and it also is at the same time doing a poly reduction on that stuff that then I'd then read back in and apply just this normal map so that's what that cop network I showed was generating the normal Maps and and the roughness and the albedo you know so yeah I think we have one minute left force integration I can't because I don't know it I don't use perforce that much but I could find someone that could talk about it for you look I can find him outside no I think we're out of time [Applause]

Channel: Houdini

Views: 18,790

Rating: 4.9810877 out of 5

Keywords:

Id: lL_jUZtaF1k

Length: 25min 55sec (1555 seconds)

Published: Fri Apr 05 2019