How to Render Vertex Colours in Arnold for Maya

this is Malcolm 341 in today's video we're gonna look at how to render vertex colors in Arnold where colors are super powerful so let's get into it the time has finally come I have been waiting 18 years to render vertex colors inside of Maya Maya software can't do it as far as I know and when we used to have mentalray in Maya which is the worst rendering software I've ever used I couldn't figure out how to do it maybe it's not possible if it is possible it was incredibly cryptic and no one on the internet had a tutorial for it so now that we have Arnold inside of Maya we can all do this it's still kind of cryptic but I'm gonna show you how it's done okay so first we need to actually have some of our colors on our object so I'm just going to go up into mesh display and then I'm gonna go to the apply color tool and I'm going to open the options box with this tool basically what you can do is you can just select some verts and apply color to them with a slider bar here replace whatever there's a bunch of other options in them but for this tutorial we'll just put some basic colors on it so the first thing that you want to do is you actually want to make sure that the color is white and we're just gonna flood fill so you hit apply and the reason that we're gonna do that is because when you go to render these if the default was gray or whatever gray is basically no vertex color and so when I was first testing this I was like oh it's not working and that's because I accidentally didn't have white as my base color so make sure you assign white first and then from there we can apply some other colors it doesn't matter because whatever we're just this is just for a test or whatever so I'm just gonna select some stuff here and let's apply some red sure enough and then I'm gonna select some stuff here let's go here let's say and we'll do shirt whatever green doesn't matter it apply okay so we've got some colors on there so what we're gonna do is we're gonna build a shader inside of Arnold that is going to take our base material which will be like the standard shader which is just gonna be grey or whatever color we assigned to it and then it's going to multiply these vertex colors on top of that so tint everything below that layer with whatever color we have in the verb all right next up I'm gonna open up the hypershade by clicking this weird little blue eyeball and then I'm going to click here and I'm going to press tab I'm gonna search for standard as its standard standards surface that's what we should be using in my 2020 that's the built-in shader that is simplified to work with Arnold's less complex than the AI standard or whatever it used to be called so we've got our standard surface and whatever one's gonna put the weight up to one the color is white which actually looks kind of gray or whatever and you're gonna notice I'm gonna apply this to my object they're assigned material by right hold clicking you're gonna see the vert colors disappear and that's because the standard shader as far as I know it doesn't display textures or word colors or anything it's just kind of like a placeholder which is super annoying I don't know why they would implement like a better shader that doesn't even give you rudimentary colors in the viewport but just so you know it's gonna look wonky in the viewport but when we render it in Arnold it will look correct and I'm just going to close the hypershade for now we're gonna come back to this and build the shader in a minute but we have to do some other stuff first so close that guy down and then I'm going to go up into mesh display again and we are looking for the color set editor I'm going to open that up and here you're gonna see this funky name and that's the name of the color set that holds the vert color that we applied to the object and I'm just going to rename this only so it's easier for me to remember you don't have to rename it but you'll see in a minute we're gonna need the name so I'm just gonna call this YouTube and there we go there's our color set whatever close this down okay next up I'm gonna select my object and press ctrl-a on the keyboard to open the attribute editor and then I'm gonna come down to the Arnold section and we need to turn on export vertex colors if you don't turn this on for the meshes that have vertex colors you obviously won't be able to render them in Arnold so make sure that guys turned on close this down okay next let's set up some basic Arnold render settings here so we can see what we're doing so let's go to the renderer and we'll click on Arnold and then we'll click on the little crop box I'm just going to move this over to the side here and sure just drag this guy out drag this guy out and then I'm gonna click play to turn on the IPR render and we're going to get nothing because I don't have a light in the scene so next I'm gonna go to create and go to lights and we'll just do a directional light okay I can't see my light so I'm gonna scale it up so I can see what I'm doing and sure whatever don't really care we just want to be able to see what's going on in the view so as you can see right away we've got our Arnold shader and I don't see the vert colors so now we have to do a little bit of material hookups to be able to render these things okay and this is gonna be really disgusting because I'm going to have to have the hypershade open big enough for everyone to see but it's also going to take up like too much of the screen so it's gonna be kind of brutal here so open the hypershade again and if you're familiar at all with the hypershade you're gonna know that all of the Arnold stuff can be found here by clicking on Arnold and then expanding these sub groups or whatever so we're gonna go into utilities stuff here and then if you click on math for example see there's all of these things so there's a ton of them so I'm going to be using the search to find stuff because I already know what it's named so what I want to do is I want to press tab and then I want to search for what is it user a I user data color perfect hit enter okay we got that so this is the thing that lets you use the vertex colors so if you actually go into here you'll see that there's an attribute setting here what you need to plug into here is the name of your vertex color set and we named our vertex color set in the earlier step to be YouTube that's why I renamed it just so it was easier to remember okay and next up we're gonna create a multiply node so again press tab and you a I multi a I multiply and there we go this Arnel thing I have to say is really awesome the material workflow the shader graph or whatever really similar to unreal so if you've ever used unreal and you kind of know what you're doing in there a little bit it's gonna be so familiar to create shader networks in here or vice versa so this isn't wasted work if you learn something like on the maya side it kind of has all the same rules and stuff so like you would do a multiply and unreal and it would like plug into the base color or whatever it's the same thing here so it's super cool okay so basically what the multiplied node does is it takes two inputs and then it has an out color from that so it's going to be like number one multiplied by number two multiplied on top of the base color so our base color being the gray color over here and some color data coming from our colors so we want to take out color and we'll just put it into input one and I'm gonna click on this node and so you can say input one here is fed by this so that's our bird color input two is white so it's nothing because it's a multiply so white is transparent and then we're gonna take the out color of that which is going to multiply that on to our base color and boom you can see as I hooked it up there it's awesome about the IP ours and updates in real time so you can see their server colours rendered inside of Arnold if yours don't show up for some reason I found almost all the time in Arnold this IPR won't update if you're like filling with stuff so going to render and say update full scene if you've set everything up and it's just not working always click that when in doubt render update full scene so that's awesome like I said I've been waiting you know 18 years to have this feature in Maya and it's so easy you can just build your own shader now and do whatever you want in the ancient times we used to actually have to light scenes with vertex colors this is an awful system terrible like hand coloring each vertice to apply lighting through it this long before the days of global illumination or even light maps and video games and so this was really helpful like when we used to have to do marketing shots it would be like sorry we can't render the lighting let's take screenshots of the viewports terrible now it's not very useful anymore to have the colors be applied as a color I mean you can use this if your game or whatever uses where colors for AO or something like that you can render that stuff right here in the viewport but what is super powerful over fix colors and what is commonly used in most games including the last game that I worked on is we would use the vertical is to actually mask between textures so you would do like a blend shader and unreal and you would paint the RGB and each different color would be like a different material that you could blend in on top so like moss on rocks there's snow on rocks or whatever grime on concrete or whatever 30 buildings broken bricks so what schools you can build those shaders right here inside of Arnold and we're going to take a quick look at that just before we finish the video up so first I'm actually going to go back into the mesh display apply color options and I'm just going to flood fill this with why just to reset it whoops not selected let's go with white okay so I did the flood fill with white and nothing happened and I met you that's because I have to use my old friend here update full scene yep there you go so when in doubt update full scene just to simplify stuff we'll just work with one color for now so I'll put some red two here red apply same thing there you go a bit full scene boom there's our red get close this guy out alright and next up we need to update the shader or build the shader or whatever so let's go ahead and press tab and search for AI image yeah yeah image and I'm just gonna duplicate that guy by pressing ctrl D so we need our two textures and so I'm just going to I'm gonna move this over it's so annoying so we go into here and then we paste the texture name here which I already got from copying pasting it in a different window so say okay okay cool select that guy paste this and it's that hit enter it's so annoying that you can't see the preview of what it is like everything in Arnold is just missing icon or just a gray you know what's up with that that's terrible maybe the film industry is used to that but I'm not used to that maybe I was spoiled working in games where you can actually see a preview of a texture okay and so the way that you make a blend material or whatever a layered material is you create a layered material inside of Arnold so same thing I'm gonna press tab and what it's at a I layered shader and a layer shader it's kind of the same as what the definition would be in Maya it takes inputs that have the different layers and each one of those layers can be a different material and I'm just gonna make this fullscreen for a bit cuz it's going to be impossible to see this on the video okay so for our layer shader let's say layer one I'm pretty sure we're gonna find out in a second but I'm pretty sure this is the absolute worst design visual design for how this thing works Maya got it wrong they do their layer share horizontally which is terrible these guys got it so wrong they do it vertically but it's inverted so layer one I think is at the bottom but they put layer two below it yes that's right Photoshop got it right people why are we trying to do anything different bottom layer is rendered at the bottom top layer is rendered at the top not top layer is rendered at the bottom so confusing so if you think about the number instead of the visual it'll make more sense like one is actually down here two is actually above it so confusing so layer one will just be our basic tiling texture so first we need to create a standard surface so tab standard surface blah blah blah whatever and then in the out color we can just delete that connection because it's gonna get plugged into this guy here and so what image is actually in here okay that's the mossy one so that's the wrong one so the base image is going to be the first one so I'm going to take the out color of this sky and plug it into the base color so basically we will have the not mossy texture being the base and then let's move this over I will take the out color of this and you want to click on this guy see through how there's no like input or whatever it's kind of weird you have to click that guy so it shows up here and then you middle Mouse drag hold middle Mouse and drag this into the input which is a standard Maya workflow it's just weird that they don't have the pins there so you say okay and then it reveals itself it adds the input another terrible design alright cool so actually let's just take a look at that so I'll just move that off the screen there select this guy and apply the AI layer shader and there we go cool okay so we got some like random tiling rock texture going on there I'm just gonna increase the light as well because it's like so dark I'm just going to an attribute editor for this whatever just make this a bit brighter so we can see what we're doing 3 sure 3 looks good man we will go back over to this and make it fullscreen and let's create our second layer so again we can delete this surface shading group or whatever because the out color of this is going to go alright so select this guy and then the out color of this is going to go into whoops I forgot to enable it so click enable layer to out color this is going to go into the input there we go connect that guy up there's one texture there's another texture plugged into it and let's just have a look and see what's going on there so that's the old part of the shader so you can see it's like rendering the old thing when we just update yeah so that's the top layer so you can see the layer shader is pretty much working already let's just take a look back at the layer shader and to test this you can just do the mixing so remember layer two is on top even though it looks like it's below so I'm gonna fade the top layer and you can see over here it's revealing what's below and the reason it's not masking yet is because we have to do a little bit of hookup to just tell the bird color to mask in the other texture instead of just rendering the bird color shader but you can see the layer shader is currently working so this is a good way to test your work when you're debugging it so what we need to do now is we need to get rid of this multiply because that's causing the vertex colors to render on top of this layer just going to delete that connection just move these guys out of the way for a second okay and then we will put the out color of this guy into the base color and that will give us the mossy texture on top there and then I'm just going to click back on the layers a derp and so I think what we want to do is let's just test this yeah so the mix is what controls the transparency between the two layers so layer two is on top and if I mix that to zero you see the texture underneath so what we want to do is we want to hook up the vert colors to talk to the mix because we have vertex colors you can do RGB a in this case we'll hook up the red vertex color to act as the mask so wherever we paint red on the vert colors it will show the moss and wherever we don't paint red it will show the underlying texture okay so we already have the vert colors from before and the multiply node and so now we just need to hook it up into the mix of this guy mix too right because that controls the transparency yep okay so again the super weird thing about these Arnold shaders is you need to middlemost drag on top to reveal the nodes I don't know why they don't show up by default and then this one's even weirder cuz it's going to bring up the old-school menu click this guy and then most dragged on top of this thing to do the mix thing layer to mix one yep and then it's going to bring up this ancient editor that I hate I hate this thing it looks like it's from the 1970s so we want to do out color our because all we care about is red and we want to do mix too I guess right click it close it ok so the mix is hooked up and it looks like something weird is happening here the moss is showing up everywhere even though we've just plugged it in to be the red panel or whatever and I think that's because I have white filled in all the other channels so let's just do a test here just gonna move this thing out of the way of it here and go back into mesh display apply color and then let's just flood fill the whole thing with black but somehow it's selected and then I believe I need to do the update full seeing seperated the bird colors there we go okay that makes sense ok and so now I'm going to set this guy to red and in theory now wherever I select and I flood fill that's weird there we go ok so I actually stop the video to figure out what was going on there because the whole tool kind of becomes pretty useless if you can't paint and see what you're painting in real time so the reason that this wasn't working is because I had deleted the history off of this asset and I guess when the history is not there it's not creating a special node that you need here which is the color per vertex piece of history and so on the other one I had created the sphere and I didn't bother deleting the history and then I applied the vert colors so let me just show you what happens here so mesh display apply color whatever I'm gonna go red and hit apply and you'll see nothing happens also see over here see nothing's going on there but if we add a bit of history to this object so let's let's go to mesh display I'm just going to tear this off here so I'm just going to reverse so flip the normals but the normal is one more time flip them back and now you'll see see I've got some history here in the stack now watch what happens when I apply the red color now hit apply boom it updates in real time and that's because over here see I've got the Holi color per-vertex node in the history stack so we don't need this guy so we can actually just go into here and go delete node and that other flip normal delete node and so we've just got this left over so if you don't have this it won't update in real time okay so I'm just going to flood fill with black here and then go back into mesh display the paint vertex color tool options box and I'll just do a reset of the tool okay so we want to paint with red and when you have this tool open you can hold down B on the keyboard and then hold down left mouse and drag left or right to increase the brush size all right cool and so now we can just paint and as we paint you can see updating a real time I'm painting the mossy rock texture on top of the regular rock texture so super cool you can build your little blend shader in here and kind of do whatever you want there we go maybe we should tile this rock texture a little bit you just find out where the moss one was I think it was this guy whoops oh I have some weird setting turned on don't I do bugs shading to save old sorry about that it's actually a really helpful feature we'll look at that in different video UV coordinates on my one let's go two by two shirt we can even palette more whatever four by four cool okay so close this guy down and just go back into the tool here so I'm just gonna flood it with black one more time and then she's red so wherever we paint red vert color it's gonna mask in the second texture and wherever we paint remove red vert color it's gonna show the underlying texture so pretty neat and then you can hook up another two or three textures and instead of using red you can use green blue and alpha and just plug those in to the shader how we did before from the other node plug them into the mix or whatever and you can also erase by holding ctrl so just erase some of that stuff there now this mask actually looks okay here but I think it's kind of tricking us because it's rock on rock with moss you can't really see the blending marks but we're just doing a straight like linear blend or lerp between the texture and the vertex colors you can see it kind of gets like a fade a transparent look there and that wouldn't work for like rust or grime or something like that on a different material but out of the scope of this tutorial because like breaking up the edge mask is like a whole nother tutorial in itself so super cool we learned how to work with vert colors render them use them for something more useful which is piping them into a vertex color mask and a little bit of Arnold stuff altogether and here's that same material applied to a quad so we can just maybe see it a little bit easier here don't forget when you create a new object that you need the poly per color vertex thing and you also need to go to the attribute editor and turn on export vertex colors and make sure that the color set editor is named the same as what's in your shader graph so paint some stuff yeah pretty cool check this out sweet I need some Moss paint some awesome awesome awesome awesome awesome awesome awesome awesome then erase some Moss it'll just make the brush smaller erase erase erase erase some more moss thank you very for watching this video without viewers like you this channel would not exist if you liked this video and enjoy the channel please support me by purchasing something from the online store each purchase goes towards creating more video content and keeps the channel ad free see you next time have a splendid day

Channel: malcolm341

Views: 8,972

Rating: 4.9756098 out of 5

Keywords: Maya, autodesk, time, saving, tricks, secret, secrets, tips, 3d modeling, 3d modelling, easy, fast, tutorial, workflow, pipeline, best, mel, script, malcolm341, 2020, 2019, 2018, 2017, 2016, 2015, video, quick, free, beginner, learning, how, to, learn, polygon, polygons, hotkey, hot, key, use, save, tool, plugin, tools, vertex, colors, vert, arnold, solid angle, material, shader, layer, layered, blend, Unreal, paint, painting, node, network, hypershade, render, rendering

Id: Cm-pqfLDHB4

Length: 21min 40sec (1300 seconds)

Published: Sat Mar 14 2020