Convert Ai generated 2D images to 3D models for use in Blender and Gravity Sketch.

all right welcome everyone so recently I posted a video um on Instagram and Linkedin showing how I use uh AI generated images and convert them to 3D models using depth map estimation using a tool called zodep or Zoe depth so here's a quick overview on the steps I take to take this image convert it bring it into blender and then convert the vertex colors to an actual texture so this model can be used in other applications such as gravity sketch or Moto Maya any other poly modeling software you want to use so let's get started so you can see here I've got a few images that were generated using mid Journey you can use stable diffusion or just use any images you want but I'm going to choose this one here what I'm going to do is take that image and come over to I'll leave this link in the description it's a hugging faces space for zodep I'm going to grab this image and you want to make sure you have image to 3D selected and then we'll grab our image and just drag it into the box here and then hit submit and you'll see this only takes a couple seconds and it will start to generate this depth map first and then I'll spit out a 3D mesh which you can rotate and preview in this window here this is looking pretty good so what I'm going to do is download this and I'll just call it helmet and hit save so as you can see here it saved this glb file in the demo folder that I created you can save it wherever you want name it whatever you want and I have it default to open with the Microsoft 3D viewer so this is just a good way to take a look at it and see what it looks like you'll notice that the colors are a bit off it's a lot brighter the conversion process it must use a different gamma setting so everything will be a bit brighter but we can fix that later on once we create the texture so that's that so now I know this uh is a usable model and we'll jump into blender here all right now we'll just jump into blender here and delete the cube we'll do import and you want gltf 2.0 it's a glb or gltf file and then we will paste in that location and you'll see the file there just hit import that'll take about a second and once it comes in you'll see it's actually a just a shaded mesh that's because the mesh file itself has a or the geometry has an attribute tag or a color attribute that's the vertex coloring there so what you want to do is come up to the top right here if you're shading and just hit attribute and that will turn that on I also like to turn it to Flat just to get rid of any other shading that's in the scene so as you can see once you have the object in the scene it doesn't quite fit the world space or anything it's just floating object in there and you'll see that because the image was kind of extracted or extruded from a single Viewpoint you have some Distortion to the model and some flat areas that aren't quite the shape that you want and this is where the manual work starts to take over and I'll be doing that with the sculpting tool but first I kind of try and align this model to the to the space itself so I'll pick like the circular object here on the on the side and make sure that that is set up on a plane that I can I can start working from so I'll just start doing that right now it's pretty much just moving this around in the scene and then I'll I'll start sculpting it and manipulating it while I'm doing this I just kind of um try and think about where the center line of the helmet might be and what what things might need to um a line to be symmetrical and I'll throw in a plane here or a cube to kind of set that so while I'm manipulating it I'll see where that Center Line lands I also like to hit I'll select the object in and edit it so this way I'll just get rid of the vertices in the background so you'll see they'll all be selected so I'll just come here and delete them I usually use the lasso tool but I can use whatever you want so once I have this plane in place this is where I start sculpting um I'm not really going to be using any of the stuff over on this side so this kind of just gives me a guess on where the the center line of the helmet will be so once you're you have the object selected you just hit sculpt and pretty much I just use the grab tool make it kind of a bigger radius and start manipulating so I know from like this view I think this this stuff needs to be a lot rounder and pulled out so I'll start to just do that here uh we do want to be conscious of where this Center Line might be this is where I kind of toggle the cube on and off let's see where things want to be but it's really just kind of getting rid of the any any Distortion that might be in the helmet and kind of uh you want to make it you know a nice form so you can start editing it in other software good thing is having the texture on here allows you to um you know use it to line up things and look at the the details of it while you're manipulating it here then once I get it where I want it I'll come back over to the edit mode and we will just do a box select around all the vertices on the other side of the symmetry line and I will delete them now we've got all that deleted I come over and add a mirror modifier I'll connect to object mode and this way we can see it actually symmetrical come over to mirror and so once you turn that on you'll be able to see if your object is too wide or too narrow so what I'll do here is just kind of scale it out a little bit more and then this way you can start manipulating it a bit again with the other side there so we'll come back to sculpt mode you can see this is a bit pointy actually we'll turn on clipping so as you can see we do have a bit of a crease in the middle here it's a bit pointy but I think that's okay I'm I will likely be manipulating this a little bit more and creating my own topology so I can fix that sort of thing later on I'm just noticing this line here is a little weird so I think you know you could take this as far as you want but for for the needs that I I need it for I'm I'm pretty happy with this the this is a little weird the shape of the the top here but again I'm going to create my own topology over the top of this so this is really just a start on on how to start manipulating this this type of geometry and and create it so it suits your needs okay so for this part I'm going to show you how to fix the color of the image here as I talked about before the gamma seems to be a lot brighter on this vertex color versus the actual rendered image so what I'm going to do is show you how to UV map this and bake the vertex coloring to an actual image texture this way you can then export the model to any other software that might not support vertex coloring it would also allow us to decimate the model down this way you don't have to export this extremely dense mesh file into other software so let's start by coming over here to our scene render settings you're going to want to make sure you're on Cycles CPU device and then you'll come down to bake and under the bake options I'm going to change this to diffuse it I don't know what it comes default but I keep it on diffuse above surface is fine and then I'm going to uncheck direct and indirect if it's selected these are the contributions to how it will be rendered all we want to do is bake the color we don't want any lights from the scene to affect the coloring of the final texture also change the margin of the pixels this will have to do with the bounds of the UV it should only be one big island but if it breaks up into a multiple that will be the the distance between the islands so that's the first setting there the next thing we'll do is come over to shading and we will hit shift a search and we'll find an image texture we want to create this image texture so when it does bake it puts it into this here so we'll do it new we'll call it uh helmet speak you can name it whatever you want and I'll just do a 2K texture I don't know how big you can actually bake from vertex coloring but the 2K texture seems to be fine then we'll hit OK we'll make sure that's here eventually we're going to replace this color attribute with this texture sometimes I do notice that some of these settings do affect the actual bake so what I'm going to do is get rid of this principled psdf we'll do a diffuse bsdf um So eventually we will plug this into it but I think for now we'll just leave this the way it is or with this diffuse here and then now we will suck the object and come over to UV and it will be in edit mode and I will come to the side view um you this is a point where I probably should have done this earlier because of the way the original model was mapped it would be good to just project it from a single front facing view but since I've sculpted it it's hard to do it from a perspective but I think I can get away with just a side view on this one I don't think there's any geometry that might be tucked behind the model along the Symmetry plane so we'll we'll just do it from our x view then I'll hit in edit mode hit a and then I will select all the faces and then lock them up to UV and I'll just do project from View and Bounds and that will fill this UV space and it should apply this helmet bake texture if it doesn't just hit the drop down and hit that now it will be all black because there's nothing baked yet and we'll just come back to our modeling View and that should be everything we should once that's selected we just hit bake so depending on the hardware your computer has this might take a few minutes so I'll just time lapse this section and we'll see the final image once it's done oh there we go so it's done and you can see it baked it into this USB space Here and Now what we'll do is just save this image so I'll just do image and Save and I've already saved it once so just testing so I'll just overwrite that one save image as and that should do it so now what I'm going to do is just use Photoshop to fix the gamma as you can see with the original image it's a lot more saturated a lot more contrast so that's what we'll do uh in a second here so you can see here I've just opened the baked file in Photoshop and I'm just going to do an image adjustment and exposure and we'll just knock this to like 0.5 I think that I found that that works pretty well so you can see once you have the two next to each other that the saturation and the the contrast is about the same so that should do it and then we'll just hit save so once that's saved I can come back here and our shading okay so what I'm going to do is now come down to our mesh and delete the color attribute there so you'll have your model turn white and then once we delete this we can bring our image plug it in there and then we'll just have to make sure we turn on texture and there you go now you can see you have a nice higher contrast and more saturated image and that should do it for that we'll just come back to modeling window and do that same thing because I forgot it that window does that okay so once we have that we can turn back on our our mirror modifier and that's that the other reason to do this is to like I was talking about decimate the model down because it is so dense that might not be usable for some people so what I'm going to do is add a decimation modifier right here decimate and I'll knock that down to like um 0.01 something like that hit enter this might take a second to do but once it does it I'll show you the wireframe and that should do it so now you see we have a much dent a lighter mesh file so when you export this you'll just apply the modifiers so you have a nice usable UV map to mesh textured mesh so you can use it in any other software okay so for this part I'll just show you how to bring the exported fbx file into gravity sketch you can realistically bring that file into any software I just prefer bringing to grab a sketch so I can look around it and kind of design new elements and and check out the proportions and volumes at scale in VR so what I'm going to do is come to my import directory or my prefab library and I'll just find the folder that I saved it to helmet demo now this fbx file usually it should come in textured for some reason the latest files I've been exporting from blender aren't so I'll show you a little work around for that if when you grab your object and put it here it's just coming in as a as a white gray mesh so what I'm going to do is just rotate it so it's vertical and I can see my symmetry plane this way 135 millimeters seems small that seems big so I didn't actually scale this in blender so I don't know what size it is but we'll leave it at 152. and hit OK so since it came in without the texture what I have done is put the actual texture in my image reference folder so we can just drag that into the scene and use it to paint the model and it should respect the UV layout so I'll just grab the image here and then when I grab the object and hit the Color Picker I could just pick the cube because the image is in the scene as a texture and then I can just delete that and that's kind of a work around if your file isn't coming in pre-textured you can just do that so let's make another folder I'm going to just import another file here so we'll call this that's there so I'm going to do now is actually import another file that I know is the correct scale it's a scan of my head let's see where did I save that helmet so helmet one or head one to one I'll put that in scene let's see what direction is this I'm just going to rotate it oops that way is that 230 tall yeah that looks about right so we'll just stick that on this layer and I'm going to lock it we'll turn down the transparency a little bit this way I can grab this model and just scale it up to kind of roughly fit this so I'll swipe to the left there and grab my manipulator here I'm going to grab scale click the object and then hold snap and drag that to zero and then I can grab this handle here and and scale it up so I think something like that will work and then I'm just going to move it around in space to where I think it fits the head that looks pretty good I'll do this once again and just rotate um this object so click hold snap drag it to zero and it should snap to the origin and then I will just rotate it a bit and then move this into place and I think that should be pretty good might be a little off but we're just eyeballing it and this is really just to get a good look at it um and spin it around in comparison to an actual known object size so I think that's pretty cool um so from here you can start building geometry sketching over it you know maybe filling in the rear here with with information so you can kind of design the back of the helmet here if you want the finalize it so overall this is a a proof of concept process this is just something I found that could be useful for me you know whether or not you find it useful is up to you but I think it's a good starting point for using all these new tools like Ai and VR together you know and trying to come up with new ways to better your your 3D or your design process so I hope you enjoyed and we'll catch you next time

Channel: James W Robbins

Views: 235,928

Keywords: ai, Artificial intelligence, Blender, Gravity Sketch, VR, Virtual reality, 3D modeling, 3D sculpting, 3d design, 3D process, Midjourney, stablediffusion, depthmap, depth estimation, Generative, generative 3D, 3D ai, ai 3d

Id: Wf-OmHyFduo

Length: 18min 31sec (1111 seconds)

Published: Wed Jun 07 2023