Convert an AI generated image to 3D

hi there my name is pete mcnally i have been doing some experiments with mid journey ai and taking the content from mid journey into full 3d and it's generated a bit of interest in social media i've been asked to do some sort of a tutorial or time lapse of my methods it's pretty simple if you have you know some rudimentary 3d skills um it doesn't really matter what apps you use i myself will be using photoshop i'll be using nulled but you can use crazybump you can use some of the substance tools um i will be using 3d studio max but you could use blender or whatever your 3d model is of choice and finally i will be rendering in marmoset toolbag 4. so let's get to it and this is this is a variation of the egg that i was using it's like a biomechanical gigaresque faberge egg dreamed up by mid journey we're going to ignore the plinth underneath at this stage we're just going to kind of concentrate on the egg so what we first need to do is extract a kind of a coarse depth map and there are numerous methods for doing this as well there's um midas there is um photoshop so for simplicity's sake i'm going to stick with photoshop for this one so up here in neural filters in the latest photoshop you'll see that there is an option called depth blur and this is to kind of add depth of field effects to a single image but if you scroll down here you will find output depth map only and that will give you a black and white depth map it's not accurate again even in the finest mid journey images perspective can be an issue so even if we had perfect perspective here from a photograph the depth map would be an approximation but hopefully this is enough to work with hit okay and i'm outputting it here to a new layer i'm going to hit control and i to invert this to something more traditional and i'm going to save this out as a png i'm not too worried about bit depth or anything like that at the moment going to save this as uh underscore depth okay that's a good start for where we're going next i'm going to jump over into nulled and i'm going to go back to the original image here from mid journey i'm going to copy it to the clipboard and i am going to paste it as a color map here so that's what we get not going to fiddle about too much um this looks pretty good to me in terms of normal just to help some final shading so if i hit update integrator over here you can see that it's also trying to give me back a pixel based shape from shading height map which is more detailed than the depth map that we pulled out with neural filters but also not particularly accurate so we have two approximations to work with but if we have a look at its effect on a cylinder here you can see how it's pushing away from the cylinder and it's giving us the type of silhouette here that's actually uh pretty desirable for for what we need okay so i'm going to take the normals copy that to the clipboard paste that into what is now a psd in photoshop i'm also going to go back here to the height tab and i'm going to copy that and i'm going to bring that back in here too so i'm going to save this second height map also as a png and let's call that biomec egg height so we can differentiate it from the depth map good stuff turn off that layer and then also save out this normal map great so that's almost everything we need i am going to go back just because i'm anticipating some shading issues and i'm going to grab the ambient occlusion from here as well now it's not true ambient occlusion but if we need to darken some of these recesses and kind of block specular from getting in there then this is going to be useful to us later on and you can output all of these quickly from null but again i'm just trying to keep it very transparent and very familiar and to somebody who might not be you know totally familiar with the other apps so this is the last one i'm going to do and this is kind of just for safety reasons okay so now i'm going to switch into my modeler which is 3d studio max and i use this reference planes script to quickly set up reference so i'm going to drag in my original mid journey ai and you can see it creates a plane here that has the proper aspect set for the geometry so nothing is distorted or pushed or pulled in either direction so that's pretty much all we need to start off with so in the front viewport then i'm going to create a cylinder or a sphere rather and i'm going to just try and pick the midpoint there and drag it out so that the radius more or less touches the edge and what you'll notice is that there's a pole here where a bunch of edges connect and i just want that to be kind of towards the top um of the image where it's less less important to hold on to detail i suppose i'm going to be reapologizing this in a little while so the next thing is to make this sphere into something more of an egg shape so i'm going to right click and make it see through and i'm going to use an ffd freeform deformer modifier on here and then manipulate these control points to kind of better fit what we have here i'm going to try and stay inside the lines and that just kind of avoids messiness when it comes to displacement and you know we want to get a nice symmetrical um egg out of this as well so something like that looks okay now obviously the image we're working from is not symmetrical but um i don't think it's going to be an issue again just to stress that the issue here is to do as little work as possible manually um you could get a lot better results where you to sculpt and paint over this and certainly in terms of you know preserving um detail in certain areas and emphasizing details in certain areas just pull that up a little bit and it'll pull these edge pieces down so whatever your favorite method is to go about making an egg shape from a sphere okay great stuff so the next thing i want to do is i want to um just use half of the sphere here so i'm going to go into the top viewport i'm going to add an edit poly i'm just going to delete the backside facing polygons from this sphere so i'm going to keep everything here and delete everything highlighted in red hit delete and it's gone and that just makes the next part a little easier to set up with uvs and stuff because what will happen now if we try and just project what we have on here um we're gonna get stretching at the edges and it looks really kind of amateurish and um just just bad kind of production values i guess um when you do it like this so i'm going to acquire the uvs from the the plane behind us i'm going to turn off i'm sorry i didn't mean to go into that turn off see through i'm going to sample the material that's on the reference plane here and then i'm going to apply it to the sphere that we have selected okay so it's on there now you can see that we have a little bit of 3dness but on these very edges because of the projection we've got all this stretching that's something that we really want to avoid because it's just going to ruin us it's going to make it very obvious this started life as a 2d image okay so to get around that we are going to add get rid of that on uvw add a new one okay i'm going to grab all of these uvs here and i am going to planar map it from this position here so planar and best align okay so you can see that there's an image of the original mid journey texture but it's just not quite right for what we want so the next thing i'm going to do is hit this button over here this is 3d studio max 2022 i'm going to hit quick peel and that's going to just relax those edges so that we don't get such a harsh projection as before and it's also upside down so i'm going to just flip that 90 degrees and 90 degrees again then i'm going to use the reference image that we have here just to go full screen on this i'm going to then just massage this these uvs in to better fit what we have here and again it's it makes life easier if you just stay inside the lines even if you're going to miss a bit of this detail at the edges okay i think that's pretty good again this is about doing the least amount of work possible this is the lazy man's answer okay so those uvs look okay to me as far as i know um let's just have a look and you can see that along these edges here like it's not perfect but that stretching is gone and that's that's kind of the key thing to avoid here okay so the next thing i'm going to do is i'm going to restore the rest of my egg with a symmetry modifier and that's just going to bring back my entire egg shape okay so i can go back to edit poly now um and you know sometimes with this weld you don't get a solid mesh this red line here represents an open mesh so what i'm going to do is i'm going to just select these vertices here and i'm going to weld them and that's going to close the mesh and the reason i'm doing that is because sometimes when you have open edges and you're using displacement you can run into issues um pieces displacing where you don't actually want them so the next step then is to add some more detail to the geometry so i'm going to add a turbo smooth which is basically just like subdivision so the egg is going to get smoother and at this point i'm also going to add i'm actually going to add a retopology modifier just to get rid of these poles at the top and bottom so let's say something like 12 000 polys we want to keep those uvs that we've just laid out and hit compute okay pretty quick and although the mesh is not perfect we have got rid of those pinches at the top so that's pretty that's pretty good all right that's good stuff so the next thing is is to actually add some displacement on there so let's go ahead and add a displaced modifier and we're going to use those uvs that we have laid out so we're going to use the existing mapping here and i'm going to bring in the depth map that we had from before okay and i'm going to go into clay mode here just so you can see the effect a little easier about the effect that it's having on the geometry okay so probably turn on luminous center for this one and then just push the strength until we see what's happening there's definitely something happening there i mean it's not crazy it's definitely you know it's introducing some noise there i can see some of those kind of uh geiger s patterns starting to fall out a little bit and sometimes you just need to play with the the luminance center to try and keep it keep its shape a little bit more so let's just maybe push that out i don't want to go much further than that at this point to be honest with you i'm going to say okay that's pretty much good enough for for where we want to be okay so i'm going to add a turbo smooth on top of that again and we can probably go one more without too much issue okay and i'm going to add another displacement modifier on here now again and again we're going to use the existing mapping and this time we're going to bring in the second the height map that we generated in null which is which is more detailed okay so now let's see what effect this has on the geometry underneath okay that's definitely getting towards the level of detail that i'd be quite happy with it's losing its egg shape a little bit but i mean that's something that's kind of easily corrected and we've got a little pole at the top there which doesn't look great but again i think we can live about it for the sake of this tutorial so some nice geiger-esque shapes going on there i really kind of like those so let's press on with that let's call that done let's have a look at how it looks when the texture is applied now this texture is is not lit at the moment so if i just bring down the self-illumination here the emissive we'll get an idea of how this actually looks so it's nice to see this kind of whitish detail against the darkish detail and it all matches in rather nicely so i'm going to make sure that this is on my origin i'm going to check the actual physical size of it and it's like 7 meters by 9 meters at the moment so i'm going to take this down to maybe yeah maybe just about a you know a meter something like that will be good enough reset the transform convert it to an editable poly and i'm going to export this as an fbx so for anybody who's just wondering about that process i think that covers it i'm going to just do a little bit of context rendering and material setup inside toolbag four right here so i'm going to drag in my biomechanical egg fbx okay there we go i'm going to build up the material you can see that at the moment it's being lit by this um environment this interior environment here so that's what's happening so the first thing we're going to do is just bring in uh just some some sort of a ground plane and we can actually create some simple primitives here inside tool bag and this is for lighting more than than anything else just to have it sitting on something okay i'm just going to put that off kilter a little bit and i'm going to copy one out just for the background embed it a little bit and also rotate it off kilter something like that and we'll do one more something like that there okay cool stuff so what i'd like to do now is turn on ray trace mode and this is going to make everything look hopefully a lot more realistic look at my camera just frame up here a little that one there should be far enough back that we don't need to worry about tangents okay so um quick material for these guys then um let's bring in the textures that we have already i'm just going to use the original mid journey um image as the albedo you know you could you could delight you could do all sorts of things here but this is going to be fine i think for our intents and purposes and you can see how the the silhouettes are really picking up the the light quite nicely there very geigerreskin style and we can play with the roughness then to make that kind of look you know um super icky and gooey which is cool and the next i'm going to bring over is the normal map and what i like to do with this one is because it can be quite heavy-handed in terms of the amount of normals that's going on so what i like to do is i like to use the details normal parameter over here and i'll copy this to the detail normal slot turn off the original and then you have this detail weight control and you can bring that up and down um which gives you a little bit more control over the application of the normal map in the runtime here so i just wanted in there you know a small amount something like that the next thing is the occlusion let's drag that in i'm going to put it as a cavity map and what you'll see is hopefully the crevices here will will darken up and not retain so much of the actual specular and we've got sliders here to control that a little bit so that's pretty good that's pretty good um just takes the uniform roughness off it and that is pretty much our material and what i also like to do with this geiger esque stuff is to add a little bit of subsurface scattering in there and that just makes the whole thing look a little bit more organic especially when you've got you know a bright light really filling out that scene a little bit more i'm going to just give that a diameter soften up those shadows and then we're going to go with um the one that requires the least amount of manual work is volumetric scattering and if we pick a kind of a an icky kind of fleshy brownie kind of a resin-esque color i guess and play with the scattering you can see that we start to get this really if we turn off the albedo here you'll see the effect that it starts to have um just lets the light in under the surface and and bounce around and just give that organic skin like feel um which is quite nice so i think the last thing then to do here is maybe just bring in a little bit of depth of field too much obviously just want to get those in the background a little bit out of focus and focus upon our guy here and then find some nice moody lighting that's appropriate for a geiger scene i'm going to bring down the background light and bring up this light over here and then maybe just add in a bit of a rim light just hitting the edge here just to pop that off from the background something like that so that's pretty much my mid journey ai to 3d workflow as it currently stands so i hope that clears up a few things thanks for watching

Channel: Game Art Experiments

Views: 320,236

Keywords: 3d, game, gameart, material, shader, model, texture, real, time, realtime, gamedev, midjourney, 2d to 3d, midjourney to 3d, ai to 3d, midjourney 3d, convert ai to 3d, transform ai to 3d, 3d from ai, aicollab, disco, diffusion, DALL·E 2, generative, art, clip, midas, conversion, 3dsmax, toolbag, marmoset, knald, artist, concept, prop, environment, model from ai, raytracing, rtxon, sss, volumetric, scattering

Id: 8uh-qFOi7lQ

Length: 20min 31sec (1231 seconds)

Published: Fri Jul 01 2022