DepthTexture for Geometry with Three.js #49

hey morning everyone my name is yuri and we're gonna live code today well probably more decompilation kind of stuff today but still let me know if this sound is alright and the video is alright and let's get started hey youtuber on youtube yeah i'd be curious to know where you're from as always we could just people home and mr waksu good morning everyone hey everyone so today i'm to be talking about this amazing website well actually recently there's been a tweet about 3js that they've just added 36 new cool showcased websites on the homepage of the 3gs and i was like okay 36 streams coming uh actually did some of those like this mav farm and maybe maybe something else i don't even remember now probably the only one that i did out of this all right so today we're going to be doing this one voice of racism i'm going to explain how it's done and we're going to create a simple demo resembling what they did there it's a pretty amazing website you should go check it out and it's done with 3.js and it was done with a new kind of new concept to me and i decided i should try to implement it and use it so first of all yeah you got money in everyone so first of all i'm going to try with well this website is kind of complicated a lot of different tech there so i'm going to start with first part of the tech which is not related to the code in any in any way but and then we're going to code to this simple demo yeah let's do it this way so when you go to the website first thing that amazed me is that they have a lot of messages there it's all about races but um the person i mean the 3d kind of person that you see is actually pronouncing every single message correctly and that was kind of like how how did they do that it's like maybe hundreds of messages there i don't know or dozens for sure and i was kind of curious how they did that and well it's really it's really interesting how they did that so if you go into the network tab and they have this face mesh glb okay i just i just i just want to download you okay actually download it already so we have this face mesh glb and then for each message they load two new files like the mp3 and then the new glb and i was like wondering what kind of glb would you need to load for just pronouncing message and that's kind of interesting if you go to gltf lawyer if you open up the face mesh this is the model that they used and it's beautiful low poly model and what's interesting about it they have a lot of morph targets there so you could just do like all kind of stuff like this kind of stuff or this i don't know all the crazy kind of stuff i think we it's mostly about the mouth here but i think there was something about the eyes too i believe maybe i'm mistaken so you could do any kind of crazy stuff here okay i gotta switch the screen probably it's gonna be much better why didn't you tell me so you can do like different kind of faces they are like any kind of faces actually and of course that was prepared in blender or some other tool but what's interesting if that i go into this each of those files that you load for each of the messages like they have 175 glb and then mp3 so mp3 is obviously the sound of it like the phrase that they say but the glb that's that's what the interesting for the glb part come on okay the glb like remember we have this joe open mouth funnel and all that kind of stuff if you open up the glb file it doesn't have anything and it's really small it's like 20 kilobytes or something so for each message it's it's really really small it's like 39 kilobytes here it's pretty small and if you open up here it has nothing inside for just one dummy object and then all the animation so they actually saved all the mouth movement that you need to do to pronounce the phrase into the glb animation i'm wondering how they like what was the pipeline to do that it got to be some automatic conversion algorithm that converts the how you pronounce the words into the mouth movement and it's pretty that's pretty cool maybe they coded each each letter and then like but then again well some some letters are not pronounced in english so i'm not even sure how they did that maybe just by hand manually i don't know but that's pretty fascinating stuff you could see all the animation like uh all the corresponding parts of the face are moving within those animation so it's pretty complex stuff i don't know how they caught it it's like 12 objects moving at the same time so that's how the saying something works but you don't really see this 3d model here right you just see some weird representation little lines and the mesh it's not it's not actually a 3d model that we see and that's the part i'm going to call it now so i don't even know how they made this pipeline and that that that is interesting yeah i hope they will make case study someday but what we will do we will do this animation well kind of it without the animation probably but just the the concept of it so let's do it uh i'm gonna call it voice i'm gonna get my 3gs starter boys i'm gonna start it over hey in the end hey philippines amsterdam too all right so template is starting and first i need some i need some object i believe i had i had this face of gal gadot and i i like gal gadot so so i'm gonna use her face and then it was face full jv i guess yeah i'm gonna just get myself a model here face food will be let's just check if it's really something i can use yes it's just a woman face and that's it there's no animation here nothing nothing whatsoever just a simple mesh i think it's like hundreds of kilobytes oh maybe but then you have to implement yeah maybe that sounds but that's still kind of a complicated thing to do just for the website you know so that's that's that's what make it cool so they went over the boundaries like you're just wondering how the hell they made it it's like it's interesting that's what makes it cool so i need to open the website now [Music] i'm going to do a short window layout kind of stuff we would be able to see both of them all right so you could see that they actually load every single new message they have lots of them all right where do we start here we need to load the model first of all we need to load the model base full glb it's called so let's load it up to load the model i need the gltf loader have it commented over here so let's just start with cheese this loader equals and then i need to move chat good moment um boom boom boom shelter floater so i need to do the loader load kind of stuff model and then i'll have gltf over here usually you could just do this but i think i gotta get steeper than that i don't see anything maybe the model is small oh yeah well that's that boomer is too small for me so i need to change that [Music] so let's see what's inside the gltf object we have a scene and we have a children so probably need the first item from the children okay then i need i think the position is zero zero but i need to do the scale set and then uh i would guess it's 2000 and then i'm gonna add this this model this in this model hey double okay i think 2000 is too big maybe 500 600. i need to move it a bit on the y-axis minus one that's too much i mean minus yeah kind of so we have the moon center now i could assign the material do i need to do that actually well just to see the that everything is correct and just to show you how it's usually done so i gotta traverse the material i mean all the objects in this scene and then if that's actually mesh because that could be anything like a camera or something and then oh calls this yeah we have the woman's kind some kind of uvs that we have on on her i don't actually need those uvs but anyway something is working so i'd probably move her a bit behind like z axis minus one should be enough it stays a little bit behind so so we have some simple setup we have the plane we have the model the next thing i would need to do the next thing i would need to do i'm going to learn about some depth so i actually took to render this kind of thing that we have here i need i need something to distort those coordinates with like something to just sort those lines so i could build those lines easily it's just a loop really let's just build them for like so we could have them for later use so this geometry one equals something kind of line so that's the width and then that the height should be real small and i want to make it really detailed on the x axis and not really the tilt on the way so then then i could create a lot of them i could do the y and then 50 let's say 50. number and i'm going to add this one to the scene and this geometry is this i have to create the mesh with this geometry one and then the material well that'll be the same material and then mesh position y equals what this has got to be i minus number divided by two divided by a number divided by two so there's got to be something between minus one and one let's see so i have lots of them maybe they're just too high yeah still too high maybe 50 is just too much for this kind of height yes now we have those kind of lines and this is like what we're going to be using for rendering i will command that out yet so i need something else later we're going to use them maybe like this yeah i think it's better and like this we have them exactly where we have the plane all right perfect let's leave them for now so what i want to do now i want to render the depth of my seam to the texture and this is done with the depth texture so let's see the depth texture example this is exactly the class used to save the depth information and what i want to do when i render my scene to the depth and see like how far each of the points in my scene is but save it to the texture and this there's an example for that not really amazing in any way but still okay okay okay thank you so there's an example here another example is doing we're actually creating another texture like this one let's do that let's do that so when we do this i'm gonna do create another texture like this target this width this height okay i have it performer do i need this should be false then the format should be the unsigned short type format unsigned short type all right and then the type should be the depth format okay so far so good now could try to render to this texture probably and this should work for me right so i have these let's see how they do that they actually setting up the orthographic camera and i think this is that would be better for my example but let's stick with the perspective camera and then they're using some shaders of that and then when they render they first render this scene to the target and only then they render it to the actual scene okay so let's do this okay let's do this so so far we just have today i'm gonna remove this just have the usual rendering of this scene i'm gonna remove that in the moment so nothing should be rendering now all right so now we have this renderer set render target to this target then we render this scene to this target and then i could use this target as a texture so to use this as a texture i'm going to create this depth info set it to null and then and then here i'm gonna do this material uniforms depth info value equals this texture right this target texture all right after that i need to actually render this scene again to see it on the screen let's render target and then just render the usual render that i have all right okay it should actually just work at the moment it's not because it is not defined and of course of course okay so it just just works now it has some errors but who cares just works at the moment some warnings probably all right so we are rendering something into some invisible targets render webgl target how do we see this to see this we should actually look how they use it to use it i would need to [Music] i would need to use this function read depth which just read depth from this texture and from uvs okay so let's let's use it i also need the camera near and camera far and obviously i need that because to store depth i need to know like what's the scale of measuring that depth like how far we start measuring and when we do finish measuring because everything could be infinitely far if you just don't have any scale all right so let's just copy paste those kind of stuff into my shaders so first of all i would like to see it in the fragment shader so i'm going to have the camera near camera farm and i have this important packing include which actually includes the useful functions and i'm going to copy with depth from the example and then i could just read depth with my uvs and then just output it okay so instead of outputting uvs i'm going to just output that and also remove the material from my from my face i don't think i need it i probably could get away with the usual should be green at the moment yeah okay so we have some shader error probably and that's why we don't see anything else why there's also a very important thing that i need to fix in my template at the moment the undeclared identifier and red dips read depth no matching reloaded function found oh yeah that's probably because i i don't have a texture and it throws an error so it's going to be depth info what else what else still t depth do you have t i have to use it okay so now it works just shows black screen i also have 30 fps for some reason and i know exactly the reason for that because i banged my head like for an hour this is just because of srgb encoding which i don't really need for this example yeah feel free to explain why srgb encoding have this effect and by the way just if you're wondering why i had 30 fps that's because of this because i've been wondering why i had this because this is this get program info log and this is some just some login probably and it just takes up like two frames to get all those things no idea why my knowledge is limited guys but if you know the reason why this is happening i'd be very interested to know all right so we just don't need this rgb encoding and we'll be all right okay so now we have face behind this now i have to set the camera near and camera for uniforms let's use them i can actually copy paste right and i could just use my current camera uniforms like not uniform but the properties yeah so now we have the camera near camera 4 they are right here they're being used in this formula where we convert the texture to actual depth it should start showing something pretty soon this shows white texture which is something new i believe we're not yet there do we have any kind of errors maybe warnings so that's texture format unknown what did i just made a mistake somewhere new 3 depth texture probably some kind of typo this is right this is right what is that unknown kind of thing so i'm rendering to this target setting it right here that for value this target texture so far so good some clearing then i'm rendering to the scene so this one works this one probably too but it's throws an error for me unknown depth texture format what kind of texture it should be def format but it's not i have it here unsigned short by type okay let's just maybe copy paste completely so format ah that's what i did here i messed up two of those things yeah it should just change places so this is the format and this is the type and it's written over there that it's actually typing this is format this happens all the time okay so now it's all black which is something new that's all white which is not really new but still okay so we have some source and destination textures of the draw are the same so probably i should use two textures and swipe them this is first thing to do and then we actually have this thing working all right i might have white because i have too big of a range here for the camera near and camera far i have like like 10 000 maybe that's the reason why 1000 so i should probably use something like this and here we go we're starting to see something and that's because the range was too huge like the face was one unit from the camera and then the whole range was between 0 and 1000 so they all looked the same the difference was too tiny too small um like this yeah starting to look interesting so maybe at this point i should just uh scale the face a bit oh it's too big so right now what we see here actually and that there's one bug that i need to face we actually see the depth information so this is a plain thing and we are rendering texture with the depth information into this so now we got the information about the depth in our scene in the texture that's cool i don't know if you get that but that's that's pretty cool and what i can do next maybe i could scale it even more maybe i don't need that at the moment what i could do next because right now if i just zoom out zoom in zoom out i just um break the scene because i'm using the same camera to to get the depth information the same camera to view so to distinguish between those let's just do the special depth camera just used to render this kind of stuff and here i'm gonna use uh uniforms uniforms where are you my uniforms by the way it's square uniform this camera one so this camera would not be moving when i change my orbit controls and then uh first of all when i'm rendering depth i'm going to use this camera and when i'm rendering the whole scene i'm going to use the other camera so here now it's kind of breaking because probably i have different positions for the cameras different starting positions so camera position set to this let's set it to this so right now it doesn't change anything and now i could change the back again like uh 100 so i could rotate the scene in any way and zoom zoom it in zoom it out so i have this face now and let's see what we can do here kinda need more depth information out of it right now it's just plain white and i need to do something like so right now you can see that we are just measuring the depth between 2 and 2.5 and we see only the sliced portion of the head i wonder why why it's so white for me at the moment what could be the reason for that i should see it's actually like a gray parts the ones that closer and the one that's further away i wonder why i see it only in white at the moment do you have any ideas kind of complicated stuff easy to make an error here so and i'm making those so we have depth infer what's this i should actually see some gray areas not just the plain white thing i wonder why it's happening okay so in fragment shader i have this inform just reading the depth info oh no idea guys we are made an error which i should see the gray areas like the one that further should be darker and the one that's closer should be brighter but i made a stupid error somewhere all broken now like in the example i could actually see some areas uh gray the ones that are further away and some areas are just brighter if i zoom in right that doesn't change in my demo so now what did i miss okay let's see we called again signed short by type okay [Music] moment of silence okay i'm gonna just randomly try to fix this maybe it's gonna work out so we have the camera it's all right we have the camera far and close things they're working good they're just returning me black and white thin for some reason probably some typo okay okay even more on z do you think it's just scale thing maybe just a number issue okay let's try to do that maybe i could just do it like right here number well the background is actually working kind of then all the other things something is wrong here i'm missing something why it doesn't change uh i believe i'm using the wrong material or something this material for this plane the render target i'm using just doesn't change anything at the moment i'm running on camera far what is happening okay camera one does it work this way it's working somehow this way i don't know i don't know what's happening why did it work this way but didn't work like the other way so we can see that the model starts at 0.5 but it's still all black and white it should be gray or something and the geometry is pretty pretty scaled i don't know okay so if you don't fix it in two minutes i'm gonna just look up the the code that i did before the stream and just show you how it's made this is probably some stupid error but yeah but i'm using camera one here it's camera one here okay let's for a moment just get back to the first camera even the first camera renders just the white black and white thing i wonder what went wrong here all right we are fixing the problem that we can't really fix did i mess up the i'm gonna check everything again try it map stencil buffer did i miss something some setting render target and actually works it kind of works rendering this scene then setting the texture and then rendering another scene it's kind of everything is in place except the except the theme right okay i'm gonna i'm gonna look up the code that i did right uh maybe just up some variables or something like that so this is the code i used for the depth texture okay nothing has changed so far i'll just copy the constructor which is just my actual template i didn't change anything but let's just try to fix it with a brutal force and then maybe after this streaming comments i'm going to point out where was the mistake in the first place it all disappeared camera one just kind of same thing that we just did and that we loaded the model maybe i should just render after i detailed everything no okay okay let's try to fix it with the brutal force brute force let's fix objects so far so good we actually get the depth like the actual information but why not depth why not depth maybe it's just too huge to render no it's just just either black or white all right maybe i messed up some variables okay let's uh change in change in the stream fix it with the brute force yeah constructor that's what happens when you try to code life yeah that's exactly what's going to happen fix mistakes mesh plane buffer geometry i know you're enjoying it but i'm feeling so stressed at the moment this is going to be mesh and this geometry and then this material okay this is how it should i should actually render you see it's the gray areas when it's further and closer so it's actually almost the same i'm gonna change it to camera i wonder what was the mistake i'm curious to fix it probably some weird attribute or typo something like that so i'm gonna switch it to camera one let's see now that how great it is and then camera one camera one is actually these two and five and let's switch it to two and four see how it looks like when when now we have the real depth information for some reason in in well yeah you know we made that mistake okay and even like that it kind of looks beautiful now because it's you see it goes to the black in the end all right so again this is this is what we should have get in one minute if i didn't make that error so right now we're just getting the depth between three and two and render into the texture well the funny thing is that they actually use the same texture inside the vertex shader use the vortex shader same packing same uniforms then same function it comes from the packing i mean those functions come from the packing okay and then and then i could just i could get the depth i could actually get the depth inside the vertex shader and i already have uvs so this depth kind of i'm curious about and then i'm gonna do like three i'm gonna render this position here all right and then i'm gonna do post c plus depth um that thing for yeah we should declare the texture as well let's copy paste the resolution as well so right now we have this well probably the other way now we have this it looks kind of weird but now we're converting the depth texture into actual geometry displacement map right oh you could multiply this by probably something like 0.6 and we'll get the normal phase and it's also colored with the depth and we could use this coloring to get like this coloring like when the depth is bigger you could just use it okay and now the interesting part i'm gonna switch it to the lines so instead of just using the and just using the face distortion and by the way i could as well do this i could just this model this model position see it's minus one by by default i think minus one plus sign this time divided by hundreds so it should oscillate just a little position of undefined okay that actually wrote the same line yes now we have this weird thing i believe this is too much so the z was minus 1.5 okay so you could see the face moving just a little bit so it's actually dynamical it's loading it's it's rendering the depth on each frame now maybe why see what's gonna happen now i could see this coolish animation where the face tries to break through the plane but it never breaks through like this so it looks a bit cooler so we're getting this depth information and converted into the actual plane all right this is really cool by the way i think it's really cool and if you've seen my face experiments this is kind of what you could do with those the other way i've turned it a little okay enough with the polishing so now i could use it with the lines how would you use it with the lines i already had that code which i commented and i also have it here so i'm just going to uncomment it probably let's make it an explanational stream so i'm just making a loop here which creates those geometries for me i'm going to remove this plane because this is just a test plane now we have those lines which don't yet work for me just the lines each of them has the texture by the way so you could see it's really stretched there because i'm using the same material at the moment what i need to change though is the uvs so i need to calculate new kind of uvs to use for those things so the x-coordinate will be the same thing like the vuvx should stay the same that means that like 2 uv equals mac2 should make it here it's going to be v u v x the x is the same because the lines are stretched on the x coordinate and the y the y should be something different and calculate why it's actually easier to set attributes on those so i'm going to just add the attribute attribute plot y which is now zero at the moment for everyone else then i'm gonna calculate the length of the buffer geometry like how many vertices are there and then i'm going to set i'm going to create an array with just the [Music] just the value like how much we traveled so the i divided by 100 would change from 0 to 100 so i mean i am going to change from 0 to 100 so this is going to be between 0 and 1 and this is exactly what i need i need something to change between 0 and 1 on the y axis then i'm going to set it as an attribute it should be should be good let's just preview those so i'm going to create the vov1 probably uv1 yeah it's starting to look good now because we're using the depth information now again for each of those lines it's kind of breaking on some places probably because the utilization could be bigger than that maybe you could choose like 300 here it's gonna be more smooth and then the texture will also have some boundaries for the texture size so probably we could change the texture size now let's try to get the colors at least the colors you're offering everyone the goodies thanks thank you i just need to take a screenshot where's my screenshot not even that and then maybe just switch to the black color and starting to look cooler than that and then we could use this depth information i mean we could just use another variable float v depth and then we def will just just the depth and then i have it here vari varying slot v depth and then instead of sampling the texture again here which i could actually use with the vue one but i'm gonna just do the mix [Music] with a v-depth it's kind of shows me something but not quite maybe the other way now it's all black and only the face is white and when the face goes behind the scene we just couldn't see it and then we could change the mesh basic material to black because we're not really rendering this per se we're just getting the depth from it and now we get this and now to get the final look maybe i could choose to this color that i actually copied actually copied the color and forgot about that so instead of mixing with the white i could mix with this color and i get in there get in there so maybe i'll i'll just make the movements faster so i could see it it gets it it got this kind of dark dark look so i should i should also play it here with the range because i believe it's not getting white i mean the closer when it gets closer to the camera it's not really still like the zero and to get it to zero this is this camera one two maybe point 0.2 it will be just a little bit brighter on the edge also maybe i should just use the smooth step function here so maybe instead of doing this i'm gonna just do this so it's i should do it before that and then mix the colors it's kind of the inverted colors so you could see it for a moment this way you could actually render anything to anything like you can distort any objects with those textures and you almost get this noisy look just because of the movement of the model of course that the model also opens the mouth and moves but yeah it's a bit different story maybe it shouldn't be it shouldn't be black actually it should be just brighter kind of this color not really black yeah i think we get in there maybe it should be between 0 and 1 i mean 2.21 yeah it's somewhere i mean i'm just trying to guess the range of the values that i get there the line here was pretty cool as well with the smooth tap there's a bit more power when it moves in and then maybe just the last touch that we're going to add so what else i'm going to add some noise show you how it will look with the noise it's pretty cool so we have this movement see let's set some noise i'm gonna get it here i have this noise which accepts vector3 so right here position y i'm gonna add something it should be something real small multiply it with this nice and then like three and then what's going to be there [Music] it's probably the ue1 and then the time we still have the time yeah i have the material uniform time here well yeah let's see yeah probably the time is too fast and then this one should be scaled a lot up as nikki as well so now we again this which already looks cool so well we have a noisy face coming down from the lane well they have it zoomed in like this it looks pretty nice for me it looks kind of dark because i have a really light room at the moment i hope you get the idea and then you could also um change like the noise values so they are more okay let's let's just do this also settings this is progress value then i'm going to set it i have to set the uniform somewhere and now i could use this thing here at the uniform uniform hello to london could multiply this z amplification with the progress so i could grow it what if i do it like it could be just real weird i'm going to change it to white for a moment just to see it better yeah so this is how it's made except that they render the model with the gltf animation and then they also add some noise and then they also add some post processing but the main concept is just rendering depth to texture and then using this texture as a displacement map on some lines and then adding some noise and it's it's actually pretty cool also show you what i did with this i've been trying to i've been trying to make my own animation which i posted on twitter it's in doodles i also added some shadows there so it's pretty cool what you could achieve with that exactly the same concept what's the notes 69 all right this is what i did also did some shadows here and because it's a texture you could actually repeat it by the way it's funny you could make lots of those faces on one plane this is just some weird experiment which didn't go anywhere in the end but it's still kind of fun that you could do because this texture could just replicate it repeat it and then use it again at the depth map and it's funny but the concept is still the same you just use it this one the gif the only difference with the demo that i did today is that here i used the physical material not the not the shader material i used the actual physical material for this one but it's pretty cool yeah so let's get we can do that down it's actually the same it's just they have it more much more detailed they have more lines here and they render it real big so they get all the depth information out of it possible the most effective way let you see this is the same noise the same thing and this is this beautiful gradient as well gotta decompile that later that would make it beautiful so lots of lots of like cool tech on this website combined together and it just makes it a real cool website i'm glad that they're using it for a good cause like for removing racism and yeah yeah it's like an apple emoji i showed in the beginning of the stream that you could actually see their model you could actually change yourself parts of the model like this yeah this is exactly my face when i try to fix the back today and this is my face when i didn't fix it yeah pretty much all right so yeah i made a i made some weird mistake which i wasn't able to fix today sorry guys i hope it still was informational to you on how this website is made and i hope you learned a new concept that you could use in your demos i think it's pretty cool and let me know which websites would you like to see decompiled what would you like to see and i'd be grateful that you slap like if you like the video but only if you like that i hope you had some fun today all righty there's a lot of minor details that you could miss so sometimes people actually ask me like i can't i can't do that i spent day fixing that well here we go i did this animation like five times already i think in some previous projects but then again i wasn't able to fix it here and now and i would probably spend an hour or two just finding that line of code where i did that mistake so thank you guys have a good day i'm gonna stay in chat for a little while i love you all tell something good to the people that you love do something good today live a happy life and have a fantastic day have a fantastic day i'm gonna switch to the big screen to show you my love i love you see you next sunday

Channel: Yuri Artiukh

Views: 7,471

Rating: 4.963964 out of 5

Keywords: webgl, triangle, tutorial, canvas, animation, ascii, three.js

Id: UaAX2_EToqk

Length: 68min 25sec (4105 seconds)

Published: Sun Dec 13 2020