Intro to Graphics 13 - Textures

well welcome to another lecture for introduction to computer graphics uh for today's lecture we're going to be talking about textures so what are textures let me see let me try to give you a definition of what textures are textures are the ways with which we define information for objects i think this is a comprehensive definition let me ex let me expand on that just a little bit there are the ways by which we define information on objects that are not necessarily connected to how the objects are defined not a very useful definition of what textures are because yeah yeah i'm trying to be comprehensive and textures are used in all sorts of places for all sorts of different types of objects you know i didn't say surfaces right they could be surfaces there could be other types of objects and we have different ways of specifying textures you may think about some some very specific things in your minds but there are various ways of specifying textures and also they are used for numerous purposes there is not just one one thing we want to do with texture there are many things that we want to do with textures so this comprehensive definition is not very helpful so what i'm going to try to do today is that i'm going to start with a very narrow definition of what the texture is we're going to keep it narrow and then we're going to expand on this a little bit actually we're going to continue to expand on the definition of textures as we start talking about other things in computer graphics so by the end of today's lecture we're not going to have a comprehensive definition of texture and by the end of next lecture also we're not going to have a very comprehensive definition of texture we're going to expand on this a little bit so we're starting narrow but i want you to understand as we start narrow that textures have wide range of uses and when you say texture people may think about something very limited but immediately but actually it means a lot of different things in computer graphics if you go beyond computer graphics textures can also mean a lot of things uh that you know don't quite fit in the definition of textures that we use in computer graphics but never mind about that we're limited to computer graphics and the narrow definition that i'm going to talk about is that a texture is going to be an image an image that we're going to put on a surface or a model right that's all we're going to do to just a very narrow definition of the texture we take take an image we put it on an object we put it on a surface and so if i take this this texture and put it on this lovely teapot to the lovely utah teapot what i'm going to get is this guy right i mean why would you put bricks on a teapot i i don't know actually there's a very good reason for that when i was teaching raytracing for the first time i would have used utah teapot then i had a brick texture so i just put it on it because i didn't really have much time to think about it it's been many years i'm still using this texture on it so that's the reason why but never mind about that this was an image i put it on my surface right this is texture right now the immediate question here is all right the immediate question is probably why break on a teapot but i explained that but the next question is how do i do this right how do i take this image and put it on a teapot i need to figure out a way of taking that image and put it on this model right all right so let's talk about a few ways of doing this um yeah i can just uh linearly map it like on a like a planar planar mapping well i guess i should do this way right i took that take that image and just flatten it on on the surface well it's already flat but i just push it on the surface of the model so it kind of it would look like this i mean i could define a function to do this fairly easily right it's not not too hard but i'm not getting there just yet uh so that that's one one thing i could do i could do something a bit more just a bit more complicated like it you know try to map this from every every direction uh i could just put this surface uh put this texture on the surface from from every direction so that could be like a cubicle mapping is what we call i i could try to wrap it around this in like a cylindrical way and maybe cap it on either side that's another thing i could do or i could do something like this like a spherical mapping does this look familiar it should look familiar this is the a representation of the planet that we're living in so but um when you look at maps most maps use this sort of spherical mapping so the funny thing that happens here is that well actually they don't use exactly this but don't be on a version of this um so the funny thing that happens here is that if you look at the top edge of this map the whole top edge turns out to be just one point on the surface so i collapsed that entire edge up up here the entire top edge onto one point at the north pole and the bottom edge becomes another one point at the south pole down here um so this is not so great because hey i'm gonna have you know a lot of pixels to define the this this top portion of my texture here which is going to map onto a very small area on that surface but if you look at the equator here like the center of my texture it's going to be you know mapped around the equator so it's going to be mapped around a large area so the texture is going to get distorted quite a bit right so yeah when you look at the world maps that what you see on the map is quite distorted it's not a very good representative of what actually is on the globe right and the same thing would happen with this kind of a mapping um but for a lot of purposes this could be a preferred way of taking a a texture and putting it on it's like a spherical like object um here's another one there's another one this is called i believe called shrink wrap so you you take this texture and you just uh you know wrap it around such that you know these um the entire perimeter of the texture all of these all of the edges and the corners they all come down to this the single one point over here they all collapse onto one point so you can you can wrap it that way all right so these are different ways of just taking in a a textured image and putting it on a surface but these seem like very simple functions right or relative is even even this one is not a very complicated function um if i and and i mean this works fine if i'm trying to map something on a sphere it's it's okay but what if i i don't know i have a dinosaur what do i do then if things get a lot more complicated right and things are a lot more complicated so um we're going to have to figure out a way to represent very complicated functions of taking this image and putting on the surface of our model and we're going to do that using what we call texture mapping sometimes called uv mapping so let's talk about texture mapping let me start by saying that texture mapping is horrible it's uh it has so many problems i can go on and on and talk about this for for a very long time but this is the main method we use for defining textures on surfaces so it's it's important that you you understand this texture method all right so here's the thing the problem with texture mapping is to find the define a function that will take my texture and map it on an object so it's a mapping from between the object space and texture space um now oftentimes this is how people think about texture mapping like you take a texture and you map it on the surface that's not actually the correct way of thinking about it the correct way of thinking about it is going to be the exact opposite of this so what we're going to map is that we're going to take our object and map our object onto the texture space so that's what texture mapping does it goes in the opposite direction and the way we're going to do that is well since we're going we're using polygonal models for representing our objects which are eventually converted to triangular meshes right so let's think about triangles and let's talk about this in the context of triangles so my object is basically a collection of triangles so to be able to define a general complicated mapping is that i just need to know where the triangle lands on the texture space right so i just need to know the corresponding position of that triangle in texture space and that's going to define my mapping function so this is not a mathematical way of defining this mapping and so we talked about spherical mapping or cube cubicle mapping or cylindrical mapping this is not any of those actually i have i don't necessarily know exactly how this mapping was defined all i know is that my triangle here this one triangle maps onto this particular point over here so this sort of abstracts the function that i use for creating this mapping from how this mapping is stored for this for defining this mapping all i got to do is for each one of my vertices let's say for this this red vertex over here i need to know where it lands on the texture space right its corresponding position on texture space same goes for this this green vertex i need to know where it is over here and for blue vertex vertex i need to know where it is in texture space and if i know all that then i am pretty much done and of course this is going to define a mapping from our object space to our texture space my texture space is going to be defined of course by it's it's 2d right so i'm going to have an origin and i need to pick two directions so origin for origin i typically we pick the bottom left corner it could be top left that could make sense in some ways it does make sense but we typically pick the bottom left corner as our origin and then i'm going to name two directions one of them is going to be u the other one is going to be v and that's the reason why we sometimes call texture mapping uv mapping basically taking any point in object space and putting it on on this uv space i'll say more about this uv space this um this image looks uh like a good mental model of what's going on but it's uh there's something minor that's not not really correct so we'll get to that later much later for the time being this is good enough all right so for any position in my object space that has x y z coordinates if i'm working in 3d obviously i'm going to have a corresponding uv coordinates in texture space right so that's the whole idea behind texture mapping and again what's important here is that i'm not defining a complicated function i'm not defining as spherical mapping or string wrap mapping or whatever cylindrical mapping i'm not defining anything okay um i just i'm just defining i'm just taking every triangle and i'm directly mapping it in texture space um okay i mean this sounds fairly nice if this mapping is given to you right it's very nice if i know the uv coordinate over here of each one of my vertices that sounds like a very nice solution i but here's the here's the thing and the reason why i said that i don't like texture mapping um somebody has to define this right somebody's gonna have to prepare this model and sort of define where each one of these vertices would go in texture space now that's where the horror of texture method is because if i'm going to take every triangle and map it on by texture space and just manually move my vertices around in texture space that's going to be quite labor intensive to say the least right so we have some automatic methods for preparing this mapping but they're mostly like semi-automatic methods they don't fully work but they work to a certain extent and then people have to go and modify them there's a lot of manual labor involved in this this is a messy messy process and in the end you get all sorts of problems when you're using texture mapping for rendering anyway so it's a it's it's um it's a very imperfect process but this is typically what we use in computer graphics for mapping textures on surfaces that's how we define textures on surfaces typically all right uh so yeah i mean get used to it and get used to dealing with the issues that comes along with it but luckily hopefully hopefully you will not be the guys who will be defining these this mapping right that you're going to just download some some model and it will come with this this mapping and you won't even need to know how this mapping was defined because every vertex will have a corresponding texture coordinate and you'll be done uh i see a question do you think there will ever be a better process there already is a better process i'll i'll get to that i'll get to that at the end of this lecture all right uh so but let's talk about how things fit in our gpu rendering pipeline so this was our gpu pipeline remember we're feeding in triangles on one side they go through our vertex shader actually each vertex goes through our vertex shader and then we form our triangle primitives here and send them to a rasterizer rasterizer forms our fragments and then we call the fragment shader to define the color of each one of our fragments and finally we get this this lovely image all good all right so where does the texture fit in well we're going to need this texture when we're defining the color of the fragment right so a natural place to put this texture would be well it can't come along with triangles so i'm going to have to feed it in separately that's what i'm going to do i'm going to i'm going to feed this my texture separately and send it to my fragment shader and say this is this is my texture right so what's going to happen is that i as i'm passing my vertex positions through my vertex shader as one of my vertex attributes i am going to be passing textured coordinates as well so my one of my vertex attributes is going to have to be my texture coordinates so if i have those texture coordinates i can just pass it along to my rasterizer and then my rasterizer can pass it along to my fragment shooter and then i i can use that to figure out what the color is at the corresponding position for my texture so here's how that works my rasterizer is going to create these lovely fragments right and it's going to have to compute the corresponding texture coordinates for these fragments now these fragments over here they look like little squares like what's the texture coordinate for add square i can't i can't do that it makes no sense so we're going to look at the centers of these fragments all right so my gpu is going to compute the corresponding texture coordinates for the centers of each one of these fragments centers of each one of these pixels if you will uh so let's take a look at one of them to this pic one in the middle over here so how do i compute the corresponding texture coordinate for for this point on this triangle well it's fairly easy we did stuff like this right remember when we were talking about triangular meshes so if i know for example let's talk about the position the 3d position at this point if i know the positions of these three vertices let's call them p0 p1 p2 and let's say that the position in the middle is p then i can just do a barycentric interpolation remember the percentage coordinates i have a very centric interpolation i found this position i can do the same thing while we talked about doing this for color i can do the same thing for the texture coordinates so if my textured coordinates are like 2d u 0 vectors 1 u 2 and i want to find u over here at this point i just need to do this uh barycentric interpolation using my percentage coordinates there it is right so we did this we talked about how to do this we talked about how to compute the percentage coordinates when we're rendering using the gpu we won't have to worry about it too much because the gpu is going to do this automatically for us either the gpu hardware is going to do this or the the software that runs on the gpu will do that automatically for us we won't have to worry about this at the end of the day in our fragment shader we will be getting the texture coordinate for this position automatically if we just define the texture coordinate for our three vertices okay simple enough and if i if i know the texture coordinates for this little dots in this texture's face then all i got to do is to just look up what color is color that what color that texture is exactly at that point simple enough right at that point i'm pretty much done right i know that point so all i need to do is to look up that value in my texture definition and i'm i'm done simple you would think no no no no no our troubles begin there we're just beginning if it was easier up until this point now things are going to get a little nasty so sorry uh get ready for it so now we're talking about texture sampling or texture filtering that is the the process of computing some color from a texture all right so i know what point i need the the the color value or texture value more generally um but let's talk about how to get that value out of our texture right so let's say that this is our texture right just that texture is made up obviously low resolution texture um it's made out of a bunch of pixels uh but remember remember remember we talked about this pixels are not little squares right pixels are not little squares they're more like values at a certain point on this space and it's important to think about them especially in the context of texture mapping like this these are just points values at certain points on the space actually these are not even pixels these are more appropriately called texels these are texture elements instead of picture elements yeah big difference yeah i mean for 2d textures textiles are sort of pixels but we call them texels all right so here's my uh set of texels and i let's say i want the value at a point somewhere on in this space like let's say like at this point over here i want to know the value of this texture at that point how am i going to do that well the simplest thing i can do is to look at that point and find the find the nearest point find the nearest texture sample and just read its value so this is the color that's closest to this point color position that's closest to this point i'm just going to take that value and and use that simple enough right that's really the simplest thing i can do well wait when i do this it kind of looks like i split this whole space into this grid and it kind of looks like yeah i have like little little squares for pixels now for pixels right so we sort of went to this definition of textiles being little squares but as i said this is no that's not what we're doing that's not what we're doing we're just saying that you know we're in this space and we're picking the closest value okay we're picking the closest spelling that's defined at this point in this case now this is the simplest thing i can do not necessarily the best thing i can do i can do a little better instead of just looking at this one value i can look at the closest values and do sort of like an interpolation of them the closest values in this case would be these four values right and i can just linearly interpolate them to find the value at this point and i don't call this linear interpolation because it's leaning interpolation in two dimensions right so i'm going to call this bi-linear interpolation so it's basically a combination of two linear interpolation or this sort of sampling the texture will will be called bi-linear filtering so bilinear filtering is not a very complicated thing i'm actually going to show you the equation at all uh not as simple as just picking the nearest color value uh it's going to involve some linear interpolation by by definition right so let's take a look at these four four textile values only so just i isolated these four textual values and i'm going to blow it up just a little bit just to give us some space all right so i am interested in this this region between these four textile positions remember i i was putting them at the centers of my texels but now i'm connecting them to form this this region in between four texture samples and if my point is in between these four texture sample positions i'm going to be using these four texels to compute the the value exactly at this point all right so let's let's do that uh i'm gonna start by naming my color values let's call them c0 c1 c2 c3 color values defined at these texture samples or textures and the color that i would like to compute is let's say it's going to be c by linear calculation is fairly simple i'm gonna have to do a number of linear interpolations i'm gonna have to do linear interpolations in two dimensions now let's take a look at this dimension first you can think of this as like the x dimension or or uh u dimension if you will all right and u dimension let's say that the distance between this vertical line and this point c is s now if this is one unit in number of pixels it's just one unit right and and if this distance is s units then the remaining distance is going to be one minus s so i can use this to linearly interpolate my colors c0 and c1 and if i were to interpolate c0 and c1 and find the corresponding value over here right underneath this the c just vertically underneath uh that's going to be i'm going to call this c01 and i can do a linear interpolation to find the value there using using this simple linear interpolation formula c01 is equal to 1 minus s times c0 plus s times c1 simple enough right we've done linear interpolation we can do that fairly easy now i'm going to do the very same thing for the the the two colors above here as well and when i do that i'm going to call it c23 because it's interpolating c2 and c3 same function it's just interpolating c2 and c3 simple enough all right so we're done with our interpolation in in this u dimension now we're going to look at the v dimension and the nice thing here is that now i have three color samples three positions here like i know the color over here i know the color over there i want to know the color in between here so i just need to look in this dimension right fairly easy let's say that you know this distance is t and so the distance above it is going to be 1 minus t i can just do a linear interpolation of these two values and i get my answer right so bi-linear filtering doing filtering in two dimensions one of them i first do linear interpolation two linear interpolations in one dimension and follow it by another linear interpolation in another dimension but you don't have to um and the the thing is this is sort of interchangeable i could first do linear interpolation in this dimension and then do linear interpolation in the other dimension i would get exactly the same thing in fact in a lot of cases this is uh implemented as just one bi-linear interpolation that is i'm just going to take these three equations and combine them together are you ready you can just combine these together by expanding these c01 and c23 and if i do that i get this function right it's just uh i just expanded it's exactly the same thing so yeah this is the binary interpolation and i can easily you know interpolate these four color values like this now uh you know here's something geometrically interesting uh at least i find it interesting uh so if you if you see if you take a look at here you see that there are four color values and four weights associated with four color values so if you look at the weight for c zero that's going to be one minus p times one minus s one minus t times one minus s is the area of this region over here right so the area of of this region is the weight of this color valley same goes for all the others like the area of let's say this region is going to correspond to this color value right and area of this region is going to be this uh the weight for this color value i kind of like it geometrically interesting right simple enough right not not too difficult just just a bunch of linear interpolations basically or you expand it and write it in one by three interpolation does it improve things though what would it look like if i were to compare this to nearest filtering binary filtering versus near skillfully i'm going to show this on my face are you ready and there it is uh so this is mirrors filtering and this is by linear filtering all right so over here i can distinctly see each sort of each one of my texels as little squares over here they're kind of more hidden i'm sort of interpolating between them and things are looking more smooth and are they looking smooth definitely more smooth um can you can you see the pixels over here okay tactiles can you can you see them i mean over here you can obviously see them like they appear like little squares over here uh yeah these two images are exactly the same resolution i'm just doing bi-linear interpolation here to find the color values in between can you sort of see the resolution of this this texture here kind of maybe can you see the size of a texel like over here you can see the size of a text though right there's a little little squares over here they're kind of hidden it's not as obvious but if you pay close attention you can actually see them like uh for example how do i show this to you i'm talking about like over over this region near the beer like there's some sort of uh bright areas you can kind of see like they're like some some vertical and horizontal shapes that form that you can recognize right so it's not perfect definitely better than than this but not quite perfect and this is not necessarily the best we can do with this we can actually do something a lot better a lot of times when people talk about texture filtering they talk about nearest filtering bilinear filtering but there is there is stuff that we can do here that we can actually do things that are better than binary filtering so for nearest filtering all i did was i looked at this one value that was closest to it for bi-linear filtering i look at the the four closest values so if i want to if i want to get better i can just need more information and the reason why so these textiles are still somewhat visible here they're not completely hidden from me is is because uh as this point moves out of this this range of four pixels and moves over here i'm going to be using a different set of four for color values for four texels and my texture filtering is not continuous across this direction as i move go from this point towards this point there is some continuity that it's not completely discontinuous like the nearest filtering but the change of color shifts so color is sort of going towards one color and then it starts moving onto abruptly starts moving on to another color and we can actually detect this our eyes are good enough and and then the processing that goes on in our heads it's good enough to to detect this this sudden change in the way that the color gradient is moving so it's moving in one one way the color is going to the the color is moving towards the color and then it abruptly shifts and starts moving towards another color and we can see this we can recognize this as like an like an edge like a vertical and horizontal edge and if we want to do something better than this we kind of need to make sure that the color changes more smoothly right uh so we need at least c1 or c2 continuity so this is going to give us no continuity this is just going to give us c0 continuity if you want to be more specific so we need to have at least c1 uh perhaps c2 so i can do something better than bi linear that would be bicubic now bicubic is going to look at not just the four closest values but it's going to look at all text texture values in a 4x4 neighborhood and it's going to do a weighted average of these such that such that the the texture the color gradient here as i move from one position to another is also uh continuous so i'm smoothly changing colors from one position to another now if you look at this bicubic example here you shouldn't be able to see any easily detectable horizontal or vertical lines right so they're sort of i mean it's still blurry yeah i'm still using the the same amount of information is still a very low resolution image all right i'm i'm not doing anything i'm not adding more information here i'm still using the same information i'm just using it a little better so i'm hiding the where the texels are just a little bit better by using bicubic interpolation so can we do better than bicubic oh yeah i could keep increasing the the function uh smoothness but it won't really help but this is smooth enough all right extra level smoothness is not going to help so what can help is doing something like like this right so from this image i could if i could construct something that has more data in it and this sort of filtering is called science fiction right so this only happens in movies when you say can you can you clear this area let's just zoom in and all of a sudden there's more information in there yeah so this doesn't exist yeah i'm sorry to tell you i mean this sort of requires adding more information now there are ways to add more information yeah there are ways but i mean i simply do not have this information in my text here i'm not i'm not going to get this i kind of need to figure out a way to add information and there are sort of maybe ways to do that but that's going to be yeah i kind of need to hallucinate extra data and if i have some extra data and somehow my extra data fits here there are things i can do but no i won't be able to reconstruct this image perfectly there's absolutely no way all right i can only hallucinate this image by sort of imagining oh i guess it looks something like this maybe and that's about it and yeah there are some some machine learning methods that could be applied here that could hallucinate an image but would it actually reconstruct the original image no no yeah no no way it will do something so there's no way to do this the only way to do this is the only way to do this is to have a high resolution image as my texture to begin with all right so uh if i use let's say by linear filtering and put this this texture on my teapot and somehow i don't like it and somehow it looks too blurry to me and i want to do something better i just need to use the high resolution texture right i just need to use the texture that has more taxes more information in it right so this is what we do right you never want to see this on a high quality model right so when you have a high quality rendering of an of an object you actually want to see all the you won't be able you shouldn't be seeing the texels in any way texels should be small textual textiles should be very very small so that you can't really tell they should be they should be on the order it's in size wise in in screen space they should be about a pixel or even smaller all right so ideally that's what they should be okay so this is what i want i want a high resolution texture on my teapot good what's the problem here's the problem i picked this resolution now he's very happy with it let's say now that this is a great texture but whatever let's say i'm happy with it what happens if i move this teapot uh far far away what happened now well it's it's gone too far so now my texture is a little too high resolution right i mean i really don't need that high resolution texture here i could get over get away with a lower resolution texture in this case but i did use a high resolution texture so that if i if this object ever comes close to the camera i'm not going to have problems but when the object is moving away from the camera then i just have all right i have unnecessary amount of data in my texture okay i mean why would i care it's okay right i have enough memory on my on my computer or my gpu whatever i'm using for rendering it's it's good enough it's it's fine or is it fine sure it's fine see like it's looking okay it's looking okay because this is not bi-linear filtering if i do bilinear filtering here are you ready if i do actual binary filtering here this is what it will look like lovely isn't it what is happening can you can you see this flickering i hope you can see this flickering with the video compression video compressions are pretty good at filtering out everything but this is this is sort of hard to get rid of um so you know like my texture is sort of dancing in the background right i mean over here closer to the camera things are okay and things are okay because my textiles the size of my textiles are on the order of a pixel but as you look away here as my textiles become very very small as compared to the size of a pixel and i start getting this this funny thing so what's going on here um let's talk about it so let's say this is my texture and i'm putting this texture on a plane all right and i'm going to render an image of this using my lovely camera here so i've just positioned my camera over here and this is the image this is the image that i'm rendering good okay now just to illustrate this let's render a very low resolution image just to illustrate what's going on otherwise the pixels are going to be too small and i won't be able to show you anything right so low resolution image they're giant pixels here so what's going on here when i'm rendering this i am going to compute the the color values at the centers of these pixels right because my fragment shader is going to generate the textured coordinates at the centers of these pixels so basically i'm going to be looking at the texture values that correspond to these points and that's what i'm going to be to be using so let's let's pick one of these pixels here i just picked this one right so this one pixel let's concentrate on this if you look at the center of this this pixel it corresponds to a particular position on the texture and the color value at that point happens to be this and that would be the color of my pixel right i'm doing bi-linear filtering i'm even doing bicubic filtering whatever i just did a very nice filtering i picked this color value at that point very precisely very nicely i got this and and then i painted my entire pixel with this because what is painting a pixel is just picking one color value all right so this is my pixel but if i take this object and just move it slightly all of a sudden that point that point at the center corresponds to a different color now my pixel has a different color so if i were to move my object back and forth i'm going to be getting one color after another color one color after another color right so this is the lovely flickering that we've been seeing and it's it's it's very annoying right we cannot possibly accept that we kind of need to figure out a way to get rid of this now what we should be doing actually it's not picking one or the other we need to pick a color that is representative of the entire texture that corresponds to that pixel so we need to pick a color that's in between in between the two of them not either one of them and such that that color would be a good representative of this portion of my surface that i'm seeing that's what i want that's what i would like to get with my desired filtering whatever that is so this portion of my texture the average color of this portion of this texture whatever that color is that's what i want to get out of my texture filtering now here's the practical problem here guys practical problem my texture here i want a high quality image right so this could be a very high resolution image my texture and this lovely area that corresponds to one pixel that can be arbitrarily large it can be very very large it can cover the entire texel sure and sorry entire texture image this pixel can cover the entire texture image how if my object is moving away from the camera and now my object is one pixel wide and that object will have a color on it that will require me to look at all of my texels and compute an average well if you haven't guessed it already that's very expensive i mean you can do this in finite amount of time but it's going it's going to it's going to require you to read the entire texture and take the average of this and that's a that's a very expensive operation especially if you're trying to do interactive rendering that's going to be way too expensive and even if even if you know you could you could live with that you know even if the pixel area is not that large it's still going to be relatively expensive right even if i look at a 10 by 10 area i'm averaging 100 values instead of four four would be just bi-linear filtering um so that's going to be expensive and i don't want to pay the cost of this because i'm not going to be doing one time you could say oh pixels are small i mean most pixels are not going to cover the entire image but guess what i have a lot of pixels if i do this computation for all of my pixels pixels are small because i have a lot of them so they they add up right my running is going to take forever if i keep doing that so i need to figure out a way to make this fast i need to figure out a way to either compute this color value or at least approximate it approximate this color value but i need to do this fast so the solution is what we call pre-filtering so i'm going to do some filtering operation before i render that will make it easier for me to compute this color efficiently and quickly all right so there are different ways of doing pre-filtering there's just one way but the way that's by far the most popular the most popular method in computer graphics by far for doing this type of pre-filtering is mid-mapping so here's how mapping works uh with knit maps i'm gonna have um my image let's say this is my original image with its original intended resolution this is a high resolution image that i want to use as my texture i'm gonna call this my mid map level zero all right this is my high resolution image and starting with this high resolution image i am going to generate a number of low resolution images right so this is going to be my level zero and from level zero i'm going to generate level one and then from level one i'm gonna generate level two and then level three and then level four and five and six and then finally let's say level seven so i started with my original original image with the desired resolution and from that i generated i added seven midnap levels in this case now if you pay attention here you should see that if you compare the resolution of a mid-map level to the previous resolution every time it's like half the resolution right in either direction and and so if my textual resolution is a power of two if my original resolution level zero is the power of two then every four pixels in a lower level will correspond to one texel and higher level so every four i average that the four i get a higher level texture and same goes for you know all of the levels so that every two by two region of a texel uh is uh converted to a single taxal value for the higher level um so this is the pre-filtering operation um that we do for generating what we call nip map levels right so what this allows me to do is that starting with my high resolution image level 0 i generated lower resolution versions of the same texture now i kept i said if my texture is at power of two technical resolution is a power of two um back in the day this was sort of like a hard constraint like we we required all textures to be to have resolutions or powers of two it's it's not like that in today's uh graphics uh you know there are ways to compute mid-map levels for any resolution image it just becomes a bit more messy but if it's a power of two then we have this nice uh structure right every four texels correspond to one texel in the higher level if it's if it's not exactly power to that the computation is going to be a bit more messy but it is still possible right uh so mid-maps are not limited to powers of two only but it's just the idea is that we get lower and lower resolution versions of our original image so here's how i'm going to use this so this is my pre-filtering operation and i have these pre-filtered images and i'm going to use these pre-filtered images at render time uh and here's how so this is the area remember that i would like to sort of approximate the color um using mid maps what i'm going to do is i'm going to draw a box i'm going to draw a a square that will encapsulate this region that i'm interested in it can be approximate it's not not too important so i i have this box that sort of corresponds to this area so instead of computing the color that corresponds to this this funny region of the pixel shape on on this texture i am going to approximate it as the color of of this region that sort of encapsulates it all right i mean it's definitely going to be much better than just picking one color one color value at the center so it's going to be approximate is what i'm trying to say so all right if this is the case then what i need to do here is just take a mid-map level that corresponds to pixels of this size so remember if my texel size was on the order of a pixel size on on screen space then things were okay right i wasn't getting any problems so if i just pick a low resolution image low resolution texture such that the size of its texels are sort of close to the size of this this projected area of a pixel then my filtering should be okay so i can just use bi-linear filtering and i'll i'll i'll be fine so here's what i do i just pick that level pick that bitmap level and then i do buy linear filtering here good enough uh almost good enough the trouble here is that remember we have midpop levels are generated such that every resolution is like half the resolution of the the previous one and and this is the case because if you do that things become very very efficient because i have my high resolution image my level zero my level one is going to have one fourth of the pixels in it right my level two is going to have one fourth of the pixels and of that right so like my successive mid-map levels will will take less and less and less storage uh as compared to my original image so i don't really spend too much memory it's extra storage to store my midmap levels but because of that the jump between the jump of textile size between midnight levels is not something i should ignore so you know i have this size if i go to one higher resolution image i'm going to have half the size now that one is going to be too small and this one is a little too big if you pay attention right this is a little too big and the other one would be too small what i would like is some level in between them but i don't have that i only have those levels so what do i do so um to to compute the color at this point i'm just going to compute it at this level right and i'm also going to compute this well no wrong direction go back go back all right i'm also going to compute this at this level uh and i don't have to choose one or the other i can just combine the two so first i'm going to do a bilinear filtering with one of the levels and now i'm going to do another bilinear filtering with the other level and then depending on the actual size of of this this area that i picked i'm going to interpolate the color i computed here and the color i computed there i'm just going to do a bi linear sorry linear interpolation based on the the size that i actually wanted all right so this is going to be my best approximation of the area uh the the average color that corresponds to that pixel area and this is what we call tri-linear filtering this is it by linear filtering and you add another linear interpolation in between that becomes tri-linear filtering that's that's what we call it so sometimes you call it map filtering sometimes we call it tri-linear filtering although you can use mid-maps without having to do tri-linear filtering you can just skip this last bit if you can just use one or the other level but you won't get the same quality all right so let's take a look at what this would look like so here's an um checkerboard texture applied on the plane if i just use binary filtering i get things that are not very good yeah like i had flickering in an animation but if even in a single frame you could start seeing some patterns here over here that actually don't exist in in the texture right texture is just this checkerboard but i am getting this this funny thing because of the way well that's what we call aliasing but we'll we'll talk about leasing later um basically i'm getting visuals that don't actually exist in my scene definition in this in this texture if i use mipmaps with tri-linear filtering i'm going to get something a lot better ready uh so much better so much better right so i sort of got rid of all this stuff and i'm just putting the average color back here things are so much better right so much so much better not great but definitely better but not not so great not so great all right let's let's see if we can do this better like here's the thing if you look at like the pixels as the pixels all the way back here they're kind of a little too blurry right they're just gray it just became gray it didn't doesn't look very realistic it's just becoming gray and the reason why this is happening is that you know a if you pick a pixel like this and put it on texture space that pixel would look like this but for computing our mid-maps with trilinear filtering we started by saying hey let's pick a box that sort of encapsulates that that area and when i did that i sort of blurred my image a little too much um because i i just needed this sliver of a texture but i just picked this box area so if i could instead if i could just compute this this range this this shape i would get something a lot better right so and if i did that i would get something that would be a lot less blurry and a lot you know much much better in terms of texture filtering and that would be what we call an isotropic filtering an isotropic filtering is uh a tricky to implement but it's conceptually very very easy all you do is instead of instead of filtering driving instead of doing trial and er filtering one time you we're going to try the linear filtering multiple times multiple times like so so in this case i have three tri-linear filters so with three filters uh i sort of created and and i sort of sampled an area that is closer to the actual area that i'm trying to sample so i get a much better approximation of the color value and if you were to compare these three different options as you can see binary filtering is not great mid-maps are you know they're much better but still they sort of uh blur a little too much uh an isotopic filter is going to get rid of that extra blur um extra blur and isoquick filtering is actually particularly important in cases like this like when you have a plane that is becoming almost surface normal becoming almost perpendicular to your view so it's just something that's going off in the distance now when that happens your pixel pixel area in the texture space becomes so elongated and when it's so elongated mint maps would just over blur trilinear filtering will over blur and you will need an isotropic filtering to minimize that blur uh minimize that over blur right but it's going to be expensive not going to be cheap because i'm doing multiple try the new filterings now instead of doing just one triangular filtering so it's yeah that isotropic filtering is is expensive because of that but it provides much much better results right okay so this was a nightmare i didn't call it nightmare but this was the this was what i was talking about when when i say troubles just began um texture filtering was the difficult part and you know as you see there's this so much involved now the good news here is that this is a very fundamental operation in computer graphics so our gpus come with dedicated hardware to do this filtering for us so for gpu i'm going to say hey gpu this is my texture and just do an isotropic filtering with mid-maps on this and gpu will do that for you so that the filtering part when you're doing real-time rendering is not something that you're going to have to implement by hand at the same time not implementing it is easy but also it's it's gpu has this hardware units that do this for us and because they have specific hardware units to do this they actually they can do this very very efficiently so by some estimates but what the gpu hardware can do for texture filtering is about 10 times faster than a software implementation that runs on the gpu uh so that these hardware support for for texture filtering is actually very very valuable it's a very very crucial component of having high quality fast rendering that we have on our existing gpus it's a very very important component of that all right so this was all about texture filtering now i want to step back and talk about textures in a more slightly more general way now textures can be all sorts of things now they could be used for all sorts of purposes well we'll get to that they could be defined for all sorts of different objects we'll get to that but i wouldn't say there are different ways of storing textures for surfaces even just just for what we've been trying to do of course yeah that could be raster images that's one thing and raster images you know you you expect them to be 2d but i can also just take one rubber raster image that would be a 1d 1d texture or i could just extend it in another dimension it could be 3d higher dimensions whatever some sort of raster image of some dimension i probably shouldn't call it an image when it's higher or lower than 2d but whatever you get the idea but they could also be procedural what do i mean by this i'll talk about this a little bit next time any kind of function any kind of function that you you think about could be could be a texture just write some function it does something that could be a texture it could be some algorithm it could be some mathematical formulation and i can call it a texture i don't have to necessarily store textures right i can just compute the texture value on the fly if i want to that's another way of representing textures but beyond these like just just for representing specific textile values there are alternative structures than just these um raster images now remember that we're trying to put we're trying to put these textures on surfaces and there are structures what we what we typically do is we take an image and put it on on the on the surface but you know as i hinted early on maybe a little more than hinted early on is defining this mapping between the texture space and and object space this defining this mapping was a very messy process now i can spend an entire lecture on this easily actually i could even maybe spend an entire course on this easily i'm not gonna do that i'm just gonna spend about five minutes but let me tell you just a few things just defining this this thing is very labor intensive very very expensive because it requires a lot of manual intervention uh and at the end of the day you get textures that that have all sorts of problems when you're rendering you actually get some visual artifacts which i'm not gonna show i'm just gonna tell you that you're getting all sorts of visual artifacts out of them as well and that's creating all sorts of issues even though this is a very popular way of generating defining textures and storing textures it's not ideal at all and at least one of you guys was asking if there's a better way than this and as there is there are alternative structures i actually gave a course on this uh with market rating silvia my favorite we uh 2017 we call this rethinking texture mapping there are a whole bunch of methods out there that do a a better job in many ways than our regular old texture mapping and from those i'm going to tell you two of them a popular one uh that's called ptex that came out of disney animation studios so ptex is a very s very brilliant and simple idea instead of trying to define a mapping for this entire model why not take each one of my low resolution quads like quads defined on my low resolution image and put a separate texture on it that's a simple mapping right take each face and put a texture on it and that it solves the one of the biggest problems of texture mapping that is defining this mapping so with this sort of structure as soon as i have a quad dominant mesh which is what i typically use for modeling anything right as soon as i have a quad dominant mesh i have my mapping implicitly defined for me i don't need to define that mapping because i'm going to take each quad and i'm just going to plop a texture on it and i'm done right so i don't need any manual defining mapping or anything and not nothing complicated and so this became very very popular in production because people didn't have to worry about this this this snapping um at the same time though i came up with with my method this was and i called it mesh colors uh so that does pretty much the same thing slightly differently that slight difference actually makes an important difference um in practice what mesh colors do is that they define the colors the texture sample positions directly on the polygonal mesh surface so if i have let's say a triangle like this i can define mesh colors with whatever resolution that i want with mesh colors i'm going to have exactly colored color values that exactly at the very season along the edges as well as on the faces it could i could have triangles or quads uh so i i can define colors this way and with this sort of storage this doesn't require anything other than just storing the the textiles okay they're of course the resolution and whatever that takes a little bit of information but it definitely does not require as much information as you would need for storing regular 2d texture mapping it actually stores less information but and it solves all the problems of texture mapping um it actually solves the problems of of ptex as well when it comes to rendering so this also eliminates all the rendering related artifacts which i'm not going to go get into that don't get me started on that that would take me a long time uh so yeah it says solves all these problems and the good news is that actually with current gpus it's possible to implement any version of this this sort of texture filtering uh or this sort of texture representation on existing gpus not exactly i have a relatively uh recent paper on that with uh my cdm aladdin my collaborator there is either uh so we talked about how to extend existing gpu so that we can do easily do mesh color filtering on the gpu so but existing gpus actually support a part of this already not not all of it but but most of it i would say the most important part is already supported uh so yeah there is a better way of doing this and probably will move along in this direction so as we're talking about textures i want to talk about textures and texture mapping because that's still the standard today but i had to mention this to you guys because hey well it would be embarrassing if i didn't teach you this it's my students i'm i'm telling the entire graphics world that this is what we're supposed to be using so if i don't tell you guys this that that would be really embarrassing so i didn't want to do that yeah so but still you still need to know texture mapping and we still need to we will still implement texture mapping as a part of our project but know you should know that something like this also exists all right i'll just leave it at that uh and this is what i plan to talk to talk about today next time in our next lecture i am planning to go over how to use textures on the gpu you know that's going to be some specific webgl instructions that we will be using for storing and accessing textures it's going to be slightly complicated but not really too complicated once you understand how this thing works it's it's it's not it's not that complicated because gpu will do the actual difficult part for us they will do the it will do the text recording interpolation automatically and then it will do a texture filtering for us so all we need to do is to tell gpu hey this is my texture just use it that will be but that will be the topic of our next lecture um and and after that i believe you will have enough to uh enough information to finish the remainder of our upcoming project so i'm gonna end it here and i'm gonna see you all next time

Channel: Cem Yuksel

Views: 1,327

Keywords:

Id: Yjv6hc4Zqjk

Length: 70min 17sec (4217 seconds)

Published: Tue May 11 2021