Intro to Graphics 03 - Raster Images (Part1)

all right well uh thank you all for coming to another introduction to computer graphics lecture um all right so we're gonna talk about some interesting stuff today and this is going to be our third lecture hopefully my video is a little better i'm using a new device now so i have a slightly better setup hopefully things are just a little bit improved for you i'm going to make more improvements as we move along so let's start today's lecture that's going to be um that's going to be about raster images so we're going to talk about browser images we're going to learn what they are and how to perform them and all that stuff all right so pretty basic stuff again we're kind of starting slow uh so an image was an image so you see this as an image right so what is this made up of you think uh probably those of you who are familiar with graphics or thinking oh pixels yay and those of you who are not familiar with graphics are probably saying oh is this stop sign there's stop there's there's an octagon whatever uh those of you who are familiar with graphics are wrong in this case because there are no pixels here this is what we call a vector image it's actually an svg file it's a svg is an excellent format i would really highly recommend you to check it out and use it frequently it's it's wonderful so this is what you see here the text is the svg format this is the contents of this svg file and this file defines this image you see it in the background um so like there's a whole bunch of stuff here but what's actually going on here is these four lines here is what's the the important part um those are defining that they're saying the first line is saying that there's going to be a box in the background in blue colored and the second line is saying there's going to be another rectangle at the bottom with some gradient on it the other one is the shape of the stop sign and then it says stop on it and all that stuff so um you might think hey we we talked about vectors last time this doesn't look like vectors to me why are you calling inductive images well vectors are actually hidden here a little bit right so um if you look at these lines there are a whole bunch of numbers here those numbers are the coordinates of the vectors that are used for defining these shapes so for um brought an octagon shape here for the stop sign uh these the positions of its vertices are listed here as x y coordinates so those are the vector coordinates all that stuff is the thing that's that's sort of related and necessary for defining the image like the color of the stop sign is supposed to be red it's it's borders are supposed to be white the thickness of the board is supposed to be something all that all that stuff is uh defined up here but what's what's important here is the vector stuff and that's why this is called a vector image so svg is just one vector image format that a whole bunch of vector image formats but this is uh probably the the most popular format used today and the nice thing about it is it's just it's a text file it's a human readable text file it's basically a form of xml file so you can actually i hand wrote parts of this and then you can do that so that's the nice thing about svg files again i would highly recommend it use it when you need vector images but today's topic is not vector images right we do use vector images in computer graphics we use vector stuff a lot in computer graphics but today's topic is going to be more specifically about raster images now when we think about raster images that's when we get pixels right so raster images are made out of pixels pixels are picture elements that's why i'm calling pixels right so a raster image will be formed by a whole bunch of pixels like this uh they're typically um they're typically defined in a scan line order from from left to right and from top to bottom and each one of these pixels will have a color associated with them and this whole thing combined would form the image that we see so this is what we call a raster image right so each pixel is storing a color value all right so um this doesn't look like much because this is a very low resolution image uh it's an eight by eight image not very high resolution here's what i mean by resolution that the the width and the height of the image in pixels would define the the resolution of the image right so when you say image resolution this is what we're talking about width times height that's that basically determines the number of pixels right uh so as you see besides resolution what's really important here is colors so let's talk about colors a little bit right this is our next opening now when when i say color probably many of you are thinking i know what color is yeah a lot of people think that they know what color is but actually most people have no idea what color so they have a very very wrong idea about what color is uh so what do you guys think when i say color you think rgb or something else i don't know all right let me try to explain color a little bit i'm not going to explain in a lot of detail color is actually a very very complicated topic but today i'm just gonna briefly overview what what color is very very broad strokes right rgba opacity right no no no we're starting with this we're starting with light and light visible light spectrum so here light is an electromagnetic wave like it has a whole bunch of possible wavelengths a part of these wavelengths of light are in the visible spectrum so if the light has these wavelengths we can actually see the lights but anything that has lower wavelengths that is infrared or sorry larger wavelengths or smaller wavelengths uh ultraviolets we won't be able to see those lights right so we have a limited range of of light uh we can we can see now i might be able to say wavelength or or frequency they're basically related um frequency is one wavelength uh so if i say one or the other don't worry about it so larger wavelength means lower frequency and smaller wavelength means higher frequency right so this is the sp light spectrum that that we see uh but when you look at some light that that you see you're not going to see just one of these and an actual light that you see will be a combination of a whole bunch of frequencies this very much like audio signals all your signals will be combined a whole bunch of frequencies light is very much the same so for example here is the the spectrum generated from some a particular led light so as you see it's not producing just one particular light frequency it's it's producing a whole bunch of wavelengths with different amplitudes so certain wavelengths are more pronounced than the others so when you look at this you're going to perceive some color from out of this but what what act what the light actually is is a is a collection of all these uh wavelengths right uh so what happens when you look at something like this when you look at this let's take a look at our eye a little bit so light is entering our eye and here we have these these cones as they call it with some photoreceptors and this is where we start seeing stuff now there are different types of cones uh there are cones that perceive color and those those cones actually we have three types of cones and this in this figure they're highlighted with with three different colors uh we have humans have three different uh types of cones and those three different types of cones would form our color perception so a proper definition of color i think would be um a shared hallucination of our species and i'm saying hallucination because color is not it's not something that exists in the real world what exists in the real world is just the the light spectrum right uh what we perceive is the color so it's completely perceptual that's why it's a hallucination and it's a shared hallucination in our species because we all most of us have these three different cones so we can see certain types of colors and so we agree on what to call them right and this is basically what what we see out of the real world but we're not quite seeing exactly what it is and it is very specific to our species because um other animals actually have different types of eye structure different types of cones and they perceive life very very differently some animals can actually perceive color like like we do but but different than we do some animals are actually more types of cones than we do actually some people even have uh a fourth kind of uh cone as far as i know so this is this is not not even even for human species it's not really universal so the whole concept of color is definitely not something universal it's very much related to how we perceive flights so three different types of cones why are they different because they have different responses to the lights um some light spectrum that enters our eyes so one of them is more sensitive to the light that we perceive as blue one of them is more sensitive to light that people perceive as green and the other one is sort of more sensitive to what we perceive as reddish although it has some you know sensitivity over here as well so but regardless of what exactly these shapes are basically we're getting three values out of this visual system it just gives us three color values um i'm seeing some questions yeah all right so yeah the stream is recorded uh so three values that means we can sort of generate what a human can perceive as color just using three different colors combinations of three different colors if you carefully pick those three different colors by with various linear combination of these three different colors you can actually generate a similar kind of uh perception for humans and that's the whole idea behind what we call rgb right red green and blue uh so in computer graphics this is what we use for defining colors and there's a red color it's green color and blue color now there are standards for this more specifically most monitors that we use uh use what's called standard rgb srgb uh and that means these these colors have uh very well defined values and spectra associated with them but i'm skipping all those details basically there are three different lights and using the combinations of those three different lights we can get all colors that humans can see not all colors specifically that's that's related to the srgb standard srgb standard cannot produce all colors we can actually see colors that are sort of outside of the spectrum a little bit uh but you know let's skip those details for the time being that that's that's good enough actually it gives us a wide range of colors that we can perceive um so as you can see when you mix all of these colors when combined all of them together this is that's what we perceive as white in the middle here so if you dim all of these colors if you dump all the rgb colors and you all the way down to no light none whatsoever it becomes black so anything between black and white and all these colors are formed by the intensities of these color values all right so then we can just use uh some if you want to represent color values we have rgb values and each for each one of these are g and d the red green and blue values we can represent them as a range between zero and one zero meaning no light one being maximum light right um so this is a standard mathematical notation for uh for the range 0 to 1 including 0 and 1. that's very important 0 and 1 must be included uh because zero zero zero will be black for us white one one one will be white over here right and red is one zero zero green is zero one zero zero zero one and so forth so basically this is forming a coordinate system for us it is like a 3d coordinate system and this red is one coordinate direction green is another coordinate direction and blue is another coordinate direction so we can actually visualize color as an as a cube like this right so black is here at the at the back that we can't see on the far end of this this cube and and white is closer up here um in the middle when all the colors combine right and so red and and if you combine red and blue you get over here if you combine red and green you're up here and so forth so this is our rgb color space and this is the color space that we will be working with good now here's the question though now if this is black zero zero zero and this is white then tell me what is this sunlight is a lot brighter than the light that i'm gonna get from my monitor but that was like the maximum one one one what happened over here okay so here's the thing this white that we call white is the maximum light that our typical monitors can generate that's what we call white yeah so the sunlight would be a lot brighter right so this um this range between zero and one for typical monitors we call this low dynamic range ldr and most monitors that we use today will be ldr monitors um if you start including light intensities that are even brighter than the definition of white that's going to be high dynamic range right and today we have high diving range displays that they can presumably generate really really bright light of course not as bright as the sun not even close but you know brighter than what one one one is supposed to be so they give you a whole a whole lot more range than what you would get from low dynamic range displays right and this is important because that's going to impact how we store color and how we store raster images it's all going to be related all right so going back i have values if i have low dynamic range and i'm storing low dynamic range images um there are various options various ways that i can store it a typical way of storing them would be instead of storing a floating point value per color channel i would use eight bits to represent them so i'm going to quantize the range between zero and one and split it into 256 quantized units between 0 and 255 so in this in this range 0 would mean 0 obviously and 255 would mean 1 right and that's how i would represent all possible intensity values for each one of my r g and d color channels this is the most commonly used format today and most image formats would be actually using this 8-bit color channels and there's a reason for that actually it turns out if you do this right um and i'm going to talk about what i mean by doing this right during the next lecture if you do this right 8-bit is actually sufficient and and it's very very hard if possible at all for us to see the difference between color values uh between these quantized steps so if your monitor is properly calibrated you should barely see the difference between these quantized units of 256 values so this is sufficient but a lot of times um a lot of times we do also serve manipulations to images uh so when you generate an image and you're going to modify it you're going to change the color values a little bit maybe you're going to draw something on it you're going to do all sorts of things with it because i mean there's sufficient now but if you start modifying it this quantization can hurt you later on because if i take an image and then i say oh i'm just going to make this image brighter take all the color values multiply them by two all of a sudden my quantization levels were multiplied by two right so i'm losing information if i use this this type of quantization for images that i'm going to modify later on so because of that i might want to use more more precision uh and and and i and we typically use for those kinds of uh ldr images 16 bits then 16 bits would give you quite a lot of precision uh way more than we can perceive so they you know they hold up pretty well in uh image manipulation as well so if you're dealing with low dynamic range images these eight bits per channel or 16 per channel would be sufficient and for most images that are designed for us to just consume that for example the images that we see on our web pages are the videos that we watch all that stuff is going to be just using eight bits per channel because that's sufficient for just consuming without manipulation all good now if we need high dynamic range however now things get a little more complicated because you know how high diamond range are we talking about right beyond one we're not contained within the this finite range anymore uh so you know it's it becomes reasonable to store colors color channels using more bits so we could use 32 bits in which case it's going to be full flaws now we can use um i should be 14.4 man and we're familiar with it and it's done actually for a lot of computations that we do in computer graphics uh we're going to use 32-bit floating points values for representing colors so during our computer graphics computations this is what we're going to be using and once we're done manipulating color once we say okay we computed the color this is going to be displayed for this pixel this one we take this value and we convert it to um the corresponding uh ldr value right and of course we're gonna have to um chop it off if needed if it's greater than one we're gonna have to uh collect that part uh so this is what we're going to be using um and this is used for for image storage as well but for image storage of course it takes a lot of space so sometimes people prefer using 16-bit float because the thing is 32-bit floats they can represent very large values and we don't really need that much large values right i mean we're not going to be concerned with the intensity of the sunlight too much anyway uh so 16-bit floats for for a lot of cases it's sufficient actually i'm going to talk about briefly about some hydraulic range storage uh and and they use more complicated structures to to compress the storage down a little bit all right so let's talk about how we're storing these faster images there are two general ways of storing color information uh we can use that we can store them in interleaved format where each pixel will have its own rgb color stored next to each other rgb rgb rgb rgb like for example in a scanline order and if i do that then i don't need to know anything about anything else about that image i just need to know its width and height okay i probably just need to know its width um and and then i can just store rgb colors one after the other for each pixel and that's going to be my image so that's a very typical way of storing images most displays will will be using this this sort of an interleaved format and actually the the image format that you guys will will get in that javascript function in our first project is going to be stored in this interleave format it's going to be colors rgb rgb rgb rgb so this looks like a 2d array of color values but you can also think about this as like a 1d array and it's typically stored as 1d array and if i know the width of an image the width resolution then i can tell based on the index of a a color i can tell where it is in this 2d plane right so you know using the scan line order each color value is going to get its its own id uh another way of storing images is going to be as you see here it's going to be as separate channels this can be used in some cases sometimes maybe you want to modify or manipulate the red channels or green channel independently or do some different manipulations with it so in some software sometimes they story in color this way and there are actually uh image file formats that will store color this way but this is um less common and and there are reasons for that because for a lot of colored images the the rgb values that we get in an image will be sort of related to each other they're not going to be completely independent so storing them together would allow you to get better better compression when you're using compression but you know if you're just keeping them in memory this is you can you can use either one whichever one makes sense i'll talk about this in a bit more detail later on so let's talk about some popular raster image formats well i'm laughing here because uh yeah this is sort of popular but not really back in the day it was sort of popular so bitmap and portable portable text map image files and bmp and ppm image files these are probably the simplest image formats actually ppm is probably the simplest image raster image format it's it has a super simple um header file just just uh just requires you to to write the the resolution of the image and then you dump all the rgb data and you're pretty much it's very very simple um and these image formats what's important about them is obviously they're ldr formats so they're not going to store high dynamic range images and the images they store are going to uncompress so they're going to store these rgb rgb rgb rgb values all right uh and those rgb values could be used could be using just a single bit per channel in which case it's going to be a black and white image or it could be using like eight bits or typically they will use eight bits or 16 bits as we talked about right and these image formats are actually not all that popular and i'm saying popular brass image format is because they are sort of used there are so many image raster image formats over the years all sort of camera manufacturers came up with their own image formats with different compression schemes and all that uh so i have no intention of going through all of them and totally unnecessary too but i'm just going to talk about the important ones and i'm listing these because they store images as uncompressed so these are as simple as it gets in terms of image storage now if you want to compress images and believe me you want to compress images uh png is a modern uh graphics format uh portable graphics format is what's called uh it does um lossless compression so we're going to take an image and it's going to compress it uh using uh different it can use a zip compression or another type of compression regardless you will be able to reproduce this image exactly when you uncompress it so png images will be used for storing files on a disk or know sending files over network but once you receive the file or once you read the file from the disk you're going to uncompress it um and then uh and then you're gonna display it as an uncompressed format you're going to store it as an uncompressed format you're going to send it to your display device in an uncompressed format all right and again these are designed for ldr images and they can be six or uh sorry eight or 16 bits per channel or or or alternatively they can store color tables so and this is kind of important now i'm saying lossless compression but if you use a color table you're going to lose step so here's how it is a color table but use a color table you're not going to store red green and blue channels per pixel instead you're going to store a color table with two to two uh 256 colors in that color table and within and then for each pixel you're just going to store the index of that color in this color table right so if you have a lot of pixels and very few colors this becomes super efficient if i have like i don't know four different colors in my image and i have a very very high resolution image i might as well just store the index of that color and that's going to compress really well as well so it's uh it makes a lot more sense so the important thing here to know is that png files are lossless unless you're using a color table if you're using a color table you're first going to quantize your colors to fit into those colored colors in your color table you may have a lot more than 256 different colors in your image so first you're going to pick the nearest color in the color table and then and then the rest of the compression is going to be lossless and it's important to note that the color tables are stored separately uh per png image so there isn't a standard color table that every png image uses uh these 256 colors will be or however many colors you're using for your png image they will be generated specifically for your image all right so that's how we use uh color tables and i'm talking about these color tables a lot because another very popular file format uh graphics interchange format that's gif also uses uh color tables and these color tables in this case can be two to 256 colors and this is a much older uh file format uh raster image file format it has been very very popular actually um world wide web made it or shall i say internet made it very very popular back in the day that was a very popular file format uh it still is used very very extensively uh in on it only over the internet today but probably png images are more popular um so it's using a similar uh compression scheme uh so after you quantize it's it's using a lossless compression scheme so a part of the compression comes from this using this color table by reducing the bits you will need for storing uh each pixel and the other one is coming from this compression scheme that this this particular file format implements all right so these are typical um common file formats used for storing uh images in a lossless form modulo the the color quantization if you're using a color table uh a very very popular um image file format is jpeg drawing photographic expert groups and generated this format back in the day uh this uses jpeg uses lossy compression uh now i'm not gonna i'm not planning to explain all the details of how jpeg compression works because i don't think it's too important but it's kind of unders it's kind of important to understand that it is lossy compression that means jpeg images cannot be uncompressed and and in a way that would uh exactly reproduce the original image so you take an image with an rgb sequence and let's say scanlan form and you compress it as a jpeg image when you when you read it off again you're not going to get the same image anymore you're going to lose some information that's why we call it glossy and there's a reason for this by using lossy compression jpg images can compress images can compress image information quite significantly so a jpeg image will be much smaller than an uncompressed image how much well we'll see and typically jpegs use typically typically use eight bits per color so okay let's take let's take a look at how these different formats fare all right so here's a low resolution uh teapots image that's that's our graphics lab logo uh saved as a bmp uncompressed bitmap file format it this one takes it's a very very small low resolution image and it takes 146 kilobytes now very large but for a small image that's actually pretty sizable um if i just use png compression in this image and png is not going to lose anything it's going to look identical i'm switching to png it's going to look identical it is identical it's lossless all of a sudden it's 5 kilobytes from you know going back from 146 i'm going all the way down to five kilowatts so this compression can make a huge difference um so what happens if i use jpeg now i'm first going to show you a very low quality jpeg jpeg has some quality knob that you can you can adjust i'm going to show you a very low quality jpeg and i'm warning you it's not going to look good you ready it's going to look pretty bad it is so oh these really awful things and here's what's going on here let me tell you just a little bit about jpeg compression so what what it typically does is that it takes these eight by eight blocks of pixels and within each eight by eight block um without getting into too much detail as broad strokes strokes um it will basically store the average color for that block and then it will store some uh for each pixel of the store the differences from that average color and because it's storing differences from that average color you can encode those differences with very few bits and that's where a lot of the compression comes from so it doesn't need to store eight bits per column check panel channel it's using very very few bits and you can adjust how many bits are being used if it's a very low quality jpeg is quality set to zero here that doesn't mean zero bits it's just the lowest quality that's what it means uh you get get something that looks uh really bad so if you crank up the quality in this case like medium quality you get something that looks a little bit better uh so depending on the quality of the stream that you're you're getting either through zoom or youtube you should be able to see these jpeg compression artifacts but if i crank it up and go to 10 maybe some of you are seeing it if you're careful maybe others are not seeing it quite but they are there this is not perfect and it's not going to be perfect because this file format is not designed to be lossless it's going to be glossy so there's still going to be some artifacts around these um these sharp edges all right well what if i use gif all right let's let's let's go back here actually i forgot to highlight some important detail here all right all the way to bitmap now 146 kilobytes to five kilobytes in losses png compression right now i'm going to low quality horrible looking jpeg and it looks at 17 kilobytes yay i'm losing image quality and storage so it seems completely useless and it is useless in this case because this is not a good image for jpeg this is a very bad image for jpeg compression i'll show you other examples where jpeg is going to do a lot better than this but here it's not good and if you crank up the quality of course you're going to need more and more bits and you know things are better but now i have a larger file it's not even worth it now if i go back to give compression and with give i'm using just 16 colors and three kilobytes no artifacts it looks just fine i there are there is color quantization here because the original image had more than 16 colors i mean you're saying where are these 16 colors i'm seeing white and red but there are colors in between here around the edges some pixels have colors between white and red um so we had something like um 48 colors or something in this image i believe something like that and we quantize them to 16 but you can't even tell the difference let me show you the original one this is lossless this is quantized version i mean you can't see the difference right 16 colors in this case turns out it's good enough so that's why the gif images are still used a lot today and png file format supports color quantization so you can get the same same level of uh compression with png as well right now okay for this example jpeg did not do so well but it's not always going to be like this so here's another example uh here you see that there's a lossless png um genji on your left side uh and on the right we have the jpeg compression image with jpeg compression with quality set to zero i should say jpeg here yeah this is jpeg quality zero right so yes quality zero is not great when you compare these two images you should be able to see some differences and some compression artifacts here uh it's not immediately as obvious as before though if the teapot was terrible this is not as terrible and the image file is actually significantly smaller in this case because the compression algorithm losses conversion of png cannot do such a good job for this particular image yeah but it is it is terrible it's pretty bad so let's crank it up let's let's go to medium quality when you switch to medium quality it's a little bit better definitely better fewer artifacts um and uh but of course we kind of paid with some extra storage we went up to 15 kilobytes but still better than png if you crank it up more go all the way to 10 we get 22 kilobytes actually 10 is not the maximum quality photoshop will allow you to set the quality to uh 12 for those of you who haven't used photoshop it's a um it's a professional photo editing and image manipulation software and it will allow you to save jpegs with quality 12 that's going to use even more bits and better quality obviously uh so in this case it's going to be very very hard to tell the difference between these two images right because jpeg compression with quality set to 10 is doing a very good job uh and the image is much smaller than png so depending on the type of image jpeg compression may make sense or not so you kind of need to know what kind of image you're using and based on what kind of image you're using what kind of image what kind of image data you have uh you need to pick your file format accordingly you kind of need to be careful about that because these um different compression algorithms are designed for for different purposes uh here's one more example that i'm going to show you uh in this image i have this this pnd and png image with wallace's compression is 35 kilobytes without compression this would be a lot larger obviously uh but here's a low resolution version of the image on my website uh so with losses png compression is 35 if i switch to gif in this gif format you get 60 with 64 colors uh you get eight kilowatts yes it's significantly smaller but but i have color quantization artifacts can you guys see the color quantization artifacts so i have only 64 colors so if a color value is in between two color values in the table we need to take one or the other right that we can't do anything about that but one thing that's um that most software that converts images to this format do is that they don't blindly converge each pixel completely independently uh they look at the neighboring pixels and the error made in the neighboring pixels and they sort of try to keep the average error minimum here's what i mean by that now i have a particular color value i don't have that color value in my color table i have one color value i have i look at the closest color values i have one that is slightly brighter i have one that is slightly darker now i'm going to pick one or the other right i have to pick one or the other i can't do anything else uh so if i always pick the brighter one for all of the colors when i convert my image to to this format the image will look just brighter right because of color quantization if i always pick the lower one the image will look darker so what these algorithms will will do is that they're going to keep the average brightness the same and they're going to do this uh sort of by considering each color channel obviously uh so they're gonna keep the average error in that in the neighborhood uh as small as possible uh this is what we call error differing so as they're converting them as they're quantizing the image they're going to accumulate the error and they're going to use that error to to decide which nearby quantized color value to go whenever they have the quantize that the neighboring pixels so the result of that is when you have um regions with in between colors you're going to see a whole bunch of pixels that are switching between two different colors so if you uh i cannot show your look over here maybe there are colors in between two color values uh or like on the cheek uh there are going to be colors between two color values and those are you're going to see some pixels switching to one color neighboring pixel switching the other color so if you look at it from a distance and you cannot make out each pixel individually um this will give you the perception of that average color this actually works out pretty well when you have a high resolution display but here i'm just blowing up all these pixels so you can see the differences that you know that that's why it's kind of uh kind of looks bad because this image is actually very very tiny all right so this is what happened when i moved from losses png compression to the gift with 64 colors only now if i use jpeg here and i'm going to set the jpeg quality such that i'm going to get the same file size more or less and it's going to look already and it's going to look like this so from here to here yeah it's not perfect but it is so much better if you ask me yeah in some places it has jpeg compression artifacts uh here you can see some cherry compression artifacts here and there but in a lot of places i'm getting in between color values and jpeg is doing a a lot better job at that um the top here you should be able to see some quantization artifacts right around here um but overall it's do it's doing well um but you know don't undermine the gift five format because it's actually very very good for other things as well and one of the things that's used very very frequently um especially on the web but sometimes during presentations is animations uh because there's that your platform match supports animations as well uh and the way that this is going to work is that uh basically it's it's storing a a bunch of different images uh with different timings and it's displaying those images either in a loop or maybe just plays once you can you can set it up in various ways for pretty much all the animations all the image-based animations you see unless they're recorded as videos they're going to be the animated gym now a png file format also supports animation there's an animated png format it's a little bit different so it's not as widely supported this file format has been around for decades so it's pretty much supported everywhere but the innovation with png even though the file format supports it it's not supported by all browsers so it's not as commonly used because of that all right so enough about enough about ldr file formats let's talk about hdr file format um there aren't as many hdr file formats because hdr is a relatively new concept because back in the day when people were worried about image file formats every bit was important storage was very expensive internet was slow so uh you know we couldn't even think about say oh i want to have more and more beds per pixel that that was beyond anyone's dreams and they didn't care um but then you know for for certain tasks people actually needed image file formats that store the full color information more specifically uh industrial light and magic um ln uh ilm uh came up with this uh specific format opendxr format or exr format uh this this format is designed for storing hdr images it can use 16 or 32 bits per channel and it can use velocity or lossless compression and these images are uh used for film production because in film production you you end up doing a lot of image manipulation so you take images and you modify them and we said you know going from eight bits to 16 bits was good enough why are you thinking about 32 bits now why are you talking about hdr it's actually needed for hdr because we don't want clipping at all so when you're generating an image and it turns out your color values are brighter than white for whatever reason because of your computation you don't want to lose that information when you save the file to when you say that image to disk that's why they're they have been using this format and this is not the only high dynamic range image format there are others but this is a very popular one that's why i'm specifically talking about this and there are different ways of storing high dynamic range images because these are going to be a lot more expensive right so just going from 8 bits to 32 bits you're going to quadruple your data right but it's not just that turns out you can compress 8-bit images uh 8-bit color channel images a lot better than high dynamic range images so the compression algorithms are not going to work as well for these uh for these as well and that that's that's another reason why these um hdr images are more expensive but they are definitely huge they have their own use cases in computer graphics especially in film production uh they are very very popular because you don't want to lose any information there when you're like imagine that it took you uh hours and hours to generate an image and then you rendered a video that contained a thousand images like this you don't want to lose that information when you save it to this because you know maybe you'll you'll need it for something and if you don't have that you're gonna have to recompute that that's really expensive storage today is not that expensive so it's okay all right i'm almost done for today but i want to talk about one more thing before before i end it uh so typically when you're when you read images from the disk or or when you receive images image files on the internet in our browser they're displaying them they're going to take those images they're going to take those compressed images and they're going to convert them into an uncompressed string of bytes of rgb values and this is what's going to be sent to our display so even if we are getting an image that is using 16 bits per per channel um you know i might be intermediately storing them as 16 bits per channel but at the end of the day when i'm pushing it to my display to show it i'm going to use 8 bits per channel and that's how i'm going to send it to my display i'm nice i'm using a high dynamic range display uh and they're going to be typically stored in this interlead format and they're going to be using this scanline order so that's a typical way of storing images and this is exactly uh the the way that images will be stored for our project in the scanline order uh but turns out this is not the most efficient way of storing images for certain algorithms uh for a lot of 3d computer graphics and applications uh we're going to be looking at images and we're going to be manipulating images in local regions so it makes sense to think about an image as like a 2d construct instead of this this 1d scan line because pixels right above and below each other in scanline can be very far away in memory uh and that's bad because for a lot of graphics algorithms if i'm accessing a pixel i'm probably going to access its horizontal neighbors and vertical neighbors too for for a lot of algorithms so it kind of makes sense to put them close by in memory so i get much better cache efficiency and so forth and because of that this um different types of swizzled uh orders are more popular and especially for gpus gpus will store images oftentimes using a swizzle order like this so you can send an image in into the format from your cpu to your gpu and the gpu will get it and swizzle it and that's how it's going to store it because it's going to be a lot more efficient for a lot of algorithms so this is a typical swizzling order that's used a lot is that what's called a z curve here i'm following this this z shape and it's like a fractal you basically follow the same shape here and then go same shape here and the same shape here and overall at the high level i ended up using the same z but larger z right and if you look at a collection of pixels here i'm i'm drawing a bigger z right so that that is the z curve again this is just one swizzling order uh because of that the way that caches are are used actually at this lower level it doesn't make too much sense to swizzle these these four pixels together because they're going to be the same cache line it doesn't matter how you store them uh so and so this at the very low end the swizzling can be disabled so there are different ways of using doing swizzling uh that's all i'm trying to communicate here this is not the only way of doing swizzling but when i say swizzling typically this is what people imagine um and the important point here is to understand that the scan line order is just one order of storing these images they can be stored in different ways as well but in memory the images are not going to be stored in compressed formats not not in in png compression with png compression or with with uh jpeg compression so that they're going to be uncompressed in in memory because we're going to be manipulating them that being said there are other compression algorithms that are suitable for uh real-time image manipulation so gpus gpus use a lot of image compression but they're going to be using different image compression algorithms not the same image compression algorithms we use for storing the images to disk right and i'm going to briefly touch on them later on just probably briefly i think suffice it to say that image compression happens on the gpus and sometimes they're even hidden gpus will compress anything losslessly if they can have lossless compression they'll just use it because you won't know that they're compressed uh and they can get the benefits of compression uh but for lossy compression of course they kind of need to be a bit more careful um all right so i'm going to stop today's lecture here i used almost our entire uh allocated uh class time uh so i'm gonna stop here and i'm gonna say do you guys have any questions about all of the stuff that we talked about maybe i should follow the chat a little more carefully are the differences between separate formats mainly how the colors are stored or what colors the algorithms considered important so the algorithms the image compression algorithms uh of these none of the ones that i know uh they're not sensitive to the the color that you're you're storing they can pretty much store any color of course if you're storing a color table there's going to be a limited number of colors but they are they can't be more sensitive to how the colors are changing for example for for jpeg uh for for for jpeg if the nearby color values are similar uh in a 8x8 neighborhood uh jpeg does a much better job in compressing them uh but if you have sharp boundaries like oh i have this red piece here and then all of a sudden i have this white piece here this this sharp changes jpeg can't can't really handle well um for formats like png or or gif or gif they are they're very good at compressing uh constant colors if i have constant value they're very very good at compressing those and that's how they can get better compression than jpeg because jfk can't do as good a job and for constant values so they are sensitive to the color information but they're not selective in in terms of what kind of color values they they would like to compress when a computer screen displays black does that mean their rgb leds for that pixel are all off they're supposed to be uh but not always uh so there are different different uh technologies used for this place uh sometimes we have a backlight that gives you most of the light and there are pixels that are sort of blocking the light uh using different forms of polarization uh so if yeah if your display is set up such that you have this uh giant led array in the back forming the backlight uh then you know your pixels will try to block that light but they may not be able to block all of the light so that's why when your computer monitors are on even if you're displaying a black screen it's not going to look pitch black you will still get sound light most of the time but some more smart displays will actually shut off that led some of them will shut up parts of those backlights uh so it varies from from display to display uh your formats with different amounts and bounce bits for each other for each color channel exist uh yes there are formats uh there are ways to store different color channels with different numbers of bits because turns out certain colors well we have three color channels turns out green is more important than red and blue because a lot of our light sensitivity comes from green so it kind of makes sense to allocate more bits to green sometimes but for various reasons you may want to allocate more bits actually there are some image manipulations happening on your displays as well and so some displays use uh ten bits for example for representing colors even though their input is eight bit per pixel they're gonna oppress it to ten bits and do some image manipulation so that so that when they are displaying it they're not going to lose data so there are a whole bunch of uh different formats and and you know the number of bits you have may vary from color channel to color channel any other questions and i had a question with hdr storage versus ltr storage so you know when we're storing in eight bits versus you know 32 bits like what what's the difference in what is being stored is it just a higher range of numbers so you want it brighter or are they just more specific color accuracy it's more about about ldr hdr if you just look think about ldr hdr it's about eliminating clipping so with ldr images white is the brightest and if you want to go beyond white you can't uh with hdr images you can go beyond white well you won't be able to display it if you're displaying it on an ldr display but your image data is going to have that information so if you're changing your image intensities then you'll be able to start seeing it so you're preserving that data not destroying that data even though you're not seeing it and that's what's important for ldr versus hdr the other important thing is of course the storing more information more in between color values within the range zero to one but this is i believe less important because if you're just storing if you're just using 16 bits per color channel and all of your colors are ldr zero to one 16 bits is really more than sufficient for for almost anything oh okay so like let's say ldr using eight bits versus ldr using 16 bits the max value for both of them is basically the same thing exactly yes oh okay just the quantization is different yeah yeah that's perfect yes in one of them you're splitting it into 256 different values uh and the other one you're splitting into 64k values uh all right i'm going to end it here then i'll um see you next time on thursday and we're going to talk about talk more about raster images thanks

Channel: Cem Yuksel

Views: 3,613

Keywords:

Id: zllIPDaiOyk

Length: 62min 49sec (3769 seconds)

Published: Sat Jan 30 2021