Camera Basics and Propagation of Light (Cyrill Stachniss, 2021)

welcome to today's lecture we want to look today in cameras and the propagation of light and basically want to understand what are the central components or building blocks of a camera in order to understand how an object from the 3d world will be mapped on to our camera image at least get a basic understanding of the so-called pinhole camera model and for that we will start with the components of a camera of a digital camera what does the camera actually measure and then go further and see how the chips on the camera actually work okay so the questions how do we obtain an image we all know we can have a camera that can be our cell phone this can be an old film camera this can be in dslr or any other type of camera that we use in order to create an image of the world and the questions is now what are elements in those cameras what do they have in common so that we're actually able to obtain an image so we all know that we can take our camera out we can take a picture of the scene but the question is how is the image generated from that scene and that's something that we want to look into today and we start with a question of what does a camera actually measure and here a camera can be a really old film camera this film camera is basically all the same ingredients that a modern digital camera has except that the film which sits inside the camera is replaced with a light sensitive chip which simply replaces the functionality of the chip there is actually not much more into this at least if you think about the basic imaging process so it doesn't matter if we think about having one of those old analog cameras or a more modern digital camera in order to capture the environment you know to take measures of the scene using cameras and so the question is so what is it actually that those cameras measure and what they actually measure is light so a camera is a light measurement device and it basically measures how much light reaches the camera and it uses its lens to channel some of the light into a certain direction in a specific way and so that this light is then projected on a certain position on either the film or the chip so that we can combine the measurement of the amount of light with the direction from where that light is actually coming and reaching the camera so how is the camera doing this the camera basically provides a 2d image of the world so it takes the 3d world in which we live in and projects it onto a so-called image plane onto a 2d plane and so all the objects in the 3d volt which consists of an xyz coordinate will actually mapped be mapped to a two-dimensional coordinate an x y-coordinate on that image plane and on this image plane there we have um so-called or sensitive chip which generates so-called pixels or picture elements that's where the name pixel comes from and we all know pixels from our images um but what these pixels actually do they represent a certain area on my chip a light sensitive area and the intensity value that we typically read out from an image is basically a counter which tells me how many photons how much light did the chip or that specific cell received from the environment so every pixel is basically a light intensity measurement device so you can see the sensor as an array of very very small light intensity measurement devices and which basically tell you how much light has reached this certain area on our chip so that means every pixel in on that chip and then in the end every pixel of my image corresponds to a measurement measurement device or measurement of the light that comes from a certain direction because every pixel corresponds to a ray that passes from that pixel through my projection center through my lens into the 3d worlds and what the pixel value basically says is how much light was coming from this direction so in sum each pixel measures the amount of light that reaches the camera from a certain direction so it's the amount of light coming from a certain direction and so this directional component is important um especially in photogrammetry where we try to do or often try to do 3d reconstructions of the scene and therefore this directional information is especially important for us here okay so let's dive a little bit deeper and look into the individual elements of a digital camera so a digital camera or a camera in general consists of different parts some parts which we can see is the lens and maybe the camera body although the sensor also belongs to the uh camera body so lens departure um this this sits in our lens we have the shutter we have the sensor which or the the old days the film so this was basically the film before um and the sensor is kind of a digital variant of the film there are some analog digital conversion happening and also some post processing inside your camera in order to make your image look beautiful and what we want to do is we want to go through these individual steps at least up to here um in order to understand what the different components actually do and i want to start actually with the sensor so starting from the back and then reverse this to see what are the different components of my camera and how do they affect the measurement so how does this sensor look like this is the size or it's in the area where typically your film sits in your analog camera and a chip may look like this so here is the light sensitive area and this consists of a large number of really tiny elements and all those tiny elements represent the individual pixel um on your uh on your for your image later on so on modern ships can be 5 million or even 100 million of those small tiny picture elements and what this chip does it basically tells you how much light has reached a certain area of the chip and turns this into a digital signal so from an analog signal into a digital signal so that this data can be interpreted and we get our intensity values for example ranging from 0 to 255 although this is of course an arbitrary number just based on how we represent our images in in our computer okay so the sensor basically converts the incoming light into intensity values and it's an array of those light sensitive cells those pixels that's important it's not a single one it's a large array and every pixel of our image basically has a corresponding picture element on that chip and the amount of light that reaches the chip is of course related to the size of the chip so if the chip is larger we typically get more light onto our chip and then also the pixels are typically larger and as a result of this um the the larger the sellers the more light it can actually collect and typically the higher higher quality your image will will be in the end however the problem is larger sensors are more expensive to produce they also make your camera larger and also require to have larger and substantially more expensive lenses therefore we have a tendency today that we see that um those chips get smaller and smaller because all lenses get smaller and smaller so you can actually embed them for example into a smartphone and don't need to carry a very bulky lens but of course the larger the chip to be the better the quality that we have and this is an example of the chip size so the kind of standard full-frame camera that we use the film camera is this size over here there's a medium format sensor and then things get smaller and smaller and kind of modern dslr cameras typically sit somewhere here this aps-c sensor size which depending on your manufacturer sits between those two elements it's typically the size um of what it sits in your dslr camera and then it gets smaller and smaller and smaller and even that's not the size of a modern cell phone anymore they are even smaller than that today so you can see um we have the tendency to make that chip smaller also the quality of the chip of course increases but generally we still have it still holds the larger the chip the better our quality and but again it comes at a higher cost at larger larger lenses that you need a whole larger camera so there are good reasons for actually going towards smaller devices and when some what this chip does as a light sensitive chip it turns light information into certain positions into an electric signal which then is converted into a number basically telling you the intensity the amount of light that has reached that chip at that specific location okay so so far we have talked about intensity values but we still have no idea how it's actually color being generated right so we only said that it takes the amount of light but we haven't said anything about is it red white is it green light is it blue light is it near infrared or anything else um so we talked so far about kind of the monochrome process doesn't matter which color the light has it will be mapped to an intensity value but today you know all the cameras or most of the cameras that we see color information is available so the question is how do we actually get this color information and there are two main ways how you can do this the first thing is is a so-called three chip design so it's a camera which consists of three light-sensitive chips not a single one but three and in front of each chip is a different filter so there's a red filter a green filter and a blue filter each in front of each ship and then you have a semi-transparent mirror so to say which takes the light which is incoming reflects a part of the light here for example to this uh chip with the red filter and some of the light passes through it's reflected here onto a blue filter and then passes through a green filter so you will have blue a green and red image being generated by basically having separate filters in front of individual chips so it's kind of the three chip camera and that's the technique using this beam splitter that has been used in professional videography or video cameras for cinema cameras for example some of them have that process but that's something that we typically don't find in a photo camera for the cameras and as well as kind of most smaller size video cameras use a single chip design and there we don't want to have three chips because this increases size it increases the cost of those cameras we just want to do with one single chip and the trick is here is not to add one single filter but to add a large number of very very tiny filters actually filters of the size of an individual pixel directly onto the chip so that in the end we have a chip which is here illustrated and we put different filters in front of each pixel so we have a certain pattern that we see here consisting of green red and blue filters in front of every um every pixel and so what we're basically doing then we just take one picture with this chip and we will only get the green information here the blue information here the red information here the blue information the green information here green here blue here red here green here and so on and so forth and then we are basically interpolating the actual color value from these three channels so we are basically combining nearby pixels into a single pixel and or a single intensity value that also comes with color information so an intensity value in the green red and blue channel and of course this is an interpolation it's not that perfect so especially if you zoom in at the pixel level you can sometimes especially add borders see those interpolation effects but it's of course much easier to do because you just need a single chip and don't need multiple um chips inside your camera so it's cheaper it's smaller but on the other hand we basically have only a third of the number of measurements that we are doing right because we have one chip which measures everything except the three chips measuring all the individual channels so the number of measurements that we have is smaller and of course the smaller the number of measurements we have the less information we have about the environment and especially the last point is important the interpolation leads to a lower quality of my resulting image so as i need to do interpolations this can actually substantially harm my image process but there are different ways how you can optimize those filter those interpolation processes or demo cycling how it is called in order to generate as good as possible images but there's also not only one way to do this pattern so you also see different patterns which are out there and you can actually see that you have a certain pattern here so it's called a buyer pattern which is today probably the most commonly used um pattern and what you can actually see in here is that for example the green pixels appear more often than the red and the blue ones actually 50 of the pixels are green um 25 are red and 25 um are blue and what we're actually then generating is um this image or this image from the chip having the same number of pixels in my image and here the green the red the green and the blue value are the actual values and the other channels need to be interpolated from the neighboring cells which of course leads to interpolation effects so the question is why do we have for example more green pixels over here rather than red and blue ones why does it make sense and this is due to the human eye how we as humans perceive the world and we are very sensitive our visual system is very sensitive to high frequency details in the luminance which is dominated by the green channel and therefore we have more green information because then the images become more beautiful or better to identify for us as humans so this is basically adapting those cameras to our human perception system that's not the only way you can do this there's a large number of other patterns that you can go for and then also need other interpolation schemes but the buyer pattern is probably the most commonly used ones here are a few examples of other patterns that you can find depending either on your on the company you're purchasing the camera from or special camera designs uh this just an example you can also imagine having patterns which um have certain filters attached to their cameras which are outside the visible spectrum so that you can add information or color information from colors outside the visible spectrum so colors that we can't see for example near-infrared information is something that you can also by um adding certain filters to your chip you can add or you can explicitly perceive certain wavelengths for example near infrared light if this is important for your application so that's not the only one you can even ask camera manufacturers to produce customized filter patterns for your application okay now consider we have those for example the bio pattern that i was talking about so now how do we actually turn this into the color information that we have and we said we do this by interpolation and just to show you that it matters how you do the interpolation i just have here a few examples of different the most hiking techniques that means recovering the rgb values from these patterns and this is the original high resolution image and you have here doing standard bilinear interpolation you can see actually here at those boundaries that you have errors in your color so it's probably a little bit hard to see in the video but if you look into the slides themselves you can see that here at the boundaries you actually have mistakes in the color because there the interpolation doesn't work well whenever you have whatever uniform colored blobs you don't have any problem with the interpolation but it's strong gradients in the individual channels there you actually see those effects or other interpolation techniques may lead to other boundary effects which then look weird compared to the original pixel or you have more advanced demosaicing techniques which then can get rid of most of those effects but nevertheless there are mistakes that we are typically doing and just to show that a little bit more to the in detail i just brought here an example where you have your original camera image or the full camera image if you would perceive it with the three chip sensor you would have the red the green and the blue one and which you can overlay in order to get the actual image what we now can do in order to illustrate the mistakes that are happening we can actually black out 75 of the red pixels 75 of the blue pixels and 50 of the green pixels exactly reconstructing the buyer pattern for example and then overlaying those images and performing the actual interpolation and then we can actually have the actual image as it would be taken with a 3 chip camera and the image taking as taken with one chip camera here side to side and the first side those images look identical but if we look to areas of high gradients for example here in that fence we can see color mistakes popping up and if we zoom in here we can actually see different effects of the um of the interpolation technique that leads to actually wrong colors in our image and that is one of the problems that i have with this single chip cameras that i need to put actually quite some brain power into the interpolation technique but still can end up having mistakes in the color values therefore if you zoom in a lot into your high resolution images especially at strong gradients or areas with strong gradients so strong changes in color information like here going from white to a very dark color there you can actually see those effects popping up and you see the mistakes that are generated in this process of actually generating the image the next thing i want to look into is the shutter and the shutter basically controls the time at which the chip is exposed to light so basically you can see this shutter as kind of a curtain which can open up and close again and therefore allow light to actually enter the camera body and reach the chip and so the shutter basically controls your exposure time or the shutter speed um also as it was referred to and there are different ways how this shutter can be done it can be done manually or mechanically so manually mechanically you also kind of have um can do that realizes an additional form so if you you don't need that for the film camera you needed that there are some digital cameras which also have these types of shutters and our cameras today um typically use an electronic shutter for actually doing this and the shutter funnily has a big impact on the images especially if we have cameras that move through the environment because what happens there are different ways of the shutter can operate either the shutter can open at once it's completely open and then completely close so that the whole image is exposed um at the same amount of time everywhere or more or less the other thing what it can do it can just open a little bit like this slide and then basically slide this over the image so this reduces the exposure time of the individual pixels so if you go to very short exposure times um you will end up in situations where this shutter or curtain is never fully open it's only partially open basically sliding through your image this however leads to the effect that you have exposed certain parts of your film or of your chip at different points in time if they slide through maybe the pixel over here is exposed two or three milliseconds later than a pixel down here and this can actually lead to problems if you have fast moving objects if either your camera is moving quickly through the environment or the object that you're picturing is actually moving um quickly to the environment so the shutters actually have strong impact on the image process but also the um in another sense so if you have for example the shutter open for a long period of time with objects moving through the environment you will actually generate motion blur so even if the shutter is completely open let's say for half a second and you have an object moving through the environment this object will be blurred in its motion so you actually want to go down to short shutter speeds to get sharp image and basically freeze the motion unless it's a whatever an artistic concept that you want to have explicitly some motion blur in there but in 3d reconstruction we typically would like to avoid motion blur and therefore we are interested typically in short shutter times or high shutter speeds short exposure times in order to get sharp and high quality images but then we need to have enough light around in the scene and we may suffer from these effects of a rolling shutter which can be problematic so longer exposure times means more lights brighter images but also the more motion blur so it's depending how much light do i have available and how fast are the objects moving through the environment if i know the scene is completely static and my camera is fixed on a tripod there is no need to bring down the exposure time and maybe not add additional light sources if i want to capture a fast fastly moving object then i need to increase the amount of avoidable light either doing this during sunshine or adding artificial lights for example there are different ways how i can control the amount of light that reaches the camera is reflected from lights through the objects and then onto my camera and if i can increase the amount of available light i can reduce my shutter speed to get the same bright or correctly exposed images i want to quickly look into these different ways how a shutter can be generated what's called a rolling shutter and the global shutter so the global shutter is the shutter which opens or exposes all pixels exactly at the same point in time through the light and then deactivates is also at the same point in time or the rolling shutter which basically exposes one row after the other and this doesn't only exist only for mechanical shutters also digital shutters have these effects because i may not be able to read out all cells of the chip due to its design and in at the same point in time i may read out this chip or these small pixel elements on the chip one after the other and this can lead to what's called these rolling shutter effects and again you can still imagine this as a completely mechanical shutter where just the curtain opens just a short slice and it's basically moved over the image that's what we call a rolling shutter and the problem that we have with this rolling shutter cam running shutter cameras or rug shutter chips is that moving objects will be distorted if they are moving fast so it's not important if let's say you walk through the environment at walking speed um but if you have objects moving quickly moving very quickly uh like very fast driving cars this can actually turn into problems in your image creation process so just as an um as an example so um in this image here so you have in the rolling shutter what this illustration basically uh shows you which pixels are exposed at which point in time so your shutter opens let's see the first lines are exposed and basically the shutter reads out one line after the other and then you have an area where the full chip is collecting light and then it deactivates um and terminates the process of reading out the sensors or collecting allow the sensors to collect light and those go dark again so a pixel over here is exposed before a pixel over here is going to be exposed and as a result of this you basically have one row after the other that is start the read out and stop the readout it basically rolls over the image and that's what the name rolling shutter comes from the problem which you have now is if you have an object which moves through the environment for example you have this laser pointer and your camera is where the camera is sitting and you move this object quickly over the scene so very fastly moving laser pointer then this will actually lead to a distorted line so the laser pointer will be actually slanted like this it's not pictured like this just through its motion because the pixels here on top are exposed first the one down here a little bit later so with the when the laser pointer is already here and so on and so forth so by the time i'm exporting the pixel down here the laser pointer is actually moved and therefore you get those slanted lines and this is something which can be of course undesirable if you want to do 3d reconstruction of fast objects in the environment so here are two examples of images taken so one here at two cars moving at opposite speeds or the blades of an airplane motor or small airplane aircraft and they can clearly see that there are distortions here at the propellers or you can see this car is clearly not the front of how a regular car looks like so these are example images taken with a rolling shutter camera at high driving speeds or at high speeds and they can clearly see the distortions with hat which happens so if you want to do a three reconstruction of the car this will be problematic because it doesn't capture the geometry of your scene well therefore if you want to do 3d reconstruction you either have to take those effects into account and need to take into account that every row of pixels is exposed at a different point in time and take that into account appropriately which makes your mathematical underlying mathematical problems typically more complex it involves much more variables because you basically need to estimate a position where was the camera for every row of pixels separately which dramatically increases the number of variables that you need to take into account or alternatively you move to what is called a global shutter camera a global shutter camera doesn't suffer from this problem it basically exposes all pixels at the same point in time so the same start time and termination time for collecting um intensity information from those cameras and they are more more expensive to produce therefore we don't find that that often in cameras but it's the preferable option if you want to go for geometric reconstruction tasks and then in the it looks like this it the the chip is deactivated then it's activated at once and deactivated again so the whole chip is exposed through the light coming from the scene and as a result and then terminated again so all pixels have been taken at the same point in time we still may have motion blur from moving objects but we can treat all pixels alike and don't need to treat individual pixels as being exposed at individual points in time which again substantially simplifies the underlying estimation problems we can also see this effect of a global shutter and a rolling shutter if we take flashlights into account so if yeah we had a flash tower camera what the camera typically does it basically limits the um shortest exposure time to a certain value depends on kind of the quality of your camera so you can't expose images when you add a flash at very high exposure times because the flash needs to be fired when the curtain is fully open or all pixels are exposed to uh in the light collection process because the flashlight will dominate the light in the scene and if you don't do that you will actually generate dark areas in your scene so here is an example of an image being taken from a scene once with a rolling shutter camera and a global shutter camera at short exposure times using a flashlight and you can see here so this was here the synchronization time of the flash and the camera was not set correctly or overwritten manually that you can see that the camera has fired when the um rolling shutter was not completely open and so in this part of the scene this was kind of the natural light that was available in the scene and this is the light generated by the flashlight and as a result of this you have this bar if you do it with a global shutter camera you have a properly exposed image and so this is an also second effect that if you work with flashlights and using a rolling shutter camera with um versus a global shutter camera so global trader cameras are better especially for 3d reconstruction tasks but also are the more expen expensive option so that was kind of the first part and then we are looking into kind of what happens now with respect to our lens so we looked into the chip so far and the shutter and now let's look into lens and aperture and see how they affect the image creation process so we are now looking into our lens and the aperture which actually sits inside your lens and it's basically this small hole over here the size of this small hole over here that is relevant before we can dive into understanding um the lenses or what lenses are for we need to quickly or briefly dive into the question on how does light propagate through the scene and it's important for understanding why we need lenses and why it is hard to deal completely without a lens although theoretically that is possible we can go to standard pinhole but that kind of limits the use of our cameras especially given the weightable light so the important thing to note is that there are different models to describe the propagation of light through the environment so they are so-called geometric optics or ray optics where which is kind of the kind of common thing you learn in physics in school that basically light travels along a straight line and that is kind of the model that we can use not to describe most of the processes especially the three reconstruction processes we can do with geometric or ray optics and we especially the models for 3d reconstruction of our cameras with this we are totally fine um we can also see light propagation as the principle of wave optics which are based on maxwell's equations and this then you have certain effects like refraction um inference of waves um if you describe the light through waves so certain phenomena you can't describe with ray optics you have to use wave optics for describing this phenomena and there's a third model for describing the propagation of light that uses particles or quantum optics where you exploit the wave particle duality or try to model is so that light can be described by a wave or it can be described by a particle and depending in which setup i am only one of the descriptions actually valuable for us and for us this is important if we want to dive deeper into the process on how the light information on a chip is turned into an electric signal because then we basically have a photon counter running over here and if we look into the error distributions that appear in the intensity measurements we need to dive a bit deeper in here but for most of the activities that we are doing so we will very briefly go over them the ray optics or geometric optic is sufficient to describe what happens if we attach a lens for example to a camera and describe the process of how a point from the 3d volt is actually mapped onto the image scene so the geometric optics has basically four axioms axioms which are your model assumptions and based on this model assumptions you can describe how the light propagates so the first important thing is if we are in homogeneous material the light travels along a straight line so ray of light or a beam is can be described by a straight line so it's not curved or anything like this it is a beautiful straight line passing through the environment if we have two homogeneous materials and at the border of those materials the the ray of light can either be reflected or refracted so that means either we have for example a reflection so we have what here one material sits over here the other material over here a reflection that the light is reflected or refracted that it enters from one material into the other material and then it typically changes the direction so the angle at which it hits the um the boundaries of the two surfaces and this is the snell's law which you need to take into account basically how much that is depends on the material property of these homogeneous materials the second thing is the optical path is reversible so if light can travel from the right to the left it can also travel from the left to the right and um if i have two rays of light or two beams which intersect somewhere in the 3d world they don't influence each other so they can pass through each other and there's no no effect if just two light rays of light actually meet so these are the four axioms so assumptions that we make and then we can use this to describe a lot of phenomena through the environment so the propagation of light is described through a large number of rays that pass through the environment um and these are rays which travel with light speed so it's around 3 times 10 to the power of 8 meters per second so that's pretty quick this is the speed of light through an environment and here measured in vacuum and in other materials the speed of light will be reduced a little bit and this is result on how fast it travels and this is related to this theoretical speed and a material property the index of refraction this is also used to describe how the angle changes between the light passing from one material another material and so the speed is this c divided by a constant and in vacuum this constant n equals to one and otherwise it has a larger value so the speed of light actually reduces and the light always travels the fastest path so if you have two materials with different refraction indices and one is larger you can basically see that the travel through that material is shorter in terms of distance so that the light always takes kind of the fastest path through the environment and this refraction is something that is used for example inside your lenses in order to build your lenses so with those basics let's see how we can see if we can describe the process of actually generating an image so the image formation process that um is actually generated so let's take a film or a sensor here just an object which is light sensitive and we want to describe how an object from the 3d world is actually mapped to this plane so consider we have an object here like this tree and there's let's say sunlight over here which shines on this object and light from this object is reflected so for example here on the top of the tree and here on the side of the tree once illustrated by red rays and one by blue rays and so we have the sun over here it hits this three over here and this is the different rays of light which are reflected from the sun to the top of the tree and then onto my film or here on the side of the tree so what happens is that the reflections will be mapped onto our film or chip basically everywhere so the question is do we actually get a reasonable image for something that we can recognize over here and the answer unfortunately is no this system would not work because all the light which is reflected from the tree is reflected everywhere on the chip so we just have basically an average intensity value from all the light that is reflected from the world onto this chip will be seen here in those pixels and there's no structure or not tree that i can actually recognize so this doesn't work so what i need to do is i need to avoid that these rays from one point will be mapped to multiple points on the film it should be just constrained that i basically just pick one of those rays and avoid that rays from um from one point are mapped to multiple points on my chip or at least when i locally constrain them and i can actually do this by adding a barrier over here with a small hole in between so what i'm doing i can add a barrier with a tiny hole a small pinhole over here so that all the light which hits the barrier is not considered and only the light which passes through this small point will be actually mapped onto the film and will actually read reach my camera and this is reduces the amount of light which actually reaches my chip so just a very small fraction of the light from the scene it actually reaches my film or reaches my my sensor but this small opening or also called aperture reduces the locations where one point from the 3d world is mapped onto the image plane so maybe we have a second ray which is very very similar to this one it will be mapped to a very nearby location over here but an object sitting over here or down here cannot be mapped to the same location so through by the through introducing this small um this small hole we are actually restricting where rays can be reflected to on our film and this allows us to then generate an image because every point in that world here sitting for example on the surface of the tree will be mapped to a specific location on my film and then i can actually see the structure reappearing and the question how's actually now how the tree look like here on that side we can see here the top of the tree will be actually mapped to the bottom of the image and the center approximately stays the center and the trunk of the tree would be mapped here on top so the object is actually flipped on sits upside down on my phone we can actually illustrate this here in this process so this is the real object in the 3d world this is my camera this is the pinhole over here this is my image plane then we can see that um this candle over here is mapped through the projection center which you see described by the pinhole and so the candle will be sitting here upside down we can actually what we can do at least mentally of course not in reality we can take this as a rotation center and rotate this image around to sit over here as a virtual image because then it's standing upside down this is a virtual image that we are often using in order to describe the the image process or can be used for described imaging process so that it's standing upside down which is basically a rotation of this plane around that pinhole but this is of course not the real image it's just a virtual image for us so that um certain considerations become easier or more natural for us as humans and we don't have to mentally rotate the image that is generated but inside your camera you actually have that rotation so the images is shown upside down or your uh your film is actually exposed the in upside down fashion so you need to rotate that around okay so this principle of the pinhole camera is also called a camera obscura and this is a concept which is around for a very long time actually longer than um than we have films or even digital sensors it's a concept which has first been documented in 1544 and the name camera obscura actually means the dark room um and it's the result that you when you're in a dark room you have a small hole inside your wall you could actually see what happens outside because it will be projected onto the other side of the wall and this is something that you can use in order to for example or was used to observe the sun because you don't want to look directly into the sun because you're ruining your eyes and you won't see actually a lot because it's too bright but by projecting the sun um onto a large wall you could actually observe the sun using the so-called camera obscura and so this is a concept which is around since a very long time nearly 500 years um and has been used to actually create images or observe the world and you can actually build your camera obscura at home very easily so if you have your room and you have blinds which are fully dark what you then can do is you can actually um close your blinds and make a very small hole into your um into your blinds and then once your eyes get adjusted to the complete darkness you can actually see the scene of the outside world being projected on the other side of the wall so this is a photo actually taken from exactly that scene so you can see the door and the light switch so it's the projection of the world outside onto the onto the wall through a camera screw and then taking a another camera or real camera to actually taking a picture of that scene so something you can create at home it however takes a while until your eyes get used to the darkness so that you can actually see that very well and this is the basic pinhole camera camera obscura and the pinhole camera is actually a pretty good mathematical model and one that we are using a lot in the context of 3d reconstruction basically due to its simplicity so what are what do we need to know in order to describe the pinhole model so again we have our barrier over here we have our 3d world outside here and this is basically rolled inside my camera and of course in reality that would be 3d now we are projecting this here on the plane so we just get a one-dimensional image so to say just to make it easier to describe this so we have a point in the world and everything which is written here in capitalized characters is in the real world and lowercase characters sits inside my camera so we have that point here which has an x and a z location over here y should be zero in this example and then this point is mapped through the projection center typically called o onto my image point so my image plane sits over here and there's a fixed plane and then basically that coordinate of this point will they will be all sitting on the plane so this one becomes it's not relevant anymore and i only have that x coordinate so this is a loss of the depth information that i have so to say um because this is inside my camera this length small lowercase z is fixed so this is basically the distance from your image plane to the um to the barrier to the uh to your pinhole so this is my distance that and this distance capitals that is basically how far is the point in the way in the world and the ratio between the lowercase that and the uppercase set called the image scale over here describes or can be used to describe how a point from the 3d volt is mapped onto the 2d image plane namely by using the coordinate of the point in the 3d world so the x coordinate of that point over here multiplying it with this image scale so lowercase that divided by uppercase that and the negative sign and then we get this vector x telling us where this point is located assuming that this is kind of the zero coordinate over here so it's a very simple mapping that can be described with the theorem of intercepting rays that can be exploited over here through those triangles that i have and um this is kind of the very simple model to explain how point from the 3d world is actually mapped to an image plane and again for any point in the 3d ball there's that an x-coordinate will differ here in the image plane this z is typically constant and so then this x is a result of a combination of this uppercase x and uppercase that so how tall is an object that's described by x and how far is the object away this is a set coordinate and this will both determine what's the size of the object in the scene and this is something that you know from your regular camera if you're going closer to an object it is larger in your image if you go further away the object gets smaller in your image and this is actually the result that can or the the size of this scaling can actually be described using this um pinhole camera model so um the problem that i have with this pinhole camera model however is that this pinhole must be very small so that this process work well because if i increase the size of my pinhole multiple rays also rays reflected from points nearby this point x will be mapped to the same location so this means that my image gets blurry and so the smaller my pinhole the better the sharper the image that i get but the longer i need to wait so that enough light actually travels through this tiny hole and reaches my film and if i increase the panel everything is faster and but the problem also gets more blurry and that's a problem that we have if we use a pinhole camera camera obscura in reality so the small hole generates sharp images but requires long exposure times and a large hole allows me to have short exposure times but generates blurry images and often that's not what i want i actually would like to reduce my exposure time to picture fast objects to make the process faster overall but still have sharp images and in order to fix this what we can do we can introduce a lens so-called thin lens so we are basically replacing this pinhole with a large opening and fit a lens into this large opening and as a result of this if i have a point x over here i need to set up the lens in the specific way that all the rays that are reflected from this point x should there be so this this would be the original ray that would pass through this tiny pinhole so now also raise from that point that gets moved over here on the lens need to be redirected through that lens so that they intersect at the image plane at exactly the same point so that multiple rays of light shoot into different directions from this point x will be actually mapped to the same point so what the lens basically does it takes more than just the single ray that would pass through the pinhole it takes more rays into account that are reflected from the same object in the 3d world from this point x and make sure that they need to map onto this lowercase x point and this is what the syn lenses are actually doing and you can actually describe with the law of the lenses the relation of how far the object needs to be away the distance from the center of the lens to the um to your image plane and your focal lengths you can set them in relation and we don't need to dive actually too much into the details how the lens works because for most of the things that we're doing here in the course um the pinhole model of this just using this pinhole at the mathematical model is sufficient and we don't need to dive dive very deep into this process we only um need to understand that a little bit in order to understand um how we can actually control if we take an image how we should control certain properties in our camera in order to get the correct images it should however be noted that this camera using a thin lens is only an approximation of a pinhole it's not precisely a pinhole camera anymore because in the pinhole camera model one of the central assumption was that all rays which come from the 3d world actually pass with single point and this is clearly not the case that i have over here um that because this ray over here and this ray over here they coming from the same point are mapped to the same point but they don't pass with the same single point through my projection center in my camera so it's important to know that the um the camera using lenses or thin lens is only an approximation of the pinhole that means i'm actually doing mistakes in this process so as i said the point on the object and in the image and the center of the lens should lie on a on one single line this is not necessarily the case um the other effect that we have is the further away um a beam pass it through the lens so the further away from the optical axis the larger the arrow actually gets sorry if i go back so the further i'm out here the further this error actually um actually gets and um what we basically do then is again we introduce an artificial pinhole and aperture and to reduce which part of the lens is being exposed so if i go back to my lens i'm basically again introducing a barrier over here which tells me or over here doesn't matter which part of the so how big is the home how many of those rays should have that path through um so they are projected into my point and the aperture is basically again a physical barrier that i'm introducing over here so in the end we will be working with the pinhole camera model for the majority of 3d reconstruction tasks it's a commonly used model for modeling a camera and it's so simple that it's still powerful that it became so popular but to always be noted it's not a perfect description because we have lenses in our cameras and there are certain setups where it becomes completely unsuitable especially if you think about cameras which have a large field of view so if you think about a fish eye lens which has a 180 degree opening angle you can just see by just drawing that that it's impossible to have a pinhole and an image plane and there should be all rays with the 180 degree opening and you're pathing through this point onto the image plane so you you don't have you can't draw a straight line that realizes or even fish eye lenses which have more than 100 degree opening angle so especially for wide angle lenses the pinhole camera model is not going to work but for normal lenses that we are often using in 3d reconstruction the pinhole model actually does a good job and the key assumptions are three main assumptions made in the pinhole camera model or for thin lenses although then just knowing there is an approximation is that all rays from an object for a point in the 3d world intersect inside the camera in a single point all image points lie on a plane so this image plane that i have over here is a plane it's not some curved surface or something like this and all rays from the object point uh to the image point is a straight line so the the straight line from a point in the 3d world onto the image plane is straight line these are the assumptions that we do in our pinhole model and that should not be violated and because otherwise my three reconstruction tasks will actually not be correct so we said in those lens setups we need to limit the aperture in some situations in order to reduce the amount of light that reaches our film we can use this to control the exposure so we get the proper brightness of the image and we also want to maybe want to reduce it in order to actually get sharp images so the aperture is actually typically looks like this so it's again a curtain kind of circular one which can be closed or open so depending on your on your aperture here the aperture is wide open here it is closed or not not fully closed but a very small hole and this is basically then in this case only a small amount of light is allowed to pass through that point and you typically describe them with this kind of f-stop number that you see down here so the larger the number the smaller your whole actually gets and what do those numbers all these f-stops basically mean it basically means if you go up an f-stop or down f-stop you double the amount of light that reaches your camera or you reduce it by half or if you go two f-stops f stops up then you have um four times more like i'm sorry that's incorrect on that slide this should be half this should be a quarter and this should be a factor of two because the larger the the opening uh the more light actually passes through so this should be two and a half and a quarter i'm sorry for that um so this well the aperture is basically the pinhole size and basically control controls how much light reaches your sensor and allows you to limit the distance under which a light ray that passes through your lens um actually reaches your chip so you cons the smaller you make the pinhole you more constrain those uh rays to be near the projection centers of the center of the optical axis and so you're basically introducing an additional barrier over here and so limiting the rays of light that pass through the outer parts of your lens so the aperture reduces the um the error that that year that we're doing here um and in this in this image processing so in the process of using a thin lens we reduce the uh the error introduced by not having an actual pinhole but you can also see the effect of the aperture on your image generation process and this is something which we call depth of field so depending on the opening of your aperture so a large open aperture will let more light into the theme but what it will actually generate is a smaller depth of field that means you focus on a certain point and how much of the image is sharp and how much of the image is blurry so basically if your aperture is wide open it basically means your depth of field is small that means you focus on a certain point and everything which lies behind that point or in front of that point is more likely to be blurry and um if you close your aperture so towards a actual pinhole then the whole scene will be sharp so your depth of field will increase so here you can see for example in this on this photo here the focus point lies somewhere over here and it's sharp everything is sharp over here the objects here in front are very blurry like the knives and also very blurry in the back and again you can use this as a style element if you take photos um but if you want to use your photos as a measurement device and you are interested in having crystal clear and sharp images and you would like to avoid having very blurry images and just again an example three times the same object just recorded with different f-stop numbers so here is a smaller pinhole size and here increasing the size of the pinhole and you can see objects are much more blurry so here for example you can still recognize that this is a lamp post which is already impossible to recognize over here so you can actually see the difference in the process um so how the aperture controls your depth of field and again this is not that relevant for us in photogrammetry we are just trying to make the aperture small to get sharp images but if you're taking photos as an art form then you can explicitly use the depth of field in order to control how your images should actually look like and then i would like to add a few final words about a lens um so why do we have a lens we said we need that lens in order to get more light on in the same amount of time onto our chip or on our form um but the goal for the lens is to not lead to any distortions to generate sharp images within high contrast and um so that's kind of the goals that we always want to have and but we need to choose a lens and the lens is always a compromise out of different factors it's kind of first the field of view so do we have a kind of a zoom camera which wants to increase the objects which are very far away or wide angle camera which tries to capture as much as possible from our scene so that's kind of the field of view or you the opening angle of your camera that again is related to the distance of the object so how far is the object away that you actually want a picture and how much light do we have available if we are in very low light scenes we want to increase our apartment have four lens which allows us to open the departure uh very wide uh the open to open very widely in order to collect more light and deal with the available light and also price pays rolls so the lenses are probably the objects in your camera which are the most expensive ones if you want to invest into a good lens um price doesn't play that much of a role here but we i just want to go quickly over the effect that we see with different lenses so we have from a tiller lens one with a with a very long focal length two normal lens around 50 millimeters related to the full frame camera 35 wide angle lens and then fish eye lenses which actually open up the field of view dramatically even over 180 degrees so the question is how do those lenses actually look like so if you look to a moderate tiller lens it's a very narrow field of view you can see here a picture of that building parallel lines remain parallel so you can see this is a these are parallel lines in the 3d world and they stay more or less parallel in the world not perfectly so this one over here and this one over here not perfectly parallel but they stay more or less parallel and we have minimal perspective distortions through that line so that's something that is that is actually good something that i typically want to have but the narrow field of view can be limiting if i want to picture the whole scene around me if i need to increase my field of view because i want to observe more of the scene at the same point in time i need to go to a wide angle lens and these are cameras let's say with an opening angle of something like 70 to 120 degrees and then the straight lines in the world are only very roughly mapped to straight lines so we're not talking about parallel lines just straight lines so straight lines are those lines over here that you can see all those buildings they are still roughly straight or more or less straight but parallel lines do not remain parallel anymore so we see distortions over here and also the proportions of those objects are not correctly displayed anymore there's an effect of these larger perspective distortions through this wide opening angle and we can even take that to the extreme and go for a fisheye lens so them is very wide open and here even straight lines are not straight lines anymore so you can see the horizon line over here on the city picture so this is not the curvature of the earth it's just the distortions that we get through the choice of our lens this high wide angle lens with more than 180 degrees opening angle can even go even over 180 degrees opening angle then we get these types of distortions and straight lines are not straight lines anymore in our world this is something that we need to be aware of if we use one of those cameras then certain assumptions that we make about our camera model such as our pinhole model will not hold anymore [Music] so all these assumptions that we made in our for our pinhole camera they will actually be violated when i'm adding lenses especially wide angle lenses i need to be aware of this that if i'm using wide angle lenses then my pinhole model will break down it can actually really break down mathematically or just the approximation error that are is actually generated through the pinhole model is so huge that it doesn't actually lead to good um approximations uh anymore that i can use nordoff in order to do 3d reconstruction so that's something that i need to take into account when i work with lenses the land different lenses also have aberrations or lead to distortions and deviations from the ideal mapping of the lens with respect to the pinhole camera models and there are different types of aberrations or mistakes kind of that the lens is doing with respect to the ideal imaging process and there is a large number of different types of aberrations that i can try to compensate for um some are very common and should be taken into account and others are less critical for 3d reconstruction tasks so typical things are barrel distortion or pincushion distortions that you see especially if you go to wide-angle lenses you see effects like this popping up and or even combinations out of those both distortions and that's something that you typically need to take into account and calibrate your camera because we know that your lens in a certain setup has this form of distortion you can of course compensate for this distortion basically by shifting those points over here more to the optical center or here moving points further away these are tricks that you can actually do in order to um you know to compensate for that uh and perform kind of a camera calibration calibrate your lens in order to generate images where this effect is not visible anymore um also professional or semi-professional digital cameras today if you buy the lens from the same producer than your camera body they will actually allow you to perform a calibration directly inside your camera so that the lens manufacturer knows or the camera manufacturer knows about the distortions of the lens and can directly compensate that by shifting the individual pixels around in your in your image actually there are different types of other types of aberrations which are harder to compensate so the spherical aberration this is something you can actually control through your aperture and this basically means that the further you go to the outside of your lens so the further you move away from the optical axis um your uh focal point over here may not be the same so that's how it should look like in in the ideal case with a perfect lens but lenses are not perfect from the production processes so you can see that not all those rays intersect in a single point and this will lead to mistakes in your imaging process and one way you can actually compensate for this is actually adding an apartment so basically deactivating the outer rays over here and in this case produce better images then beside the spherical aberration there's also the chromatic aberration which is a result that the that light that comes at different wavelengths so for example blue light red light or green light is refracted slightly differently so especially at the outside parts of your lens you can see that the refraction of the of the blue light is stronger than of the red light and as a result of this they won't intersect in the same point that means different colors are differently refracted this is only a little bit this effect is not very strong but if you zoom in you can actually see especially in the outside areas this chromatic aberration popping up where you see you basically have add boundaries different colors being mapped not exactly in the same pixel but on neighboring pixels and you have this kind of slide color around certain objects this was called a chromatic aberration um that is another form of distortions there are other things like astigmatism which some of you may know if you have you need glasses and suffering from astigmatism this is basically having a different focal point in the vertical and the horizontal direction so this is an effect of your lens that in one direction you have a different focal length another length so you can't really focus on this and you basically need to compensate for that by having a second lens in front of your the lens of your eye which actually compensates for those effects so that here kind of um we end up having the same point over here for the horizontal and in the vertical direction and other things like chromatic aberration or the kumar that is basically if your rays come from a certain side directions or not parallel to the optical axis onto your lens then they are not mapped appropriately on the same point here and you basically have those ring effects that you sometimes see um but these are again mistakes which are a little bit harder to pinpoint so a circle is not mapped into a circle but in kind of this like a droplet shape that is an effect that you sometimes also see or vignetting basically means there's more light in the center of the image and if the image gets darker to the outside this is something which you can actually compensate for by just brightening up the intensity values in the outside or making it darker in the inside if you if you know about the mediating effect of your camera even today you see this sometimes as an artistic form of putting focus to the object which are in the center of your image by adding and vignetting artificially in the end to your image just as a process of steering the eye of the viewer again that's can be a style element in photography and photogrammetry we are typically interested in actually getting rid of those effects so that's it from my side for the first part of this lecture on the camera basics in the next lecture we will continue with the propagation of light and look into wave optics and particle optics and reflections as a short part of that course over here and so this now brings me to the end of dealing with lenses and i now want to quickly dive into wave optics and particle optics or quantum optics and in order to describe a few things that we are exploiting when working here with our cameras so starting with wave optics what sits behind wave optics again we have discussed the geometric optics what is what means wave optics that means light is considered as an electromagnetic wave that can be described by max by the maxwell equations and then with this kind of treating light as wave you can describe effects like inference and diffraction um or influence and boiling in german that i think are things that you probably have seen in your physics practicals describing how the the light behaves for example at small slits and and certain effects that then come if you treat light as well if you can describe certain phenomena that you cannot describe with the ray optics our visible light spectrum is in the area of around 400 to 700 nanometers and this is just a very very small part of the spectrum of electromagnetic waves and the question is how can we use um the or how can we use not just sensors which capture the visible light but also the non-visible light in order to obtain for example more information about objects and taking them into account into image interpretation processes and if you have the spectrum of electromagnetic waves from gamma rays x-ray ultraviolet radiation then we have the visible light spectrum over here infrared microwave radio waves um so this is the whole spectrum of electromagnetic waves and the visible spectrum is just a very very small tiny part of that spectrum in the area in which we actually see and there are different applications where you want to extend that spectrum so for example near infrared photography is often used depending on your application where you take additional information into account or microwave information to infer certain properties about the earth there this becomes relevant and you are looking explicitly outside the visible spectrum for generating information or obtaining information about the environment there we have to the key property is the the frequency so the frequency is the speed of light divided by the wavelength so the wavelength was what we had been written over here 400 around 400 to 700 nanometers is the wavelength of the visible spectrum and this then creates with these combined with the speed of light basically frequency of the wave that we are taking into account and it also depends the wave depends also on the on the fraction index of the material you're operating in so this would hold for vacuum this changes as the speed of light changes in the different materials i just want to say that here in this course we are focusing only on regular images we are not using any other wavelengths taking them into account so most of the photography business takes red green and blue into accounts around 650 nanometers 550 nanometers and 450 nanometers at least approximately and we use this to turn them into our color images as we use them and that's typically the images that we are using here again depending on special applications that you have you may want to go to into different spectra even outside the visible spectrum for us humans to acquire additional information that are also not accessible to our human eyes if this is a property that's important for you one example is the near infrared spectrum so this is around 1 000 nanometers or 1 millimeter in terms of wavelengths and there you have an effect that this area of the visible spectrum is strongly reflected from chlorophyll and chlorophyll appears in vegetation or healthy vegetation and therefore it's highly highly reflective so you have here for example a photo of the same scene once with a regular camera you can see this rather dark green tree over here which becomes super bright same hose for the for the grass area over here if i take the picture in the near infrared spectrum because it's highly highly reflective um [Music] in this in the spectrum of the near infrared radiation you can also see here that the wrong color image is where the red channel is replaced with the with the near infrared channel and you can see that here all the typically green vegetation pops up in extreme red and this is something which can be relevant for you if you're actually monitoring vegetation because it for example allows you to very easily distinguish which part of the scene is healthy vegetation and which is not so depending on your application you want to may take other wavelengths into account you can actually put this to the extreme to go from having multiple different spectrum to what's called hyperspectral for multi-spectral imaging to hyperspectral imaging where you basically have a whole cube for example with 200 different wavelengths that you need to take into account so your image basically turns into you can see your image already the rgb image is a 3d structure with an x dimension y dimension and basically three channels in the or three discrete steps in the third dimension here your wavelength becomes actually larger and you have you so called higher dimensional data cubes um where you have three dimensions also the wavelength dimension is then a full dimension for example covering 200 different frequencies and then of course those image data becomes much more difficult or much more challenging to process but also the process of actually generating those images is is typically different so if you have that many wavelengths you typically have line sensors and not sensors which measure whole area all at once and then all the struggles from the rolling shutter system actually come into the game again and aligning those data cubes also becomes more challenging but something that we are not explicitly doing here in that course today or in this photogrammetry one course so we're not looking into hyperspectral sensing here i also want to briefly touch quantum optics or particle optics here the key idea is to describe light by particles and it's a way where you can exploit the particle wave duality using uh quantum because they're so-called quantums of light um the reason for introducing this or there was a law for a long time in the past a debate about if light should be described by waves or by particles there are two interpretations there are certain phenomena that you can describe very well with the waves and which can't be described by particles but you also have other phenomena that are actually described all by particles and so you have this duality that certain things can be described with waves and certain things can be described with particles and the quantum optics here is the way to describe the things that or describe the phenomena that cannot be explained just using wave optics and the interaction it also is very useful especially for us here because it allows us to describe the interaction of light and matter and this becomes important when in our chips when we receive light and turn the light into electric energy or into a current we then measure because this tells us how much light has reached that chip and actually this process of describing why their current is coming up from this light sensitive chip the reason is because the uh the quantums of light actually have reached that chip and turn into a current and for describing this is actually a very useful process so um it's based on the idea of the photon so photon is the elementary particle and describe the kind of smallest quantity of light through the quantum of light and you basically when you white reaches you these are actually multiples of these quantum of light so you have quanta of light and that are actually reaching your and has this quantum of light has a certain energy q is h times nu where um [Music] the nu is the frequency and h is the planck constant which is given here so it's a very small number uh roughly six to times ten to the power of minus 34 so the energy that one quantum of light covers one photon or brings with it is an extremely small number that means in the current let's say sun radiation that we are exposed to there are a huge number of those uh quantum speakers otherwise the energy that we would get out here would be far too small why is this idea useful why is this helpful for us because this allows us to describe how photons are turned through the chip into intensity values because they generate an electric charge on that chip and this can be explained by saying yes it's a particle which has a certain energy and this then can be or allows us to describe the process how we can measure intensity values through the amount of light that reaches a certain area on our chip so we can see each cell on that chip so each pixel basically has a photon counter so every pixel is a photon counter and it tells us how many of these quantile of light how many photons reach that certain area on the chip through the pinhole within the exposure time so we're saying okay how much light travels through the environment through the small pinhole and reaches the chip during the exposure time and so basically every pixel is a photon counter and since every pixel as we said in the beginning is associated with the direction it basically says how many photons reach that chip from that specific location and the more photons i have the larger the intensity values that the pixel is actually reporting a few more words how we can actually control those intensity values so there are of course external factors to generate more photons that reach my camera that means bringing light sources into the scene so the more light i put into the scene the more light is reflected from the object onto my camera and the more photons are actually generated and um typically i need to have a certain number of photons not to get good intensity measurements onto my camera but i can also these are kind of the external factors so what i can modify the scene to generate more photons reaching my chip in a given time frame or i can play with the parameters of my camera so i can vary the exposure time typically abbreviated as t v on your camera um and this is basically the time how long you open your chip and allow the camera or the individual pixels to actually collect photons this of course determines it if you let your um exposure time open twice as long you will receive twice as many photons the aperture or the pinhole size typically a previous av is of course an information which is important the smaller you make that pinhole the smaller the number of photons that will actually reach your chip or something we haven't just talked about yet is the sensitivity of your chip is to be called the iso iso value um of your chip which basically defines the sensitivity so the mapping from photon counts actually to intensity values and you can set that up or set that down to be the lower you set it the better the quality of your image gets and the larger you set your iso value the more noise from the sensor you are actually um generating that is generated onto the chip itself so that's typically what's called often referred to as the exposure triangle so you can actually play around with certain things you can vary your shutter speed from a long exposure time to a short exposure time to high shutter speed low shutter speed you can vary your brightness by changing your iso values or low sensitivity of your chip and high sensitivity of your chip and you can open or close your aperture which allows the amount of light that you're adding and basically if you move in this direction you always get brighter images so increasing the iso leads to brighter images leaving the shutter open for a longer period of time collects more light creates larger images and opening your patra creates larger images but with doing this all side effects come into the game so if you increase the iso you will get more noise into your images or in the old film time this was the larger grain that you had on your film um which basically was a change in the ice or value this can be again sometimes a desired property for artistic purposes but often you actually when you want to take measurements you don't want to have a larger screen if you make your leave your shutter open for longer periods of time you collect more light but you actually generate motion blur which is something that you don't want if you especially if you work with moving objects and if you increase your aperture then you let more light into your camera which increases some of the um errors or the spheric aberration increases you get more blurry images and you're violating the pinhole assumption more and more and the other effect is is the depth of field so you're reducing your depth of field if you open your aperture so by playing around with this triangle you get an idea on what are the external factors that you can actually control in order to get properly exposed images and again as i said lighting plays a role so a few words about lighting and the reflectivity of objects in the scene so the lighting itself is essential i said before you can add a light source to your scene and then you have more light available that you can exploit but the light intensity that you actually register on your chip depends also on the material it depends on the light source but it also depends on the on the on the material that you have and on the on the surface on the properties of that surface this can be the location orientation of that surface but it can also be the material and this affects how much of that light is reflected into a certain direction that it passes through your pinhole or your lens and reaches your scene and you can actually try to describe that and kind of what's the reflectivity of these objects over here and the different ways how i can describe the reflectivity of objects the fairly simple form is so called albedo and this measures the diffuse reflection of the of solar radiation of different objects and we can see different objects here like ice snow desert uh clouds and they tell you how much of the light is being reflected it has values between zero and one or zero percent and hundred percent which is the same um where basically means 100 that the full radiation everything is reflected and uh zero means basically it's a black body nothing is reflected from those objects and it gives you an idea on how different um objects will reflect the um the the radiation that you're getting so water for example is fairly low numbers here something between five and ten um so a fairly small amount is actually reflected by water and this is something that you may take into account for example in remote sensing this is an important information if you look into the reflectivity of of physical objects that we have we sometimes wanna be um more precise than just having one value for a certain object and then the so-called brdfs or bi-directional reflection distribution functions come into the game and these are it's a general model for light scattering and it depends on the geometry of the surface and the wavelengths and basically describes the material and basically says if i have an incoming ray so the two angles of the incoming ray onto the surface and the directions in which the ray is being reflected and the wavelengths how much of the light is actually being reflected and this is what this function tells so it tells us how much light of each wavelength arriving at an incident direction described for those two angles is emitted in this outgoing direction r over here and so by knowing the incoming direction by knowing the outcoming direction or that's the one that you typically query and the wavelengths you can actually then for example virtually render a material for which you know this brdf so the brdf is a property of the material and the and the surface or how the material uh material of that surface that you can then describe for example to render them so you have these these two angles for the incoming direction those two over here and the outgoing direction with respect to your um your normal and then you can actually describe how much light is emitted into a certain direction or reflected direction and then if you want to then take the amount of light into account then you have the basically the incoming light you have an integral um so the reflected light is the incoming light um you have this brdf function over here and then um your the incident angle um that you need to take into account of course only in one direction because you don't want to look underneath the surface and then you can solve this integral and actually get the amount of reflect light out here and um so this is so this is something which is independent of the incoming light it only depends on the geometry and the wavelengths and then you can combine this with the incoming light and this integral and then to estimate how much light is actually reflected into the direction you are interested in and then you can for example estimate how much light will reach your camera given this object just brought two examples over here of this brdf estimation so this was a project done by mark crossman and luke van gaal where you have an object here this is a marble surface where they estimated the vrdf properties and here basically different color values tell you the or illustrate different properties of this brdf function so you can see that they're different material or different parts of that object have different materials and then you can actually use this information to then render those objects so this is not a real image it's a rendering of the image where you can actually see the marble and the reflections over here that are being generated in a three reconstruction then in a rendering of this three reconstruction so that you actually get surface properties out that are realistic and are in line on what they should look like and you can do this by estimating this brdf and then using this brdfs again to render these objects over here so this was an example on how we need to take into account which incoming light we have in which direction light is reflected so that we and this depends on the surface what material it is and what's the orientation of that surface and then this is the amount of light which reaches our camera and then this is the thing that we actually see our observations and this information that we then can take into account for for example modeling of three reconstruction tasks and this actually brings me to the end of the lecture today so i introduced the basic elements of a regular camera this can be a digital camera which you looked into which are the cameras that we are obviously using but it's actually not that much different from a film camera except that the sensor the digital sensor is replaced by a film uh which is light sensitive and then the position of the film corresponds to the position of the pixels or the location on your chip but there's not that much of a difference in there we described what a camera actually measures and what impacts the intensity measurement that the camera is actually producing and for that we looked into the different physical models that we can use to describe how light travel through the environment so ray optics wave optics and quantum optics or particle optics and we will stick with the ray optics for all the considerations that we're doing here in that course you need to know that there are other things exist and you may want to exploit this but these are basically side notes here for the photogrammetry one and two course at least we introduced the pinhole camera model um as a way for describing cameras and an approximation of a thin lens and we will stick with this pinhole camera model throughout this course because it's a very simple model and and as long as we don't work with for example very wide angle cameras it actually does a good job and we can do a lot of things with these pinhole camera models we however need to take into account that they're always mistakes the world is not perfect and all lenses are not perfect we have seen different types of aberrations some of them we must take into account and consider them especially for the three reconstruction tasks these are this pincushion distortion barrel distortion for example others are less relevant for at least 3d reconstruction tasks in here and we also looked in how to how objects reflect the light so that by now you should have an idea how a camera works what are the key properties of a camera which parameters can i change and what does it mean for my resulting image and here we have the mixture of photogrammetry and photography actually and so that you have an idea how the cameras work so we can use them as a tool as a measurement device for our further tasks so with this thank you very much for your attention and see you in the next lecture thank you

Channel: Cyrill Stachniss

Views: 2,859

Keywords: robotics, photogrammetry, computer vision, lecture, bonn, StachnissLab

Id: 2KR-b2Fmjxk

Length: 91min 16sec (5476 seconds)

Published: Thu Apr 01 2021