Image Processing Made Easy - Previous Version

hello and welcome to the image processing made easy webinar my name is Andy thay and I'm a Technical Marketing Manager here at MathWorks image and video processing is an extremely popular field and you see it being integrated into a lot of different application spaces anything from medical to robotics automotive machine vision the list is pretty endless unfortunately with all this integration comes a lot of challenges a lot of people are new to image processing and therefore we get a lot of questions such as how do I deal with all these different file formats how do I create and actually test my algorithms how do I visualize and maybe find the cause of the of the problems and then finally how do I deal with large images and speed up the execution of my algorithms so in this webinar we're going to go over the first four topics you see here so the goal of this webinar really is we hope to equip you with some common strategies that you can apply to very typical image and video processing workflows and with that if you see this image below you see this is just some simple toys and some candy we're going to use this image throughout many of our examples here and I'll have about three different major examples to walk you through but using this simple image we're going to use several processing techniques and by the end of it you'll be able to see on the right hand side how we can greatly simplify the image and really segment it and be able to do maybe some statistical analysis to get some details out of the image so the agenda for today is we're going to spend some time going over an introduction will go through image enhancement we'll do some image segmentation and then we'll do image registration and then we'll have a quick summary let's go over a typical image and video processing workflow of course in the beginning you're going to want to acquire the data that's what we call access so this can come in the form of files like JPEG images come in form of output from other software and of course from hardware such as a camera so that's very common to capture live images and video and you want to process those on-the-fly once you acquire your data of course you're going to want to explore it also do some discovery find out you know what you need to do to create that algorithm so a lot of time is going to be spent here actually making your application and your algorithms and finally you're going to share it so you'll share with colleagues either in the form of documentation maybe it's a form of libraries that they're going to consume and it could be consumed for third-party software that you might use to design an end product for example a camcorder or maybe some sort of automotive vision application so although I've showed a single streamlined path the reality is this is a really an iterative path and so once you acquire data you're probably going to develop your algorithms and then you're going to come back and acquire some more increase your data sets to really solidify and validate your algorithms once that's developed you're going to share it of course get feedback and you're probably going to come back and iterate on it to refine your application so this is really needs to be automated with MATLAB you'll find out that MATLAB is not only a language but it's also a scripting language so you'll be able to automate your entire process what we'll be using throughout all these demonstrations is in addition to MATLAB is the image processing toolbox so there's a lot of functions there's over 200 functions at your disposal but at a high level you'll be able to do things like displaying and exploring the images will do some enhancement some image analysis morphological operators which is shape based processing or shape based filtering image registration geometric transformations and region based processing and in these demos and what we're doing this webinar will cover many of these topics highlighted in blue and so this brings us to our first topic what is image enhancement so simply this is the process of adjusting an image so that the results are more suitable not only for display but also for further processing so as you see down in the cameraman image you can do such things as deep blurring or you can do some filtering or otherwise known as pre-processing that can be a preliminary step for you to do more enhance processing or more advanced processing later so that takes us to our first demonstration where we're at you're going to take a Landsat image so this is a satellite image of Paris and you see on the left-hand side there's definitely some issues here we see you know a lot of purple not a lot of contrast on the right hand side is what we want to drive towards so the goal really is to import this this map of Paris we're going to correct for the poor contrast and the unbalanced colors and make the image hopefully a lot more useable so here's MATLAB and just so it's not overwhelming I'm only going to cover the aspects of MATLAB or features as I come across them so let's go ahead before I get into the image enhancement demonstration I'm just going to show you some of the basic features so here we have our toy and candy image right here it's a JPEG file I can import that into the environment as simply I equals I am read and this model complete and then there and so what you'll see is I created this variable I I imported the image and you can see here it's inside of the workspace so this is where all my variables are stored and you'll see it's about a 24 hundred by 2800 pixel the image and there's three layers of it and the reason why they're three layers is there's a red a blue and a green layer so all three colors and so we can view this image just to verify it's there if we click on it you can go to plots and here the different plotting tools you can use and so we can simply do an IM show and here we have the image so here the toys as well as the candy and you can see at a command line what was entered so if you prefer to operate things at a command line you have that option as well so another tool I'm going to bring up in the closest window is just using the IM tool function so using that same variable I'm going to click on iam tool and you'll see here it entered in I am tool with variable I and it looks like a similar window except this is one of the nice apps that are within the image processing toolbox and what that gives you access to is things like let's say we want to measure distance so I can click on this ruler and I can simply connect and draw these distance lines so that's about 700 pixels long that car about 260 seven pixels wide how how bigger these pieces of candy so this type of pixel information and measurement is very helpful and we'll play a little bit with this later as we do some so maybe a shape based processing so just to kind of give you a little foreshadowing of what we'll be doing later and also you can do such things as some more detailed analysis so here's the button to inspect pixel values if I click on that what you'll see let me move this over is this little crosshair appeared and here is a zoom in of what that crosshair is pointing at we actually see the pixel values on this little hex bug and this is showing us the RGB values or intensity values so I'm going to go ahead and zoom in here so you can see more or less what's going on and you'll see as I drag around this crosshair as I can ex inspect the pixels of where that crosshairs are pointing so here you can see RGB values are very low because this piece of candy is close to essentially at the color black so when you move to the desktop here's closer to white that's why the RGB values are a lot higher and then the hex bug is like going to shade of turquoise and here's basically the RGB values for that color so again this is just a really nice way of exploring images and really giving you some maybe strategies that you may want to tackle later based on maybe some shape based processing and you see these colors here we're going to actually be doing some color based processing as well okay and so if we go to our first one this is an example that you'll actually find inside of the product and what we're dealing with here is actually multispectral data so this is a Landsat image and if I scroll down a little bit you'll see how I'll be breeding the image in because this is multiband data we're going to use a function instead of I am Reed called multiband Reed and to pull up help on anything you might want inside of MATLAB you can hit f1 and simply it'll bring up the help window and you can see this is to read banded interleaved data from a binary file so here so tells you different parameters a description showing you the different bands here in a nice graphic and and here is the syntax and telling you what are the different parameters and and even giving you an example so a lot of different ways to get to help one is just by clicking f1 and of course you can click inside of the documentation or search bar right here as well okay but let's go ahead and run this code so here we are right here at the read I'm going to just run and here we say run in advance the next layer so the here the advance button so this is what's really nice about MATLAB as well as I can run what's called sections of code so every time you see these double percent signs that denotes another section a single % is comments so what's really nice about MATLAB is you can actually run code out of order based on just moving around these different sections so here we are inside of this section here I'm going to go ahead and run in advance the next one and here we see an item show of our Landsat image so this is the image we showed earlier not really a great image everything looks kind of purple it's kind of very difficult to work with so let's go ahead and close that and let's try to analyze what's going on with that image so I'm going to use an item his function so what this is I just opened it up is just going to give me a histogram of the image data and so let's go ahead and run this section in advance and then we'll see here this is showing us where the problem is coming from so we see here all of the data is clustered together right here around 50 and so that's what this is the red band that's what's giving us pretty much poor contrast and to look at the data a little in more detail we're going to do something else here we're going to actually run a scatter plot so what this is is plotting the red blue and green in a three-dimensional plot for us so here's all the data points and you can see it's not hard from this what this means also to see what the problem is in all three planes we see everything clustered along this line right here so again not really great you kind of want to see some as far as distribution here to get some more dynamic range and to see different colors okay so let's go ahead and see how we can correct this if we scroll down here this is where we're going to try to do what's called a contrast stretch and so this is using a function called image s or image adjust and you'll see here hitting f1 I'm going to adjust the intensity values of the color map okay and so you simply pass it in here and you'll scroll down and I'm going to be using here's a just another example of it I'm going to be passing in the true color and then you'll see I'm just going to be doing a stretch limiter and you'll see here what this stretch limb is the function it finds the limits to the contrast stretch of the image so we're going to go ahead and pass that in and we're going to run it and you can see instantly we have a much nicer image we have actually some some shades as far as even dark dark green all the way to some shades of white so this is a pretty nice image and so we can actually kind of do some processing on this we can really work with it but we can even take this a little further if you look although there's some good contrast there still as far as the colors are quite muted and we want let's see if we can actually make the colors a little more dynamic so this was a great step forward just using a contrast stretch let's go ahead and check the histogram on why this actually worked and so you see here before remember everything was clustered in a narrow band around 50 now we see the histogram is is pretty evenly distributed and so that's why we're able to see so much more detail so to go one step further and maybe fix the color we're going to do something what's called a D correlation stretch okay so this is going to enhance the color separation across the highly correlated channels remember that earlier that the scatterplot everything was banded a correlated really tight across one line so with this D correlation stretch right here let's go ahead and run this and go to the next section you'll see again not only do we have a nice get a contrast difference now we have a pretty dynamic range of color everything from these nice dynamic greens to blue to shades of red so we have a lot more tangible data that we can deal with and really you can use the image as is or continue down the path and do some more image processing okay so let's go ahead and take one last look at this data from a plotting perspective and if we look here here is the scatter plot and you can see instantly is that we have a pretty good distribution of color and that's why we have such a dynamic image so just to summarize our image enhancement example you saw we were able to visualize not only the images but we were able to look at histograms and scatter plots to help analyze the problems in the image and then based on that we were able to do some performance enhancement like contrast stretch as well as a decorrelation stretch and so we essentially move from this poorly illustrated image here that was all purple to this nice you knows dynamic image with lots of shades of color as well as correct contrast so this takes us now to our next subject we're going to talk about what is image segmentation image segmentation is simply the process of dividing an image into multiple parts and typically this is a foreground as well as a background this process is typically used to help identify and isolate objects so that you can use other functions to extract meaningful information from an image so here below you see you know a densely packed image here and we use watershed segmentation to essentially isolate and start counting some of the objects here in this point image you can see we we segmented it and we have a nice binary image here that we can actually do analysis on to maybe even count how much money is here based on the size so let's go ahead and get into the demonstration here so we're going to deal with our toy cars and candy and Hexbug image and we're going to segment and analyze the image we're going to turn it into this binary image here of everything that we care about and then we're going to extract some statistical information from it we'll go ahead and open up a script I wrote earlier so here you can see I'm doing exactly what I you saw me do earlier I'm going to read in the toys and the candy image okay and let's go ahead and dock that so it's a little easier to see and I'm going to minimize this tool strip just to make some more room here okay and so next thing I'm going to do is I'm going to convert it to grayscale so let's go ahead and convert it and you see here I've removed all the color and if I look at my workspace over here you can see I move from three layers to one so I don't have an a red blue green layer anymore just a grayscale image with just single intensities for each pixel and then the last thing I'm going to do is a IM 2bw functions so this is a threshold in this is probably one of the most simplest forms of segmentation but it's also quite powerful if we look at this function and let's go ahead and go to our documentation I am - BW I'm going to do it this way and Here I am inside of the help and if I look at this function you can see this converts a grayscale image to a binary image and uses a level so this level value essentially sets the cutoff so it's between zero and one right here and then wherever this cutoff is everything above the cutoff is going to go white to a value of 1 and everything below it is going to go to zero and that's what makes the binary image so and also before I go I'll just show you here if you go to documentation this way this is another great way to navigate as far as to the functions that you might care about for example image enhancement here we are a contrast adjustment these are some of the functions we dealt with earlier so again and just another way to get to help and get some real guidance as far as what you can actually do so let's go ahead and do this and so I'm going to initially set this level value to 0.5 and just at the halfway point and we're going to do an I'm to BW and create this valuable eye Thresh so let's go ahead and run this section so I just hit control enter just to run that section and you can see let me go ahead and blow this up here we have our original image here we have a thresholded image so we did a pretty good job you can see just by putting a level of 0.5 we have some of the candies some of them pretty well segmented we have the cars and then some of them you can see we're starting to lose it so these lighter shades don't threshold as well we didn't pick the right value but at least we got the edges but there's a problem you can see the candy the most of them were segmented except for you see these yellow ones and you can probably guess why this is and that's because the car as well as these candies the yellows are very close to the desktop here and so when you have colors that are very close like this the threshold inquiry and tricky when you have high levels of contrast basic threshold again when you have colors that are close you know these might only vary by a couple different intensity levels so that's where it can be very challenging so let's go ahead and think about this a little more before we do that we can still experiment though so look at this level value we can right-click on it and we can say increment value and run section here's a little window that popped up I can say I want to increment that by 0.1 so the tenths place and then let's go ahead and increment that and you can see as I click on it you see changes taking place I'm now letting in some more black here you can see oh but the problem is I get a lot more noise however some things are segmented pretty well but I still can't get these yellow pieces of candy to show up any better so there's definitely some issues but the nice thing is I was able to iterate pretty quickly and see what was working and what isn't so you see point six was pretty good but still it wasn't good enough I still can't get these yellow pieces of candy that actually brings us to a question we really need to maybe take a step back and try to maybe think about you know are there any unique attributes or characteristics about this image that maybe we can take to our advantage so if you look at it what are some things that jump out at you so one is of course sizes of the objects we've got the cars are pretty large and mass compared to the candy the hex buggers are kind of in the middle so we've got sizes we could work with we could also do some image processing based on shape right we've got these very rectangular objects here with a with the toy cars and hex bugs but the candy is actually round so we could actually play a bit with shape based processing maybe with morphology and then finally another thing we could look at is color there's quite a bit of color being represented here and so that is a pre-processing technique that many people overlook but it's actually quite beneficial so let's actually start with color so if I come back to here actually wrote this function earlier and this is called color spaces dot M if I open this up let me go ahead and minimize that you can see I'm going to just read my candy image in but what this is going to do is we're going to print out each one of the color spaces remember I said this was three layers here well we're going to print out the red the green and the blue and you can see here's the plots I'm doing as well so let's go ahead and run this section here you can see the easier here's my original image but now here's each one of the grayscale images for each one of the color planes so I'm actually going to do pre-processing in color and then I'm going to threshold each one of the different color planes so coming back to our help here you can actually see if we go to image processing toolbox you can see there's a whole section on color and you can deal with things like the ICC color profile the LAV color spaces you can even deal with HSV for the hue so there's a lot of different ways you can actually do processing just based on color so let's go ahead and go back to my lab here and so I'm going to open up on another file so this is similar it's a combination of the color processing file or script I wrote earlier as well as the threshold in one except here we're just going to read in the image we're going to do what we did before and we plotted out our three color spaces and then now we're actually going to threshold on each one of the layers so here let's go ahead and open this up you can see I did a IM to black and white i'm to BW on the red plane the green plane and the blue plane and we pick the values that would get the objects isolated correctly without bringing in too much noise in the background and what we did is we summed and added all the planes together and so we have a correctly as far as isolated image i mean there's still some noise in here but we'll deal with that in a minute if you look back at our script here you can see what happened essentially again thresholded each different color plane here's the red the green and the blue and then I sum them all together so I simply just did an end and put them all together and then we plotted them out what I did with this next step here if you looked here this is great but it's more intuitive to work with black as a background and white as the foreground so I'm just going to compliment the image and so that's what's being done here and I'm going to call a function call I am Phil so I am Phil is a morphological function so remember I mentioned morphologies to study shapes so this is a shape based filter and what this does is going to fill up the hole so all these holes that you see in the image here are going to be filled up and made solid so let's go ahead and do that and you can see it filled up all the holes very nicely but we still have some random noise we have these frayed edges we have these little dots around so we're going to deal with some more morphological operators or shape based filtering and before I get into that let me actually show you the function and we'll talk about what it's actually doing if we look here the next section right here uses a morphological operator call I am open so this is going to open the image and what all morphological operators tend to use is something called a structuring element so if I hit f1 you can see this structuring element is essentially the shape that you are going to select to help do your processing and so you see the structuring element you can pick things like a ball or you can pick diamond disks lines octagons and so on we're going to deal with a disk so let's go ahead and run this and so here I'm going to do an opening and you can see if I come back to my image here and you can see it's cleaned up quite a bit all the little dots are gone all the frayed edges are cleaned up and you've got a pretty well segmented image right here and you can see let me go ahead and dock this for a minute I could have played with that structuring element a little more so let me go ahead and blow this up so you can see it a little better so I can right click again on the size of that disk that's the radius and you can say increment value section there I can say five and now if I click I can make that 30 third five 40 and eventually you'll see the size of that disc is actually going to get too big to fit inside of those pieces of candy the skittles so you'll see 65 70 75 now you see it couldn't fit in any of the candy so it wouldn't redraw those pixels and now can only fit in the hex bugs and the cars so this is a way of I could deal with shape based or pixel densities as far as filtering and I can keep incrementing until all the hex bugs are gone so you can see how with morphological operators on a binary file is pretty powerful you can really do some nice clean up as well as some shape based filtering so let's go ahead and move it back to what it was I actually want all the pieces of candy as well as X bug so I'm going to change this back to 25 okay and run that section and so I have everything back and then the final step what I'm going to do is I'm going to extract some details so the function I'm going to use is called region props or region properties and this is a very powerful function so hit f1 you can see what makes it so powerful is the fact that you can extract a lot of detail from a binary image and so you can get things for example such as area how many pixels they are in a particular object you can find things like the centroid the centers the eccentricity the diameters intensities of the of the actual pixels so there's a lot of things you can pull from it even the number of actual objects in the binary image so a lot of neat statistics can be extracted with this region property function and so we're going to go ahead and run this and then in the end here the final step is we're actually going to draw some nice bounding boxes and label all of them and also count how many objects that are so you can see I drew nice bounding boxes around all the objects for the user so it can be easily identified and also we counted them and there we saw that there are 18 objects in the image and so you saw through using this image of toys and cars and using a lot of the features of MATLAB we were able to identify some unique attributes right so at first we did some basic threshold mend that as it's a very simple approach to segmenting an image but you saw we had some issues with color here especially these yellow pieces of candy here they created problems because it wasn't enough contrast so what we did is some pre-processing if you will we did some color based processing we separated the color spaces into red blue and green and we did now a threshold anon each one of the different planes and as a result we were able to sum them all together and get a complete segmented image of everything that we wanted to isolate and then finally we did some region property functions to gather some statistics to count the objects and to surround them with nice bounding boxes for the user for easy identification and then finally I kind of talked about morphological operators it's a very powerful tool for pre and post processing but also you saw how I can really segment an image based on sizes of different structuring elements so I can basically filter based on shapes so now that brings us to some additional things so other utilities and tools that can help you you saw me play with like I am tool there's a lot of fantastic apps that are at MATLAB central so this is our online community and where you see a lot of people contributing for example here's a lot of tools you can use for example segmentation tool morph tool to play with morphology image registration app so these are fantastic utilities to not only help you explore but also really converge on a solution you can play with the different parameters and a lot of these utilities actually output MATLAB code so you'll be able to integrate it directly into your application so I encourage you to take a look at these and you'll find them on the file exchange on MATLAB central and so this brings us to our final topic and that's image registration so what is image registration so simply this is the process of aligning images from different data sets for either visual comparison or computational analysis and you can see here from the images below you can be using image registration for maps may be collected from multiple satellites maybe it's the same satellite on the right we have you know 3d medical imaging where you're oh you're registering multiple images to give you a more comprehensive data so you can do some more processing as well so as when it comes to the topic of automatic image registration there's actually two major categories one is intensity based this is exactly what it sounds you actually are going to be aligning two images based on intensities and in this case it's maybe aligning the skull you can see here it's the intensity is very bright with maybe another image that has a high intensity there as well and then in a map that's exactly what's going on - it's aligning things like streets houses any sort of characteristics in the image that where the intensities are going to align up the other type of registration is feature based so with feature based that's exactly what it sounds like you're extracting different features for example corners could be edges could be particularly unique features within it and the difference between these two is intensity based registration algorithm is found in the image processing toolbox the feature base because features is a big part of computer vision is found in the computer vision system toolbox so the example I'm about to show you is the intensity based automatic registration and that's from the image processing toolbox so in this demonstration what I'm going to show you is how to work with and in this case a webcam image and an infrared image and the goal of this example is to register the two images together because we want to measure body temperature and because the locations of the inner corners of the eye is the most accurate place to measure body temperature we want to register these images so then I can run an algorithm on the webcam image to find out where the location of the eyes are and then I'll be able to correlate on the infrared image that exact location to read the correct body temperature so let's go ahead and get back to lab and so here's our script here and you'll see what I'm doing is I'm importing in the two images into my workspace I'm going to be reading in the infrared image as the fixed image okay so that's the one that's going to remain stationary and then I'm going to read in the webcam image so this is the one that's the moving image and the one that's going to try to move and line itself against the fixed image so let's go ahead and read those in and then what I'm going to do quickly is I am show pair so here showing you the two images next to one another again we want to register these two so what we're going to do to configure that is we're going to call this function call I am reg config and so this is really the inputs if I hit f1 you can see it's only expecting really one input argument and you can see here's the modality so all that means are your images captured from a single camera or a single device if so then you use the input argument mono modal if it's captured from different devices like we are here a webcam and an infrared camera then we're going to input multi modal okay so that's our input argument is multi modal and go ahead and run this and then we're going to do our first try at registration so we're going to use the item register function you can see here it's going to do intensity based image registration and the first thing the transform type we're going to pick first is simply translation and so all that means is going to move the moving image in the XY direction to see if they can align the two so let's go ahead and and run that and you can see here this is our results so here we're doing an item show pair and what's called false color so we're overlaying them and this kind of gives you at least some nice color to see what's going on and you see it did a pretty decent job on the right-hand side but there's definitely some issues you see this ghosting here and even the shoulders are misaligned so this is kind of telling me let's take another view of this let's do a blend instead of a false color this gives us a little better viewpoint so you can see here that there's actually the scaling issues going on so it aligned up properly on on the right side here but on the left-hand side you can see one of the images is clearly larger than the other one so the simple translation the default transform is not going to work so let's go ahead and change things so we're going to change the transform type to f-fine and if you scroll down here just remind you what that is so we use translation that didn't work f-fine is more advanced it's going to use rotation as well as going to change scale and some shear as well so let's go ahead and run that and see what happens and so you see here oh it's gotten a little bit better so we have not as much ghosting around that the head but we still have issues here around the neck and the ears even the shoulders seem to be off quite a bit but you see that the image was rotated so that part worked we're getting closer but let's try something else let's see if we can try a different transform we're going to try the similarity transform and all that is is simply one additional change it's going to deal with non-reflective similarity okay but it's still going to do a translation rotation and scale but it's not going to play with shear it's a little simpler so let's actually try this one and here's a results you can instantly see that this is actually a very good registration so you see the head is aligned very well see the shoulders you have a little bit of gap here but to be honest this is really what we're after is we want the eyes aligned quite accurately because we want to detect again where the eyes are so we can accurately read the body temperature so the final step here is we're going to use a function from the computer vision system toolbox and that's a Cascade object detector and if you look here if I break it f1 you see it's using the Viola Jones algorithm which is a great algorithm for anything dealing with faces and that only detects where the faces are but it helps to detect facial features such as eyes nose mouth upper bodies and you and see here we're going to try to look for the pair of eyes so if we go ahead and run that section and then we're going to compute the temperature near the eyes so this is where it's going to convert it and what we're going to do here is we're going to embed the temperature and do a text inserter and write it on the screen so it's easy for the user to see so if we go back to our image here we have our nice registered image and we have a nice bounding box around the eyes and you see here the temperature read is 98 degrees Fahrenheit which is totally normal and this person does not have a fever to summarize the demo you saw how we acquired images both infrared and standard webcam images we visualize the data both before and throughout our algorithm process we used automatic image registration to align the images and then finally once aligned we use feature detection using the viola Jones detector in the computer vision system tool box and then finally we found the relative temperature and we wrote that on the image for the user and for more information here are some great resources first we have a product page we also have Steve's blog where Steve Evans one of our technical experts related to image processing authors this blog and there's a lot of great interesting topics there and then books wise if you're new to image processing digital image processing using MATLAB which is also co-authored by Steve Evans is a great resource to learn about the key concepts with image processing and how they are actually executed in MATLAB and then finally a product trial where you can try out a lot of the examples you saw today as well as experiment with your own algorithms and that brings us to the end of our webinar first of all I'd just like to say thank you for the time and hopefully you found this session helpful and that maybe you walked away with some new techniques or ideas that you can implement in your own image and video processing workflow thanks again

Channel: MATLAB

Views: 452,109

Rating: 4.9485908 out of 5

Keywords: Image Processing (Field Of Study), The MathWorks (Business Operation), MATLAB (Programming Language), Image Processing Toolbox, image enhancement, removing noise, image segmentation; image registration

Id: 1-jURfDzP1s

Length: 38min 40sec (2320 seconds)

Published: Fri Sep 05 2014