Feature Detection and Matching + Image Classifier Project | OPENCV PYTHON 2020

hey everyone welcome to my channel in this video we will learn how to create an image classifier using feature detection we will first look at the basic code of feature detection and description and then we will move on to the classifier project I upload videos on a weekly basis so don't forget to subscribe and hit that like button if you find this video useful so let's get started [Music] so here we are in pycharm and we have our main directory and you can see here we have image images security and images strain folder and what we have inside are three images of each and you can see that we have covers of games so the idea here is to detect these games and what we are going to detect in is our train image so we will try to find our Kure image in this given image which we will call as train so the first thing we will do and you can see there is another image outside for Assassin's Creed I will explain why that is later on now the first thing we will do is to import our libraries we'll go to file settings and our project interpreter and we are going to import open CV and numpy open CV Python and lamp so once we are done we are going to import our libraries so import cv2 and import numpy as NP now the very first thing we have to do is to grab our images so we are going to use the IM read function so image one is equals to c v2 dot i am read and we are going to grab our curie image so we will say inside images curie we have which one should we pick let's pick the Xbox so Xbox Kinect Sports and then image 2 is equals to C v2 dot I am read and then we will write train image and inside we will grab what is the name of it it is connect so now we can test it out if it's working so we will write C v2 dot I am show and we will say image 1 and image 1 and then C we 2 dots I am sure image image 2 and image 2 and then we can write C v2 dot weight key and we will give it zero so if we run this and here you can see we have our two images image one which is our cutie image and image two which is our train image so next we are going to create our detector now the detector we are going to use is known as ORV which is oriented fast and a rotated brief so the idea here is to find features so what are features so let's have a look at the opencv example so this is given in their documentation so here they are explaining the features and we can see that we have a building and they have shown ABCD and EF so if these are the features how easily can a human being detect these features so if I were to ask you how to find a you will say that a could be anywhere here and B could be anywhere on the right hand side of the building here if you look at C and D they are over here so these are the side rails of your building but if you look at E and F you can pinpoint the exact location of them so E is your corner point over here and F is your corner point over here so we can say that E and F could be considered good features whereas a and B or C or D not so much so a and B are the worst ones C and D are better and E and F are the best ones so the idea is to find these features and once we find these features we are going to translate them into computer language so that part we will call as descriptor so when we translate our feature to computer language that will be a descriptor and then using those descriptors we will match two different images so if it's a little bit confusing don't worry we will go ahead explain this step-by-step as we go along so for feature detection or be uses the fast algorithm so actually a modified version of fast or which is oriented fast so let's initialize our detector so we will right o RB is equals to C v2 dot o RB underscore create now you might be asking why are we using o RB why not other algorithms there are some common ones like Swift or surf well o RB is one of the fast ones and the one that is actually free to use if you use Swift or surf you will have to pay a yearly fees because they have patents so now we will find our key points which are our features and the descriptors for both images so let's write down KP 1 as our key points and we will write D as one as descriptors is equals to or RB dot detect and compute and in sight we are going to write image 1 and none for the mask so the same thing we can do for the second one and we can write here 2 and we can write here 2 and 2 so in order to understand these let's print it out let's show these key points so what we can do is we can use the built-in function for drawing key points so we will say image KP one key points one is equals to M which is equals to C v2 dot draw key points and we will send in our image 1 and we will send in our key points KP 1 and we will write out image as none now one thing we did not do so far actually let's do it for the other one as well let's space that and 2 2 and 2 TV 2 . I am sure and we will write KP 1 and this will be image KP 1 and we can copy this we will write here 2 and 2 so one thing that we did not do before that we should is to import the images in grayscale so that will ease our process so we can just write here comma and 0 that will import it as grayscale so let's run this so this is our image yeah so here we have our images now with the key points shown so all these key points are the ones that the algorithm has found to be useful so you can see here where it says season 2 you can find similar key points over here right next to the connect icon you can see that we have similar key points now we have these features now but we need to describe them and that's why we have our descriptors now what are these descriptors these descriptors are basically an array or you can say they are a bin of numbers so if we print them out let's see so if we write print descriptor 1 and if we run this you can see it's just a bunch of numbers but let's see what is the shape of it so if we run this we can see the shape is 500 by 32 now that looks like an odd number like what is 500 and what is 32 now the idea here is that the orb detector uses by default 500 features so it will try to find 500 features in your image and for each feature it will describe it in 32 values so if we look at descriptor 1 let's print that out so you can see that it's 32 numbers put together in an array so this is one descriptor so we have 500 of these for each image so how can we use these descriptors now so what we can do is we can use a matcher to actually match these descriptors together so I have the descriptor of image 1 and I have the descriptors of image 2 and I can match these two together and see how much similarity am I getting so we have different types of matters as well now one of the most common ones is the brute force matter so basically it takes one descriptor and it matches to all of the other descriptors of the other image then it goes to the second one and it matches all the descriptors of the second image with our new descriptor so this process is very computationally expensive so there is another technique of the brute force which uses the K nearest approach K nearest neighbors so we are going to use that instead so let's have a look at that so what we will do should we remove this or let's keep it let's keep that there and over here we are going to write our matter so we will say that our brute force matcher is equals to C V 2 dots brute-force matcher and we are going to say that matches is equals to brute force not K and n match and then we are going to send our descriptor 1 and we are going to send our descriptor 2 and we are going to give our K value as 2 because we want two values that we can compare later on so this will give us our matches but how do we decide that which is a good match so what we can do is we can go through all the matches and we can find the distance between two matches and the two features and then we can decide if that is a good match or not so we can write here good is equals to an empty list and for M and n in matches so because we said K is equals to 2 we will have two values that we can unpack so we are going to say that if m dot distance is less than 0.75 multiplied by n dot distance then we are going to append and we will decide that this is a good match so we will say good dot append and we are going to write in between M so whenever the distance is low the distance between them is low we are going to say that it is a good match if the distance between them is high we are going to say that it's a bad match so you can change the value over here based on the results you are getting 0.7 0.75 is a good approximation so now that we have all our good matches what we can do is we can plot them out to see what we getting so we can write image three is equals to see v2 dot draw matches okay and N and we will write image 1 then k p1 then image 2 then K P 2 and then we will write we need to give the good matches and then we will write here out image to be none and for the flag the flag is how do you want to show so we are going to write it as 2 so let's output image 3 so if we run this and there we have it so now we can see that it's showing us where did it find the appropriate matches so you can see that here it's by the end of the racket here it's by the end of the racket here you can see it's found this to be here which is wrong then you can see the S over here is detected here so there are some good matches and then there are some bad ones so we can change the value to actually fit this also what we can do is we can increase the number of features it has to find so we can write n features is equals to by default it's one 500 we can put for example 1000 so let's run that again and you can see now we have much more matches so what we can do is we can actually print out the length of our good matches so we know that how many good matches we are getting so we can write print length off good and if we run that we are getting 42 so we are getting 42 good matches now if we were to do this for another image so right now we are using the Kinect image which is this one which has the actual image but what if we used another image for example this one let's see how many matches do we get there so our image to this time will be the other one so let's comment this out and if we read on this and there you have it so now it did find some matches but you can see that the matches we found are only 5 so earlier we got 42 matches and now we are only getting 5 matches so we can clearly say that this is not a good match this image our Curie image is not present in our train image just by looking at the length of our good list we can also do it with the other one let's write that down last and let's rerun it and there we have it so again we are getting very low numbers we are getting 7 so again we can tell that this is not a good match because the number of good matches are very low so next we are going to create a project out of this so what we will do is we will create an image classifier in which we have these three images our Curie images and whatever image we feed in we are going to find out which of these three classes it belongs to so this means that let's make it in a way that it is generic so that whatever number of images we put here it will automatically create that number of classes and run the code for us so let's create a new file first so we'll go to peyten and here we will create image classifier using feature detectors so let's just copy our original code we will copy this and we will paste it here now the very first thing we should do is - yeah let's let's import our images now we can import our images one by one so image one image - image 3 but that is not a good way to do this and as we mentioned before we want to make a generic code in which it will automatically detect how many images are inside our folder and it will create that many classes so in order to do that we need to use OS so we will import imports or s and then we are going to find how many images are here and we will store the name of all these images so that we can import them so let's comment all of this out and over here we are going to write first of all our path so our path is basically the image curing and what we will do is we will create two lists in which we will store all our images and the second list we will store all the names so the names will be ps4 Detroit ps4 The Last of Us and Xbox Kinect Sports so the reason we are storing the name so that we can display whenever the class is detected we can display the name of the class instead of just the ID so we will create a list called images is equals to empty and class names class names is equals to empty then we are going to find the list of the names that we have in our path which is image images curiam so we will use my list is equals to OS dot list directory and we will give it our path so if we print this out print our my list and let's see it is running the previous code right click and run and there you go so now you can see that we have a list of names of our images which we can use to import so here we can write or should we write let's let's write print and we will write total glasses detected and over here we are going to write the length of my list so whenever we run it it will first tell us how many images it detected how many classes there will be so if we run that it will say total classes detected is three and we know that we have three images so we are good to go okay so next we can import our images so we know that we are using I am read function to import but we have to give it a loop to actually go one by one and import so then we can say that for class in my list we are going to import so we are going to say that our current image let's write IMG current is equals to c v2 dot i am read now we have to put in our path our path is basically so let's write our path then we will have a slash and then we will have the name of our image with the extension which is basically CL so we can write that and as you know we are importing it in grayscale so we will write 0 once we have imported that we will appendix in images so images dot append we will append our current image and then we are also going to append the name so we will write here class name names dot append and and we cannot append CL directly let me show you why so if I print CL you will see that it contains the extension as well dot jpg so when we are displaying the name we don't need to display dot jpg so we need to remove that so in order to remove it we can write OS dot path dot split text and then we are going to write here CL and 0 so now it will be stored without the file extension so we can print it out and see if it works so we can print class names and there you go so now we have the names without the extension and we have our list of images what is wrong with the Spelling's him so the next thing will be to find all the descriptors of our given image now all of this these descriptors we only have to find once because these are constants and the image we will get from the webcam we will find the descriptor for it when we get it so all of this we can do outside the while loop so what we can do is we can create a simple function that will take the list of your images and then one by one it will create the descriptors and it will store it in a new list so let's write that function so we will say let's say find descriptor and we will send it our images so images and then we will first declare our list so this will be our descriptor list and then we are going to loop through all the images that we have so we will say that for image in images we are going to find the key points and the descriptor for it using row RB dot detects and compute and we are going to send image and then learn now it says o RB is missing so we have to uncomment this so all of this code is basically to import images so this whole code is for importing and so now we have the ORV and once we have that we can append our descriptor in the descriptor list so we can strike the Earth's thought list dot append and we will write our descriptive and at the end we are simply going to return our descriptor list now all we need to do is to call this function so we can call it simply by saying descriptor list is equals to find descriptors and we will send our images images so that should do it so if we print the descriptor list length then that should give us three because we have three images and there you go so it gives us three so it means it's storing all the descriptors of each image properly so the next thing we need to do is to now define our while loop so let's remove some of the code from here so this part we have already done which is importing the images then we do not need to draw the key points and we have done the detection part so let's remove that and we have to define the matcher and this still so we will keep that so now let's write down the code for our while loop so we will say wine true we are going to get the camera feed and for the camera we have to write cap cap is equals to C v2 dot video capture and we are going to use device number 0 here we will write sux and image is equals to gap dot read and now that we have the image we are going to convert it into grayscale actually let's copy it because we want to see that the colored image at the final output so we will say image original is equals to IMG to dot copy so let's put this as image two and then we can simply convert it so image 2 is equals to C v2 dot CVT color and then we will write image 2 and we will write cv 2 dots color PG are two gray so this will convert it into grayscale and then we can write CV - let's copy one of these oh we have one already so let's write this here and we can also copy the white key and we will put this as once because we want one millisecond so if we run this and there you have my webcam and you can see we have all these games that we are going to test so as I mentioned before we want to see the final output as colored so we will use image original instead of image 2 so now you can see we have the colored images okay so now we have found the descriptors of our cutie images and but we did not find the descriptor of our webcam image so the the one that is coming from our webcam so we need to find the descriptor of that and once we have that then we can use our matching algorithm to find the difference between the both descriptors so let's create a new function for that let's call it find ID so the the idea here is that we are going to send in our image and we will send in the descriptors so inside the function we will find the new descriptor of our current frame and then we will match all the descriptors and at the end we are going to send in one single ID that it matches the most with so our return value should be just a single number for example zero one or two so if it's matching with Detroit it should be zero if it's matching with last of us it should be one and then two for Xbox Kinect so let's write that down so we will write find ID and we will send in as I mentioned we are going to send in two things we will send in our image and we will send in our list of our descriptors so the first thing we will do is we will find the descriptor of our new image so we will say KP 2 and our descriptor 2 is equals to RB dots detect and compute detects and compute and inside we are going to write image and none so once we have that we are going to use our previous code to find the matches so over here you can see that we are using this so we will copy this this is common sound and then we will copy this and we will paste it here so now we can remove that so now you can see that we have the descriptor 2 which is good which is our descriptor of the current frame but what is the descriptor 1 now the descriptor 1 we don't have just a single value we have multiples of them so we have 3 different descriptors that we want to compare with so we cannot write descriptor 1 here so what we can do is we can loop through all the descriptors and match them one by one so here we are going to write for descriptor in descriptor list we are going to match so we are going to match our descriptor the current descriptor that we have and then inside we are going to apply our same method that we did before to find the good matches so we will copy we will cut this and we will paste where is it we will paste it here and this time we are going to find again the good matches and we will find the length of our good matches and then we will put it in a new list so what we will do is we will declare a new list we will call it match list is equals to empty and we instead of printing it out we will just append it match match list dot append this way it will append the number of good matches it is finding for each of these images and it will give us an output list so let me just print it out so that you can understand it better so friends match list but we didn't call this function so it will not work so let's call it over here so we can say find ID or is I define what is it called how it should be defined ID find ID and then we will send in our image - and the descriptor list and let me run that and there you go so now you are seeing a list which is our match list and what these values are is basically the number of good matches it is finding for each of these Kure images so it's saying that it found 10 matches with the Detroit image it found 27 matches with the last of us and it found 7 matches with the Jeannette sports game so as you can see the image we were looking at was actually the last of us so this means it is pretty much correct because the most matches the most good matches we are finding are for the last of us so what we want to do next is to find the maximum value and then we just want to send out the index so here the maximum value is 38 so we will send the index 0 & 1 so we will send the index of 1 so in our final image we can write the name of image 1 which will be the last of us so how can we do that we can define here a new variable by the name final value and we can put an initial value of minus 1 we cannot put 0 because 0 is also a value so 0 will be Detroit so this is 0 1 & 2 so we will initially put minus 1 and then what we will do is we will find the maximum value but before we do that we need to set out initial threshold the minimum threshold that it should have at least for example 10 good matches or 15 good matches to qualify it as an actual match otherwise if nothing is available it will just randomly keep matching which will not be accurate so in this case we can see that we are getting 12 we are getting four we are getting six seven so most of the time if the value we are getting the good matches is less than 15 if it's not the actual match so here it's 12 again so here 13 12 13 and pretty much that's it so we can say that if it's above 15 it means it is a good match and we need to consider it otherwise we will not consider that value at all so let's do that so we will remove our prints and down here we are going to write first of all we will check whether it's empty list or not because sometimes you might not find any matches and it might give you an error or something actually that reminds me we should put here a try and exception as well because sometimes the matcher does not match anything and it gives out an error so we can write here try and over here we can write accept pass so all of this should be inside the try okay then let's write down here so here we are going to write down first of all we will check whether our list is empty or not so we will say that if length of match list is not equals to 0 then we are going to check if our max value of our math list so this max value will be for example 32 in this case 28 in this case 20 in this case and so on so if that value is greater than let's say a threshold so that threshold we can ask the user to define actually by default we can put 15 and if the user wants to they can define so if the maximum of it is greater than 15 then we are going to find the index of it so we will write final value is equals to match list dot index of max of our match list so this here will find the maximum value which is let's say 32 and this here will find the index of it which is 1 so 0 and then 1 and then we are sending it to final value so final value was minus 1 it will become 1 and then we will return this value so we are always going to return so return final value okay so that is pretty much it now what we can do is we can save our final value so we can say that ID is equals to find the value and do we need to define the Thresher let's let's not define the threshold and then we will just check that if ID is not equals to minus 1 if it's minus 1 we will not print out anything but if it's an actual index then we will write down the name of our class so here we will say c v2 dot put text and we will put it on the image original and then we will write our ID so here we can write ID but writing the ID will not give us the name so it will not be very clear this is why we had this class names so what we can do is we can say that find the index at the ID given so that will give us the name and because it's already string we do not need to convert it and we can place it like 50/50 and then we have to give it a font CV to dot font let's pick any font then we will give it a scale and then color we will give it let's say red so zero zero two five five and thickness let's give it one so let's run that and see what happens so there we have and if I put it like this so it says the last of us actually the thickness is a little bit thin let me increase that so here we have PS for the last of us this one is not defined so Xbox Kinect Sports and ps4 Detroit now you remember that I mentioned in the beginning that this one was not defined and if I don't put any of these you can see it does not show anything here so this means that our function here this maximum threshold part is working properly so as you remember I mentioned that this image we are keeping outside the folder of our image Curie now the reason I put it outside is because I wanted to show you how easy is it to add a class so if I just drag this image inside our Curie images folder now if I run it you will see that it will not display three it will display four classes and it will be able to find the assassin creed as well so if we run that you can see already here it says four and you do not need to worry about their order it will automatically detect and place it in the correct order so if we go here it still says Detroit and if we go to Assassin's Creed and now it's saying Assassin's Creed and you have Detroit then you have Xbox Kinect and then you have the Last of Us so this is how you can create a simple project with feature detectors which can classify your images now you do have some limitations on this mostly the images you are trying to find should be 2d they should not be a 3d object and most of the time it should have good texture so the more texture it has the better features it will be able to extract the better descriptors we will get and the better matching that we will get at the end so this is it for today's video I hope you have learned something new if you enjoyed the video if you found it useful give it a thumbs up and I will see you in the next video

Channel: Murtaza's Workshop - Robotics and AI

Views: 61,134

Rating: 4.9816995 out of 5

Keywords: feature matching, brute force, opencv, python, tutorial, features detection, sift, surf, obr, Feature Detection and Matching, feature detection, image classifier, image classify, opencv image classifier, opencv image matching, Image Classifier Project, OPENCV PYTHON 2020, OpenCV, Object Recognition, object detection, ORB, SURF, opencv tutorial, opencv tutorial python, python (programming language), computer vision, image processing

Id: nnH55-zD38I

Length: 45min 32sec (2732 seconds)

Published: Tue Jun 16 2020