Identify objects moving on a conveyor belt using Opencv with Python

so everything is working correctly we should see this is going to stop right nice hi welcome to this new video tutorial we're going to demonstrate today how to build a prototype using a conveyor belt a Raspberry Pi and OpenCV we will show in this specific case how to detect nuts we are exactement nuts with different sizes so we have a bigger one and smaller one our goal in this project is to put all the small ones inside this box so if this runs all the small one will go inside this box then when we get through the big one we want just to take this one out of course I'm doing that now manually but we will do that using OpenCV so Raspberry Pi will detect what's exactly here on the belt and so this is a really useful project for companies for industries because on a large scale you can detect the objects passing on the belt you can detect the quality of the objects and if you connect this with a robot arm you can for example separate on each boxes like put the correct not on the correct box or different objects on different boxes just on one conveyor belt so it will take small one in each box then the arm will take the big one on some other place but you can use this in many different ways so if you are curious to see how this project is going to work and how to build it keep watching and let's start let's start by importing the library so we're going to import cb2 and then we're going to import numpy S&P as first thing we're going to load the webcam cap is equal to C V so that video capture 0 and here is the first webcam if you have more webcams the index can be 1 2 and so on then we take the frame in real time from the webcam while true is then underscore because we don't need this value that's why we put underscore and then we just need the frame is equal stock cap dot rate then we want to show the frame on the screen so see we do that in show the name of the window let's just call this one frame and then what do we want to show we want to show frames so we put frame right here then a wet key event so that we can press the key in real time while we're running the video or key is equal to weight key 1 so we're going to just wait one milliseconds between each frames and then if the key is equals to 27 and 27 is the asking on the keyboard that case we want to stop so we break the load once we break the loop we're going to release the camera because we don't need it anymore cap dot release and then we want to close all the windows because they're not going to close just themselves automatically so we need to close them by writing C if it's without destroy all windows how just a typo here is not just weight key but it is CV to dot weight kid and now before proceeding I'm going to run the script to be sure that everything is written correctly so let's press run and everything is running fine right now here it is that conveyor belt that I have at the moment now what's the goal of the project if you see here we have different exact groundnut we have one three small of them and then a big one we want to examine all the nuts passing in real-time and for example we can say that we want to focus on the small ones and we want to stop the belt if there is some big one so let's ask first thing just try to detect the nuts on the belt how do we do that well let's take a look at this frame we see here we have the conveyor belt then we have many other things on the screen that we don't need the first impression that we have to make is to simplify everything so what we do we just cut the belt because we just want to focus on the belt and so belt is equals to well now we need to cut this from the frame I don't know exactly the position of the belt so we're going to check where the belt is located on the screen and it is really simple when you use OpenCV on the Raspberry Pi you you can see here that we had the coordinates of the mouse on the screen so we just place the mouse on the top left on the screen or on the conveyor belt and I see in my case X 139 and y 210 so let's keep these informations X I already forgot them X 137 in 2009 137 and y 209 and then we need to see where this end so we take the top left and then we take the bottom right point bottom right is equals to 5:30 and 327 X - 5 3 and I don't have a good memory 23-7 against countries and so now that we know the coordinates we're going to use these coordinates to cut a region where is the belt from the screen so the belt is going to be equals to the frame and from the frame we're going to cut the specific area so frame we have from our keep in mind that when we work with the frame first we work with the rows which responds to the Y and then we the columns which corresponds to the axes so first we safe from we want to take from 29 to 2327 7 and then we want to take from 137 so 530 and to make sure that this is correct we're going to show on the screen just the belt see it so a team show belt and then belt and now let's I mean stop everything and let's run this to see if we have just the belt and that's exactly what I wanted to achieve so we just cut it a portion of the belt so that our detection is more simple now how do we detect the nuts on the conveyor belt it's always better to use the most simple approach possible and in this case as if we consider the nuts have most likely always this color and the belt is almost always black at least this belt is black then we can just use a simple threshold operation so the threshold based on the value of the pixel and in this way we can simply detect the nuts so let's do that in case it's not clear now everything will make much more sense first thing we're going to convert the frame to a gray scale formats of frame equals the Grif frame is equal to see if it's without CVT color so we want to convert the frame and then see if it's without color we want to convert the BGR format to the gray format so if we show seem to the MSHA frame gray and then frame no grave frame great great frame let's run this one here is the frame in a grayscale format now we can apply the threshold on this one I said just we want focus of course on the path so we did the conversion on the grid frame but we will focus on the belt so better let's do this on the belt so instead of gray frame which most likely we don't need let's say great on the belt gray belt is equals to Belle see if it's got CVT color we want to convert belt and then see it to that color underscore PG our format to gray format on the gray belt we will apply a threshold and if it is not clear what is threshold let me show what it is and you will get it underscored trash trash old is equals to see to that threshold we want a very threshold on the grey belt now we say that all the values above 50 will be converted to 2055 C dot Thresh Thresh banner okay and let's show the threshold threshold and then threshold and let's run this one okay what you see here exactly we are converting all the let's say gradient colors close to the black we're converting them to a complete black color all the things like this exact now that are not so close to black we will convert them to completely white so the threshold it is a way to convert a great image to instead of having different gradients of gray only white or only black the thing the fact that that you see here this thing is because the lightning even if it's not clear to our eye here there is some lightning change that's why here we have this we can solve this just changing the threshold value so instead of 50 we can let's say increase this one so let's see this okay not perfect but for sure a really decent result so this is just some really small noise but it is in general great result so if you don't get it right you have to play around with the threshold value and tweak to understand well what is threshold I will not go deeper in this one but there is some other video I've done about thresholds so you can check it out instead let's now move on now we're moving on by detecting exactly these white objects so let me show this again now we somehow made a distinction black and white so black we don't need the black edges deccan failure and white what is the white white are the exact nuts that we have so now using the obesity function fine contour we're going to find a contour so the boundaries all of all the white objects that we have on the screen so let's do that also I'm going to put some comments so here are our eyes the region of interest so in this case is the belt that's instead of all right let's say that let's keep simple words why make things complicated what it is not necessary belt and here detect the nut so underscore contour then underscore is equal to CD to that find contours and then we're going to find contour on the threshold then CD to dot red external and I will explain this quickly save it to that chain up Crocs simple let's now draw the contour so for CNT in contours let's print them I will quickly show what is what is this so here in CNT we have a lot of coordinates and these coordinates represent the boundaries of each white object that we have so in this case the white objects are the nuts so we have the coordinates of the nuts and now we will take a rectangle and we will draw the coordinates of this so we have a function from OpenCV which bounds a rectangle from this coordinates so we have X Y width and height of the rectangle is equal to save it so that bounding rag of C and T and let's short angle for them sieve it to that rectangle and rectangle we want to draw the rectangle on that belt and then we have the top-left point of the rectangle which is x and y that we just got on this function and then the right bottom point of the rectangle so we had X plus the width and then Y plus the height let's make green rectangles so it will be the colors geoblue 2055 of green and 0 of red and let's make them Figg three pixels and let's throw this on the screen okay now you see this interesting result we have boundaries surrounding the nuts one two three four and also we have some noise here so of course it is not perfect so some lightning change ores like the conveyor belt doesn't have a complete black belt but here there is some difference in lightning but anyway that's not a big deal it is not a problem for our project so even if there are ways to remove them they don't disturb us so we can keep them instead let's focus on our real goal our real goal is to distinguish between the small nuts so small we have one two three small nuts and the big one to understand the difference between them we can calculate the area that they have so let's calculate the area calculate area area is equals to see it so that contour area of CNT and we can show the area for each one of them we can say for example see if it's not put text we want to put the text on each contour so that we can see the area in real-time we want to put the text on the belt of course and we need to define which text we want to show we want to show the string of the area so area is a number but when you put the text you need to convert it into string format so that that's why we use STR area then where do we want to place the text we want to place the text just on X&Y so on the top left point let's put it there and then we have the font of the text so your different number corresponds to different font and then we have the size of the text so I would put I don't really care about the font so I will just put zero of whatever phone it is and then we define the color of the text let's use maybe green color 0 to 255 0 also for this one let's let's see if this is visible okay we see a lot of numbers and it is a bit of a mess so maybe let's try changing the font a smaller font let's try one and also I want to change the color of the boundaries because if we use green for everything then it is not so clear and also let's show the text as last element just after the rectangle always the rectangle it is covering the text let's run this okay well much nicer let's see what is happening here we see this mauled by a small nut have a decides around 20 78 this one to 27 to 30 while this big one has value above 500 so just using the area we are right away able to distinguish them so we can say if the nut is bigger than 500 for example after the areas or distinguish distinguish small and big nuts and we say if the area is greater than 500 so we have a big nut so in this case were big nut and what do we want so the we can for example draw a red rectangle to make it stand out on the video so instead of to 5500 blue green red with 0 blue your green and 2055 of red instead if the area else if the area is smaller so it is a small nut and we draw just a blue rectangle let's run this okay now you can clearly see the difference between them right away even with the code we know that this is a big nut and they are small nuts plus this small value that we see here we can remove them actually let's remove them so we will make this project even more clear so if there is greater than 500 and if the area is more than 5 under let's say that the small nuts are between 100 and 400 pixels of of emphasizes the area so if 100 is less than area and if area is less less than 4500 let's keep it this way so all the elements below 100 we don't take them into consideration even if we see the numbers we're not surrounding them we could even remove the numbers but I don't want to waste time on these leaders that don't really matter right now now next step we need to find a way to move the belt so switch on and switch off the belt the belt is connected to a relay which is connected to the Raspberry Pi and I have a library to switch it on and switch it off so let's load our conveyor belt library what is a custom library I quickly create myself I just go here called it conveyor lead and we need to import this one import from the your clip and convey your deep dot conveyor we create an orbit an object relay against it's written this way really and we want to turn on and turn off the relay when we turn on the relay of course the conveyor will start running when we turn it off it will stop and it is really simple now I will give Manuel common with the keyboard so here I will just add two lines regarding that I leave the key is equals to let's say that if we press the letter n on the keyboard we are going to switch on the belt so art N and then relay dot turn on it is just that simple I leave the key is equals to if we press the letter M instead we're going to turn it off or M relay dot turn off and let's see if this works correctly so let's focus now on this big image so I press letter M and you see it is running correctly and for some reason okay now we see okay I press I press m and it stopped but we see that of course because of the distance of the camera now the object there when they are in this part of the of the belt they look smaller so our condition 500 plus and 500 less it is not working correctly so we need just to change the values but for the moment let's keep this going okay I will just change this one 400 and 400 now we are at the last part of the project and in this part we just want to stop the belt if we detect the DRI's a big knot on the belt so which we know that here area the text big nut and here it detects small knot so here we need to add something that stopped the belt if then we detect big knot and we stop the belt so relay dot turn off it is the same function I used before let's try this but before I have to take all of them I have them here so that I can place them again on the belt so we can test any real time okay I'm going where the belt is just here on my side to test them to test it okay the belt is empty so let's now switch this on and it is going even if we don't see so there is something to work on it wasn't supposed to stop right away so one thing that we can do is here is the start of the belt we want just not to focus at the start but at the end of the belt only so we can somehow cut the area that we are considering for the belt so instead of taking from let's see the value that we have we are taking from 29 28 27 is the height so that's the same always from 137 to 530 instead let's take from 137 to 250 or or bit more 200 hitting maybe and let's see what we get we have just a small portion of the belt which probably is enough so let's let's test this one now in a rough time I press n and it is on let's see when we see this one passing on this small screen well that's nice so that's what we need exactly so I'm going now to test it is working correctly we should see this is going to stop right now nice that's exactly what we wanted so I take it out nice well as you can see this a really simple version of this project of course it can be improved in many ways but we achieved our goal with just open CV and a really simple code I can say and these of course is really useful on industries if you want for example if you have let's say for example nuts you have many of them on the bed of different sizes and you want to put each size on a separate box so there could be a robot arm which takes them in based on their size and they put in the right box for each of them you could use this approach to detect if there is some element which is not the nut but of course you can use this for any object passing on the conveyor belt you can check the quality of the products you can check if there are bottles if there is if there is something wrong with the bottle if there is some sticker you can see if it's not printed correctly and so many other things that you can do with this hope you enjoyed this project I will be still also releasing soon other interesting projects like this one so if you want to see them stay updated and see you in the next video

Channel: Pysource

Views: 22,488

Rating: 4.9741268 out of 5

Keywords: conveyor belt, detect object, opencv tutorial, python tutorial, computer vision tutorial, detect objects conveyor belt

Id: A29IqeahI84

Length: 34min 4sec (2044 seconds)

Published: Tue May 19 2020