Distance detection with Depth Camera (Intel Realsense d435i) - Opencv with Python tutorial

430 millimeters so around 443 centimeters uh hi there my name is sergio and i help companies students and freelancers to easily and efficiently apply visual recognition to their projects we're going to see today a simple implementation of the intel realsense d435i camera this is a depth camera that you can use to track not only the object but also the depth of the frame that you get so you can get exactly the distance of each single object and i used this in the past for some commercial project and it's essential when you need to work with commercial projects especially for manufacturing companies where you need also to get not only the the x and y position of the objects but also the z position so that the robot arm knows exactly where to go also on the z axis so that you don't smash the object if you go too far or if you are too high especially when you have object which have different sizes but you can also use this one for different projects for emotional recognition when you have the depth you get better accuracy and also you can use this camera for three-dimensional objects so you can get the objects in three dimensions because you have a clue of where is the distance of each part of that object but for today's tutorial we're going to keep things simple and we'll focus only on distance detection uh we'll do this in two steps on the first one we are going to get the frames and information from the camera so we'll get the image in real time also the information of how far are the objects and in the second part we're going to integrate this with opencv so that we can show in real time using the mouse the position of each single pixel i hope you're ready for this and let's start now before proceeding you need to make sure that we you have everything installed to use this camera and we need two things we need opencv which is the library that we use for computer vision and second we need pi realsense 2 which is the module to run this camera and i will go quick with the installation cmd and then you can install them with the pip python command pip install opencv python that's one of course i have a requirement already satisfied and so i have it and you need also to make sure that you have it and then peep install the second one pi real sense 2. and requirement is already satisfied in this case as well and so if you want to be 100 sure that the installation is correct we are going to import cv2 which is opencv then we import pi realsense 2 and you if you run this one and you don't get any error it means that the installation is done correctly now to work with the camera there are some functionalities and libraries released by realsense to make things more simple i created a new file which is this one real sense depth and we will be using this one so if you don't have this file you need to download this file from the link below it's called real sense that from the blog below you will find all the instructions and once you download this file there are two files real sense depth which will be the file that we will be importing on our project and then the attack distance is the main file that i am coding right now so you need to import this one and this one you will get as well but i am that's what you will see right now that i am doing and once you have this one real sense that we are going to import real sense that but uh better we're going to from real sense that we're going to import everything so from real sense that we're going to import everything and i made this file in a way that in just a few lines of code we can extract all the information that we need from the camera and it will be really easy even if you are a beginner uh first of all let's initialize the camera so initialized camera of course this will for the interval sense and it's simple we can say dc which stands for that camera equals depth camera and this is the this class is inside real sense depth now from that camera we can easily import the vgr frame so the color frame also the depth frame and in just one line red then depth frame and also color frame equals dc dot get frame uh to see and make sure that everything is working correctly let's show this frame on the screen cv2.imshow and we're going to show the color frame color frame and then color frame now let's make a weight key event to keep the image open cv2.weight key 0. and now we can run this one and this is the color frame that we are getting from the camera and now let's also get some information from the from the depth frame we made the depth frame also in order that from the original frame we can extract the distance of each object i'm going to show you the depth frame even if it's not the way that we use it but let's now show the depth frame shift in show depth frame now that frame the depth frame is a frame which has the same shape of the color frame but instead of containing the image with color so the the original frame from the camera this arrays is containing for each pixel the distance of on the space from the camera to the object and we're going to make this uh a simple example right now we're going to take some specific point of the frame and show the distance for that specific point let's see that let's take for example um show distance for a specific point and we are going to take i would take some random point point let's say position 400 and 300 let's show also this point to have a clue of where we're taking the distance from sieve it so dot how do we show it of course uh the best way here would be to draw a circle on this point so that we know where the point is on the frame to that circle we want to draw a circle where we're going to draw the circle on color frame and we will put the circle on the position point so position 400 300 and then let's make radius 4 so that we can see it well and let's make this circle red so zero blue zero of green and the maximum of the red which is 255 and now we can run this one and as you see we have here the circle which is getting the chair this red circle right here as we now have this circle how do we know how far is from the camera this specific point well in order to get this information we need to refer to the depth frame so on the depth frame we need to go on the same position which is this one and from that position get the value of the point and it's really simple we have distance equals we take the distance from where we take the distance from that frame and then from that frame where exactly we have an exact position so we define the indexes uh we have 400 from the x and 300 from the y so 400 starting counting from the left side and 300 started counting from the top and the only thing that you need to know that when we work with opencv uh when we work with uh more specifically with the arrays instead of putting first the x position then the y position we need to put first the y position so we will put point index one so we're taking first the position 300 and then point index 0 and let's show the distance print distance and let's wait that this initialize and we see now a value which is 802. 802 is the distance in millimeters this means that the camera the depth camera is around 80 centimeters far from the chair so 800 millimeters or the same 80 centimeters and i will make note this in real time so that we can try to play around with this so we can do that by instead of taking only one frame we want to switch to real time detection so from after we initialize it that camera we also do while true and we put everything inside the while loop so the video in real time it's nothing more than images one after another so we're getting a frame after frame and do all the processing for each frame that we get just one last thing that we need to change is the cv2.weight key 0 because if now i run this one even if there is a while loop it doesn't move farther unless i press a key when i press the key each time i press a key we go to the next frame and we don't want that so we remove instead of weight key 0 we use weight key 1 with key 0 freezes the frame with key 1 waits 1 millisecond and move to the next frame and then we say also key equals weight key 1 so if the key is equals to 27 which is the s key on the keyboard we can break the loop so i'm doing this in order that if we want to quit we can otherwise there will be no way to quit normally from the window and now let's run this one as you see now we do not only have the frame but we have this in real time and now let's play around with this one when i go closer to the point you see now we have around 305 so we have around 30 centimeters distance and the farther i go the more this number is increasing sometimes we have some errors so it's not 100 precise this will require some filtering some more advanced method which is not the case of this tutorial for the moment but you need to know that this camera is quite precise and detects the distance uh from 10 centimeters to 2 meters with a 2 margin error maximum this means that if the distance the object is around one meter there can be an error on the depth detection of maximum two centimeters if it's uh two meter distance the error can be up to four centimeters which it's quite good detection we can say now we we've done with the first part of this tutorial which was getting the information from the camera we have the color frame we have the depth frame and also we have the position of one specific pixel now on this second part of this tutorial we are going to focus on integrating opencv with the mouse functions we will have a function that when we move the mouse we get the position of where the mouse is and for that position we get the depth for example the idea would be that if i go i put the mouse on my nose it will tell the distance from the camera to my nose if i put the mouse on the wall will get the distance of from the camera to the wall let's do that oh the first thing that we have to do is creating a mouse event so that we get the position of the mouse when we move it inside the window so let's do that create mouse event to create a mouse event we need first to create a window cv2.name it window and this window needs to be the same where we're showing the frame so if we create a window then let's make sure that it it has the same name of this color frame so color frame to this window we're going to associate a mouse event see if it's suit dot set mouse callback and now we need to pass two parameters set mouse callback where of course on the color frame so we're going to say that on this color frame we want to activate the mouse functions and each time the mouse is over the color frame window we want to use this function which we can call as you wish let's call this show distances oh let's say show distance and now let's create also this function and we will hear this at the beginning depth show distance and there are a few parameters we need to pass and that's how it works with opencv for mouse callback functions we need to pass first event x y arcs and params uh don't worry about all these things we will use almost none of them for from this we will just use x and y and we just print x and y x and y and let's now run this one and i will show what this is doing right now once we run this have a look right here on the printing when i go with the mouse not on any window but on the window where we associated the mouse callback function so you see color frame when i go on that window we see the printing as output the position where the mouse is so if i go on the top of course we have around zero zero almost so each time i move the mouse i get the new position and we have that specific position and using this we can change the our position of interest in real time for example there is this red circle that we have right here by changing this position in real time we could also move this red circle in real time and we're going to do that so that later we can get the distance from this circle and let's do that right now so how do we do this we need to find a way to take these coordinates which are x and y and put them inside point right here of course we can't do just taking x and placing the x here you will see this is an error because we cannot take the the variables from a function and just take it outside of the function but we can do something similar we take point we're going to put the point outside the loop because we don't need the point on the loop and let's put the point just here at the beginning point 400 and 300 the thing that we can do is changing this point in real time so now if i run this one it doesn't change much point was 400 before it's the same right now so we see the circular rate always here but we can do one thing which is global so we're assigning global to point and then we can say point equals x and y so each time that we first of all at the beginning we have point four hundred three hundred let's see that so i just see we have the circle there but when we move the mouse inside the window you see now the red circle is where the mouse is and having this now makes things really easy for us from this point position we can take the position from the depth frame and then of course we get the exact position the exact distance from that specific position where we have the mouse let me do this we have distance uh we have already listened so let's show the distance on the screen in real time cv2 dot cd2 dot put text we're going to put the text where we will put the text on call or frame and then what is the text we want to show the distance um we will do it this way dot format and then distance and now let's put mm for millimeters and then what else we need the the position okay i forgot we need the position where to put the text on the screen and we're going to put the text on point zero the exposition and point one is the y position i will we could also put just point right here but i will explain why i did it this way later then font face cv2 dot font i will just take the first which i see for hashtag play we don't really care now about which font we are using as long as we see the information font scale one and then finally the color let's make the text black so zero of each color zero blue zero green zero red and thickness of that text two and that's it let's run this one okay we have two thousand me okay as you can see probably the text is too small so there there is something to change so let me quit this one and let's change something of the formatting of the text so two is one is too small so let's put font size two but also if here we have the point and we have the mouse i want to put the text a bit more up from the point position so that's why i extracted the x and y from point one so that i can add minus 20 so that we will put the position of the text a bit above the position of the point and so it should be everything that we needed and let's run and test this one right now okay now we have the position of the text this one you see 240 me no 2400 millimeters which is around 240 centimeters if i go on my head is around 42 420 430 millimeters so around 443 centimeters the chair is around 80 centimeters and so on uh the position is not always uh 100 exact in some point in some part we see zero so there will be some function that we will need to implement like some filtering in order to avoid these errors where we see zero also it's important to know that this camera if there is the reflection of the lightning that does uh effect also the distance detection so it's important to know uh this information uh well i i think there is nothing more to add regarding this one please let me know what you think about this tutorial and let me know if you use the camera and what else you would like to see using the interior sense the 45i camera and this is all for this video i want to remember also that i have an academy so on pistours.com you will find the section academy where instead of these demos in short videos i have full video courses also for beginners where you will get detailed lessons and also more advanced source code about different computer vision topics i hope you enjoyed this lesson see you on the next one bye

Channel: Pysource

Views: 71,007

Keywords: intel, realsense, d435i, distance detection, depth camera

Id: mFLZkdH1yLE

Length: 24min 23sec (1463 seconds)

Published: Thu Mar 11 2021