Connect Intel Realsense with Python and OpenCV

in this video I'm going to show you how to take your Intel realsense camera and connect it with python to show a color image and a depth image [Music] the Intel real sense is a line of cameras that Intel puts out that incorporate both depth sensing and RGB Imaging in one small camera now this camera is cool because it's relatively low cost now the cost is going to vary this is a d435 maybe it's about 300 now a couple years ago it was a bit less and the price is going to vary based on the model you have so they have models like the d435i which has some gyroscope inside of it they have a D415 they have a d455 and really the major difference is the the range that the depth sensor can see the angle the resolution so they if you're looking for a specific application you should check out the different specifications to choose a sensor that fits your needs now I am just using the 435 this 435 right here has a USBC cable which is what we use to connect it to the computer so so hopefully my program doesn't crash again but the important thing about this USBC cable is you're using a USB 3.0 I'll try to show you the error you get if you use a USB 2.0 cable and that error doesn't always occur it occurs at well at least for me it occurs when I'm trying to run the camera at higher resolutions for RGB and I get some sort of error that says I can't get any can't get any frames from my camera so I'm gonna so use the USB 3.0 cable to try to eliminate those problems first let's install the dependencies we need this is not Linux specific I'm just using Linux because I want to eventually move this camera on to Ross so I'm going to pip install uh pi real sense too I'm going to pip install opencv python and I'm going to pip install numpy and all of these are already installed for me so now what I'm going to do is I'm going to go to my editor and I'm going to import those packages so import Pi real sense two as real sense I want to import numpy as NP and I want to import CV2 so those are the three packages I'm using and now let's make a connection to the webcam so first uh let's create a pipe and this is going to be r s dot pipe line next we're going to create this variable that's called config and it's going to be rs.config and this is where we're going to make all the initiation calls that set up the settings we want in our webcam so the first thing we're going to do is cfg.enable and enable stream and what we need to do is this tells us we're going to start our uh RGB streaming so rs.stream dot color and then we can pick the actual dimension of the color image we want so I'm just going to use 640 by 480 right now in my format is going to be RS dot format I'm using opencv so I want to do BGR eight so an eight bit uh and 30 frames per second we're going to do the same thing CFG dot enable stream rs.stream dot depth again I'm going to use 640 by 480. in RS format this time is going to be rs.format.z for depth 16 bit and again I'm going to do 30 frames per second so at this point I said okay this is my color stream this is the size of my color image this is the format of my color image and this is my frames per second if you're looking for any of these options so this is depth if you're looking for any of these options you can install the Intel realsense SDK and they'll show you the available options that you have with your cable that you're using okay so now let's start this config so now we've started the the streaming and now we'll create an infinite for Loop and we'll wait for our frames to become available so frame is equal to pipe dot weight for frames so this is going to get me my depth and my color image so my depth frame is going to be equal to frames or frame dot get depth frame my collar frame is going to be equal to frame dot get color right now this is in a different format than what we can use so what I want to do is I want to convert my images to numpy arrays so depth image is numpy dot as any array I want to say depth frame dot get data so this is going to convert this depth Frame data to an array that's a numpy array so then we can have it in a format that opencv can use all right so now let's do the same thing with color image so color image is numpy dot as any array color frame dot get data and let's show these so cv2. I am show this will be my RGB color and I'm going to show this color image then we'll have cv2.im show this will be my depth and for my depth I'm going to show my depth image then I can do something with the weight key so it automatically runs again and again so if CV2 dot weight key one is equal to ord of Q then we can break if we break out of this Loop so if we press Q on one of these uh if we press Q it's going to break out of this uh stream and then we just want to stop pipe dot stop okay so let's run this and what do we get uh the error we get is it's a bad argument so my color image is a bad argument oh I need I need to make this a function right here so it gets the data so now when I run it I I do get something so I now need to change the screen capture so so I get something that looks like this I have two images the one on the bottom is my RGB image the one on the top is my depth image if I look over here you can see uh the difference in depth it's a gradient that's gray so as I move my hand forward and backwards the well the depth should change sort of hard to see so let's actually put a let's put a color map on this so let's put this color map on so my depth we'll say color map is going to be equal to CV2 dot apply Color map and we can use the CB2 dot convert scale and abs and what this does is we're going to take our depth image and we're going to have an alpha value of 0.5 so we're creating the scale right now uh on the step image Alpha value 0.5 and right here we'll use CV2 dot color map jet so now instead of instead of displaying our depth image we'll display our depth color map so let's run this and now our depth color map is a jet color map and it's a little bit easier to see what the depth is so the different gradients are different depth values that we have I believe blue the blue value you see is actually zero depth because it can't actually figure out the depth for uh whatever reason so there you go so the really powerful thing about this is we can just use opencv once we have this color image to do whatever manipulation we want so let's say we want to take a grayscale image so my gray image is going to be CB2 dot convert color I'm going to take my color image and I'm going to convert that so CV2 dot can uh it's going to be CV2 dot color and we're going to go from BGR to gray right BGR is how my image is coming in and we want to go to grayscale and then we can show this gray image so let's run this and now I am showing a grayscale a grayscale image so you can do any type of manipulation that you want finally I want to show you some errors that you might that you might get so what we're going to do is we're going to comment out my depth image right here and we're going to increase my video stream to 1920 by 1080. and we're going to run this and I get a video stream so my RGB that is 1920 by 1080. what I'd like to do now is so we know this code works right we have this code the code works so what I'd like to do is not touch the code at all but change what I'm using from a USB 3.0 cable to a USB 2.0 cable so this is my 2.0 cable and I'm going to run the same codes let me hide this RGB I'm just gonna run it and I'm gonna get this problem right here I get this problem that I couldn't resolve requests and the only change I've done is I've changed from a USB 3.0 to a 2.0 now if I change back to the 3.0 so the 3.0 is now reconnected and I try to run this again it runs and and I'm actually able to get my my image again so that's one of the most annoying problems that I've worked with with this with this Intel real sense is sometimes before I knew what was happening I'd be using a USB 2.0 cable and I would be having a problem because my I knew my code worked but I wouldn't be able to get the higher resolution streams to come through so hopefully this is showing you how you can connect your Intel real sense with using pyreal sense 2 and displaying the image with python and opencv

Channel: Engineering Corner

Views: 13,967

Keywords:

Id: CmDO-w56qso

Length: 13min 20sec (800 seconds)

Published: Thu Apr 13 2023