Raspberry Pi LESSON 45: Using the Raspberry Pi Camera in Bullseye with OpenCV

hello guys this is Paul McCarter with toptechboy.com and we're here today with episode number 45 and our incredible new tutorial Series where you're teaching your Raspberry Pi who's boss what I'm going to need you to do is pour yourself a nice tall glass of ice cold coffee that would be straight up black coffee poured over ice no sugar no sweeteners none needed and as you're pouring your coffee as always I want to give a shout out to our friends over at Sun founder Sun founder is actually sponsoring this series of video lessons and in this class we will be using the sun founder Raphael kit the sun founder Raphael kit for Raspberry Pi now most of you guys probably already have your gear if you don't have your gear yet look down in the description there is a link over to Amazon you can hop over there and pick this kit up and believe me your life and my life are going to be a whole lot easier if we are working on identical Hardware but enough of this Shameless self-promotion let's jump in and let's talk about what I am going to teach you today in the day is the day that we have all been waiting for because today is the day that we are going to learn how to use the pi camera on the Raspberry Pi yes we are going to get the camera up and running today so by the end of today's lesson you will be able to shoot and display live video from your Raspberry Pi now how cool is that so I am going to need you I am going to need you to let me get out of your way here and what you will need to do is you will need to get your Raspberry Pi kit and you will need to find this camera module it's in a little static protection sleeve and what I need to warn you is I have found that the Raspberry Pi camera is one of those components that can be sensitive to electrostatic damage and so when you do not have your camera connected up to the Raspberry Pi you need to keep it in the little static protective bag and also you want to be very careful as you are connecting the camera because it is possible to damage it with electrostatics and so I'll kind of show you but it's just be a little bit more careful with this than what you usually have to be with most components now first of all where do you plug the camera in here is a connector called a CSI connector this is where it is going to plug in notice that there are two places that you could plug it in but only one place that you should plug it in because I have known people that have plugged the Raspberry Pi camera in here and in fact fried it and so you don't want to be the guide to fry the camera and so we're going to plug it in here now also when you look at this ribbon cable what you're going to see is down near the connectors one side of the cable is white and on the other side there is a blue strip when you plug this into this connector the blue strip needs to be pointed back towards you when it's configured like this or the blue strip needs to be facing the USB connector the blue strip needs to be facing the USB connector also as I'm connecting I like to go ahead and touch one of these metal parts and by touching the metal part it helps me kind of ground myself to the Raspberry Pi so that there's less chance that I'm going to get electrostatic discharge now this connector to connect to it you first have to put your fingernails underneath the little tabs on this black plastic piece and you have to pull up but understand it's only supposed to come up a millimeter or two and if you pull too hard too far you can damage this and once you damage this you've kind of messed up your pie and so you want to be very careful with this step so I come with my fingers to the left and right my fingernails and I pull it up I can feel it pull up and stop Don't Force It at that point now while I'm touching this USB I'm going to pull the camera out and what you can see is on this ribbon cable blue on this side and white on this side the blue side needs to be facing towards you and towards this USB connector and then you come down in the slot and you want to hold this up you want to hold the little plastic piece up and you really want to get this firmly down in there then once it's firmly in there press down on the left press down on the right and then press down both at the same time now the reason is when you come down here it goes like this when you press on the other side sometimes it comes up a little like this and then you press on both sides to get it firmly and snugly in there okay I had this goofy little tripod and what I'm going to do is I'm going to go ahead and tape the ribbon to the tripod so that the camera isn't just pointing up at the ceiling will give it something to look at I think I know a lot of you guys are taking the fusion 360 class I think it would be pretty cool if you guys would design and then print some sort of little tripods or holders for these uh for these Raspberry Pi cameras if you do leave a link share your design file so maybe other people could download and print it but let's see if people come up with some good ideas for some little holders or or uh you know little tripods I guess is what I'm saying for the for the camera okay so now the camera is connected and now we are ready to actually fire up the Raspberry Pi very important and I should have said this earlier you want to do this without power to the Raspberry Pi you want to do it with the Raspberry Pi unpowered as you saw here my power was not connected but now that the camera is connected I can boot up the pi and so let me switch over here to this View and this is the Raspberry Pi screen and so now as I plug in the Raspberry Pi hopefully we'll start seeing some signs of life here on our Raspberry Pi screen giddy up did you see the little multi-colored uh page there indicating that we do have a live pie okay and it's booting now the first thing that you need to understand is is that moving forward in these lessons I am I am on the Raspberry Pi Bullseye 64-bit operating system now most of you guys for the first 40 something lessons have been operating on the old Legacy Buster operating system now what I could have done was I could have shown you how to do this on Buster but the thing is on the new operating system it you know interacting with the camera is completely different than what it was on the old operating system so I had two choices I could teach you how to do it on the old operating system which was simple and easy and straightforward and I know everything about it but then very quickly that is going to become deprecated and Obsolete and the guy that comes in a year or two from now and takes this class it's not going to work for him so I decided I needed to bite the bullet and move to Bullseye 64-bit now the things that I'm teaching you today if there's some reason that you need to be on the 32-bit operating system the things that I'm showing you should work on Bullseye 32-bit but the things that I'm showing you today will not work on the old Buster operating system so what is it that you need in order to move forward you need to be on the the 64-bit Bullseye operating system does that make sense I hope it does now with that we're booted up in the bullseye system and now the first thing that we need to do is the way that we're going to interact with the camera is we're going to interact with the camera through a library in Python called opencv and so in order to do that I will need you to install opencv on your Raspberry Pi I'm going to show you how to do it but you need to be on the same version of opencv as I am and so if you've been already playing around with this and maybe you have opencv flash a fresh card for this class and then do the opencv like I'm going to show you here so what we're going to do is we're going to fire up a terminal and then we're going to install it with Pip so we're going to say pip install opencv Dash python that is the library but this is how you get the exact same version that I'm using you put an equal equal and then I am on version [Music] 4.6.0.66 4.6.0 0.66 now when we do this what we're going to hope is we're going to hope that we see things going out looking and downloading and we are waiting and hoping and it's looking for it that's good it's looking it's collecting okay and it's going so it found it and it is downloading and installing it very good news [Music] okay it'll take just a second here I'm not going to cause to pause the video it takes about 30 seconds for this to install foreign now I want to just check that so I'm going to just say python you can see that I get the python command prompt that my python interpreter is opening open I'm just going to see can I import CV2 which is what the opencv library is called and in fact it imported it without an error now like let me just say CV2 dot underscore underscore version underscore underscore and you can see that we got 4.6.0 which is what we wanted so everything is right on track at this point I think this video is going extremely smoothly at this point and let's hope that continues now let's go ahead and open up thoni so that we can begin trying to get this camera up and running so I'm going to come over open Thane and then I am going to go ahead and save this so I'm going to save and then I am going to save it as I'm going to go to my documents and then uh I'm going to go ahead in my documents folder and I'm going to save this as open [Music] cv-1 Dot py and the dot py is kind of important but we've gone ahead and saved it now to get that business out of the way and now we're going to start writing our uh writing our program now what I want to do is I want to Pro write a program that will go to the camera grab a frame from the camera and then show a frame and then I want to do that in a loop grab a frame show a frame grab a frame show a frame if I grab frame show frame in a loop what am I doing I'm creating a video all right and so what I want is I want a window on my Raspberry Pi screen that will show a live video that's coming from the camera so let's look and see how we would go about doing that well the first thing is we are going to need to import CV2 which is the program that is going to allow us to interact with the camera and create the the video now now if you guys are familiar with opencv in past versions of Raspberry Pi and in past versions of uh use of opencv you would use the opencv program itself to run out and grab the frame but in the new operating system the new operating system Bullseye you are not able to from opencv to grab the frame using an opencv method you have to grab the frame using a new library which is called Pi camera two so we're going to have to open that library in order to talk to the camera so we're going to say from PI camera to import Pi with the upper P Pi camera two so Pi camera 2 all lowercase import Pi camera to uppercase P you got to get your P your your cases right now we're going to go ahead and create a camera object so I'm just going to say pi can is equal to the library Pi camera two and here we don't have to uh here we don't have to put any parameters in you just fire it up like that okay that's all good now we're going to have to create some parameters that tell the camera how we got to give it some settings okay and so the way we do that we're going to do PI pad cam which is our camera object is pi cam dot preview [Music] underscore configuration so we're going to configure the camera we're going to set up the preview configuration and it's got to be dot Main and then what are we going to set I want to set the size all right so how big do we want the image that we grab from the Raspberry Pi camera now the bigger you grab the image the more detail you're going to have but the slower it's going to run the smaller you grab the image the faster it will run the more frames per second you can get but the lower the resolution and what I found is a good balance between something that's big and easy to see and fast is to make it 12 80. by 720 like that excuse me 1280 by 720 looks good you could also try 640 by 360 or the largest size would be it's 1920 by 1080. okay it's 1920 by 1080 would be the biggest that you could bring it in but if you do that you're probably not going to be getting 30 frames a second as I go through this I really like to try to keep things at 30 frames a second so we got that and so now we've given our first parameter that we have configured the camera to and then also we're going to want to do a pie cam Dot preview underscore configuration dot main again now we've got to tell it what format because there's a lot of different formats that you can use for video well this is the really strange thing opencv wants the blue green red or BGR format so we have to tell this Pi cam configuration that we want BGR and how do we do that we set this equal to wait for it in a kind of unusual thing the way you tell it that you want BGR is you set it to RGB sounds backwards but this is the way that works RGB and then 888 and probably what that 888 is I'm guessing it's like eight bits eight bits eight bits eight bits on the red eight bits on the green and eight bits on the blue is probably what that means now we're going to do a pi cam 2 dot preview and that's what time what we're going to do is config you can configuration uh ah not Pi cam 2. pack cam I like the ones above it preview configuration and this time we're going to do an align open close now what is this a line there are certain standard video image sizes that operate really fast it's like 1280 what did we say 1920 by 1080 the 640 by 360 all those standard formats automatically load really really quickly well what happens if I ask for a 1285 by 730 it will do that but it has to go in and do some manual work and so if it's having to do that manual work to create that non-standard size it's going to run slower well what this Pi cam preview configuration a line does is it takes the format or the size that you've asked for and it picks the closest standard format to make sure that it runs fast now if you care more about getting some strange exact size that you want than having it run fast you would leave this command out okay so now we have created this preview configuration now we have to apply it and the way we do that and I need an uppercase C there doesn't count as a mistake if you catch it before you run it so pi cam dot now we're going to configure the whole thing and how are we going to configure it well we just set up the preview parameters and we're now going to apply those preview parameters now if where did this preview come from it came from this leading text here now there are kind of three different things you can configure preview is for this case where you're going to be grabbing frames and showing them video is for the case that you're going to be grabbing frames and recording them and then there's a third one that's sort of like still images if you're going to go out and just be grabbing still images there's a different set of configuration parameters as we go through this class we're going to be looking at a lot of different cases but I'm just showing you today the easiest way to get your camera up and running so we've got the camera configured so now what do we do we just start it so dot start like that okay now the camera would be up and it would be running so really on almost all the things we're going to be doing in this class this is going to be the main part of your setup okay so you would do this every time and usually okay maybe maybe you would play around maybe you would play a little around a little bit with what size you wanted but most of this stuff is going to be the same every time you run it so that's good we've got the we've got the camera started so what is the camera doing it's up and it's running but now we've got to do something with those pictures that are taken well what do we want to do we want to grab a frame from the camera and we want to show it we want to grab a frame and we want to show it well we should do that how we should create a loop so I'm going to say while true one is true true is always true and then I need to grab that image or grab that frame so I'm just going to say frame is going to be equal to my camera pi Cam and then what do I want to do with pi cam I want to capture the array like that and you've got to think of a picture as an array it's an array of numbers it's an array of data and it each pixel at each intersection of row and column there's a pixel and that pixel has three numbers the red value the green value the blue value or it goes the other way blue green red and so you've got at each intersection of a row and a column you have a pixel and inside that pixel are three numbers the blue value the green value and the red value and those would be numbers between 0 and 255 okay and that's how an image is created so you've got to start thinking of this Frame you've got to start thinking of this Frame not it's just an arbitrary picture you've got to think of it as a data array of a large two-dimensional Matrix of rows and columns and at each intersection at each point you have an array of three numbers blue green red okay hopefully that makes sense and we will be working with that as a data structure as time goes on so we grabbed a frame what do we want to do now we want to show it so we'll say CV2 dot IM show so this is the library the opencv library and we want to give it the command to show now we've got to tell it a window to show in so I'm going to name my window camera or I think I'll name my my window pi cam like that okay and now what do I want to put in pi cam I want to put frame which is the picture that I just grabbed all right now I need to give myself a clean way to exit the program I need to give one of those dogs barking I want to make sure I don't have someone [Music] I hear them but I don't see them out there but I digress I've gotten sidetracked let's come back let's come back to the program so we have grabbed the frame and we have shown the frame and if you catch your mistakes before you're in the program it does not count and so I have grabbed the frame and I've shown a frame and now I need to do two things I need to pause just for a tiny amount just so it has time to show and then I need to give myself a clean way to exit this Loop and so what I'm going to do is say if on the next line if CV2 Dot weight key weight key [Music] and how long am I going to wait I'm going to wait for one millisecond just a tiny little amount to weight and what am I going to do I'm going to see if the letter Q has been pressed and if the letter Q has been pressed then I'm going to exit the program so if CV2 weight key so it stops here for one millisecond and looks for what to see if a key has been pressed what key is it going to be looking for it's going to be looking to see if I press the q key but you've got to tell it a little bit differently you have to look for the ORD of Q the ordinance of q but whatever letter you put here that is going to be the key that it's looking to see if you have you you have pressed it so it's going to be looking if I've pressed the q key that's the way you look at that and then if I press the q key what do you want to do you want to break out of here and so that will break out of this while loop and then it will come down here and then as I break out of this movie Loop what do I want to do I made a window so I want to go ahead and destroy destroy all windows like that okay so up here I set the camera up and then I start the camera and then here I am grabbing a frame showing a frame grabbing a frame showing a frame and then all the time I'm looking to see did I press q and if I press Q then I want to drop out of the program and come down here and destroy the windows does that make sense that is pretty exciting I'm trying to think is there anything that I forgot here I don't think so I probably have some typos I will need you to hold your breath giddy up look at that we have live video wow look at that we've got live video coming in on the Raspberry Pi camera the thing I want you to look at is look at how low latency now me in the bottom right of your screen that's the real me from the studio and then the other larger me is the me on the screen of the Raspberry Pi and watch this no latency at all one two three four five and so this thing is coming in really really nice really really good and you can see that I'm getting a very very nice image it is moving pretty good it looks like it's probably about 30 frames per second that that is operating at now how do we quit quit the program by hitting Escape no no by hitting W no no how do we quit the program we click on the letter what Q I do that boom it destroys the window is that not neat I want to run that again okay look at that that is so cool is it just me or is this a pretty exciting advantage that we have a pretty exciting advancement that we have made in our Raspberry Pi skills now this is the thing that I'm thinking you remember that video that you remember that security system that we made in about four or five lessons uh ago you remember how we were building that security system well imagine now you would be able to add a Raspberry Pi camera to that security system and you could do something like if you detected motion if you detected motion with that PIR sensor then what you could do is you could start recording video and I'll show you in future lessons how you can save the video to the disk and how you can do all types of different things and so there's just just really really some incredible things that you can do now what I want you to see here is is that what you're doing is you are grabbing a frame here and then you are displaying the frame here but I want you to see in the program you have the space between grab a frame and show a frame even though you think a movie is going really fast you know it's going at 30 frames a second and in the computer world that is pretty slow so between grabbing the frame in showing the frame you have the opportunity to do all types of different things right you have this is where the magic happens the space between grabbing the frame and showing the frame and like what you have to think is you have to think that this Frame again is a two-dimensional array it is like you could think of it as the frame it has uh the frame has rows and the frame has columns like this okay and then what you could do is you could come in and you could you could address a given row in a given column and you could change that pixel okay and so that means you can come in and you can do whatever you want to that image in between the time that you grab the frame and you show the frame and also this is the area between grabbing and showing a frame that we can go in and like we could add annotation we could add text we could add shapes we could edit we could do all types of things to the image that we're capturing after we capture it and before we show it and so those are the things that we're going to be learning this is also where we would be doing the artificial intelligence where we could be looking at the image and like trying to find a face or trying to find an i or trying to find some sort of object of interest and so that is what we're going to be doing in these lessons moving forward is all the magical things that you can do between grabbing the frame and showing the frame if that makes sense I want to run this thing one more time I am just really loving this that is really really really cool and I like how we can cleanly exit with the queue okay so now this is this is going to be your homework assignment and I need to do a little Windows management so you can see this here okay so what we really care about in a lot of these lessons moving forward you can see when I run this thing that this is really really smooth video right I mean it's just absolutely Smooth video but what you got to see is as we start doing more and more complex things between grabbing the frame and showing the frame if we're doing so much computation between those two steps it's going to start slowing down and getting jittery it's going to slow down and get start getting Jagged and so what do we care about we care about our frame rate how many frames per second are we able to operate at and so what's your homework assignment is is to take this uh is to take this program that we've written and then and then what you need to do is you need to look at it and you need to have the program calculate what frames per second it is operating at now there's one coming and maybe that I haven't shown you before so I'll show you that command and that is you can import time okay and then you can go time Dot time like this and then it shows you a number and that number is the number of seconds since the epoch and so I could go time dot time again and then you can see that it's almost the same number okay but we went from almost the same number 185 to 195. so what does that mean you can see that this time stamp was taken 10 seconds after the earlier one so that sort of makes sense so what is the epoch well the epoch is like it's this real strange thing but it was like some date some specific date some specific time in 1970 something that was declared the epoch so that point was point zero and all time is measured from that zero time but you don't have to worry about that all you need to know is is that what the time elapsed time is so like I can say time uh start is equal to time dot time like that okay now I can say time delayed is equal to time start minus time dot time okay and this is how much uh and now print time delayed ah what did I do wrong ah I spilled it a strange way a e d I spelled it okay and there you can see that there were 14 seconds that it elapsed from the first time I gave the command and the last time I gave the command and now you have nice easy to use nice easy to use numbers because you're looking not versus the epoch but versus when you started making the measurements if that made sense but just showing a little bit about how you can do some timing and so what your homework assignment is now is to go in and instrument this code where you're measuring and displaying your frame rate your frames per second you know how many frames per second are you operating this thing on and then we'll be using that in future videos so that as we start doing more and more advanced things as we're doing more and more advanced things we see the point that okay you go from 30 frames per second to 15 frames per second and then you get down to two frames per second and you can see okay that doesn't look like a very good video but as we're moving forward we can keep track of that by being able to measure and display and know what our frames per second are does that make sense I hope that makes a lot of sense okay guys I have had a lot of fun with this lesson it has really been a very uh a very big step forward in my mind that we can now uh use the camera with the Raspberry Pi and so what we'll do is next week I'll show you my homework solution to the frames per second problem and then I'll also show you kind of like the first thing where we start manipulating the image and I'll show you how you can go in and put annotated text you can put text you can superimpose text live on top of your video okay guys I hope you are having as much fun taking these classes as I am making them this was like a really big a really big day for me to finally get to start to play with the camera I'm really looking forward with what we're going to be able to do with this in the future lessons if you enjoyed this lesson be sure to give us a thumbs up helps me a lot with the old YouTube juice if you'll leave a comment down below subscribe to the channel if you haven't already when you do make sure you ring that Bell so you'll get notified when these future lessons come out and then most importantly share this video with other people because the world needs more people doing coding and fewer people sitting around watching silly cat videos Paul McWhorter with toptechboy.com I will talk to you guys later [Music]

Channel: Paul McWhorter

Views: 32,571

Keywords:

Id: kuJpdAf07WQ

Length: 36min 22sec (2182 seconds)

Published: Thu Jan 19 2023