Pixy2 Camera - Image Recognition for Arduino & Raspberry Pi

Video Statistics and Information

Video
Captions Word Cloud
Reddit Comments
Captions
today in the workshop we'll be working with the pixi 2 camera we'll learn how the pixie works and how to use it to detect objects follow lines and read barcodes we'll also hook our Pixy 2 up to an Arduino we have a lot to look at today so welcome to the workshop [Music] hello and welcome to the workshop today I'm going to be working with a really amazing little camera something called a pixie 2 now before I explain why I think the pixie 2 is so amazing I need to let you know that this pixie 2 was provided to me by the folks over at EF robot and I'd like to thank them a great deal for sending this wonderful device to me you will find a link to the product on DF robot below this video now why do I think the pixie 2 is such a great camera well unlike a regular camera such as the one that you would use with the Raspberry Pi the pixie 2 is an intelligent camera this device has an onboard processor and this processor allows it to do things such as object recognition line following intersection detection and even simple barcode reading now you can use the pixie 2 in your projects with just about any microcomputer the pixie 2 has all kinds of outputs that can be used with all sorts of microcontrollers and microcomputers so it's useful with the Raspberry Pi it's also useful with an Arduino a BeagleBone or any sort of microcontroller a microcomputer you might have now what we're going to do today is I'm going to explain how the pixie 2 works I'm going to show you how to use pixie Mon which is software that you run on your PC to monitor in the pixie and I'll also show you some sample code for using the pixie 2 with the Arduino to do things like object detection and line following and barcode detection so we've got a lot of things to cover today so let's get started and learn about the pixie 2 so let's take a closer look at the pixie 2 camera the pixie 2 is a small video camera with an onboard image processor chip it was designed by charmed labs and Carnegie Mellon University's robotics Institute the original pixie was released in March of 2014 as part of a very successful Kickstarter campaign the pixie 2 can be used with any microcontroller or computer the pixi 2 is trainable it learns the detect objects which we teach it it can detect lines intersections and very simple barcodes it uses a color based or hue based object detection system and has tracking algorithms to track devices anywhere within its own field of view it has an integrated LED light source for low-light environments the pixie 2 can also directly drive 2 servo motors for a pan and tilt mount now here are the connections on the pixi - camera there's a microUSB port for connecting an external computer an i/o port for connecting additional computers or micro computers and two servo motor outputs for connecting the pixie cam pan and tilt mechanism the pixie 2 has several interfaces aside from the USB port there's also a serial UART spi I to see and a digital and analog output the thick see two processes an entire image frames 60 times per second the device can remember up to seven distinct color signatures the pixie 2 tracks every object which it detects within its field of view the application for controlling the pixi - is called pixie mod and it runs on Windows Mac and Linux pixie Mon uses the pixie to USB port to communicate with the device pixie mine will allow you to see both the raw and processed video it allows you to configure the pixie tube and to manage the color signatures one neat fact is that pixie Mon can be used to debug the pixie - while the device is connected to an Arduino now here's a demonstration of pics Iman showing the barcode detection capabilities here is another demo showing the line tracking capabilities of the pixie 2 there are many coat samples and libraries available for the pixi two for both the Arduino and the Raspberry Pi the device supports C++ and Python coding all of the software and firmware for pixie 2 is open-source and there is a very large wiki and a user community that can give you lots of information for using your pixie 2 so now let's use our pixie 2 camera so let's take a look inside the pixi toucan box now when you open it up the first thing you see is the actual pixie to cam itself along with a little indication to go to pixie cam comm slash start where you'll find the whe key that gives you all kinds of instructions for using the camera now the cameras just mounted to the top of the cardboard here we'll take it out and take a little look at it the camera module itself on the top here and all the supporting electronics there's a number of holes at the bottom here these are used as mounting holes as you can see there's a push button on the top here this is used when you're training the Pixy to cam and I'll show you that in a moment there's a microUSB connector you would use that to connect this either to your PC or to something like the Raspberry Pi there is also a connector over here which brings out the other pixie cam outputs and you can use these to connect to an Arduino so let's put the pixie cam itself down obviously we'll be using that later else is in the box we've got a microUSB cable actually a fairly nice quality micro USB cable for connecting it to the computer another cable this is one you would use to connect to the Arduino this will actually go to the Arduino SPI connector and so you don't even need jumper wires to hook this thing to an Arduino that's great and finally some mounting brackets mounting brackets and screws they have on the pixi wiki they show you a number of different mounting arrangements you can use with these brackets the brackets are kind of unique they're not all the same some of them have threaded holes some of them don't and the wiki explains a number of different methods you can use to mount this on a robot or another project and so there you have it the internal of the pixie 2 cam box now let's hook our pixie 2 cam up to pick C Mon and actually start recognizing objects with it now the first step in using the pixie 2 is 2 get to your computer and use the pixie Mon software to configure the device and to update its firmware now as I mentioned pixie Mon is available for Windows for Linux and for the Mac I'm running it on a Windows computer here in my workbench right now for this demonstration now live first hooked it up it prompted me to do a firmware update and that's a very good thing to do because it'll bring your pixie to up to the latest version of the firmware so go ahead and do that also mix certain that when you connect the pixie to using a micro USB cable that you either use the high quality cable that came with the pixie 2 or use another similar high quality cable the reason for using a good cable is that you're powering up the pixie 2 using the 5 volts from the USB port and on a long or low quality cable the voltage may drop a little bit and be insufficient to power the pixie 2 this is especially important if you're using the pan and tilt mechanism because you'll also be driving a couple of servo motors off of the pixie and of course you'll need voltage for those as well so assuming you've got the pixie 2 hooked up to your computer and you started pixie Mon this is what you're going to see on your screen essentially you're going to see a display of what the pixie 2 camera is showing and the first thing you'll notice about this display is that the quality is not particularly high now this is actually typical of image recognition systems image recognition systems actually wouldn't benefit from high fidelity video it would just simply be too much information to process instead they use a rather grainy type of video which still gives them enough information in order to do object detection but doesn't overwhelm them remember the pixie is doing this 60 times a second so it really has to be very quick and processing every frame of video that it gets now let's take a look at the screen on pixie Mon and some of the features if you go to the file you'll see a configure now this configure screen a key configuration and the pixi itself now this tab over here as the parameters are on the pixi and so you can do tuning for the different color signatures I'll actually be showing you how we can use that in the moment there's an expert mode over here for setting up some advanced parameters on the pixi some signature labels as I mentioned the pixi can detect up to seven distinct color hue signatures or seven distinct objects and you can expand upon that by the way using color codes and we'll discuss that a little bit later but when those objects are detected to come up as signature one signature to etc etc this dis lets you label them as something like red ball or purple dinosaur or something that lets you describe what the actual device is so it's more human-friendly so to speak here are some parameters for the pixie camera and here's an interface tab now as you recall I said that the pixie could be used a pixie Mon excuse me can be used as a debug monitor when you're running the pixie on something like an Arduino and this is where you set this up you'll notice over here that by default my data out port is actually the ice is the SPI port on the Arduino and you can set things like the i2c address the baud rate for the UART etc you can put the data out onto a different port and so this lets you set up the connection you're making to your microcontroller or microcomputer while simultaneously using pixie Mon to monitor things so it's a really handy feature now over here we have some pixie Mon parameters and they're fairly basic the folders that you want to save documents into and a couple of other parameters that you probably won't need to need to adjust now one thing to note on the configure that when you change things nothing is actually saved until you hit the apply button so make sure that you keep note of that now under program we have four different programs that we can run using pixie Mon color connected components is the one that we're running right now and this lets us do hue based object detection line tracking another one that we're going to be experimenting with in a moment the pan-tilt demo is for the pixie cam pan and tilt mechanism I don't own that mechanism so I won't be showing you that and video dis lets you see the raw video which is coming out of the pixie cam because you may indeed want to use that video for something else other than object detection now under action we can set up parameters for the color signatures now this is where we can actually configure the pixie to detect objects and assign a signature to it although as I'm going to show you you can even do that without using pixie Mon so it's quite a versatile device over here under view we can have a couple of different view modes now right now I've got console tapped off and you'll notice at the very bottom of my screen I've got a little area here it's actually going to print out the commands that are being sent out to the pixie so I like to keep that open now over here we've got three other different selections for blocks blocks with video blocks video and detected pixels and these will make more sense when we start detecting objects I'll show you what the differences on these they just give you a different method of seeing things on the screen to interpret what the pixies actually seeing so now that we've taken a look at the interface for pixie Mon itself I'm sure you're anxious to get started recognizing objects as am I so that's the next thing that we're going to do now in order to recognize objects the pixie to assigns a color signature to the object the color signature is a combination of the hue or color of the object along with the objects shape the pixie too will memorize these and then it can use that to try to attempt to visualize the object when it comes into its field of view now you need to train the pixie to on the object that you want to recognize and there are actually two ways of doing this there is a manual method of training the pixie - which doesn't even require it to be hooked up to a computer it just needs to have a power supply and this is a method using the push button on the top of the pixie as well as the LED that is on the and I'm going to demonstrate that to you in a moment there is another way of doing it and that's using the pic C Mon software that we were just looking at and you can also use the two in combination training it manually while using the pic C Mon software to see how the training is actually working so let's take a look at how the manual training works and then we'll train a couple of objects to be recognized by our pixie to camera manually training the pixie to involves two of the components on the pixie 2 device the push button on top of the pixie 2 and an RGB LED mounted in the center near the bottom of the pixie 2 in order to train the pixie two you first push the push button and observe the RGB LED after a little bit of flashing it'll start cycling through different colors and these colors represent the signature you are going to train they are as follows first a red then an orange then a yellow a green a cyan a blue and signature number seven which is violet once you get the correct color release the button and the pixie two will now start to learn the object when you've released the button the pixi 2 is now in light pipe mode the fix e 2 will try to lock on to the object that it sees in the center of its image the RGB LED will glow trying to match the color of the object that it sees while in light pipe mode a brighter LED means a better lock once you have a good lock you press and release the button again to learn the object and you have now trained your pixie to now the first object that I'm going to train my pixie to to recognize is this yellow golf ball I'm going to use the manual method that we just illustrated although I'll also be using the pixie Mon to kind of verify I'm walking on to the correct thing instead of using the LED on the pixie - you can do either although when you're first getting started it's advisable to use pixie Mon because it takes a little bit of a technique before you get this correctly working when you just have the pixie in a standalone mode so let's see if we can get the pixie to to recognize our yellow golf ball I'm gonna hold the ball about the center of the pixie which is a good place to hold it press it down and release it when it gets red and it's not sucking a bit under my fingers there no that's not too bad right there let's get okay and as you can see it seems to be locking in on the OL golf ball now it's got a label on it that says s equals one it actually might be nicer to label that something different so we're going to configure go to signature labels and for number one we'll just call it a golf ball and I'll hit apply and there I have a golf ball so it now recognizes the golf ball let's recognize something else only this time we're going to use pick C Mon to recognize it I've got a little blue flashlight over here let's put the flashlight down and try to Center it if one can in the screen because it makes it a little easier get a nice and flat to be be good now what you could do is go up to action I'm going to set signature to and they use my mouse to go over the area of the device and now it is set this is signature number two and again I can go in here and configure that as as a flashlight so now it can spot the flashlight pretty well anywhere in its field of view the same time it can spot the golf ball in fact they can spot a second golf ball as well now as you'll notice when I move the objects around the pixi 2 is tracking them and as long as they're in the field of view the pixi 2 cannot attract the objects so there we've trained our pixie 2 to recognize two different objects and as you can see it tracks the objects as they move through the field so that's great so of course this is wonderful for using pixie Mon and all that but as I said at the beginning of the video we would be using an Arduino with the pixie 2 so what we are going to do is see what we can do now by connecting in our dwee know now that our pixie 2 knows a couple of objects and see what data we can get back from it so let's go ahead and take a look at how we do that now connecting a pixie 2 - when our do we know is very simple thanks to the cable that was provided with the pixie - the cable connects to the SPI connector on your Arduino and the other end is connected to the i/o connector on the pixie now I've got mine connected to an Arduino Uno over here connecting the cable end onto the pixie is very simple because the cable is keyed on the Arduino end you have to just make sure that you oriented in the right direction now in an Arduino Uno it's set so that the cable is facing away from the Arduino if you are using an Arduino Nano by the way it's reversed and the cable will be facing toward the interior of the nan but other than that the hook-up is very very simple and you don't need any jumper wires or anything so hook up the pixie to your Arduino and also hook it up to pick C Mon so that you can monitor what's happening on the pixie now the pixie comes with a library for the Arduino that you can install into your Arduino IDE it's available as a zip file once you install it there are a number of different examples that you can run and we're going to use the CCC hello world example as the first one this example is going to look for some of the blocks or some of the objects that we have trained the pixie to look at and it's going to provide some information back to the Arduino so let's take a quick look at that code right now and then we'll see how it works with my pixie now I've got the code over here and it starts off with a number of just comments etc but the licensing on the code etc and then we include the pixie to library now we create an object called pixie and then we go into our setup and we set the stereo port now notice the speed that we're setting the serial port up act it's going at this high speed because we're going to be getting quite a bit of data back over here so 9600 baud probably won't work after that we just print the line saying that we are starting with a carriage return character at the end of it and then we initialize the pixie to now in our loop it's actually quite simple we're going to go grabbing blocks blocks are actually detected items so for example in our last example every golf ball or every flashlight we saw would be considered to be a block now it uses the get blocks function now if there are blocks detected this number ox function will return the number of actual blocks and so if there are blocks we're going to print some details about them so we just print the word detected then we print out the the number of blocks that we've actually detected and then we go into a for loop for every block that we detect we print out the block we print out the number of the block and then we actually print out this array over here called blocks and this array contains a wealth of data that we're getting back from the block and that's the sent to the serial monitor and we just do it over and over and over again so the code is actually fairly simple so now that I've got it set up over here let's actually take a look at it running now for our demo I've also activated pick C Mon so we can see what the pick C is seeing while we observe the data that's coming back from the arduino so i've got them both on the same screen here and i'm going to put one of the objects that we've detected this golf ball and put it down and as we can see we're detecting the golf ball now let's take our flashlight put it down over here and now we're detecting two objects the flashlight and the golf ball and let's take our third golf ball place it down here and now we have three objects so let's take a look at some of the data that we're seeing now one thing you can do with the pixi is you can just hit the stop button and the data will stop flowing and it makes it a lot easier to take a look at what you're seeing so let's go take a look at some of what this data actually says now we're detecting three blocks over here which is correct we've got two blocks with your golf balls and one with it's a flashlight block number zero has a signature of one and if you'll recall signature one was our golf ball now these are the x and y coordinates of that particular golf ball block number zero here's the width and height of that particular block the index number is a unique number that is assigned to every block so if you look down here block number two is also signature one but it has a different index number so these are two independent objects that happen to have the same signature now this age perimeter over here is a number that goes from 1 to 255 and when an object first appears in your screen it's assigned on the first frame an age of 1 the second frame and age of 2 etc and it goes up to 255 so this is something you can use when you're tracking when an object actually entered the screen which may be useful information for your program let's take a look at a couple of other things that we can do over here we'll start our program gain and now going into pixie Mon right now I'm viewing both the blocks and the actual video but if you want to you can just view the blocks and this way it makes a display that's a little easier for you to actually see because you're not being distracted by all the external video in the background it also makes the pixie run a bit faster because it doesn't have to dump the video back down on the output as well on the other end you can do this blocks video and detected pixels and this is showing you the view we had before along with the detected pixels what the pixie is actually locking into and if you find that it's actually not locking in correctly you can go down here in the file and configure and start adjusting these signature sensitivity pots over here or sliders if you wish and use that to fine-tune the pixies detection capabilities so if you're having a lot of false triggers and such this is a good way of rectifying that so as you can see using the pixie with the arduino to detect objects is actually very simple thanks to the code library that they provide to do that so there's another mode we can use the pixie in as well and that's doing things like line following and barcode reading and so I want to take a look at that right now the pixi 2 is capable of advanced line tracking the device will look ahead to anticipate curves and intersections on the line it can be used with dark lines on a light surface or light lines on a dark surface the pixie 2 is also capable of detecting intersections between lines the line that the pixi 2 is currently following is called a vector the pixi 2 determines the best vector candidate from a group of lines it will report the start and the end position of each vector in every frame this vector Direction can be used for navigation the pixi to can detect intersections the device can automatically determine which way to turn you can also make turns programmatically the pixi 2 can handle a 3 4 or 5 way intersection the device can also read 16 simple barcodes you can download these barcodes in a PNG format these barcodes can be used as inputs or Direction indicators so now that we've seen how the pixi 2 can handle line following intersection detection and barcode detection let's run a simple Arduino sketch to exploit those things now this is another one of the sketches that came with the library for the Arduino that I loaded earlier so let's take a quick look at the code it starts off again with some licensing information after which we include the pixie library and define an object that we're going to call pixie in this setup we setup our serial monitor we initialize the pixie and this line here changes the mode of the pixie into line following mode then we go into the loop now in the loop we use get main features of the line following mode in order to get all the parameters of the line following mode and deliver back now if we happen to have vectors we'll end up printing the vectors over here if we detect intersections we'll print them over here and if we detect barcodes we will print the barcode number over here so once again using the pixie library the code is very easy so now that we've seen that let's actually do a demo and see how it works now those of you with acute observational powers well notice that I've placed a white sheet of paper against the wall over here in the field of view of my pixie - now the reason I did this is because when we're in line detecting mode the pixie too will be looking for lines everywhere and every junction in my workbench every object that it sees will actually have a number of lines on it now in the real world this isn't really a problem the pixie too can just simply lock on to a vector or a barcode or an intersection and ignore everything else but on the pic seam on display at least in its default mode you will see all the other lines and it makes it a little bit busy so I just put the piece of paper there simply to make the display on pick seam on a little cleaner now I'll also show you a method on pick seam on of eliminating all the extraneous lines without having to resort to something like so but any rate what I'm going to do is I've taken a piece of paper and I've drawn a couple of lines on it so we'll see what happens is when we track the lines and there's an intersection in yours you'll notice I've also got this printout of the barcodes that the pic c2 is capable of reading there are 16 of them labeled number 0 to 15 and we can put this in front of the screen and make sure the depicts you can capture these correctly so let's go and take a look at how this works we'll start off with my line and so here again I have picks Iman and my serial port monitor in the same place and as you can see it's caught on to a vector it's found a 4-way intersection caught onto a different vector now okay we've got an intersection in there let's just stop this for a second just so we can examine what the data actually looks like let's go back up so we see an intersection as well there okay first of all for a vector we get the coordinates of the start position of the vector and we get the coordinates of the end position of it every vector just like the blocks that we were doing or the objects we were doing in the previous display gets a unique index number so we can be looking at more than one vector and they'll each get assigned a different number here picked up a different vector over here this thing that says flags just simply indicates that there are additional parameters that we can get if we are in advanced mode called flags and there are actually four of them for every vector but in the simple mode we are right now we don't display what those flags are because we've got the basic information over here we go to an intersection we find the coordinates of where the intersection actually is the center of the intersection and the intersection in this case has four different lines protruding from it labeled zero to three each one of them gets a unique index number and the angle that what is going now this one was going straight ahead so it's got an angle of zero but this is in degree so negative 115 degrees positive 74 degrees and positive 157 degrees as well as you see it does a very good job of detecting both vectors and intersections now it's clearly output here start to program again we'll give it our barcodes and as you can see it is picking up on the barcodes they're actually upside down let's try it right way up that's better and there we go now again let's look at the data that we get back for a barcode and it's all fairly basic we get the coordinates the x and y-coordinate of the barcode we get the value of the barcode here's a SiC here's a 10 here's a 6 here's a 12 and there are no Flags included with barcodes so very basically it will read back the value of the barcode and the location of the barcode now one thing I want to show you you remember I told you that everything got kind of busy when I took my piece of paper away and as you can see it does there are all kinds of lines and things even my hand will give you a lot of data but there's a way to cut a lot of that down and one thing you can do is go into view and you'll notice the options in line fall and mode have changed right now we're at all features but they do primary features no background it's just simply picking up on what it thinks is the vector I'll put my sheet of paper in front of it and then you can see it's actually only sending back data that's useful and it's filtering out everything else is extraneous so for purposes of programming this is very very good now under view and set it down to primary features primary features again cuts out all the extraneous lines but it does show you video in that so you can see the video of where the vector is and then go to all features it starts picking up on basically everything so as you can see the pic c2 is quite capable of detecting lines intersections and barcodes and of course for a robot this would be wonderful to make a line following robots that could respond to barcodes you can use the barcodes to tell the robot to turn left at the next intersection because it looks ahead it is much superior to a line following system that simply follows the line that is under or immediately in front of the robot it's like when you drive down the street you're looking ahead to see if you need to make a left turn or a right turn down the road well the pic c2 is capable of doing that as well so for line following robots this is an ideal component and so that wraps up my quick look at the pic c2 but if by no means wraps up what you can do with the pixi - in fact I've only scratched the surface of the capabilities of this amazing little camera module now I'd like to thank the folks over at DF robot for sending this to me I look forward to incorporating the pic c2 into some of my future robotics projects now if you'd like to learn about some of those future robotics projects I would urge you to subscribe to the YouTube channel so if you haven't done that already please hit the subscribe button below this video and become one of my subscribers I would very much appreciate that I'd also like to point out that there is an article that accompanies this video as always it's on the drone bots workshop comm website and there's a link below the video to that article and while you're on the website please consider subscribing to buy newsletter this is not a sales newsletter but it's just simply a way of keeping in touch with you letting you know what I'm doing in the workshop and occasionally surveying you to find out what things you would like me to build for you or talk about on my videos and in my articles so until next time please take care of yourself and I hope to see you again very soon here in the workshop good bye for now [Music]
Info
Channel: DroneBot Workshop
Views: 288,008
Rating: 4.9316969 out of 5
Keywords: pixy2, pixy2 camera, pixy2 tutorial, dfrobot, arduino, pixycam, cmucam5, cmucam5 pixy, cmucam5 pixy arduino
Id: 391dXDjqzXA
Channel Id: undefined
Length: 38min 38sec (2318 seconds)
Published: Fri Oct 19 2018
Related Videos
Note
Please note that this website is currently a work in progress! Lots of interesting data and statistics to come.