Easy Machine Learning on Arduino/Raspberry Pi with DFRobots HuskyLens!

hey welcome back to geek toolkit i'm joe farrow and this episode we're going to talk about df robot's husky lens sensor this is a sponsored video dfrobot reached out to me and they said hey would you like to do a video about this and when i did some research i was like absolutely you see this is a very unusual device and on geek toolkit if you're new here i like to talk about gadgets talk about home automation and some of this stuff is diy and we build it out of arduinos raspberry pi's things like that this sensor adds the ability to do machine learning to those devices in a very unique way i'll talk a bit about what machine learning is and what a model is because that's very important to understand if you don't have that background then it's gonna be hard to understand why this is unique so i'll talk about that and why it makes this so easy to use and then we'll talk about the hardware of this sensor we'll talk about the models that are on here and how they work and then we'll connect it to an arduino and talk about that real quick to talk about the sensor itself a machine learning vision sensor is a sensor that actually has a camera on it so that all of the stuff that's needed is included on this little package right here the camera is a 1080p camera and it's able to take that data and pass it to a processor that's on the board the processor on the board allows you to do machine learning tasks such as facebook recognition object detection line recognition color recognition and things like that before we get into the hardware let me just take a quick minute to talk about what machine learning is machine learning is a foundational technology for artificial intelligence and what's really mind-blowing about it is it is a formula that can actually get smarter so in computer science we call these algorithms and an algorithm is just anything in a computer that helps solve a problem so you can think about alphabetizing a list of names that would be a problem there be algorithms of how to do that and some are faster some are slower now machine learning algorithms are interesting because you can give them data sets and the more data you give them the smarter they get or they appear to get it's really the accuracy goes up an example of a machine learning problem is say that you want to detect cats in an image the way that you would approach this with machine learning is you would create your data set now your data set is going to be in this case where since we're doing image-based recognition it would be a bunch of pictures of cats there's actually data sets available on the internet for all sorts of machine learning things and i'll show you one here of some cat pictures the thing to notice is they're all very different there's different breeds different camera angles different lighting different parts of the cat and the whole point of this is that when you feed this data set into a machine learning algorithm then it actually analyzes these images and comes up with a concept of what is it that makes a cat a cat then out of the end of this that data set plus algorithm is what your model is okay so if i have a model then i can then use that model and apply an image to it of a cat that it's never seen and it can detect the cat this is what machine learning is and this is why it's so important to understand what a model is now back to this sensor this sensor has several different models on it the reason that's important is a lot of sensors that are doing machine learning for an arduino you actually have to go program your own model which means you have to come up with your own algorithm or you have to utilize an algorithm and you have to train it and so on now this is a different approach and there's pros and cons to anything here this basically trade-offs that you're dealing with but at the end of the day the biggest benefit of this is incredibly easy to use since they've already given this thing the training data and the models then the things that this does like facial detection it does very quickly very efficiently and when you go to use it you don't have to deal with any of that back-end machine learning stuff you can simply say i want to use this model take the camera image point it at a face and it will tell you okay i see a face here's where the coordinate is here's the what's called the bounding box that will draw a box around the face and it will tell you the coordinates of that bounding box of where the face is in the image i don't know of a lot of devices that do machine learning for things like an arduino but is definitely a differentiator between this device and say something like the arduino nano 33 bld sense which i didn't name um i i don't know if i agree with the marketing on that name but uh that is a device where you would use a tensorflow model you train your own model you can upload it to the device and uh use it as an arduino device now another differentiator between this is this has the processor on it it does all of the processing it's a dual core processor it's quite fast in matters of fact you can actually do facial detection on like a moving video for instance with this which is pretty wild but because it does all the processing it actually exposes everything through a uart pin or an i squared c uh it's on the bottom here and it's based on basically how you select it that's really powerful because that means that this would work with not only an arduino but a raspberry pi a latte panda micro bits like quite a few different controllers because of the way this was implemented so i thought that was really really clever what we're going to do now is we're going to do a hardware walkthrough of the entire device so on the front here you've got the camera and on either side you've got a white led that helps light up the camera when you're doing machine learning stuff you really want to have a decent brightness or lighting so having auxiliary lights is kind of nice there's also a rgb light or a red green blue light on top that can be different colors this is really good because it can indicate what's going on with the device in the sensor on the flip side here you've got a two inch lcd screen now this screen right here is 320 by 240 resolution this is used to help kind of decide what you care about when we'll talk about that when we show the model demos later on on the top here is a like a scroll wheel that goes back and forth but also pushes in and then on the left side is a button this button has both press and hold and press functions this is basically how you teach it what again what you care about so if you're doing facial recognition you want to do identification i could point the camera at somebody and click the button and say okay i care about that person and we're gonna do a little demo of that in a second on the bottom you've got two power ports here one port is a micro usb this is to provide power this device here when we do the demo i'll show you this right here will let you walk through all the algorithms and and basically set it up without anything hooked up to it so you don't have to have an arduino you have to load up code you can just power this thing on and start using it immediately the other port though is your uart or your i squared c port again selectable in software so that you can get data off of this and down to arduino here's how this thing works you use this to decide what you care about and you select one of the models now the models are you only select one at a time so you can't do facial recognition and line recognition or facial recognition and object recognition you say okay i want this thing to be a facial recognition sensor i want to use that model and then you set it up to basically register faces you care about and when it detects those faces it sends a data stream down off of this uart to the arduino that you can listen for and then process on the arduino code side the beauty of this is once you pull in their libraries what you get back in that data stream is literally do i see a face and you can have uh you can register multiple faces and say these are different face ids and in the arduino code you'll get i see face id one i see face id two or i don't see face id at all or i see four faces but none of them are the ones you registered that makes it incredibly incredibly easy to program on whatever you're using arduino's raspberry pi it's in python they have python libraries whatever you're programming in it makes it super easy because you don't have to deal with any of the complexity of machine learning and to me i think that's the brilliance of this device don't get me wrong if you're somebody that's deep into machine learning you're looking to learn you're looking to build your own models you want to recognize you know the things that you care about then writing your own models and such there's going to be boards out there that can do that but for hobbyists that are just trying to apply that or add that to their device as simple as possible this thing is really really incredible and impressive the other thing i really like about this is because the processor and everything is on here i don't have to worry about overhead or threading or any of the complexities of dealing with what's going on here i can just read this as a serial stream like any other sensor and decide what i care about that makes it really really easy to incorporate in the projects so some really cool things about the design the next thing we're going to do is really the exciting part we're going to go through the different models on here and i'll show you demos of each one i'm going to try to make it as fun as i can these demos will really show you the power of this and what it can do now that i've explained how the hardware is set up how to use it and how it gets data off of here all right one of the things we're going to show here is object classification so i'm going to bring up object classification and what we're going to try to do is determine which can we're looking at here so we're going to have a couple of cans lined up and so what i'm going to do is press the learning button and i'll rotate this can as i hold it down so it can learn it from different angles and then we're going to bring this next can in and you can see it already turns green it already recognizes that it's a different can and then we're gonna bring in the third can and we'll learn this can you see it's orange now it sees that it's something different and we'll learn one more character just to kind of give this thing fully tested here so we're going to bring in ryu okay and now i'll click the other button there we go now it's ended so now we're going to do is we're going to do a line up of these characters and then i'll just pin from left to right and show you so there we go object two object one object three object four and object five that's crazy that was it took that long to differentiate between five objects you can see it's just switching fluidly right between them there that is crazy but it's crazy powerful the next thing i want to show is facial recognition and the first thing i've got is i've got the cast of marbles one division up here and we're going to recognize these faces and then we'll do a demo on that you put the crosshair in front of the face and click on it and then i'm going to click to recognize multiple faces so we've got elizabeth olsen recognized we've got kat dennings recognized we'll do paul bettany tayana paris and catherine okay there we go all right so i had to redo paul but you see the but based on the colors here it recognizes them as different and so now for the next demo we're going to do is go through a bunch of elizabeth olsen faces and see if it recognizes her with different hairstyles different poses different lighting and also does it recognize her as any of these other characters that it's stored right now so we'll do that right now [Music] [Music] bye [Music] okay so we're going to do the same thing for object recognition here i've said it's object recognition mode and we're going to see how this does so okay so it sees a boat in that picture dog good dog um this one it sees a chair sees a chair okay that is a chair people good it's these people see is the cat there still thinks there's a chair airplane cat cat chair good job people more people so this is pretty impressive it's got 20 objects that can differentiate between and the nice thing about this mode is it's just doing it based on what it sees so what i want to show here is this is what it looks like when it's recognizing a video in real time you see that it's doing a cat here and as it transitions to other cats you'll see it's quite quick it recognizes it in a couple of frames and it holds pretty steady now it's not a hundred percent there are things that it sees sometimes it'll see a bird or a human uh based on how things look but just the fact that i'm holding a machine learning device in my hand that is this quick and that's doing this well is still mind-blowing to me it's it's just absolutely amazing the other thing that kind of gets me about this is that it's doing it on live video is really amazing just to kind of show you what the what it looks like when things don't quite go 100 it is recognizing that there's something there but it is misrecognizing that and this is what i was talking about my facial recognition test that's important when it's when it misrecognizes but keep in mind this is the same mode that was doing humans it does chairs it does cats and dogs so it's got quite a wide variety of objects it's trying to recognize and also from different angles so i still am very impressed with this uh despite despite having a couple of glitches here and there here's a sample of the line following uh let's see it's a little bit there we go it's a little bit better when it's got a clean line to follow but it can find a line and guess where that line is going to go so that's just a quick sample of that one there here's another one this is actually color recognition so i can lock onto a color like this orange here and it recognizes it and then it doesn't care about any of these other colors but when that one comes back in line it immediately alerts on it so you can use colors to actually trigger things or look for other colors and so on so that's pretty cool for the tag recognition feature you have something that looks like these these are what are called april tags and you see these squares on each of these robots now to do vision based imaging on these robots wouldn't make a lot of sense they all look identical except for the tags by putting these squares on there and then blocking out some of them you can get a unique identifier now typically the way these are red is you know if you go left or right top to bottom a white square is a zero a black square is a one and then at the end you would have a unique binary number and that could be the tag that's there so what i've done is i've actually already done the recognition here and you can see that it's able to discern different robots based on their let's say i memorized three i think over here i got another one i just learned another one so there's tag id six tag id one tag id2 and tag id3 so it's actually differentiating these robots even though they all look identical just based on the tags tags are also useful for things like if you want a robot to know what room it's in uh you know detecting a room can be really hard but you could put a tag in the room all around areas that the robot can see and then as the robot's going through the room it would know its room it's in nasa is also using april tags they do things like put them on the ceiling and their robot can actually see where it is in the room by analyzing the tags and so on so these are called april tags and that's what kind of things you do with these the next thing we're going to do is track the space shuttle landing because there is an object tracking thing on here there it is object tracking and i thought wow it'd be cool to actually see it follow the space shuttle down this is an old nasa video spatial landing we're going to click our learn button up here to lock onto it there it goes and we're going to let this video play and see how it tracks along this video here so it is now the video is a bit jerky i'm going to hold this as still as i can but really this is a video of a space shuttle coming in so it's gonna be a little bit a little bit jerky on its own but you can see it followed it pretty close down until it became a little dot now we're going to see where we pick it up here we're not going to pick it up there that's a back angle and the machine learning algorithms the way they work it kind of needs something that looks similar to what it memorized right so oh there it goes wow look at that i didn't expect it to pick it up from the front there but it and that's interesting i did expect it to pick it up there there it goes so not bad not bad it really locked onto the side view there that's a demo of the object tracking functionality okay now let's talk about how to get the arduino portion this up and running this is going to be very simple matter of fact it's almost deceptively simple when you realize how this gets put together first thing you want to do is go to github.com husky lens you're going to want to grab the arduino code now if you're on a raspberry pi there's a raspberry pi uh or i'm sorry a python setting here that you juice raspberry pi there's also the firmware up loader what we're going to do is here's the arduino api and it basically tells you that when a block comes in which is a like a frame of recognition this is what you're going to get back the center of the block the width the height and then the id of the block which is really really important we're going to download this here we'll download this code and we're going to put that into our libraries for arduino now in windows this is under user account arduino libraries if you did a default install we're going to go into husky lens create a folder there and then dump that code into there once you have that into there you start up arduino you will have the intro basically um there's gonna be let's see here we'll move this up i'll show you the examples that get pulled in here so under examples you will have a husky lens folder and you'll have these examples here the get started one is the one we're going to start with here so we're going to go with get started and get started is a very very simple we'll actually do a code walkthrough here but the first thing you want to do is get everything set up as far as your device so make sure that your ports are correct i'm using an arduino leonardo because my next tutorial we'll talk about doing some really cool stuff with that and subscribe for that one that's going to be where we actually use facial recognition to send stuff to arcade cabinets things like that select your board and then make sure that you are correct set up i like having the serial monitor up because you're going to want it when you are talking to this board now i'm going to bring the camera up so you can see what's going on here there's the camera here's there i am and here's the board okay so now that we've got some faces up we're going to show you what happens so it says nothing learned press learn button button on husky lens and what we're going to do is point up to the faces that we want to recognize okay so at this point what i'm going to show you is the serial monitor output here i have it not scrolling but it says no block is on the screen i've actually i'm using the same data set we programmed earlier to keep things consistent for faces so elizabeth olsen cat dennings and so on so what i'm going to do is i will point the device up and start recognizing faces and then i will scroll down and we should see there we go we saw the data come in here in the serial monitor so now what we can do is copy this and i'm going to basically make it bigger for you let me show you again this is this is the page i was using i had these two on screen and you can see it saw id one id one is what it registered elizabeth olsen as and id2 is what registered kat dennings as so what you're seeing here actually if you look at it closely is it saw both of them in this frame and this is basically like almost like a frame of video in this frame of video at about the same y about the same height and about 140 pixels 160 pixels over it sees kat denning so it's showing id1 was here and then about the same height over to the right was id2 and that's exactly what we're seeing here and then we're seeing it consistently all the way down so that is basically how you would do facial recognition so how would you handle that in code if you wanted to do something well let's do the code walkthrough real quick we've got our headers our setup is basically going to set up our serial for talking to this device and again this is off the husky lens uh get started here so when it sees the husky lens and it gets some data back from a read then it creates this result object and we talked about the result object earlier that is going to be the thing that has the center point the x and y the width and height of the bounding box and that also important id right there so if i wanted to do something when the face id 1 was recognized i could just say if result id equals 1 then go do something cool and that's exactly what we're going to do in our next video that's why i want to get you here because now you know how to recognize faces you could do the same thing with the object recognition which we did earlier or any of those tutorials i showed the only one that's going to be really different is going to be the line following and i won't get into that one in my video series but if you're interested let me know and i could do something on it but other than that i think the object recognition is the one that i'm very excited about and face recognition because we have a really cool arcade project i'm going to show you in the next video with this okay uh let's do the outro thanks for watching that thank you for watching this episode of geek toolkit i'd like to thank df robot for sending this device out from here you're probably wondering where do i go with this well this is geek toolkit and when i'm not talking about gadgets i'm building them and so of course i've got a gadget in mind for this if you're following me on instagram you saw a preview of where i'm going with this facial recognition in arcade cabinets is an interesting combination i've got to do one video about the arcade cabinet part that i want to do that way people can understand how i did the theming and so on and then when i combine that with this sensor i'll do a third video in this series that will actually combine both of those videos together into one project and i will show you the power of facial recognition in an arcade cabinet being put together i think that's going to be a really cool little series that i'm putting together here this is something new i'm trying out in the channel of splitting up my videos into segments so that i you know there's going to be people that don't care about arcade cabinets that care about arduinos there's many people i care about arcade cameras that don't care about arduinos and then there's going to be people that want to see projects and i want to basically have a video for each of those people in the audience the people that want to see the project so i think you're going to get the best show because this was the intro this was the warm-up the next one will really show you some cool stuff and then the final product i think is going to be mind-blowing so i'm very excited there thank you so much for watching the support if you have any questions let me know down in the comments i'll link this uh to everything that's related to this in the library's down in the description as always and i'll have my instagram link there if you want to follow previews of what's going on for my videos thanks for the support i'm joe faire with geek toolkit and until next time [Music] you

Channel: GeekToolkit

Views: 6,312

Keywords: arduino, robotics, face detection, robot, ai vision sensor, ai camera, huskylens, husky lens, husky, lens, lense, huskylens tutorial, AI sensor, Face recognition arduino, raspberry pi, object tracking, object tracking arduino, ai line follower, arduino AI, arduino machine learning, arduino tutorial, arduino ai tutorial, electronics, machine learning, neural network, arduino project, how to, programming, beginners, introduction, easy, arduino programming, arudino code

Id: AgU_lKmhBn0

Length: 24min 39sec (1479 seconds)

Published: Tue Mar 09 2021