Machine Vision with HuskyLens

today in the workshop we are working with the husky lens an inexpensive ai powered machine vision sensor we'll see how we can use all of the features of this amazing device both by itself and with an arduino it's an ai opener of an episode today so welcome to the workshop [Music] well hello and welcome to the workshop where we've been working on the edge lately and what i mean by that is edge computing devices now edge computing devices are iot or sensor devices that have intelligence themselves so they can not only collect data they also pre-process the data for you and that alleviates a lot of the tasks off of the host microcontroller or host microcomputer now a good example of that was the ultrasonic sensors that we worked with recently they not only collect the data they actually process it and give you back a reading of how many millimeters you are distant from the object that you're trying to measure and so this is a good example of pre-processing and edge computing well another bunch of edge computing devices that i've been working with lately have to do with video recognition and i've got a few from a number of different manufacturers now all of these devices have a few things in common all of them obviously have video cameras they wouldn't work without video cameras they also have small display screens so that you can work with these devices without having to attach an external screen or computer to them and one thing they have in common as well is that they're all based upon an incredible microcontroller something called the kendrick k210 now calling the k210 a microcontroller almost seems kind of weird considering the specs it has this is a dual core 64-bit unit it has an integrated floating point unit as well and together they form something that kendrick calls a kpu which is useful for neural network development now these devices are specifically targeted for both video recognition and for audio recognition they have inputs for up to eight different array microphones and we're going to be using them in their video application today with a device from df robot called the husky lens and so let's go and take a look at the husky lens now and see what it can do the husky lens is an artificial intelligence machine vision sensor it's available in two models the regular husky lens with a 2 megapixel camera or the husky lens pro which has a 5 megapixel camera both models feature a 2-inch ips display the husky lens operates on either 3.3 or 5 volts making it suitable for virtually any microcontroller or microcomputer the interface on the husky lens can be operated as either a serial uart or an i2c mode the husky lens can be used in standalone mode for training purposes without connection to a microcomputer or microcontroller the device also has a microsd card that can be used to save training models or images from the camera let's take a look at the husky lens in the center of the husky lens we can see the camera this is either the 2 or 5 megapixel device depending upon which model you've purchased above the camera we see an rgb led this is used to indicate status during training mode and is useful when you are using it in selfie mode in other words trying to do face recognition on your own face where you can't see the screen on the back the selector wheel is used to navigate through the menus on the husky lens it can be moved to the left or the right and can be pushed down to make a selection the training button is depressed when you want to train the husky mode to learn an object there are two white leds on the husky lens these can be used in line tracking mode to illuminate the subject for better performance below one of the white leds you can see the micro sd card slot the micro usb connection on the husky lens is used to supply power the 4-pin data connector is a jst connector that's used for both data and power you can have power on this as well as the micro usb connector if you've powered them both then the micro usb will be used to power the device the two inch screen on the husky lens measures 240 by 320 and is divided into the following coordinates these are the coordinates used by the husky lens when doing object tracking or line tracking in face recognition mode the husky lens detects face contours you can save the detected faces and you can track those faces as well the husky lens is capable of working with single or multiple faces with object tracking the husky lens can learn and track a specified object only one object can be tracked and multiple objects are not supported in this mode the husky lens also supports the detection of gestures in object recognition mode the husky lens can recognize some predefined objects the device has 20 different objects predefined you cannot however add any new objects it also can't distinguish between objects of the same type for example two pictures of different cats will be recognized as a cat in line tracking mode the husky lens can track a specified colored line it's capable of working with single or multiple lines it's also capable of making predictions as to which direction the line will continue in this mode the white leds can be used to illuminate the lines for better performance in color recognition mode the husky lens learns and recognizes specific colors this can be used to track colored objects the accuracy of color recognition mode can be greatly affected by the ambient lighting the device also has problems with two objects with similar colors in tag recognition mode the husky lens can detect learn and recognize tags the device supports april tags a tagging mechanism developed at the university of michigan both single and multiple tags are supported in object classification mode the husky lens can be trained to recognize different objects multiple objects are supported for example apples oranges pears and bananas unlike object recognition mode however the husky lens does not report the object's position and will only report one object at a time the data connector on the husky lens is a 4-pin jst connector which is used in both serial uart and i2c mode this chart shows the pin outs of the connector in both of those modes so now let's take a closer look at the husky lens now here's what you get when you purchase the husky lens and it actually comes in two separate boxes this box contains the husky lens itself the cable and the mounting brackets that you get over here and this box contains a little silicone holder that you can use to put on your husky lens to cover it up and it's got a hole for the camera in it plus it also has two little ears on it so it actually does look a little like a husky now the main component of the course is the husky lens itself and this is the husky lens you can see the camera on it over here here is a push button and here is a rotary dial that you use as a function thing to navigate through the menus and select things over here we have a micro sd card slot and the micro sd inserts from the inside over here if you're going to be using it this is actually the kendrick k210 processor over here by the way and down in the bottom we'll find a couple of connectors this connector here is a micro usb and it is used just for power and this one over here is a four pin jst connector and it's used for power as well as data and it supports both uart serial data and i2c data using the same connector now they give you this cable that will plug into there and the cable is terminated with a couple of dupont female connectors over here and so you can use that to connect the husky lens to your microcontroller or microcomputer now when we flip this over what we'll see on this side is a screen and this is a color screen it's a two inch screen that displays everything the husky lens is seeing and also the menus so that you can work with it now when you first get it you can power it up right away so i've got uh usb cable over here i'll connect it to my husky lens and as you see it boots up and it comes up right away and starts working and you use this dial over here to go through the different modes that you can get in and when you get to a mode that you want to be in you just press the dial the thing down over here and now it's going to be in that particular mode and you can use the button on this side for training it and i'll show you that in more detail in a few moments before using the husky lens you'll want to check the current firmware version as chances are that you'll need to update it use the dial and move to the general settings and then press down to select the lower menu use the dial again to move all the way over to version and check your version number now mine 0.5.1 a which is a recent version but if yours is earlier than that you'll need to do a firmware update although you can do this update under any operating system it is much easier to do it using microsoft windows to start you'll need to download a universal windows driver from silicon labs the link for that is in the article accompanying this video you'll also need to go to the husky lens github page and download two different things first of all the husky lens uploader for microsoft windows and secondly you'll also need to download the latest version of the huskyware firmware after that you can install the universal windows driver that you downloaded earlier it's a simple windows installation program and then you'll want to connect the husky lens up to your computer using the micro usb cable and look in your device manager to make certain it is seen look for a silicon labs device the next step is to run the uploader itself in the select file box you're going to want to select the new version of the firmware that you downloaded and then just click the upload button it will cycle several times between zero and a hundred percent and the whole procedure takes about five minutes but once you're finished the firmware should be updated and you're ready to use your husky lens so now that we understand how the husky lens works it's time to actually start using it so what we're going to do next is go through all the different modes of operation that the husky lens is capable of performing now you'll have to have updated your husky lens to the latest version of the firmware in order to follow along because some of the modes of operation were only available with the latest versions of firmware so make sure you go ahead and do that otherwise all you're going to need to do is to power up your husky lens with the usb cable it doesn't necessarily have to be connected to a computer it can just be a usb power supply so let's work through all of the different modes of operation starting with face detection the default in face recognition mode is to recognize a single face and it's quite easy to do just dial face recognition from the top dial focus on the face and press the learning button if you hold down the learning button while the face is moving you'll get better results as it can learn all the different contours of the face now this face has been recognized and other faces will just come up as face however in default mode you can only recognize one face and you have no option to recognize other ones to do that you'll need to put it into multiple face mode first you'll need to delete the face you've already learned so get it back in the screen click the learning button once and then click again to forget the face now you're ready to set it up for multiple faces and face recognition mode press down the dial and navigate to learn multiple on the bottom screen you'll note that there's a toggle switch that you can toggle now using the dial toggle it into learn multiple mode and press it down to save the value then go to save and return and press down again to confirm you wish to save this now you can learn multiple faces point at the first face you wish to learn and learn it in the standard fashion after you've done that click the button again to indicate that you wish to learn more faces point at another face and once again you can learn it in the standard fashion if you're finished you can just leave it alone or do a quick press on the dial button and now the unit is capable of displaying and learning both of the faces in object tracking mode the husky lens can be set to learn and track a single object you'll need to dial object tracking on the top screen then press the dial down and navigate to learn enable set the toggle switch as we did with the previous example you'll also want to go to auto save and do the same thing with its toggle switch after that you'll navigate to save and return and press the dial down to confirm that now we can begin to learn and track our object place the object in the focus area and hold down the learning key continue to hold the key down while you look at different angles of the object this will give you the best results once you let go of the key it will still be learning the object even though it is already in its memory this way it continues to improve its performance note that right now we are tracking our learned object whereas the second object which is quite similar is being ignored in object recognition mode the husky lens has been pre-programmed to track from 20 different objects getting into object recognition is quite simple just dial object recognition at the top there are no additional setup steps after that just aim the camera at the object that you wish to track note that the husky lens will not distinguish between two objects of the same type so one person or two people will each just be tracked as a separate person it is however very good at tracking people and despite movements and things it does a pretty fair job of keeping them in track i tried the husky lens with a number of different objects it is capable of tracking and got reasonable results although i found with animals it was often confused often confusing a cow for a horse and the sheep for a horse or a dog it did a pretty good job on this video of a horse but on some of my other horse videos it couldn't distinguish them from dogs so it really does depend upon the source material that you're tracking birds seem to be particularly problematic often being confused for persons dogs or even bottles it didn't have very much trouble with a cat at least this cat with some cats that was confusing them with dogs but otherwise it did a pretty good job on that so your friendly feline can be easily tracked a fast-moving train did a pretty good job occasionally confusing it for a bus but admittedly this style of train is a bit unique all in all the performance here was pretty good i was particularly impressed on how it did persons and motorbikes over here now this is actually a bicycle built for two and so will go between bicycle and motor bike and that's understandable it's not a very standard type of bicycle but what really impressed me were these scenes of amsterdam with a number of bicycles cars and people it did a great job of tracking them no matter how many seem to be in the scene and i found this extremely impressive and so for object recognition this device does a pretty good job right out of the box the husky lens is capable of tracking single or multiple lines you'll get better performance with single lines however dial line tracking press down and then navigate to learn multiple you can turn it on or off as you wish also might want to turn on the white leds using the led switch this will give you improved performance when tracking lines press down to save the settings and then aim at the target line press the learning key and the arrow will turn yellow as it learns the line's parameters when you let go the arrow will be blue indicating that it is tracking the line you will notice that the husky lens is capable of tracking vertical lines quite well but it doesn't seem to work very well with horizontal lines if you're building a project like a line tracking robot however this is actually what you would want you would want to track a vertical line but wouldn't be interested in the horizontal ones in this respect the lusty lens performs quite well in line tracking mode in color recognition mode the husky lens is capable of distinguishing between one or many colors dial color recognition at the top and press down for the lower menu navigate to learn multiple and use the toggle switch for that go to save data and save your settings now focus on the first color you want to learn and hold down the learning switch press it again to learn a second color move over to that color and repeat the process you can continue to do this for as many colors as you wish in my case i'm going to do the four colors you see on the screen once you've done the last color you could even leave it alone or press down the dial button to complete the process now the husky lens can identify the four different colors note that the ambient lighting can really affect the performance of this but i thought it did a reasonable job of picking up most of the colors on this screen in tag recognition mode the husky lens has been programmed to recognize different april tags only april tags can be recognized dial tag recognition at the top press down the dial and then navigate to learn multiple to learn multiple tags use the toggle switch with the dial and then go to save and return and press down to save your settings now give the husky lens a number of tags that you wish to learn focus on the first tag and press down the learning and then press it down again if you want to continue on to another tag note that you don't need to press the button down very long to learn a tag as it's a simple two dimensional object continue to learn as many different tags as you wish you'll notice that the husky lens assigns a different id for all of the tags when you're finished learning the tags you can just ignore it or press down on the dial button to complete the process and now the husky lens has memorized the four different tags that i've learned note that the tags are recognized even when turned to different angles object classification mode allows the husky lens to learn a number of different objects enter object classification on the top press down in the dial and navigate to learn multiple select that and then save and return once you have set it up you can take your first object and place it in the crosshair area press the learning button as you learn the object from a number of different angles the more angles that you can learn your object at the better the performance will be here i'm learning a raspberry pi board from a number of different angles once you've learned all the angles release the button click again to learn a second object and repeat the process as i'm doing with this arduino uno board once again you're going to try to learn your object as many possible angles as you can for increased performance i found the quirk with the husky lens however in that it will always display the last object recognized when there is nothing on the screen in order to get around that i just learn a blank object like my background as i did over here once you're finished you can click the other button in order to save everything now the husky lens will have learned the object so as i place the arduino in front of it it will display that object id number 2 is being seen as you'll note with nothing object id3 was seen here it correctly recognizes the raspberry pi board as object id1 so now that we've seen the husky lens in its different modes of operation in a standalone mode it's time to hook it up to a microcontroller and for this we're going to use just a simple arduino uno now if you recall from the beginning there are two different methods of hooking up a husky lens you can use the uart method or you could use i2c and we are going to try both methods now we are going to run some very simple demonstrations and these demonstrations are included with the husky lens library that you can get up on github now this library installs in a little bit of a different fashion than what you might be used to with arduino libraries but it's not that difficult and i'll walk you through the whole thing so let's start off with our husky lens in uart mode for our first experiment we'll require an arduino uno and the husky lens we'll begin by connecting the transmit or green lead from the husky lens to arduino pin 10. the receive or blue lead from the husky lens will be connected to arduino io pin 11. the husky lens ground which is the black lead will be connected to the ground lead on the arduino uno and finally the vcc which is the red lead will be connected to the arduino's 5 volt output and this completes our wiring now the installation of the library for the husky lens is not as straightforward as some other arduino libraries but it's also not that difficult we'll begin by going to the husky lens github page and then selecting husky lens arduino once we're on the arduino page you'll see a code button over here and you can click that and download a zip file save it in your downloads directory and once it's downloaded you can open it with your extractor now once you've got your file extractor open you'll need to navigate within a go under husky lens over here the main directory and then there's another husky lens directory here right click on that directory and do extract now go and find your arduino library files you'll go into your arduino directory and then you'll find libraries and you can extract over here you can close once the extraction is complete and you can also close your web browser now open your arduino ide we'll expand that and then go into file go into examples and scroll down for examples for custom libraries and you'll now see husky lens with a number of different examples and this completes the installation of the library now we're going to use one of the example sketches provided in the husky lens library for our uart connection and the sketch we're going to use is one that they call husky lens get starred i actually think they meant to say husky lens gets started but whatever it is starter started we're going to begin with this sketch now the sketch starts off by including the husky lens library and we're also using the software serial library for our uart connection we define an object called husky lens with the husky lens library and we use the software serial in order to establish a serial connection on the two pins that we've connected the husky lens to our arduino with now you could use two other pins any pins other than zero and one which are already used by the hardware serial could be used you would just have to change the sketch over here and we also define a function called print result and we just do a blank definition of the function up here this is a common technique used with compilers like platform io which will give an error message if this function is called before it's actually defined now we go into setup we set up our serial monitor and we also set up our software serial connection at 9600 baud which is the speed that the husky lens works at and we take a look to see if the husky lens has been initialized if it has been initialized on that port we move on otherwise we print out some error messages and end now before we go into the loop let's move down to the function called print result we input a result from the husky lens and we use commands within the library to print that result back if it is a block we break the string down over here and print it out to the serial monitor if it is an arrow we break it down and print out over here otherwise we just print that the object is unknown note the use of the f macro during the prints the f macro keeps all the data in prog mem as opposed to passing it into the arduino's free memory and this is important especially with an arduino uno that only has 2k of memory and it could be easily flooded with all this data now let's go back up into the loop we check a number of connections first of all we see if we're getting data from the husky lens if not we print out an error message we see if we've learned something and if we haven't learned something again we print out an error message and we also see if there's something available in other words are we actually looking at something that we have learned and if not we'll print out an error message assuming we get past all of those points all we do is we print out a whole bunch of hash marks in order to distinguish this data from the previous line and then while the husky lens is actually available we will go and print out the results using the print result function that we just saw before so the sketch has been simplified by using the husky lens library let's load it up connect our husky lens and take a look at a few things so i've got the husky lens tracking of face that i've already memorized and as you can see in the serial monitor we're getting some data and the data gives us the location of the block that is tracking the face so if we had written a sketch that needed to know that kind of information we can now get it back from the husky lens and it's giving us everything including the id of the face which is face id number one so if we were tracking multiple faces we would get data from both of them and as you can see when the face moves the data changes to indicate the movement in the block and if we aren't tracking the face you will see that we get no block or arrow appears on the screen so it's pretty good at relaying the data back from the husky lens and displaying it on the serial monitor and as i said you could use that data in another sketch if you wished and so now let's go and do the same thing all over again but this time do it with the i2c interface now when you're using the husky lens with a microcontroller or a microcomputer you can use either the i2c or a serial interface and the serial interface is also available at a number of different baud rates now the husky lens is capable of auto detecting the protocol so as they both use the same connections it should just be able to figure out which one you're using and set itself up accordingly but it doesn't always work and so you can manually set the protocol as well so what you're going to want to do is going to go over to general so move the wheel here until you get to general and press down on general and you'll get protocol type over here and you can see that you've got a selection of a number of different serials you've got auto detect which is the fault serial 9600 which is what we're using in the experiments we're using today because we're using software serial so we don't want to go that fast but you've got a couple of other baud rates as well or i2c so if you see the one that you want let's say you want i2c you press down on the wheel and then go back to save and return press down again and say yes you wish to save that and then after you do that it'll just be set to i2c and you'll need to remember that if you're going to go and use it in serial mode after that and so just a handy hint in case you're having problems communicating with your husky lens for our next experiment we'll be using the same two components but we'll be connecting the husky lens using the i2c bus so disconnect the leads from the transmit and receive and reconnect them as follows the green lead which is the sda lead will be connected to arduino pin a4 and the blue lead which is the scl will be connected to arduino pin a5 the connection to the ground and the 5 volt will remain as it was in the last experiment and this completes the wiring of our husky lens in i2c mode now for our i2c connecting we'll use another example that's included with the husky lens library the husky lens i2c example and when you look at this you will see that this is pretty well exactly the same code that we looked at for the uart in fact there really only is one difference and is down over here since we're using an i2c connection we use the arduino's built-in wire library which we start over here and then we bind it to the husky lens object over here instead of binding the uart like we did in the last one otherwise this code is absolutely identical to what we looked at before so there's really no reason that we need to go over it a second time let's just load it up to our arduino and give it a quick test to make sure that everything is also working in i2c mode okay i've got the i2c demo set up and this time i'm going to be trying to track a line and the line that i'm going to be tracking is simply my own finger i've just used that and trained it to be aligned now right now you can see i don't have my finger in the display there's no lines on the husky lens screen and the serial monitor is just reading no block or arrow appears in the screen and so if i put my finger in there and i create a line you can see the arrow on the screen and you'll also see the data on the serial monitor and this time the data that it's giving is the origin and the target coordinates so the bottom and the top of the arrow and as i move my finger and change the direction and the size of the arrow you can see those coordinates change and again you could be using those coordinates in a sketch of yours something that is trying to follow a line for example could use those coordinates to guide itself and so once again another method of getting data back from the husky lens this time using the i2c connection now when we ran the husky lens in face and object recognition mode it assigned the label to every face and object that we detected and it usually went something like id1 id2 etc and while that's fine it's not particularly descriptive so now that we have an arduino hooked up to our husky lens we can go back and change those descriptions to something a little bit more useful to us so let's go and use our arduino with the husky lens to put a name to the face so to set up our experiment i've scanned four faces into our husky lens it's a little hard to get them all to display at the same time but they're currently face id's one through four now what we'll do is we'll run a little bit of arduino code that can change those face ids into something more memorable now here's a sketch that we can run to assign names to the faces that we've detected in the husky lens and you can actually use this for more than faces if you detected objects for example if you've classified them you can use this same method to name them you can even use this with the april tags if you wish it's a very simple sketch it starts off with the libraries we need which is the husky lens library and the wire library for i2c then we create an object called husky lens to represent our husky lens now i've defined the function over here and this function is actually where everything happens it's called set new name and it takes as input parameters the new name and the id number that we're trying to set the name for and what it does is it calls a method that's part of the husky lens library that does exactly this it's called set custom name and you insert the name and the id now the reason i've put this inside a function is i've it's enabled me to put it in this while loop because sometimes it doesn't work the first time and you need to delay and go and do it again so instead of just calling set custom name a number of times and inserting delays i found this to be a bit more reliable by just sticking it in its own function at any rate everything happens in the setup we start our serial monitor and the only purpose of the serial monitor is that the set new name function does use it in case the name setting fails so you can use it to troubleshoot then we connect to the i2c bus as a master by not providing any address inside here and we'll bind the husky lens to the i2c connection as we've done before and then we insert a short delay to allow everything to stabilize and then after that we can call set new name four different times to change the four different names so we're going to change id number one to john number two to paul number three to george and number four to ringo and then once we've defined the fab four we can exit our setup in the loop there's nothing here because we've done all the work inside the setup so let's go and upload this to our husky lens and see if we can put some names to the faces so i'm about to upload our sketch to the husky lens and when i do the upload there's a very brief little bit of indication on the screen so if you observe very quickly you might see that i'm going to do the upload right now and there you go on the screen and now it's uploaded and now you'll notice my second one now says paul and if i can just move this around let's see there's dawn george and ringo and so we have now identified the faces that the husky lens is memorized now one other feature of the husky lens is that it has a micro sd card and this card can be used for a couple of purposes you can use it to store photos or screen shots and the difference between a photo and a screen shot is that a screenshot has all of the text data that we've seen on our screen whereas the photo is everything without the text data another thing you can use the microsd card for is to store the training models that you've used and so let's go and take a look at how we do that with the husky lens micro sd so i've inserted the micro sd card into my husky lens there's a socket on the front of the husky lens for the card and i'd advise that you angle yourself so that you can see the inside of that socket and it will make it a lot easier for you to install the card once you've done that you can start saving the things that you've memorized so i've memorized the id of these four guys over here i'm going to try to save that to my micro sd card so i'm in face recognition mode press down to get the menu at the bottom and go all the way to the end to export the sd card and i'll press down and i get these numbers here and i can scroll through these to let me know what number i want you've got zero through five i'm going to press down so i get zero and so now it's been exported to the sd card and so let's just get out of here and let's go now to general settings press down on that and i'm going to go do a factory reset now so it's going to reset my husky lens and now as you can see all i'm seeing now are faces i'm not recognizing any of these fellows anymore so i'm in face recognition mode i'm going to go down the bottom again i'm going to do an import from the sd card i'm going to import number zero and then what i have to do is i have to reset my husky lens by powering it off and on and there you go now we're recognizing the faces again you can see paul george and ringo over here so the data from the micro sd card has been imported back into the husky lens and you can save five parameters for every husky lens function and so that wraps up today's look at the husky lens i think you can see that this very powerful device can allow you to add artificial intelligence-based object recognition to any of your projects and so you can use these for a number of applications whether you want to build a front doorbell that actually recognizes who's at the door or if you just want to count cows now i've got another application i'm using the husky lens for and i've got it right over here it's something i've been working on for a little while and as you can see it's one of these little robot car bases to which i've attached a husky lens and this can already follow objects around i'm going to get it to try to do a few of the other husky len features and of course once i have it together i will create a video and an article about it so if you want to see that video the best way to find out about it is to subscribe to the channel and all you need to do is hit that subscribe button and also click on the bell notification and if you do that and if you have notifications enabled on your youtube account you'll get notified every time i make a new video be it on the husky lens or anything else now if you want a little bit more information about what we did today or you want to get some of the sketches just head over to the article of the companies this video on the dronebotworkshop.com website and you'll find a link to that article right below the video while you're on the website please consider joining the newsletter it's a letter i send out every now and then just to let you know what's going on in the workshop and of course is absolutely free and another thing that's absolutely free is the dronebot workshop forums where you can discuss this video and anything else electronic with a number of like-minded people who are there just to help you out so a lot of resources for you so until we meet the next time please take care of yourself please stay safe and i'll see you again very soon here in the dronebot workshop goodbye for now [Music] you

Channel: DroneBot Workshop

Views: 61,262

Keywords: Huskylens, Kendryte K210, Machine Vision, Machine Learning

Id: E140gPLPz4A

Length: 44min 20sec (2660 seconds)

Published: Mon Nov 08 2021