I Built an AI Controlled Drone (2/3) - Gesture Control with OpenCV AI Kit

in the last episode in my attempt to build my  own drone and get it to fly I nearly destroyed it   that's right in this episode I'm going to add in  some AI to this drone so that it will recognize   gesture commands to take off and land just  like how the DJI does with its air gestures   to find out how I did it and how I recovered the  drone from a near fatal crash we have to go back   two weeks back when I began the planning phase  of the AI in computer vision development in this   phase I define the gestures that it should have  which is to take off land and follow now it would   be really nice to integrate all the ingestions  that DJI features on these drones but due to time   constraints I had to stick with the basics I also  heard a rumor that if you like and subscribe to   this channel it will add another gesture command  to this drone so how it works is that on boot up   the smart camera will detect me and once I give  the command to take off which looks like this yeah I know I look like a cheerleader doing this  but anyways the drone will spin his roses and fly   into the sky until a certain altitude is reached  then the drone should on detection follow me along   it's your axes until I give the "naruto command"  to land now keep in mind that we want to avoid   unintentional commands so when the drone is flying  we don't want it to take off again and when it's   landed we don't want to take off by mistake i  mean imagine if i go to pick up the drone and it   throws a spun right into my face so we'll need to  code in some additional logic to ensure that this   does not happen now if you remember in the last  video we got the drone flying in position hold   so i'm happy with that there's a few things  that we need to do first before we can add   in the ai and computer vision module the first  step is sort of a side mission in the a-game   i need to 3d print lego extensions for the drone  that will give me additional space to place   a larger lipo battery at the bottom of the drone  without these legs the drone will end up landing   flat on this battery which could end up like the  galaxy note 7s yeah we all know how that ended up   now i could have used these legs that came out  of the box with my drone but if you watched my   last video we don't want this drone to topple over  every time we land it so i printed out some legs   as funny as it sounds and attach them to the drone  as far away from the center of gravity as possible   to avoid the drone from toppling over but after  landing the drone it became apparent to me that   these 3d printed legs weren't as strong as i  initially expected so after some searching i   found this really cool spider leg 3d print design  i'll have a link to these files down below as well   as the kit and course on how to build your own  drone besides looking really sick they were much   stronger and could handle the impact upon landing  the drone almost reminds me of sherlock from the   lord of the rings ok cool so side quest completed  let's stop wasting time and get back to the main   quest which is programming the drone now how do we  go about programming this drone to do our bidding   well on board this drone we have a raspberry pi  4. okay i actually destroyed my raspberry pi 4   in the last video i have a backup by gb plus which  docks the pixel autopilot and uses the drone kit   framework for intercommunication between the two  devices now i have no knowledge or experience with   drunk yet so i spoke to my friend caleb from the  drone dojo and it turns out that he actually has   a drone programming course in addition to his  drone building course which made things super   easy for a simpleton like me to program a drone  i'll have a link down below to these courses if   you are interested but anyhow with this course  the first thing i needed to do was to implement   the code that would get my drone to take off  and land on its own okay so this is our first   autonomous drone mission we essentially loaded  a pre-programmed mission and we're going to get   our drone to fly itself without any inputs from  the controls let's see if it works slide confirm   then our drone should take off by  itself hopefully the battery is not flat   you can see there's no input from the controls  it's flying on its own completing its mission it's holding there for a couple of seconds  and then it must go and land right over there   awesome it worked okay cool so today is another  big day for we can do another flight test   and essentially what's going to happen is i'm  going to execute a line of code and that drone   over there should take off fly for one meter and  then land let's hope that's actually the case   but nervous regarding whether it'll work or not  they did recommend that we had to leash the drone   so if you look down here we've got a little bit  of a leash connected to some weights so in case it   flies off into space which we don't want that to  happen we covered in that respect other failsafe   is our radio control so if it does do anything  unexpected we should change our modes and that   will automatically bring it into stabilized  mode where i can take it back down to earth   let's see how it goes ready to take control at  any time there you go here we go okay look at   that it's flying by itself and then it's going to  land oh that worked better than i expected let's   try that again i'm not sure why it takes so long  for it to start up but i guess it gives it time   for me to get away in some dangerous cases that  could occur all right here we go are you ready   one meter and then sweet this is the two meter test it works all right so this is probably yet another  test that we are running you can see the drone   over there so this is autonomous flight where the  drone will lift up to a safe altitude and then   it will go to the north south east west attack  you get the idea okay so let's see how it goes yes i think it worked it did move to the side but  it was very subtle let's try increasing it a bit here we go here we go again  that did not work here we go let's flight it's flying by itself and landing as well flawless victory now that we're able to  autonomously take off and land we can start   with the integration of the ai and computer vision  but first we have to go on another side quest   now in this course caleb mentioned that it is  highly advisable to first simulate your drone   code before implementing real-world tests and so  that's what i did in caleb's precision landing   and drone delivery course i learned how to use  opencv to detect rgco markers within the gazebo   simulation environment in the real world  however i wanted to detect myself so i   just used different ar markers as placeholders  for now look at that my virtual drone crashed   i think it's virtual batteries died out somehow  so now on to the integration of the ai onto the   drone i'll be using the opencv ai kit by luxonos  for implementing real-time person detection at 30   frames per second the reason for using this unit  is that all of the ai is done on board this module   so that it frees the raspberry pi to do other  dos like giving commands to the flight controller   based on the detections so first up let's test to  see if the entire system will work while in flight   okay guys so i've managed to integrate the code of  the opencv air kit depth ai api onto the raspberry   pi and i've mixed it up with the takeoff and  landing script this is what we have at the   moment we have our opencv air kit which will do  some ai processing in this case it'll just detect   my hands or different sign language signals and  then relate back to the pi and this is where it   will decide whether to take off and land but right  now i'm just testing to see if the code works at   all there's no integration between drone control  and the ai part of it so i'm just going to see   if on take off are we able to use the opencv ai  kit and if so is the image from the opencv air kit   clear enough or is it prone to vibrations  from the drone let's see how it goes okay so the first test field i think  the raspberry pi was shutting down   when i had to activate the opencv air kit it's  very strange i'm not sure how i'm going to solve   this um one way i could do it is by starting  the opencv air kit first and then starting the   takeoff sequence let's see if that works okay so  i think i've sorted out the power supply issue   i was actually very stupid because i was trying  to power it off the power pins which is not ideal   now i'm supplying power directly from the battery  to a 5 volt power supply so it steps down the   voltage from 12 volts to 5 volts and then it  feeds it into the micro usb connector so that   is the much more ideal way of powering your  raspberry pi and this way the power won't cut   when we spin up our propellers and run  our opencv ai kit let's see how it runs okay getting up to target let's see if we're able to get a stream from my  open cva there we go that is from the drone it is   updating and be able to see what's happening it's  going to keep it there and see if it can detect me success okay so the problem that we have in now  is that our camera is moving all over the place   as you've seen now to do any sort of computer  vision stuff it makes it a bit tricky okay if   you're giving commands like land or go up go down  that would be fine but now the problem comes in   is if you're tracking it now if i'm moving my hand  to the side and the camera is moving all over it's   going to be hard to keep our image center for  example and to lock onto the hand now the way   we can solve this is by using a gimbal stabilizer  that will help stabilize our motion so depending   on where we are it will stabilize that motion so  you can either have a single axis or triple axis   gimbal stabilizer i'm not sure which one will  be ideal i would need to test that for this   application also you can see that we're running  out of quite a lot of real estate at the moment   so we have our battery taking up a lot of space  i could move it to the center that would help a   little bit that would actually make more  sense but if i have my beefier battery   my 6000 milliamp battery that would that one i  could only mount at the bottom so i have to be   really picky of where i place my gimbal so a  shortcut that i can take is for now to dampen   any vibrations because this one you can see is  very loose so maybe it will move around all over   the place i could 3d print a part that could be  mounted on here because this is where vibrations   are actually dampened so that could make things  a little bit easier let's try the the cheap   alternative which i mentioned now and then and if  things still don't work out you could maybe try   buying a cable stabilizer let's see how that works  okay i think some double-sided tape might work   let's give it a try just take  it and mount it right over there now at this point we have our hardware issues  sorted out we just need to build our models to   detect various gestures so that is take off land  and follow i probably spent like half the day   running around my garden with my dog chasing me  to accumulate enough data to build up my data set   which i then annotated and trained using the  roboflow platform if you don't already know   what roboflow is it is this all-in-one computer  vision platform that allows you to easily annotate   your dataset train your model and deploy for  your own custom computer vision applications   so i spoke to joseph from roverflow and they  released this nifty feature which allows you to   train the opencv ai kit with just a single click  insane right you can get this dataset and free   training credits to the first 50 people when you  sign up at public.roboflow.com forward slash aes   so the younger me wanted to use hand gestures  as well as body gestures to control the drone if only he knew a few moments later great now that our model is  trained let's test it out let's start the sequence   right now it's detecting me in follow mode i want  it to take off so let's give the take off command   and there we go all right and right now we want it to land  so go in range over here give the command   and there we go let's get there so it works so there we have it if you thought  this was cool let me know in the comments down   below and if you have any suggestions on how to  improve it so to wrap up we are now able to get   a drone to autonomously take off and land given  specific gesture commands but this is one thing   missing can you guess what it is well it's the  follow command as you'll see in the next episode   i'll show you how i got my drone to follow me  along is your axes this is pretty cool and risky   at the same time because my drone crashed a couple  of times on its own to find out what exactly   happened gently tap on that subscribe button put  that bell icon to get notified when the next video   comes out and lastly if you would like the  source code parts list video tutorials blueprints   and the behind the scenes of how i made the  gesture control drone as well as using the opencv   and raspberry pi consider checking out my six in  one mega course in computer vision and ai all of   the links will be down below and a huge shout  out to channel members and sponsors provoflow   which you can use for annotation and training of  your own am models thank you as well to our elite   augmented engineers andre shane prokop and claudio  person you guys are great supporters of this   channel ah nothing i like more than letting my  drone loose fast and furious out at the rc field   okay i think some double-sided tape  might work let's give it a try just   take it and mount it right over there just  like how the dji just like how dj dji dji ah

Channel: Augmented AI

Views: 37,604

Keywords: artificial intelligence, AI, opencv, opencv ai kit github, opencv ai kit crash course, opencv ai kit oak—1, opencv ai kit with depth (oak-d), opencv ai kit kickstarter, opencv ai kit review, OAK-D, oak-d depth, oak-d raspberry pi, oak-d camera, oak-d getting started, OAK on drone, Autonomous drone, gesture controlled drone, drone documentary, DJI Drone, DJI, dji fpv, unmanned aerial vehicle course, AI drone

Id: TYiiLTioecg

Length: 17min 46sec (1066 seconds)

Published: Tue Jun 01 2021