Teaching my custom AI drone to track humans

last time on the drone series i built stanley my drone powered by an nvidia jackson nano in this episode i show you how i programmed him to follow me around with just ai [Music] hahaha [Music] come on stanley well we've lost an arm it's completely sheared off here the gps antenna it's broken on the top plate so that's now just wobbling on its own and we've completely lost the mounting point for the foot to attach this leg like it just disintegrated he's not in a good shape that's for sure hey can you get the tools please okay you're gonna be okay stanley all right you're gonna be all right [Music] well stanley i reckon you are in a good enough position to fly again let's do this [Music] come on stanley [Music] appear to be chopping grass [Music] i'm shaking with adrenaline right now like look at my hand oh wow like this is loiter mode where he's able to control his own position with just a gps antennae pretty cool [Music] at this point stanley is nicely calibrated which means we can move on to detecting people with the oak d light do you remember how groundbreaking the xbox connect was well you might not realize but it's incredibly heavily used in the research community due to its pretty good quality depth maps and its comparatively low cost i believe though that the oak d light is a spiritual successor don't be fooled by its small size though on board is a myriad x vision processing unit vpu for short and that is able to run at four trillion operations per second you can run ai models such as mobile net ssd and yolo directly on board the camera and you can also convert your own models to run also that is crazy talk to see the full specifications for this camera make sure to head over to its website which i've linked in the description of this video in terms of programming the camera there's an sdk available from exonus named depth ai and it comes in both a python and c plus variants i've had to play around with it and it makes it so easy to get depth data working with inference so i tried it with tiny yolo and was able to get it detecting me in a single evening there's another camera which i've also got my eye on for this year which is the bottlenose which comes in stereo and mono from lab forge this is currently available in pre-order and i'm pretty excited to see how it performs maybe i'll uh do a review on that if you guys want so drop a comment below if you want to see that now you might remember from previous parts of this series where i covered how to generate depth maps using stereo bm and stereo sgbm as well as how to run tiny yellow on the jets and nano so naturally i ran experiments to compare my results to what the camera can produce look at that fps 25 it's just frozen that's off this guy here i'm really really really happy with that that is pretty much real time enough for what i need we've basically established at this point that the okd light is perfect for what i'm trying to do so i went ahead and started writing the code needed for stanley to follow me around using this as the input source for data when i write code i lean heavily into a trial and error approach so i build something see where it breaks and then just keep fixing that until everything works but the thing is if i tried that with stanley it's a surefire way to end up in hospital because well you've seen what bugs in my previous projects can do to make sure the development process goes smoothly i enlist the help of a virtual drone instead of running everything on stanley straight away i did this by using cyto which is short for software in the loop the idea is you're able to simulate a real enough environment for your code to run inside to find any issues before you run anything on real hardware on top of all this cycle stuff i was able to build a visualization tool which is what you can see on screen right now i was able to control a virtual person on the map using just a keyboard and then observe how my code reacts to it changing position i've just found a problem now when i'm sending the drone up i'm initially setting it to run in loiter mode which means hey keep your position in 3d space latitude longitude and altitude and then turn around to try and search for people the problem with that is loiter mode expects there to be pilot input on the transmitter so i send the drone up set it to loiter it looks at the pilot input it's zero because we're running everything in code and it crashes because that is interpreted as go down as fast as you can there's two solutions to this fake the pilot input which i'm kind of scared of doing because i need to be able to take control of the drone at any time if something goes wrong or two try to write everything in only guided mode which is my initial idea so i've got a little bit more work to do but thanks to this tool i figured out a few issues before they happened in the air in this project i ended up using some computer science robotics theory in this case subsumption architecture let me explain over here we have a rule we pass input in such as camera state and get output out in this case hover in place next i'll add a second rule this one is going to be configured such that if there's a person visible it'll direct the drone to fly towards them now the important thing here is to note that this second rule will subsume the output of the first meaning it will control the drone over the output of the first one if it seems it's necessary the first rule has no idea this is happening so it'll happily keep commanding a hover but the system will ignore that now let's see what happens when we add another rule this particular one is going to back off so if the drone gets too close to a person that rule will fire and start backing the drone away from them and what we've created is a model where the highest priority rule is the one that takes precedence at all times but the lowest one keeps running as if nothing has happened and as a result we get a complex system modeled by the precedence of the rules within it one of my favorite things about this project is that all of my code is written with python i've put all of that online over at github and you can find a link to that in the description below this video today is saturday the 23rd of april and i promised myself that stanley would be done by tomorrow to start i'm going to look at getting all the dependencies my code uses actually operational on the jetson nano the most important one is one called depth ai which is the sdk used to communicate to od light now fingers crossed i don't run into integration hell but i pretty much guarantee you there's going to be some problems in this process it took a bit of time to install python 3.9 on stanley because the jets and nanos image it only comes with python 3.6 and that is too old for my code to run i also spent some time hooking up some code to look at files and folders on a usb drive the way that works is we have this little guy here when it's plugged in my code will save the video from the od to that as well as any log files but also if there's a file present called kill switch everything stops and i have full control over the drone again so if i decide i want to fly stanley in just a normal way all i've got to do stick that filename kill switch on plug in the usb and it's like nothing out of the ordinary i've still got a lot more i want to do like i don't trust my code whatsoever meaning yeah i want to avoid getting cut up by these guys so i'm going to write some more integration tests and when i'm happy that everything is running as it should then i'll take him out to the field and we'll see what happens we're really close now you've probably noticed i've already installed the od light on stanley by now my initial idea to do that was to reuse the gopro mount i had on board to get the footage you saw earlier in the video but after i was reviewing it i noticed this sort of jelly effect in the footage and i think that was due to vibrations from the propellers to combat that i went and got myself a two axis gimbal off ebay and installed the od light onto that instead now currently it's set at a zero degrees of rotation so it's pointing slightly upwards now when it's in the air this drone is gonna need to be able to point slightly downwards so i'm just gonna set the gimbal position to be 30 degrees and you can see it rotate here it can go all the way down to 85 degrees which i'm also going to do here as well [Music] but i think that sort of positioning is pointing a bit too low down i took the gimbaled okd light out for a test flight just to make sure that everything was performing as it should be there was a bit of jitter in the resulting video from the camera but the gimbal kept a decently level horizon throughout all the roll and pitch maneuvers that i was doing so that's a thumbs up from me now was just a case of doing a fully integrated system test on the bench i've got the od light hooked up to my laptop where i'm running all of my flight code now this is basically bypassing running everything on the jetson nano up here so i've got full control over everything all the output of my flight code is being routed over my proxy to the pc behind you where i've got a virtual drone setup with a mac visualizer so i can see quite what's happening as i'm walking that way and that way towards the camera so it looks like when i get too close to the camera the back off rule is kicking in like it's supposed to now you can see that where the virtual drone here is moving pretty quickly backwards now as i walk a bit more backwards you can see that you know the drone is slowing down and then actually starts accelerating slowly forwards which proves that the following rule is kicking in and if i move out of frame it should start searching [Music] i think that's as good as we're gonna get at this point take it out to the real world so we're here at the test location got stanley all ready to roll hopefully i don't end up going home in an ambulance but we'll see so to start off with i'm going to send stanley up in just a normal flight just to make sure everything's okay and then i'm going to flip the switch into guided mode and see how he takes off from the ground just to make sure that he stays in a fixed position and then we're going to do the full test of him facing me to see if he follows me so in the interest of safety i've got this stop switch still set up on my transmitter if anything goes wrong i can flip that to the on position and the flight will be cancelled now i'm going to be holding the transmitter at all times so please don't think i'm faking this it's actually gonna be my code running so i'm gonna be holding this in one hand to prove to you that's the case so i'm gonna go in and start the motors start the motors uh looks like there's something happening like the gps on here so i'm gonna go back home and figure out quite what's going on because it makes zero sense to me it was flying literally the other day so hello welcome to my slow and inevitable descent into madness so i have been staring at this for i don't know how many hours now i have tried so many things now to fix it and i have no idea what the hell i'm doing anymore help please i've had enough i was about to give up but then i found a forum thread covering radio interference generated by a jetson nano so i tried putting some tinfoil over the top of mine and no more gps glitches were locked [Music] it's rf interference holy okay so what was happening was we had this wireless antenna installed underneath the jetson inside here i don't know quite when it started happening but one of the wires on the antenna started getting pretty twisted so i assume is that was causing interference so what i've done is now removed and i've replaced it with just a usb dongle for now because you know it's small and it does the job just as well but it means there's no gps glitching and we got 10 sats inside the only thing now i've got left to do is get another battery for the transmitter because i've gone and just destroyed that but other than that we should be ready for another attempt at testing flight so before i take stanley out to the field again i just want to double check everything works properly so i'm going to toggle the switch on my transmitter to turn it onto the guided mode which should start my code and then have stanley attempt to take off hi stanley it's only taking me two months hell yeah this has got to be the best place i've ever flown a drone look at this place it is gorgeous [Music] this is going to be the making sure that everything works and in theory it should you saw nothing i'm going to try it i'm terrified and i don't know quite how this is going to go so i've instructed my camera woman to drop the phone and run if something goes wrong i'm going to toggle it onto guided mode and let it take off we're live that test didn't quite go as planned it went up came towards me seemed to not detect me try again [Music] so that test whilst it wasn't quite what i was expecting it showed that at least my code to lift the drone off the ground from arming that works but basically code broke i gotta fix it i found that there was a divide by zero problem in one of my rules which was an issue always do it on my own so i gotta get through it and the only thing i know [Music] [Music] way too close as well i don't belong tell my thoughts to move along push myself to be the best die with no regrets live with every breath see my message [Music] never give up never slow till almost at a fully working system i think [Music] a year ago i set myself a challenge to get a drone to fly around and follow me just using ai and you know what [Music] well it's been a year since i've been back here where it all began i can't believe i've gone from a raspberry pi drone that couldn't fly to stanley you can follow me around it's been a hell of a journey and thank you for coming with me on it i'll see you the next one [Music] [Music] i'm grateful

Channel: aka: Matchstic

Views: 26,717

Keywords: drone, nvidia jetson nano, jetson nano, oak-d, computer vision, ai, diy drone

Id: Zq_2_BEfU_s

Length: 19min 51sec (1191 seconds)

Published: Mon Jul 25 2022