How to Control a LEGO Mindstorms kit with AI and Raspberry Pi 5

welcome back to element 14 presents I'm Andy and today we're going to build an AI robot with a Raspberry Pi 5 and a LEGO Mindstorms kit from 1998 let's get [Music] started so I found this Lego invention system from the late 90s and what's interesting about it is it's apparently never been used it's kind of a shame because somebody probably would have you know enjoyed owning this back in the day and now it's completely obsolete I mean the software designed for Windows 95 um and it requires a Serial port to program it now I have a couple of kids who are getting into STEM related toys and I think this would be perfect uh for them so we're going to modernize this and basically keep it from being thrown in a landfill speaking of landfills one of the common uses of machine learning in robotics is in recycling so we're going to build a miniature trash sorting robot so the first thing we're going to do is connect the IR Tower to a Raspberry Pi because it supports USB to serial right out of the box then we're going to replace the old software and I'm going to show you how to build a web- based graphical programming interface to control the robot in real time we will connect a camera to the Raspberry Pi so that we can connect to gp4 or Claude or some other Vision API and then we'll build a robot out of these actual Lego Parts uh but first let's take a closer look at the condition of the parts in this box because I can see already there's going to be at least one major problem so as you can see everything's still in its original packaging there are some empty spaces where I've taken things out including what they call the microcomputer as well as the IR Tower and some cables but uh most importantly I took out the wires for the motors and sensors and I put them in separate containers because the insulation on these wires is disintegrating and it's making a big mess so we're going to want to replace the cables right away and clean up the residue that's already starting to get on things okay we'll start with this one oh man look at that it just like flakes right off when I touch it so we'll have to figure out how to get into this brick without damaging it I prefer to use these soft plastic tools because they don't scratch harder Plastics and after plenty of prying I managed to get it open okay so this is interesting there's no solder in here uh it's more of like a vampire tap sort of connector like the old computer networks where it uh pierces through the insulation okay well we can work with that all right I'm going to cut this cable down and toss it cuz it's this stuff is getting everywhere we just want to push the new wires into there without damaging them and then we'll press the top part of the brick on with some pressure to pierce the insulation all right that looks pretty solid let's just give it a test with the multimeter perfect now the light sensor it looks a little trickier I don't think I can remove these tabs without damaging something well I got to get in here somehow so I'm going to try and cut these tabs off cleanly with a blade and see how that goes well I I didn't show it on camera but opening the sensor brick took a lot of time and effort uh I did break one of the plastic tabs inside that was unavoidable but if you look here at the PCB it's obvious where the new cable needs to be soldered in so let's do that so as always we start with flux and then we're going to grab the tweezers and gently pull the old wires out while we heat up the solder with the iron then we strip the new wires and strip the new wires no give me just a second yeah it's always challenging to solder for the camera I have to do it at a weird angle okay well that's not proper soldering technique but it got the job done it looks good and I've cleaned the junk out of the blue brick you could see how the board fits back in there and here it is with the wire situated I'll probably just glue that back together and the other end has another one of these black bricks we just press the pieces together until they snap and that looks really quite good we managed to do it without ruining the brick and it still looks nice so that's awesome now that we've restored the wires on our bricks we need to test them so I figured now is a good time to power up the yellow programmable brick I'm just going to call it the computer for Simplicity sake I've been going through the various documentation that came with this and it really seems like they wanted you to learn how to use it from the official Windows 95 software the information in the user guide is pretty limited but I did find something about the built-in programs that we can use so if we put the light sensor on Port two and the motor on Port A and then run program 3 we should be able to control the motor with a light sensor so we'll load this up with batteries and plug in the motor with one of our repaired wires and then the sensor goes up here and let's see if this still works after all these years nice now we'll select the program and we'll press run now the motor spinning that's a good sign and if we block the sensor it works well that's uh I mean that's pretty definitive yeah that totally works so this proves that the computer the wire the light sensor and the motor all [Music] work so as I mentioned in the beginning we're not going to be using the original software that came with this I've done a little research and found that the Raspberry Pi supports USB serial adapters like this one out of the box I did try connecting this to a Windows 11 machine and Device Manager basically said don't even bother trying so uh I'm not going to the Raspberry Pi website has great instructions on how to install the OS for your Pi so I'm not going to cover that here but I will however put a link in the description to the element 14 Community website where I'll include all the resources uh for this project like the code blog posts and everything else I used we're going to install something called nqc or not quite C which is a c likee language that can be used to program the Mind Storm's computer now there is a bit of setup involved here but somebody was kind enough to document the process nqc can also be used to send single commands directly to the computer and one of those commands is to upload new firmware there are various ways to get the firmware file I actually have an optical drive and was able to extract the firmware from the CD to prepare for the upload we plug the USB to serial adapter into the pi and the serial cable into the adapter the serial C then plugs into the IR Tower which requires a 9volt battery all right then once we have nqc installed and the firmware file in the right place we can execute the command to upload the firmware look at that it's counting I don't know what we're looking at there are those bites all right it's done let's go look at some code do you like free stuff you can join the road test program you can get free Dev kits test equipment and even online training courses in exchange for a detailed review join our road test program learn more at the link below ah free Stu so I said before that nqc is a programming language and typically the next thing we would do is write a program with a text editor compile it and upload it to the programmable brick uh but that's a little Advanced for kids who have never written code and it's why Lego originally included a graphical programming environment with the robotics invention system but rather than dig up a Windows 95 computer just so we can run a very outdated program uh we're going to make our own modern UI with a Google web Library called blockley uh this is the same Library used by scratch which is a graphical programming environment uh aimed at kids so if you ever seen that this is going to look familiar now the simplest blockly application I know how to create is just a web page with some JavaScript and as you can see I've created exactly that you don't have to install blockly locally uh here I'm adding the uh the scripts from the unpackaged CDN just by pointing to the URLs and after that there's a number of JavaScript files that I've created to Define custom blocks and code generation so I'm going to go ahead and run the application which I'm calling block storms to demonstrate how it works and on the left are all these groups of blocks everything from logic to functions are built in and if I go into text I can uh drag a a print block then I'll grab this literal text block and we can type hello world and when I click the generate python code button I get a Python program which represents what we've added to the workspace so this is really cool and it works because I included block Le's python code generator and I have this function that executes when you click the button that puts the python code in a div tag under the button but what if we want a block that can turn on one of our Lego motors well we can go into the motor group and drag drag the set motor block onto the workspace and if you click on the drop down you can see that we can select which motor we want and we can also turn the motor on and off or have it Coast now when we generate the python code we get a call to nqc which executes a rock command and this command happens to be the one that turns motor a on uh there's a whole documented list of numeric op codes and parameters that I use for reference to figure out what commands to send to the programmable brick and control it in real time and you can control the motors read values from the sensors and uh even do things like play sounds so these blocks are all defined by custom JavaScript code so let's take a look so this is the code that defines the set motor block and you can see how the drop- down values are defined uh as well as things like the block color and the tool tip and below that is the uh the python code generator for the block uh it takes the parameters that you selected and turns them into a call to nqc and subprocess is python way of calling a child process ideally we'd want to replace this generate python code button with a button that says something like run on rcx that would probably involve adding in uh like the flash framework and it's a bit more complicated so for now we'll just copy the python code into a file and run it from the command line now I should point out that there's a drawback to this approach of sending one command at a time to the rcx rather than uploading a full program and that's lag there is a noticeable delay when executing commands that makes this unsuitable for certain types of projects the theoretically we could make a blockly code generator that generates nqc code but that's a whole other level and I'm not ready for that yet so let me show you a quick example that plays a few notes to show you how much of a delay there actually [Music] is so there you have it uh fortunately I've come up with a project idea where the delay shouldn't be a big deal I've got an old piie camera module in my parts bin and we can use Python subprocess again to call Lib camera still to capture an image and I've added a parameter so we can specify the file name the last block I'm going to show you is the most interesting one it uses the new GPT 40 model and it lets you upload an image and ask questions about it with a customizable prompt so you'll have to provide your own API key and the way I'm doing it here isn't the most secure but it is simple and it works those are all the elements we need to program our robot let's go back to the Lego and I'll put my extremely amateur design skills to the test well it's been decades since I've built anything original with toy blocks so not sure how this is going to go but uh it should be fun regardless so let's open these all up and see what we have to work with I'm not going to actually count the pieces in this set but from what I could tell they're all here sadly a few of the rubber tires have started to deteriorate but everything else looks brand new now since I have no idea how long this is going to take I'm going to go spend some time building building and I'll come back when I have something that works it'll seem instant to you but for me it's probably going to take a few hours of real [Music] time and here is the trash sorting robot I found some small carts which are fun and the cameras mounted on a stand which is part 3D printed and part Lego so originally I was going to put actual trash on the conveyor belt which is really it's just a couple of tank treads uh but the size here is really limiting so what I've done is I've printed some images on uh some pieces of card stock and that should work just as well so the way this works is we put our image up there and then I've got a touch sensor set up as a button to activate it so the camera takes a picture sends the picture to GPT 40 which classifies this image as trash or Recycling and then it activates the motor in the right direction to put the item in the correct bin I've also made a block storms program to control everything there's an infinite Loop that continually watches for button input and when the button's pressed it queries the AI model with a custom prompt The Prompt is designed to always produce a single word response that can easily be parsed by our program logic and here's an example of what the camera actually sees I guess there's nothing left to do but try it out we start by running the Python program generated by the BL Storm's workspace then I give the touch sensor a firm press the image is captured and classified by GPT t40 and it got it right let's try styrofoam this time which is generally considered not recyclable we'll push the button again yep it's seen it is trash this thing's working great I should build a fullsize version that's all we have for today have you ever revived an old toy or device with modern Electronics let us know at the element 14 Community at the link in the description and we'll see you next time [Music]

Channel: element14 presents

Views: 5,078

Keywords: electronics, hardware, hacking, mods, element14, maker, engineering, element14presents, Raspberry Pi, Raspberry Pi 5, Lego, Lego Mindstorm, AI, Robotics, AI Robot

Id: UqUG2VUFsrg

Length: 14min 23sec (863 seconds)

Published: Fri Jun 28 2024