I Made an AI with just Redstone!

I made a fully working neural network in Minecraft you can draw a number on this drawing pad and it'll tell you what you drew this is the story of how I made this project from researching to running simulations to building all the Redstone I hope you [Music] enjoy this story starts a few years ago when I was taking a machine learning class at University on the first day of class the professor showed us some basic neural networks and how they could learn things like humans do and as someone who did Redstone I wondered if it was possible to make a neural network in Minecraft I mean how cool would it be to build something in your world and watch it learn especially if it could learn something complex like image classification so later that day I went on YouTube to see if anyone else had done it before to my surprise almost no one had at least not with just Redstone there were a few people that did it with command blocks but I could only find one person who built a redstone only neural network yanic Kure after watching the video I was really impressed but at the same time I was a little bit disappointed not in yanic he did a great job but just in the size of the network if neural networks are already this big and complex for just a few neurons then making something like an image classifier seemed completely out of reach so I scrapped the idea and I just made other kinds of projects for my channel but then in the summer of 2022 this video popped up on my recommended and I immediately clicked on it a team of people made a convolutional neural network and they used it to recognize hand handwritten digits you could draw a number on the screen and the network would tell you which number it thinks it is along with the confidence value suddenly my opinion on neural networks completely changed clearly it was not only possible to do cool stuff with them but also keep them in a relatively small footprint now For Better or For Worse I'm a very competitive person so I quickly became fixated on making an even better neural network than this I wanted to make a handwritten digit recognizer that was faster and had better accuracy whether this was possible or not it didn't matter my competitiveness meant that I was about to spend a lot of time trying to make this happen there are lots of different kinds of neural networks so first I had to decide which type of network to use after looking at some handwritten digit networks online it seemed like most of them either used a multi-layer perceptron or a convolutional neural network which is what the team used both MLPs and CNN's can get pretty high accuracy on handwritten digits so I actually decided to go with an MLP because it seemed easier this was a bold Choice considering the team didn't do this and they clearly knew what they were doing but I just had a gut feeling that an MLP would be better suited for Redstone now if you haven't seen MLPs before 3 blue one brown has an absolutely incredible video on them and it's on handwritten digit recognition too so I highly recommend watching it but I'll give a shorter explanation here too an MLP is a function with input and output in my case the input is all the pixel values of the handwritten digit zero being black all the way to one being white and the output is 10 numbers representing the confidence the network has for each digit if you put in the pixel values of this image for example example a smart MLP would output a high confidence for three and a low confidence for everything else but for Simplicity consider an MLP with just two inputs and two outputs inside the MLP there are neurons and connections between the neurons every connection has a weight which is a number that represents the strength of that connection and every neuron has a bias which represents How likely or unlikely it is to fire when you input something to the network like 1 and two for example they get sent out by the connections and multiplied by the weights then for each neuron in the next layer all the numbers that arrived get added up and the bias of each neuron gets added as well then these neurons send their values to the next layer and the process repeats itself every column of neurons is called a layer and these layers have special names the first one is the input layer the last one is the output layer and anything in between is a hidden layer in general you can have as many hidden layers as you want and within any layer you can have as many neurons as you want so there's a huge variety for what an MLP can look like in the case of handwritten digit recognition the input layer has as many neurons as there are pixels in the image and the output layer has 10 neurons now as I've described the network so far it's just doing a bunch of multiplications and additions that's great but it also means that if you're good at linear algebra you can reduce the entire network down to an equivalent network with just an input and output layer so there's actually one more thing an MLP does after a hidden layer outputs its numbers an activation function is applied before passing it along one of the simplest activation functions is called rectified linear unit or railu railu just says if it's negative set it to zero otherwise don't change it so let's apply a rud to the output of the Hidden layer and as you can see the negatives become zero then they get passed along another way to think about activation functions is that they're kind of analogous to a real neuron the output is zero until you hit a certain threshold which makes the neuron fire so that's how an MLP is executed but how does it actually learn to recognize digits well it has to be trained in other words it has to learn the weights and biases that give the best accuracy this is typically done by showing the network tons of examples and using an algorithm called back propagation to continuously update the weights and biases and hopefully after showing it lots and lots of examples you can then show it a brand new digit and it'll make a prediction based on what it's learned now that I had a better grasp on MLPs I created a plan for the rest of the project first I wanted to simulate and train the entire network in Python that way if the Redstone had an issue later I'd have a reference to compare it to then I wanted to build it in Minecraft and import the weights and biases from the code so the Redstone would only execute a forward pass through the network in Python I started by importing the M data set which has thousands of examples of handwritten digits every example is 28x 28 pixels and it's labeled with the correct number then I used a package called caras to build the network I made the input layer 784 neurons one for every pixel in the image and then I didn't know how many hidden layers I was going to need so so I started off with just one hidden layer of 32 neurons and a Ru activation and then I made an output layer of 10 neurons once that was done I used this code to train the network and after testing it it got a really high accuracy of 96% then I continuously reduced the number of neurons in the hidden layer seeing how it affects the accuracy and I ended up finding a sweet spot at about 10 neurons which still gave pretty good accuracy 92% and now I had an actual structure for my network 784 input neurons 10 and hidden layer neurons a Rayo and 10 output neurons but this is when I ran into my first big problem the images in the mnus data set are grayscale and there's no easy way to show a grayscale image in Minecraft I mean technically there is but how would the player even draw one anyways the best you can do is draw an image on Redstone lamps which are only on or off so using this code I made all the pixels in the data set snap to either black or white depending on which one they're closer to I retrained the network and the accuracy was still 92% however I found another big problem the weights and biases are stored internally as floating Point numbers which are really difficult to work with in Minecraft in fact almost no one does it most people just work with integers I tried rounding all the weights to the nearest integer but because of how precise they were this lost a ton of accuracy but then I had a pretty good idea I decided to multiply the weights by 100 and then round them this works because the exact value of a weight doesn't really matter what matters more is how the weights relate to each other each other so this strategy allowed me to get more Precision convert them to integers and keep their relative relationships I tested the network again and it still got decent accuracy 83% obviously it's not amazing but it was still better than my self-imposed competition with the simulation working it was time to start building the Redstone I started with the drawing tablet which would allow the player to draw a number by moving around on a platform luckily I had already built this exact thing before in my Redstone paint program it used trip wires and string to detect where the player was standing and sent that information to the screen so I ripped it out of the paint build and I modified it to be 28x 28 pixels I tested it out and it worked beautifully when I sped up the game using carpet mod I was able to draw in nearly real time I also explored some other options like pressure plates and throwing snowballs at the screen but it seemed like nothing could beat the Elegance of trip wires the only problem was that the data was hard to access because the XY decoder was right behind the screen so I duplicated the Redstone and kind of mirrored it like this that way when I drew something it created two copies one on the screen and one over here so that I could easily access the data and with that the input layer was basically done these are the 784 ones and zeros that get put into the network next I started working on the hidden layer thinking about all 10 neurons at once was really overwhelming so I started by focusing on just one remember a neuron takes all the numbers from the previous layer multiplies them by the weights sums them up and adds the bias at first this sounded like a Monumental task 784 multiplications that's insane for a redstone build but then I remembered that this isn't grayscale anymore the input numbers are just ones and zeros so the multiplication is Trivial just keep the weight if it's a one or throw out the weight if it's a zero then just add up the weights you kept and add the bias the other cool thing was that the weights and biases only had a range of -5 to 5 that's only 11 numbers so I decided to represent each weight as a signal strength value with a barrel so here's the first prototype I made for one neuron in the hidden layer every weight in these barrels is being read by a comparator but if the pixel behind the weight is zero the comparator gets cancelled when I press this button all the weights that aren't cancelled get put onto this main line and they come out over here in a stream so if the weights are like this and every pixel is on you get all of them but if only the Four Corners are on you get this then to sum them up I converted them to Binary and fed them into a special binary Adder to keep a running total by the way I didn't make this adder and I don't really understand understand how it works but it's really good at adding up a ton of numbers it's called a looping carry save Adder designed by Amino instead of a normal Adder where you would add two numbers and get an output it just has one input and every two ticks it adds that input to the total which is extremely fast for example if you input 1 2 3 4 all two ticks apart from each other you get 10 almost immediately this made it a perfect fit for summing up the weights if all the pixels are on you get 18 or with the the Four Corners you get eight and to add the bias I plan to just add it to the total at the end after finishing the Prototype I made the first hidden layer neuron for real honestly this went really well after I finished it I put in some test weights and it seemed to be working the only problem was that it took a long time to run 784 editions is no joke even with a really fast Adder so I split up the work into four adders running simultaneously and that made it way faster then to import the real weights I used a python package called MC schematic I used this code to extract the weights and generate a schematic of barrels which I could just paste directly into the world I also used MC schematic to put in the first example from the real data set once that was on the screen I ran the neuron and I got the correct value then to make the rest of the neurons it was pretty easy I just used world edit to copy and paste it nine more times and I pasted in nine more sets of weights and biases instead of testing anything I decided to just send it and run all 10 at once which unfortunately didn't work the first time the other nine neurons revealed a bunch of bugs that I didn't even realize I had but eventually I got the output to look like this which was exactly correct the only thing left for the hidden layer was the railu remember Ru just sets negative numbers to zero to my surprise this was really simple to do every number already had a sign bit which was one for a negative number and zero for a positive number so I just wired the sign bit into a tower of comparators that set it to zero that way if the number was negative it got canceled out now that I was confident the hidden layer was working I moved on to the output layer just like before I started by focusing on just one neuron unlike the hidden layer there are 10 multiplications instead of 784 but these multiplications are not necessarily by zero or 1 so now I needed to use actual multipliers I looked in my schematics folder to see if I had any multipliers already and I found two designs but I actually didn't want to use either of these they didn't work with negative numbers and they were really big so big that they couldn't fit side by side here without making me spread the circuit out so I designed a new multiplier which was skinny enough to fit 10 of them side by side then once I confirmed that they were actually working I made them all output to this line which feeds into another looping adder and now I was already ready to test the neuron I imported the weights and biases from output neuron number N9 and when I ran it it was right suddenly I could see the light at the end of the tunnel I duplicated it nine more times pasted in the rest of the weights and biases and just like that the full network was ready to run for the very first time [Music] at this point the network was working perfectly I considered stopping the project here but there was still something about it that I really didn't like I had to read the confidence values as binary numbers so I wanted to figure out how to make the final output easy for anyone to understand my first idea for this was to make a circuit that finds the maximum and displays it on a screen so in this case it would see that s has the maximum confidence and it would display it here but then I thought why not display everything if I could take these values and display them on some kind of bar chart that would be really cool like when you draw a three it would be cool to see the bar for eight go up a little bit since they're similar shapes however there was an annoying problem with the bar chart it needed to be tall enough to account for the smallest and largest possible confidence values which meant that most of the time the bars would be really small but I really really wanted a bar chart so I went to the internet to see how people solve this problem one solution is called softmax softmax is a function that takes a list of numbers and creates a new list of numbers where they're all between 0o and one and they add up to one so it basically takes the values and converts them to probabilities this was tempting to make in Redstone but when I saw that there's exponentiation in the formula I got scared and started looking for another solution then I found this formula which is similar to softmax but doesn't involve any crazy operations it basically squishes the list down to a specified range like 0 to 1 for example so I thought why not make that range 0 to 15 and floor it that way it converts the list to Signal strengths and so this is the formula I decided to build ironically implementing this formula took up more space than the entire network so far I used a chain of comparisons to find the Min and max value I used shifting and subtraction to multiply by 15 and I used a bunch of dividers to perform that final division this was obviously a ton of work but it was worth it now the confidence values were just the signal strengths of these 10 Redstone dusts so finally I made the bar chart every bar was 15 lamps High which allowed me to show all the signal strengths in a really nice way overall I could not be happier with the final build with 83% accuracy and a real-time speed of Just 2 minutes this thing is a beast if you'd like to try it yourself there's a world download in the description I'll see you in the Showcase [Music] [Music] learning about neural networks can feel really overwhelming as it did for me many times during this project but when it comes to learning about AI one great resource is called brilliant the sponsor of this video brilliant gets you Hands-On with Concepts like math data analysis programming and AI it's a learning platform that builds your understanding from the ground up every lesson has Hands-On activities which are way more effective than watching a video by actually solving problems you're not just memorizing things you'll build critical thinking skills that will stick with you for Life Learning for just a few minutes every day is pretty important for both personal and professional growth brilliant helps you do this with lessons that are available 24/7 if you thought handwritten digit recognition was cool then you'll love the course called how llms work and llm is another kind of neural network and this course gets you Hands-On with how they build vocabulary choose their next word and more you'll even learn how to fine tune in llm to generate different kinds of output from poetry to cover letters to try everything brilliant has to offer for free for a full 30 days visit brilliant.org slmap batwings or click the link in the description you'll also get 20% off an annual premium subscription [Music]

Channel: mattbatwings

Views: 723,963

Keywords:

Id: DQ0lCm0J3PM

Length: 17min 23sec (1043 seconds)

Published: Sat May 25 2024