MarI/O - Machine Learning for Video Games

welcome back settling here you're watching a skilled player play Super Mario World but this player is not human it's a computer program I wrote called Mario this program started out knowing absolutely nothing about Super Mario World or super Nintendos in fact it didn't even know that pressing right on the controller would make the player go towards the end of the level learned all of these things through a process called neuro evolution in this video I wouldn't teach you about how Mario learned to beat this level don't explains one what his brain looks like and how it's all based on actual biological evolution so let's start out by actually looking at Mario's brain let's play it again but this time we'll look at Mario's brain as it's making the decisions of what buttons to press it's going to look a bit complicated at first but don't worry I'll help break it down for you this structure of colored lines and blinking boxes is called a neural network it's a simple mathematical model for how our brain works but it can produce some very complicated behavior with enough computational power a neural network could come close to simulating a real human brain but modern technology isn't there yet on the left side you have the inputs this is what Mario sees it's a simplified view of the level the white square of stand for blocks the player can stand on and the black squares stand for moving objects like enemies or mushrooms on the right side you have the outputs these are the eight buttons that Mario is able to press by using its neural network in between the inputs and the outputs all those lines and boxes those are the neural network each free-floating box is called a neuron and the lines connecting those boxes are like the axons and dendrites in a human brain I did make given time only some of these neurons and connections are to being used and this is what people talk about when they say you only use 10% of your brain the neural network you're seeing is a pretty complicated one and it got so complicated as a result of a 24 hour evolutionary learning session so to explain how neural networks work let's rewind about 24 hours and look at how the whole process started this is what mario looked like at the beginning of its training session all the way back in generation number zero the program is probably even dumber than you thought at this point often it just stands there and doesn't even press any buttons if my iö stands still for too long it'll cut off a simulation and try the next neural network so it's mostly just jumping from one simulation to the next but occasionally the neural network says to press the right button and the player starts walking right behavior isn't complicated but it's enough to make at least some progress in the level let's take a look at a sample neural network to understand just how that works this is one of the randomly generated neural networks that appeared in the first generation of the simulation there are some green lines in a red line and one neuron in the middle here's how it works a green line is a positive connection and red line is a negative connection a green line reading from a black or white Square will turn its output the same color a red line reading from a black our white square will turn its output the opposite color in this case the green lines read from the platform that the player is standing on and make the neural network press the right button as long as the player is standing on it however when the red line reads a black square representing one of those capes Koopas it presses the a button and Mesa player jump is that this puts the player in a position where the green lines are no longer reading a white square so the right button turns off and Mario just stands there this is a really basic example that illustrates how a more complicated neural network might operate the more lines and neurons you have the more nuance to the decisions can be so how exactly do we get those more complicated neural networks the answer is evolution when mario gets further right on the screen its Fitness goes up in this case Fitness is a function of how far right it gets and how quickly it gets there only the neural networks that produce the highest Fitness are selected to be bred creating the next generation it took 34 generations of genetic breeding and Fitness evaluation before Mario was able to finish the level without dying and get a fitness score about 4000 you can see there were several places it got stuck for a few generations but it always evolved out of those ruts let's take a look at a few of those ruts you can look at the top-left corner of the screen to see what generation number each rut occurred on this process of picking the fittest individuals from each generation breeding them together and adding random mutations very closely matches the actual process of biological evolution that took single-celled organism and produced intelligent humans that's the power of neuro evolution and though we don't yet have enough computational resources to produce something on the level of the human brain this way it's kind of neat to see what I can do on one of my favorite games I didn't come up with this idea on my own this algorithm is called NEET which stands for neuro evolution of augmenting topologies and it's based on a paper by Kenneth Stanley and Ari stow me cool einen it's a really great paper that describes how to use genetic algorithms to build up neural networks from bare bones without presupposing the best structure for the neurons and their connections also includes some really cool ideas for separating genomes into species which a lot of genetic algorithms don't really try and do I wrote Mario from scratch in Lua as a plugin for an emulator called bizarre as I close out the video let's take a look at the fittest neural network in each generation it's kind of cool how sometimes you can see them make modifications to each other and improve performance but sometimes an entirely new species becomes prominent and dominates the others if you'd like to do more reading about the concepts I talked about in this video I provided some links the video description I had a lot of fun working on this project and I learned a ton hopefully you learned something too it's about it thanks for watching you

Channel: SethBling

Views: 10,684,498

Keywords: Super Mario World, Machine Learning, Neural Network, Genetic Algorithm, SethBling, Neuro Evolution, NEAT, MarI/O

Id: qv6UVOQ0F44

Length: 5min 57sec (357 seconds)

Published: Sat Jun 13 2015

Reddit Comments

[u/dat0]

Source Code available here.

[u/VeloCity666]

If you thought that was cool, you'd enjoy this.

(This one is more general and is tested on more games)

It was so succesful that he made 2 more videos trying different games.

[u/retrogamer500]

I think this is an example of overfitting. I'd like to see an example that wasn't in the training set.

[u/jellybeanerr]

Watching how close it comes to dying makes me uncomfortable.

[u/Waveseeker]

I see "Top Super Mario speedrunner" And I did not think sethbling!!!! Holy hell this guy got me into reddit 2-3 years ago!

[u/[deleted]]

SethBling never ceases to amaze.

[u/[deleted]]

If he were to run this system again, starting MarI/O at generation 0 again, would the final neural network look the same, or is the algorithm's progression stochastic?

[u/Minerva89]

That random mythical error of the 10% usage... note that the point he was trying to make is that after a complex neural network is formed, it's likely that certain parts of the network are more active than others at any given point, depending on context.

Just to be clear, it's not that 90% of the network is not used, simply quieter, or filtered to be quieter.

[u/CabbageSwag]

Now do it with Dark Souls