Let's Build A Video Game With Unity and TensorFlow

I'm gonna talk about three things today I'm gonna talk about unity which is a game engine I'm gonna talk about tensorflow which is a machine learning framework that comes out of Google I'm gonna talk about something to glue them together which is the unity machine learning agents toolkit which I may occasionally refer to as unity ml agents unity ml or various other things all these words are wrong it is called the unity machine like if they get very angry when you call it something different unity again game engine this is a quick introduction to the power of using a game engine or a simulation engine for machine learning I think this is really cool and more people should be doing it so hopefully you'll get enthused by this we havein got time to go to in depth first thing I'm gonna do is look at unity I'm gonna quickly tear out of my slides because I think it's best to do this live because I lost half my slides anyway so please excuse me while I tell my my slides in a second but basically unity is a game engine unity is 60 to 80 percent of the video games industry depending on how you count which is terrifying no one company should have that much power but there you have it we have that problem that's our one problem to deal with but unity is mostly free and it's really useful to play with the terms that a game engine can simulate enough of the real world to be useful for people who need to simulate the real world to pretend something's in the real world which is actually quite a fun thing to do so alongside game engines being a big thing in unity being a ridiculously terrifying percentage of the games industry machine learning has really taken off lately in three very specific different domains those three domains a cognitive physical and visual so we've got alpha go doing cognitive things solving cognitive problems we've got deep mind and like the open and eye gems and stuff like that making things walk with like weird throwing motions and stuff so physical problems and we've got visual problems like visitor room where they talk about to play doom just by giving it the visual input of doom ok doom is the old video game so cognitive physical visual as it turns out a video game engine can seem like cognitive physical and visual things very easily and very quickly it's a really good place to do that stuff so that's what unity comes in this is the MLA just toolkit it's on github we'll come back to that that's FAR's just there in case I didn't have an internet connection I'm gonna turn out now and show you this live just because it's easier you can do you write a paper on it if you want to read more about this from an academic sense this paper is really good you do see a general platform for intelligent agents it's on archive because every machine learning paper is on archive it's very it's very readable check it out so now it's almost like time time so I'm gonna push this guy and go into unity there's going to pick the right unity because I have three of them open to show you three different things that's this one okay so ignore those errors this is unity in unity you may think it looks like most other 3d software this is nothing to do with machine learning this is just a scene to acclimatize you with how unity works so in this scene I've put a ball into the scene it's called sphere I'm gonna rename it to ball you'll see it over here I've got a cube in the skirt scene which I've flattened so you can squish things I've flattened it I'm gonna call it floor and on the ball I've said this ball is made up of components it has a position component which is called a transform which is this thing over here it has a mesh which is a sphere it has a mesh renderer which is the thing that makes the sphere appear in the real world in the game and has a sphere Collider and a rigidbody these things make it affected by the physics engine which unity ships with I've also applied a thing called bouncing material to it which I'll show you in a second but most importantly I've ticked this box here use gravity and that means it'll be affected by gravity as you'd expect now if I delete this one because that's a bit of a spoiler and run this simulation you'll see the ball falls to the ground and not much else happens okay if I want it to bounce I need to give it some bounciness so I've made this bouncing material which is a thing that says this is bouncy not much not much to it and it's got bounce combined and maximum which means every time it bounces it'll get more bouncy very exciting okay so I'm gonna drag that onto my ball and I'll show you what happens then you can probably guess right very exciting okay that's my talk not um this is not machine learning this is an idea with machine like this is just a cool simulation of a ball bounces that ball will eventually bounce off screen because we told to multiply its bounce every time it's very exciting hopefully this really quick example shows you how easy it is to assemble something that would take a little bit more than the 30 seconds I took to build this to code so you can build it very visually very quickly so unity lets you build 3d complicated scenes very very quickly and that is why it is interesting for machine learning so I'm gonna switch to a completely different scene now we've made a little racetrack which some of you may have seen before you saw me speak at PyCon it might move a little racetrack it's pretty much what you'd expect it's a really boring little racecourse for the machine learning experts in this room yes if we train a car to drive around this track it will not drive around other tracks very well it will local maximum problems all that that's not the point points to how easy this is in this track we have a car this car is actually some sort of dozer it's not actually so it's a self-driving dozer more than as a self-driving car we're gonna make this self drive around the track and I'm going to show you how easy it is to do that so the first thing I'll show you is I have made this car this is basically a game okay so I hit play now assuming I set everything up correctly earlier you'll see that this lets me drive around the track okay this is me steering I'm very bad at steering I cannot drive in the real world I can't drive on computers okay so when you crash it just reset you to the nearest point okay so hopefully you all have seen no racing game before or played a racing game as exciting as this one you can see that you know we can drive around this track it's very good that's not very interesting from a machine-learning point of view so we're gonna use machine learning to make the car drive itself the way we're gonna do is we're gonna make sure the cars camera is failing to some sort of machine learning system so if you look at this car if I expand it over here you'll see that it has a camera mounted to the front right here and it points forward so you can kind of see the view frustum of the camera there's this like cowan thing coming forward and you can see a little preview down here what the camera can see so this camera is mounted to the front of the car and if the car drives around the track which I'll show you in a second you'll see the camera the front of the car is connected to it okay so it's moving with the car we're gonna feed that camera into a machine learning system and teach the car to drive around the track by itself the way we're gonna do that is we're gonna use these white points if you look on the track there's these orange things here which are spaced around the track relatively evenly and if you like my scene to show you those so you can see one there and then if I go forward okay so they're going on the track when the car leaves the the track and goes through one of these colliders on the outside we reset to the nearest yellow thing okay make sense to everyone it's pretty straightforward that that's code I'm not gonna show you just trust me that works that's what what's happening when I was crashing earlier now I'm gonna turn on some machine learning and show you how this works but first I'm going to show you the cars code and if you don't like c-sharp it's time to avert your eyes I I really like c-sharp it's actually grown what got over the years it's one of my favorite languages it's not suitable for the same things python suited before but it's a very good language in its own right so I'm gonna open the cars code and this is the code attached to the car itself so there's a lot of junk here because unity is a game engine and comes with a lot of baggage because it's basically we have we have a system that simulates a car so if I look back in unity for a second you'll see on our car there's a whole bunch of parameters over here that you know relate to cars because if you want to build a car game you need all these parameters when we're not using most of them in fact the car automatically steers forward drives forward it accelerates and we have only control we have is left and right because we don't want to make a machine learning but thanks to it too long to train so if I go back to the code you'll see we have all that all the junk to set that up the most important thing here is this agent action thing which is the thing we implement to adhere to the requirements of this ml agents toolkit and this agent action says when this action comes in we either go left or right to get to get me minus one on one and we tell the kind of move with that action that's all that's doing okay and then we check in with collided so if we've hit something we've gone outside the edge of the track we penalized the car with a minus one floating point and we reset the car to its nearest point and if we did not hit something we give the car a very tiny reward very small reward very large penalty okay it's really important that you tune these numbers these numbers are more up than they are science if anyone's done your machine learning you'll know that this kind of takes a while to get right this is something that we came up with it kind of works okay okay there's a collect observations thing here which allows us to pass information about the scene into the machine learning system we can give any sort of vector input we want in here so if we wanted to measure the distance from the walls we could send invisible red lasers called ray casts in the video game world out into the walls and measure how far away those are we're not going to do that here we're actually going use visual observations so we're just using that camera on the front and that's I think we can set up visually inside unity so here we have an agent action the car can steer left or right - 101 and even hit something it gets penalized if it's still driving it gets a small reward okay that's literally it back in unity you'll see we have our camera attached to the car and we have this thing called a car Academy which is the thing that just mediates between the ability to learn and the ability to control unity and here we have a learning brain I'm gonna click control on this learning brain this means we're using an inter process bridge to talk to Python that's all that control textbox does and then I show you the contents of that brain you'll see that it has a visual observation of 32 by 24 in grayscale and that's that camera attached to the car okay that's that literally it has a space type space size of 1 1 1 1 action which basically means that vector has one thing inside it which is a minus 1 or a 1 that's literally it ok that's the only setup we need to do in unity so I'm actually I live there's one more thing I need to do there's this if I connect the learning brain to the car I had a player brain there before which allows me to drive the car with keys so this is what a player brain looks like a player brain just exists so you can use the machine learning subsystem to test your world and drive it around the keyboard so I had exactly the same setup here but it was mapped to keys colleges for sniffing along so now we've set the car to use the learning brain we're going to go into a Python script very exciting first thing I'm going to show you there is some hyper parameters all machine learning these type of parameters very straightforward lots of magic numbers don't worry about it okay has anyone done machine learning and familiar I'm talking about ok yep high parameters lots of magic numbers these have been tuned over the last few weeks of us iterating on this project to get it right more interesting we're using a trainer called PPO which is called proximal policy optimization it's a very generic but useful machine learning algorithm it kind of does lots of things kind of well it's not perfect for all situations though and here we have a specific set of parameters for our car that's really it ok so I'm gonna fire up this Python script and I'm going to show you how this can work with more interesting Python I'm gonna run this Python command ml agents learn which is just a nice wrapper around PPO you're gonna point it to our Yama file and I'm gonna give it an ID ok so that's gonna fire up unities ml agents and then ask us to push play in the unity editor ok start training by pressing play so if I push play in unity this will stop training hence the flow will complain because that's what tensor does and then eventually it will start training with a bit of luck again this is very sorry ok so you can see here shrink a little bit this is training and slowly getting a reward what this is doing is taking completely random actions in that - 1 - 1 action space and seeing how its rewarded or penalized if I got a community it will be lagging but it will be kind of driving so you'll see it's moving the car around when it gets penalized gets penalized when it gets rewarded gets rewarded and we can see that reward happening here if you're familiar with tensor board from more serious machine learning projects you can use that as well so if I go here and fire that up you can see here this is all over the place right now because it's really early in its training run but it does show you what's going on so you want the reward to go up slowly over time this is reinforcement learning so we're pairing actions with observations and rewards and trying to guide it towards an optimal policy of driving now realistically this will take you know 10 or so hours if we're leaving it to train in my computer to get somewhere we haven't got that much time so I'm going to show you what I've prepared earlier because that's what we do so I'm gonna kill that you'll see here it right at an end file which is just a neural net from tensorflow that's a PP file I read here one of those so I'm just gonna show you how it works very exciting I'm gonna go back into unity and I'm going to go on to my brain which is our learning brain here then you'll see here we've got a slot for a model and I've got one I prepared earlier which is this one to infer on the GPU this is actually not a problem not the kind of situation we'd want to actually use a GPU for but I'm gonna leave it alone and then I'm gonna turn off that control thing because we don't want places to control it you needyou does not use Python for inference it only uses Python for training and I'll show you some more pipe in a second for those of you prefer python to see sharp things I'm gonna click play now this brain was trained for about a day and you'll see it's driving itself it's as bad as I am basically at this point but it is at least driving itself which is a positive thing that's very good that's very interesting we can make a game that's really fun no that's not the fun bit okay Python is the fun bit so that's great we made it work but now I turn this control thing back on and remove the brain remove the model from the brain and go into Jupiter so I have Jupiter running here everyone loves doop it up I'm gonna run some code here you know zoom in so you can see it you know say train mode true I'm gonna run that I'm gonna run some Python code so we're gonna import map plot numpy pill the usual stuff and I'm going to import some stuff from unity okay so I'm gonna run that one as well I'm gonna just check the Python throws just correct machine learning mostly often usually kind of requires Python 3.6 don't worry about it well I'll talk about it later if you have a question I'm not gonna ask unity to give me a hand while its environment you'll see this message we saw before start training by pressing playing unity so if I go back to unity and hit play then come back here you'll see it gets a hand on me the environment here and it knows there's a learning brain with one visual observation no vector observations so there's no numbers being fed into it only that one visual observation and it has an action space of one okay so it can take that - one-on-one action there's also the player brain which is the thing I was using to steer it with the keyboard it can see them both now we can grab a handle on that and just make sure we've got it so yeah we've got a handle on the learning bone here I can now have a look at that environment and print it out so there's no vector observations but I can see out that camera so that's my 32 by 24 grayscale camera coming out of unity I can keep running out so I can get more of it was moving you know it's very very interesting but now I can take random actions in their environment so here I am using Python to basically say take a random step using that continuous action range so minus one to one in our environment and do it 10 times then I click play you'll see I can spit this out in Jupiter here in this place that means you have complete control of Lili from within Python and you can make it do your bidding you can ignore the machine learning side you want and just use this to control and build a simulation I know lots of people using unity to build environments that are high enough fidelity representations of the real world then they're using unity cameras to capture images of that and using it to train machine learning for things that work in the real world completely unrelated to the simulation so if you need a really high fidelity simulation and you want to capture it with Python this is a really good solution for that this is kind of mind-blowing ly useful and when we finish we close the environment which disconnects it then releases unity back to us so this is a really useful thing if you need to do something with machine learning so I'm not going to go into any more depth about the plateau stuff there's a really good Python API that lets you go through everything this thing so there's a full API here that you can use to basically figure out whatever you want to do and there's a big Python API here which lets you grab stuff from the Union environment so you could ignore the machine learning side and just use this to control unity which is something I recommend you start with the thing I want to show you next in my last 10 minutes or so is how ridiculous this can get so this is an environment that we didn't build but I think is really cool so if you've ever seen the those ridiculous walking demos where the thing is negative self walk you can implement that yourself so this this thing has taught itself to walk kind of this is about 12 hours of training because I didn't have time because I my computer booked itself yesterday or the day before so I had to retrain all these this is about 12 hours of training this thing I pull up the docks because I keep forgetting this thing this thing has this thing has 26 degrees of freedom on a whole bunch of body parts hips chest spine head thighs shins feet arms forearms and hands its goal is to move towards the goal direction and it gets rewarded for velocity a velocity in the goal Direction head position up so staying upright body direction alignment with goal so it's not veering off its goal and it gets penalized for head velocity difference from body velocity so yeah that's basically how humans walk if you unpack it it's not definitely there's not meant to be funny that is how humans walk and it gets observation space from the world it's it's in so it can see the rotation velocity angular velocity and the position of all of those 215 different variables if it's joints that muscle it means it ends up doing this because it's not it's not really walking so much as its kind of flinging itself in the direction of where it needs to be however this has not been animated and you can build anything you like using a system like this so I'm just going to unpack this a little bit more for you because I think it's worth learning so I still have ten minutes left I think I'll show you one more thing we've built a little warehouse okay so this is a little warehouse with a little robot in it this robot needs to push this cube into the goal over here okay so we've given this robot lasers he has perfect lasers that can show where he needs to be so his lasers are these things have you see here he has one two three four five six seven lasers going out at different angles around his body he can detect the crate he can detect the goal he needs to get to and he knows what's a wall and he sends those that set of lasers out twice one at one level and one in another level okay he gets rewarded actually he gets penalized for every single move he makes which is designed to make him take economy of movement so he's designed he needs to take as minimal steps as possible to achieve his goal and when he delivers his package to the correct place he gets five okay so this is him this is him going and you can see the lasers flashing I can turn those off those are not right but you see the little guy pushing it around he's pretty good at his job right but we can make him really confused really quickly which kind of demonstrates how and why machine learning does what it does so we could code this pretty quickly we were just coding this as a heuristic but this is learnt to push this around so if I open a different versions of warehouse where there's multiple goals of different colors and multiple crates but leave the machine learning brain exactly the same he's gonna freak out so if you don't feel comfortable seeing here robot freaked out now is the time to look away this robot knows that he needs to push crates into goals but he doesn't have any concept of color so he just rapidly gets very stressed he'll occasionally do something correct he doesn't know he doesn't know why he's getting he's basically getting hit every time something happens here if he gets punished too much the environment resets itself so you may see everything disappear basically if he files too much we kill him um I'm gonna put him out of his misery but before I put him at ease I'm just gonna show you one quick really sad thing there's actually hundreds of them all going at once it's because when you do machine learning you want to train in parallel so the more information you can feed into the brain the better so often these environments are duplicated repeatedly so this is just a really quick taste of what you can do unity is completely free if you're not using it to build video games and making over $100,000 it's not open-source the Python bridge lets you pretty much control anything you like within unity and comes with a bunch of scripts that allow you to train things that do all sorts of stuff without having to think about it much so if you are interested in learning about machine learning this is a very accessible way to getting into it it uses tensorflow it uses all the industry standard de facto standards for this stuff so you're learning real useful skills and you're doing it in a really visual way okay you could even visualize just a regular on your network but put it in a visual form so you can see what's going on so we teach a lot of children how to do machine learning with this and they instantly grasp it when they can see it moving and it's really open people's eyes like nothing else I'm gonna show you one other thing I'm just gonna give you a bit of a content warning before I show you it looks like a spider to some extent so if you do not like spiders you might wanna look away it's not very spider like it just has lots of legs okay so that is your warning here is our spider it's gonna clear those areas okay so this is a spider Bop that has taught itself to walk we built this one it is its goal is to follow that that orange cube around the orange cube is just moving around the environment randomly the spider knows where it is but doesn't know much else it can be hit by it now it gets rewarded for facing it and approaching it and getting close to it it gets penalized for pretty much everything else and it's totally self to walk so this is pretty simple example pretty straightforward but we thought you know video game engine we need to make this more something that gamer would like so we gave the spider bot a gun so here is AI spider bot and here's a gun Milt which I've turned off and is going to turn the gun mount on so the gun is actually a completely separate a AI agent that is sitting on top of the spider so it's effectively a spider bot with a parasitic gun on top they're both controlled by two completely separate machine learning brains that were trained with Python the bot is designed to seek the cube the gun is designed to kill the cube and they work together so your thing's got a gun now it's pretty efficient so we did this we did a version of this at PAX the Penny Arcade Expo which is a consumer gaming show where we were trying to explain how machine learning works to video gamers with a capital video game and they we are we hooked this up to a controller so they could also play the robot and attempt to destroy the cube quicker than the bots and in the 30 seconds we let them play the bot killed the cube 60 times in the human killed twice so the machine overlords will win eventually but this is about this is about an hour's worth of work in in Python immunity to get this going excluding the visual aspects the thing I want to show you before I finish is that this behavior is embedded in the object so if I go here and I do placate the robot without doing anything else and just move it slightly so there's a couple of robots duplicate it again duplicate it again alright so there's a couple of robots there and run this there's now more than one robot in the environment in theory there are so our camera is designed to track the main robot substr go into the scene here they have no concept of each other existing but they know each other exist in the physics engine so if you have enough of them they will eventually make a big pile of robots where one will ascend to the top they will they will use the others as a platform with which to shoot the cube and then I'll just stay in this big during chattering spider bot pile it's horrifying it's legitimately horrifying so I'm gonna show you one more one more thing quickly because I've still got three minutes left so this is another unity demo that we didn't make this unity so I have lots of unities open this is tennis so this is tennis which is two hands which are robots trained with Python that can play tennis okay and the fun thing about this one is you can do imitation learning or behavioral cloning so you can hook one hand up to a human and have the other one learn how to play the tennis based on the human instead of based on reward signals so you can do behavioral cloning online and a recording which is real-time using this algorithm and then you can feed all the information into Python and tweak it to your heart's content so that's unity with machine learning for Python I'm going to go back to my slides in quickly finish I'm gonna jump to here so the brain is the thing that encapsulates logic you can swap a brain out for being controlled by human or you can route your blame to Python to be controlled by either a neural net you've trained or for training purposes it's really powerful the agent is the thing inside unity which is the thing you're controlling with your brain it assigns rewards it's linked to a brain it performs actions okay so the agent in this case was the robot the hands the spider bought the car hopefully that makes sense and these three objects exist inside unity and then talk to Python to let you do the training okay what we saw was reinforcement learning today but you can also do imitation learning as I said which is the behavior for anything where you teach it as a human what to do really really powerful stuff can come out of this so the reinforcement learning we did today was actions observations and rewards we had some actions we wanted to do which we rewarded based on you know how it went and we've got certain observations to generate those rewards or not actions are just a number so you know you can handle it however you like in your game engine as you saw I was taking minus one for one direction and one for the other direction when we had the car there can be whatever observations can be vectors so you can you know distance to a thing or a count or a number or B pictures coming from you news cameras rewards are just more numbers okay reinforcement learning is just this cycle of learning until you converge on or fewer can actually do the thing you wanted to do brain academy agent this is how it looks tends to flow is communicating down the bottom when you build an environment you kind of want to think about how you construct it this is pretty straightforward stuff coming up with the actions observations is the really tricky bit that's the art not the science I recommend using the Python bridge to start with via the Unity ml agents learn script which is the pre-made things and then moving to using Jupiter and doing it yourself but you'll cope with the process here's what you need you need unity you can get it from here it's really cool it's mostly free got a simulation with the self-driving car you play it like a game make it make sure it works you also need Python I like anaconda but you can use whatever environment you like and you need very specific versions of tensorflow because machine learning something and then you need ml agents which you can get from Kipp that's pretty much it I really really really like this I think a game engine is a really great way to learn machine learning and I hope you try it out if you're interested in doing this with a completely open source game engine there's a game engine called got ogod T o gog OT which is open source completely free and has a similar project in the works which works with tensorflow as well thank you very much thanks powers

Channel: Coding Tech

Views: 70,818

Keywords: unity, unity game development, game development, machine learning, artificial intelligence, tensorflow, python, c#

Id: aGU2la0fHds

Length: 28min 27sec (1707 seconds)

Published: Sat Nov 30 2019