How Forza's Racing AI Uses Neural Networks To Evolve | War Stories | Ars Technica

[Music] a car is very hard to control like it is a very complex system and a simulation engine it has loop upon loop upon loop a tire itself is a spring then there's the springs of the suspension the car has moments of inertia and weight and the way the car moves and all these things interact the car gets heat and wind resistance it's so complex and the ai has to be able to predict that i'm dan greenwald creative director on the forza franchise and this is how we've evolved to take better advantage of deep neural networks to make ai i had a pretty atypical background coming into games i was a martial arts instructor but it turns out that um martial arts does not necessarily pay the bills that great so i started looking for jobs and a friend of mine my roommate actually said that they were hiring at microsoft to test games i loved games i liked working on games so i started testing and i really really enjoyed the work i enjoyed the teams i liked how it was so collaborative after working on a bunch of sports titles nba inside drive and then some racing titles like midtown madness and motocross madness and monster truck madness and a whole bunch of madness titles eventually the original xbox was announced internally and when some jobs opened up to go full time i started working full time and immediately started working bizarre creations on project gotham racing i ended up moving to the uk lived there for a while and really enjoyed the work they ended up giving me credit as a designer which was not something i asked for but i was incredibly honored and humbled that they did that they were a great team i learned so much working with martin chudley and the guys from bizarre so we founded turn 10 about 2001 the original xbox had been out for a little while the group believed that we needed something to compete with gran turismo on sony's box you know gran turismo was a juggernaut and that really cleared the business case and put the onus onto this very small team there are four of us to come up with a creative pitch we got together and put this pitch together for forza and it was a it was a new take you know we were looking at the racing space differently we're embracing new trends web 2.0 customization ugc that was something we thought was really going to be important and it worked so well with car culture you know i lived in car culture i love cars i tuned cars i raced cars i loved that stuff and so did other people on the team so that idea of customization and building your car instead of just buying it was something we just we loved forza obviously it's a racing game it's in the car genre i think different players are looking for different things in forza when you think about motorsport they're looking for simulation racing at ten tenths competition it's being under threat it's driving a car under duress it's mastery horizon's about exploration it's about hanging out with your friends it's about finding new places freedom fun jumping the car around having a great time the first forza motorsport the original that shipped in 2005 it started on the original xbox [Music] so we made a prototype and that was a track that it was stitching together a whole bunch of different ideas and we built our physics engine and quickly realized that our physics engine needed some assists on top and at that point we were faced with a fork in the road we could either layer assists on top that you could turn off or we could change the foundational layer of the physics engine itself and the engine would become more quote unquote arcadey we immediately went to this idea that the physics engine was sacred and it has to be cutting edge at all times and the layers we add on top are things you can enable or disable that make it more controllable to control that first prototype the physics engine was so hard to control the controller because you a hundred percent input left 100 input right you can get on the throttle off the throttle immediately and anyone who's played racing games you watch their hands that's how most players play it's incredibly digital even though they have analog sticks so we need to put assists on and buffers that would change your input they would kind of interpolate and guess what were you really trying to do that doesn't happen in a real car but we didn't add that to the physics layer we put that on top so sometimes the inspiration is new tech like the original xbox had a hard drive and it had a nick card and that allowed us to create new experiences so we were inspired by things that would use that that actually birthed drivatar because drivatar needed to create big data caches and you really couldn't have done that without a hard drive drivatar is a ai technology that we worked on in the original forza motorsport with microsoft research in cambridge and at start it was a machine learning system on a hard drive using a bayesian neural network to record lines and characteristics of how somebody drove a car and that's how it started since then drivatar has evolved considerably as new technologies have come in new ways of doing ai have come in and we've learned so much from how our players play and all the data we get from how the community in general has interacted with the system so the drivatar that we work on today bears very little resemblance to the original one so in the original forza motorsport there was this team in microsoft research from cambridge and they had a thought that why is it that juan pablo montoya who is a f1 driver at the time and michael schumacher who is another f1 driver at the time could both be so incredibly fast but they drive so differently in fact you could tell which driver was driving which car based on how the car yawed and how it drove in the rain and how strictly they went into corners so they started looking at bayesian neural networks learning networks to learn a line based on how somebody drove so they would have people like me drive a car in our physics engine and that would give them a line and then they would classify that line and then generalize it to a new application so they might take a given corner that i was in and say okay we're going to classify that as a right angle corner and you took it at ten tenths or eight tenths of the car's capability you came in with this much yaw now we're going to go to another track that you've never driven in another car you've never driven and we're going to replay that but of course you can't replay it because it's got new variables so it has to then use the neural network to develop a new line that was a really interesting challenge but in some ways it was a mismatch with forza which is about building cars falling in love with cars which means we have hundreds of cars and they all have upgrades so that whole like how different is one driver versus another in the same car that's a level of subtlety that requires the cars to be same when you've got two people in cars that are radically different a car may have 400 more horsepower or a thousand more pounds of weight that really changes how a car drives so from the very beginning we had an interesting piece of technology but it was a slight mismatch with the goals of the game ever since then we have encountered new mismatches that are for us to either design around as designers or implement around with technology it's hard to know even where to start when it comes to ai challenges i think you gotta start with what players expect out of an opponent in a racing game what they love is seeing their friends names and kind of a a de-rezzed version of what they think their players are almost like a cartoon or a characterization and we found that when we gave really subtle representations of how your friends drive people didn't even recognize it they had to get kind of bombastic they had to go beyond how they really drive and that creates unpredictable ai which is one of the problems players don't want meanwhile you're meeting players where their skill level is if you're a very skilled player you can lap a car incredibly quickly so how do we keep the opponents grouped around you or predictably around you and what if you have a catastrophic mistake you don't want to necessarily restart the lap but if you go off track it could be quite a while to get out of the sand and back on the track well now you know the opponents are long gone so we've got a set of problems here but underlying those problems are a much deeper set of problems that actually is where more of our technology goes so those are more the design challenges that our technology then looks to match part of those technological problems are and a car is very hard to control like it is a very complex system and we're amazing i mean humans we will throw a car into a corner and just intuit that it's going to be able to take this corner 60 miles an hour because i've done a couple of laps on it it probably can go at 80 but we are safely hitting the limits of you know 80 percent what the car is capable of really quickly if you're experienced that's amazing that's a big challenge for ai and then on top of that dealing with us humans on the track so the opponents the ai opponents have to deal with this incredibly unpredictable nature of a human that sometimes we're looking for that simulation we want to take it seriously or you come in and you're just trying to smash things up because some players want to do that the opponent the ai opponents have to deal with all that so the set of problems we have to deal with in ai is multi-factor it's design challenges of how do we delight the player action as technological challenges of how do we get this computer to deal with the complexity of this whole situation so on the xbox one the original i still get that confused on the original xbox and the xbox 360 our drivatar system was local to the hard drive and on forza motorsport 5 we switched that to being on servers so those these were thunderhead servers for xbox live this was a pretty cool change and brought in a whole new set of problems we had drivatar fairly well tamed on the local box because it wasn't really being infected by other people's data it was really about local drivatars we trained and your own drivatar which dramatically reduced the amount of variables coming into the user's experience when we went to thunderhead all of a sudden it was a massive amount of data which is awesome it's great for training a neural net like data is your friend when you're training learning networks however i feel like we've had a tiger by the tail ever since then has been that your friends are coming into your game and their behaviors are showing up in your game so technologically it was really cool and from a user experience it's had a couple of sparklers that are incredible great user experiences but it's also had some irritants that we've had to look at how do we shackle and how do we tame and how do we change the user experience [Music] all of a sudden the drive to our system wasn't just using your data so you might train it with a hundred laps that's pretty ambitious that you actually did 100 laps in various cars on various tracks and that would train up the system for how you drive and then we had trained the data that was on the disk and and was in the game based on you know hundreds of thousands of our own laps right all of a sudden going to thunderhead we had millions of laps within what a week it was crazy and that allowed us to take a system that would classify and that's what neural nets do right there they're classifying you think about the uh the joke in silicon valley right not a hot dog that's that's it that's what ai does it classifies and the thing it classifies best is not a it can tell you it's not this it's not that and the truth is that's magic i know it sounds not very ambitious to most people but it's pretty amazing so all of a sudden we had millions of data points for it to classify so the ai got very fast it got very weird in spots it did things we didn't expect and we had to start like clamping that we didn't want it to do this behavior but the problem is it's a bit of a black box you don't know why it's doing the behavior that's what ai is you feed in data and it feeds out what it wants to do it doesn't tell you why you know it's like a two-year-old you just kind of have to take the output and then you can put a restrictor on it and have it do less of that or modify its output in some fashion so the data was a huge boom we started learning about how players drive we started getting laps we didn't expect things got fast in ways we didn't expect they got slow in ways we didn't expect and weird inaccuracies showed up in our system that we really didn't expect now i'm going to have to dive super deep for a second so the way that we classify a track is using what we call a chi value it's basically a value from zero to one where zero is the inside of the track and one is the outside of the track and when we record a line it's a series of points now those points are said in that chi value so you might say that if you are towards the outside of the track you're at like 0.9 chi value and if you're at the inside of a track maybe you're at a point one or maybe you're lower than that like a point zero one so all the way around every single track a track is expressed by these chi values which gives you three lines an outside line an inside line and then a middle line the line the ai is going to drive or that a player recorded in these chi values here's the issue when you want to classify and generalize you're trying to say this corner is like that corner and that corner is like this corner and you're starting to stitch together virtual lines based on a whole bunch of information now it's going to look at the lines ahead and the lines behind and new inaccuracies come in because you know the corkscrew at laguna seca is a very complex corner and you can't really generalize it to any other corner in the world and the system may try and we don't have control over it because it's a black box inaccuracies show up because if you think about a track it's easy to conceptualize it as there's an outside and inside but if you really think about a track the outside and the inside move you know a curbing shows up or doesn't show up it gets a little bit of extra concrete over here and so all of a sudden the scale of that chi value changes because the track's really wide and when you go from section to section it changes in width so the chi value is still accurate but you're getting little changes that make it harder for the system to deal with what that would end up with is lines that have an imperceptible small kink as they get replayed because they go from one track which has no curbing to a track that has curving or whatever right there's a whole host of these things that could make it different while the line follower sees the kink and says how am i going to make that clearly i have to hit the brakes and so we got this braking behavior that showed up out of nowhere so what do you do you say well hey don't hit the brakes so sensitively you know kind of overlook some of that stuff but now you've actually told the controller hey ignore some data sometimes you do need to hit the brakes but this time we're telling you not to do it so all of a sudden we're putting programming on top of the black box which is more like traditional ai where you're actually programming if this then that so we kind of got a hybrid now where we've got a black box training some data and then this thing on top of it meanwhile we wanted to keep some of the behavior your player did so we we wanted you to make contact if you were a player that made contact players don't always appreciate having contact made with them so we then limited that to only your friends so strangers drivatars weren't allowed to make contact even if they wanted to so this is yet again another limitation put on to this set of data the sparklers that happened were water cooler talk we'd get people coming around saying hey i drove against your drivatar and it was doing this or it was doing that it was really fast it was really slow and it kind of reminds you of them when you play in the game these were awesome experiences really delightful for our players but they came along with a bag of behavior like brake checking that nobody wanted to see or getting pit maneuvered in a corner here's another complexity watching all of the other cars around you is very complex in fact most players have very poor perception of the cars around them if they're not on screen in front of them they don't realize there's a car in the rear quarter panel even real drivers struggle with that great drivers have almost like radar they know where everybody is around them just like a great soccer player a great basketball player now give that to an ai where we have discussions of performance and we can't all of a sudden have it go exponentially that every single opponent is tracking 16 other opponents so we have to limit that so for the most part our ai is only watching one car and if it's not you it might run into you without even knowing in fact it might even hit you and be continuing to hit you and not know that it's hitting you at all when we get more performance we're able to apply that performance to watching more things but it's just been one thing after another one way you can make things more efficient is you can run something offline maybe on a server or it could be run during load time so that is when the game is loading but before you're on track we could run something and cash a table full of information so for example there's a bugatti veyron on track it's a heavy car it's incredibly powerful it's a tricky car for ai to drive because it's limits are ridiculous it's so heavy it is so performant we could run that track through a lightweight simulation during load time and then cache the results so that when the ai has to look up how fast can i take this corner oh through my simulation i can take this corner of 60 so i should throw myself at it at 60. i have to break at this point to hit that that allows us to run more ai in the track because we're now more performant so every one of these cars could have look up table now if our cars were all the same then we wouldn't need as much but the truth is it's not just a veyron it could be a veyron that you've thrown upgrades on maybe racing tires or downforce or some other piece of technology so we have to run in load time because the number of permutations across 700 cars and all of those different upgrades is innumerable we could not cache that amount even if we did it on a server so we have to do that in load time and we choose to do that in load time so that we're more performant in real time but now there's other ai on the track and maybe there's a puddle or maybe there's rain or maybe there's a human on the track how am i going to deal with this should i pit should i not pit am i damaged i mean there's a whole lot of things that are part of what we call racecraft now each one of these have their own set of problems places where you can apply learning neural networks deep neural networks places where you're better off with caching and then places where you look to simplify and we've got numerous places that we have looked at simplifying over the years and honestly when i say when we get more powerful boxes we get greedy that's what we remove we remove simplifications on things like the ai or the physics engine itself we will do new research projects to find new complexities in real world physics and integrate them into our game and then in low time real time or even offline we will use caching techniques to reduce some of that complexity so we can run at 60 frames per second so in most racing games there's this concept called rubber banding it's often highly maligned by players but i'm going to try and impartially explain why it exists and then i can tell you how we use it so in racing you don't restart very often right so you start a race let's say it's five minutes long you run off the track and uh getting back on track is a little bit difficult so as a result you fall way behind the opponent players we need a way for you to catch up to those players or else you feel like all is lost and we have a rewind system in forza motorsport and forza horizon which also helps take care of this so a catastrophic failure you can actually rewind and redo it drive a chart and rewind help us because we've got a myriad of lines to follow and we have the ability for you to fix your catastrophic mistake but you still end up with wanting to have more pack around you it's just more entertaining racing so we've looked at a couple different ways of doing rubber banding we don't change the opponent's characteristics but if they're far ahead of you we start scaling back their torque we start scaling down their friction we start adding weight we play with aerodynamics the reason we have to play with so many parameters because if you play with any one of them too much it does crazy stuff right you can't double the torque the tires will just spin you can't do too much air friction or the car slows to you know almost a crawl at like 160 miles per hour so we've tried to be very sparing of where we use it because if you come in contact with the car that has let's say 20 30 percent more friction in its tires it's not going to look right it's not going to feel right when you collide them because they they have more grip i think the reason it's maligned by players is when it's put on really heavy and you feel like you were going to win the race and on the final 500 meters some car comes whizzing up from behind you and pass you at the last second that is really irritating that's something we've avoided but there's a connection between rubber banding and the inaccuracies that have come out of our black box we have a setting in motorsport it's in horizon as well called unbeatable unbeatable eye is in fact beatable our all of our professional esports players toy with that ai they are so fast and they are so good that they can beat that ai without problem we could not make it fast enough because of these kinks in the chi value it just would break itself a little bit it would sort of stop occasionally the other issue is the line follower so it sees it wants to follow this line at a certain speed but our line follower was digital so that would mean it would full break or it would full throttle creating a controller that intuitively knows exactly how much throttle and how much brake to give on a very complex system is pretty hard so we were doing it that way and then we would let the ai use our abs system and our traction control system to limit how much power it got but that's not the fastest way of driving our pro drivers myself even i don't drive with traction control often i don't drive with abs which means i get more out of the tires more out of the car which means i'm faster with that car the ai is already driving with one hand tied behind its back by using these assists because it's so binary so we ended up having to use rubber banding not just in the pocket when it's around you but actually to close the gap to drivers not even pro drivers but drivers like me since then we've done a ton of experimentation this is what's so exciting we've been experimenting with a new controller that's able to give analog and we've turned off the assists so the ai now drives it can do modulated throttle it can do modulated braking that's awesome we also got rid of the chi value error by going to a new system of classifying the track basically as a result our test opponents and our test bed have gotten so crazy fast it's ridiculous our hope is to get rid of rubber banding for difficulty entirely have it completely off and the only reason there'd be any rubber banding ever would be outside of the pocket when you've had a catastrophic failure and we don't want to stop the opponents we never do that we want to slow them down a bit that would be a great use of rubber band so we've organized a team that's just been doing experiments on drivatar as well as on neural networks and how they could be applied to different places in our game our build system our tools all over the place and then we see the results of how much faster the opponents get with and without cheating what about a new controller what if we cache do things in real time and they just try new experiments and we see what works we design something we think players are going to react this way and it turns out they react that way [Music] you gotta be really careful when you ask players what they want and even when you ask them what they did we found that over the years that players have told us they used rewind x amount and then we look at the data and they actually use it sometimes 30 sometimes 50 sometimes 100 percent more than they remembered using it the feature was incredibly well used by people but it wasn't always well loved it broke up the action it caused problems with our line recording for uh ai so that was one of those features we have a look at what do players perceive versus what do players actually do same thing with uh self-reporting from players on how rough the ai was we had a look at what was really causing the ai opponent to run into them was it that the ai didn't see them which is a problem was it the ai didn't have control of the car because it was being too binary with its inputs or was it that the ai thought i should hit this person because that's what i was trained to do by my drive a chart well we can take care of the third one we can you know get rid of that but we need to do radical experiments to figure out how to get the car to control better and how to give it better vision oh yeah drivers coming from everywhere now it's a party we've constantly had to to grapple and wrestle with what players perceive and what they expect and what we're capable of i love what we figured out over the years though i think we've made a much more robust system and then i think it was sports motorsport six we added a parameter where you could toggle on or off ai aggression just on your friend opponent because we found players that really loved it they loved having the rougher race and they loved having the reminder of oh yeah that drives like john versus i i need a predictable field that never hits me the biggest thing we're tackling is we've been improving our controller you know don't all the way break or all the way accelerate you know and turning off those assists so the drivatar the actual ai is able to control the car in a very nuanced fashion where a player can a more complex simulation of the car heading into our caching of that player data so that when the ai is trying to play the car back it's trying to control that car it has a much better intuition of if i break the car is going to slow down at this rate judging what is a collision and what's not a collision and whose fault it is that's a great thing for deep neural nets and we've been applying that the thing that's impressed me the most that we've been playing with is by turning off all the assists all the scalers any sort of cheats or anything like that but improving the line value the ai has gotten so fast so it destroys our unbeatable setting which was the one that i could sometimes compete with and the unbeatable setting was cheating and now it's being destroyed by this car controller and and fixed line follower these are just cool things to see happen in technology you know so we're doing that with the research team in in redmond and we're just you know playing around with this stuff and seeing where it goes and and where it's useful we started looking at neural networks pretty early you know as part of the original xbox and the hard drive 2005. i mean there weren't a lot of game teams looking at neural networks as a result we maybe we got jaded maybe we got experience i don't know how you want to say it but when all of a sudden everyone was talking about solve everything with ai we had a very realistic eye on that there are some things ai and dnn deep neural networks are incredibly powerful for but you know they're not like peanut butter they're not good on everything right you you got to decide where you put them so a lot of the learning has been the right amount of experimentation to make these features come to life ai doesn't just have to be applied to opponents it can be applied to build systems it can be applied to optimization it can be applied to a whole lot of different areas of your pipeline and your experience so i think a lot of the learning we've had is going really eyes wide open into deep neural networks it's a black box you put in a ton of data but what you're going to get out of that is going to be a bit of a black box you got to take what you get and then you're going to have to add scalars and things on top so it's not this like one size fits all solution you're kind of doing a deal with the devil and it's powerful it's powerful but you gotta know what you're doing but it's just been super interesting we learn so much about our players and what they love we learn more about racing in the real world and on top of that we learn more about what our technology can and can't do and what's really magic about it

Channel: Ars Technica

Views: 156,859

Keywords: ars technica, ars technica gaming, war stories, ars technica war stories, forza war stories, war stories forza, ars technica forza, forza ars technica, forza racing, forza driveatar, forza ai, forza motorsports ai, forza 7 ai, ars technica forza motorsports, forza 7 war stories, war stories forza motorsports, forza driveatar ai, forza game, forza games, forza game ai, forza games ai, forza motorsports game, dan greenawalt, greenawalt forza, ars, technology

Id: XB9lf7iJbRw

Length: 28min 26sec (1706 seconds)

Published: Sun Sep 13 2020