Tales from the Trenches: AI Disaster Stories

[Music] um this is something we decided to to throw in this year Steve's idea of you know we always talk about the really cool things that we do in AI uh and wow we had great success using this architecture or this technique um what about when things get really haed up completely um because there's educational value in that those of you who have read post-mortems in the past it's what went right what went wrong and uh what did we learn from it so I think it's time for a little bit of what went wrong and what we can learn from it so I'm not going to bother introducing these guys you know most of them anyway but we're going to kick things off with Steve Ren Tales from the tent trenches AI disaster [Applause] Stories the uh portable mic here we go okay so I'm going to tell you about Microsoft baseball 3D what Microsoft made a baseball game when did this happen this happened in 1998 um actually I was working for a small company called whizbang and only about 10 people and we made the uh the code side and Microsoft did the animation and uh the graphics for this thing but uh in 1998 I gave a a uh game developers talk on the cooperation in this game the AI cooperation between the uh players and uh I'm actually still proud of that work today um but it's interesting to come back 17 years later and tell you all the horrible things that happened okay disaster number one why were we making a baseball game I kind of know why we were making a baseball game because well see nobody at the company was that big a fan of baseball why were we doing this but uh I know I know kind of why because we just had made Activision's hyperblade and there just weren't that many companies in the world who knew how to make 3D games at that point um but this was a huge disaster because in me I was the sole AI programmer I didn't know baseball well enough and I didn't have that [ __ ] Det detector so that when I made Behavior I could just it looked reasonable but to someone who really knows the game it's wrong right and I was missing that and it's a huge detriment if like no one in the company has that that that detector um disaster number two was the program manager at Microsoft happened to also be the game designer and a solution to telling me how to design the uh AI was to hand me this book which had about 120 diagrams like this and so you can imagine that uh you know there no mean you know you don't know why anyone's doing what they're doing it just says yeah go in this direction if this thing happens um so another ridiculous disasters and and he wasn't even on site so I couldn't even talk to him easily he was at Microsoft I was at this little company disaster number three is what happens when you combine a megalomaniac with an advanced degree who's also one of the owners of the company the answer is you get a custom scripting language is what you what you get and it was it was it was [ __ ] it was and I was the only one programming in it like the reason to have a scripting language maybe you have lots of designers using it or something but it's just the the the errors in it were non-existent to compile errors and you couldn't debug it for anything and one of the benefits of a scripting language is you could reload it while the game's running well it wasn't trustworthy enough to do that um probably took you know 10 times as long to do anything um using this thing disaster number four was this is a two-year long project and for about the first year we had no 3D graphics and so I had to do all the AI on this isn't it I found this on the web but something like that looked like this I was making a 2d baseball game really and I was getting bug reports on ah this wasn't right and so I'm fixing it but I'm hyper optimizing the damn thing to 2D and once we did get the 3D oh my God it was terrible people aren't facing the right way the transitions look like crap like half of the AI it really is making it like look good and uh it was sort of just we gone I made a great 2D baseball game is what I did and uh so this this was kind of horrible the last thing kind of an animation thing but AI is so hooked to animation is that and this was touted as a feature like our company had this technology was the animation drives the motion notice how his right foot is nailed to the ground now this is a ridiculous example of it but as an AI programmer to get people to move some where I had to start an animation and hope the feet would drive them in the right direction and sort of twist the hips to get them steering right and when I got to the right spot I just slurp to a stand or something like that another example is this guy who's rounding third base kind of notice he doesn't touch the base that's sort of that's sort of an important part of baseball but there was just nothing in the animation engine to make this happen um here's another uh here's the picture walking up to the mound you can see it's kind of this stilted animation is his ankles and stuff don't move right so they walking looks a little weird and then he slurps at the end anyway so his feet slide so like what the hell okay last thing I want to tell you about this is that something actually very funny happened is one day we uh imported the uh the batter character and he uh came into the game as just a puddle of polygons right it's like he stepped into a teleporter as a fully formed human being and he came out the other Ides is just Tangled mesh of polygons with a bat sticking out of them cuz he was still holding a bat well this system was so robust that somewhere there were feet in those polygons and this poor thing was trying to wiggle toward the plate and it was slowly making progress and the whole time this bat is like waving like a flag out of the puddle as he's walking up to the that was awesome I wish I had a movie I'm sorry okay please learn from these disasters thank [Applause] you does this work too can everybody hear me all right um I have a couple of little a couple of stories actually um the first one happened probably a little after you probably like 2000 2001 and it was a game it was it was probably 15 levels or so I I I'm not going to name any names I'm not that person so um the this is not a screenshot of it even but it's very representative it was it was a fairly realistic uh World there was a lot of stuff in it and we were about I want to say two weeks from going to a wide a wide beta and this was somewhat early on when it it was a PC game and and um it was when a lot of games were having their editors released with with with the title itself so it was uh you know you could you could fuss with the levels and that kind of thing and it was like I said it was a couple weeks before open beta and there was some AI issue where the AI couldn't get into a particular spot and I figured what it was was the the the classic right the the um some object had a bounding spere on it that was too big and the guy couldn't squeeze by even though he thought he could and so I went into the editor and I clicked on that particular Coke machine and uh sure enough you know great big sphere on it so I'm I'm moving it down and out of the corner of my eye I just happened to like glance at the the name of this particular object and it was quite possibly the most offensive thing I have ever read in my entire life and I was like who who who who is the level editor on this game you know like like who is building these levels because I started clicking around because it wasn't just this Coke machine it was every stupid little object in this level had a crazy racist or offensive or some form of just ungodly name like every single object in the game and it turns out it was our best level designer he had had like seven or eight of the 15 levels were just him we we counted it up I think there was over 4,000 like horrid things accordingly apparently over the course of this guy's career this was his thing like he was this very you would never guess very quiet kind of hi guys how's it going and what he liked to do when he named levels objects was like just go berserk like that was where he got rid of his anger I guess and so the problem was is that this was going to go out into the world and anybody that wanted to could open the level it had never happened before apparently and so he was okay up until this point and so what we ended up doing was we ended up uh uh using the vulgarity filter in the editor and and it worked out well because not only could we find all of the instances of you know we also put him in a corner and said get to work guy um um but but then if somebody else using the editor out in the world made their own map it wouldn't let you save it until you changed anything that was vulgar or whatever and so anyways that was that was that one um the other one this is this is later this was when I was at Sony I've I've talked about that the that giant engine I did for the basketball game at Sony uh uh several times and it was a huge hierarchical like expert system and and over the course of years because it was a basketball game we were iterating on it every year and so content kept going more and more and more and by the end uh uh there was like six Megs of XML AI data in this in this engine it was huge right so it was this huge tree and um we had shipped and we were we were like I want to say like a month or so into pre-production on the next year and one of the designers who had had like a a huge chunk of the tree kind of uh uh to himself the year before he came to me and he said hey you know I'm having problems with this one situation it just won't seem to Fire and so I I go in there and I'm looking at him and uh I said oh oh well uh you know this you you've got this thing commented out here and and he just went white and so basically if you think about he had wanted to test out this particular part of the tree let's say and see how it's it's it's a two a a two Branch right there over over here is the other side and so he he just didn't want the AI in any way to use that side he just wanted to test out the other one so he just put a comment out right there right well when he went white I said well when did you comment this out and he said few months ago and so effectively this poor guy had commented out 75% of the years worth of work and we had shipped itow and tough noogies for him now lucky luckily the AI system was robust enough that it still looked like basketball and luckily also he had been working on ball handler types that were not in the top 10 most played teams so nobody really noticed lucky for him not only that but we got a nice feature that was done for next year's game for free but the thing to take away from this is if you are if you do have some ability in a hugely data driven system to comment stuff out especially in a tree structure that you can just go way the hell up and say oh up here is fine um make sure that you turn anything that's commented out bright green pulsing something so that you can make sure that it's not that way when you ship and that's it all right so the year is 2009 and I've just left the games industry to work at a big military contractor where I'm working on this character who is a tech demo uh to how we can do cult cultural training right so this is a mixed reality character she's projected on the screen um the Marines are in this facility where we've got live role players and all sorts of stuff to make it feel really immersive um and you know she's supposed to to help them learn how to win hearts and Minds make friends not enemies right like they're good at combat but we need to teach them to do all the other stuff that lets us actually win Wars and not just shoot people um which is supposed to be where we're over there so the previous mixed reality scenarios were combat training right there they they're very fast it's hard to see but there's an Insurgent in the back there who's holding a gun on the the poor woman in the white dress um so you shoot the Insurgent not the woman we wanted to move Beyond this to more cultural training scenarios that would last a little longer um you know even 10 to 15 minutes um non-kinetic scenarios we called them so kinetic is things that shoot or blow up right so non-kinetic scenarios um and she needs to speak pastu because the Marines are going to go to Afghanistan and what people speak in The Villages where they're going to be going is pashu um oops so uh we got one of the role players who worked in the facility there um we recorded all the audio in the map Studio that we had also built for them as part of this project um so we had all of the animation and all the voice recorded together so you play a line of dialogue you play the animation that goes with it right I mean I'd like to do better fancier stuff like I was talking about yesterday but I've got three months to build this character right and almost no money I've got an artist halftime and me and you know that's it so um but I found out when we sent it to the customer to get them let them see it for the first time that she speaks farsy which is what they speak in Iraq not P to [ __ ] so so this one has a happy ending though happily so we re-recorded all of the audio with a real actress well I mean she was real too but another actress who did speak past you um had a better quality because we did it in a Sound Studio instead of an a mocap studio um then we sliced all the animations into gestures that were you know about a second long so you know youve got a gesture like this and a gesture like this um and then you pick a gesture sequence randomly the gestures are tagged for the types of lines that they're good for the lines are tagged for the type of gestures that they like you pick a gesture sequence so every time you go you know when you replay a gesture you're blending from a different gesture and into a different gesture and those Blends are different so the motion is different so it feels like you're doing it again um stupidly simple worked much better than I would have thought I was really worried about the problem of getting the Beats to synchronize on the emphasis of this speech um it turns out that didn't matter at all in fact I had people compliment how well that worked um which really surprised me I don't know if it's because we started the gesture with the speech or if it really just doesn't matter that it happens to happen every once in a while and people see it and think it's good and they don't notice when it doesn't happen um and it breaks up the rep repetition really nicely um because when you play the same animation um you get different blends when you play the same line of dialogue you get different animations um so even though though we had to reuse assets a lot because this was a very fast project um it didn't get noticed nearly as much as it would have otherwise thanks hi there I'm Mark Sanger um I am currently a where's the thing ah okay there we go currently working as a uh uh consultant freelance but uh back in the day I worked at redstorm entertainment and um this is from the very early days of our stuff we um on top of making our um more uh larger scale games we also had a number of very very small games that were um that were on like an 8 n month release so we were just churning them out and it was uh in the um Power Play series it started with uh politica and ruthless. comom and our very last one which I'm going to talk about was Shadow watch and and it was designed to be a kind of um top- down orthogonal strategy sort of excom Syndicate e uh fallout e sort of thing um and um there was sort of an onus on us and that that it was coming out so damn quick to have some really Kick-Ass stuff especially with the with the um with the squad strategy AI so um we spent a lot of time on the AI like lots and lots of time like like lots and lots and lot lots of time but one of the things that we really wanted to do was um have the uh enemy squads be intelligent enough about their environment to recognize when they were pinned and double back on the player and come through another door sneak attack everything's really cool and we managed to do this two ways one that it was very very room based and this actually is the game um so uh the um our enemies just had total knowledge of the whole um of the whole level and we kept the players knowledge both from them not knowing that and also by using a fog of war did you mess with my slides Dave okay so so it was a fog of War all right it was a fog of war and um so they could not see so you would have your enemy vanish and it's uh turn-based much like XCOM um so you know the guy would go around and come out through the other door and stuff like that and um you know it actually you know for for the for the the product it was it actually did fairly good um and we were kind of proud of this and you know like a ton of time went into this and this is back this game released in 2000 so we had a slightly more you know smaller but more Savvy audience that kind of knows you know what what the current state of games is so we were looking on the forums and there was lots of stuff and we were particularly interested in what people are going to be saying about the AI and um they remarked on that doubling back thing and they were amazed at how well we had faked it so their their basic thought was they couldn't we couldn't possibly be doing pathing like that so we must be pulling the guy off the map waiting a couple of turns and then popping him back at the door and it was kind of disheartening cuz we really we could have done that but instead we did this really neat thing and this this was very early in my career and really informed my career in that um when you're doing really really cool AI make sure that there's some indication of what you're doing because if you don't then your audience might just think you're faking it and thank you very much this here hi everyone uh My Name Is Res Graham I'm an independent um I was most recently at Maxis working on the Sims 4 um so this is going to be a story about a game that I worked on in 2010 called The Sims Medieval for a little bit of History what the Sims Medieval is is sending the Sims 3 back in time uh so that they can play in this fantasy world uh it wasn't just like Sims 3 with a new skin though we added things like professions so we had a merchant a Bard a monarch and so on and you could play as any of these professions um and have multiple Sims and so on we also had quests so um what we really needed were NPCs that were smarter and I've given talks at previous gdcs about how we accomplished the let's make Sims uh constrained to a particular uh area or a particular thing that they're trying to do they have Quest NPCs and so on So This Disaster story takes place here I took this screenshot in a brand new game um and uh this is an empty world where in the very center there's a pit right there that's my nice little pit and in this pit lives a pit Beast who looks like that so this pit Beast goes and uh you can do all sorts of you can you can execute people so you have the execution here who was like tossing someone down in there uh and all sorts of things like that the pit Beast was this really cool thing that added to the flavor of the game now the guy there on the right with the axe is Ron the Executioner who's this guy with his amazing Viking beard Ron The Executioner's job is not only to execute people but also to throw in gigantic hunks of meat uh to the bit Beast who then consumes them that's our basic system that's Ron he's our NPC and that's what he that that's what he can do in this world so of course this is a a talk about bug so we got a bug the bug was basically the Executioner ran up to the the edge of the pit and leapt in that seemed bad my first thought was okay is there some kind of weird animation ik thing did he like you know sort of tweak in there no he was in this pose um when we saw the screenshot it's pretty clear in this case this was a screenshot of my player Sim doing this um yeah so this is what can happen is the player can click on the pit and say jump in it's meant to be high level content you're a you know maximum level character you go in fight the pit beast and if you win you get Epic Loot a standard role playing game mechanic uh it's not available autonomously no player or NPC Sim should be able to do this autonomously so we're like what happened well fortunately uh and I gave this talk as well several years ago about uh tools we have tools that let us see we got really really lucky because this is one of those bugs where it happened once and they never got it again well we managed to get it just through sheer chance um and we have really great tools specifically this history down here that lets us go oh [ __ ] what did he do why did he choose this and we got to walk into it so we got to figure out exactly what it was it was the insane trait on the Sims 3 this was fine so here's how the insane trait worked and and I won't name any names either but this was rather crazy basically what it does is if you have the insane trait it ignores tests which somehow was fine on the Sims 3 um because there's you don't have a lot of Sims where just who need to be locked into places doing things in a fantasy simulation game where there's an RPG and you need like a quest Sim to stand there with a big Arrow above his head that becomes more important uh or you have Ron the Executioner who needs to be at a particular place doing a particular thing Um this can also completely break the game right if we have the Constable or other certain NPCs that if they die we're hosed um the entire game breaks down now the cool thing is is that the designers came to us and like you know we like this we like the idea of Ron the badass right we just we just want to be able to control it that's the thing that we really want and so we want to allow designers to have this this control over the game well we fixed it uh by fixing the horrendous that was the insane trait to make it do something more reasonable um and incidentally for The Sims 4 we did the same thing uh we made it so that it wasn't that the insane trait wasn't so insane um cool so what is what is the purpose of all this the first one seems Seems really obvious to me my first conclusion point but see it is not in my my many years in the games industry which is don't bypass existing systems they're there for a reason there's a reason why we have autonomous tests in our game and if you completely bypass them if you just short circuit stuff then it breaks um fixing this bug fixed all sorts of those it happened once bugs all over the place the second one which is slightly less obvious is use automated testing it it was painful to try and find those we got super lucky with that one bug on medieval on the Sims 4 we're like you know we don't really want to have to have to fight this anymore let's do automated testing on the4 we had a huge massive automated testing Suite that we just ran this this smoke test basically that over and over and over that would find all of these little Annoying bugs so do those things thank [Applause] [Music] you

Channel: GDC

Views: 260,077

Keywords: gdc, talk, panel, game, games, gaming, development, hd, design

Id: __5whYgSTV0

Length: 24min 42sec (1482 seconds)

Published: Mon Oct 31 2016