What You Need To Know About Video Game Quality Assurance

hi everyone welcome to siege 2018 convention my name is Jonathan Betancourt I'm a junior game designer at tripwire interactive and what this panel is gonna be about is what you need to know about video game quality assurance so in this panel we're going to be asking questions to everyone here and without further ado I'm gonna let everyone introduce themselves so we'll start with I am a programmer active I work directly with Jonathan and I'm at the other side of QA and that's hopefully the perspective I'd like to give you this what what we think and what we do right Jonathan yes hey I'm Brian marques I work at adult home games I'm on the production team they don't really have a title but I do production there but I'm similar I will be talking about QA from the other side my name is Cody Ulrich I'm a Quality Assurance tester at tripwire interactive for tripwire I have worked as an intern for game production at Cartoon Network and I've also volunteered at Tyra's is hrx a couple years graduate of Kennesaw State University with a bachelor's and computer game design development so I'm basically here to speak about the two different sides of QA on the developer and production side of things my name is Toby section I'm with game scribes we do localization we also do game testing compliance testing compatibility testing all that kind of stuff representing basically the third party anybody you might need just different ideas that might help you with your QA needs awesome so we got to meet everyone um so I'm gonna start off with a question just to get things going about QA so what exactly is video game QA my perspective in QA is playing the lead QA is just get to play games all day I'm sure very few of you have that misconception but you do but you have to play it clinically and that can be boring and you have to have focus and there's really two sides of QA there's the play games because like breaking stuff play the games and find the problems with them and there there is that that is that's one big but there's the other half and that's the targeted side that's what's not often talked about where there's a bug we know about and we're trying to find out how to reproduce it we say we know that the game crashes around here when we're doing things like this QA Department find it and tell me how to reproduce it because once a bug is reproducible once I can follow steps and do it it's dead it doesn't live long at all it's the bug that happens one time in a hundred or some time during this level maybe it happens that are really difficult to find so that's that that's for me what QA does is they find bugs both from in cold just find things that are wrong and find things that we know are wrong and figure out exactly how they do it in addition to all the other things like yes for three parts is just does the game do what it's supposed to you know sometimes bugs are no typos is a bug you know or it's something that we want to make sure that the game is what's written on the box pretty much nail bad basically it's testing the game out so you make sure there runs well for the player so that when you try get all the problems that might come up or that they would anticipate so therefore they don't you know reach out to you on the threads and come back to you on the boards and then and try you to die and that's that's really it will - yeah all the time yeah really what you want to do you want to make sure that you find everything through the spread testing like you was saying or through comprehensive testing to make sure you get every single corner and if you miss it well that's too bad so since we found out kind of what cueing is is what kind of qualifications are needed for someone who wants to pursue a career in Quality Assurance for a video game studio oh man I would definitely say don't just say you have a love for games you definitely need some sort of technical background I can't tell you how many people have come up to me and they're just like man I love playing games I would be perfect for quality assurance it's true like you really you really need to not just love games but also hate them and learn that you need to destroy a game no matter how much you love that game you need a you need to destroy it I don't care if it's your baby I don't care if it's like your favorite game ever you need to kill it and that's something you need to learn over time but that's something you still want to have like some sort of technical background on whether you went to school for coding or whatever you just need to understand that games are not just playing them if you're in QA yeah you will be playing a game and you'll play it from start to finish and then you'll do it again and then you do it again you'll do it again so you just need to have an understanding of what you're getting into on top of you know how many people here didn't get into queueing that's actually one to address how many are looking to get in cuz they think that's their foot in the door to get in to do something they really want to do don't be afraid to put your hand down that's probably a lot of people okay well you should and that's okay but I just want to dispel a myth I have I heard really crystallize in a previous talk which was everyone thinks to gain the games industry the breaking insertion and probably a lot of you over there was become QA and that's your way in a game serious and the problem is that's kind of true but it's not always true and it's actually very rarely the case you want to it's an entry-level position or can be but the senior the mid-level q ways they're very valuable and I want them to stay there a good QA person is extremely valuable and it's a high level position and it's very skilled and a lot of people don't think that they think that the QA department is just a whole bunch of kids right out of college or interns now let's play the game and the real work happens somewhere else and that's so not the case I just wanted to make sure someone said that to you Chua is extremely important for the entire like production pipeline wearily yeah it's like if the QA doesn't get done the producers can't do their work or the marketers can't do their work or the designers can't do their work like the game won't release unless QA is done so it's super important and yeah it can be a good foot in the door at least for like my experience I'm sure Cody you've had the same different thing you the QA team they're the ones that work with everybody the most so it's like you'll find an issue it might be something programming wise you could go talk to a programmer if you find something art wise you could go talk to an artist like you have the ability to talk to everybody on the team so it's a good way to quote/unquote put your foot in the door for that but there is also that understanding that QA is probably the most important part of the production pipeline so they might want to keep you there as well if you have to know who to talk to that requires quite a bit of knowledge of the way games work I mean there's a big difference between an asset that has the wrong texture and an asset that's performing incorrectly one's an LD artist and to know that difference is to not waste someone's time that's what makes you valuable we want you to come take the right problem to the right person and that's not your boss's job that's not the seniors jump think it will do that but it really ought to be yours when you file a bug and figure something out you should already have in your mind this is probably programming it's probably code this is this is probably our taxes and I certainly appreciate when an QA person comes to me and says this sword doesn't work I mean what you mean by doesn't work is it's got a bad particle effect I'm a programmer we talking to me for well because it doesn't work in its essence yeah I appreciate someone like Jonathan is very good at this who knows not to talk to me and waste my time but some things are talking about her like in-house QA departments yeah which not that many companies actually have that it's actually it's a little bit of a luxury but within the person that they do have to have somebody at least who understands QA and basically can head it up because otherwise you go to a third party so where the company doesn't have like any extra expenses or anything else like that so another way to get and not just through a company like they were saying is also to work with any kind of third-party testing facilities or something like that be able to work with them as well but then they show to be able to find the bugs spell it out be able to replicate it give them at least at least at least the pathway to be able to possibly replicate something because that's where it's getting its held up a lot and sometimes I could take like days be able to replicate a major bug and then you up with what we call known issues oh yeah yeah but just to kind of go back real quick as far as like liking to kill things thank you saying we had fun test her new bed she was like our league or lead tester that we utilized and he was very very proud of himself and that he was able to kill a favorite Li loved childs you know character over and over again kissing kids games the characters are not supposed to die and the kids are supposed to be successful through the game which which unfortunately that doesn't happen a lot so he was very happy he was able to kill Dorothy's for like five times they thought they fixed it and then he was able to figure out another way to take her out so basically he's you know that was kind of that's that's kind of funny that is incredibly valuable yeah it really is yeah and then you have to be able to be on the hunt you have to find it and be able to try to find the problems agreed oh yeah and some people have the touch they just hand them a product say this works great like yeah but when I push this this and this it dies like I need to make it back to my desk I mean I've got a feature this is not something that's that's hypothetical I have a really major feature it's being rolled out on our next release and I can't call it done until QA has had time to vet it and it's complicated and that means it's going to take lots of time three or four QA is taken offline for a week to test this feature and it's a big deal if it's a little wrong our community's going to be up in arms believe me and I know there's bugs I know it but I and I'm waiting what are they gonna be and and so far everything is coming back as almost working it's working what's not there's got to be something wrong people from tripwire I know what I'm talking about brings me to something that is like really important understand is every games gonna have bugs like no matter what you need to understand what bugs are unacceptable and what bugs might be acceptable like yeah in a sense they're like everybody's played this might be a bad example but everybody's played at Bethesda game I'm sure those games are yeah everybody not everybody laughs those games are synonymous with bugs they're still phenomenal games but they understand what bugs are acceptable to release with in their deadlines and they know how to prioritize you know what bugs aren't and that's one thing to definitely focus on in QA it's like if you find a bug that you know is gonna be a minor I you know always keep a note of it but if it's something that you know is you know not the worst thing just push it aside if you find something that clearly breaks the game you can't progress further and that whatever level you're playing like that's got to be fixed so like you always have to keep an understanding of what's the most important and sometimes it goes both ways but the way I'm gonna talk about it sometimes you'll see a bug that's kind of minor and you think how can they haven't fixed this and the program is so slow because I have to refactor an entire system to do that it'll take me offline for three days and it's the sword is green instead of forest green so we're gonna let that go you have to be willing to let that go as a QA person you have to not take it personally we found your bug we believe you we're not going to fix it does it mean it was a bad bug does it mean to you it means that we ran it through the pipeline and it would take more resources than we have that happens all the time and when you see a game that has bugs a lot of bugs what you want I want you to remember is that's the bugs they left in you should see the ones they fixed those are the ones they couldn't ship with and there were a lot of them I guarantee it these are the ones leftover that they were acceptable but there's an expression which is kill the engineers it's time to ship it kind of goes both ways - it's like you know a lot of times as a QA person you'll like B and there you'll be digging you refining you give the example you say this has happened this happened in the unit sorry with too many bugs then some of the designers and some of the people that are actually making the game the devs is they start getting a little upset they start that's as if as if you're sir trying to like fine problems just to tell them that they don't know their job you know cuz doors back and forth you know because they're artists they are there are creative people there and it's their baby I mean as a QA tester well I'll go back as a developer and a designer your job is to make the game awesome your job is and your job as a you know working for somebody else to just say what I did it's amazing it's beautiful it's perfect look at this because you know all the things to avoid in the back your head and as human beings we try to avoid things that are gonna cause us pain right the UAE's department job is to bring out the pain your job is to basically find those problems and say yeah you thought it worked but did you go over here to this far corner they're like why would I go to that corner the whole game will stop you're like right I know you need to go to that corner and figure it out it's it's a real thing and I've experienced as much as an engineer would say here are this good sports great no test and say no when you when you equip this and do this it doesn't work as well don't do that and I know that sounds dumb and I hear myself thinking it and I have to not say it because they're right I would never do it because I don't know what to know you know the game you can test your own code to some degree but you can't test it properly can't be done yeah and that's why at least from my perspective when I first started a tripwire interactive I started as QA I work at Cartoon Network for about six months I graduated from Georgia Tech I got an intern Cartoon Network and they gave me a lot of experiences with the workflow that developers go through and you learn different methodologies depending on the company but for the most part you always use systems like JIRA Zero to be able to maintain bugs and you'll be always in constant communication with other developers so it's always a really neat experience to be able to talk to people to be able to see what they're working on and to work with them to be able to fix their problems because winds up happening is their job is to develop to add features to code features but a lot of times they don't have a lot of time to test their own materials so that's why the QA team comes in and verifies every little thing about the so that when we have a list of issues they go through and they're able fix each one correctly that's kind of what QA is you're a detective you go through the entire game you find out who killed who what the crime scenes are like and then from there you write reports and basically report those and they get organized in this giant priority list and then each one gets fixed hopefully and then passed back to the QA to be able to regress them and I know for me personally when I first started QA was gonna be a stepping stone my passion was game design and it was always kind of a question of like will I be able to become a game designer if I'm working in QA because if you're really good at QA yeah they want to keep you in QA so you kind of go into your head like you always want to do the best you can at the work that you're into at the same time you want to make sure to cater your work towards what you like so just because you're doing QA doesn't mean that you just need to do one thing and be stuck with it you can explore different alternatives and solutions or I guess for example for me was game design we didn't have a lot of game documentation so - QA something and not have any clue how a system works is god-awful it's like I don't know what I'm doing with the system I'm just playing it so you don't know how it works and what I did was I ended up writing a lot of the documentation myself and with that documentation it helped the q18 so eventually after about three years I went from being a QA to a senior and that's when I started really getting into documentation into giving advice for designs and it's good to definitely not take it personally if your ideas aren't taken into consideration especially with a lot of people being in time crunches sometimes an idea can't be taken into consideration because of that but that's okay when they see that you're working into some subject that you're very into and you keep going on with it eventually you can get into a position and at the same time the position has to be available as well if the position is not available there's really the week for someone yeah you pretty much have to wait for someone to either get promoted or leave the company and be honest about it yeah I'm here for QA but I really want to be a game designer you know that's okay to say that's okay to be and then we know you're working yeah speaking of writing bugs over communicate something else I want to talk about when you write up a bug one of those there's several parts to and I'm not gonna go into it now but one most important part is the Reaper steps and it's how do you do I'm sure you all know what I'm talking go here do this make sure that is as stupid as possible you would believe you would not believe how many apparently intelligent smart brilliant programmers will go it didn't work recall more information and I'll tell you two reasons that one is it really didn't work the other is we're kind of punting like I'll do the steps I did this I did this I did this the bug didn't happen needs more info back over the wall that books gone for another day now we can work on something else if the steps work properly I can't do that oh god Danny's right and more than once I have gone to QA I've walked over there and said show me your steps don't work but I believe there's a I believe you I don't lying but they didn't work for me can you show me this be prepared to do that and by over communicating by saying exactly pedantically what you did it it stops the round-trips and that doesn't that keeps everyone's time from being wasted yeah I've always been told like don't don't make the developers feel like they're stupid when they're reading it but make sure to have your repro steps understandable for like a five-year-old like you need to make sure they're so easy to follow that literally anybody can do it and there's another one says something else especially when I'm in the debug phase of a proper project My Mind's everywhere and when I need to shift context to whatever this system is I don't know anything about it I'm not gonna load that in my head I'm just gonna bring up the bug and I'm gonna do that and unless I'm unless it's a system that I've just been working I have no idea how it works I figured I would take me 15 minutes I got 50 bucks to fix literally I have 20 or 30 bugs to get through today or tomorrow enough time to dig into each system and fully comprehend it I just want to do your steps and fix exactly what's wrong I move on so that's why you make them simple not because you're insulting your intelligence so not only do you have to like love to play games I only get to like one like hate games we also have to be able to be good at English be able to write clearly communication skills are super important to / important and you have to not take it personally because you are you're insulting everyone who's bug yousa not on purpose but that's the meta message the meta message or the troll right you are the troll you your code sucks because of this that's everything you do your feature sucks because of this so a good developer won't take the personally and I certainly don't and a good QA doesn't mean it personally but still that message gets across and sometimes you'll get designers and developers particularly artists who really start to get mad at you for finding their bugs in their system try just be aware it's going to happen let it roll off your back it's going to happen just commit and one thing I want to add kind of jumping-off point communication is so very key in this position not only are you going to be dealing with a lot of different people and different departments whether it's programming our UI stuff like that you also want to be able to communicate with your own team itself because there's gonna be a lot of times where you are thinking to yourself well I'm going to spend my time doing this or I'm assigned it has to do and you'll run into situations where somebody else is also doing a task and congratulations you just wasted a couple hours because you were doing the same thing you always want to be talking with your team making sure you know what everybody's doing any single time if you have any kind of issue you're not sure about something make sure to bring it up to someone immediately like you always want to be talking with somebody at almost any point in the day it's like you don't want to be goofing off or stuff like that but just being able to be clear and concise with your team just to helps avoid those kind of missteps and saves you time speaking of communication delete the word should from your vocabulary nothing should do anything I expect this to happen that's what you say that's what you're right don't say I grabbed the sword and it should have killed the troll no I grabbed the sword I expected it to kill it what actually happens is it didn't it's a very neutral non-confrontational way of saying things and it automatically should as an arguments being even argument it should have done this well no it shouldn't that's that's what the developers going to think I expected oh yeah well I guess that's a reasonable thing to have perspective and it didn't happen it's it's a way of defusing a situation immediately you just don't use the word should and and also whenever you're doing it make sure you set up your database you mentioned JIRA here is a great one there's mantas there's others are actually good someone's mentioned that their use Trello using Trello for it they've set up their system so it works well you can use it there's even excel files which please but but if that's what you got that's what you got but in general make sure that you do you communicate properly what the bug was the the route that it took to get there but then also the steps that it took to fix it and then it was fixed make sure you make sure you add columns that actually so you can verify that the fixes have them and implemented properly and they did work and then on the fifth time you play the game after you've made all the changes a couple of those may come back again because it may spawn something else so just be aware of that I guess it means they just make sure you have enough time keep the communication open make sure you track it properly but make sure you have time or just cuz your game plays in like ten hours let's say right allow yourself like sixty hours to good to get through it yeah you'd think having that history and that database is so important because you'd think when a bug comes back as fixed and it's fixed depending on the end the QA gets to know you when when when person a sends a bug in is fixed QA just oh yeah I probably check yep service and person becomes in yeah right okay we're going through and testing it again and again yeah what do you know isn't fixed kick it right back it becomes like a girl after back there he does exactly what I'm not kidding and having the history of that bug is incredibly valuable unless it becomes like troubleshooting because what's gonna happen is if you're working QA for a department for a live game that's being developed and you're keeping up weather as you go because if they finish this level or that level in some companies are so large that they'll be working on multiple levels at different times based on the doc and the design and what's happening so they may be this may be you're doing great then they're over here then they add something and they go back to go hmm wait a minute because we added this now we got to change this over here hallo testing yeah so basically you've got to be aware of that that it's gonna be constant change so don't expect it to start and play the same way every time because you may have started fine one day and then and then the next day you load in the build and you're like wait it's it's not what's not working and they go what's not working you say well it's not working like I can't it won't even start like what did you do I expected I have read that but game expected to start result they didn't believe me that's a very neutral way of telling someone they made a mistake and also sometimes you know making sure that you share it in the proper formats and when you do share was like a third party make sure that there's time for them to get it to load it into their pipeline to make sure that they can smoke test it make sure that it works within their system because a lot of times just going over the Internet's you end up with some issues sometimes time so I'm gonna ask a a pretty experienced question about what was your most positive experience with QA or working ask you a and can you also name like a negative or really bad experience you had with it and what's something that people should expect or be prepared for what you're doing QA my best experience was leaving QA know just being able to find a major bug and see that bug get fixed is really rewarding as much as it may sound like it's not a lot my very first day working QA I interned at Cartoon Network and I found a bug on a game that literally switched your account with somebody else literally across the world and that really upset the producer on the game as expected game had just launched and it's like hey if you do this this and this you literally switched your account with somebody in Australia and he's just like okay um so like he hated me for the first two weeks now we're really great friends but I guess to answer your question Jonathan that was both like one of the best and worst experiences my first couple weeks there because I found out like yeah I know what Q is I can figure out what this bug is I can repro at nobody else is able to do it but then at the same time I started and everybody hated me so that also something to understand is like you were gonna get angry they're gonna they're gonna get mad at you they don't they're not actually mad at you they're just mad at you no very positive experience I was working at CCP I was working on something and a feature hadn't been tested for a while because hadn't come up and a tester did tested and it was something seriously wrong with it and the problem was that it was an a' system no one had touched and that happens what must have happened is something someone did touch affected it in a way we didn't expect and the problem with this particular feature you know I wanna be too specific because it's not not that I can't be it just wouldn't interest you is that I had to find out when the problem happened I couldn't say what happened but it has been saved two weeks of check-ins one of those broken I need you to find the build exactly a check in which he had broken because it's you're gonna be asked to do this yeah this bloke builds broken when did that happen because it's not obvious I need you to tell me well it wasn't broken now and it was broken then which means one of these files broken and the QA came back to me next day and told me that it was this build and I said great I built it three and a half weeks old did you guys load every single check-in and they had and they had bracketed it and gotten in and like three developers and I didn't realize it all I was like wow that guy I really appreciated how much work and effort that that took and I told them so and I was a very positive experience and I that is incredibly value it saved so much of my time and it narrowed it down like three files ago God founded immediately so that was very positive experience and his opinion I'm sorry [Laughter] one of my most frustrating was whenever we were brought in a build and the company said yeah we need you to look over this creature run through it it's a huge massive MMO with giant RPG game like massive right and it's basically we need to do some sprint testing we had like two Sprint testers working on trying to get through the major parts of the game that we had like we're gonna take like for comprehensive testers to kind of like take out the whole world and check everything out me again massive game we were like okay great no problem then they came back to us and we're like checking in with them and they're like okay we're still gonna need you to do this like okay great fantastic then they then they came to us and said okay we're ready to go like fantastic ready to help you like what are you looking for the like well you know we ship next week but if you can you know just we need you to run through the game and see if there's any problems and and we're like you you're releasing next week and it was it was it was like what if we find one it was the beta right it was it was the beta but we had like so many people and they had like it that everybody was gonna be ready for it but the problem was we just said okay well we're not gonna make it because they're like what what do you mean we said well what's gonna happen is we're gonna play we only worked like eight hours a day five days a week new switch nine you know how to weekend like a day and a half like that that Saturday we're gonna get overrun like we're gonna overrun and they did they did they got hit overrun so bad that they end up having to shut down the game and shut it down and basically wait like three weeks before they reopened it again for the public because the the public started finding problems and because they started finding problems than other processor coming up and just being created within within the I guess the matrix of it all they would that that was the most frustrating thing that I remember I've ever had to deal with it's nice to have the time when you do to the first time I saw this it's like oh right you can't just say we're gonna spend 60 days or days testing what you say this room has been 15 days testing and 15 days fixing what we find and setting those two blocks of time aside is important even if that's what you mean when you say 30 days is to say no for the first 10 or 15 we're gonna find all the criticals and that's it and then we're going to decide we're gonna triage it and we're gonna then spend the time fixing and figuring it out and we're not gonna do any more testing except for the stuff we absolutely have to yeah regress it back progressive back and that'll that'll be handled by a competent management scheduling team but as QA you have to be aware of that that's actually what I'm on now I'm in production so I'm I'm in charge of making those schedules for those things and it's always important to leave that buffer time in there it's like yeah we we want to schedule two weeks of testing ok schedule for like that's two weeks we don't know yet chances are nine times out of ten you're gonna need it QA just never ends one positive experience I had and I always think back to it makes me feel a little bit smug thinking about it every time I reflect upon it started out pretty negative there was an issue that came up that was a pretty big blocker that happened kind of at the beginning of the game that I thought up all the repo steps everything I'm like oh this is a really obvious bug it's like let me get the same let me speak to the developers let me speak to the designers it's like this is what's going on this is what needs change and getting told hmm we're not gonna do that but we're gonna go ahead and try and do this alternate solution already kind of realizing it's like I don't think that's going to work but okay I mean you're the developers you're the designers it's like I'm just lowly QA that's fine a we can go by and that issue doesn't get resolved it's like yeah it's still happening I think this is a solution you need to go we're gonna try this other step now that we can go by they keep trying steps until eventually they come back it's like oh this issue was fake it's like yeah we just decided to end up removing it entirely it's like oh so like I originally stated we could have saved in three weeks to a month with it but all right and I always think back to that one yeah it goes back to just make sure you have enough time and take it seriously don't don't take it lightly okay that's what happens if you run up against the wall and you don't have time to fix it it just gets deleted and I've seen some pretty great features get just disabled because they were buggy in the no time fixing yeah I mean I would say that I mean one of the I think were the greatest things is basically digital download I think it's great for marketing but one of the worst has happened for actual like QA and being able to actually like get your game out to where it works is digital download because everybody thinks now they can they can do a patch for they can do will not fix it yeah and and honestly I've seen I've seen a couple of times that that's come back and really hurt the company especially if if they're running morning on a certain deadline where there's another game coming out that's very similar to it that's coming out like within the same week or two like they've been to saying hey we're games gonna be ready to go but then the bigger competition comes out and says yes but we have a game that's very similar to it you know where people dance a lot in it and it and basically either there's dancing characters that do flossing and things perfect but it but it comes out like right at the exact same time and one has been has been alpha tested for five years and one has been basically even working and working and they're they're ready to go and they released it they got here with tons of bugs the one that was out potestas five years came out and became one of the biggest games in the world possibly you know I mean but they got crushed they got crushed you know I mean that's you it can sometimes you won't have the opportunity because sometimes the the community gets so upset that gets upset as the developers when they start finding the bug oh yeah yeah and so you want more so yes you want to make sure you have time please give yourself time and make sure you track them and I can't only games to do awesome yeah one of my most positive experiences working in QA was when you find those really funny bugs that unfortunately whenever they get fix makes everyone sad and sometimes you do find gold like sometimes you might find like a floating teapot it's like okay well I'm just to put this bug in it's a minor or you find out that one of the weapons has friendly fire on and I you could play a game of tag with your co-workers for about until your manager is like did you write the bug up you're like I'll get it right so those are always really fun like we I have multiple stories like for example one of our weapons called the pulverizer so it has the ability to right-click and whatever you hit it explodes so during one of our plane test sessions I was playing with it and one of my co-workers was talking to me and he walked in front of me right as I right clicked and then he died and we found out that the explosion on it can hit players and it can kill them so for the next hour we were all running at each other with the weapon hitting us the play test we didn't really get a lot of great feedback just because it didn't really turn into let's test the pulverizer and see if it's fun well we found it was fun but not really from how it was intended so the designer was a little sad on that and then it was fixed the next day and it was unfortunate so it's not there anymore but yeah it's always whenever you find a really good bug yeah it's like we've had stuff like that though something is more and more more games are doing this I don't think it not putting more there is they'll involve QA like from the very beginning they give when there isn't a game when it's just a tech demo when they're just showing the features and they'll have to weigh in on it and they think they should and one of the big reasons is sometimes the bug to find become features and I know that I know that's a mean right it's not a bug it's a feature it really happens say this bug looks awful yeah but we ran around for two hours killing each other yeah okay put it in the game leave it that way we'll polish it a little bit get done that really happens I can't think of many games where it has it that's another like huge positive experience in QA is where my team at least at Adult Swim like we like to make people laugh so when we find a bug that's gold we will literally make a meme out of it and like tweet it out all the time it's like that helps break the monotony of just testing all the time if somebody finds something like everybody in our office is an artist they'll either make a meme or a short little video and just like send it around the office and then if they're really good like do something cool and then cool now you have your bug all over the internet that's something to look forward to in some places not necessarily everywhere yeah it's a fun little bloopers we do that a tripwire too where we house all of our bugs that we find during the game process our community manager does and like we have a folder full of at least a hundred videos from from the very beginning of its production where like you could Crouch a nun Crouch and start living tithing into the air until you're just gone - like when we introduced the new model into the game where you're able to like spin around actually the the feet did an anime as well as lower body so you can like twist into a pretzel so we had stuff like that going on so we have like a little home video that we make and those are the very fun aspects of qat it's basically where you find an exploit a bug and you kind of own it and you keep it and then once it gets fixed it's like okay what time to find a new one so that's that's where it gets very rewarding but uh just because we have about 20 minutes I'm gonna leave it open to anyone who wants to ask a question from the group so yeah yeah I'm gonna walk around so that we can get everyone [Music] okay so my name is Edward I'm from detective so this question is a chief organism because he's harder to take yeah just kidding if because you're passed from QA design is a path I manifested into you but uh you guys mentioned that QA is actually sort of an enemy of a developer right you point out their mistakes and they get angry at you at some point so how do you manage that relationship like how do you move from QA to - or game designers if you you know get angry at you we don't have sandbags in machine gun yeah they don't mess you give it to them I mean they have nerf guns sometimes you gotta be careful at that but ya know that's a really good point um so it's not that they're really getting angry it's just most of the time you want to have you want to direct it like whenever you're communicated with someone so when you do find a bug and it's on a developer their frustration is not on you their frustrations on the bug it's usually because if it's something that they're not really sure what's causing it and your repro steps don't really lead to a cause a lot of their time has spent investigating it themselves so most of the time you want to be there as their support you want to sure that you're there communicating with them to see if you can provide a help of any kind to them and that's kind of what led me into the designer role I found out we didn't have a lot of design documentation there was a lot of points where like I would find exploits with the designs that we've had and I'd work with our designer at the time to basically figure these out with her and we were able to come up with solutions together so it's kind of putting yourself out to proactively look for these kind of bugs and definitely having the ability to assess things calmly because it is true whenever it's like very busy a lot of people are trying to get their stuff in and we have a time schedule to me people are not going to always be in the best of moods and again it's not that they're directing it to you it's not personal at all it's just kind of the moment it's like if you have a homework assignment due tomorrow your friend probably won't be as excited to play video games at the time or they might be you yet okay you're holding up a target look at this bug I found all right and the developer angry the target but as soon as you say again now they're angry at you and you keep finding issues for that one person that target keeps getting bigger and bigger so just try to make sure that you keep it clinical and non personal I mean everybody wants the game to be the best you're working on the same team hi I'm Karen I'm a programmer dev designer I'm not specifically within QA but I guess you could say I kind of have like a very observability and something that I noticed as far as things that I observe to help me become better as a designer is sometimes people think that I hate things when I pick them apart like for example like even when I watch anime like I can sit and enjoy it and then someone's like Oh what do you think and I'm like yadda yadda yadda yadda the character blah blah blah days likes okay so you you hate it you hate it don't you no III love it but um I guess as far as it goes with how do you package your feedback that you've experienced with whatever that you're doing in a way that doesn't come off as say disliking it like you were saying using a expect as opposed to should so yeah packaging your opinions to other people that may not be as intense into it as other people may be yeah so one thing I've learned is when you're in QA to know that you're not expected to give design feedback most of the time like a lot of times they will ask you like hey what do you think like are you having fun because you will be playing that game more than most people but your main role is to give feedback on you know the bugs whether you can make it from point A to point B whether this specific item or this specific ability works as it should expected as expected yeah I'm sorry as it should as expected be like taking something apart critically and you know giving like a real analysis on that is not necessarily the job of QA I bet something you could talk to you with like a designer about it and get their feedback on it and like that's always something that's encouraged but that's not the expected role of QA yeah I guess a lot of times it can depend on the person I know for me personally I'm somebody that's always actually seeking out critique like if somebody just comes me and says that something seems fine they only have good things to say it's like that always makes me nervous because it's like I feel like you're holding something back to actually like not try to hurt my feelings and that's actually the other side of it but a lot of times I feel like you can kind of go with a a pro some people may have heard of it it's the compliment sandwich it's the own you kind of yeah you just always provide like something positive you found about something that negative kind of meat too it's like what you didn't so much or it's like what you found negative about and then finishing up with something positive usually it's pretty good with most people whenever you're watching anything in general you know or coming up with something keep in mind the budget how much time they had and what the intention was of like the filmmaker or the game designer or whoever they are and and see if they were able to reach what they were what was possible or capable to be able to get to within the confines of what they had so do whatever you're looking at anything just be aware of that and be aware of what the designers intended and there's a bear trap you can sometimes step into and that is when one designer thinks it should be a certain way he's the one in charge and everybody disagrees with him and then you make the mistake of talking that designer and telling him the same thing he's been hearing from everyone else that he's been that he's been disagreeing with that's a bear trap because now he hates you because you're against me too and you will you will run into that so just be aware that that it might not be you you just might be bringing up something that he's been bothering and if you do see something say like you're you're like I said when you're watching something on TV or something like this is great I really enjoyed it it's a month given what it was always capable and what restrictions they may or may not have and then you'll be able to figure out I say that was great for this you know but if it's not a triple-a game they don't have like like how many billion millions of dollars to make the game they only had like a few hundred thousand to make the game or ten thousand dollars or maybe they're making it in their basement and the game's amazing and it was blowing everyone's mind you know there's three different levels of things and what have different expectations so like scope focus complements yeah scope is extremely important in games like you like in my experience I've received a DD DD for a game that's like 150 pages long it's like wow this looks great how can we Whittle that down to 50 pages or how can we get that to fit our one-year timeline or a nine-month timeline with this budget so that another thing to understand when you're like giving feedback is like as they mentioned there's I can't talk specific time line that they have a specific budget that they have and you know they had to tweak their entire project to fit that right I mean Final Fantasy one is way different than Final Fantasy seven right no you can't look play Final Fantasy one and go oh this would be great if it had all these extra things in it no now you know because at the time it was mind-blowing right good question by the way I actually wanted to ask something similar to that I see a hand cuz language two really does matter a lot and sort of on the flip side of that how do you I worked in not a QA but customer service for a while and some things and you know you you you get bombarded with all the things that go wrong all the time and you get a population of bias of like does anything work you know what you're talking about yes broken for everybody knows we're for the ten people called you the ten thousand who didn't it's working fine for ya the ones that you hear you know it's a hundred percent of what you hear it's called selection bias yeah yeah so how do you sort of quell that in yourself like how many hours a day do you meditate you know to get rid of that what do you how do you cope you know this big bag of weed there's this great invention called the alcohol you have to you you learn very quickly in the games industry an industry is that there's a very vocal minority and everyone learns that really quick so you always take that with a grain of salt like this seems you've broken forever Michael really according to our metrics it's not or you try and find corroboration you have to find out what's really going on or you test yourself it's a very limited thing it's not really affecting it we have those conversations almost explicitly is this effecting too many people I know that sounds like it's crazy thing to say is it's not affecting that many okay we'll fix it but not right now or the other side is affecting a lot of people very badly well then we gotta go blind it could be a compatibility issue too sometimes yeah you know which which is hard to test on PCs I mean because there are so many different types of designs so many other things so sometimes a certain segment or a certain group or a region they they're not quite where they would like to be as far as their computer bills go and things like that their video cards so that causes some problems you know in certain regions so if you're getting tickets from a specific region try to find out like where may be and what their systems are out there running because it may be very maybe you find out their average pig is 350 so I mean that you know because you're in Russia yeah I mean that's part of you know basically it's charge customer services jobs you know which actually work with that sometimes where you know you have to be able to ask the right questions and be able to get to the center of it if it seems like it's happening in one area or one region you know besides just to go there are trolls so managing the scope of it in your head I guess really really hopes all right thank you okay cool so I'll come around here so specifically my question is about I know speedrunning is a very big thing and I'm speed writing game well I know that's where you find all the frames up to marry or run because speedrunners they'll find an exploit bugs to get a better game time and I know specifically there's the evil within - there was one where if he walked up with an axe - a bunch of rocks you could launch yourself across the map and skip huge parts of the game have you encountered any like footage or something about a game that you've worked on from a speedrunner are from a perspective of a casual player where they showed a bug that then got fixed because that was brought to your attention or anything like that I could think of one time that happened where we saw a video of someone demonstrating a feature and I saw him thought hey that shouldn't happen then I went to QA to confirm it and then they wrote up a book and we fix this I could think of that happening once but it's not common ya might be able to actually chime in on that because like recently a lot of times we look at YouTube videos of people who put up exploits so that's the fun part about the Internet is that when you find an exploit that makes the game a lot easier people post it and love talking about it so we do occasionally look up like we'll see a YouTube video of someone's like the sick ex boy that makes you attack 50 billion times faster of a melee weapon and then it gets fixed and then you know their documentation like they wrote out this whole exploit list like how to do it and then we fixed it so it kind of Nold it out and sometimes it's a hard decision because a lot of those exploits like you don't want them to be in your game if they make the game phenomenally easier but some of them aren't harmful they're just they make an aspect fun and it takes us to evaluate it as designers with QA to kind of consider what the what the ripple effects are so thankfully a lot of times we might find an issue that someone is exploiting but isn't like a major problem or you know they find an issue where it's like hey I'm able to get five billion dollars that just makes the game basically pointless and that's something we fix as we know that's huge and that's game-changing so I had a question about practicing or getting some experience for a keyway if or possibly like doing something they could put be put into the portfolio do you have any ideas on that in my personal experience I just made games myself in college I practiced with unity I tried to make a small game and then because I wanted to get into a as well when I was younger I would just start playing the game myself if I saw a problem with the collider I try to figure out what I did wrong there and try to fix it if I found a problem with like yeah I walked into a wall and I get launched through the other side of my level like I try to fix myself just to get myself an understanding of what's going on here and though in my like professional experience I didn't have to do any of the debugging myself but at least gave me an understanding what you know the programmers might actually go through and it definitely looked a lot better for me then like I mentioned before just being somebody that's like oh I love to play games you know what to look for right let you know how to exploit like especially different kinds of systems and how to communicate that to someone else who is too busy right so just building your own system of whatever kind and then also if you like Havanese like you to local like chapter of like a group a meetup group or anything like where you live a game designers people they were just like coding or trying to do anything or doing coding classes like a colleges or anything volunteer to like QA it well and that would be a good way you actually could get some hands-on experience on a lot of bugs probably more than likely yeah there are a lot of games too sorry real quick there are a lot of games too that have like early access and they'll have like discord groups so the developers will actually like asking for feedback and be like hey if you find anything let us know and that's a good way to practice as well because then you're also directly talking to the developer and then you can mention that you know in your resume or something like that and that is something that we look for and we actually you know higher than most people see a bug I submitted yet acted upon yeah yes try to be active about it in different games that you definitely so currently there's about five or six minutes so we might be able to cover maybe one or two more questions but definitely after the panel feel free to come to any of us if you have any other questions we'll be able to stick around and answer any of them so I could probably get one or two more questions thanks what about can you talk a little bit about creating the testing document what you need to do it from and what a good testing document looks like tell me what you don't think about testing doc what I don't think about them testing documents sound like a good idea and they are I've never really seen it work super well the best icing arguments I've seen have been very generic they've been here's a list of features and here's what they're supposed to do confirm that it works trying to come and I've been in a company tried to do that here's the features we want here's how to test them and it just it it never ends it just never meets the it never survives meeting the enemy it always change it and it's like well why didn't why do we bother writing that up when we ended up doing something clearly different anyway that's my view of it it's probably wrong so I'm gonna defer the people to actually manage this stuff um in my experience I don't know anybody here has anybody put in adults home game like over time okay so I was in charge of battle chef brigade most recently that came out last year I'm still in charge of that game but that was a game where I wrote a really extensive test plan on so for that I played that game many times I've probably beaten that game well over 500 times at this point but I played that for a good solid like three weeks and learned every single part of that game and my extensive I guess test plan for me it was probably about 30 pages for some games that might be short we had a game where we had to testify and that was closer to like 70 but to put as much detail into those and note things that you think should be looked at and things that you know the average player would definitely see and also put things that you know the average player would not do like you mentioned walking into a corner trying to kill Dora the Explorer like five times like that was you know be as detailed as possible and they at least in our group they worked for a good chunk of time and then after a while most people kind of know the game and they just kind of do this the game develops they become out of date really right yeah you have to constantly update those but really my test plan for battle chef Brigade started at like 22 pages and over time with DLC and we just locked some DLC last month or two months ago it's gone up to like 45 pages like it it keeps growing as you're doing more and you have to stay consistent with it awesome so it looks like the I think I know wraps up at 8 o'clock so I don't think would take oh we have two minutes okay so we've got time for one more question anyone up here real quick and then again after the panel we can still answer questions we should have like an awesome this a little bit earlier you touched on game tracking software well above tracking soccer has got it do you like have like a list of like software that can be used because I mean I do QA for one of the companies over here and I just been doing it the old-fashioned way JIRA bugs plant and bugzilla and mantis are the four that just jumping I prefer JIRA yeah we used Eric's constantly JIRA jir a sweet sounds yeah jura jura and bug tracker most people use mantas a lot of asian mantas also you can deploy yourself I think it's actually a freeware sure where JIRA what's nice about bug tracking software it's especially the good ones and they and they all do this now all the good ones they integrate directly to your email so when you file a bug it automatically emails the people who it needs to and when they do stuff it keeps people connected directly you can have a real conversation back and forth yeah through you're actually talking through JIRA we couldn't realize it and that's nice for history and triage someone comments on a bug instantly everyone gets that comment ok the comment is it's not my fault I'm sure there are tutorials all over yeah definitely on YouTube that's what I would advise there's a lot of video as I show just how it works one thing I want to add real quick as far as like the overall QA things getting involved everything there's a lot of different aspects to QA right now it's not just basically sit down at the computer and playing there's also a lot of ER QA where actually you have to be fairly active and very athletic actually to be able to keep playing through and through the game again like work on raw data when it came out that was actually pretty intensive yeah we shipped out clean floor incursion I was on that and we had to queue waste working pretty much continuously and they were just in the room and they had to be physically fit it's just they had to be yeah and there's all different kinds of games different platforms you made me QA on some instance like those apps and stuff like that I mean it's all there there's more to do it than just games there's a lot of aspects to it here to run to all the pentacle things accessibility Q is a big thing that is super important now and should have been important all the time but like you may be able to play a VR game no problem but like I can't so that's one thing to keep in mind as well what you're doing QA it's something that's hard to test like this worm game works great unless you're colorblind we have anyone who's colorblind nope well we're gonna have to fake it yeah so don that's on the one side on the other side you know that's not a huge part of most gaming games but they can be very vocal there's also QA all $0.16 also compliance testing as well for the different platforms yes participant as well as for linguistic testing as well if you're gonna submit a game on a console QA is your friend I mean you need them because you're gonna keep you you send it off to Microsoft for Xbox or ps4 Sony or something they're not going to tell you that afternoon the first few things I found now they're gonna send you a report in two weeks with a huge laundry list of everything they found well hopefully they find the three things and they stopped yeah sometimes if it's major enough the point is that you don't have the feedback that you need and that's why you need the QA Department constantly hammering on all the things that might be wrong because you do have the feedback right there well thank you guys very much and thank you all very much for attending this panel and if anyone has any questions I'm sure roll stick around so we'll be able to answer anything so thank you all very much for attending the panel [Applause]

Channel: Georgia Game Developers Association

Views: 1,816

Keywords: Association, Atlanta, Conference, Design, Dev, Development, Discussion, Entertainment, Expo, Game, Games, Georgia, GGDA, Growth, Industry, Interactive, Media, Networking, Presentation, SIEGE, SIEGECon, Southeast, Southern, Trade Show, Jonathan Betancourt, Brian Marquez, Tobe Sexton, Cody Ulrich, Tripwire, Killing Floor, Red Orchestra, Playtesting, QA, quality, assurance, gamescribes, Adult Swim

Id: XjiZNe8LW-U

Length: 60min 44sec (3644 seconds)

Published: Mon Jan 07 2019