AI Driven Test Automation: A Tour De Force with Tariq King

[Music] all right cool i will go ahead and make the announcement um i wanted carlos to do it because he's he's self-promoting his twitch channel so uh he is uh streaming again on twitch every weekday at noon that's mountain time and that's a free easy way to support qap there's also additional ways to support curate the point you can join our youtube channel subscribe to the youtube channel and then of course it's really easy to go to whatever social media outlet you use and like retweet re-quote post uh whatever else you can do we're on instagram twitter facebook linkedin and if there's something else that people are really interested in then let us know and we'll go ahead and add those as well all right and then expansion is coming and this is july 28th so just next week if you haven't already signed up hopefully you can still do so and a lot of great speakers again carlos i think you're self-promoting quite a bit here since you're not defending yourself i get to talk about you but here we go carlos is speaking at this event and for again for those who are local to utah um this is a great opportunity to meet up and uh get to hear from the qa community here in utah all right and then manual button pushers this is promoted by q8 point they are meeting august 12th as you can read here on the slide uh 7 p.m the topic will be accessibility tools and the presenters are russell johnson and amber ostendorf thank you manual button pushers for pushing this along uh doing a great job and the topics have been really good so thank you so promoting that and then um we want to thank again test ai and for those who are curious when we will do the drawing the drawing will take place after treats presentation maybe during the q a maybe after the q a we kind of like the suspense and so it'll be sometime in there but now you know when you definitely need to be here um of course listening to drew's presentation is the highlight of the night so we uh you definitely want to be here for all of that as well all right so what's up this is part of the exciting news at least in my opinion it is so we have a great lineup um [Music] coming soon we've got julia pottinger um she's going to you can see the topics here i'm not going to read through all the topics but joe colantonio and then i'm going to highlight carlos again carlos holy smokes you're everywhere tonight so in october carlos is going to speak uh but the exciting part besides carlos speaking is that we're actually going to meet in person we do not have the location decided yet but we're excited that we're finally going to meet in person again so we invite um everyone local and hey if you're not local and you want to fly in and join us for the meetup we invite you as well we are going to work on streaming it so it will be available for everyone um around the world just like this event but we're excited that we finally get to meet again in person cool all right so hello everyone i mean this is great uh thanks for for taking the time to join number one and then let me just say a special thank you to the organizers of qa at the point like i've always keep hearing about qa at the point a really cool name for a meetup number one and i even worked with kevin now but still for months went by and never got an invite so when he finally pinged me and was like hey um i want to have you on the point i was like this is this is great so um really happy to be here really pleased uh to have a big turnout like this as well so today i'm going to take you on uh what i'm going to call a tour de force of ai for software testing and the idea for this talk really came from something jason urban our cto and my buddy said one day he was talking about a keynote that he gave at eurostar and he was like you know that that was just like a quick tour of the force of our coolest tech at tesla and i must say honestly like the sentence itself sounded a bit strange right uh especially because the word quit makes you think of especially in the context of tor the whatever makes you think of the tour de france right uh which is very different from the tour de force uh it's probably a tour de force in itself uh but uh you know the tour of france is really a annual cycling event that predominantly takes place in france but also makes an occasional pass through other uh adjacent countries right and you know this was the route in 2021 um but authoritative force is a feat or display of strength skill and ingenuity right and so you know i thought about it a bit and what jason said and kind of how it resonated with me and i said well it might be cool to give a talk that puts these two things together and and not just for tesla ai but really the entire space of ai for software testing and so our agenda really is uh one that kind of combines a little tour of the ai for software testing space right and you know we got our little mascot riding around as part of the theme um but first i'll really get into a background on what is ai for software testing including different ai ml approaches that are commonly used for test automation and then we'll get into our tour which um like the aist kind of displays a feat is looked at from different perspectives from over the years right so we're going to look at it by testing level by quality attribute and then by problem domain and of course we won't be able to cover every problem but just a few that are pretty interesting and then we'll wrap up and take questions and hang out so all right so some quick background on terminology and stuff so that we're on the same page so when it comes to artificial intelligence that's this broad field of computer science where we're really trying to get software to learn and reason like humans uh should right intelligently um and you know that's that's a very big and broad field and it encompasses what is called machine learning which is a subset of ai that really deals with the programming model and so you know traditional programming models give computers step-by-step instructions right in this imperative manner do this and do that but with ml programming algorithms enable the machines to learn from data examples rather than these hard-coded instructions and so this is a major paradigm shift where where even instructions are becoming data and then last but not least deep learning is really a subset of machine learning that deals with large-scale computational learning and so when you hear this term you should be thinking about training neural networks that have many many layers and uh using relatively large data sets and we'll we'll get into neural networks a bit as we look at this all right so when it comes to machine learning itself there are three roughly broad categories of machine learning there's supervised learning unsupervised learning and reinforcement learning so under supervised learning humans are providing the machines with these labeled examples right so here's a type of a cat or here's different types of dogs and you're you know sending those labels to the machine and when the machine encounters an example that it hasn't seen before here's a new picture it tries to take a educated guess to classify it based on the previous examples that it has seen and so in essence what it boils down to is that you're really solving a classification problem after learning based on known patterns with unsupervised learning you're kind of letting the machine loose on label data right the machine then will partition that data based on certain features or characteristics and this is a process typically called clustering right it's the most common form of unsupervised learning you run into and whereas classification assigns a label right to your data like some aspect of the domain this is a flower um but clustering just takes place on that whole bucket you don't need to know the label and this type of learning is good for discovering new patterns in data in other words the machine teaches you something new that you didn't really recognize before about the data that you have and last but not least unsupervised sorry reinforcement learning is really about uh you know learning by trial and error right so you have an agent in an environment uh in some state and it takes some actions to transition to a new state and what you're using is some type of a reward system that helps the agent to learn the desired behavior to reach whatever goal you have right and so reinforcement learning in combination with deep learning is where you see a lot of the modern advances in ai being realized right so from things like playing gold to atari games to self-driving cars it's all a lot of reinforcement learning combined with deep learning or deep reinforcement learning all right so this idea of artificial intelligence for software testing or aist and this is really this field that has emerged where we're looking at three areas in particular right number one the development of ai systems that help to test software different methods for testing ai systems and ultimately we'll talk about this briefly at the end this future where the machines might actually be testing themselves however the tour de force is specifically focused right now on ai-driven test automation and we'll we'll see um maybe how over time this might expand into other areas because i think uh it'll become necessary and these three things as you see uh actually intercept all right so back in 2017 2018 i gave a keynote call rise of the machines can artificial intelligence terminate manual testing now this is actually like my very first keynote believe it or not uh folks always like tell me like this is the favorite keynote they ever seen from me and i was like that's great because that was my first one and i gave that at star west and then pnsqc and i would go on to give it all over the place like here i'm in romania at code camp in front of 2800 people right and you know the goal of the keynote was really to start to inspire the testing community to embrace the possibility that ai could really be a power up to traditional or conventional test automation techniques right in other words like traditional automation are those smaller part pellets i would say and you know ai is this big power pellet right so these smaller pellets don't really give pacman any kind of ability to do anything against the ghosts in which case these are the testing challenges uh whereas the power pellet actually can help them to overcome or eat the ghosts so to speak and so this talk is really just a follow-up check along the way to see sentence then where are we right and we'll get into some different dimensions on our tour de force right as i mentioned um the different levels but it's important to note that even though we'll look down these different dimensions they're really very intertwined and interconnected and so you'll probably be able to identify the interconnections as we go through all right so first let's look at how aisd is being applied at the ui level uh api level and the unit level right so we'll look at these so-called levels of testing and you know much of the industry is actually focused right now on applying ai for software testing on building the next generation of gui level automation and so there's companies and startups including us that are leveraging ai to do functional testing of web and mobile applications and so you know our platform which we internally referred to as carbon is like our bread and butter in our price possession that allows us to do this type of testing uh with ai and what i really like about tesla a in particular is that we haven't just stopped with a single suite or even enterprise um or mobile applications right we've gone into other areas and looked at the bigger picture so we're leveraging ai for testing entire app domains or the entire app store right our mission is to test the world's apps and of course when you get a mission statement it should be bold and there's nothing bolder than saying that we're basically going to test everything we're also applying it to testing modern video games and doing things like testing video quality and video streaming and so for us ai is really like our superpower right uh especially at this level the ui level you know we like to think that we can use ai to test anything with the screen right and you know if there's a superpower the next thing that you naturally might think about is that there may be something that lessens that superpower right some form of kryptonite so to speak and you know for me personally i believe that the kryptonite in this space uh is the complexity you know ai is kind of like magical it's but it's kind of the scary kind of magic right it's not the fun kind of family magic that you go watch um and so one of the goals that i usually have when i speak is to educate folks on how this technology works the different strengths and weaknesses uh and so hopefully we can start to build trust into uh the community for this software and bring value where it makes sense to apply and so you'll see that at each point in you know these stops on this tour de force we'll look at how it works underneath the hood as well all right so i threw this in here as a reminder that it's not just the flashy stuff like games and streaming services that makes ai for software testing powerful right uh we've been working with my buddy jeff marr over at dell trying to figure out how we can tackle some of the not so flashy automation challenges that uh you know large customers might face and so this is our product actually running and identifying elements on a system bios configuration screen and so challenges like this like i said aren't the prettiest thing but they have value to organizations that don't have an automated solution for this right i would never think that you know we could automate a bioscreen well except for very rudimentary trivial keyboard type stuff but actually verifying the screen and other things like that's pretty powerful all right so how does this work how does this ai for ui testing actually do its thing right the first thing is that we leverage computer vision to recognize screens and elements just like humans do right that's why it's about anything with a screen and along the way the bots will build a model of the application to help deal with uncertainty and help uh you know even with test generation which is the next important step right you've got to be able to generate test inputs and verify outcomes all using machine learning and then uh last but not least obviously the whole point here is that over time the bots will improve right you know they learn they get better you know as you train them the more labels you add or the more they can explore on their own they get better at testing the app and giving you feedback and using ml at this level really allows us to build more robust and resilient automation and so the bots are not tied to the dom you don't have to use like css selectors and which you've got to update whenever the application structure really changes but instead they're trained on thousands of images and examples so that when you update the look or feel of your application or your game it still works right and then the bots can use reinforcement learning that trial and error to relearn and rediscover newer updated paths through the application and so that's really the value point of using ai at this level and i couldn't really talk about this without mentioning um this guy diony santiago he's a personal friend and also director of engineering at tesla ai um he does that during the day and then at night he's this like superhero that does his masters and phd i don't know where he finds the time to do all this stuff right um but in his master's thesis his approach to generating uh ui level tests um for form based applications and he does that using things like boundary value analysis equivalence partitions trip script injection and he's incorporated all that into an open source prototype called agent that you can find out there for educational purposes but it gives you uh running code examples of how this stuff works so you don't have to take my word for how it works you can go and download that project and look at it just search for ai generation and exploration and tests on github and you'll find that and without getting into too much uh of the nitty gritty of his master's thesis what diani does here is basically combine ml based text generation techniques with this abstract testing language right and so this allows you to see some preconditions in and then have the machine keep generating new tests right and a lot of this technology you'll see up there is used for what it does to generate you know text right news articles or books and stuff like that so it's really just a translation of some of that work into the testing space and so for me this is definitely a tour de force um and i know the only recently completed this proposal i actually sit on his committee as well and and he's about to extend his work using hierarchical models so if you guys don't already follow him or have him on linkedin on twitter please feel free to do so uh i'm excited to see where he's gonna go with this all right not the api level right and this is also some very very cool work uh that's coming out of liberty mutual from gordon merritt um so gordon i ran into gordon because i saw him on linkedin talking about the parent and then i looked up the pattern and it's really a project that he called named seresha or sirisha and uh kudos to him on the patent and just even having the boldness to kind of reach out into the space and do something different because there's a real big gap right now in the space where a lot of folks are focused on ai for ui testing and there's not as much attention on the api level which we know is very very important especially when it comes to test stability and that sort of stuff right and so how this works at the high level again is that he uses test patterns and the past history of api runs to really teach the machines how to generate test inputs right how to determine the correctness of outputs uh which test to prioritize and other things right and so what he does is he leverages the api specification language probably something like swagger to help with the test generation and the learning process and you know as i mentioned this is a big big gap and uh you know even though this is an internal tool and there's nothing that's out there for everyone to see i think uh just having the idea pushed out to the community is a big value out so yeah major respect to gordon for that another approach on the um automated api testing uh front which you'll see out there is this idea of recording service traffic right from manual testing and then using that information to train some ml based test generation system and this approach stems from your popular record and playback that you're probably all very familiar with in the early days of functional automation tools and so you'll see this pattern in other places like performance testing um the thing that is disadvantageous about this is just that you really need to have the ui available to be able to kind of even start with this right and then what i've seen is that there's a bunch of vendor tools that are touting this approach and they let you do the recording and they generate you know the the apa test but they're missing the learning right they're kind of more translating those things into api tests than actually generating new tests and so on and so forth like things like the ani's approach but nonetheless i think that there's promise here to to do more work in investigating this and it's good to see at least some traction in the industry around this type of approach all right at the unit level uh if you as you see as we start going down the levels we start increasing the gaps that we see in the industry around this stuff right um but here uh practitioners are trying to train ai to automatically write unit tests in high-level programming languages such as java right and there's two popular features around this type of framework for unit testing uh the first one is just automatic source analysis to generate unit tests right that reflect program behavior right and so it's not really connected to your oracle right so uh warning flag should come up there uh what i liked about gordon's approach for example from the api level is at least it took some external uh specification of behavior from swagger right so here is definitely needing something like that um the other kind of feature that you'll see with these tools is the integration with uh version control systems to monitor source code changes and keep your tests up to date right to kind of help deal with test maintenance uh and so this is a interactive widget that diff blue has on their website uh for their product this blue cover right there's um uh interesting uh online widget that you can just go and use and see kind of how it works right um but that's again very interesting to to see some traction in that um that space right so there's a firm out of the uk i actually met their ceo and a quest for quality conference a few years ago so check it out if you are interested there and how this kind of works again without getting too technical or too deep um much of the work in this area builds on advances in ai for generating text and natural language right so initiatives such as open ai is gpt3 which combines natural language processing with deep learning models to generate tests that you probably couldn't tell if a human wrote it or you know it was written by a machine in this case right and so here you'll see that there's quite a bit of work this is on the right you'll see um copilot and github release the copilot um product which really is generating full functions right and this are you know kind of replacing the programmers right so if we can replace the programmers we certainly can replace the test automation code uh maybe right we'll see all right so now looking at another dimension looking down the uh quality attribute row and so you know we'll start with performance and uh of course there's different levels of performance like i mentioned our whole thing uh desktop ai is about scale right and so uh we are looking to test the rose apps not just one app but all the apps and part of that is addressing the application domain um as you go through an application and you're testing it you can obviously at the ui level you can track steps and gather performance metrics and that's what we can do with our product and not only can you now start to see how your app performs on certain key scenarios but you can compare its performance to similar applications from your competitors right so for example um you can generate reports that compare the results of any retail app to all other applications in that domain right and the way that this works is that the tests are running uh the bots are running through these goal-based tests on these key scenarios and applying that same test across the board within a given category and then they compute a percentile rank that puts your app you know against all the others in that category and that's really powerful test information to have uh if you're a leader in your space like you know why is our performance in this particular function so much worse than one of our competitors that drives change and that's viable information the other level of that is just not looking at the domain but trying to test the whole store and test everything in it and you know uh again works in a similar way where the bots are crawling through the apps right and looking at key user flows and using computer vision to make decisions uh there's an action pool so now you're a little bit more exploratory and autonomous at times where you're clicking swiping entering text you're doing things like maximizing windows you're looking for all these strange things that could happen and the impressive thing here is really just the scale that ai allows us you know to test so we talk about a thousand apps per night and every month doing runs of 5000 apps at a time like just like if you want to talk about a tour de force or a feat of strength like that level of testing uh even at a rudimentary level where you're just kind of looking uh and comparing metrics is impressive right and there's tons of benefits and and returns on that investment uh if you are the owner of the app store and executive you want to know the health of the overall store because you're accountable to that uh you also have vendors people who are submitting their apps to you and you know identifying issues like that helps with the vendor relations and helps you to to get those things um and get them to get their stuff fixed right so you're providing them valuable information and then if you're a software engineer a software tester we can do things like show you the the path that you know most of these bugs and issues occurred on and that's good for debugging and fixing and improving quality which is what testing is all about right all right another um aspect is a lot of the work that we have done on testing data centers right and uh you know this is really about uh doing some top of the rack upgrade testing and the context here is game streaming right so as i said these these dimensions uh interconnect right and so how this works is that you know you have a very large scale where you have a 165 parallel accounts and sustain load running for 10 hour period you let the bots lose at scale and so we execute 55 sorry 15 500 tests in that space of time everything is fully automated and there's secure communication between the bots and the cloud environment and that's just crazy to to imagine that you can take uh what would normally take a human uh 21.52 man days right uh you know or what is 1291 hours or 75 500 minutes right all of that time squished into 10 hours for for bots to do it right and um you know this is i think one of the whole points of of not only leveraging uh cloud compute to do testing but combining that with ai to be able to do it across the board and you know just kind of knock things off the park and obviously there's no point to doing uh testing if you're not gonna find bugs and issues and so you'll see um our system picking up you know game streaming bugs right fail to start stream or you know the stream connection issue is lost you get black screens after you load a game and that sort of stuff and all can be detected just using computer vision right and one of the key features that i love uh especially the way that the world has been going with uh you know really uh testing as a feature and embedding testing and feedback into their apps is that our bots go through and find these issues which is again impressive in itself but then they actually go back and submit or on the flight also can submit those bugs uh straight into the system to go straight into engineering um so that's that's again pretty direct value um uh feedback for these stakeholders so there's still gaps in ai for performance in particular on the end-to-end space right so you know application benchmarking and even doing data centers testing is not the same as doing end-to-end performance testing of your application where system level performance testing really should emulate production traffic using hundreds and thousands of concurrent users uh under real world scenarios and and transactions right and you know again i think when you start to look at our community and you see like little pockets of engineering happening and innovation happening uh it's pretty exciting so uh cultural dolly over at the ultimate kronos group has been building a tool for end-to-end performance right and the vision there is to it's pretty ambitious right is to have full self-service system that automatically produces load runner scripts complete with parameterization correlation and logic and you know the internal tool emulates the uh basically emulates what the performance engineer would have to do so you eliminate this need for manually rescripting and all that stuff right and so you know i think the long-term goal and the long-term vision there is to uh really use ml to drive smart rules and smart parameterization and stuff like that and smart correlation so uh also if you're not tracking culture please check him out he gave a presentation at star east last year uh which should still be recorded somewhere so check it out all right last uh dimension here is to look at uh usability actually i think there's there's two because there's usability and accessibility that we'll look at um but uh one of the things that we always hear is so hard or even impossible to test is usability right you could test all sorts of stuff two plus two is four but you can't check if there's a good look and feel well ai is kind of making uh that not true anymore and giving us some power to really test what are these um uh good designs versus bad designs right so each mobile development um platform has some sort of guidelines like whether you're not talking about apple there's these human interface guidelines and for android you have uh their design for android guidelines and you know a great way to catch these issues is with ai you train the blocks to examine the screen right you feed them examples of good designs and bad designs or areas where these design guidelines have been violated and you don't have to look at code or any application specific checks instead only visual elements right and you know we can find these in a repeatable manner and alert folks and this is a major hurdle especially for the app stores who try to run these things manually and it's a huge effort that goes on to do that all right on accessibility uh keith briggs also over at the ultimate chronos group uh in 2019 uh published a paper around site-wide accessibility testing right um a11y or ally i like to call it ally testing and and this is pretty cool because the whole point here is that uh you're starting to combine uh the power of the bots to explore right so your bots can explore and build models and this is a this is a short video of ally in action where it uses bots to to build a an application graph with all the pages and then it can easily plug in tools like axe right which if you guys have ever used axe or lighthouse you know the pain of these tools is that the developer the tester has to go and fire up their app to a specific page and then run the analysis and then keep doing that so having an automatic way to explore and then add on these behaviors with um accessibility testing tools is pretty legit right we saw that there's going to be some talks on accessibility tools that should be pretty cool i think one good thing to note here is that um you know there's still a lot of room in this space for automation of accessibility i mean a lot of the tools don't cover a lot of the higher level automation uh test cases that need to be run if you're truly gonna have something meaningful for accessibility but being able to start here is just a a good step for uh you know an industry who's that is trying very much to keep pace with what is a very important area of testing all right so now let's look down at the problem domain and we're really going to take a look at two very specialized domains uh which you probably know by now because we talked about them uh first one being video quality testing or video streaming and you know the thing is that how we do these things today is really really really painful right uh so you know how this is really done is it's 100 manual the best practice is really to literally stream the video and ask a bunch of people to watch it and rate it right using this mean opinion score uh scale and it's just a real nice fancy scientific way to collect these scores but clearly this is slow it's expensive and it doesn't scale right and when it comes to automation there is some right you have um a lot of frame by frame pixel by pixel automation being applied here where you comb through every frame and look for different failure modes like the blank screens poor frame rates etc um but this this is also very painful right it's almost like you know find find the video bug in the haystack here right um and so there's just a need for something better and this is where ai can really thrive right and so um what this is all about is first training the bots on those common failure modes right and so they want to use those failure modes as features in our model along with the category of video because not all video genres are created equally like live action versus slow scenes versus animations and that sort of stuff and then the third major feature is to have the bots actually watch this video and and focus on the areas that humans would right because a bug in a video stream may or may not be noticeable or you may or may not care about it depending on if you even notice it on that part of the screen right and so again just like humans would uh we train classifiers so we collect these features we trained um basically on a set of videos with associated mean opinion scores what you're really trying to do is train the bot to mimic the human judgment and again this is where ai really thrives and you can see here after several iterations uh in a particular genre i think this is our asian fusion genre you can clearly see uh from this curve that at this point there's a 93 percent correlation between what the bots are predicting and what the actual human scores were and at that point 93 accuracy no one's going to complain it's pretty accurate and actually when i look at the curve sometimes i tend to favor the predicted score curve over the sharp painful kind of weird things that kind of tend to go on in the other one and so that's pretty pretty awesome to see uh that kind of stuff happen and you know this opens up i mean that kind of testing really opens up the uh the space for uh just doing better and different types of automation um but there's still one other little piece that needs to be taken care of and we've also done some work with that where we use raspberry pi's to allow us to just plug into any kind of hdmi interface or we use it to interface with the game controllers and that sort of stuff and then add a driven automation so that we can now automate things like roku and you know the amazon fire sticks and those those kind of things right which we've never really seen that kind of stuff before and then you'll see that we also do it on you know playstation and xbox and that sort of stuff and that's just really cool in terms of a super power right to have all right so last topic here is gaming i think i'm good on time uh but ai has been playing games for as long as we can remember right um but it was in 2017 where we had uh open ai creating this bot that went out and and defeated the world's best dota player right and uh in a small match right a 1v1 tournament but really they bought learn from scratch right using reinforcement learning and self-play and uh and then uh two years later they'd create open ai five which was really using reinforcement learning at an even larger scale to train the bots to play together as a team of fights right and so we were talking about having bots learned by playing over 10 000 years of games against itself and then reaching an expert level right human ai corporation and when i think about that right then it makes me think that extending the bots while they're playing games with classifiers for detecting cosmetic bugs and gameplay bugs all of a sudden doesn't seem so hard right like it doesn't seem to compare to the gravity of beating an expert at one of these games or beating a team of the experts at these games and so as you can imagine traditional approaches to game testing are pretty manual right and a manual effort but again not fun it's not smiles enjoy the game be happy kind of thing it's more repetitive like i need a different life kind of uh automation right uh or sorry testing and uh when it comes to automation there's not much help there either right there are at the lower levels a lot of frameworks that help you with the game logic but when it comes to doing end-to-end testing either it goes back to the whole manual piece or uh there's automation that's either very brittle very you know prone to breakage because it's very kind of hard code you know move move north move right you know that kind of stuff uh or it does the pixel comparison which we know is very uh brittle as well um but you know uh most of the bugs right uh will happen not in this vanilla environment right that's only a dev environment for a plane uh it'll happen for that end user experience right and that's where they always revert back to the manual work um but game testing is a huge huge area right very important especially now the gaming market is booming due to the pandemic and stuff like that but there's several dimensions to this right so there's the core functionality of the gaming application itself and making sure that that works there's the game stores which is a source of revenue for all these major game companies right so that's important um there's the visual look and feel and how it sounds right all of those things which again traditional automation um doesn't necessarily tackle and then there's gameplay right and again if you think about automation frameworks from an entwine standpoint you know your seleniums aren't going to help you with this right or your opiums right um so ai is almost like the only alternative that you have right um again there's there's some other ways that you can do it but it's a very powerful way to step into this world of game testing and so you know again anything with a screen so it doesn't matter it might matter if it's a game and it doesn't matter if that game goes into game play like you know you can compose tests that use elements and then actually write a flow that goes straight into the gameplay and then interact or verify the game elements all using computer vision and machine learning and you know it gets deeper right because you want to be able to train the ais to detect different objects right to classify different things as we saw from the the foundational slide so here you can detect the character right in this case maverick and you can train a classifier for him you can also train a classifier for the accessories that he might put on like in this case the adventure pack and then now that you can traverse through the application and select different things from the menus you can now do a test that runs through selects different uh configurations of the visual assets and then verify that it is what you think it is at the end of the day right and so that is pretty pretty powerful in terms of automation those are the kind of tests that people are running manually and finding glitches um but of course it doesn't stop there like you got to be able to identify if you do find a glitch where the glitch is right and so again machine learning can help you with that we train models to really remove the background uh and be able to deal with the acid itself and compare a baseline image that we trained on with the current observation to see if there's any changes and if there's any changes you can create a difference mask that really highlights exactly where the change was right so here this is a typical you know change where you know obviously just for illustration purposes it's like okay it's obvious what chains here uh but if there was just a little pixel here a little pixel there it didn't matter the death mask would allow you to see where that is and that's where you know having a pixel by pixel comparison can be useful but not for the base level of automation only for showing me really when i want to find out exactly where something is wrong or what it looks like so i can go back and show this to the animator and say hey we we changed something and it broke something in our um in our system and what we do with this in the production setting is actually lay this div mask over your original asset and fade it in and out so that you can actually overlay it on the original asset to see exactly what the difference is so that's pretty cool and um you know similar to the visual diffing problem you can also now look at comparing audio for similarity right and so again without getting into all the technical details and you know you can use ml models to train this or you don't you know it's still within python there's a lot of different ways to look at the cross-correlation between things to look at the similarity of things right and so uh it's all available there as an extension and again it's not just about using a ml to do everything but if it's available and you can extend it and mix and match these things i think the result the end result can be very very powerful right so here this is just again base observation that you have right some base image and you you you'll see what that looks like your test observation and what you do with the cross correlation is that you're you're really seeing if this graph uh really fits over or sweeps through this other uh graph right here as well so pretty cool all right and you know the big thing is really about gameplay and we're seeing that you know there's just more and more demand for uh really validating uh things in game because there's a lot of manual testing happening around games right and so um you know this again might look like magic but we can drill into exactly how it works uh on one side there's real time object detection so by now you should be familiar with the fact that you can just train things on images right so we train here the bots to recognize enemies or friendlies or generic objects or the active weapon right are also things like the the hp right we can do things like this this bar here for example for the for the health is you know somewhat depleted right and we can calculate by looking at the images and classifying we cannot write a calculator that estimates how much hp we have just visually looking at it and of course the object detection is just one thing because you're going to leverage that to validate that things look and feel how you assume they would but you also need to train the bots with uh goal based reinforcement learning so you model the actions that the bots can do in this case go forward back aim strafe left and right there's a mobile there's mobile call of duty here so the shoot happens automatically uh in the default setting but then you give it rewards right so we we look at the hud and we can pull out using ocr what the kill account is and assign a reward to the block for increasing the kill com and we can also do things like decrease uh give a negative reward for decreasing health right for taking on hits and things like aiming at enemies right so here visually once you lock on to an enemy the cross hair becomes red right and so we can detect when that becomes red and use that in a reward system to say okay i want you to do more of this i want you to change that cross here from white to red by targeting on enemies and learning that and again pretty cool because what you're saying is that all that research that was applied to uh gaming and that is being applied to gaming still to play games can now be combined with object detection and now things like visual diffing to put a comprehensive solution together for testing gameplay using ai all right so to round it up you know there's just this rapid growth i have shown this light over the years and almost every talk that i give uh once some time goes by and i've just seen it grow and grow and grow right and so the rapid growth in the use of ai for software testing just continues and there's vendors there's uh you know i focus here on industry related things right internal tools there's still a lot of academic research and a big body of academic research out there as well that needs to be transitioned into industry um but it's exciting right and for me uh i'm just like at all in the fact that we're we're seeing um basically ai driven testing turn into this kind of full stock solution where we have ai for the core testing concerns at the different levels and now we're seeing a being used to address cross-cutting concerns like usability performance and security and that sort of stuff and i'm i'm just really happy to to be a part of it and work for a company that's a part of doing these things i think um without folks kind of working together on these problems we're not going to make much traction at all and it'll take some you know trial and error and failures and all that stuff and um it's the exciting thing about being at a startup company i think for me um but then i'm also thinking as a quality practitioner doing this stuff is becoming in my mind a necessary step because we are building ai like we're building it like they're building it even if we're not testing it they're building it right and i think that um the key thing that i've learned over the years from all of my research is that when you build something that can learn or adapt at runtime right you know it's no longer just um you know test it beforehand and release it like while it's out there it's going to be learning and it's going to be changing right and you need a different strategy for testing and um that's a big passion of mine to look at testing ai systems and even that the future might involve embedding some of this ai driven test automation technology into the runtime of the system itself to be able to have it you know test itself as it learns online uh but for now we need to be thinking about how to test these systems both offline and online until all this automation and full stack automation uh catches up and so with that i wrap it up and um you know open any questions you can always connect with me on social media kevin knows how to find me all the time i try to hide from him but he works it out and so you can open up kevin yeah okay cool thank you you

Channel: QA at the Point | Carlos Kidman

Views: 783

Keywords: qa, quality assurance, software testing, automation, qa automation, selenium, testing, testing framework, carlos kidman, qa on point, api testing, katalon studio, robot framework, devops

Id: NnitgX0kjo8

Length: 50min 16sec (3016 seconds)

Published: Mon Aug 02 2021