How AI is transforming software testing - Raj Subramanian | SeleniumConf Chicago

how's everyone doing today great Wow I know you're lying because it's just lunch everyone should be sleepy right now but anyhow so it's at least good to know you're doing well a couple of things before it gets started first thing is I want to make sure my t-shirt and everything is good because this is going on YouTube live so good okay yeah that's good yes secondly I have a really thick Texas accent so in case you're not able to understand anything I'm saying yeah please feel free to shout out and then I'll repeat what I'm trying to say man and also after just having a talk after having it back after Jason gives attack yeah that's the expectation really high right so yeah hopefully you get something out of the stock so a couple of questions I also wanted to ask before I get started so first thing how many of you think so how many of you are confused with terminologies with AI neural networks what machine learning is and how would a place of software testing great great okay that's good to know okay second question how many of you think in another three years AI powered robots are going to take over the world and we're gonna be slaves to them okay yeah see yeah let's go so I have some good news and some bad news good news is I have the answer to the first question I am going to talk about how AI is influencing software testing and you're in the right top bad news is I don't have the answer to the second question but the next Terminator movie is definitely going to answer that as well so so with that being said let's get into my talk today so a little bit about myself again my name is Raj and I'm a developer evangelist at this to my o a couple of sentences about my company before I tell what I do for the company so testing my own use of artificial intelligence for the authoring execution and maintain about a minute test our focus is on functional testing end-to-end testing and UI testing right and we are based out of San Francisco so that's my company so what I do for the company is I'm a developer evangelist yeah it's a super awesome way of describing my job but basically I'm jack of all trades master of none so I do 30% I do conference speaking and I do a lot of writing which I'm really passionate about another 30 40 % I work with R&D to make our tool better I work directly with customers to see how they're using our tool and make sure you know put in features to hug them out and the last 30% I guess I know you're following whether I'm doing the math right but anyways I don't care but the point is the last 30% does the I do support customers as well for the entire US region anything related to technology comes to me and I work with the founder of my company pretty much to help them out and make the tool better so that's a long description about what I do yeah I've been in the development and testing industry for over 12 to 13 years now and you can see that I share my ideas on testing regularly on my blog and other channels as well so you can look at the links and you'll figure out stuff so this one I'm gonna cover today right so first I'm gonna start with the basics of AI because everyone keeps getting confused with the terminologies and I want to explain any AI from my perspective because when I got into a I feel everything was really confusing right so I wanted to dumb it down to really simple things for make me understand what it is and hopefully what I'm gonna say also gives you a clarity and the terminologies then I'm going to talk about what are the current challenges in test automation how AI can help to solve these challenges and finally talk about the future of artificial intelligence so first let's set some context right even before talking about I can do this eight and do that it's really an important to understand how test automation in general as evolved over a period of time in the past few decades so in the 1980s and 1990s right so testing was predominantly manual testing and yeah we didn't have any tools or automation and the development methodology was also basically waterfall methodology where developers used to work for six months and finally build a product and then it'll come to the QA testers hands and then they'll find bugs and then they'll send it back and then you have to fix it and then send it back then retest it so there's a whole thing right there was so much time basis so that was 1980s 1990s era then we had 1990s in the 2003 period right so we started seeing finally some tools coming out to help the testers and the on the bright side testers that man now we have automation tools to help us out but on the other side of it it was a problem because how many of you have had situations where you were using automation tools and the whole laptop froze pretty much you can do anything right I don't cure DP eyes but yeah so those kind of tools so the point is they were really bulky automation tools right and then at the same time people started experimenting with different approaches like rad that rapid application development XP scrum and so on right then we have 2003 and 2010 timeframe where we saw more tools coming up and then it was mainly the era of open source frameworks rate we know some open source frameworks right because we are at a selenium conference so it's like people like in selenium their liver group of like-minded people who wanted it to wanted to solve different problems with test automation they started involving other people as well and then build a community right so open source was a big thing and at the same time we had agile approaches like our XP scrum Kanban became a normal thing everyone started following and the name of the game was faster release cycles how fast he can released to the customer and then 2010 to current rate so it's more about scale so what does it mean by that we need we need we need to run a lot of tests in parallel multiple times we want to find bugs faster we want to release faster and also during this era we had cloud testing you have sauce labs and other folks who said you know what don't worry about the hardware and software let me figure it out you just do your testing and then we had crowd testing where there was this group of people who gave their software to the testers and get feedback from them either paid or not paid version rate so those were the two major things which happened and then we keep hearing about DevOps continuous testing see ICD which is kind of normal right now from 2010 to the current state now we're going to talk about the future right the future is going to be about autonomous testing machine learning and it's all I'll it's from my point of view it's going to be about dynamic locators which I'm going to talk about in detail in a bit and it's all about smarter testing so that's the name of the game and that's my point of view going to be the future so I just wanted to give that context before we actually talk about AI software testing test automation so first thing I want to start with is a bit some basics of AI my goal is by the time you leave the session at least you'll have some idea on what different terminologies mean and then you can have a conversation about it so in the on a really high level you have artificial intelligence right so what is a ai ai is the area of computer science that deals with giving the machines ability giving building machines which can think and work like human beings right so that this artificial intelligence under artificial intelligence you have machine learning which is a subset of artificial intelligence and it's a field of study that deals with giving the computers the computers the ability to learn with being explicitly programmed so that is what machine learning is it evolved from the study of computational theory and pattern recognition computational theory who has pretty much study of learning algorithms so that's a machine learning evolved right so that is machine learning and then under that you have what is called deep learning which everyone know talks about so deep learning is just one of the areas of machine learning yeah Bayesian networks and in depth of logical programming and so many other subsets of machine machine learning but deep learning just one of them right so what is deep learning so deep learning is basically a model based on the neurons of the human body so in our human body we have neurons each neuron starts learning from another urine starts reacting and giving reacting and we act according to all these interactions in a human human body in the neurons the neural network right and they just took that to the field of computer science and that's why it's called a neural network because it meant it's mimicking the human body neural network rate so deep learning neural networks so you know they use it synonymously so that is what is deep learning and just to give you an example of what people do to train AI or machines to learn rate you have heard about all these things quite a bit but I just wanted to dumb it down to really high-level right so say you wanna you move to Chicago and you're trying to find a place in Chicago right and then you build a neural network for it so the way you may go about doing that would be say give the size and the price of the house and then you want to see what happens right so the size and price the input the common combination you give is color training data set so a training data set that's what they keep calling it so say you give the size and price of the house right and then you think about you know what let me give the number of bedrooms I want right because that could also and we're and live in Chicago and after that probably after giving the number of bedrooms probably the AI would figure out family says that something is really important for you you know what so let me look at areas based on family size next you can give zip code and the price right and then it could figure out maybe how walkable the area is right so that is something it may start building that relationship and then you would give wealth and the price right it probably but could figure out what's the school quality school quality is something important for you so let me try to give you a data base on that right so whatever happens here and here right out in the outside so those are the training data sets we give but what are these relationships the family says the walkable walkability school quality all these strengths these things are done by the area in the black box right it's like a black box so this is basically how they act like in general how they train the AI rate so I just wanted to give you a example and then when you talk about deep learning they usually talk about three things right under deep learning so one is supervised learning so what supervised learning super is learning is you give different training data and then make the Machine learn and then monitor it and see how it's doing right so a lot of examples for supervised learning for example how many of you have this experience where you're using M Hulu or Netflix and then once you log in then it gives you recommended movies or TV shows right so those are oh that's because you have an AI an algorithm which actually does that based on your viewing pattern rate so that's an example for supervised learning and then giving us giving audio and getting texts from it so that's an example then giving a bunch of emails and then seeing which is spam right so those are some of the things are good example supervised learning and of course you have autonomous cars as well that's a great example for supervised learning as well right then you have unsupervised learning so that is giving a bunch of data and then see what they can come up with rate and some examples could be cellular companies like AT&T and Verizon rate they want to target different states based on age group like they want to give targeted ads for 16 year old people to targeted ads for 88 year old people rate so market segmentation is somewhere something where unsupervised learning is being used and then you can give a image and create a 3d model effort as well so that doesn't those are some examples unsupervised learning then finally you have reinforced learning that is basically the concert concept of rewarding good and bad behavior such like training a dog you'll give something and then you then you get some output and say bad dog bad input right and then next time you give another set and then you say good input good good right so you just keep training and the kind of example reinforce learning I got I'm trying to simplify all these concepts so those are pretty much the things you want to know about AI and now in terms of autonomous cars for example for the past three four years they became really famous but the first research for example for autonomous cars where it was done in 1989 at Stanford right we just don't know about it right so I know looking at your faces you're skeptical and you don't believe me so that's why I have a video to show you that autonomous cars this video was released making it is just here I'm watching a person drive Alvin is designed to control the navlab to a modified army Humvee equipped with sensors computers and actuators or miss navigation experiments the initial step in configuring Alvin is training a network just year during training a person drives the vehicle while Alvin watches once every two seconds Alban digitizes a video image with the road ahead and records the person steering direction and again this was in 1992 so this is their experimental car Alvin and they're training it great and you'll see more cool stuff happening 32 pixels and provided as input to Alvin's three layered network using the back propagation learning algorithm Alvin is trained to output the same steering direction as the human driver for that image initially the network steering response is random after about two minutes of training the network learns to accurately imitate this oh sorry there's my bed Alvin is a system of artificial neural networks that learners dispute most confident Network in this case so what's happening is they're feeding it images and training it and then at one point at this point what they need ominous and I love obedient driving they've put an autonomous car mode or whatever it's got driving based on the data per second how the digitizes an image and feeds it to its neural networks each Network running in parallel produces a steering direction and the measure of its confidence set this a confidence level in terms of how confident it is it can perform in an action so this is really important because they're going to make sense and actually related your testing there's a few interruption from the most confident network in this case the network training for the one lane road is used to control the vehicle get to some interesting part so it's coming to a junction right and now see this confidence level decrease because it does not not as the vehicle approaches the intersection the confidence of the one line or decreases as it crosses the intersection and the two-lane road ahead comes into view the confidence of the two-lane network rises again when it's confidence Rises the template Network is selected to steer safely guiding the vehicle into his lane on the two-lane road there you go and the car drives and everyone lives happily ever after but the point is so this autonomous cars and all those AI stuff which are hearing the research was done while ago it's been happening read me we just now come to know about all the stuff right so I just want to show you stuff and just one more last thing I wanted to show you as a lot of people when I'm speaking at conferences ask me hey can you show visually me training in AI you know I want to get that experience right so I was thinking about different ways to do that I found out a really good way to show you how we're training in AI would look like so we have one second hold on so we have an open source website called quick drop by Google where millions of people draw different diagrams based on what the AI suggest and then it yet training the AI by drawing different things where it will start predicting that's a really good example of training in air so I just want to show you for example how would work so this how it works note draw a bandage in under 20 seconds so let's see I'm really bad at drawing but I see lime or ocean or canoe I see skateboard or shoe or spoon or crocodile I see hot dog now that I see River I see submarine or belt or flying saucer can I see a helicopter or toothbrush so sorry I couldn't I fail in that so let's let's try another one so let's write panda okay think I should be able to get panda let's see I see Circle or blueberry I see frying pan or shoe or teapot I see snowmen or alarm clock I'm not sure what that is I see koala or bear sorry I couldn't guess oh man I need to get one okay skyscraper eight thing I should add he said she get skyscraper come on man let's I see you lime or room oh I know it's a skyscraper see yeah yeah okay let's just say one more a crown I see fork or arrow oh I know it's crown right okay let's just do one more I know it's addictive right okay it's a neck I see paperclip or garden hose or roller coaster I see a blueberry or ring or lollipop or not are smashing satellite or yoga or hula hoop or flamingo I see Sun or short monster or fan or tree I see you but I couldn't get it but anyways so so this is a live example of actually training in AI and I just want to show you how it actually looks like so yeah one thing two things we learned from this exercises now we get got a visual image of how it will look like and second thing is I'm really bad at drawing now I admit it in front of everyone right so so those were basically the things related basics of AI want to cover now let's get into the real deal great with test automation and the challenges and how a I can help to overcome this challenges so recently we did a survey in terms of how fast teams release software the highlights of the survey was 57% a majority of the people said they started testing really early but just kept testing testing testing and then they missed deadlines and they couldn't release at all right how many of you have been in that situation rate most of us rate and that and the same thing happened the survey as well and only 2% of people really deployed every minute of every day right so why is this so once we started analyzing why this problem was we figured out there were five set of challenges which people face currently with this animation right and those are the things we really need to think about so let's first talk about skill set right so getting skilled testers is expensive and hard to get right and usually many companies suffer not have suffer with not having skilled testers and who can do automation as well so that affects release cycles rate so that skill set is a major problem which are affecting release cycles then you have authoring so now we have hired screw testers and then they start authoring tests but they fail to pay attention to good practices of automation such as reusability how to do parametrizations or do data-driven interesting giving optimum optimum weight rate reusable weights so those kind of things people forget and that in turn also affect releases because they don't release until our automation tests pass and then bit because you have pranams the authoring you never seem to release on time next challenge is the initialization States right so what do you mean by that see you're automating a webpage saying the Amazon shopping cart right you write an item awesome automation code to actually add items to a cart so you write your automation code and then and it adds an item to the cart right you're super happy then you run it again what happens right you forgot to initialize like bring it back to the initial state of zero right because every time you gotta run it it's gonna start one two three four keep increasing rate those are common problems which people fail to realize like setup and their teardown activities right so those are things which are affecting the test automation as well so next one is maintenance right the biggest problem with this automation and it's been and it's been the same case for the past 20 years is the maintenance rate so say you have 10 tests you wrote and then you're super happy at all passes make sure you come to work and then everything fails right because the state of the application exchange rate now and then you have to spend a lot of time maintaining it and based on a recent survey 30 percent of our testers time goes into just maintaining automated tests that's crazy rate imagine I'm on a stuff we could do and that even if you shave up some percentage of thing you spend on maintenance right so maintenance been a really big challenge and finally scale I know I talked about scale in terms of resources like cloud computing and cloud testing services but another thing in terms of scale is your automation suite rate how scalable it is is it could you add more functionalities to it can you extend the functionalities and can you run more tests in parallel faster right so those are some of the things you want to think about so these are the five main challenges which exist currently in test automation in so that's why we probably need AI for some of these and then I'm going to tell you about how AI to certain extent can overcome these challenges especially the maintenance piece which is the biggest challenge at especially at estimates our goal that's why we're trying to solve right so that Segway segwaying into the next section so now I discuss a challenger so let's talk about how a I can solve these challenges right so a test oh my oh where I work V is a concept called dynamic location strategy right and from my point of view I think that's going to be the next big thing and future of test automation so what do I mean by that say you have a log and button on a web page and you change the name of login to summit and for some reason say you're locating that button by name the login name and if a person changed the name from login to submit and you run the automation code what happens that just breaks exactly great and and as testers what we do is ok man the test broke ok now let's see what happened oh ok the name change developers the name change please change name oh no I'm on vacation for two weeks I can't do this right and then you go to another developer there's so much wastage of time and the maintenance piece rate it's been a really big problem because you're using one single location strategy for every element for example the login button using the ID or the name right and and the dynamic location strategy which that's why we are using AI for it is wow it's getting multiple attributes for every single element you interact with right so what happens in real time is we have this rai underneath the hood which extracts all the object trees and properties of the element after Dom and then creates a list of location strategies right and for the same example of logon name changing to summit if we are using dynamic location strategy with the AI it won't break because what's gonna happen is it's gonna say home the name change you know what I'm not gonna fail the test because the functionality hasn't changed but yes they are to be address changed but let me go to the next best location strategy it'll go to the ID of the idea change and go to the next special location strategy it could be class name the next place location strategy tag name parent-child relationship so it will go through the entire location strategy list which is it has extracted in real-time rate data skull that this is a concept and it's called dynamic location strategy that is getting multiple attributes so that's how you can actually in help and maintaining and creating more stable tests and more tests you run the AI keeps seeing what now is happening and the tests become more stable because it finds different ways to optimize your tests right so just to show you the concept of dynamic locators I just wanted to show you a demo and different people implement dynamic location strategy in different ways but I'm just showing you what we are trying to do so say so I have a test and just let's quickly record at this right I just created a sample webpage just for demo purposes I'm gonna type in name password and then select dates from say a date picker and I'm gonna go ahead and book this itinerary and then enter a name email ID and yeah I think I should that should be good for now so I just quickly recorded a test and then yeah let's just make sure it and mess up anything like that most always are successful right so don't worry always also I have a recording of this so in case this bombs I have a video for it but let's give this a shot first so just going through the steps I just recorded selecting dates from a date picker selecting a button clicking on book cool seems like the yeah this will pass okay cool so test completed successfully so remember the dynamic locators which I was talking about so just how it works underneath the hood so say for example take the select destination button and here you can see how the attributes which have been extracted for that particular select destination button and displayed over here so you have the text add to be a tag and to be a Class A to B and so on and remember I showed that at Animus car video yet that that I was just showing up with videos but there was a purpose because if you see over here you had the confidence level so it says a hundred percent which means that when I ran the test with a hundred percent confidence it said I can look at that select destination button and then for each attribute you also have a scoring system so for example say this text attribute write it that's a score of 0.9 which means that that's the best way it tried it's going to locate that element save the text attribute change then you go to the next best location strategy based on the score it could either be the class or type attribute maybe because both have the same score but again all those additions are made by the AI underneath hood we don't have to worry about it right and to show you how dynamic locators the beauty of dynamic locators and how they can make your test stable so in real time I'm gonna do a change and let's see what happens right so I'll just put a breakpoint quickly and again I'm this mainly to show you the concept of dynamic locators and this how we implemented so what was happening is just running through the test until the point I click on the select destination button because they put a breakpoint to do cool so it reached a breakpoint right so what I'm going to do is let's do an inspect man the screen is so small here and I'm pretty much blind I can let let's say we move this button right so I've moved a button the select destination button over here right you're seeing it and you know what to go and step further let me say selenium con Chicago right ok so if this was any other automation framework which we have right now so what's going to happen if I continue running that this it's going to break because the location has changed and the name has changed right but now because the beauty of dynamic locator so this let's see what happens to to do to to if I did everything right if I play there you go so the test tool continues to run and yeah so test will pass so the this is the beauty of dynamic location strategy because the act had to be a change but the functionary still remains the same so why do you want to fail the test the thing you could do is show that hey there's a change so for example the way we do it is if you look at the screenshots for example it says that hey I expected the Select destination button but what I found was selling him conference Chicago right so it tells you that hey there's a change but the functional didn't change so it still kept continuing to run and if you go back to the locator locaters again so if you notice now it's 83% confidence because I actually changed an attribute rate similar to training that car and also if you see the text attribute over here okay it's really for people in the back you can see it but if you see that the X attribute there's a color coding there showing that that attribute has changed so what what have happened in real time is it went to the next location strategy which could either be the type or the class again the AI does it for you right so this cell dynamic locators work and a lot of people are currently working on it and this R we implemented but I showed you the demo mainly to explain the concept e-rate so that's going to be the next big thing this diagram is from a person I know a person named Jason urban so I really loved this graph because aster test coverage is growing linearly right what's happening is the complexity of the software has been increasing exponentially and there's this coverage gap right and that's what we're trained to focus on with AI base testing so that is something to keep in mind now let's talk about the future of AI right so some of the things I'm showing you are things which we are currently working on and things which are possible because of AI right and this is going to give you an idea about how a is been going to be used in software test automation first thing is autonomous testing so you no longer have to assume how the user is going to use your apps in the production in production right so what we could do is we have this AI actually looking at what are you doing in live production and then based on that it's going to start finding out patterns in terms of common things people are doing say for example I can come back to the login button scenario say you have this login page and this you logged in 50 times and logged out 50 times the AI based on this production looking at the production the behavior it would say you know what you logged in 50 times and logged out 50 times why don't I create a reusable component for it right so it's gonna create a reusable component and automatically inject that into your test right so it's going to start authoring tests based on your production data right so you'd no longer have to assume things but you're gonna have already written tests based on user behavior so next is GUI TDD you have you have heard of BDD TDD a DD s DD or whatever all the DDS you have just to make things more complicated very injecting one more thing called UI based test-driven development so what I mean by that so yeah I worked in industry for 14 years now yeah based on my experience too so usually development works after that works where you get design mocks and then based on the design mocks the developer will know what feature to build and he starts building the feature rate based on the marks that's how I'm used to so now with UI based TDD what you could do is when you have the mock ups right so the developer can look at the mock ups awesome when he's starting to build a feature you can use the AI and recognize all these images and based on the images you can start creating tests right so why it's eventually going to happen is by the time the developer has completed building the feature you already have tests written and then you can run the tests on the built feature and see what happens right so that is UI based test-driven traveller development and that is something which is coming up and something to keep in mind next is automatic responses right so when you're doing UI based testing some quite often people have this problem so how many of you had this problem where the same test when you run at something the page takes priest I can still load sometimes take 15 seconds to load when doing you it exactly right so the point is because of that what happens is then you have to start looking at what optimum weight statements to give what's happening and all that stuff but now with a I why do you do is the air is going to recognize that dude for if the past 50 times you tried to login with the same credentials at testing that I oh right and every time the response goes to a server and comes back and because of that the page is load late and because of that your tests are failing so what I'm going to do for you is I'm gonna record all this responses within myself right and then whenever you want to log and I'm just going to relay it so you don't even have to go to the server for the request so you can just record all the server responses and then give it to the user UI test in that way UI testing is much faster and much more stable and you don't have to worry about these timing issues as well right so that is something you're actually currently working on as well and something is going to come up so how desam is paving the way for the future right so it's easier to for QA snow to max my user coverage maximize use a coverage rate so the point is you already creating user scenarios based on production data so you already have a lot of user coverage rate now QA is no like more it's just more than just Q is like a science right now because you're making all the decisions based on based on data and this based approach to testing rate so that's something which you can achieve with with use of AI and you're preventing bugs as opposed to fixing them so what I mean by that so with AI you have the self-healing mechanism so what the means is as and when you you run the test the AI sees what amount of things you're running and becomes smarter and smarter and starts optimizing different things especially wait times right how many people have to do put X inclusive eights with except for example selenium rate rate and or any other automation framework rate but now the a I can start optimizing all those weights and you don't even have to do that anymore right when if you with the help of a a you can overcome that issue as well so those are some things you can expect with a I've and you use it for software testing so yeah that's pretty much what I wanted to cover today my goal was to give you an idea from my point of view what AIA means and what we were working on what he can expect as future in the future and I've given a summary of my talk written a post about it in this link and you can find all the resources and extra things which I mentioned and yeah you can look at it in your own time so yeah that's pretty much what I had thanks for coming thank you you

Channel: Selenium Conference

Views: 22,033

Rating: 4.5955057 out of 5

Keywords: seconf, seleniumconf, seconf 2018, seleniumconf 2018, selenium 2018, selenium chicago, testing conference, Raj Subramanian, AI software testing

Id: pMd1L1IZrxk

Length: 40min 54sec (2454 seconds)

Published: Fri Nov 16 2018