iOS Automated Testing Strategies Q&A (Unit, Integration, UI...) | iOS Lead Essentials Podcast #013

okay we're live hello everyone welcome to another edition of the IRS lead essentials podcast I'm Mike and I'm Caillou and this week we're gonna be replying to some questions about automated testing strategies in iOS so we're not going to cover every testing strategy but the most common ones exactly start him with what is unit testing or isolated or micro testing yes the industry cannot agree on a term for a unit so what is a unit right it depends who you ask but a central idea behind unit testing it should test a single component at a time in isolation that's why you may call it isolated tests or micro tests because they are testing just a tiny portion of your code base at a time exactly and you do so without communicating with other real components we're talking about isolated tests isolated components with mocked or stubbed behaviors so no network connections no network requests no database requests nothing no flakiness no flake exactly no flakiness we're talking about deterministic behaviors that they will yield every single time the same result green hopefully right so unit tests should be reliable every time you run the test you should get the same result regardless of the system state because you don't keep state in unit tests exactly so you should be able to run those tests several times a day they are fast and reliable that's why you should be able to do so because they are extremely fast and reliable well they should be fast they should be yes yes they're very cheap or again they should be very cheap to to write and to maintain exactly and to maintain that's why in the testing pyramid of allocating tests in your system the unit tests are the foundation the layers the base of the pyramid that's it unit tests are fast reliable and cheap to create and maintain does they are the foundation for a testing strategy or they should be yes and they are written by developers yes you write it in the same language that you're writing your application exactly there is no business language or any QA type of tool that can be used we're talking about the programming language that you have learned in your system in because you are going to reference your system in your unit tests implementation is gonna be referenced in the test code and since you're not talking to real back-end servers you don't even need to have a back-end ready to write them as long as you establish a contract with the backend you can start writing those tests at any time and that's that's exactly right and that's very useful because you're not blocked you're never blocked yes for example in iOS when you're just testing a class in isolation without talking to any other system that's a unit test yes for example if you're testing a view model you invoke a method passing some input and you assert the value get back the output mm-hmm in isolation with no views no controllers nothing just the view model you can prove that if your model does exactly what it's supposed to without having to instantiate the whole object graph or even run the application you don't even need a simulator exactly so that's unit testing or isolated testing or micro testing which should be the foundation of your testing strategies but should not be the only testing strategy as so next question what is integration testing integration tests check the interaction of two components as a group two or more two or more components yes exactly ideally you don't wanna stash a to the whole application and test the integration of everything at this level that's why you'll normally limit C to three components collaborating yes but there is no limit ideally just a few components a small portion of your application collaborating that's it exactly and integration tests are the second layer of the testing pyramid above unit tests so that means we allocate fewer tests to check components in integration and of course that's because checking more than one component is going to be more expensive that means that the tests are a bit harder to write and they can take a lot more time to execute right and in your development process that means that you don't want to be blocked by the test running all day long you want fast and reliable tests and integration tests well sometimes might not be fast and might not be that reliable exactly especially if you are testing the integration with core data for example mmm-hmm that's gonna actually persist data in disk or you are testing the integration with an external system through some kind of HTTP connection right so that can be slow and flaky because if the system is unresponsive the test is gonna fail exactly well if you're gonna hit the network if you're gonna hit the database there are many many things that you do not control so if you don't control them then the result is not gonna be 100% deterministic something gonna be 100% exactly the same every single time exactly and there are some behaviors you cannot easily replica in integration tests for example it's very hard to replicate a database error or a network error because it's just and reliable yes just like you cannot reliably reproduce a successful connection exactly because it's out of our control yes so if you need this level of control to test edge cases or error cases should probably use unit tests for that and it tests more of the happy path in integration to make sure that the components collaborate correctly because maybe just a unit test is not enough but the ratio of number of unit tests per integration test it's completely different should have a lot of unit tests in a few integration tests that's why it's the second level of the pyramid yes it represents a small small portion of your tests so integration tests are important but they should not be the foundation of your testing strategy or the only testing strategy as well they should go on top of unit tests you should be 100% confident in your unit tests anyway the integration tests to make sure the collaboration also works and again you don't need to test the whole integration the whole application working you can instantiate just a small portion of your objects of your components and test them in integration exactly I mean an example of that would be the network request welcoming the response mapping it and propagating it to the rest of the system monitoring how many components are involved or how many actions or operations rather are involved in such operation right you can even go one step further then and check integration as well the database let's say you ask for the data from the network you get it and then you save it you persist it in a database and you want to make sure that all the the round-trip of a data actually works so there's a lot of objects integration there yes there is the HTTP client there is the matter the models there is the persistence there is some kind of coordination in between those objects maybe some business rules validating the data adding some expiration date to the cache and etc so maybe you can even break this down and say I'm going to test the network connection in integration just the mapping and the HTTP client and in another test I'm gonna just test the persistence if I can prove that they both work on their own these also proves that they work in integration so don't need to compose all those fragile operations together it's a choice exactly I really like what you're saying because it must mean if you have to check all the error cases all the possible error cases in integration right yes the tests would take so long and you have so many error cases that you will need to check that it's just gonna be like instantly it's gonna become a bottleneck if you're after a 100% code coverage there so yeah I like that the integration tests you want a few of them above the unit tests foundation of your testing strategies yeah just covering the things you cannot covert with unit tests if you can cover a hundred percent a feature with unit tests that might be enough you might not need integration tests yes I agree but if there's any kind of complicated collaboration between objects might be valuable to use integration tests just to cover the integration part and in integration tests you can also replace some of the objects in the integration with some kind of test double you don't need to use the home integration has real implementations right for example if you want to test a network failure you might have to replace the network client with some kind of test double so you can control that failure you can force a failure right so can replace just one instance in that integration just to control some edge cases that's fine as well so that's integration testing next what is UI testing okay we have a the definition by Apple here in Apple States UI testing gives you the ability to find and interact with the UI of your app in order to validate the properties and state of the UI elements UI testing includes UI recording which gives you the ability to generate code that exercises your apps UI the same way you do in which you can expand the bond to implement UI tests right so Apple gives us a framework for performing the UI testing which allows us to record us interacting with the application and then repeating that automating that process so we don't need to do it manually every time yes so your why testing is a kind of black box testing it tested the UI you don't have control over the application you are actually initiating the whole object graph running an application and interacting with it so you have much less control over replacing behavior or controlling the state of the app yes you need to manually perform the actions to get to the state you want that's exactly right and the keywords that stood out to me of what you just said is you have to run the app hmm and that makes it very very expensive that's why UI tests are at the top of the pyramid of the testing pyramid because you want very few of them because they take a lot of time to run because you need to run by actually an application and press buttons and wait for animations and etc exactly of course you can speed up that but if you compare a unit test in a UI test a unit test takes 0.01 second a UI test is going to take at least 5 to 10 seconds it's yeah it's like two orders of magnitude more unit tests play at the millisecond level you artists play at the level so if you have a bunch of them then it can easily easily become a both link and when it does well you're gonna stop testing hold together yeah yes of course there are strategies to make you a test faster and more reliable for example replacing the network stack with some kind of stub or the database or passing some large arguments to the application so the application can put itself in a state you expect but still it's a costly strategy right even the frameworks used to interact with a user interface they're not very reliable they crash for many reasons so you don't be running your tests for hours and hours and hours it can become a bottleneck in your process but you want to feel them if you have a very complex user interface or interactions they're hard to test in isolation or in integration you must have to run the app and actually test that yes so you might need it not always but you might need it and if you do who write UI tests well developers and QA testers exactly like unit tests and integration tests developers can write them but you don't need to be a very experienced Swift developer to write those tests with a little bit of experience with Swift and Xcode and exit tests QA engineers can also write those tests and if you have QA engineers in your team you'll be very beneficial if they wrote it yeah and when should you write those tests well ideally you should write those tests with the QA engineers in the beginning so ideally sure also write the UI test first exactly and it's a it's a live process basically you can write the UI test first and of course the system is not ready so you jump into another testing strategy gradually start implementing your components by unit testing them first and then maybe if you need you can have a couple integration tests making sure your components work together and at the end your tour we'll be passing if everything works correctly in integration that's it so your white testing is a testing strategy is going to validate your application through the user interface like a black box you have little control even though we can pass some large arguments and do some trickery with the application it's still a black box and it can only evaluate values you see on the screen can only perform operations with the user interface with UI elements invalidate what you see on the screen like labels buttons etc and they are slow and expensive does we want only a few of them if any exactly next question what is acceptance testing acceptance testing is the process of validating that that your app is compliant to the business requirement or the acceptance criteria right so who should write acceptance tests right hopefully the business folks can write the acceptance tests right so business folks meaning a business analyst there has direct contact with the business with the decisions exactly exactly these are high-level tests that do not touch the implementation of the system they check everything in integration for specific scenarios specific operations the system must do and since they are not written by developers they are written in a high-level language like English exactly that's that's correct and we have some guidelines for that for example they're given when then from BDD yes for writing these acceptance tests so donate to get developer to write those tests down in a high-level language but then a developer or a QA engineer is gonna translate the high-level language to some tests the camera run against your system exactly so there's this high-level language that everyone can understand in the business and it needs to be true related to something that can execute your system to validate the acceptance criteria and there are tools for that right like Fitness and cucumber exactly the excellence tests also live in the top part of the pyramid because they are expensive they are very expensive to run to execute taking a lot of time because they're checking basically the whole system whole features of the system in integration and usually as a black box as well yes so usually you don't want to mock anything you want to use test doubles for anything gonna actually perform operations in your system in some kind of staging environment that replicates the production environment so it can be slow because you're gonna be running network requests you're going to be interacting with the file system so the results can be under termina sztyc yes it depends on external systems working as well so you want just a few of them just to validate the acceptance criteria remember the foundation is unit tests on top of that go integration tests and then you start writing more high-level tests like UI tests in acceptance test but you need this solid foundation because if you have to test everything with high-level tests that's gonna be extremely expensive yes it's gonna take a long time to run the test is not gonna be reliable thus you're gonna have to test things manually it's gonna be cannibal tonight exactly so you need to build a solid foundation first so those can be important tests but acceptance testing should not be your only strategy next question what is end-to-end testing well we covered already I believe and 20 sting is a form of integration testing that checks an operation from start to finish in the example we gave before about making an actual network request getting the response mapping it propagating into the rest of the system then you save you persist that response in a database well that's an end to end test where you check what happens to the data from the network to the database so talking to multiple systems in integration yes so it is an integration test specifically going into end without any mocking yes so they are also written by developers yes probably in this same language you're writing the application that's exactly right because you're gonna reference the implementations of your up inside the tests that's it so they are also expensive to run stay going to end talking to your network database it can be slow and fragile they can fail for many reasons out of control so you want just a few and to end tests we have critical systems they need to operate correctly and you don't trust those systems for example if you're talking to your backhand you don't trust or if you're using a framework for persisting data that is like a black box they don't have access to the code does you want to make sure that it does what you expect yes so that's it end-to-end test is a kind of integration tests you can also say that UI test is also a kind of integration test because there are multiple components integration in acceptance test is also a kind of integration test next what is black box testing ok black box testing refers to the process of checking the system's validity without having access to its design or its implementation right so it's literary test into the binary somehow yes like in the case of UI tests that's exactly running the application and tap on buttons exactly you don't know how the system is being designed or written in the case of UI tests in the case of acceptance tests that's exactly the same in the case of an 20s or integration tests or unit tests that's not the case because you would have to reference the implementation code in your tests right so you can say that UI tests is a kind of black box testing yes an acceptance testing is a kind of black box testing as well if your testing application not through the implementation to some kind of high-level framework then it's a black box just like a user would test your application they don't know what kind of pattern you are using they don't know anything about your application no you can only interact with what they see in what is typical yes that's exactly right that's it can think of it as the automated version of manual testing where yes you basically don't know anything about the guts of the application of the design of the structure but you can interact through the UI there yes so black box testing doesn't need to be implemented by developers no if you have a high level tool like UI tests where you can record the interaction with a user interface anyone can record those tests so that's it every time you test some kind of private source framework where they don't have access to the implementation or you're using some kind of tool there's been a test application from a high-level to the user interface that's black box testing in there leaving the top of the pyramid as well yeah together review eye tests what only a few of them because I'm not very reliable and they interact with video systems so it can be slow again you need a solid foundation with unit tests in a few low-level integration tests then you build on top of that if necessary yes next what is snapshot testing so snapshot testing refers to the process of taking a snapshot of your system or parts of your system and comparing it to a previous state for example you can take a snapshot of some data structure it was the output of an operation then you can run the operation again take a snapshot of the output and compare it this is usually used for validating a user interface the rendering of a view for example because you can render a view take a screen shot which is a kind of snapshot and store it so as you refactor your code as you add more features you can keep taking snapshots or screenshots of that view and comparing to a previous version is they are the same actually checking the pixels if they are the same the visuals are is still the same yes the deef check checking the deef of the two pictures there yes we should say here that snapshot testing is not like UI testing in many regards actually first of all snapshot tests take a lot less time to run the new artists and that's because you can render your UI without actually running the application while in UI test you can't do that you need to run the application yes so you can't take screenshots of your application during your eye tests but you don't need UI tests to take screenshots of your application that's exactly right yes you can render just a portion of your screen you don't even need to render the whole screen yes for example you can render just a cell take a snapshot or a screenshot of debts and store that state in your git repo so next time you run the test it's going to take a new screenshot and compared to the old version yes as long as they are the same your UI is rendering the exact same zeal as you expect and you don't need to run the application to do that now and this can be very fast can only render the view a small portion of the view and compared to a previous version that can be very quick that's a good thing exactly so snapshot tests are often written by developers but since you can also take screenshots from UI tests QA engineers often write those tests as well when they are a bunch of tools that can help you take screenshots of your application store them and compare with previous versions exactly again snapshot test is not just about user interface you can snapshot anything in your application any data structure any result of operation and compared to your previous version the problem was using only snapshot tests as your strategy is that if the tests fail for example two versions of a data structure don't match you have no idea why it could be anything the feedback can be not good exactly so that's why you also want few some sort as you perhaps in certain areas of your application and your system you don't want to rely on them although there they can be fast you don't want to rely on them yet they are faster than UI tests yes but there's too much slower than simple unit tests exactly so you can add it to your strategy but not as the only strategy and never as the foundation yes they can go on top of unit tests yes but they usually test more than one component in integration so there are a kind of integration test does they go above unit tests in the pyramid you want just a few of them all right next question which testing strategy is best which one should I choose so we have many testing strategies but doesn't mean if I choose one I'm saying no to all the other ones that's exactly right you need to create your testing strategy by combining all those different approaches so if you have to say which testing strategy is best it depends on your application but you need a solid foundation the foundation should be fast and reliable so unit tests on top of that you can test the integration of your components with integration tests they can test integration of specific parts of application you don't need to run the whole application if you're still not confident you may need some acceptance or UI tests on top of that one thing I would like to mention here is that this testing pyramid we've been talking about and it's out there of course it's like the standard in the industry sometimes it is being shown as the the three layers are like equal the three layers meaning the foundation of the unit then the integration and then atop the UI acceptance and the rest these are not equal parts in the founded in the in the pyramid excuse me the unit test should be the majority of the test and we can't stress that enough yes the testing pyramid is a good analogy how to distribute your tests yes doesn't mean you need UI tests every time in the top of the pyramid it could have a lot of confidence just with unit tests and integration tests depending on your use case yes examine some strategies might require a little bit more UI tests but never more UI tests then integration tests and never more integration tests than unit tests that's a good rule yes that's it the foundation should be fast and reliable if you have that then you can be used on top but until we have the solid foundation no other strategy is going to help you in the long term and at the end of the day the goal should not be to you know favor you need over integration or over UI testing right the goal is to go fast and keep building the vision of the business perpetually so testing is a fantastic tool that can help you do that so like I think of it in our terms of the goal is testing it should not become a liability you know that's that's that's very very important actually you want it to be very very fast you want to be reliable now if you meet these two criteria then you're good yes because the goal of testing is to give you confidence confidence so you can add new features change the current features refactor your code minimal cost yes because you want to go fast and going fast is risky the only way you can take big risks is by having a solid foundation if you have the solid foundation you can take risks because there will be no systemic crash in the system that's that's exactly right so it doesn't matter like again what frameworks are used for testing what like all these preferences are secondary the goal should be is it fast is it reliable does it give me confidence am i confident enough to ship my version of the app yes that's the question you need to to reply to when achieve a testing strategy then when you run the tests they are fast reliable and if they pass you are confident to ship the app you achieved a good balance in your testing strategy he would achieve to the goal now if you run all your tests they are green and you're still not confident and you're gonna manually test everything you spend all the time building the test suite and you don't trust it started helping you achieve the you tomato goal of going fast taking risks and creating profitable solutions exactly so think of the goal first and then you find a way to get there and start with a solid foundation next question I read about the benefits of testing but it takes too long to implement am i doing something wrong should I abandon the idea of testing no should not abandon the idea of testing definitely not because maybe it's taking too long for you to implement it right now but you need to look at other teams if there's a team 100% confident they can ship when the tests pass and it's fast and reliable they proof that it's possible there are bazillion of teams out there doing this and creating amazing software an amazing code bases there is a delight to work on yes so it's been proven that it's possible the problem is that you don't have these skills yet and that's fine they're not born knowing this you need to learn that it's one of those processes that there is no shortcut you need to put the hours you need to put the work and by definition like we've weird we're saying that openly you shouldn't expect results to click and come in stalling you know no the shortcut is to be part of teams that are good at doing this yes it's learning from people they achieved it they can teach you exactly because try to figure this out on your own may never happen yes it may happen but we have take a long time so this is a very hard foundation to build on your own but you don't need to learn these on your own there are a bunch of techniques practices and disciplines you can follow to achieve good results 100% of the time because it's you learn them so don't abandon the idea of testing no but find people taking teacher they can guide you don't try to learn on your own reading random blog posts and everywhere it's just gonna make you more confused exactly and abandoning you I think that's like treats to extremely the testing is one of those processes the remarkable professionals will master like the value that they can bring through testing to the business it's I mean it's amazing simply amazing it's what will allow the business to keep moving forward confidently in most businesses understand that already and it's becoming more and more a mandatory skill yes writing good automated tests it's one of the mandatory skills to remarkable career exactly high paying positions working in exciting projects with remarkable people that's what I got to do need to learn those hard techniques and it's hard it's hard on your own but you can learn from others so try to get some training from your company ask your manager for some training if you cannot get that you might have to go out of your way to learn to pay for those courses out of your own pocket it's an investment it's not a cost you're not paying something as a cost you're investing in yourself you're investing in your career that's for sure it returns yes if you are building valuable skills skills that is gonna make you more valuable in the market it's gonna return you learn as he warms and he delivers forever as long as you practice it yes so you need to be careful with your sources so you don't waste time that's it testing takes time to write and makes you less productive when you're not used to it but don't want burned on the idea now get better in faster eight so you become more productive than before yeah I agree next question I work on a legacy code base with no tests what testing strategies do you suggest to use well that's a hard one yeah you have this legacy code base probably with a bunch of components they're not testable at all because there are no tests they were not mindful about testing sorry be to gradually and strategically start implementing a little bit of each yes when you have a bunch of components they are hard to test in isolation in a legacy code base we might have to use some very high-level UI tests just to guarantee that the whole composition works the whole application works just to give the confidence just changing the code and breaking down the dependencies so I suggest you start with high-level tests they give you confidence to change the current code yes but as soon as you start writing new code create a solid foundation with unit tests and build on top of the solid foundation so you might need some integration tests and maybe some acceptance tests or your white tests by start with the solid foundation for new code and you don't need to write tests for your whole codebase in one go don't stop delivering features to refactor the whole codebase yes every time you need to change anything in the legacy code base you write a little bit of tests around that thing you changed maybe some UI tests to give you confidence to change it then as you change it you break down the dependencies and it tests things in isolation and little by little you clean the code base so that take time but don't do it in one go do it as you go that's it that's it very good guidelines and advice balance basically you need to jump from one strategy to another and it's not easy to yes but the idea is to create a solid foundation of unit tests yes they're fast and reliable so it can run it several times a day yeah every time you make a tiny change you run a test make a tiny change run a test make a tiny change you run a test if your tests are fast and reliable you can do this every 10 seconds every 30 seconds that's the goal and the test should take less than a second to run yes the unit tests but if you cannot do it because the code base is a mess you might have to write some high-level tests and clean as you go next question my tests sometimes pass and sometimes fail can you suggest how can I improve them okay well that's what we mentioned with flaky tests that's the definition here flaky or flakiness it's when your tests sometimes are plussing and sometimes are not well if it's a unit test that sometimes pass and sometimes does not pass you have some unpredictable side effects in your codebase yes exactly you should eliminate the side effects when testing now if you're testing things in India raishin and you're actually talking to some external services out of their control well they're gonna fail from time to time perhaps increase the time out but that's it basically it's gonna make things as lower it's just gonna take longer to fail right thank you yes maybe so if you have a unit test making network requests replace that with a test double so you can control this system and it tests both the success and failure case test all the edge cases with unit tests because you have absolute control in integration if you're testing a bunch of components in integration and there's one part system there's no reliable for example a back-end replace it the integration test yes with something you can control like a stub now you actually want to test talking to this external system there isn't reliable then it's gonna fail from time to time because it's an external system it's out of your control and it's unreliable but a problem of testing and a reliable system that fails from time to time is that you're gonna start ignoring failures and maybe it's a fatal failure right you're not gonna pay attention because oh this system always fails so it's probably another flaky run just ignore it yeah other things that make tests flaky is if there's any dependency between tests for example you have three tests and test two only passes if that's one run before so the order here of execution you mean yes there is any order dependency you cannot run tests you in isolation for example right because it depends on the result of test one it's leaving some side effects in the system yes you don't want that no you should be able to run only one test or run the whole suite and it should be fast and reliable yes so one thing you can do to find those flake tests go to Xcode go to your scheme select your test configuration select the test target and enable randomize test order and run the test a couple times yeah that should do it that should reveal the the flaky ones exactly because I've seen test tests failing because the test-fire was renamed right because normally the tests run you know for betta coordinating the test you change the alphabetical order of it you may start failing yeah that's a flaky test you don't want that yeah they can drive you insane yes how we're naming a test make you fail it makes no sense well if there's an order dependency it does make sense yes everything the fails make sense somehow we just cannot see it exactly so enable random order other things that make tests leaking race conditions enable the thread sanitizer yes for the things that make you white tests and reliable executing coding background thread a white coat in Pegram's head activates the main thread sanitizer so there are a bunch of tools that can help you but ultimately you need to have the discipline of creating those isolated tests that don't fluence each other eliminating side effects exactly and you need to be mindful when you're creating side effects in the first place exactly so Global's I don't know no yes they are just they make things more unreliable yes they are shared State yes exactly so you should eliminate those Global's as much as possible okay I think that's it for this episode okay and if you want to learn more about automated testing strategies go to academy dot essential developer.com that's gonna do it for this edition of the iOS Live essentials podcast we'll see you again next time bye y'all see ya [Music]

Channel: Essential Developer

Views: 1,295

Keywords: ios, swift, ios development, ios engineering, ios app development, xcode, tdd, architecture, advanced ios development, iphone, ipad, advanced swift, clean code, unit testing, testing, xctest, solid principles, uikit, ios testing, advanced ios, advanced testing, how to, senior ios developer, iOS lead essentials, podcast, clean iOS architecture, iOS architecture, integration testing, snapshot, end to end, acceptance, slow tests, snapshot tests, snapshot testing, acceptance tests

Id: f0akKJ_9kIo

Length: 43min 12sec (2592 seconds)

Published: Thu Nov 21 2019