Great Debate - Testers' Role in DevOps Continuous Testing

let me bring up the panelists fuller first Wolfgang plotz founder of tri senta's neck next we have mcurtin CEO of tasks top technologies and finally Jason Steele senior executive at Accenture first of all Mick Jason I'm really sorry that we didn't have a wrap made just for you maybe by the end of this we will so let's have some memorable thoughts let me just start the whole conversation out guys I so a couple facts first of all if you searched LinkedIn roughly three years ago for the word software development engineering test you would have found a hundred titles and you guys can do this today on your phones if you search it today in multiple incarnations there's 40 2000s deaths in LinkedIn there is something going on in the role of the tester Wolfgang you brought it up you said with Google it looks like they're going away or a Google set test is dead we've had the conversation multiple times about the role the tester shifting and potentially shrinking so let's start it off with you what is the future of the tester so thanks for for opening it up like this what we see is actually a huge discrepancy these days between death and test when it comes to numbers the sheer numbers of people if you do a research you're gonna find out that today 21 million of people who call themselves developers these are the latest from from IDC and if you break it down to the guys who would be really focused on development and make the living out of that it would still be a stunning one for 14 million people who would call themselves developers so if you on the other hand take the number of people who do test for the living you're not gonna go way beyond the million all of a sudden we know that 35% of the money spent on tests so 14 million 35 percent is certainly way more than just 1 million so something is wrong in that and this is actually bound to the phenomenon that you've been talking about what we found out is that a very large number of developers to some extent do testing today but guess what even if they do just 10% development and 90% testing they would already call themselves a stet they would already call themselves more a developer than a tester and that it seems still to be the case that the prestige of being a developer is so much higher than being a tester and I think that is one of the reasons why we see this incredible incline I I can only encourage people to still understand testing as a as it is to plan to be proud of and and it's not the developers are the better people so from my perspective yes it's happening but don't be shy stand out there be proud there you go Jason we were just talking off stage and you say you get a number of resumes today what are the titles on those resumes well I think I think a lot of clients are looking for s tetes in the market and let's just sort of unpack what that looks like as a job spec typically clients are insisting on deep technical skills they want to see a history a developer history and they want to see deep skills in automation we even have clients that insist on a four hour exam right to pass their onboarding so I think the clients are getting very serious about technical skills in testing right now why is this is to try and crack the automation problem okay so is that what you do if you have challenges in automation find some s tats drop them in sit back and it all gets fixed no because automation is pretty complex a lot of things we've got to come to get you've got to have the right to lling you've gotta have the right strategy kind of environments at work but s debts can help right and I think as a direction of travel testing having more technical skills in its armory is a good thing but we need balance right it's not the only game in town so let me maybe come on to the port maligns manual tester right the person who's been set as extinct job disappearing in a number of forums to me I hate the word manual tester I think functional tester and at the core of testing is understanding the business domain the context without that you can automate it all you like but you're not adding value so I think functional stir testers are still very relevant we need the right balance but the functional testers are also going to have to learn some new tricks they're gonna have to learn how they can contribute to automation how they can leverage things by AI and analytics to get deeper business understanding and they can even start to participate in in API testing so I think it's getting the balance right I guess one final point is I think if a directional travel is more integration between dev and test it's very useful to have more technical skills in our testing teams right you can go more head-to-head with a developer talk on their level influence them better so I think that's it as a good thing as long as we keep the balance got it so Mick I know you've been doing some research in this area and how teams especially in the DevOps motion are reforming what do you think the impact of this is for the tester yeah I think there's been a pretty significant misunderstanding of the role of tests and the way that companies try to emulate what startups do and what the tech giants have done so I think we have seen a lot of enterprise IT organizations we're trying to try to take this very lightweight approach and bring as much testing as possible into development and leverage all the open source frameworks that you've heard about but if you actually look under the covers of what's happening in a in a tech startup or a growing tech company or what's happening in in some the tech giant's the amount of investment in what we all think of as testing is actually tremendous it's just it's just done a bit differently so I'll give a couple at my company at a stop we we have to integrate many things we integrate 60 different tools but in addition to envy and make those tools work with tools like Tosca so you can connect them to everything you use - Tosca Connect but that integration effort not only do the scrum teams have an S that on each team there's a test product we call the integration factory and half the staff is actually on that test product if you actually dig under the covers and you look at the way that Google does testing the amount of investment they have across the company some of the other tech joints as well at Microsoft they think of quality as a product and within their products across their product they have different levels to fill out that test pyramid and I think what a lot of us have discovered is that the way that you do this the way we've done at a stop so you actually have to model your business your applications and invest in this model-based approach to to test automation testing all the way up and down the pyramid and this idea that you'll actually shift everything left and just embed a single tester on to every scrum team and from that and you know become a software innovator I think is just a is a it's a complete wrong path and a disaster for the companies that take this approach as part of the digital transformation without considering those other layers of the pyramid and the technological investment they need to make those layers work so let's let's do a little prediction so Wolfgang brought this up when he was talking about shift left Jason I know you've have an opinion of this so the idea of shifting testing laps seems to be highly attractive right now but what is gonna be the ultimate impact of this seemingly you know amazing investment to the left side of the equation what do we get to see well I I think one of the questions always shift left but how how left right and you know this is complex because we might have an architecture with many applications in it we might have multiple different types of testing right and now we're trying to integrate different people dev test assets ops all together to work collectively so a question I often ask to clients if they if they talk about where they are in their agile journey and often they'll say hey where we can only release say monthly want to get to weekly is to try and understand where do they get to by the end of the Sprint so this is a kind of conceptual question how much of the testing quote or quality have we achieved by the end of the spring and if the answer is not a lot we've just got developers working still a little bit siloed and then we have to run you know many many weeks of testing well then we're not there I think the more progressive clients are further down the maturity curve look to load in as much as they can and they set the bar very very high in terms of the expectations of what gets delivered and this is no longer successes deliver some code this is deliver code fully unit tested we've got some functional coverage we've done some integration we're running regression right it might still mean we have to go through further test phases right and that's not necessarily the end of the world it depends on your business priorities but I think this is part of the challenge is to kind of know where you are on that maturity curve and then where are you heading in terms of business cadence I'm really unreleased to production but let's look at one thing so you mentioned one thing unit testing right so by the way the first focus usually gets this idea of unit testing and Wolfgang I want I want your opinion on this one how much can how far can you get with this unit testing mentality only and what's the risk of the shift or focused towards that versus the broader life cycle that we used to have before so I'm a big believer in unit testing first of all I've done it myself and and I kept on doing it which a lot of developers don't but I think it's very very useful I would like to give you an analogy with an orchestra okay so see all the different sections of an orchestra as the teams that go for the whole entire landscape you would have two violins here and here would have to Trump it and and you would have other of the types of instruments and the unit tests really make sure that they all can play their instruments nicely each and every one of them should better train should better make sure that they know how to play their instruments and that is great but all of a sudden you these need to play together because we do see these end-to-end processes and when you want to play together you gotta accept that there needs to be some orchestration okay it's simply not gonna work and this orchestration is required and here comes something which was very very interesting and very very relevant each of us becomes tired of practicing I don't know who if you have learned an instrument the comes the situation when you're a little tired of practicing and your skills funnily enough if you have to play in an orchestra this is gonna provide ya an imperative back to you to keep on practicing meaning if you just drop the higher levels of testing like an end to end like an integration test in the higher levels of the orchestra of the orchestration you will not find that extrinsic massive motivation to keep you practicing meaning to keep your unit tests going so it is actually something that only works if it's embedded in a bigger ecosystems and that is something which I'm afraid people tend to forget these days so let me shift the conversation just a little bit here and I think with the DevOps movement we've seen this idea where devs influence is certainly greater today in the Indian Ischl faces of the DevOps movement where does ops come in now and and Mick hopefully you have an opinion on this one but we've seen ops as I guess what was the song that you played Wolf Gang all by myself all by myself so what happens to ops now because there's some theories out there that's you know by the way we structured QA wrong in the first place it should have been in ops so I think this isn't interesting when I but I want to just add something here because I think that the thing I've realized certainly in the research that you mentioned waned all this reaches by the way went to the book called project a product that that I wrote and it will be announced that the DevOps Enterprise Summit which talks about how quality works in agile and DevOps I think the main thing I realized in in doing the research for this book is that ad and DevOps are not enough and it's not a question of embedding test in depth right and that that will get you what you need I'll get you a successful transformation or embedding quality and ops or making quality part of ops and so on I think what we've seen DevOps do is help focus on you know the three principles of DevOps basically flow between Devon Ops and feedback back and hopefully towards the business and then this learning process but the biggest problem that I've noticed in terms of organizations having their digital transformations go sideways is that if you think of the entrant value stream because in the end all that matters is how much quality software you deliver to the business how quickly and how much better you get at that so that you can move fast enough to not have a you know one of 50 startups or or two tech giants take a significant chunk of your business so the thing that I learned is that through this research is that the biggest cause of failure is that companies look at optimizing only a segment of their value stream just because it's simpler right agile was very nice because it helped development teams plan were iteratively but companies if we look at the poster child in 2009 of agile transformations Nokia they were amazing at how they did agile they had a really great metric that the leadership of the company understood and it was called you know became called the Nokia test from Nokia Siemens Networks on how many people you trained on scrum so did that make Nokia more agile or did they actually just optimize that small part of their end-to-end delivery process and completely missed the boat on this mountain of technical debt that they had in a series 16 series for the operating systems that did not allow them to innovate once the screens became bigger it did not allow them to build enough quality software quickly enough so I think the the really large failures of agile I've seen is when it's treated as you know just you're only optimizing just for agile and you're tracking these activity metrics we're now seeing some where DevOps failures where you're just optimizing continuous delivery continuous integration you're only focusing on your unit testing when you do that because it's an obvious thing you need to do it but then of course the entire focus become let's make our automated testing gate as quick as possible it's just unit testing and then the problem is you realize and I've lived as the hard way you realize after some time you're not actually delivering better quality software to the market faster you've just optimized how quickly you can turn the crank on delivery but you haven't actually optimized for antenna quality so I think the really big thing that we need to look at right now because the continuous integration and the unit testing and shifting left is actually it's much easier than understanding your quality at an end-to-end value stream level and actually making the investment across the entire test pyramid and looking at how you know through this cultural change of bringing more testing into development but also making testing itself be a product making your automation layer itself treating that as a product as the tech giant's that stars have done I think that needs to be the focus the answer is not at all in shifting left that's a almost a necessary condition of success right now but if treating quality as an end-to-end product and what does that mean how do you surface a dashboard that's meaningful to your CEO who's really going to care that as you're moving faster as you're doing continuous integration as you're moving some of you your products cloud negative you're actually delivering at a quality that's meaningful to your customers so I think this this idea that's going to happen in ops yes Ops is a part of it but it's it's actually a small part of it if you look at the end-to-end value stream so it all seems very logical what you said seems very reasonable why the hell haven't we done this sooner we why are we in this position now we're it's such a massive transformation to get testing into the mainstream yeah so I'll just answer quickly though the others but I think it's we've been taking this overly siloed approach in these transformations and so in the book I actually detail at large banks at top 25 by revenue banks their transformation story that I got to watch from the sidelines for six years so it was one transformation started six years ago took two years to fail another and of course the VP's and some of the executives involved move on get moved on then it's another one started the first one was just a it was a LM transformation that next one was an adult transformation and now they're doing their agile devops transformation and they're getting towards the tail end of it and again it's that exact same approaches rather than looking at these end-to-end metrics and an end-to-end way of measuring things they're looking at how continuous integration continuous delivery is doing and that's it that's the focus so there's actually no business level metric as you're transforming a really large Bank and this transformation by the way the current one the budget the incremental budget for it's a billion dollars yet there's no visibility of what's happening to quality at the business level and this is what I realized is we actually need to come up with these common metrics that not only all of us technologists understand which which we do but that the business understands and quality is one of those those top-level metrics we need to understand the velocity and quality and we need to do this for each of our value streams so I think we're doing the easy things I think this is the problem is that in these transformations they're decomposed into these two year one year project and the easy thing is let's shift left let's use open source let's get developers to do more testing without actually looking at and to end value stream quality and so the results predictable is you succeed at having a bunch of assets you don't succeed it's bringing quality products to market faster and and your market declines so the way I would look at it is and I think this is something we see everywhere there has been waterfall waterfall has been very very strict you have certain faces and and it was all clear how you would need to go and all of a sudden it became very very clunky and very very slow and and it turned out to be the wrong way of doing things in a in an age where we are and now the pendulum swung completely to the antia to the other side and we say we gonna break this building down completely entirely cut it down no towers no walls no nothing get down Zak we move over to a gel and what we and and and what happens is that with the same religious fundamentalism people were defending waterfall they're now defending and I'm a big believer in in in like thesis antithesis synthesis okay so what's gonna happen is I think the pendulum is going to come back to kind of a medium position that that's gonna free us from from some observe developments here you ask the question why are we in this position my answer to that is we can't help it that's the way things are somebody needs to come up with a completely disruptive idea otherwise things are not happening and in that regard you're gonna throw everything overboard we've seen this again and again and again in our political systems right and then it comes back hopefully but then there's gonna be the next one in another direction right it's always gonna go like this you can't help it it's always wrestling with the Beast yeah I think from my perspective ideally if you're trying to transform delivery that starts top-down and it's pervasive across the enterprise how often do we see that very rarely right because it is so big what we see much more frequently is that most clients there'll be some kind of agile initiative a DevOps initiative and in each case it could be you know as as small as just some very tactical to put talling focus or it could be pretty pervasive right trying to integrate all these multiple programs is difficult and it's difficult because we're cutting across many different aspects of the organization right and that then starts to get often quite political I've been in workshops where dev will say Co quality is great QA will say no it is not right and there's a kind of pitch battle with the you know kind of people on the same team right so there are political boundaries to cover as well and I think then the last thing which is it's kind of simple but it's a real problem and the business still wants delivery to happen how much of our capacity are we assigning to transformation versus day-to-day delivery right and the more that is a small amount five ten percent we're not going to transform quick right so I think the organizations that have done this the quickest have done it top-down and they've been prepared to compromise on near-term delivery to themselves fully transformed but there's fewer clients of us that we've seen that do that for most clients they're trying to kind of grow organically transform over a longer period of time got it very interesting so what we're also seeing now is another trend which is in sourcing which is pretty controversial right now because we actually committed a lot of resources to support a third party to actually test the software manually how is this gonna impact the organization and I'd like to start with wolfgang on this one so yes we see this starting more and more we see we see organizations trying to make use of like pick the raisins right make use of the labor arbitrage on one hand but also in source the the testing efforts by having their own delivery centers set up in India for instance right but they want to in sauce it why because they have been some of them have been heavily disappointed by what they've got right I mean this guy from the champions world which I picked up today morning it's not an exception he said we have been so highly disappointed by what we have come from testing and since you bring it up I gotta tell him a story okay you give me 30 seconds on this it is funny one of her customers a large bank they had an automation going OK by one of the sourcing partners and they automated 3,000 test cases so and one year later they would find out that the execution time of these 3,000 test cases had been reduced in half so everybody was going to hey this is cool right I mean you speed it up massively right and and and they said yeah yeah yeah we did some optimizations here ok so but one of the deaf guys didn't really like the answer and they had an open-source framework going and he looked into the test scripts and found out that whenever these tests would run into a problem they would say return true meaning whenever the maintenance issue occurred they would just break right in front of the maintenance agency returned true so all of a sudden the the execution time of the test cases would be just reduced so this is a real example this is not that's something I invented so I can fully understand that there is a certain level of a frustration that happened at some customers and and so they see that they will be better off in insourcing I believe the key the key to avoid situations like this is that you still are able to understand what the tests do if you have a technical representation of a test you're not going to be able to understand what they do you can't be blackmailed whenever you want and that is not a good thing because chances even if they're not good are gonna be taken and insourcing I don't think it's so much about insourcing versus outsourcing to be true it is what you expect the artifacts to look like and there is nothing more healthy than being able to understand these tests and be able to run them on your own if need be even though you don't have the capacity but only if people know that you could do it in theory because you understand what these tests do is gonna be super super healthy interesting Mick and by the way we're gonna have Accenture the last comment on this one yeah so I'm gonna I'm a bit dogmatic on this one I think quality is core you need to treat quality as a product internally and you do not outsource a core product so now that's said there exceptions to this I think what you'll know this happened and of course there's some organizations where key parts of your portfolio will actually be assisted by you know resources outside your company and that's a little bit different but you treat it as core as you keep treat your other core product wherever the resources sit it's as important to your business and this is exactly how we see this happen inside again the successful tech companies the unicorns and the tech giants and I think basically you business level visibility into your quality you need to understand the way that you create the models that will allow your testing the scale is actually very closely related to your business so again I think the key thing is however you build your core products you build your quality platforms the same way your automation platforms the same way and they are as escorts your success and in many cases I know in some of our cases we'll have written more lines of code we'll have created more models or extended more models around the quality called the aspects of the product then for the functionality and features itself interesting Jason a bit dying to hear yeah I was so pleased with that question yeah thank you so yeah I think there's there are different options out there in the market I think in source outsource it comes down to the clients strategic focus whether or not they believe they can build the right resource and get access to the right skills there's probably a commercial angle in there as well I don't think there's a single right answer I think for clients they can work in in different ways I think what I would say I think it's always important even in an outsourced situation that the client retains enough knowledge of what's going on from a kind of architecture perspective so we've seen clients before and this is this example some dev I realize they don't actually know how their application works anymore because all of the knowledge is in the heads of the outsourcing vendor so that's not that's not a good situation I think the client should always retain enough that they are steering the ship they understand how things work and then and then they have flexibility right over over where they go they go next thank you for that so we're just about at the end of time here so let's ask one more question and I'd like this one to be more advice so when I go to these software testing conferences I think one of the biggest questions I hear from the audience is I'm a tester or I'm a manager of testers what do I need to do I'm the world shifting around me all my colleagues have changed their resume to say yes that what do I need to do as a tester today to survive in the future world who wants to go first on this one Jase so I would say testing has still got an awful lot to add it's still very important and Taran as testers QA professionals we absolutely should be pragmatic right we have a role to play we need to be pragmatic but also proactive driving the agenda right testing can be critical to driving more automation smarter testing approaches and we should embrace the change okay and for me the teams that add the most value are the ones that the forefront of the change agenda they're the ones going to dev say hey you guys should be testing more thoroughly right you should be building better virtualization capabilities so I think you know in any organization there's there's there's a political nature right and there's multiple initiatives normally to kind of think through but I think in essence we should be proud of what QA brings and we should absolutely be proactive at driving change thank you for that Wolfgang so I think there's two options that you could do so if you have a look at this pyramid thing that we have been talking about today and the horizon of unit test integration which is very much bound to the dev side and the higher levels that are more bumped to the upside and gonna be even more I think that there's two different flavors one is very tech focused where you're gonna have a very close cooperation with the developers and the other one is pointing towards business and when you want to be a testa and you want to be relevant in the future you've got to ask yourself how am I gonna distinguish myself from the others and there's two ways one is you learn how deafs work and you learn how to do tech that is one thing that's gonna help of course and this is what most of the people do today the other route however which I think is going to gain dramatic importance given what's happening now with the scaling issue that everybody has wants to switch over to DevOps is learn business learn business understand how business works and understand it for certain areas of business in the banking space here and there in the insurance base here and there ideally even tied to some packaged application products that are there that is how you're gonna have a great great value to proceed in the future your decision right it's an intersection take the left take the right if you stay in the middle let's calm Mick so the main point I make in this book project the product is that to survive this digital disruption that's gonna get even even more significant with a tech giant's growing even faster and then this new breed of unicorns on them turning tech giants project this idea of project management the old ways of looking at things just do an agile just doing their vibes it's over companies that try to shift this way will not succeed and they will decline and what every organization has to do is create their internal value Street Network and basically become software innovators by understanding their product portfolio and by measuring the flow the quality the time to market of each of these products and I think what's happened in the last ten years of agile and I think Wolfgang made a really good point on this and more recently in DevOps is again we've looked too small we've actually forgotten the role of quality and I think it's now at the point it's basically on this community to define for the business for the CEO for the CIO what a product value stream what the quality for that value stream needs we're missing these definitions we have testing strategies that simply will not scale we have these completely in-house custom frameworks that every tech giant has built for itself but that no one else has access to so we need both the vendors to provide off-the-shelf testing software open platforms for testing so that Enterprise IT organizations can leverage them and I think we really need you to define what quality is for each product value stream as every single company needs to shift away from this way of managing ideas project and ending up with 70 80 percent of legacy and this mass to becoming innovators in a product oriented way where that again the time to market the cost the revenue and the quality of each product is understood by the business and I think it's it's really time to elevate that discussion and again for you to make the case to the business of this is what quality is and the world of continuous in the world of agile and DevOps and lean so gentlemen first of all thank you Jason Wolfgang MIT big round of applause for our for the big brains in the room I think the most interesting thing as we wrap up the second issue of the great debate is that the the concept of change is beyond us we're over the threshold and it's now about transformation and that's what I took away from you guys today saying that we are we are beyond the tipping point on this and digital transformation is here and the transition needs to happen now thank you so much guys I really appreciate your time [Applause]

Channel: TRICENTIS

Views: 3,668

Rating: 5 out of 5

Keywords:

Id: xPlGmtAXzeE

Length: 33min 55sec (2035 seconds)

Published: Tue Oct 23 2018