Frankenbuilds: If Agile is so Good, Why Are Our Products so Bad? • Gabrielle Benefield • GOTO 2012

and now we're going to talk about how agilistas might be killing the planet who was here for Jazz's talk earlier on value hands up most of you and who was here for Josh's talk okay there's going to be some overlap hopefully I can get into a little more depth because um I came along to their sessions to make sure that we weren't going to have too much overlap so similar theme let's go through it agilistas are killing the planet so agile lean scrum cban they have all the answers do they are they really solving our problems or are they just helping us to build the wrong thing writer there's been a really big focus on let's make our development work better the issue with that is we're delivering all these Milestones we're delivering working software so the great promise of agile was that hey you know waterfall is so terrible because they look at phases we say that we've done planning we've done delivery we're making all this progress and we sort of trashed that in the agile Community we said well hey that's that's useless all we could have is a pile of documents we might not have anything useful to show for it so instead we're going to quickly build working software show it to our clients let's see how that's turned out for us in fact all they did was push more through the system faster so now what we've done is we've gone from a model where we tracked activity right we tracked that we'd actually finished some work activity we've gone to tracking outputs we have built more stuff but is that actually a good thing so I hear so many people saying we get our products out on time and on budget does it matter if it's the wrong thing for the wrong people agile takes a machine gun to product development so what we're effectively doing is taking lots of shots and hoping that some of them hit our Target the issue with that is we cake out a lot of innocent bystanders on the way and effectively we could be creating a whole lot of waste so output is not necessarily a good thing right what we've done is effectively by building more increased the odds of success by using the machine gun whereas what we really want is a very targeted sniper gun I was at a meeting once and the VP stood up and said hey team it's so great we delivered 35 features on time and on budget and everyone's clapping away and I'm at the back saying yeah but did we make money did we deliver results in fact would two have done the same job capus Jones had an interesting statistic in one of his books on estimation he said that 30 development accounted for about 30% of product development costs which means there's 70% surrounding it over the years because I've been doing agile for a very long time in various forms I found that um you know it's not that agile has been a bad thing it's cleaned up a lot of the mess in the middle Zone that develop velopment Zone and the great promise of Frameworks like scrum is to make the dysfunctions visible so what we've now found is that we clean that middle mess up we're making stuff get through the pipe quicker and instead what we found is that what's coming in the requests and what's getting out are the wrong things so instead of building the wrong thing writer so instead of saying you know the big question is being which is better caran or scrum completely the wrong question it does not matter if you're building the wrong thing what we're doing is gold plating the development mechanisms we're not actually improving the product itself instead we want to build the right thing so I'm going to talk to you a lot about building outcomes over outputs now uh people who know of Tom Gil back in the 60s he's been saying this he's been saying hey we need to measure value and it's taken a long time for the rest of the world to start catching up Tom's a brilliant guy almost too smart I think it took this long for everyone to finish competitive engineering book but uh what what Tom's been saying for a long time is you need to measure value Nothing Else Matters his whole model is based on Evo which is little one week iterative steps to get feedback and to look at your progress measures so I've been working with Tom a long time to figure out you know how do we make this more used and more acceptable to the world so that's been a bit of a Lynch pin in what I've been doing to give a bit of background in the late '90s um we were doing some XP just very technically focused 200 three I was in a San Francisco startup I came in to run the product team and then they ended up giving me everything they gave me Product Development operations Etc and they said in six months we want to take the company public so we want to do a big IPO and at this point this is a company that has like six Waring VPS this is like Afghanistan in practice we have everybody disagreeing we have separate lists think of those as six different back logs for all the agile people and we don't have that big development team so we have everyone building lots of things lots of work in progress nothing actually getting done so when I walked in and I said okay well let's start cleaning up with this mess and we started to use a bit of lightweight sort of scrum I started bringing in lean thinking lean portfolio thinking and the number one standout item was that we were on this massive feature farming Drive what we were trying to do is put so many features out at light speed and we were going to make lots of money who's seen those proposal which come through all of these features each featur is worth x amount of money I'm having a big argument with my co-authors of the scrum primer there's a little column in the backround uh backlog which I sort of missed the first time which assigns value to each product backlog item I'm like how do you know you know what's the value of something like a login that's way too detailed and um so these people thought that each feature was worth money I looked at them I thought that's that's not our problem at all so I went up to the board of directors and I looked at them and I said okay so who goes to Amazon and eBay and I'll ask you all this who goes and uses Amazon and eBay regularly like at least once once a month right sure we do all the shopping there it's CU we don't like to get up from programming uh so uh I said okay do you uh who was one of the people who uses it a lot who would say that like a hardcore user of Amazon or Ebay hands up okay I'll ask you do you go to Amazon or Ebay every week because you're wanting to see what's the new feature that they're going to for you to use no no what do you go there for Amazon for the books for the books and what else eBay no no okay uh so you go there for the books you want to buy something it's pretty simple if you're Amazon simple value statement I want to buy something if you're someone like Skype simple value statement I want to make a call and yet most of these products need to go out of their way to put as many obstacles and features in your way as they possibly can so the the more features you send out the worse it actually gets so this was our problem we were on this feature farming exercise and in fact that wasn't our problem at all so I said we're not going to build lots of new features we're actually going to perform uh improve the performance and usability all the qualities of our website our Revenue curve went from being small hardly going up until so steeply vertical that in 6 months time when we were ready to go public the bank has called us in and said stop you're not allowed to go public I'm like what we've done a great job we're almost down to zero defects we'd actually halfway through being actually replacing Legacy code we had the best XP is on the planet everything was going so well and they said I said look at the revenue curve and they said that's the problem I don't know if anyone's everever tried to take a company public why would our Revenue curve be a problem because it goes up any ideas okay be careful of this if you get really good at what you're doing uh the problem is that it's not sustainable they're like no one goes into a vertical Cliff jump it looks like you're actually fiddling the books so I had to literally go back to my team and say okay guys stop what you're doing let's uh actually tail that off get a sustainable curve so we could take the company public so that's the dramatic difference that stuff we're hearing today about lean startups Etc that's what matters we were lucky enough to have uh the chief scientist from Amazon at the time Amazon is a company that drives themselves by day everything they do they test it's amazing Amazon in the early days if anyone used it back then but it used to take about 3 weeks to get a book two to three weeks right your daughter get a book someone went up and said well how can we get a book and maybe you know let's get it down to a week three days one day then who had the brilliant idea you know if someone wants to get a book would you like to get a book in under one minute of course right it's the worst worst product in the world for me I'm sitting at something like a conference I'm there and I'm like one click byy it's downloading to my Kindle fantastic so they get the whole idea that's all about that user experience so uh one thing that I've been um working a lot with lately is a guy called Ryan Sher in the US and I've been creating something called the target outcomes framework so we really want to drive home you've got to go for outcomes not outputs okay something that uh we need to re-educate our clients in and as you heard Josh and Iran talking about earlier velocity all these estimated measures they they're just getting in the way of what we need to do which is to deliver value faster so think of outcomes outcomes are the results we expect they're the destination we want to get to options uh once you create outcomes the these are the results there's many ways to get there so we're going to talk through how to get started I want to give you a little bit of practical information we've only got a little bit less than an hour this something we're going to have uh a book coming out on this and some more detailed guidance what I found found I love The Lean Startup book but when you go through it it sounds fantastic until you try to do it and if you've got a little website that's okay but when you're running this at scale I run it with a company who has literally millions of users and very complicated product sets if you're doing this at Enterprise levels I find that it starts falling down they can be a little there's not a lot of information out there on how to get this going um big scale uh one thing I like to do there's a lot of lean people in the room who knows about A3 thinking coming out of Toyota okay not a lot of lean people uh lean gets kind of bastardized in our community it tends to be about reducing waste it's all about in a way trying to make things more efficient great ideas but one of the key tenants is about this idea of learning now A3 thinking there's some good books on it there's a Soc book on A3 thinking there's another managing to learn book and it's a very deep discipline it makes you think about systemic reasons and why we have problems or causes for things so this is actually something we've blended into our model so that we understand the context the users the environment extremely deeply before we start deciding where to go I think sometimes we sort of leap frog that we jump straight into we want to get here but actually we don't know the problems we don't know the reasons why so one thing you can do we're hearing about user experience I'm going to tell you to be the method actor of user experience you know Robert daero these famous actors go out they actually if they have a a movie where they're going to be working behind the counter in Supermarket they get a job in the counter of the supermarket uh so never ask your users what they want and in fact the worst thing you could ever do is ask your developers what they think the users want or even your on-site customer right they might be a million away miles away from your actual users I always love that quote which you probably seen if I'd asked my customers what they wanted we would have built Faster Horses so one thing you never do is go and get that product requirements backlog list from users you'll build the wrong thing almost guaranteed get out of your chair and go find out what they need uh I'll talk a bit more about this later but uh this can be done very quickly and easily there's a great story idio uh one of the Silicon Valley think tanks they build great products they were investigating they got a job to uh create children's toothbrush so everybody at that time thought children are little people right little people have little hands we're going to build a little toothbrush so everybody had built these little toothbrushes right thinking little hands Little People little toothbrush they went out and actually watched little people doing their toothbrush stuff and guess what they found I don't know if anyone's got kids little kids with little toothbrushes they brush their teeth like that right they have no fine motor skills so actually completely the wrong solution they came up with those big toothbrushes they were the guys who came up that invention so the kids could grasp them easily they went from something like under 20% market share to 75% market share sharely through going out and observing the users uh Yahoo we used to do a lot of the stuff we'd beg borrow steel go and get friends whatever we could do to get a sense and to understand them so remember I said earlier you want to go to Amazon to get a book you are on Skype to make a call those are some of the things that when we go in we both look at those goals if you're in existing products which a lot of us are could even be your business processes doesn't matter figure out what it is they want to do use some of your lean techniques like cycle times to measure those what we're going to do is keep iterating to make those better and better and better so figure out what the core jobs are and always be very specific about making that as easy and quick for our users to do create strategies for getting to Market at the right time I saw our jez had a little Target with the one person in the middle to Etc Great Idea be aware that product development is a bit of a black art it's far more sophisticated if you're at a startup you have different users at different life cycles for example if I'm a startup I want to get viral thought leaders so I want someone like one of our um personas we created I was in The Innovation group at Yahoo we went out and we were looking at a youth product and we targeted this girl call Susan because we saw in the breaks she had a piece of technology she'd be looking at it there'd be like eight kids all around her watching what Susan was doing so she was the thought leader and we thought you know what if we want to get this product out to many kids we want her to be our Target that was one of our personas after a while we then want to go for the kids who are going to be the repeat users so again a lot of this gets into a lot of sophis icated thinking about our users I mean at the moment with some teams I pretty much start with drawing a little stick figure bubble head it's better to have a user than no user so I don't want you to think you have to get super sophisticated but again create empathy and get people to drive products uh software is used by real people um once we do this analysis again I'm going through it fairly quickly but we've decided um what's really going on with our products um I a lot of G data Gathering it's one of the first things I do at companies one of our clients who uses builds a product you probably all would have used millions of people are using it around the world we went in and said okay what what are the usage patterns what are people really doing with this product it's really interesting 3 days later one of the product owners came back and said you know how you told me to investigate all this stuff and get some of this data back he said we found this one feature that's been on the backlog for a while if we launch it now well actually they were they said we're about to launch it tomorrow it's pretty simple that's going to save us 12 million a year so immediately by looking at the data they could figure out what's going on that was great that was our first week of Consulting at a client so that pretty much paid for us so um once we've done this data Gathering we've understood our users because what we're trying to do is do deep deep thinking we can use techniques like five wi value stream mapping what's really going on once we understand it now we can do the impact mapping if you know the severity of the problem or the opportunity that helps you immediately prioritize I love those agile classes just a quick as side where they say you can do bubble sorting you get two stories you go let's do a or b which one goes higher and it's like seriously is that a way to build a product you know how do we that makes no sense to me instead where are we going to get the most impact um put on your 8020 thinking caps we've all seen those Standish group reports but are the ones that show that about 20 % of the features are used 80% of the time this is what we're doing as the agile Community we have a great method for getting products out quickly optimize for that 20% and we'll talk about ways you know which is the right 20% but it starts before you waste all your time building user stories understand where is the greatest upside the problems that we experience we set the target outcomes so Target outcomes I'm going to call them the Pokey yoke of product development anyone heard that ter poke before yeah see if you throw Japanese words everyone's like how cool is that so poke make it easy to do the right thing and hard to do the wrong thing I hate using this example because it makes me sound like I'm about 200 but floppy discs when you opened up the little floppy disc container there was one way to put it in right and not the way to put it in wrong right so they designed in idiot proofing right they made it very easy to choose and this is what we need to do as idiot proof our products so that we can tell which features should go in and better still which features not to build remember we're dealing with the world that we're getting so good at building stuff fast that's making us lazy and it's making easy to build too many things my best product owners I work with are very good at saying no well not now we're not going to do that now Target outcomes need to be clear and measurable uh we've kind of used a simplified version of Gil's work so we have very clear Baseline again you need data to set a good Baseline we have a very good Target this is the point we want to get to for example at Amazon it takes two weeks to buy a book we want to get that down to 3 days let's keep building things and features and enhancements to get that cycle down okay that's a simple version of setting Target outcomes half time okay and if you don't measure how you know when you get there um it is a discipline I get people it's hard to get this statick at first so I get people saying well we can't quantify things you know we can't quantify the user experience right who thinks that who thinks it's hard to quantify everything any hands yeah okay so I had this recently so one of the other things I'm doing at the moment is writing a book on legal contracts that can be used for Agile development it's actually why I spent a lot of time building a Target outcomes framework because fundamentally legal contracts are based on a goods and service contract it's based on coming up with description of the goods you want like you go to a shop hand over I want buttermilk eggs they give you buttermilk eggs you get paid for it fundamentally floored for the complex development we do on software so now we need a way to say let's be results driven let's come up with outcomes that you measure and uh based on that uh we have to make everything measurable so I was working with big uh charity I went in to do a quick contract review just to see what they were looking at they had some words in there we want our system they were building an internal um CRM so they get a lot of people contributing and donating to their cuse they want to track what they're doing at the moment everybody's using things like Excel 10 different products Etc they want to centralize it be able to do better data reporting Etc and so they had written a contract for the suppliers which said we want our system to be easy to use and to be very consistent okay so what does consistent mean any ideas throw some ideas out consistent first thing that pops to mind same thing not go down anyone elseed predictable what does predictable mean see I like that version of predictable that should be for agile teams you can guess what they might do okay right okay so I'm going through it and I'm mentally assuming consistent well it means that it should work like if you click on a right button here it should be the same right button and you should go back and it should act that way and I said is that what you mean let's really Define it and she said no no no no no I mean that once you run your report the data is correct so consistent meant that it was actually data that was validated and it was actually correct and more precise data whoa who knew right so you can imagine the kind of systems we were about to build with these people so when people say you can't measure everything yeah there are some difficulties last week I was down with a very big video gaming company helping them to build their video games very cool industry and I was kind of thinking how do I get this working what a Target outcomes for gamers right so first I said to them who's your persona who are you building it for a gamer I'm like okay you've got at least three I can think of you got a hardcore gamer a call gamer and a Casual Gamer very different use cases one has to get into the game as a Casual Gamer has to be easy to use hardcore Gamers need skill difficulty I said okay and what are the goals for these Gamers well we're going to build a game where you win at the end imagine it's whatever racing game you hit the set track I'm like wow that's going to be a great game I can make that now I can build a little racing track where they hit a Finish Line we're good to go well no it's about more than that what is it about oh it's about fun it's about what is fun okay so we we started breaking it down really interesting we started building a little I was calling them an emotion map so we had things like attractors how do you attract people into the game we had frustrators if something's not annoying enough then you get a bit bored with the game right we had to get to a point that someone gets to massive Elation and winning and then this tail off what do they do next how do you make it so that they can re-experience that feeling get addicted back to your game so there are ways to kind of figure out what are the pieces that matter how do we measure them right a measure can be Quantified but there's other qualitative measures as well but what we're trying to do and this is the thing that I think most people Miss we're not measuring to get the most precise number just like we don't do things like uh poke planning to actually get the best estimates why do we do something like poke planning wow you're a really crowd to share knowledge right yeah so what we're trying to do when we quantify these things is get down to a level of understanding so that we've created pooke easy to do the right thing hard to do the right wrong thing okay so that you don't want your product own or your customer who's got customers that they can't even get a hold of anyone work with those people right I get people where they're dealing with Generals in the Army who are like okay product development team or war in Afghanistan guess who's going to win right when you got those people you've got one opportunity to really try and understand their greater outcomes that make this work cuz you can't sit there going should it be blue or green what do you think armig go like you know those aren't the sort of things they can spend time on so what we're trying to do is we set a direction once we have a very very clear direction if you do nothing else then this will give you a lot of power over choosing what to do next in your product then we want to go into something called options thinking so back to Agile or whatever methodology a lot of them based on outputs what you're doing as soon as you start writing up outputs be it your product backlog your road map you're committing to a certain future you're saying this is what we think we're going to build this is the danger of current legal contracts they force you into defining up front and committing even if you can change it around it's harder because once you give a feature a name or put it in a document has a life of its own and people feel like you're killing it at that point if you take that life away so we're going to stop pretending that you can see the future and again you've had this theme pushed at you today about Lean Startup create options so instead of having one thing and have multiple ideas right we've got an outcome to get to but there's many ways to reach that outcome so keep your options open what we want to do is again this is back to responsible moment thinking you want to make decisions as reversible as possible and irreversible decisions as late as possible options thinking is all about setting up potential ideas and then being able to execute on them rapidly I was talking the other day oh that's interesting but I had a different slide up there uh we were talking about we were talking about yeah we were talking about uh Darwin's theory survival of the fittest it's not about survival survival there's a movie called dudes it was like a teen movie I think they F rout survival is the slowest film with suicide that's a way of Treading Water arrival is the of the fittest is where it's actually at so it's the people who can get to the market with the right product the quickest not the ones who can just survive in that market we want to be better than surviving the outcomes options idea is fractal so what you can do is have high level outcomes for the company things like reach engagement with our customers monetization they should link through and align to every area of development right down to the qualities of your system right so every big outcome for a company breaks down into multiple options think of that as your portfolio your road map all of those break down into different options and streams and so forth okay so it's a very flexible more of a model that looks like a very complex system as opposed to the linear stuff that we've got ourselves into an agile so if you think about your product backlogs they're very linear they lead to very tactical thinking that's why originally the talk was called Franken builds we keep hacking things together and hoping for a good result so all this does that sound simple you can go out and do that all right how do you know what the right thing is we don't we have noing clue right we never know I like the idea we keep saying there's no such thing as product requirements they're product guesses they are options and experiments we could run never think you know in advance because we don't so stop guessing start testing we've talked about the idea running multiple options fast the thing is we're all about Nile making the unknowns known that's why we do vertical slices of our product Etc we're trying to highlight risks we're trying to get ideas and information back quickly we talked about set-based design with Josh already so I won't talk about that that was a slide I thought I'd have so get it in front of your users early and often um another big tip I'd give you I noticed a dramatic change we had very Advanced researchers at Yahoo user experienced people we had people from apples Innovation lands we had the whole gamut what I found is they would say hey this doesn't work the people don't use it the developers never quite got it so we started taking our development teams out on field trips with us seeing as believing you need to experience it even in our usability Labs I'll give you an example Ajax had just come out so you could drag and drop and so the team had designed this great photo album feature where we could drag and drop your photos all the execs were like ooh ah this the best thing ever you know it's going to be the breaking ground and uh we got the lab going so we had someone coming in behind that two-way glass and they get in and they're looking at it and someone says okay so how would you create an album and move your photos into it and that clicking around they're like I don't know I give up behind the screen is the developer he's sitting there going no what' you do you movie you stupid person you know because of course that user was an idiot because the developer you know obviously smart to design this very cool system so uh anyway they walk out and Scott just said give me a laptop runs over we push it through to the next room before the next person got in for the next test problem fixed we' been telling him this for 2 months Scott they don't get it people don't get this drag and drop thing it's not it's not intuitive so by taking them out we Chang the whole model um ignore your mother be fast dirty and cheap hopefully my mother's not watching this um everyone comes to me I got someone the other day where i'm like they were building something and I'm like why are you building it that way you don't even know if it works and they said well in Q4 we're going to get a user testing group to come in and do all this research I went what do you mean don't you have like friends or family I'm like go and get couple computers will set up this test so immediately we get feedback um to give an example I was in New Zealand giving a class there and I had someone from the uh police department in the government they were building all the software all the tracking of the police records and she used to be a policewoman and she said you know what our software is so unusable it makes me sick but I tell the guys and they don't get it I said you're a police woman right yep were you local yep still got friends there yep okay I said how much will it cost for a little taxi to take everyone in I don't know 20 bucks I said what about do police people like donuts is that a myth or true and she went actually they really do and I said okay get a box of donuts get your team go and spend the afternoon with the police people so they did so for 20 bucks in the cost of a box of donuts they went and sat beside the police people immediately they saw Joe who was 56 with two kids exhausted after being all down the the beat coming in trying to add his time and it was just miserable they felt so bad they created a little picture of Joe put it on the wall and every time they went to build something they went yeah but would Joe really get that probably not right so this is like Fast Dirty Cheap a team in Bangalore India every lunchtime they go out on a Friday say hey we'll give you a free lunch come and try out our Yahoo product that's all it costs to get rapid feedback so don't think it has to be expensive to do this remember that features not the experience again Pokey create this is the goal this is the job our customers need to get done more features are actually a problem what we need to do is remove the inessential so the essential can speak right the more features on the page the harder it is to use when the iPod came out they had about 40% of the feature set of most of the other MP3 players because they got what the job people wanted to get done was to actually get music quickly and listen to it right which meant you had to be able to connect to someone to get somewhere to get the music and get it and listen to it so again the game is to hit the target with this few bullets as possible that's why I'm very concerned by current legal contracts they incentivize building more I'm concerned about things like velocity cycle time measures which measure how much throughput we can get to me that's telling your developers to build as much as possible right I want you to do the opposite so the game we are very smart people we work with we set a game Target which is we need to get to here as quickly and with as few features as possible if you do that maybe you can go home early right I don't want more code so this is back to that idea of Lan startup we're building minimal viable not maximum possible your job who are the product owners in the room heny yeah your your job is to try and filter that to get as little through that pipe as possible measure as you go now everyone's listening to metrics going yeah I get that we say that we need 20 new signups or make x amount of Revenue what they do is they put in a pitch has anyone seen those uh bids for work we want these 200 resources and a year and we're going to make $50 million it's the greatest product ever right we've all seen those don't believe it what we do is we constantly measure that way we get feedback are we in the right direction we set up automated metric Frameworks so that we can get this real time feedback back at that beginning step where we talked about go gather the data this will when when you'll start to see the holes in your organizations do you have the usage statistics do you know how people are using your products if you don't start building those in as quickly as possible this is what will drive your company this is why Amazon's so good they constantly look at the number numers they set up very sophisticated Frameworks so if your metrics lag your product will lag you need to get those quickly I won't talk too much about continuous deployment this is where the agile group can come in again that we say our job is to make it as fast as possible to get that product to Market user testing we talked about that I although I said I use some lab test Tes in um I really like setting up multivariate testing the idea of split testing with multiple outcomes if you do that why test with five people in a lab can actually give you substandard results we used to be able to very quickly at a startup in San Francisco flip the switch and immediately we could siphon off a couple of percent and be testing Real Time with thousands of users that gave us much more contextual data uh one thing I haven't heard a lot about is we talk about metrics and G Gathering them and a lot of people think that's for the managers or the product owners I always treat my developers I think essentially they have a gam of mind so they like playing games maybe but also they like things like The Thrill The Kill finding defects you know they have a gam of mindset so when I set targets transparency is key give all those numbers back to your developers at the startup every month we'd have a pizza meeting we would give them all the spreadsheets we would show which stuff we'd launched what feedback we got what changed in the company so again this is not for management this is for everybody to drive by try to have very visual progress indicators so ways that you can say you know if we want a certain outcome are we making incremental progress keep those very front of mind you can drop your burndown charts I don't need any of that stuff I want to see are we actually getting valid business results back as we go so how do we know we've succeeded we get to where we're going and so we hit the target outcomes and if we don't we very quickly adapt so we pivot at this point to use Lean Startup language so I get people saying well that's all well and good but does it really work so which sounds better our velocity is 36 point 46 points and we've delivered 100 new features or we just implemented one feature in 3 days which will save us 12 million over the next here what would you choose to hear we uh we did a pilot with a company recently and trained up one of our favorite product owners and they had an all hands meeting they had a big offsite with all the product managers all of them put up their PowerPoints saying we delivered these features on time and on budget of course and so it went and Jamie just got off and went yeah well we made x million and this is what we're doing and this is how we did it and all the managers just went oh that's the guy and he's you know going up in the company pretty rapidly what industry can you do this with we're currently building with building these huge automated um metrics Frameworks we not only measure products but also transformation areas so I've gone to companies where they have like a n test are you doing daily standups like a maturity model they're like oh my God why do you do daily stand-ups to communicate so let's measure communication saturation if we're looking at Cost reduction let's measure total cost of ownership for systems so we're putting all these measures in and not only at a big energy Trading Company um we're doing metrics video gaming and we're doing it for a big Voiceover IP Telco so this stuff I'm again huge adro person I've been doing it for a long time to me this is way way evolved right this is so much better than what I was doing before I mean yeah we were kind of doing it at the startup and then I got distracted trying to make agile better but actually this is is the game we should be playing so I won't to talk too much about how this changes everything I mentioned legal contracts um but let's go back CU I know we're almost out of time where we started why do you think I said agilistas are killing the planet that was a pretty strong statement especially for a CST by the way come to my class on Thursday Friday I'm just kidding uh when you use the machine gun approach to develop your products you're basically building more features faster the more features you build the more resources you use so if you think about it every feature you build takes a certain amount of Machinery power Etc to use each time you have to maintain it the more servers that build all your crap and store it up the more beautiful feature Farms that you develop that's using a lot of power right so you're actually helping to kill the planet faster good on you so save the features save the planet um any questions yes yeah I was wondering measure how many money you're going to make on a ah great question question was uh you know I talked about this 12 million we can make how do you measure the money uh well again back to sort of the thinking of test things um we can get down to quite sophisticated Financial models because at the end of the day I look at it that we're asking for Investments and we're showing that we can create value back on that investment right each bit of money you give to development and investment so uh let's say usually I do it um to give a simplistic example I can't use that one cuz I get killed apparently in my NDA but um uh Amazon let's say that I go through Amazon it's taking two weeks to get books or maybe actually a very easy one is um if you look at the customer conversion funnel so from when they land on something like Amazon to when they actually get through and get the book where do we lose people so often you know in the early days people found that they'd get 40% drop off at certain pages you can look at them and say oh weird they get to the shipping page and then they disappear turns out all they wanted was the shipping information maybe they get through into the payment scheme and find that they can't actually use a certain payment type this is where we' start doing split testing to try the things so now we know that actually there's a lifetime Revenue amount that many companies can uh workout that for each person who converts there's an immediate money but also you turn them into a longer lifetime customer so you might have a revenue uh model where 50 bucks a year for each person that we have is a certain conversion statistic um so now I know um based on that for each one that completes that conversion that if they're worth $50 to me then say if I have a million people um we're up at 10% okay then you can go through and say for each percentage that's worth x amount of money to the company right so now if I know that hey when I've got to choose I'll give you an example let's say we found that there was a big drop off and conversions when people bounced if the page loaded slowly right so that causes a huge drop off I could look at three options to fix that one could be by looking at I say well our hostings based in England and a lot of the drop off happens for a lot of our users in Australia because the data center is so far away that causes a Slowdown it could be Network latency um maybe we could just cache some page elements there's a few options right this is where we're getting into options now I can start costing it out I know that if we improve that we could potentially make x amount of money um which one of these options do I want to play out so I'd probably start with caching the page elements I'd look at maybe some Network improvements what is it but there's a point where I know know that maybe the data center is too expensive for the amount of value I potentially get back this is where you get into fine testing I don't have time to go through all that is that okay yeah wow you look puzzled still good come to one of my other classes about stuff yeah yeah there is and actually it's a good point so the question was how much is is it do you measure too much and again think back to investment what's the value um we actually are very clear about trying to measure very few things to start with so we might only take three simple measures uh if you measure too much it's confusing also this is like you know if I launch 100 features all at once this is the problem with big batch product releases how do I know which of the killer features within it right by doing that we've taken away our ability to understand what's going on same with metrics you've got to be very careful because it's a system each thing you change can change five other things so actually you want to be very specific about what you measure and when for example if I'm a startup I'm going to be looking at verality how quickly does my product get taken out there stickiness do people come back to it um I'm probably not going to be looking at Revenue so much to begin with because if no one uses it who cares right then I'll bring in other measures as we go so you do have to know what you're doing it's not a simple thing to get going but figure out if this is our outcome what are the things to measure at this point okay all right so you say about that maybe a b split testing and stuff like that can help when you release products Etc so what I'm thinking is what about the marketing side they are doing stuff as well so how do I make sure that they are not affecting the same stuff I'm creating do you understand what I mean yeah I might have two answers for that about marketing one is involve them I get them involved in all of the product Discovery thinking because they can work at odds for example if the game is to get someone to buy a book marketing goes and puts 200 packages up that might get the marketing income up but doesn't actually help the user experience in fact it can detract from it that's a problem and then the other part of the question was Remy I got 2:00 in the morning iot what was it oh he can't remember either who can remember what was it what was it about marketing there was a really important thing there I totally forgot oh oh oh oh causation this is a big thing so one of the nanas is that we'll have an outcome based contract we've actually got three one-page contracts that I've developed with the UK lawyer called minimal viable contracts how original and uh one of them is a pure outcome based model that you only get paid if you deliver value and if you deliver it early you can get huge upside all the Consultants can you imagine a world I save that company 12 million in 3 days if I got 10% of that how cool would that be right so now you get into quation ah but we just launched 200 other things how do we know that what you've done has impacted the results this is where you need to measure at different levels so remember I talked about the model being fractal so we have Target outcomes and options underneath it for example you could say we want to increase Revenue that's a huge outcome for the company like Amazon I might be at the level where I know if I'm in the backend server area if I can um keep the system make it more available we go from 96% availability to 99 then we actually keep more people on the site I can show that that has an alignment to the higher level outcome right so what you got to be able to do is agree that this is a level that you're comfortable with you want to keep linking it you should see some variation in the revenue but just as we could do something that improves that bottom line someone else could do something that detracts by it so we have to agree that our our product level this is the right level to measure it there's a little bit of an active Faith there but if you do enough testing and can get feedback again this gets into how you do split testing Etc we can show that there is a link this is the hardest part about writing the contracts is to get that linkage correctly done so that we get the suppliers saying okay we're going to do this that actually helps the greater level outcome any other questions other questions okay thank you [Applause] everybody

Channel: GOTO Conferences

Views: 179,541

Keywords: Software Development, GOTO aarhus, agile, scrum, Agile Software Development (Industry), Gabrielle Benefield, Videos for Developers, GOTO, GOTOcon, GOTO Conference

Id: 2JNXx8VdbAE

Length: 46min 30sec (2790 seconds)

Published: Wed Apr 03 2013