Scaling Agile @ Spotify with Henrik Kniberg

right nice to be here why you guys so far away Sweden's like a long way to travel but luckily I came a week ago so I've had time to kind of get D jet-lagged um I don't know about you but I tend to learn best from concrete examples so that's what this is all about give a concrete example of how one company works it's interesting as I go to different places around the world and meet different outdoor communities the the big question is always scaling how do we scale scaly scaly I like the phrase don't scale agile D scale your organization if you think of it doesn't D scale an organization D scale your project scaling it and cultures is the other one that keeps coming up I think Spotify is an interesting example of both another thing that I find interesting about Spotify spot if I never went through an agile transitioning because they can achieve it they were born into this around 2006 so kind of agile DNA was planted in from the very beginning and and what is and what is that resulted in now when we're about 1,500 people and I've been lucky to be able to be on pretty much most of that journey in one way or another and it's very very interesting so I'm going to tell you yeah basically tell you where we are now and some of our challenges and just a little bit of context how many of you use this product or have tried it just understand its most ok that's most right um what what it basically attempts to do is make these ways of listening to music obsolete right bits of plastic or the notion of buying and owning songs or the notion of pirate coughing's trying to find a better way of doing that what I find is interesting is it looks like it's just a music streaming product but but in if in fact the company is really trying to change the whole music industry turn it upside down and doing so doing quite well with that so far so it's from a product perspective it's very interesting story as well although in this talk I'm going to focus more on the kind of how we work as a company yeah it's been going very fast so we we like a lot of what we what we've been doing has been triggered by growth pain right no matter what we do it tends to break after you double in size and then it breaks again when they double in size again so it gets kind of crazy now we've slowed down a little bit growth but generally most of what we do is oriented around how do we deal with all these people right right now here's how it looks like most people in Stockholm mainly because that's where the company was born and but we got development spread across about five cities us in Sweden and one interesting thing in our journey was we were pretty much a scrum company for for at least a couple years I would say everything with scrum everywhere and what we found was that that was great kind of to create some semblance of order in a chaos but then after a while we started noticing that some of this in shoe level like like specific scrum practices we're actually getting a little bit in the way so we decided to relax a little bit on the practices and instead focus on the principles so we associate ups having scrum teams that that always do sprints we said how about we're agile teams we call them squads that do whatever flavor of agile they like they may or may not do sprints etc and as part of that we renamed the scrum master role to two agile coach because we want somebody who doesn't sound like a master because that kind of implies that there are slaves and instead we want to emphasize a certain leadership part to want to use word coach said so in effect it's a similar type of role but we just call it a jock coach and we don't we don't prescribe sprints specifically right so right so what is this thing called us called a squad it is really the equivalent of a scrum team that doesn't necessarily do Sprint's right so yes end-to-end responsibility cross-functional self-organizing pretty much your canonical scrum team they all own some aspect of the product such as search or playlists management very very end to end very vertical some of them are are internally facing some of them build tools for other squads etc but the big thing the big driving force is is what we call autonomy which is interested cuz that's kind of built into scrums that's it's it's it's no coincidence but we use autonomy to guide the whole or the whole evolution of the company pretty much and autonomy trust means not only deciding how we do our work but also deciding what are we going to work on next as a team right and the reason why that matters so much to us is because we've found that this is a this enables us to move fast because girls don't have to get stuck waiting for decision I think they can always move somewhere and it helps to scale because they don't need as much if squads are fundamentally self-directing we don't need as much central functions so yes we do have managers yes we have coordination roles and things like that but not as much as you otherwise would need when being this many people and it's fun it's math it's motivating and maybe that's the most important so I have this little formula that I use it's very unscientific I just I discovered just like a month ago that I've been using this formula in my head unconsciously um productivity equals right you ready for this got your notepads right you calculators right do this math effort because you actually got to put in some work you know sorry but you have to work to get stuff done times competence because it kind of got to know a little bit about what you're doing and environment the team needs access to tools and just support things like that and guess what it's not enough right what's last one hint I told it to your two minutes ago yeah what motivation because if you have all these three things but you don't know that one then yeah sorry not much is going to happen you're not going to get great products at least so that's why that one's squared this is the formula got it okay you can go ahead and apply it um so this is what matters the most if you have a very highly motivated team like these guys who are hell-bent on solving a problem struggling with right now but you really want to solve it they'll fix the environment they'll get a hold of the competence they need they'll put in the hours they need they'll solve it if they're motivated and if they're not then the other stuff doesn't really help so we really focus a lot on anything that will raise the motivation of the team right but we still have this problem okay so 70 70 squads how do we make sure they're not just all running in different directions right that's where alignment comes in and I used to think of it like a trade-off scale like you have to do you have to take your pick where do I want to be along this scale do I want to be on that side which is a manager decides what people do I do what I say right or do I want to be close to that side which is a team's decide and the manager says do whatever and they team's sign right and have to pick but I found that this is actually false that false dichotomy it's more useful to think of it as two dimensions right and don't worry this is the only you know consultant type 2x2 I'm going to put up on this presentation but this one actually is quite useful as you know all two-by-twos have a good corner and three bad corners right so we'll start in the one of the bad corners a low alignment low autonomy that means people don't know why they're going to work rather than getting their salary right go to work and you do what the manager says and don't ask him any questions right so yeah that's that's kind of micromanagement culture not fun hardline means high alignment low autonomy means managers actually are good at you know painting the big picture describing what's the purpose of what we're doing so yeah we need to cross this river because of whatever right but they're also telling people how to solve it so we need to cross this river so build a bridge right high lineman still low autonomy a little bit better but this is the this is the good corner hey that's the good corner at least the corner we like High Line and high autonomy is the managers leaders are good at at describing the problem that needs to be solved we need to cross the river but they're also letting teams figure out how to do it they're not telling the team's how to do it we need to cross the river figure out how so for example how we might you what are some other ways you're getting across this river all right a bike does someone say a bike yeah you know I haven't had my morning coffee yet right yeah there's lots of ways right you can you can swim you can build a tunnel you can build a ferry can reroute the river you can move the village to both the same side of the river or I don't know write lots of crazy ideas so let people figure out the best solution now you get innovation now you get collaboration you get all kinds of good stuff and down on the right side we have this other not-so-good corner which is low alignment high autonomy that's the kind of 70 mini startups all running in different directions and managers are like I hope someone is working on the river problem but I have no idea so what what we find is that a high lineman enables high autonomy if you're good at aligned if you're good at really paintings inspiring vision or where we're going and why it's important to this company you don't need to control people as much you can they'll tend to autonomously help help you get there right different teams will do different things but they're all aligned towards same goal that makes sense it's it's a very powerful concept idea that high alignment enables high autonomy it doesn't conflict with autonomy so this means that in practice a leaders job this is the CEO Daniel Lake is is to explain what problem needs to be solved and why at a high level and then the squad's jobs collaborate with each other to find the best solution this is also an anti anti siloing strategy right if squaws see some bigger picture they may be less likely to optimize just for their own spot which otherwise known as some optimizing right and the opposites is physically optimized to kind of invite cross squad collaboration so there's lots of inviting you know couches and spots between the squads we can just go sit down and chat with somebody so you don't really need to go and you know book a meeting and find a room and things like that um there's of course a bit of that but a lot of lots of informal spaces are helps a lot so in general although you do need some kind of structure in a bigger organization we find that what really matters is is community so I'll talk more about that in a moment but first a little bit about the structure we do have some of you probably read some of the articles videos I've been putting up on this I'm not really sure why we make up these weird words there's probably some good reason for it but I forget what that reason was but we we call a tribe is a bucket of squads that have some kind of shared purpose and sit near each other so for example a tribe over here they they focus on the music playing experience itself how do we make a good music player while that tribe over there they make infrastructure from four other squads etc so we got about what I think it's seven or eight or nine tribes now and inside each tribe if you zoom and you find out matrix matrix models are very old they've been around for a long time what's a little different about this one is it it's a matrix that's focused very much on delivery so the primary dimension is the squad which is the vertical that's what people sit physically grouped together with their product owner and their so a squad typically includes a product owner a back-end developer front-end developer a designer a QA which is quality assistance and TA which is test automation agile coach very very wide depending on what to do of course a chapter is the horizontal dimension that's the secondary dimension that's competency areas so we could have a back in chapter all the back end developers are in the same chapter and here's a front-end chapter and there's a agile coach chapter so they communicate along that dimension that that's mostly about personal growth and about developing my competency within that within that area so yeah each tribe is this little matrix thing and sometimes you want to communicate across the whole company on some items on some areas such as quality or leadership or whatnot so we have this notion of a guild which is a lightweight community of interest for example the leadership killed anybody interest in leadership doesn't matter what role you have you're interested in this topic you can join that guild and we'll have conferences like this and we'll have a mailing lists and stuff like that so these are the kind of the the this is the glossary of terms we use when talking about organization but we're not very very strict about it I guess the main you know first order thing here is squats right squads are very much real like you sit with your squad that's your day-to-day work everything else tribes chapters guilds are a little bit secondary it's the squad which is a primary thing and reality is messy it's not as nice and pretty as it looks like in a slide right so here's a real version tribe lead this was quite a long time ago asked them so how does your tribe actually look like right now and it took out a piece of paper I need that from his head you just T just it through this picture and as you can see it's not a whole lot of straight lines right it's not symmetrical and it changes changes quite fast and that's okay we don't try to make this too stable instead we allow things to change because basically basic idea is if you need to talk to somebody about something to solve a problem just go do it doesn't matter what chapter or squad different people are in just go talk to the people you need to talk to and if we notice that a lot of people need to talk to talk to each other very often then maybe that group should be a squad so we form a squad out of that right so yeah we try to form the organization based on what communication needs to happen and not vice versa right and one kind of key thing to make this whole thing work is small frequent releases and here's a piece of problem I see in men in many companies especially large companies when you have a long release cycles let's say this is half a year right when you have long release cycle then each release is like a project releases drama it's something scary you need like roles and you need you know maybe a release manager and documents and artifacts and meetings and stuff right because it's scary and when things go wrong they tend to well first of their likely to go wrong because half a year a lot of stuff happens in half a year right and also I got to come together plus when things go wrong it takes time to fix and there's risk involved etc so this is the equivalent of 50 at a train station and the train leaves every hour now you want to make sure you arrive in time you want to check the time schedule and becomes a bit of a project to get on the right train and if you miss it you get very frustrated but if the train leaves very often then you don't need to look at the time schedule right you just go and the trains there you get on it if you miss it again on the next one right it's not a big deal but anyway yeah so big release cycles tend to cause trouble and there's a vicious cycle here too this is the kind of kicker because if you really seldom the release is big right because a lot of stuff happens and if the release is big it tends to be hard and if the release is hard we're going to tend to not want to release too often right oops yeah so anybody recognize this hand up if you feel like this is kind of us we're in this vicious cycle yeah it's okay you're you're among friends here right we won't tell anyone it's okay thanks it's hard to get out of but once you do get out of it it's very nice because then you get this other you know everything becomes opposite releases routine releases boring right you don't talk about how everybody's because it happens all the time it's just a mostly automated release often means the release is small because you know if you release every week what's happened since last week not a whole lot so if things go wrong we can probably fix it and if we can't we can just roll back it's not a big deal right so it takes a lot of effort to get from the top to the bottom and once you're down at the bottom on it's it's quite easy to stay inside it because yeah release is simple so you can keep releasing often so we put a lot of effort into this and here's an example of when things started going south for a while as we grew it became harder and harder to release the actual client the actual core product the music player itself on the desktop it's interesting it Spotify we try to maintain this illusion to the users that this is just a music player but actually behind it's like this complete ecosystem for music almost like a global operating system for music but from the customer view is just that's this extreme product right so getting this thing out the door was started getting more complicated as we had more and more people involved everybody was you know spraying in code right and what's the chance that this thing is stable at any given moment in time well zero right so is somebody - working on something so it got really hard but what I really liked was and this where culture comes in instead of saying okay release is probably better add a lots of roles and structure and process around them and said let's make release not hard right what's making release hard let's let's change the underlying reasons for why this is hard well it's because there's arced architecture so we change architecture and now it's more like we use a something called chromium embedded framework which basically means this product is actually like web browser it just doesn't look like one so the each part is like a frame and squads can put their own self into production without having to synchronize with neighbor squads really nice they can just release as often as often as you like sometimes we need to release the whole container itself so but but things got a lot simpler and this is in the past say one and a half years we put a lot of effort into this notion of taking it even further and we've tried to implement this thing called continuous delivery and I'll see a lot of teams around the world on the same path right and the basic idea banking is delivery for those who aren't familiar is is that every commit is a release candidate every time someone commits that's you get a new release candidate that's the vision right it doesn't mean you're going to release everything but it means that you can it's just a business decision not a tactical decision and the guiding question is is a how long does it take to deploy a change that just involves one single line of code if you ask yourself that question and answer ideally should be just minutes than you're getting close right if it's hours or days or weeks or months or years then you know I've got some work to do right and can we do this on a repeatable reliable basis right once again this is about making release routine rather than making release drama so when teams manage to do this their reality looks something like this I push some code and it lots of stuff so it's happening automatically well I you know keep working or go have some coffee it gets built automatically tested and integrated it gets deployed to a production like environment all automatically then then it's got any release candidates sitting right there right and I can do manual tests on it so we want to automate as much test as possible but you can't automate all testing so whatever's left is manual testing and so yeah now I can go do some manual testing and if I'm happy with this I can put it into production which is just one click that's the vision um we're far from this today we have some squads that have done really well as some that are way behind but this is what we're focusing on all the time it's driving a lot of organizational change architectural change tools etc but I'll just give an example of one one squad actually has nailed this really nicely and their vision is kind of like deployment should be so easy that even an agile coach can do it right and I think they manage because here's what he built look at that it's just a just a little joystick and a button that that's all so let's put something into production let's deploy the metadata service right let's put that production where do you put in production or shared environment I'm sure you are all right and which version we want to deploy the latest one or that one over there am I feeling lucky yeah all right let's go right it's going it's going it's going it's going it's going and done there I like that turn it at the game right yeah so uh of course there are some things we don't want to release to offense for example you know inside here is a Spotify client and every time that gets updated I get up you know it tells me there's new version of Spotify and that's a bit annoying right if that happens every say five minutes right so um like I'm in the middle of playing a song it's like a new version has come oh okay update it and then you know one song later a new version has come like what the hell so yeah we need to batch do a bit of action there so we settled on release training frequency and decided on three weeks now I think we're shrinking it's two weeks and the idea there is just like the like I told you about that the the train metaphors train leaves every five minutes right in this case the train leaves every three weeks and the Train always leaves so whatever is whoever's on the train gets to go whoever miss the train gets to take the next train there's no waiting right so suppose we have three squads here that are all building stuff that's going to impact this client right feature a is done feature B is not HB is done feature C is done but it's always going to be something that's in progress right so feature D is in progress what happens with that one well it actually gets released as well so everything gets released even the stuff that's not finished and all the squad members know that so they're kind of careful about how they commit things and they hide things behind feature flags that means that in practice there's lots of unfinished features inside this client right now but they're hidden from me using feature flags so I can actually go in and turn on and off features and you can do that centrally as well and turn on turn on and off features for certain population groups etc it's it's a great tool to make sure that we're releasing often rather than letting code sit around on branches right but it's also a useful testing tool in general like what happens veteran this app does it how will people react right so feature flags and release trains is very powerful so we experiment a little bit with how long the release trains are we don't need release trains for everything but it certainly helps to make sure that stuff is always going out the door right what one thing we learned also was that even though there's release going out every three weeks on for example Android we do want feedback faster than that so just recently we finally finish this really nice thing which is a nightly build for mobile it's it's a bit harder to do the nightly build thing on a web things it's easier but they actually managed to crack it using various tools so what happens now is I have two versions of Spotify on my phone I have you know the Spotify internal in them to the right and see it looks like a moon right like a moon with Spotify so that one is updated every night automatically just automatically gets a new version downloads in my phone so right now I'll pick it up and I can run the latest code which is very useful because that means that as a developer when I commit code I know that by tomorrow there's there's 1500 spot employees that are actually using my code a lot in their you know actual music listening habits so if something is broken I'm going to find out right and find out so yeah anything you can do to release more often and get early feedback is worth a lot but it takes time it's a lot of work right you need people teams dedicated to infrastructure and automation things like that um one cultural principle that has turned out to be very powerful for us is the idea of a self-service model and what that basically means there's no handoffs right you don't hand off something to someone else you take responsibility all the way yourself others may support you but you can't you know hand the stick too over to someone else and this principle has has it has caused us to end up with pretty much three flavors of squats we have feature squads that are like say the one on the top left there is search squad Dippold search features they do that on all platforms they own that that thing all the way to deployment on all platforms and maintenance and operations say they press the release button and they they're the ones who get woken up the middle of the night if things break with search etc so they really own this thing kind of like a mini startup but to make that feasible because it's almost impossible to have a squad that has enough breadth of skills to be able to fill around with iOS and Android and web and desktop and various API is you have like a you know thirty person squad right so to make that feasible we've evolved this notion of a client app squads where each client app squad owns some platform so we have iOS squad the Android squad etc on that so the iOS squad their job isn't to put your stuff in production on the iPhone right their job is to enable you to do that yourself but they platform eyes that it's like they pave the road they build the road that you as a feature squad can drive on to get your stuff into production does that make sense all right so they platform eyes that so that way this feature squad can put code into production on an iPhone without having to muck about with too much plumbing on the iPhone or iOS and we also have other infrastructure squad squads the build things like monitoring services continuous delivery tools and squads that work with more operations like things right and but once again they don't put stuff into production for you you do it yourself they enable you to do it yourself so I like them I like the metaphor of a buffet right you're going to have if I were if I were to serve you all lunch right the most scalable way to do that is not to have a bunch of waiters going to your tables and taking orders I would need a lot of people a lot of coordination instead it most scalable way it's just to put up a big buffet table and have you come and serve yourselves right but that's based on trust that only works if I trust you to not misuse that right trusting you not to come up and just take the best parts and put in your backpack and go home right so yeah self-service model is is a fundamentally good trick for scaling but it is based on trust generally speaking I find that trust is more ism is more powerful than control in fact you indirectly get control through trust which is interesting I like this example I'm not an expert on traffic systems but I read up a little bit up on this particular thing about different ways of designing crossings very interesting turns out that this is a four-way crossing with with traffic lights right signal signaling system this is a roundabout in there in the traffic signals version it's central control right the lights determine who gets to go so if there's an accident you're more likely to know whose fault it was right who ran the light right and is very clear rules in a roundabout it based on trust and self-organization you need to open your eyes you need to look around right there's no lights telling you when to go when to stop there's some basic principles but that's it so this works because we have a shared goal of you know surviving this crossing and getting to the side right everybody wants people to be able to get through right nobody wants us to just get blocked and if people start crashing that's lose-lose so to the left we got 32 conflict points which to me reads as 32 ways things can go wrong right to the right we have just eight conflict points not sure exactly what that means but it sounds like more as bad and research indicates you get faster flow and roundabouts in most cases you get significantly faster flow through this and you get a lot fewer accidents there are probably cases where it makes sense to build crossings but it seems like in general we tend to build crossings too often when we should be building roundabouts crossings give dis tonight.this this false sense of control right this false false security the hayward control that actually makes things worse in general around abouts yeah you have to trust people so yeah here's a famous racecar driver Mario Andretti if everything is under control you're going too slow and that's a useful and if you think about it right to think of a really big complex project and if you want control as a project manager you know what's how can you get control we ask everybody to do nothing right ask everybody to stay home in bed and don't touch a phone or a computer now you're in control right now you're in perfect control and as soon as anybody does anything now you start to lose control so this notion of taking control is a little bit you know counterproductive so here's that what I want to really coach managers in an agile organization is trust support people don't try to control them assume that people want to act in the best interest of the company I know it's hard right assume that people want to act in the best interest of the company and guess what the chance that they will increases in that case it's a bit of a self-fulfilling because people tend to want to live up to to the trust again and but in order to act in the best interest of the company people need some support and that is autonomy they need to actually be able to act at all right without asking for permission they need transparency that's where the whole scrum values come in to inspect and adapt the transparency they need to go see what's going on right what are we doing what's the vision was where we're going what are the customers think of my latest release some yet a whole lot of transparency around priorities and things like that I need a short feedback loop because as this this team does things they need to find out if whatever they're doing is is the right thing or the wrong thing they needs we're finding out so releasing often is one way of doing that promoting face to face conversation is another way anything you can do to shorten that feedback loop these are the management actions in company it's not controlling people it is putting in place these things so that self-organization works you are enabling self-organization right so manage for the normal treat exceptions as exceptional yes there will be some people who aren't acting in the best interests of the company there are going to be bozos sometimes right but treat those as exceptions treat those on a case-by-case basis don't organize a company based on those people right so what I noticed companies like this the mindset of management is less around what are you doing what are you going to be done the types of things to talk about are more like what what what do you need what support you need how can I help you it's kind of servant leadership in a nutshell so this is this seems to be embedded inside the culture at Spotify this is what they consider obvious if I were to show this slide they'd be like why are you saying this this is does this is obvious right so um of course people screw up sometimes people make mistakes and you need to let people do that because the people are punished for making mistakes guess what's going to happen if people get a hide the mistakes next time right they're going to hide them and that that's that's not good right so you want to say it to be visible and therefore you need to accept it people make mistakes so here's a good quote from from Daniel Ek the CEO of Spotify founder we aim to make mistakes faster than anyone else it's like that's interesting but he really means that he says it quite quite often and what that means is people are going to make mistakes anyway because we're humans right darn humans make mistakes so let's get it over with just you know make the mistakes if we're going to make mistakes anyway let's make them fast so we can learn fast and improve fast um so in practice what this means is we need to focus more on failure recovery than failure avoidance and here's another useful metaphor for anyone has kids right who has kids here's yeah that's most to you well this community is growing up right I have four small kids and here's the thing if you want to keep your kids safe yeah lock them into the crib right safe right so my kids are between age of 4 and 11 and they're still in the crib they're very safe nothing's happened just kidding yeah this is how you keep a kid safe but the kid won't learn very fast won't develop very fast won't be very happy right let the baby run around and guess what happens well they fall over sometimes they get hurt right but this kid is going to be happier a kid is going to learn faster it's gonna develop faster and you know babies heal fast right they heal pretty fast just got to make sure that it's not lethal right so evidence of this inside cultures like Spotify and other companies that have this failure friendly culture you see things like this is our internal blog people are putting up their failure stories right they're displaying them proudly here's exactly how we shot ourselves in the foot right and what we learned from it and things like the fail wall I people put up sticky notes for failures like hey look I screwed up this thing yesterday and here's what I learned right so failure is considered yeah something you do you display proudly okay but of course we don't want to just let it pass we when I pick up the learnings as well otherwise we'll just fail again so we want to learn from failures how to do that while using standard agile techniques like retrospectives but also at a little bit higher level if something goes wrong we want to do it postmortem something went wrong a crash happens we get all the people who know anything about it into a room and talk about so what happened and what did we learn but we don't talk about as whose fault it was because again if you start looking at you know I think it was his fault right then yeah people are going to start hiding their failures right so we don't care whose fault it was we care about what what did we learn what happened what did we learn what will we change so this is very much a cultural thing all over the place there is retrospectives and post-mortems and various things like that happening um often good often guided by it by guided by it by the coaches we have about 30 or 25 or so agile coaches what they bring to the table is these skills right skills such as well how scrum works but also other agile methods and how to run a retrospective things like that all these little coaching voodoo things right now another thing about failures it has to be non-lethal so we are architectures is optimized around something we call limited blast radius which means if something going to blow up it's like hopefully not going to blow up the whole system so behind the scenes it's actually about either up to about 300 different systems at our fairly autonomous running them separately the chance you're going to bring them all down is quite small unless you really screw up it does happen but very rarely instead when you screw up and normally it just impacts one part of the system for example the album art is looking kind of wacky right now what or maybe this little notification thing up there is not showing up for example so normally that the blast radius is limited and since whenever we do something a little bit scary we roll out just to like 1% of all users that's still like 200,000 people but it's still just 1% right so if we screw up its not going to screw up for everybody and the effect of this is that when a squad makes a mistake which happens from time to time it's normally only going to impact a small number of users and only a small part of the product and only for a short period of time because of the no handoffs principle right we own this feature if it breaks that's embarrassing for us there's nobody else to blame this doesn't it's our problem we have to fix it right it's embarrassing so things tend to get fixed quite quickly and what's interesting about this it's a bit counterintuitive because I might look like wait a sec you're letting people make mistakes and in production what does not make the quality worse but I find that no it makes the quality better because now squads can learn faster they can try things out and get faster feedback so they can improve the product more quickly and as a result we our product today that is viral most people love it and they tell their friends about and it spreads very quickly any other will be a bug here and there sometimes but most people don't even notice it it gets fixed quite quickly and then we we move on right in contrast if we were very paranoid about bugs in the beginning we'd probably have a product that was you know maybe zero bugs today but it'd be a pretty bad product with zero bugs right and it's not yeah we wouldn't win this match by just having the least bugs in general it seems to be more about the winner in a very competitive industry like this is the fastest learner so that's our strategy make mistakes faster than anybody else right um when it comes to building an actual product we're very inspired by this notion of lean startup there's going to be at least one talk today on that we have our own twist on we call it think it build it ship it tweak it which is like our framework for small things yeah just just build a release right squads just put stuff into production we don't add too much process but when there's something bigger that's going to imply forming new squads and maybe hairy people then we need to be a bit more careful so then we have this notion of a thicket phase that's when we formulate the problem what problem are we trying to solve right so let's say the problem is people who don't know what to listen to can't use our product because you have to type the name of exactly what you want to listen to right so people who just want to wash the dishes they just want some background music how they're going to use our product so we verify that that's a problem using research we have user researchers that you know talk to real users through focus groups etc to verify that this actually is a problem it's not just something we think is a problem that's part of it the other part of it is defining a press release to write the press release before if this product was done today how are the press release look like we call that the narrative what's the story how would we show off this product if we can't come up with a compelling narrative then don't build don't build the product right and then metrics how will we know it's exceeded how do we know that this if this product was done today how do we know that it was a success or a failure so we look at things like monthly active users daily active users various forms of engagement how much music do people listen to how much do they share a lot of stuff can be measured and we can we look at so if this product was done what we think will happen let me set up hypotheses for that that's measurable right which is quite typical Lean Startup technique and then I build a bunch of prototypes low-five prototypes paper prototypes mock-ups etc to get a sense of what this might look like when it's done so this is our thinking phase right we're trying to figure out the constraints of this thing we're building it's not a requirement spec there's no list of features there's no promise dates it's not that it's just understanding the problem right and then once we feel that this is actually worth building right then typically the people who did the think it work will also build it the same Squam typically just continue again try not to have handoffs right so what to do now is they work in it typically in sprints or something like that and they iteratively build them MVP the Minimum Viable Product which to us kind of maps to the minimum lovable product right the smallest thing we can put in production without feeling very embarrassed you might say it's narrative complete not feature complete so something we can put in production that will fulfill the promise of radio you can save but yeah there's still lots of stuff to do and so we put that introduction for just a few percent of users the squad makes that call themselves but they put it up and they start measuring and comparing right so they have these dashboards they look at how are these 1% acting compared to these other 99% and based on what they see they'll start tweaking the feature maybe the feature isn't performing as well as they were hoping so they start tweaking and they release again and they're releasing over and over and over again sometimes several times per day and looking at the data to find ways to achieve the desired impact so it's all about the impact and not the output right so all my impact and and at the same time they're sorting out operational things like you know the how much load is this on a server do we need more memory or do need to optimize something so they're learning both how how to build the right product but also how to build a product right right and then as I gradually roll out to more and more people and finally when it's all fully rolled out yeah that's ship it so ship it starts when when when the 1% have the product and it ends when hundreds end of the world has the product okay then what all they were in tweak it right the product is released everybody in the world but it's by no means done we can still keep improving it so this quad will keep owning it keep tweaking it keep keep improving it until it gets chopped down pretty much so this is our high level process we're not very strict about it we don't have like strict rolls and hand off the toll gates but this is the kind of mindset any given major initiative we have we can pretty much point and say this one isn't builded this one is and think it right this one is in tweak it so experimenting with various forms our portfolio level dashboards to visualize this so the general mindset it's maximize value and then not output so think of this person loading rocks right we're not trying to load as many rocks as possible that's not the goal the goal is to slow down a little bit and look at what what are we loading some of these are bricks and some are gold right so don't just write a lot as much as possible load thick load the good stuff right we get less stuff loaded maybe but we get better stuff loaded and that's what's going to win in the end of the day one of the things about that industry since Spotify kind of created a new industry this music streaming industry managed to convince labels and artists that it's a good idea to actually let people stream your music and there's a way to earn money from it we notice that that industry is all about innovation and then the competitors are companies like Google and Apple and like a hundred different startups it's getting pretty yep right and basically we think innovation is the thing is the key and not predictability and these standard odds to each other because if you want a hundred percent predictability the only way to do that is if you do something you've done before exactly right only then do you have predictability if you're doing innovation that means you're by definition trying exploring new territory so predictability has to be sacrificed to some extent so on this scale yes sometimes we're on that side a little bit because we have a partner integration project or something and there's a deadline involved that we do things like standard scrum release planning techniques right there's like burn up charts and velocity but that's really the exception the norm is more like we're on on that side focusing more on laying people experiment and innovate and not caring too much about plan fulfillment right so what one example of this is something that we call Spotify hack week it twice per year we had one just a couple of weeks actually couple weeks ago this is really interesting on it's a whole week of everybody gets to do whatever the hell they want right a lot of the companies have different variants of this but this is our variant it's a whole week of build whatever with whoever in whatever way you want originally it was just a tech people when I was actually the whole company so it's even sales and marketing and everybody right and yeah hack the product hack the company hack the office hack the process hack yourself hack your desk hack stuff right and it's it's amazing it's really it really blows us away every time whether or not of a creativity people have um and there's a big demo on a party on a Friday and we you know look at what what came out of this sometimes they're like really useful new products come out some are even ready for production is just that one week which is like whoa some are more like a proof of concept and they later on spawn into actual real projects and some are just crazy ideas that we don't know if they're going to turn into something or not such as this one yeah it's really fun you imagine that time is hundred to get a hundred crazy things it's like whoa and but but maybe one of the most important things is new connections people who don't normally work together will start working together so you build new connections across the company really interesting plus it also reinforces the message that listen everybody we're here to innovate right we're here to try things out and act we kind of it's a demonstration of that um since we're a big big an innovation a lot of what we do is based on experiments and not so much an argument because you can sit and argue about whether should we do it this way or that way and you know you can argue it to death and we still don't know who's right so it's better to just try it out so generally our cultural default is did instead of arguing too long about a or B just try both and compare right sometimes it's about the product should we organise the layout like that or like that we'll do both will hey be tests sometimes it's about should we should be do we need sprint planning meetings or not I don't know let's try a month without sprint planning meetings and a month with and then see how what it was like so that means by definition we allow things to vary from place to place so some squads are doing very much scrum by the book and some are doing like more like on done like some are doing I don't know what all right it varies some but what this does it gives us more decisions that are data-driven not all decisions but it gives us more decisions that are data-driven and less decisions that are more like ego driven or or opinion driven or Authority driven right I don't know hooking up with that term hippo the highest paid person in the room the highest paid person's opinion the hippo right people make some decisions and we normally don't do that so here in the hallway conversations you hear quite a lot of things like what's the hypothesis people are talking about that well they talk about things like what did we learn right what is it data say what are we going to try next but a really important thing for this to work in practice is you also need a waste repellent culture because as you try lots of things some will inevitably not work right you can't keep the things that don't work right because you get it to mess so what I really like is that the very waste waste repellent if something if you try something and didn't work will throw it away you know without remorse without blame just okay we try this they didn't work let's stop doing it so if it works keep it otherwise dump it and so something that I've turned out to be keepers are retrospectives great stuff daily stand-ups and scrum boards or things like that very big on that not because we standardize on it but because people have tried it long enough and we realize that this actually works Google Drive or talk document sharing a git for version control and what we call guild on conferences which is like an open space like thing for Gil's just some examples and things that are more like skip slash dump or time reports handoffs of any kind separate test teams or test phases task estimates estimating a task level I don't think it I don't think any squads do that anymore any and any meetings are perceived as useless for whatever reason right very quick to just stop doing them as part of this we try to reinforce a message that there is no right or wrong it's just trade-offs for the most part so we have two big team okay if we have two big team we might split it but if we go overboard with splitting we suddenly have too many teens right and that's another category of problems or like to top down or to bottom up we don't pick your poison right to short planning horizon to long planning horizon if you don't like a deep hierarchy maybe like local let's make it flatter hierarchy but now he has no I have too many direct reports as a manager darn right you know one problem gives birth to the next or solving one problem will give birth to the next problem too much standardisation too much variation or too much too fast change versus too slow change in general I find that it's not really possible to find the right spot to be at but it is possible to get a sense of are way too far to the right at you're far to the left what are the symptoms of being too far here or too far there and which symptoms are we seeing most of and as a result you might conclude that okay we have too much variation we should do a little bit more standardization right for example now I know that if you read my articles or talks about this it might come across as modifies is some kind of a jewel near Varna where everything just works and that's just not true we have a bunch of problems and are always struggling with things um and I thought give you a long list but but due to lack of time I won't give you that specifics but generally there's a lot of pain with a lot of challenges all the time and mostly related to growth pain but the interesting thing is I find that quite often when I'm this is kind of job to be looking for these things and help them sort it out quite often when I'm tracking the root cause of a problem and I turn the corner and I find out that there's people working on it already that happens in most cases somebody already working on that problem and making pretty good progress as well so what I find is that yeah healthy culture heals broken process process breaks into the left and right but generally speaking problems are seldom long-lived they normally get fixed by somebody and yeah I think that's because of the culture there's a culture of yeah this is a problem let's fix it let's not whine about it let's just go talk to the people that we need to talk to and fix it right and if maybe today's solution causes new problems next year then okay then we'll fix it again in a different way we find ourselves always trying to straddle this balance between chaos and burocracy so as the company has been growing get more and more people we risk chaos right absolute chaos so we need to add a little bit of structure to compensate for that but if we go overboard with structure and process and roles then we're suddenly on this other side which is you know stuck on those spikes their bureaucracy like it turns out that my experience is is easier to to fix chaos and a fixed bureaucracy if you have too much chaos it's easier to add a little bit of structure to fix that if you're stuck in bureaucracy as harder it's harder to remove things right so that's why this guy up there you see is leaning to the left it prefers if I have to fall down somewhere I'll fall down to the left preferably and then climb up again right hold on to the right and I'm stuck right so so the question is what's the minimum viable bureaucracy that's kind of the guiding question right the minimum file the MVB right remember you heard it from me first what's the MVB when I go to some big companies they like that phrase and we it started triggering a whole series of workshops around yeah what is our Minimum Viable graphic our answer evaluating that and it's interesting so the engine behind this is of course continuous improvement right this stuff I talked all this stuff I've been talking about it's not really something we designed it we've evolved into it right it matches our context some of that stuff here might match your context so might not what do I know but the engine behind this is of course continuous improvement and the ways of doing that is very simple typical techniques like visualizing impediments right so as I walk around the hallways I see a lot of boards like this what are the top three impediments for this squad for this project for this whatever and what are we actually doing about it or fancier boards with to do doing done and lots of all kinds of boards I pretty much amount to this right what problems are we trying to solve and I really like this concept definition of awesome with using that more and more often it's basically like a squad sits down talks about in a perfect world what would that be like for this squad what is our definition of awesome or for this tribe or this project what is awesome look like in this case yeah don't break in production having complete set of skills in the squad really finishing stuff right a lot of you know this is not their reality right this is their vision they may or may not ever get there but it helps them decide with which direction they want to move in the basic thinking there is if you can't agree on what awesome looks like you're going to be less likely to get there right so start by defining what it is here's our definition of awesome architecture which we did a couple of years ago which was I can build test and ship my feature within a week and I use data to learn from it and my improved version is live week 2 like yeah that would be really cool is it realistic I don't know it doesn't matter it's just a direction to move towards what's funny is we made a lot of progress on this thing so now when I meet some squads and I show them this there they almost like they're like laughing like what did we did we think that was fast two years ago this seems kind of slow now right so some squads have gone way beyond this well some squads are still struggling to get to here so it varies but it's been very helpful to articulate what would also look like right we also experiment a lot with various forms of visualization this is what we call a health check model we haven't standardized on this but some tribes use it and it's basically a workshop technique where you get a squad into a room and I take a deck of cards and you should deck contains some aspect of Awesomeness okay such as easy to release and we have an example of awesome releasing it's simple pain safe painless and mostly automated and an example of horrible releasing is risky painful lots of many work it takes forever right here's an example of fun we love going to work and have great fun working together example of horrible boring right Scott so Lisa it really is fun delivering value learning beed how much support are we getting just like a 360 view on how how yeah how good is our our work environment and we do a workshop on that and the outcome is pretty much a chart like that for each squad green means this is awesome red means it's horrible yellow means somewhere in between right and the arrows are the trends is it getting better is it getting worse so this is a whole tribe you see any interesting patterns what do you see generally getting better almost every arrow is up except for like two so that's that's a good sign that continuous improvement engine is working right good any other patterns but look at the code bases seems to be in trouble here it's getting better for most but they're two squads here where's red and not getting better so it's like okay we have some systemic issue here any other patterns have you fun yeah look at that all green right and getting better maybe they're having too much fun right but we don't need to you know look at any kind of motivational schemes here they're having fun that that's not the issue here but we easy to release that's another problem right it's hard to get stuff into production here for some reason so there may be a systemic issue here if that problem persists maybe fun will get worse as well over time there's not fun if you can't get stuff into production so we'll look for systemic problems this actually led to a big push towards continuous delivery and code quality things like test automation by seeing that it's actually a systemic issue in the tribe we can start looking at the common root causes things like that it's also interesting to compare like squat to here is having a lot of trouble right scroll for appears to be not having any trouble at all but that might not necessarily be true right this doesn't give you the answers it gives you the questions to ask it turned out that squad 4 was actually mostly new hires and they were still in some kind of honeymoon period I'm not really done right so they hadn't noticed what was not working yet they were still blinded about what is working but yeah just it's used for visualizing problems look for systemic issues this is just one way of doing it but it's turned out to be quite helpful um so wrap up this is my big takeaway so far from working at Spotify for a number of years which is a culture you know beats process culture is more important and and culture to me is just stuff people stuff that people do without noticing it it's easier for me to notice because I'm part-time I'm there part-time at halftime about but I try to help help them notice it as well just create self organizational self awareness if you notice the good stuff and the bad stuff you can do something about the bad stuff and strengthen the good stuff right but I find that culture you can't you can't delegate that to somebody to own culture you can't point your fingers that you should change right it's kind of about the only way I know is this - that you are the culture the culture is the sum of everybody's behavior and attitudes right so start with yourself you're going to change the culture look at what can I change and how I act and sometimes that'll cause ripple effects so model the behavior you want to see that's that's my big takeaway from this and yeah I think I think that's it we're out of time and I'll be around for questions in open space or this afternoon or find me actually there's a coffee break now so if you have questions you can come up here we can have a chat right but thank you everybody

Channel: Scrum Australia

Views: 94,093

Rating: 4.9338479 out of 5

Keywords: Agile, Agile Software Development, Scrum, Scrum Australia, Regional Scrum Gathering Sydney, Scrum Alliance, Scrum Training, Learn Scrum, jtribe, love your apps, Best Mobile Apps Melbourne, Best Mobile Apps Developer Australia, jtribe Mobile Technology Solutions, jtribe Innovative Mobile Apps, Agile Tools, Agile Tips, Scrum Tools, Scrum Tips, Henrik Kniberg, Crisp, Spotify, Lean from the Trenches, Scrum and XP from the Trenches

Id: jyZEikKWhAU

Length: 53min 15sec (3195 seconds)

Published: Mon Mar 30 2015