Surge 2016 - Mike Bland - The Convergence of Wills

some of you folks might remember a few years ago in October 2013 healthcare.gov launched and it was crushed under the load so of course that's good news right people actually want this service they need this but health care reform was in danger of failing because of a website that seem right to you I mean think about all the talent and success in the US tech sector and this United States government couldn't get a website so a month later they finally got the sense to open the doors to industry experts that could come in and lead the recovery effort said that by the end of the following April not only had they enrolled their revised target of 6 million people or their original target of 7 but cumulatively they had eight million people enrolled in health care so this website recovery was a huge success thanks to a dramatic change in the government tech culture and it was a triumph for DevOps principles he had a small integrated team that could come in and turn around a troubled project that was years and hundreds of millions of dollars in the making it's because the technology actually was never the problem it was the contracting culture managing those waterfall waterfall style contracts took priority and people just kind of assumed oh yeah the software is going to be good it'll have quality so the key to the story here was that once the administration decided it going to fix the real problem instead of adhering to decade's worth of you know contracting and procurement conventional wisdom they actually fixed the real problem but the big question then is what's next how do you how did this happen in the first place so how can you keep it from happening again and how can the momentum from this recovery effort helped to reform development culture throughout the government and you know what am I doing here talking about this how did I get loop done so I had left Google and the tech industry in September 2011 I thought that's it I'm done I went back to music school I went to Berklee College of Music in Boston but then I got this curious email from old colleague through LinkedIn and he wrote me something that went like this as you may or may or may not be aware I help late last year with the healthcare.gov website rescue it's going to take a while before the government gets good at software development and testing and they're going to need a culture change the White House gets it now that fundamental change is needed and how they create and Tesla and more importantly that change imposed top-down is likely to fail again no pressure I respect your desire to focus on other things but there's this whole your country needs you think so I figured I'd just put that out there for you to think about and I don't think he realized excuse me grew up in a military town so he had just basically slammed my big red button I dropped out of school I joined the government in November 2014 and I just left this past March but to understand why Jason recruited me let's go back in time to when I joined google in 2005 so of course the company was already hugely successful everybody could see everybody wanted to work there and it was as famous for its stimulating work environment as it was where its products I mean it was like a big grad school like atmosphere and you know we had our twenty percent time to you know innovate and trying things and of course there was plenty of food so I think right after I joined I gained like 30 pounds off of chocolate-covered espresso beans that were in one of these bulk things like I just kept going back to the trough it was ugly but with all this freedom in food actually did come a great deal of responsibility I mean not just to our users and customers to one another it was baked into the development culture very early that people were expected as one of their first tasks to pass what was called a readability review he had to write a bit of code and get it reviewed by somebody that made sure that you were doing things in line with the style guide for that language so that you know not just the formatting rules but the idioms and this enabled people to actually quickly understand all throughout our single shared repository and of course that help people collaborate move from project to project and find and fix bugs I mean I consider it a career achievement I actually found and identified a legitimate bug in BigTable one sounds like whew so we are also expected to send all of our production code changes out for review and we are expected to review code for others is one of our highest priority tasks and not only not that not only kept the code at a pretty high quality level but it created the opportunity for discussion around each change an issue you know ideally that meant that knowledge about a particular feature what have you wasn't limited to one person's brain and then this repository a discussion people could mine it later for history if they need to revisit that code in that decision and also enable people to hone their skills they would share like insights things they learn tips and tricks we are expected to participate in the interviewing pool provide clear thoughtful meaningful meaningful feedback on candidates of course we actually did have a lot of internal documentation had a huge wiki for a number of years it was eventually replaced by Google Sites there was like this early culture of doing design Doc's where you know you basically talked about the pros and cons of the thing you want to build before people go forward with it and that became a less prominent practice but it was still alive and well in a lot of other forums but the bottom line was communicate with people collaborate and support one another across the company in order to improve our operations and improve the capabilities expand the capabilities of our products so basically we had a responsibility to help one another improve our code our products our company and ultimately ourselves and so on the whole things were working pretty well for Google had to be doing something right a lot of ways it was but there was a side no one outside the company could really see or appreciate we were potentially in danger of collapsing under the weight of our own success because as you can imagine as the company grew as more people came in writing more code more features more products the complexity and everything just exploded and even though Google did the best it could hire the best people look at fine and to support them there was ultimately never going to be any amount of brainy heroics that was going to overcome the challenges of sheer scale so the company was risking holding itself back slowing itself down as the fear of change and all the things that might go wrong threatened to stifle courage and innovation which would lead to missed opportunities empire building mediocrity and ultimately irrelevant and there is no project more visibly susceptible to these forces than the Google web server or what we call excuse me what we call gris for short so this is the team that's responsible for google com homepage and I wasn't a member of that team but I work very very closely with some of the folks as we'll see and the thing is back in 2004-2005 whisk was not a glamour assignment it was basically a dumping ground for all these different changes that people had four features they wanted a ship and they're on different teams telling gris integrate this integrate that things were a mess and just imagine if you break google com or allow it to be broken I mean the search results might not be as good as they could be they might suck they might come back too slow and so all of these thousands of queries per second add up pretty quickly to thousands of broken promises and you're talking about losing not only revenue in that case but trust the same time though like I said you know a lot of people thought hey things are great everything's working we're doing all these other things that are great of course there was a lot of press you know inflating egos people probably believe the hype a little bit and then you had like the frauds like me that somehow slipped through the system and we were just you know sweating bullets the one day we were going to get found out and kicked out of the company most people at the time though actually they didn't know how to write testable code and I'll hammer on that in a minute and the development tools we were using at the time they were really breaking down under the load and when I say friction here I mean it in a very physical sense things just took time but the biggest obstacle convincing people the testing was a valuable practice by far was you're trying to convince people the value of preventing problems rather than just addressing them as they arise so we can and I have pointed two notable examples where automated testing discipline may have led to the prevention of some pretty high-profile bugs but people kind of underestimate the tendency of these things happening to them especially when the core of their value system depends entirely on objective measurements and certainly Google's famous for data-driven decision making it served the company very well its customers society is benefited but it did make it extremely difficult to convince people that investing and learning how to do better testing was going to be worth their time and we could not really blame them because again people back then especially all the MIT and Stanford grads they kept scooping up they had no experience with testing outside the slowness and brittleness of the status quo and they were always under delivery pressure you know ship something that you could show your grandma so giving given our inability to communicate using this language of data they really couldn't understand why we thought it was so important and we couldn't blame them for not trying to do any testing when they couldn't afford the time to learn how to do it so we had to find other ways around the problem and before I get into the specifics when I talked a little bit in the abstract about how cultures change in general one thing for sure it doesn't happen like this because there's nothing worse than Cartman with the Thorat ah which basically means no matter how good an idea if you try to shove it down from the top people are likely going to fight it and it's not going to work fortunately for us at Google at the time so many of our executives had learned this lesson before at Microsoft novell son digital Bell Labs but it also doesn't change like this there's no rock star guru ninja that's going to come in and fix all your problems for you in fact I fight against people using these words because I think they ascribe a certain type of power to other human beings that's supposed to be beyond the reach of us mere mortals and I believe that it's wasteful at best and dangerous at worst to assume that change is only possible through the power the magic the charisma of a selected few now not anti power mythology I actually think they're very useful tools you just have to be very careful about what you're doing with them and how you cultivate them and a useful mythology is going to be repeatable to some extent in other words it provides a model I mean we can never say Google did this we're going to do exactly that and it's going to work different situations products people dynamics but I think the most important part when we tell these stories is the principles from which all of these processes and technologies emerge and kind of the core principle driving the story I want to tell today is that organizations should be serving the people in them not the other way around they shouldn't just be coercing people to just follow orders in order for that to happen people need to know what the right thing to do is and they need to know how to actually do it so back in 2005 relatively few people especially in Google but throughout much of the industry nobody really had that much experience with automated testing it or had any experience you know not just knowing how to do it but why it was important and the tools were getting slower and slower we had to make the right thing to do the easy thing to do so let's get into specifics why was the right thing so hard to do back in Google in 2005 well back then as I mentioned testing wasn't as big a thing it was picking up a momentum clearly but it wasn't quite the kind of thing that a lot of people take for granted these days and internally somehow the term unit test was applied to anything that ran in five minutes or less and otherwise it was a regression that's that's what you do that's how it was expressed in our build language unit test regression test and like I said the tools they were getting slower with every new developer every new project every new line of code and again back then everything was on your workstation to and one of the most common reasons people said oh this is why I don't write tests I said my codes too hard to test I mean these people figured out how to scale web search and other applications to an entire planet and they say they can't figure out automated testing does that sound right to you what they were really saying they just don't know how to do it which I just got out Stanford and I joined Google I know everything not going to MIT that they certainly don't teach testable design in most graduate courses most folks had never been on a really good project or a really bad one and on top of that there were few good examples to follow so absent this knowledge and experience everybody just kind of felt like Jon Snow when it came to testing like they knew nothing they wouldn't admit it but they didn't what they were really saying when they came up with the I don't have time to test excuses takes forever to do anything and the test that we have take too long and I got to deliver some stuff these excuses they were valid but they contributed to a company culture that was actively hostile to the adoption of automated testing and there are a few of us I'm about to introduce that we felt the culture had the change for the good of the company our products our customers and ultimately for society but merely just arguing people down that wasn't going to be a productive path so the things they were complaining about signaled real problems but business as usual would not suffice so we had to stand up take a stand which is difficult in any organizational culture to go against the grain but the thing is just because a piece of the culture is broken doesn't mean that the rest of it is so by applying the cultural tools at your disposal effectively you can eventually reach a point where you can make the right thing the easy thing to do and despite the limitations and challenges that we face there were a lot of advantages to the fact that hey we were at Google we had access to information like crazy and all kinds of tools to help us discover it we were empowered to develop and pursue a vision and to collaborate with like-minded folks in pursuit of that vision and we ended up adhering to a common model as well that Geoffrey Moore described in his book crossing the chasm he was describing you know adoption of new tech products in general but certainly adoption of any practice follows the same pattern and if you're not familiar with it over on the left the thin end of the wedge are these two groups called the innovators the really bleeding edge people and the early adopters and I lump them together and I call them instigators swear my title comes from self-applied the idea is like these are like the the like-minded seekers have changed they they know things can be better and they want to make it happen but they have to cross this chasm to the early majority in the blue here and the early majority is actually very open to new things new ideas and change but they're not the expert they need solutions that are accessible they can take and they can just start using the late majority the purple there you know there was a stupid ever everybody else is doing and the the laggards over here they're useless just forget about them um but the key is you got to get the right messaging to the right people in the right order and crossing that chasm in between the instigators and the early majority is what will make or break your initiative and we'll see i plugged in a little bridge here and i'm going to illustrate that in a few slides but first let's go back to the glist example so the tech lead up with guy named bart Matata he thought automated testing would go a long way to solving whistles problems with complexity and fragility so the team took a hard line no more changes would be accepted into gris without accompanying automated tests and so they set up a continuous build and they religiously kept it passing they set up coverage monitoring they made sure it was up into the right over time and they instituted a policy and wrote some testing guides and they made sure that all contributors both within the team and from other teams adhere to those policies and practices and despite the initial unpopularity of this hard stand the gwiz team held firm and they did eventually turn a corner they became one of the most productive teams in the company they were in true integrating all these different changes from all these other players and they were still keeping up a really brisk release schedule and as the team grew new people were productive very very quickly because of all the automated tests and the resulting good healthy code that people could just dive in and contribute so ultimately this radical policy enabled the google com homepage to very rapidly expand its capabilities and thrive in an incredibly fast moving and competitive technology landscape but yeah I mean it goes without saying which is an example this is what automated testing can do and why it's good but it actually in the context of larger google it was still a pretty small team in a large and growing company so we had to take its message its principles its model and kind of amplified this some health and build that bridge across the chasm so Bart partnered up with a fellow named Nicholas ecchi and started what I proudly remember as the testing group light and a group 'let basically was hey we're a group of people that have twenty percent time we're going to basically create our own team and try to solve a problem for the entire company and eventually I you know was handed the torch I led this group after Barton Nick at some point we had very little budget zero authoritah but we had so much latitude to try creative solutions to the problems we are facing and we could rely on the Gris experience as our foundational model so one of the first things we did is we worked with ng to you this was a fully staffed in-house training organization and we help them develop these training materials called code labs and then based on those code labs we produce this lecture and lab session for new hires because when you get hired and gdu walks you through like a two-week session of classes and they also helped us coordinate and promote Tech Talks both from people within the company and people that we want to invite from outside we ended up working really really closely with our internal tools teams as well and eventually as we'll see unleashed this toolkit board tool chain that essentially took away the I don't have time to test excuse but our biggest breakthrough I think by far testing on the toilet if you haven't heard of it it's been running for ten years still running it's a weekly newsletter we put up and just about every bathroom throughout the company i still stay we isn't that strange and the idea is we would use this medium to gradually increase awareness of testing and people's you know sophistication level of comfort with it and so I picked this particular episode you can't read it here but if you don't you know look at the slides you could blow it up but I picked it not just because I did happen to write it but I accidentally also encapsulated to other hugely important testing group initiatives the first was test certified so this was a road map to better testing practices that we based on the glist model and it did two important things it hacked our culture because it was comprised of you know installing tools it allowed you to measure things and then the tasks themselves added up to levels and then when you were at a certain level you could see it on this app the test certified ladder and compare where you were and then the other thing is that it got people over that big scary psychological obstacle of yeah I want to do better but where do I start and it went a team agreed to participate in the program we also found mentors volunteers that would you know encourage and advise the team on things that they wanted to do and these mentors would also validate the progress as they climb the lab or what the ladder and one of the things we figured out and about mid-2007 is hey this is a great mission let's get the whole company to test certified level 3 by the end of 2009 and we didn't care if a bow is actually in the test certified program we just wanted them operating as though they were so the other program that is encapsulated in this is the test mercenaries which I also became a member of and we were like hands-on internal consultants we would join teams work on their project for a few months and we would be using all the things that we talked about and testing on the toilet the tools and techniques we would be doing that with side by side with the team on their own code and we would use test certified as sort of like a you know a guide and a bowl for our progress and then on top of that especially the the tight integration loop between the mercenaries and the tools teams led to this tool suite that changed everything and took away the I don't have time to test excuse and one of my favorite things that we also did we ordered these we organized several of these big company wide events traditionally called fixit's where you put a call out to the whole company to say let's spin this day fixing something it wasn't you know no a permission was sought new approval was granted we just did it and we could get people to focus on doing important but not urgent things like writing tests for uncovered code fixing in a broken flaky test making progress up the test certified ladder and then we found out it was an incredibly powerful way of rolling out tools to the whole company at the same time and so the other aspect of this is it created a sense of urgency and that urgency created like this critical mass of attention and effort and energy and it ratcheted up the state of the art and our tools and our techniques and the entire culture change mission would just be up on a new plateau after that but did not happen overnight success was very far from certain for most of the five years that we spent on this problem and remember we had to just gradually increase the knowledge gradually get the tools better in order to provide people with the experience of testing because data was not going to help us I mean yeah we told the goose story over and over and over again and you know it's just really easy for teams to be like wow that's so cool I love hearing about how Wis did it and that's great but you know my project we've got like these other things that keep us from doing it and so you know it's just different wish we could but we're not going to focus on that we're not going to focus on the negative today I you know get me a beer later and we'll talk but let's take the pieces that I showed you of this testing group 'let puzzle see how they fit into the bridge across the cows I can't take credit for this model that goes to Albert Wong he's another ex googler who's now at the VA digital service and he had actually one of his earlier engagements was with United States Citizenship and Immigration Services and he did a tech talk on his experience about what they did and he had this great model and I said you know hey Albert can I steal your slide he said sure what I'm gonna give it a weird name though because he called it framework for helped for helping and I thought it needs to stick in people's bring more than that so welcome to the rainbow of death but what it does here is it kind of outlines I don't know if you can read the red one very well so it's intervene it outlines all the functions necessary to carry that initiative from one side of the chasm to the other in terms of the needs of that early majority and the progression is also linear so over on the left you need the experts they're doing the work hands-on but eventually you progress to the point where the early majority has the knowledge and the power to do the right thing so bear mine when I fill these in like certainly each of these things could spread across multiple buckets but it would get pretty crazy if I did that so let's just go with one at a time the mercenaries and the tools teams well that's not showing up test mercenaries tool development with the tools teams these were the hands on experts the ones that were like the deepest in the trenches trying to solve problems and make things better chest certified did a great deal to inform and inspire a mentor people but the validation was perhaps one of its most important functions had all our information channels testing on the toilet the Tech Talks the lectures the labs the fixit's were very inspirational and high energy and we would give out when people joined test sort of certified they get a build orb from us and they sit in their cube so that everybody can see when the build was was was passing the orb was green and happy and when it wasn't passing it was red and sad people kind of anthropomorphised it a bit funny and in New York we built ones that had like the Statue of Liberty and her torch was the orb is pretty obviously the testing group lit and the broader test certified Minter community we mentored down here the last big steps these two big fixes this revolution fix it was in January 2008 that's when we rolled out the tool chain that took care of the I don't have time to test excuse because everything suddenly could build more quickly all you had to do is declare your dependencies correct and your header files in C++ yada yada and you can use these shiny new tools that just shave literally hours off a build time sometimes but it also meant hey now you can do more testing and then in March 2010 we rolled out the test automation platform during the tap fix it and this is still in use today it's the centralized continuous integration service and at the time they've had to throttle it somewhat since then and you'll see why momentarily but at the time it could test every change as it went in figure out the dependency graph run all but only those tests that were affected by the change and it was so clear and precise and fast that as a build cop after tap was everywhere I would see my build is broken I go investigate through tap and I'd see that a dozen other people had already identified it and if the change was either already fixed or rolled back but then i looked up again the bill was passing again that's some incredible power so to put some numbers on it rachel potman from the tools team gave a talk at the at scale conference last year so these numbers are a year old and I'm not going to belabor them you can see for yourself that's a lot of code a lot of changes and in her talk she talks about a whole ecosystem of tools that contribute to this environment but she makes a point of saying specifically tap is our automated test infrastructure without which this model would completely fall apart now there's obviously plenty of room for improvement but instead of arguing whether or not people should test anymore which is what we were up against they're just arguing about how best to do it the fear is of changing things it's largely gone and Googlers today they can you know see tangible progress towards exciting new features instead of being held back by chronic outbreaks of high-priority bugs and you know from the outside view things just not only continue to work but they keep getting better largely because the developers can innovate experiment and implement things without the fear that they're going to break something else even after the 2008 financial crisis and I spell it funny for old theatre reason we lost most of our manual testers but there was no quality crisis that followed somehow we had kind of reached a tipping point where people knew enough about testing and how to do it and they had the tools and certainly automated testing wasn't the only reason that Google survived the financial crisis but I'm fairly well convinced it played a huge part so after years of grassroots teamwork our little team done the impossible and that made us mighty now thank you I'm waiting for somebody to get that reference so this is Caravaggio is David and Goliath and I love putting it up here at this point to make to make an observation this is a self-portrait of the artist as Goliath and the reason I put this up here is because you know there were no outside obstacles resisting us the technical problems were the pretty easy ones to solve the the real problem was our own identity and culture it took a gradual years-long effort to overcome this self-image and appreciate what automated testing could do for us and we had to provide that knowledge in that power necessary to change people's perceptions and experiences because often the greatest obstacle to the changes we want to make in the world is the way that we as individuals or teams or organizations already see it now I want to take another abstract side path here how do we know still that automated testing provides value we still can't really have we don't have definitive metrics that people can clearly agree upon up front like we talked about stories we talked about experiences side effects and things like that but realistically how can we be certain that anything we do ahead of time is going to provide value how do we know that a feature is going to work or not so it's almost impossible to demonstrate the value of a preventative measure or a supportive measure whether it's testing or security review or even documentation and a book just came out by the CIO of Citizenship and Immigration Services fellow named mark Schwartz he's a very very forward-thinking CIO in the federal community and he wrote this book called the art of business value and first off he talks about why the standard measures that that the agile literature talks about in terms of demonstrating business value they're not any good they talk a lot about ROI but ROI is focused on factors that have nothing to do with kind of the incidentals that come up during development and excuse me instead what mark wants to replace roi with is the proposition that value is actually this emergent property that arises from the experience of a team trying to deliver value and to provide context for his definition he talks about how organizations should see themselves as complex adaptive systems that self organized under leadership's influence so in contra asked a hierarchical command and control you know do what you're told systems that resist change and before you know going any further just really quickly hit on the basics of systems thinking here there's an author Donella Meadows she has an introductory book thinking and systems and she talks about how systems are comprised of elements or in our case people interconnections relationships and then the purpose of a system and that the the system's behavior depends on that structure that systemic structure itself you take an identical stimulus apply it to two different systems you get two different outcomes and this behavior is sensitive to information flow which is itself sensitive to the interconnections between elements or more tangibly to the relationships between people and that the purpose of the system is another key factor that changes the behavior the whole thing the punch line for me being any time you try to devise a system that tries to shape how people behave without accounting for their nature it's going to fail because in my experience the secret to Google's success wasn't that it had you know the smartest people or the most money or the best products or the best cafes and micro kitchens and ski trips and you know not stuff that we had the key was that it created a system that works with the best tendencies of human nature instead of against them but that system also worked against those human factors that tend to compromise quality and innovation in the long term the purpose of the system was continued innovation based on the principle that smart creative people will thrive in an environment that empowers them to do so but this empowerment depends upon the responsibility of every individual to foster constant knowledge sharing learning and improvement in short Google understood early on the innovation and success is an emergent property of a properly established system and that brings me to the federal government so you know the momentum of the healthcare gov recovery created this political will we're going to start innovating across all these agencies specifically we're going to move away from these overly restrictive catastrophic we flog waterfalls saw contracts and we're going to do we're going to do devops we're going to do these you know agile methodologies that are accepted throughout most of industry but if innovation is something government innovation government something people say they want then why didn't it happen before october2013 very quickly let's look at the obstacles to innovation as I see them through the lens i looked at google's problems so not going to belabor any of these obviously the context is different the manifestations are different but those same basic forces of human nature are parallel to those that we faced at Google and to those that are really frankly present in almost any organization I mean the government case people are afraid of taking risks they don't want to be held accountable which is sort of government speak for getting fired and so you know the desire here is to replace this you know culture of fear with one of shared can shared effort shared success and I think knowledge sharing is a critical component of this so that you can eventually make the right things easy for people in the government and to illustrate again why I have this deeply held convictions take a look at my first day at Google I thought you know I had the world at my fingertips I pretty much did I mean I tell people it was like jumping into the fire to drink from the hose but my first day in government not so much I had to do a lot of digging around bothering interrupting people chasing people down going from this document to that whatever and that didn't just use up my time energy but all the people that I had to interrupt on my journey to find out what I needed to do to start doing something and I realized at some point everything the testing group look did we did based on a foundation of you know values in technology this whole combined platform this system that existed before I joined we needed a similar foundation in government but didn't exist yet so what I do well apparently i got the term for it later without realizing i was trying to create a learning organization and so there's another book I've only begun reading called the fifth discipline by Peter singe and he kind of outlines the introduction like the five big things you need to have a learning organization quickly he talks about personal mastery shared vision mental models dialogue and systems thinking so this whole systemic idea is coming up again and so everything he said in the intro resonated with me especially this quote I've met many people who have experienced this sort of profound teamwork in sports or in the performing arts are in business many say that they've spent much of their life looking for that experience again what they experienced was a learning organization the team that became great didn't start off great it learned how to produce extraordinary results so without even being consciously aware of this I did very consciously start trying to rip off you know some of the best ideas from google and implement them in this new context to suit the new environment one of the first things I did I created this prototype that called the hub and you know it's just kind of this amalgam of documentation and you know employee directory and things like that the idea was to try and just give people access to this information like what do I need to know who knows it who's working on what what do they what are their skills and eventually this kind of morphed into what's now handbook 18f gov which you can actually visit I finally opened it up and from that we also extracted the team api because there is a lot of data munging going on it's like why not expose it as json endpoints so now we have this graph of things that people can put whatever sort of interface they want and a lot of that data eventually started coming from these metadata files we would put in our repos just a little Gamal file that says what the project is who is for the impact who works on it the technologies that sort of thing feeds into the API and then if anybody saw heidi's little lightning talk from yesterday she mentioned at one point hey how do you capture knowledge in slack what you ought to do is put you know designate an emoji like a tree and you know somebody will come back you know somebody says that's important put a tree on it and somebody will come and triage that make sure it gets that knowledge is shared in the right place so we started doing that but then the people who were triaging were like yeah be great if it would automatically file issues against the handbook and the temptation was too great so I ended up writing this cubot plugin and this is the first successful actual integration of this where I posted a little test message I put a tree on it you can see Hugh bot created the github issue automatically and then it was the bot that put the check mark which is another little signal hey don't create a new issue for this and then somebody else was like yeah I put tada so what this does is this radically reduces the friction necessary to capture knowledge produce documentation especially for our non-technical users and by scaling up these systems making them more accessible to team members I was trying to create the space where these insta instigators could discover one another and connect and band together to create group 'let's of course in our parlance we didn't use this term as much I try to promote it but they're like oh we're working groups were guilds same thing I organizing Court nice three the first one I started was documentation to promote the hub and some of the other tools we'll see what yeah no that's right and then I started a testing group 'let and then I you know started a the working group working group to try and you know create a space where people who want to do this stuff you know could share ideas commiserate whatever and to make it easier for people to document their knowledge and share up the organization as well as with the public I launched this little github pages rip off and called it pages we were told in no uncertain terms that it was no longer acceptable to put canonical government content on a particular domain and so this ran on our own infrastructure with the dot gov HTTPS and all that and then you know we also have this common format figured out a way to package it make it easy I wrote this guides template which was a guide that told you how to fork it and to hack on it till you had your own guide and then you can see this is a you know this isn't even all the guides now I mean they just kind of exploded all these different working groups we had you know an accessibility guild agile guild course testing group that did the testing ones and these artifacts would not only create discussion and you know some sort of tangible work that people could produce within the team but other government teams would take notice and start to contribute and even the public we had a couple of public contributions and so this is like the dream come true like you know a participating citizenry and all that kind of stuff so just quickly let's just see I'm not even going to Blaber this one this is just going to filter in a bunch of the stuff I talked about a bunch of stuff I didn't but the point is the same model kind of helps us see that even though it looks like oh we're doing a whole bunch of stuff all over the place this whole scattered array of initiatives actually serves to reinforce one another to achieve the mission and the point of developing the tools in particular is that all of these things map to the same principles that I believe made Google a great place to work and consequently a great force for good in our industry and society and if I seem to be a little too critical of government particular I want to remind everyone that the bulk of the industry was about 30 or 40 years behind the curve fame i remembers fred brooks from IBM system/360 project he had this little book in the 70s the mythical man month they talked about how you can't add people to a late project it makes it later because suddenly you have so much more communication complexity and overhead and he argued for you know careful thought put into design which in his day was data structures not CSS or anything because good design often Trump's the need for complex algorithms then he followed that up in 1986 with the no silver bullet si he talked about the difference between accidental complexity problems of complexity you can solve with better editors although evolution stopped at vim we all know that and in better languages for example but then he talked about essential complexities that you can't get around that you have to manage using this list of methods he talks about which are based on ongoing analysis and communication he talks about building over buying when it makes sense or buying over building i should say rapid prototyping and iteration growing software organically adding features as users gain experience in cultivating great designers in particular and the thing is mainstream practice is still trying to catch up to his insights from 30 and 40 years ago so why is it so hard for a good idea to get traction well i found this really interesting article is freely available online from the Harvard Business Review it's called why employees stay by Vincent flowers and Charles Hughes and their research wasn't into the typical why are you leaving a company the whole exit interview he won know why employees are they when know why employees were staying and they have all kinds of really amazing insights that I'm just going to hit on a couple here but he noted that managers assume that what's right for them is naturally going to be right for the employees without actually considering what might be right for the employees so that's a common trap and it resonates with my own experience where you know even new organizations they tend to backslide into command and control structures because founders and executives and managers most of the time they don't really have the experience of working in a high functioning learning organization and as a result many management teams they were they tend to remain focused on short-term goals particularly financial goals this they'll start believing in their own importance especially if there's all sorts of you know shallow and undeserved media attention or surrounding them and they neglect to cultivate the creativity and aspirations of their employees instead just you know treating them like servants there to do as they're told for the cold comfort of receiving a regular paycheck and in their conclusion they note that and I quote most organizations historically have been and still are created and perpetuated by manipulative and conformist philosophies if management wants employees to stay for reasons that are right for the individual the corporation and the society it must develop existentially managed organizations that truly accept and respect people with differing values and this article came out a month before I did this statement however maybe it should sound familiar should have deep echoes of an ideal system with which we should pretty much all be familiar based on a philosophy that was drilled into us at a very early age imagine your lives without the US federal government and what it's produced in terms of Liberty security innovation prosperity the system the founders created it was based on service to the people not the system itself and it provided the foundation for this sustained innovation and success I mean so happens our own industry has developed a rather extreme version of this culture beyond anything that Washington or Jefferson or Madison might have imagined many of you may be familiar with Eric Raymond's cathedral and bazaar paper where he dissected the dynamics of the Linux kernel development community and then also experimented with his own fetchmail project and he came up with a number of different principles that drive open-source culture but I was really struck towards the end he included a quote from a russian anarchist named pioter LexA Fitch Kropotkin I apologized to any of our Russian friends he had written a book called memoirs of a revolutionist in which he said I began to appreciate the difference between acting on the principle of command a disciplined acting on the principle of common understanding the former works admirably in a military parade but it's worth noting where real life is concerned the aim can be achieved only through the severe effort of many converging wills now it seems kind of weird to quote a russian anarchist after extolling the virtues of American republicanism but i think the spirit is largely similar the founders and Kropotkin they understood the great accomplishments are often the product of liberated individuals acting in concert and while Kropotkin thought anarchy was the way to that path or the best path to that end I think the American experiment has proven more scarab ille scalable durable and resilient because it provides a framework that values and protects individual rights and liberty rather than sacrificing them and the system can be amended over time they understood that continued innovation is an emergent property of a properly established system and citizens have been free to experiment with crossing chasm after chasm ever since as we saw yesterday crossing literally crossing chasms on other planets that America got this part right really early on I mean it's celebrated all the time but the tragedy of this is that we often think the system of government was just you know immediately handed down as a blessing from these immaculate geniuses the true story of it's painstaking development i have been reading a lot of history lately it was so horrible and the chance it was about to fall apart pretty much from the outside of the revolution until about the war of 1812 this struggle would speak very loudly to those of us are trying to produce change in any organization and they kind of top this all off we often talk about DevOps we promote the promise and the practice of DevOps supposed to affect organizational greatness by promoting transparency autonomy and collaboration all these other benefits but I want everyone here I challenge you to think about your assumptions of this DevOps model and what the real intent is I mean yes it's a powerful organ tower phul model for producing business value and achieving business objectives but at its essence it is a toolkit for producing complex adaptive systems designed to shape human behavior and at their best these practices cultivate teamwork due to the convergence not the coercion of wills and when that happens you got an organization of the people by the people and for the people that will produce insights methods products services that will bridge the chasm and not only satisfy the expectations of customers and society at large but actually exceed them thank you so much so we do have a few minutes for questions I believe and if you have a question please walk up to the microphone otherwise it won't they won't be able to hear it on the recording what yeah the sound guys were very appreciative of the fact that I emphasize that oh we have a brave soul some curious when you work on the the culture of testing a lot of organizations have gone to you know a single engineering role rather than separate testers and developers and on the one hand I love that because you know their developers you on the one hand I love back is every developer should test and on the other hand on my wife I spent a lot of time testing and I always found it fascinating how delighted she was when she broke something which is I think a skill that most developers are missing and so I'm curious I'm just curious what your thoughts are on that in terms of like getting excitement about hey this great test and I broke something as part of the culture well I mean that's what we were driving towards it's a much different experience when you get that feedback after a few milliseconds then after five minutes so that was kind of a big thing and also like I said people didn't know where to start in a lot of cases like okay we want to do testing now so what do we do and that's where test certified and things like that came in and in particular with manual testers it turned out our what was then called test engineering force which was the the test engineers who are more manual and the software engineers and tasked who are focused you know they're full developers but they're focused on testing tools and issues managed to get them excited about test certified because they were wonder what do we have to do to connect with our client teams you know they were kind of matrix down at the time and you know how can we have a conversation that's meaningful and productive and so I started hammering them I said test certified man you bring test certified to them and you say look just start doing these things and you'll catch so many things up front and then we our time is better used we can do something you know provide some higher value out here because we're not just chasing down stupid crashes are off by ones or any of that kind of stuff and eventually that that really what you know that spread like wildfire throughout test engineering and then with that kind of fan out between that and all the other things we were doing we we managed to saturate the culture to the point where it just became what you did but took five years awesome talker really appreciate it thank you um I noted how in the Google example it seemed to me like this is more of a kind of like a peer-led relationship bridge building and at 18f you or it seemed to be more of kind of like a leadership led culture change I'm curious about how that modified your actions and behavior talk to me later I would say it was well no talk to me later I I will kind of say this in the abstract like I said I left in March you've seen my my principles and mo like I can help you through the math later hi oh you can pull that down a little thing Thanks I noticed in your google example you talked about this twenty percent time principle I was curious if you have any recommendations what are you doing a culture that doesn't have twenty percent that expects 100% billable time all the time well um part of me wants to say talk to me later um yeah so I think at that point becomes a very personal decision either you take a risk and try to find other crazy people to fight the power as it were or you make a decision to leave there's i'm sure all sorts of various you know points along that spectrum where you can make different decisions in context but i do think at some point if the fundamental structure isn't there to eventually accept and support the kind of change that you want to create that there's no point in spending the best years of your life banging your head against a wall but if you can find even just one other crazy person and just do stuff you know kind of a skunkworks go for it's fun okay timers running out not sure if we should we cut off the questions here can we anymore did mean intimidate anybody yes if anybody comes up and asks a question right now it's no longer me holding people from lunch it's you sure all right well thank you so much

Channel: OmniTI

Views: 157

Rating: 5 out of 5

Keywords: OmniTI, surge 2016, mike bland, google, government

Id: D5ld7NJPXkI

Length: 60min 40sec (3640 seconds)

Published: Fri Oct 21 2016