How Google sets goals: OKRs / Startup Lab Workshop

you good morning my name is Rick Clow I'm a partner at Google Ventures for those of you I haven't met before thank you for coming this morning thank you to those of you watching online we're here to talk about objectives and key results this is one of the critical elements of how Google managed its early execution and really grew the company and remains pretty critical element of its of its DNA today I came to Google a little over five years ago I was part of a company as some of you know called feedburner feed burn was acquired in June of 7 the very first day that I joined Google was the first day of a quarter which meant that Google was already about a week and a half two weeks into that quarters okay our process I didn't know what ok ours meant I was asked to draft my ok ours within my new team I really didn't have any idea what was expected of me certainly it startups prior to Google had never had a formal process of setting my objectives we're measuring goals but in the last five years I've set ok ours as a biz dev lead as a product manager on a couple of different products and then more recently in my time as a partner Google Ventures process had a real clarifying effect for me personally and for the teams that I was a part of helped me understand what mattered it just as importantly helped me understand what I wasn't going to be working on today we're going to talk about what are ok ours how do we implement them at Google how to adopt this process at your own company I think one of the things startups often struggle with and one of the things I spend a lot of time working with you all on is fighting the urge to just say well that's Google right you guys you're different we're not Google well as you'll see in this deck Google wasn't Google when this was first adopted at Google Google was under a year old when John Doerr came to Google and pitched Larry Sergey Marissa others on how to adopt this process at a very young and an ambitious company and I'm going to apologize in advance because I've committed at least two cardinal sins of in this case keynote not PowerPoint so much so it wouldn't surprise me if somebody from Cupertino drove over and wrestled control of this remote from me not only have I used slides that are almost entirely text which is bad enough on its own but in addition I've also embedded an alt text presentation inside of an alt text presentation which is a new low for me as a presenter but I think as you'll see there's there's a reason for having done that quick show of hands how many of you read the book in the Plex by by Steven Levy it's a book about Google's origins and growth up until about two years ago okay maybe 20% or so misrule is a great book Steven was given really incredibly direct access to not only our founders and many early employees but a number of people across Google that gave Steven a very broad sense of who we are how we worked and he talked about how in 1999 a John Doerr who is a partner some of you may know at Kleiner Perkins remains a board member for Google to this day and he came to Google to talk about a process that he'd witnessed at Intel called ok ours objectives and key results and Steven in his book talks about and I know that those of you on the livestream can't see this I won't read the whole pages here but talks about how how Dora came into Google and and okay ours were an immediate hit that unlike what you might expect where this might be seen as some kind of Dilbert ization of of a tech company instead it was immediately apparent to Googlers that this was data right that what this gave the company access to was a way to quantify instead of just qualifying what it was they were working on and how well they were doing towards those goals and periodically reviewing did they meet expectations for what it was that they were trying to do and a highlighted element here a couple of things that are critical to how we think about okay ours and and I think you'll see themes resonate throughout this discussion it was essential that okay ours be measurable an employee didn't say I'll make Gmail a success but I'll launch Gmail in September and have a million users by November real critical there are two different points right in the key result launched by September and a million users by November those are both measurable right the key here is that not only is the objective clear but the means by which you would grade whether or not you had met those that objective is really what's key here and Marissa goes on to say it's not a key result unless it has a number the okay are embodied ambition it sanctions the ability to take risks says door and then it goes on to talk a little bit more about what else makes okay ours key and and it's important to note that everyone's okay ours from Larry and Sergey on down our public within the company everyone it's part of your directory listing I went and what's called MoMA Jaya Steven references it by name here in the book if you go into MoMA you look up an employee right next to their cell phone their email address their title is a link to their okay ours that means that not only can you see what they're working on this quarter but you can see what they were working on last quarter quarter before you can see their grades and we'll talk more about the the process of how to grade them and and why different approaches may be beneficial in your companies so I got ahold of John doors actual deck this is the embedded alt text presentation I was talking about a few minutes ago but I actually got the original deck from 1999 and I think it helps I'm not including the entire deck I'm tried to take what I felt were the most critical elements that Illustrated both how we think about them at Google and how I think you should put them into context for understanding how they might apply in in your company so objective of this deck imagine that I'm John Doerr for a moment presenting this to Larry Sergey Marissa he he started his deck with his OKR for that meeting and the objective was I'm going to develop a workable model for planning as measured by an established three key results note how each of these three are inherently measurable one finishing the presentation on time now I haven't told any of you how long this workshop is going to take so I think I have a pretty good chance of finishing on time since only I know it on time is completing a sample set of three months objectives and key results so at the end of the meeting they were going to have a set of their okay ours three having management agree to institute a trial system for a three month period pretty straightforward so John goes on to give examples of and okay are in the context of a football team now I'm a big 49er fan so I'm going to use the 49ers as my illustration I think the key here is is one of the things that when I've talked to with many of you individually you've struggled with which is how to connect the individual objectives with the team objectives with the company objectives and I think John's example of the foot all team actually goes a long way to helping clarify what that what that looks like so general manager I'll say the team owner Jed York his his job is he's going to win the Superbowl and fill the stands to 88% now 49ers would tell you we're going to have sellouts all season but these are measurable key results with an objective that in this case is defined as the GM making money for the owners you can can get a sense of what that looks like now filter that down the head coach notice that he doesn't care about filling the stands all he cares about is winning the game right winning the Super Bowl and the way he's going to win the Super Bowl is he's going to have a passing attack now in the case of the 49ers bit of a crapshoot if Alex Smith is going to throw for 200 yards in a game but maybe it's more important that we get Frank Gore the ball and so we might say that you know we're going to we're going to cumulatively run for more than 200 yards or number three and defensive stats given the 49ers defense you might say things like we'll have a net positive take away differential where you know we'll have more interceptions or fumble recoveries than the other team right these are measurable things that at a high level in this case the head coach is focused on now these roll up into the prior comment which was you know the GM is going to win the Super Bowl fill the stands but obviously there are some things that the GM is not worried about that the head coach is intensely focused on pivot from the head coach to public relations their responsibility is they own filling the stands right we we're going to hire colorful players I'm not sure that if I were in charge of PR for the 49ers that that would be my focus but if I think back to 99 I think that's when they had hired Deion Sanders so maybe that exactly was the strategy back at the time get media coverage highlight key players obviously this is a way that you generate enthusiasm amongst your fans that may be a byproduct of what the coach focuses on but it is absolutely a hundred percent of what the PR function is focused on here now filter that further down defensive coordinator all the defensive coordinator is focused on is obviously the defensive stats holding the other team under a hundred yards passing intercepting at least twice in the game recording at least three sacks whatever you want to extend that analogy whatever might be important given the players that that defensive coordinator has at his disposal offense on the other hand of course doesn't care about interceptions except for not throwing them but they're going to set an objective of seventy-five percent completion rate now I should note here for those of you that follow football seventy five percent completion rate would be pretty astounding particularly in a in a Super Bowl part of the goal of an objective we'll talk about this again as we get into the mechanics of how you define okay ours is that it'd be a little uncomfortable you don't want to always get on a grading scale of zero to one you don't always want to get a one in fact the target ought to be a point six point seven so you should always have goals that are slightly out of reach that feel a little uncomfortable so that you are always striving to do more to do better than perhaps you did the quarter before where you think is reasonable in the context of the current quarter think about now we've talked about defense offense there is a special teams unit right they're not focused on interceptions they're not focused on how how well the quarterback does their focus specifically on running back kicks trying to block the punt trying to block the field goal they have very specific goals that are completely distinct from the other guys same with the new staff same the scouts right they may not even be on the field they may be off watching spring practice for a bunch of colleges the idea here and what what John doar was trying to reinforce to that the the Google team at the time was that if you roll all of the things up from this foundation level in their objectives and key results are reflected the ultimate company's priorities but not every single one of them is going to be considered a company priority right is it here the new staff has a key result of three Sunday features articles or it might be a you know an interview in ESPN or it might be something along those lines is that that's not going to be a company okay r-right that's not going to be that the focus of what the 49ers are are thinking about that the GM or the owner of the team is going to be waking up every day saying you know how are we doing on those Sunday features but you better believe the head of the news team the PR team within 49ers organization is absolutely thinking only about those things now what is that further to the company objective well ties back up into filling the stands right so you can start to see how you can pretty clearly connect these dots even if not all of the foundational objectives and key results ultimately show up in the company okay ours now want to talk about why you use objectives and key results now this was John trying to convince Google Google now has lived these this this approach for 13 years for those of you that have not yet made the commitment to embracing this method of measuring both your objectives looking forward and grading yourselves looking backwards let me talk about some of the things that less to john's bullets here where he was trying to make the case let me tell you about as a Googler who's now lived this 4/5 of Google's 13 years using okay ours what is it what does it do it is absolutely amazing for imposing discipline on the organization it helps me individually understand what I'm working on and why it also helps me make conscious decisions about what I'm not working on which is often just as important make sure that by making these okay ours public everyone on my team anybody in the company that cares can see exactly what I'm working on can see what my priorities are why is that helpful well to give you one example when I get dinged by a team at Google who wants let's say when I was the PM four youtubes homepage somebody pings me and says hey we would love to promo this new product on the youtube homepage well everyone would love to promo things on the youtube homepage the person who's going to make that pitch it might help them to know what my priorities were for the quarter in which we were working and they can immediately know even before they talk to me what my answer is likely to be if the thing that they're promoting is somehow connected to the things that I care about for that quarter they've got a very easy case to make they're helping me deliver on my objectives if on the other hand it's a distraction they at least know going into the conversation they're unlikely to get much traction in that quarter now it might help to plant the seed with me let's say in late October so that as I start thinking about cue ones okrs coming up that I have them on my radar by writing objectives and key results specifically the key results that are inherently measurable you're establishing how you're going to measure what it is you're working on and in the process of drafting those okrs and we'll talk about how how particularly individuals set their okrs you come up with really consensus from the individual to the manager manager to the to the team to the company depending on how big you are that might be individual to CEO where everyone understands at a pretty basic level what is it we're trying to do and more importantly how to measure whether we're there or not and how close we are and then finally focusing effort I think we'll talk more about that in a little bit thing I really like about how this process ends up getting embraced is that there's a really virtuous cycle that happens company objectives are generally set by the CEO your board the leadership team who should have a pretty strong sense in any given quarter of what the things are that matter most to the company right take my Feedburner example back back in the day Dick Costolo was our CEO and he had a somewhat comical way of establishing it but the mission was get all the feeds right if we get all the feeds syndicating all these RSS feeds through feed burners platform where we could both measure all the traffic that was happening and give publishers a way to monetize that traffic it was a very clear statement of what it was we were trying to do that meant that when I would go off to New York and come back and say well such-and-such company wants us to do X the question always that always followed was does that get us closer to getting all the feeds and if it was a distraction might be an interesting revenue opportunity but it was not delivering on the main core focus of what it was we were trying to achieve then the answer was no so these corporate objectives are set and generally communicated down throughout the organization individual objectives by contrast are often what the individual wants to work on and what the manager wants that person working on instead of this being an all top-down discussion it often ends up being a bit of a cycle we know generally what the company wants to be accomplishing as an individual I want to work on important things so I'm going to find the things that are within my sphere that I can do that are most beneficial to the company and I'll come up with a list and then communicate that to my manager or the CEO or whomever it is I report to and we can have a bit of a give-and-take now what's interesting in this process is that occasionally things that individuals want to be working on might end up opening up new discussions for what the company should be doing a good example of that at Google engineer called named Paul Buchheit was really annoyed with the state of male clients in in in Linux so he started he thought maybe I could make mail searchable as sort of the core interface and he started what became Gmail well over time Gmail became its own practice group at Google effectively and became one of the core strategic goals at Google which then had a way of creating its own objectives and key results for the people who were working on on Gmail so let's talk about the communication one-on-one meetings where as an individual contributor I talk to my manager my boss the CEO whomever it is about what it is I want to be working on and what I think is is the best use of my time the one-on-ones at the end of the quarter and the very beginning of the quarter best to John talks here about negotiating what those key results might be that's I think actually a pretty good term for it because you are ultimately offering what it is I want to work on and what I'm hearing back is what we want you to be working on and somewhere in the middle we find the best combination of the two at a company-wide meeting the company okrs are communicated and the team owners identify what it is they're going to be working on there is also a grading process that happens here that we'll talk about in a little bit so typical cycle I'm not going to dwell on this timeline only because I drafted one specifically for the time frame we're in about the end of October that I think will help clarify what the general rhythm of you know when you should be thinking about which steps in the process so we'll come back to come back to that it's basic hygiene you don't want to have 12 objectives it's too much for any one person to be working on really if you think about a quarter being effectively 12 weeks if you had in total only a week to give to any given objective it's unlikely that you could move the needle much on any of them so you really want to limit the number of objectives you have and you want to make sure that you're very clear about what the key results from that objective are don't the interest of limiting your objectives don't stuff every objective with 12 key results you want to be very clear fine to be ambitious but I'll tell you personally there was one core when I had seven objectives I have never been more exhausted and and just mentally drained from the amount of intellectual juggling it took to keep all of those different objectives sort of clear in my head and keeping track on where I was with each and who I was relying on and what I needed to coordinate it was just it was too much and it in the end ultimately took its toll I mean it was it was clear at the end of that quarter that I had been stretched a little too thin John here has a number I don't know that the number here matters as much as the overall intent which is sixty percent of the objectives ought to come from the bottom up I think I would phrase this a little differently I'd say more than half of the company's objectives I need to be coming from the individuals up through the organization if there is too much top-down dictation it's going to be hard to inspire a lot of those people to be working on things because they're going to be told what to do instead of telling you what they think is the best use of their time and and talent there's a comment here about everybody must mutually agree no dictating I think that's really just another way of saying what I said a minute ago interesting that they say one page best we do all of this electronically now so there's no issue of page length here's a really important point this surprised a few of you when I talked ahead of giving this workshop ok ours are not a performance evaluation tool what that means is in the annual performance review cycle that Google goes through ok ours are not a standard element of that evaluation let me be clear when I was reviewed each of the last five years I would sit down and you know right up my summary of what I did in the year prior what I think that means for the company whether I wanted to go up for promotion or not at no point in time were my quarterly okay our bullets factored in into that evaluation now there it's always possible to just include your okay ours as part of that packet I found individually they were always tremendously useful because it gave me really a summary glance what I'd been up to over the last year I could look at the last four quarters of okrs I can look at the grades again we'll talk about the grading process in a little bit and I knew really within a few minutes what were the biggest things I've worked on what were the most dramatic results as a byproduct of that work and I could then summarize my contribution to the company my impact in a way that previously when I thought about you know the prior startups prior to coming to Google it's much harder to recall back what did I do six nine twelve months ago I mentioned this in passing at the outset your target should be 0.6 to 0.7 on a zero to one scale if you consistently get ones you're not crushing it you're sandbagging anything below 40% is definitely in the red that that means you missed the key result entirely and that should be a forcing function for re-evaluating whether or not that is an objective you want to continue to work on okay last slide of the embedded slide promise will we'll move on after this John here talks about this has an alignment from and that it it ensures everybody in the organization's working towards the same set of results I agree with that that was certainly my experience at Google it was very easy to see amongst my team members where we were working on complementary things where we were working on dependent things where there was something I was doing that I needed three other people to deliver something and so by keeping track of where they were in their progress helped me understand how likely I was to deliver on on my okay ours it fairly organically insures the organization's tuned in to what everybody is up to and it that virtuous cycle I talked about a few slides ago there's a so wonderful I use the word organic a minute ago it's the best I can think of here that just ensures that you're working on things that support what the company is focused on as do the people sitting next to you and across from you and down the hall and there is a confidence and candidly a comfort that comes from knowing that you're not treading water you're not walking through molasses you know that everyone shares those commitments bizarrely John even says here in the last bullet that okay ours are fun that I think is open to some interpretation but but I do think that there are elements of how to make this part of the teaming process that I'll talk about in a little bit okay Kay's too good okay ours just in terms of rhythm and process okay ours are both quarterly and annually set annual okay ours are not written in stone if you set them in December and in May you've assessed you say you know things that we thought were true in november/december just aren't proving to be true don't obviously feel beholden to CML revised them they are they are directional aids not not blinders as I've said and I'll continue to say I think this is one of the critical elements of okrs they are measurable the fact that I know whether or not we shipped in September like I said we would well that's measurable if we shipped in September great give it give it give it a one if in fact that was the entirety of the key result talk a little bit more but important to note here that there are ok RS for each individual there are ok ours the team level that summarize what the collection of people are working on and then there are ok ours up at the company level which tend to reflect the most important three for no more than five things the company cares about and of course ok ours are publicly available to the entire company these cannot be siloed it is critical for everyone in the OP and the organization to know what everyone else is working on that's true from the CEO on down grading is also fundamental to this exercise ok before diving into my I pulled some of my personal ok RS from my days when I was p.m. on blogger I've got two more comments here two more slides I want to talk about differentiating between the objective and the key results and then talk a little bit about differentiating personal versus team versus company objectives should feel uncomfortable going into a quarter you should be uncertain as to whether or not you can deliver on the thing that you've just said you will deliver if you are certain you will nail it you're not pushing hard enough you're not thinking broadly enough you're not thinking big the key results are the things that will in or that you deliver on the objective they have to be quantifiable they have to lead to you being able to objectively grade them right don't set an okay R that says I will make Gmail better because in your opinion you've made it better because you made it prettier but if that degrades usage it leads to user attrition if it means people send fewer messages per minute hour whatever the metric is then you have not made Gmail better and it's impossible to objectively assess that don't say make it better make it say that you know you'll be able to help users respond to messages more quickly right well that's easy you can and in fact the better way to say that is I will reduce the average time to response sent from whatever it is now by 30% okay personal versus team versus company personal care is pretty straightforward what are you working on what is most important to you what are the things that at the end of the quarter you'll be able to show you did team okay ours are the priorities for the team not just the bucket of all of the personal okay ours think about our example from John Doerr slides of the 49ers personal okay ours that's Alex Smith Frank Gore David Akers Vernon Davis Crabtree moss you name it it is each individual player has their own okay ours and obviously those are going to be quite a bit different the team okay ours offensive coordinator defensive coordinator PR head ticket sales the new stadium person who's coordinating you know the season tickets those are the team okay ours and then the company who cares big picture top-level focus right when the Super Bowl might have a few others they're secure home-field advantage through the playoffs might be another company level okay are things that help clarify for each individual what the most critical things are to the organization is all right as I said we're going to look at a couple of my Oh cares from my blogger days some helpful context before we dive into this at the time that I took took on blogger blogger was the 8th largest web property in the world still incidentally the top in the top ten worldwide so it's a good trivia question next time you're at a bar and somebody asks you know what is Google's third largest web property by page view traffic not by visitors anymore you know first is obviously Google second obviously YouTube almost no one knows that bloggers the third I take a little bit of pride in this as you can tell but again for context blooder was a fairly ignored product internally fairly forgotten at least within the certainly the US tech echo chamber even though it remained this amazing huge engine of traffic worldwide it was the middle of the recession Google was looking at different products that it owned and trying to figure out whether any of them were revenue generating or could be revenue generating and blogger was making a little bit of money at the time but not a lot so I was asked to come in and turn blogger into more of a business so in these personal okrs that I'm going to share i've removed some of the specific details because for illustrative purposes they're not relevant and i don't want to put you in possession of any material information about some of our products but my first objective was always accelerate revenue growth every quarter that I managed blogger my first objective the thing that I needed to think about every week that I was running that product make more money key results first element this was the first quarter that I was on blogger at the time it was I think I had counted for clicks for any blogger owner who wanted to put Adsense on their site which is ridiculous Google owned blogger Google owned Adsense I wanted to launch a monetize tab that would make that a one-click operation so first key result launch launched a tab and and by the way I had other key results as part of this that that quantified the number of publishers that we would bring into Adsense as a result of this another very specifically written key result implement Adsense host channel placement targeting it's a bunch of words I'll explain in a moment to increase rpms by xx percent note here this key result is written in a way that the core goal is increasing rpms by a set percentage if I had implemented and launched this host channel placement targeting which was just a way of bundling a bunch of similarly themed blogs into a targetable channel for advertisers if I had launched that and had no impact on increasing revenue on those blogs then I'd get a zero or maybe a point two on this key result the key for nailing this key result is ensuring that it had measurable uplift impact on revenues and then finally this was something that that really was disciplined from Joe Kraus who hired me to run blogger he's who brought me over to Google Ventures launched three revenue specific experiments to learn what drives revenue growth in late Oh 8 early oh nine there was a lot you know we didn't know what we didn't know about making money on blogger so Joe's thing was don't assume you have the answers do lots of experiments get into a normal rhythm of launching something to test whether it has an impact and if it does great do more of if it doesn't well file that away know that it doesn't have an impact and don't do more like that and then finally this was a couple of quarters after I'd been on blogger finalized the PRD product requirements doc for the blogger ad network secure ang allocation to build it out the following quarter so this is again a measurable okay are our key result rather in that it showed preparation work that I'd be doing in this case in q4 so that something would get launched in q1 now if I wrote the PRD entirely under my own control whether I did or I didn't but failed to ensure that the engineers would be on on the hook for building it out in q1 you know at best I might get a point five on this okay our I would argue that far more important to get the anjelica ssin than it is to just write the PRD so again in grading this you know that'll that'll come into play when we when we look at grading these and I will be grading them in just a second next objective is to grow blogger traffic by a certain percentage over our organic growth now blogger was this wonderful gift in that it just it had hundreds of millions of visitors every month and would continue to have hundreds of millions of visitors a month whether I got out of bed or not and they would continue to read blog posts and they would continue to visit through search and they would just it just was this engine at that point it was turning ten years old that year it was inevitable that it would continue to get traffic in it as the internet continued to grow and more people came online almost certain the traffic would go up over what it had been the year before so my objective here was to not sandbag and say hey grow grow traffic but to ensure that I was the one moving the lever up so the key here is growing traffic over and above what would do normally so key results to define this launch three features that will have a measurable impact on blogger traffic improve our 404 handling very tactical but if you think about a site that gets hundreds of millions of visitors not all of them land on pages that still exist people delete posts blogs go away if we had wanted to create a worse 404 experience for users I don't think it would have been possible at the time we were you know blogs had their own style had their own themes we were serving an error page that was not using the the default style so if you were familiar with a blog but landed on a 404 it looked like a system error page not a an error page served up by that blog we weren't giving you access to archives on that blog we weren't giving you a search box we weren't even telling you that this put belonged to a blog it wasn't even a link to the blog on which the 404 was being served so we looked at our logs and we realized good lord were dumb as a box of rocks like that's a key area where we could dramatically improve the user experience and probably increase our traffic while we're at it improving bloggers reputation was another objective I had again blogger was about to turn 10 years old that year largely forgotten in the u.s. even though it continued to have a lot of use and we had not been proactive in communicating with press with users with partners and I saw a part of my job as reestablishing blogger as a as a member of the community that we were committed to to being engaged so that meant going out and speaking at events getting bloggers back out in front of people who pay attention to these things again measureable key result I said I'd speak at three industry events coordinators 10th birthday PR efforts figured out that bloggers would 10 in August of that year so it was important to say all right you know what are we going to do to draw attention to the fact that blogger has reached this milestone and how can we how can we use that to further our other objectives increasing traffic increasing revenue so some of the partnerships that I launched in conjunction with this 10th birthday we're done specifically to drive revenue others were done specifically to drive traffic we partnered with a company that had built an iPhone app so that our mobile usage would grow each of these ultimately served a couple of different objectives I pulled a list of the top end users as measured by traffic and personally send each of them an email many of them had been on the platform for years and had never heard from anybody on our team and I felt like it was critical that you know in some cases we were making considerable money as they were I was one of the really great things about working on that product any amount of money I made they were making more because Google would get a percentage of whatever ads were running on their site if they were using Adsense and they would get the majority but I gave them my email address my cell phone number my Twitter account so that they knew how to get in touch if they needed help and they no longer felt like they were you know the using this nameless faceless product anymore a very tactical issue for those that understand the US law of the DMCA as a host of content we are legally obligated to take content down if it is alleged to be infringing the rights of another well a lot of music blogs on blogger the RIAA and the and it's its international counterpart are fans of sending DMCA notices and the DMCA process if you were a blog or user was pretty broken and I felt this was hurting our reputation with some of our our core constituents not to mention it was just it was it was a degraded user experience overall whether you were the blogger whether you were the reader who wanted to read the post so I had as a key result fixing this process and eliminating miss dinner should be another word in here mistaken takedowns of entire blogs when the goal was just to take down a post and then finally improving our reputation getting us active on Twitter that was where a lot of our core demographic were actively chatting with their readers with each other with the world and if bloggers had a problem they would often rant on Twitter about us and you know it was sort of into a void so I wanted to get us active on on Twitter as at blogger it's convenient when the founders of bloggers are the founders of Twitter they can help get you started on that path and and and so we did that so let's look at the grades as I said accelerating revenue growth was always a key for me and these grades are written to be measurable launching the monetize tab well we did we launched it was it was a fairly significant change to the overall UI so that got a 1 here you know as I said earlier the goal is a point 6 or a point 7 but if there's a very tangible binary key result you don't give yourself a point 7 because you never get a 1.0 in this case I did what I said we were going to do it was considered critical so you get a 1 on implement Adsense host channel placement targeting to increase rpms by X percent well we implemented a version of this and it didn't have much of an impact on rpms so this this one was was was a fail launched three revenue specific experiments we launched three experiments only two of them really told us anything of about driving revenue growth that was largely because the third experiment that we launched that quarter that we thought would tell us something interesting about revenue really in the end didn't so that was as much our our error my error in not crafting the experiment more effectively and then finally the the the PRD for the ad network wrote the PRD was still at the end of this quarter doing some horse trading with my engine on which engineers would be building it out but we were fairly agreed at the point at which we were grading these that that it would get built there was still a few open questions so I didn't consider this a one but overall it was a point eight now quick note on scoring you'll note that the point seven in white at the top here next to the objective is just an average of the four key results below I've seen some get really cute in how they grade their okay ours by trying to determine by weight each of the key results and say oh well you know that that's twice as important as this other one so I'm going to give it a point seven or of the objectives this is the most important objective so that's that's fifty it's ridiculous the goal of this is to provide feedback not only to the individual who owns the okay are but to the group and the company ultimately it is far less critical to get these down to the second third fourth decimal point I always felt continue to feel grades don't matter except as directional indicators of how you're doing if you're spending more than a few minutes at the end of a quarter summarizing your grades you're doing something wrong the work should go into doing delivering on the okay ours not not grading them on growing traffic we said we'd launch three features that have a measurable impact on blog or traffic we lon - each of which had a measurable impact so that's good but we said we were going to do three and candidly to be totally honest we probably should have been able to do four or five so so this this could have been quite a bit better on improving bloggers for handling by the end of this quarter in which this this one was graded we had not proceeded past the mock stage was pretty sure we knew how we were going to do it I just hadn't prioritized getting engineers to build it out so that was that was a big mess on improving bloggers reputation um not least of which because so much of this was it solely on my shoulders this was a much easier one to to deliver ah so let's talk a little bit more about grading as I alluded to on the prior slide simplicity is key grade each key result then average them up to give yourself an objective grade and then average your objective grades to give yourself an overall grade for the quarter if you like I don't altom utley think that much matters but it is helpful to get a sense of you know where you are I think in my best quarter I hit a point it was north of 0.7 I don't think I ever had point eight averaged across all of my objectives pretty sure I'm pretty sure that's right they might have been a point seven eight somewhere there abouts that was a phenomenal quarter and the grade reflected that I had a bunch of quarters between a point six and point seven not because it was overly engineered to get there that just was I was in the you know sort of sweet spot of I was delivering on more than I was missing but there was still a lot of room for improvement and there were probably two or three quarters where I was below a point six where I was absolutely not delivering to a standard I felt was was good high scores are possible right pointing to 1.0 again if it's binary and you did it you get a one Hey high scores are possible they're unlikely you know I was I was going to say they're as unlikely as my kherington rejoining TechCrunch but then he did last night so sort of a maybe a bad example I guess um a when it happens it's exceptional be proud of the accomplishment be absolutely eager to share with your colleagues your manager your team what-have-you but I think the converse is probably more important to focus on which is the low scores right 0.4 below it's not failure right one of the reasons when when Steven Levy talked about how this became so embraced so early at Google was that this was data the point for the point three the point to the zero it's not a failure it's figure out what to stop doing or what did you learn what did you not know when you said you would do that thing that prevented you from delivering it well do you have any control over removing that obstacle in the quarter ahead or is it a moveable if it's immovable go around it figure out either don't focus on that objective anymore or figure out different ways of achieving it the scores ultimately benefit everyone in the process more by what they help you know what not to do what to change what to continue to do more of that that is ultimately where they're most helpful company-wide scoring reinforces the commitment here you your commitment individually your commitment as a CEO to repeatedly grade at the company level at the individual-level helps everyone understand that this really does matter and so you get into that rhythm it's just a byproduct of doing your job it doesn't take a ton of time but over the the quarters and years they become extremely useful so the company-wide process you're not unless the company is only four or five people you're not going to be grading everyone's okay ours in front of the entire company that's just that's impractical but it is practical that at least once every quarter sometime after the beginning of the next quarter that the company is going to get together and they're going to discuss how they did and the owners of those individual ok ours sorry of the team ok ours so in a larger company that's the SVP you know in a small growing company it might be the you've got the head of product the head of engineering and had a head of marketing maybe or sales are each going to get up and say ok this was the this was the product ok our for last quarter here's how we did we got a point six here's why we got a point six it's important to explain to everyone in the company why you got the grade you did and what you're going to do differently in the quarter ahead this calibration ends up being extremely useful for keeping everybody honest insuring the company's as transparent as it can be but then also saying ok what did what did we learn for me that was always the most important part and then at this company meeting the last part is setting the the upcoming quarters ok or is now often there's continuity from the prior quarter to the next quarter sometimes it's you know we put that project to bed we are there finished it shipped it and it's done or we learned that that's not what we want to be focused on moving forward now we're shifting our attention to this other thing let's talk about how to implement for a moment at Google we've and we eventually built our own system it's not a complex system it's a bunch of text fields with text boxes next to the text fields where you can enter in your grades and then the most sophisticated that it gets is there's a button you can push that says average my scores and it does an immediate roll up of your key results and gives that objective a score and if you want to get even fancier then you can average your objective scores to get your overall score for the court that's not something we make available to the public so that's obviously not a tool you have at your disposal but cars to me that I think some relatively low-tech options are just as valuable Google Sites you can create a page a sites page for the quarter and each person can add themselves to that page where their name becomes the their quarters objectives and key results just a bulleted list very simple and then at the end of the quarter they can go back they can edit their page and they can add their grades next to it pretty low-tech slightly more flexible might be a wiki you could use Google Docs or Google Spreadsheets I think that starts to get a little unwieldy particularly as the company grows but it is potentially a way in which you can do it the one benefit of using something like Google Spreadsheets of course is that if you've got a lot of people the ability to calculations across all of them to get sort of directional indications of by team by by Department what-have-you might be really useful at least as important as the tool you use is the communication you do about the commitment itself what I mean by that is this needs to not be a half-hearted effort it needs to be clear to everyone in the company that for at least a three month trial period ideally longer that everyone in the company is behind this idea at Google it was always clear not only because you could see leering Serge's okay ours it was always clear that from senior management all the way down okay ours mattered when I was in the product team I to tell you how much of an impact this this next email had I can tell you that when I was putting these slides together over the last week or so I knew when I got to this point in the presentation I was going to have a screenshot of this email because over two years later I still remembered the email and the email is one from Jonathan Rosenberg who used to be the head of product at Google still very close to Google and advises a bunch of folks in the company on a number of different things but every quarter he would send out a version of this email and this was the one it was particularly bad which is why I remembered it but the email that he would send out every quarter within the first few weeks was a public shaming of whoever on his team had not updated their okay ours for the quarter and this particular quarter he didn't send it out until the beginning of November so it was already about five weeks into the quarter and I'll just read to you a little bit our performance on okay ours this quarter was poor as of today nine PM's have still not posted their q4 okay ours out of several hundred that's a full month into the new quarter the offenders are listed below and then right below you can see right here he has he has publicly shamed my colleagues two friends of mine are on this list as not having taken time to tell the company what it was they were working on he goes on to say I don't understand why it's so hard for so many of you to take just a few minutes and list your priority for the quarter so last paragraph for now I recommend we all torture the offender's below by asking them in the halls why the heck they're late on their okay ours for those of you who find the previous statement offensive I'll be happy for you to chastise me as long as you're okay ours are posted and current so the key here is when you're on a team and you receive an email like this from the person ultimately responsible for your in this case the product team but this could easily have come from the CEO this could have easily come from you know if your engineer the head of engineering I know this matters right and as an individual I never wanted to be on that list I never wanted the person who was responsible for my team to identify me as as as holding the rest of the group back pretty powerful and simple so here's a timeline we're coming into the beginning of November a couple of thoughts on things that are really critical I think in the next month I think there are two things that matter most for a company thinking about embracing okay ours for a three-month trial first is you got to be thinking about what your q1 objectives are going to be you probably also want to be thinking about annual okay hours but but tactically thinking about what are the what are the immediate things that you know we're going to be we're going to be worried about if I'm the CEO I'm thinking just about what are the three or four most important things for 2013 the other thing that's really critical to be working on right now is deciding how you are going to capture what the okay ours are and how they're going to be shareable inside the company this process will fail unless everyone in the company has a way to see what everybody else is working on so I think it's critical over the next four or five weeks that you figure out whether you're going to use a wiki whether you're going to use Google Sites whether some engineer is going to build a really simple rails app that's just going to have a bunch of text fields and you know connect to their username and whatever right keep it simple don't over engineer it initially it's always easy to migrate to something more more powerful down the road I think the key here is just know what it's going to be so that if there's any kinks to work out you get them well it worked out well ahead of the end of the year now into December I think the company-wide objectives have got to be communicated for q1 and for the year that way everyone else in the company you also by the way have to communicate what the hell okay ours are so that there's a general understanding within the company and by the way this video is being recorded is available for anyone in your company to watch so if anybody has questions about what it is you can just as easily share this video with them communicate annual and quarterly q1 objectives for the company now it's up to the individuals in the company at the end of December by the very beginning of January to have their personal ok ours drafted and that should be at least one draft with a meeting with the manager and at least in iteration to get to consensus this is that negotiation that John Doerr was talking about company-wide meeting probably no later than about the second week of January where the company-wide objectives are presented and if appropriate depending on the size of your team the team objectives within that company are presented now throughout the rest of k1 important that there's individual meetings happening between individual contributors and their manager at least once in a quarter it doesn't need to be overdone doesn't need to be weekly check-ins but at least once or twice so that there's a general point in the process where everyone in the company has gone through that exercise of saying you know this is what I said at the beginning of the quarter I would be doing by February 15th am I in the ballpark if I am fantastic keep doing what I'm doing if I'm not I need to now recalibrate do I need to stop doing certain things so that I can deliver on the objectives I said or it where am I going to miss and what can I stop focusing on entirely so that I can at least deliver on a few that are really important and then finally you repeat this whole process so that towards the end of March now it's up to each individual to start thinking about grading there okay ours team owners start start coming up with the team scores grades and then the CEO leadership team are going to grade the companies okay ours and in the beginning of April there's going to be a company-wide meeting where you grade Q ones okay ours explain why you missed or hid and what you've learned about what's going to happen in q2 and this just this process then just happens on a rolling quarterly basis all right believe it or not we are at the end of my formal presentation here so I am going to pull up the QA that have come in from from you all online and in the room so thank you for that let's see what kind of tools do you use Luke asked for managing and communicating okay ours well we talked a little bit about that I've seen it go through at least three revs at Google right today we have a very simple tool that is really note no more than I log in with my username our Clow and I click on the quarter and I add type in an objective and then under each objective I have a button to add a key result and I add each individual key result I'd showed to but again its internal its proprietary so it's not the sort of thing that we would share in a workshop here but you get that you get the idea of what it is very very simple at the end of the quarter I go in and I click the score button and that's when I am now just assigning a number in the text box next to each key result and as I mentioned earlier then I have the ability to average them really sophisticated math going on there I think in most of the cases for folks that are here for folks that are watching online I think something as simple as a wiki if you use a wiki internally I think Google Sites would probably be enough I you know I think this is the sort of thing you could over engineer pretty quickly the keys are not in my opinion are not the tool it's the commitment to being measurable the transparency that everything is available internally to everyone in the company and the grading process itself I think beyond that I think that the tool matters less what's the cadence for planning for the planning portion of an okay our period for example is it all one week or spread over multiple how many meetings do you have with whom in attendance for reference at puppet labs has one exact retrospective meeting than an exact planning meeting and team meetings and executive you meeting all in one week I think each company will likely be a little bit different in my own assessment if I were if I were implementing this at my own company tomorrow I want this to be a relatively lightweight process I don't think it necessarily needs to be in a meeting I think this can be the sort of collaborative drafting that can happen where everyone can take a look at a doc make some suggested changes then have a meeting to wrestle over or negotiate over key points so that there can be some consensus I think if you run the risk of meeting over meeting the the the process which which I think could be problematic because then that's that's reinforcing the wrong thing that the point here is to be doing the work not meeting about doing the work so what would I what would I say there I think in terms of grading I think it's important that people have that mid-quarter check-in so they have a sense of where they're at I think there's probably some I think diligence maybe an hour towards the end of the quarter and then a final meeting at the very beginning of the following quarter to come up with a grade in terms of planning for the quarter ahead I think often that's a natural byproduct of the grading process where I think about what did I get done last quarter are there things that are rolling from last quarter into the next in which case that's you know sort of a copy and paste operation are there things that I wanted to be working on that I that I deliberately chose not to do in Prior quarters for prioritization purposes and add them in I think cramming it all into one week is probably a little too tight I would I would say at least bound it from a week before the end of the quarter to a week after the end of the quarter maybe a little bit of a buffer on either side I think that will ultimately give you enough room in which to to give people the feeling that it's it's aiding in the planning process not removing time from their schedule so Jeff asked how do you ensure okay hours roll up or or decompose down the org structure trickle down might be better decompose suggest that they go away but I think I know what Jeff meant there um so how do we do this um I talked in the deck about the sort of virtuous cycle and I really do is not just I believe it I've seen it work the way we ensure is used to be Eric when I was when I first joined Google now Larry is the CEO there is a quarterly meeting and it is given by Larry and Larry says these are the most important things the company is working on these are the grades for what we did last quarter and now I'm going to have Vic and ORS and Susan and Jeff and each individual group head get up and explain how they did last quarter and what they're working on next quarter now that has a really important clarifying point I think most people particularly in startups want to be working on the most important things it was always very helpful for me as I was going through the process of drafting my okay ours that I would I would draft them at least in draft form and then I would attend the company okay our meeting and I would do a reality check against the company okay ours and would want to know can I see in my okay ours the company okay ours you know is what I'm focused on reflected in what the company is most focused on think about you know the the punter the quarterback the running back is what they're focused on reflected in the GM or the team owner I can tell you you know at one extreme there was one quarter when I was running the home page at YouTube that I owned two of the company's five five or six okay ours that was terrifying because I I didn't just my o Kerry's weren't just reflected in the company who cares they were the company okay ours that's exciting but it's also for a company of at the time you know about 30,000 employees a little scary when you're you're the your your running the team that is responsible for a significant proportion of what the company deems to be most important at the time the reason that was a helpful process was if I couldn't see the company okay ours in my individual okrs then I would you know that's when you take the the red pen to your personal okay ours and and revise Jeff asked cadence for checking process progress against okay ours I different teams will do it differently you might it might be more natural to do it every few weeks particularly in larger teams I think it's probably realistic that each individual is going to have to own the discipline of managing themselves and and maybe having only one or two check-ins and a quarter with their manager specifically against okay ours so I was asked was I using okay or is it blogger before I joined Google to clarify I joined Google in oh seven Google acquired bloggers in oh three and I was not obviously part of the blogger team until late oh eight by that point blogger was you know five six years under Google's ownership and and had really embraced okay ours so sorry if I if I didn't make that clear at last question unless there's any others here in the room I see several nodding heads that you know they've been jotting them down on the dock so that's great at what point in a in a company's evolution should the management team Institute ok ours is it five people ten people fifteen more easy answer as soon as possible the sooner this is part of the DNA of the company the sooner this is accepted as soon as this is part of the normal rhythm that the company gets in the better off you'll be you are there is always a point at which you can do this obviously I'm hosting this workshop because I believe every one of you should have some version of this approach in place at your companies so even if you've been around a year - it's not too late but the longer you wait the longer you are going to have habits form within the company about how things are done and the more inertia you're going to have to overcome so even if it feels a little artificial if you're a company of five the discipline that it brings is immeasurable I committed to doing okay ours even though at the startup lab right now I'm a team of one now technically I'm part of Google Ventures it worked a large and growing team and I have a manager who I check in with periodically on progress but for me the discipline of looking ahead and saying these are the things I'm going to focus on this quarter these are the things I'm not right and then evaluating how am i doing that it's hard for me to imagine doing my job without this discipline at the end of the quarter it makes it unbelievably easy to summarize what we've done and and and whether we did well or not and I say we I mean me the for the startup labs so how many workshops did we schedule this quarter one of my objectives was launching the video archives which we launched you know in the second week I had been an objective in q3 I'd done most of the heavy lifting in q3 couldn't get the trigger pulled on getting the archives up so it slipped into queue for my grade for q3 wasn't as good my grade for q4 was less about launching because I knew that was going to happen in a week or two after the start of the quarter was more important to focus on okay now we're going to measure how much traffic are the archives getting how many playbacks are there how many people have asked for access to the archives that's the sort of thing that starts to give you a sense of how that mentality can can really be embraced and and drive action and an accomplishment so with that I'm going to wrap up let me again thank you for taking the time I teased you about not telling you how long this was going to go i budgeted 90 minutes I think I'm at about 88 so you know hey I'll score that at one on the Y okay ours please fill out the feedback form for those of you here in person or watching online the the data is critical as you know not only from this presentation but from others we are very data-driven as a as a group not just the quantitative score that gives us a sense of you know how we're doing on that 1 to 7 scale but also the qualitative what takeaways did you take did you get from this event knowing what resonates helped me when I think about planning future topics really know what to focus on and how to steer presenters to so that's that's really critical topics again as I said at the beginning of this workshop if you've got a topic that you would like to see us cover I want to know what it is I want to I want to hear from you my job is to find the expertise and experience at Google and make it available to you it's a great job I love doing it but I'd rather not be prescriptive and tell you what I think you should know I'd rather know what you think you need to know so that I can make that available stay tuned on the announcement list we've got a lot of stuff coming up for the balance of q4 and starting to do a lot of planning for q1 and really excited about some some changes in approach and and format so looking forward to seeing you again online in person until then thank you again see you soon bye-bye

Channel: GV

Views: 1,076,048

Rating: 4.8958077 out of 5

Keywords: google, management, objectives and key results, okr, startup lab, OKRs, leadership, john doerr, Strategic Planning (Field Of Study), startups

Id: mJB83EZtAjc

Length: 81min 50sec (4910 seconds)

Published: Tue May 14 2013

Reddit Comments

[u/KrustyFNL]

Google's original post

[u/[deleted]]

If everyone is to set personal OKRs within the company, what sort of objective might someone with a fairly monotonous workflow set? – a call center agent for example

[u/AceyJuan]

TL;DR? I don't have 80 minutes for this video right now.