Surge 2013 Speaker: Bryan Cantrill

started here okay hi good afternoon I'm Bryan Cantrell I am the SVP of engineering at giant I've got some some mixed feelings for even though I love Serge do II I I am in the douchebag track so I feel at some point I'll have to go on a technical rant and hopefully write some code on stage just to remind you that I actually have technical chops so I'm Alex I'm a little I'm a little embarrassed for being here in the douchebag track but hopefully you can bear with me um so this is the the title of my talk is purely leadership without management scaling organizations by scaling engineers largely because I knew if I wanted to get a talk accepted in the douche bag track I really need to come up with a very douchey title so that was successful because I'm here here's the actual title of my talk software engineering no management toxin or cancer reasonable people may disagree um so let's uh let's explore that a little bit so let's talk about an engineering organization a little bit because we know that when we add people to an organization we had engineers to an organization that has known drag people love to talk about oh yeah we need to add more people we need to add more people of course but when we add people we add drag to the organization we know what that does all right I mean you've got a certain amount of constant drag just associated with being a human which is really it's really annoying that we're all humans but and we all require we require sleep and we have kids and we get sick and got to get on dog's die and stuff like that and and which is all fine I mean but that's just that's part of being a human so you're gonna get a certain amount of drag that's gonna come just individual Drive which is fine but then you also get communication related drag and organizational energy related drag that actually grow nonlinearly with people that is say that adding people makes it worse and you know I feel this is one of these things that we have known for a very long time people love to talk about the mythical man-month as if they've already and um you don't have to I'm not gonna embarrass you by asking you to put up your hands I know you like to say how many say they read the mythical man-month of course every hand goes up yes whether you've actually read it or not and the reality is of the mythical man-month a lot of it is actually dated a lot of it is very specific to os/360 some of it is very kind of like weird and sexist like about what the secretaries should be doing in your organization on but there are some real truisms there that are actually important and I think that the truest truism is Brooks law Brooks is law which is that adding people to a late software project only makes it later this is as true now as it was in the nineteen sixties and I feel that this is something that we have not internalized it's like why if if everyone agrees that we've all read that book and even kind of the the the pointy head types all acknowledge it why do we keep adding people or trying to add people too late software projects only makes them later and it seems that most of the thinking around how do i scale an organization comes with trying to repeal Brooks's law or minimize the effects of it they're trying to deal with how do I deal with now I've got all these people and they don't communicate so I need to put all these structures in place and I need to all this middle management and so on that's how we kind of think about scaling but all of that thinking really only makes sense that it's a the thinking about these structures we put in place to scale an organization really only makes sense if all software engineers are essentially equal I've got five cogs I need to add three more cogs to my machine oh one of the cogs broke I'll replace it with another cog which of course you know there are people who like to fantasize about this but that's not the case software engineers are not equal and again we know this but let's just be sure that we all know it know it and know what its ramifications are the best software engineers are at least at least an order of magnitude more productive than your merely average engineers how many agree with that statement very good um believe it or not there are some people that don't still and actually Steve McConnell I've you know code complete I've got kind of mixed feelings about it but I don't think I would you liked about code you played Steve McConnell to his credit has done really really thorough research on this which is another way of saying he's been very thoroughly trolled on this issue um because I mean my attitude is like yes sovereigns are quitting on equal the best are clearly an order of magnitude more productive than then to meet your engineers and off if you disagree because that's obviously the case but McConnell no no he's being trolled and McConnell has gone through all of this research on sovereign earing research and it's really I mean this is really kind of incontrovertible even with folks that are trying to study this formally which has got also with the problems associated of course and Makoto calls does the 10x effect but this probably understates the multiplier so the thing about top engineers the top engineers are 10x more productive great um they don't induce 10x more drag right unless I mean top software engineers are now I'm not sure what kind of person would have to induce like 10x more organizational drag I guess like a like a gossip or somebody who had like smallpox I'm not sure like the I'm not sure what exactly that would look like but sue me that's not true of these engineers these engineers the the the amount of drag that they induce is is effectively linear it's so different than any other engineer probably actually even less because they tend to be better communicators and better natural leaders so if this is true if we accept that software engineers are not equal the best are an order of magnitude a factor of 10 that I can actually deploy I can deploy either an IV bridge or a raspberry pie that's what you're looking at right it's like well I and I figured out how to do this with 10,000 raspberry pies it's like well why don't you do it with one Ivybridge and be done with it it mean to kind of put it in more technical terms so if that is the case then the way we should scale an organization is by finding the fewest number of best people a best software engineers only top performers an organization that consists of only the best people you've ever worked with and no one else as many of those as you can find yeah okay that's all yeah yeah yeah we we hire best people yeah that makes sense of course of course like who knew I had such like motherhood apple pie makes sense well ok it seems obvious it seems like an obvious aspiration to build an organization that consists of only top engineers everyone would agree with that and yet if that aspiration is really that obvious why do we put in place the structures that demotivate the superlative engineers why don't we put in place the things that actually motivate them there's a fundamental misunderstandings I think about what motivates top software engineers and what D motivates them and how you should structure an organization to to attract them and not repeal them repel them rather how and of course the big one and I'll get this to the end how does one actually find them because this is kind of the classic excuse yeah yeah yeah the best or a factor of 10 more but better than yeah blah blah blah but they don't exist anywhere I can't find them they're too expensive so I've got a higher these five over here it's like well huh and the UM the reality is you you actually can find them I mean it is hard but you can't find them if you have all the other structures in place all right so let's talk about the motive a little bit um just to give you I shouldn't now might be a little a good time to give you a little bit of my background so um I spent I spent 14 years at Sun Microsystems may she rest in peace I rode that vessel to her watery grave um I I spent and and and I know I know I know I know you're thinking like am I about to sit through another cantrell Oracle talk because because because I got mixed feelings about that on the one hand like I've seen that on YouTube and I'm a little titillated I could and yes I did spend five months so as you may know the Nazis invaded Sun and as a member of the Benelux I did ultimately flee um but it it did it took me five months to figure it out it's like well maybe we will take the the Nazi military strength and combine it with the Belgian idea of fine wine or beer so no no that's not it's the Nazis man come on like having you have you heard of Poland I mean get out of here um so Poland being PeopleSoft and you know the really funny thing is that actually wasn't as a joke I actually I didn't do that for laughs um that's that's actually sad um and yeah yeah yeah yeah I went there Nazi allegory but just just one thing to be clear about Oracle and the Nazis so yes it is true that that that every descriptive Oracle ends in Nazi allegory but if it doesn't if you describe Oracle without ending a Nazi allegory you have left some understanding of Oracle on the table you're hidden you haven't fully conveyed the essence and indeed indeed I I have argued in will continue to argue that if you were attempting to explain the Nazis to someone who had never heard of World War two but was an Oracle customer you would actually explain the Nazis in Oracle allegory it's like I'm imagine that Larry Ellison control the country oh Jesus thank you my god the license audits yeah yeah noise ins on it that what we called it blitzkrieg it was really really ugly um anyway you should look out look out World War two is terrible love people died um alright so I spent five months in the Nazi Empire that was kind of entertaining in some kind of after the fact left that and then came to join um we'd start up and and built a team adjoining and you know one of the things that I saw at Sun so we had many many people and many talented people but many many fewer people actually doing all the work and and and not just like fewer people that you've heard of but lots of people that you haven't that were terrific engineers that didn't get elevated in there within their own organizations because they they weren't interested in abiding by these this kind of middle management rules so I think that even at Sun and I think Sun did better than most of the large companies we didn't emphasize the motivators enough and our in particular had the the D motivators but so what are the motivators um the the primary motivator for superlative engineers and this again is a misunderstanding the primary motivator is mission purpose that is what point what's the prohibition ears are motivated by we are doing what we are doing because we believe in it we want to this is how we serve humanity by delivering utility to humanity this is what we are about this is in our marrow and people don't understand that that that is the most important thing the why of an endeavor if you can't explain why your company exists don't expect to ever find superlative talent ever it doesn't matter how much money you have oh you know give money and and masseuses and and sushi and all this other that is not what motivates superlative what motivates the prologue talent is mission why the soul that is what motivates it's a part of talent so you need to be able to very clearly explain that why and you know for all of its other flaws there's a wide that I stay at Sun when why was I in the so when it was so queerly listing and I had cold water up to my hips it's because the Y was always actually clear that Y can be tremendously powerful and people will stick with you through remarkably thin times if the Y is clear emphasize the Y that is the very first thing to do so if you are trying to to Rick and I don't care if you're trying to recruit them out of a university or of a senior engineer they're motivated by the Y the larger sense of purpose that purpose goes kind of hand-in-hand with the next two things the team the actual people are going to be working with and the technological problem we want to solve hard commercially relevant problems with people that inspire us achieving a mission that we believe in that's the top priority of French Cheers note I didn't even mention compensation now I would say compensation is no higher and no lower than the next thing you need to compensate people fare away this is not an excuse for not compensating people fair way and more than fair way you need if you're expecting professionals you need to you need to be willing to pay a professional wage but that doesn't mean any price that means that you're going to pay people fair way and you're going to really emphasize the Y and the team and the technical problem and you will find the the the right people are attracted to that the compensation is not nearly the issue that people think it is I have beat out Google and Facebook many many times hiring engineers and with Google and Facebook offering redonkulous redonkulous amounts of money at one point on if they're still doing it Google was simply offering massive stock grants not options and if you're a startup I know I can always go compete with options right because we can if you'd agree that compensation is important we can talk about growth potential it's impossible for a start-up to compete with a stock grant because it's like I mean Google is just like giving you a you know they're giving you a million dollars worth of the company over four years of course yeah by the way you need to be shackled to that vessel for four years so but I beat the I've I have beat out Google even when they're offering such terms how because of the why and the team one thing I began aren't they're still doing this one of the things that was amazing to me about Google's you would never know who you're working with but don't worry it's Google we're all brilliant it's like yeah you're all brilliant but some of you were I just know like who I'm working with and when I came out and actually actually the and I I mean in terms of like the mission and the team the technical problem in terms of what will be the actual top priority these things are so interrelated that the fluctuate terms of why I came to Sun Microsystems when I came out to interview at Sun I was convinced I did not want to work for Sun every exposure I had with Sun was like dealing with some sort of Soviet DMV it was kind of this very kind of like sleepy bureaucracy and I all of that changed on the second I met Jeff on work and Jeff was this just bolt of lightning and all of a sudden someone who was unbar Nishtha their opinions someone who was who absolutely believed in what they were doing I came to son because of Jeff I don't got that for a second so the team alone can trump a lot of these things I would not I would not have matriculated at Sun had it not been for Jeff and the technical problem it um and again like see these these things are are all interrelated um I would say that also if you focus exclusively on compensation and you don't emphasize these other things you will find you get the wrong kind of engineers you get engineers who by the way like if someone else comes along with a better Dumber offer they will leave because they don't believe in the mission don't believe in the team they don't believe in the technical problem so all this is not just a compensation isn't important it is very important just understand that it is not the most important thing it is the fourth most important thing no higher no lower all right so enough for the motivators let's talk D motivators and now we're gonna take a tour through my my lifetime of middle management mediocrity um so when I was approaching when I you know clearly like III was in the the kursk there and Oracle and one believed and was looking around and was talking to giant my major apprehension was oh boy me as a VP of engineering I would talk about the animals walking upright and every VP I have ever known at Sun without exception was a jackass or became a jackass when they became a VP now that's that's a reflection on son as well it's not just that that's not that's not endemic but there was not a single person that I could look to in my entire 14 years of history at Sun that was inspiring in terms of the way they manage folks and I was very depressed about this because I'm like I don't know how to do this i-i've been raised by wolves I I'm an orphan from I don't I don't know how to actually manage a group of people and then I kind of disposition like wait a minute I dunno what not to do but they know I know a lot of what not to do I can rattle off what not to do so I'm gonna do an experiment my experiment is I'm just gonna not do what I know I shouldn't do and I'm gonna hope like a sculptor I'm gonna hope that what I'm left with is is plausible and I'm sure I'm making my own mistake so we're making our own mistakes at joint but I have been sure to not do on the things that I saw that that we're profound deep motivators at son and the thing that that I think is is crushing to me is that when I look out at these demotivate errs I see them very widespread in our industry and actually son was much better than most and there are some things that seem like they're like obvious just kind of like good things to do to scale an organization that are profound deep deep motivators so that's I mean that's the real reason why I wanted to give this talk it's because I that there are some of these things I want to get off my chest because I think that there's some truisms that aren't being spoken on and I I think this is especially true for those of you who are similar to some more position to join it where you kind of at this inflection point where it's like okay now we really want to be able to grow the organization when scaled your organization and now we need to get and you know you might hear if you know if you're a VP of engineering you might hear you know CEO say this you might have kind of hear that your manager says like we need to get more formal about things like time to grow up a little bit time and these are all euphemisms for toxins and cancers and III want to kind of give you the strength for those who are in those kind of organizations that's kind of you know the 50 to 200 person organizations that are not this kind of inflection point I want to give you the strength to resist some of these structures that are incredibly corrosive and incredibly motivating so the and I'm going to start off with what makes you like the most controversial but the first profound demotivator is a formalized annual review and this is one of those things that just doesn't seem like really formalized review like that doesn't seem that bad well I'll tell you what's not bad feedback feedbacks great feedback is really really important and you know I would put formalized feedback like maybe an air quotes like a feedback that is I would say concrete feedback is really important don't just tell someone that you liked their work but it but explaining you know I really like the way you solve this about this problem that's great problem with the annual review is it's too formal and it's you know goes in your HR file like that's a thing why is that a thing why do we cut talk about your file like that's a thing that sound like a legal thing that's something the HR people made up to keep themselves employed we don't need to have an HR file and it's it's the wrong kind of feedback it's and it's the it's the just a wrong cadence annually you're gonna get feedback annually really annually wow that's a long time so basically I should if I want to burn the place down I should do it like immediately after my last review so you'll forget it by the time the fire finally simmers down you actually forget that I did that and we can talk about the things I did in the last 45 days and the in my expert I'm my experience I never once found an annual performance review to be uplifting it always pissed me off in some level it always pissed me off it often pissed me off because they there's this kind of obligation I can't it's an annual review so I can't just give you positive feedback I have to give you something to work on like what's this something to work on I'm the one he had like allowing people showing people where they can improve is always valuable but something that work on inevitably devolves to dinging you for something I gotta find something to ding you with I can't not ding you because by the way if I don't ding you I have to promote you it's like all right I guess I and this problem um and so what you what do you get dinged on you get dinged on your personality you get dinged on the things that you can't change are you an introvert you're too shy you're not talking enough to people oh are you allowed mouth you talk too much you're not shy enough you could use a lot more shyness I may have heard this one or actually what I heard is the so what I did what I used to get is so I had a manager that we nicknamed moody because of his galactic ass um I I was like the first day on the job like is anyone else notice it he's got an extremely large ass I'm like I can I can act like not like it like large man large ass but kind of like medium sized man galactic ass it was like kind of like oh um and my other colleagues were like new we had not noticed that my go okay this is gonna be well hi I'm here so we Nick in new booty and booty used to give us what's called the booty dance now I'm in the booty dance is where the booty would come in and booty would start to refer to you in the third person so well Brian well there's an idea that Brian is always technically correct oh my god who's it who's Brian I want to meet this guy oh me I'm right here I'm right here you can actually use the second person you ever gonna have to use a third person bug so Brian is always technically correct sometimes the way he presents his opinions does so in a way that is not necessarily sensitive and he would always kind of like flutter his eyes and look at the ceiling and the other thing was kind of funny about this guy is um he he would kind of like scratch his back against the wall when he did this he can't like a skin condition or something on his back and my lights would go off when he did this and because he would hit my light switch and he was have done this 500 times and over the years and years I reported to this guy which was way too long he did this happen many many times and every time he's like I can't wait to go out and he'd just be like frightened animal like oh my god like black it's a blackout or an earthquake or something it's like no no booty it's your back just turn the light switch on um and be actually just because I have to tell this we were had a huge All Hands yeah and this is you know like 600 people in the room galactic room and all of a sudden all the lights go holding was out and everyone was like oh god this is like a power outage every kind of like has that moment and I look back in the back of the room and there's bootie like scratching his back and he kind of looks around and there's this whole Bank of switches and he's like run and I like Wow bring that Bell that's impressive that's impressive but so the booty dance I hated the booty dance because the booty dance was it was it was just this it was a fiction that was invented it's like I know you're right you know you're right but I need to ding you so I'm going to I mean here to ding you and I it's it's really unfortunate that we feel an obligation to give to give negative feedback to people because by the way negative feedback doesn't do anything and if you don't know this have some kids and then have some more kids and trust me if beating children worked we would all beat our kids I am NOT morally opposed to beating my children it just doesn't work at all and and there's a reason it doesn't work because you don't learn anything you know it's like all you're doing is being you're being punished all it does is it instills fear it doesn't actually make you do the right thing it just walks you down you just do less and you're the only thing we actually learn from is positive feedback and we know this I mean and and neuroscience is finally figuring this out what animal trainers have known for a long time you do not train a dolphin with a shot-caller go to an animal show go to a marine show write a lot of fish a lot of fish and that's all it's positive reinforcement oh they're being bribed nothing not being bribed I was at work today and I'm getting paid for it I was not bribe to go to work that's part just it's just positive reinforcement so the kind of there's there's a natural kind of fixation on these kind of negative this negative reinforcement and it carries a very heavy burden these formalized reviews how many people have had to write their own review I want to like that wasn't so see that quite that many hands I think I that was just me like a good third of you so I could just like and say look at this for third of the room that was not supposed to be all of you god that's depressing um that is so so so so so deeply wrong that you're a technologist it's like hey by the way this is kind of hard enough over here without me having to write my own damn review and or in my case like you'd write a review and they didn't they'd it up when they gave it back to you so literally I had a manager who to only introduce grammatical errors in the review that I had written which is very crushing to read because and so in particular I'm reading this I couldn't read in this review it's like I just want to kill myself and be among positives it says Brian is a first-grade debugger I'm like I think you mean right here not grade and I made a huge mistake of telling my mom about this who almost died laughing and now whenever anything comes out Tegrity SEC O'Brien is a first grade debugger like mom just shut up so um these reviews are there they're just a moralizing they're a heavy burden you should never have to write your own review everybody wastes way too much time on it um do not do this and if this sounds like a radical opinion it's not this is not a radical opinion so everyone knows Peter damning the the seven deadly diseases of management annual performance review is on there so if you've got someone pressuring you to adopt annual performance reviews cuz that's what need to do to grow up you just go Peter Deming on their ass because you actually it's actually corrosive okay so along with that hierarchical titles so one thing that Silicon Valley did write is the invention of the dual ladder so you did not have to go into management it used to be back in the day that you would have to go into management in order to advance from salaried perspective one of the things that happen in kind of V in in the 80s maybe in the late seventies buts only in the 80s 90s is many Silicon Valley companies and other high tech companies develop a dual ladder system that allowed you to advance so that having to go into management which is great the problem is it's like well there's got to be a ladder if we're gonna have a dual ladder it has to be a ladder and this is what you end up with the member of technical staff and the staff engineer and the senior staff engineer and these are all rigid titles in fact when I first came to sign I'm like I was told to make a business car and I'm like well I saw some other guy's business card needs a staff engineer that sounds good so I guess most App Engine then I only of you guys later it's like no no that's like saying you've got tenure like you be a staff and staff engineer you may be a staff engineer after a decade of working here so you you had these very rigid ranks and the problem with these hierarchical titles is they create what I call the n plus one Butthead problem or n plus one problem I don't want to I don't put that on the slide but the n plus one problem is this and I don't care how enlightened a thinker you are that if you are at a certain title when when someone at the next grade up acts like a jackass it just deflates you and it said like wait a minute if if Johnny jackass is n plus one why am I not in plus one and I always actually counsel people that like you should actually instead of fixating on the one jackass that's at the grade above you just to make yourself feel better focus on on the best possible person at that grade who's probably been under promoted and just think yourself like well should I be the same rank as as Alan awesome the and so that's kind of what the way to try to think about it but the reality is that you always have this problem you always naturally think about the rank above you and when they're when they're moronic it just deflate cyou and means that when you actually promote people the if you promote other people you're never happy for them it's like being in minor-league baseball right minor league baseball no one is ever happy for anyone else because they're all trying to get to the show and so no one's actually happy when other people do well it's very that's a very corrosive environment and the same thing is true someone gets promoted you're actually angry why not me and when when you're when you're promoted every time I was promoted my attitude was guys it's about time meanwhile Johnny jackass has already been promoted to n plus one hierarchical titles are not uplifting they are corrosive adjoint we have one title software engineer with the exception of one engineer Jerry jelinek is a senior software engineer because he worked at Sun for 22 years and if you have a problem with that show some respect that's that so that I think you can make exceptions to this obviously but we are all software engineers that is the title and I think that that making that I think leveling the titles and any new but people have our have titles that are non hierarchical where they are leading different aspects of what we're doing and that's great titles that uplift people are terrific but hierarchical titles are not the corrosive um the mother of all demotivate errs ranking how many of you are in organizations that rank that right people how many of you have been in organizations that rank people there you go okay what more hands that's good actually that's good so this may be I'm hoping this is becoming less less of a problem so the idea of ranking is that you force an absolute ordering of engineer performance you've got the top that are at 20% the middle and it's always kind of comical to figure out like what they named these right so the I think at-at son did get better than most and named the middle like excellent is to say that Oh everyone is excellent in the middle and the top 20% is superlative and then you know the bottom is cancerous or whatever the title is that they use for those but the you have some top that you're gonna reward some middle that that you're gonna do basically gonna ignore and some bottom that you're gonna terminate and oh this seems like an obviously stupid idea but but because of formalized review companies kind of back into this well formalize review i have to formalize your performance and I'm gonna formalize your performance I'm gonna quantify it I got to give you a kind of a grade and by the way I can't give everybody a top grade because then I'm jockeying around with other organizations right there need to be a finite number of top grades and indeed so in terms of what this is known as is known as top grading at Amazon its stack ranking at Microsoft stack ranking may have certainly if it didn't if it didn't do it single-handedly exacerbated the decline of Microsoft stack ranking it metastasized at Microsoft and it and there's been a lot of a kind of a lot of interesting reports with that on this the system was born at GE ranking yank Jack Welch is the devil and and and you know Jack Welch is getting his right I mean he I think you look at his he'd there there was a time when he was lionized among American management I think we now actually see what that actually is Jack Welch should not be my lyin eyes should be demonized it was called ranking and rating at Intel where the the bottom 5% had to be termed and then I love this is like wow he individual dignity entitlement Wow that is yeah you have hired a sociopath to distract HR I mean he's like Oh delicious individual dig to the inside of it oh the irony I love it um I took Wow that was I that was renamed after a couple of years to something that was a little more cloaked but it's all the same thing it is it is mandating that your organization can have no more than a certain percentage performing at the top level and this is by the way good news self-fulfilling prophecy because that those are the only top performers you'll have left everyone else will leave because everyone else does your other top performance you're saying we can't possibly have 100 top performers in this organization indeed we can only have 20 well what if there are 100 top performers 80 top performers gonna walk out the door as a result of that now if the way this was done at Sun was a little more benevolent in that the but it was also Dumber in that when you had the top ranked they had to promote you that was like that was like in the HR software so if you had the top rank you had to be voted so if you weren't up for promotion they couldn't give you the top rank so you would oscillate between getting the top rank in which you're promoted and then getting beat and then getting back into like the middle category where you would and and it was stupid because everyone's kind of Wingate well you know we can't give you the top rank because then we have to write you and we just revert you so we can't promote you again like okay are you hearing what you're saying because this makes no sense so again it's a self-fulfilling prophecy and it starts harmlessly it starts with this idea of like I've got this annual performance review as long as we got actually quantified this I think it comes by the way I mean one of the dangers and we see this at Google is that you know we engineers love to quantify things but you need to know what not to quantify and quantifying people is a very dangerous idea you need to be extremely careful when you quantify people Google emphatically believes you quantify everything including people including performance and that leads you down this path and it doesn't necessarily lead you all way down this path but it does lead you down this path and it's done to kind of level standardized level compensation and the problem is that the second you do this the second you allow the numbers to be formalized you're gonna get organizational jockeying because you can't have everyone be a top performer by the way that's in part because you have some some jackass director out there they will actually say all my guys are top performers even though when when they're off on some some Iowa inbred island that everyone knows is a is a jail sentence right so he actually will make all or she will make all their reports highest performers so you have to kind of divvy these up and quoted them quota of them around then you've got organizational jockeying and then you ultimately have the mother of all perverse incentives you are actually incentivized to not work with good people Wow so you start off with this thing that I'm gonna scale and now you arrive at a sparked a part where you are undermining one of the core motivations of top technical talent you are motivating them to not work with highly technical but with great people people that are better than they are I feel lucky that I were I am surrounded by people that are better than I am on any given issue that we are we as I say join everyday everyone leads and is led and and it's great to be led I'm in an organization and and to feel inspired by your coercion Wow how did you figure that out that was awesome thank God for you what a great feeling that is what a great feeling that is and to to incentivize someone to not seek that out indeed to seek out the opposite it's okay you seem like driftwood won't come into my organization and then you have like you got the junior engineer that's like I can't believe Bob is still here Bob's been dead for two years he's he's mummified in there it's like yeah yeah I know I know it seems ridiculous it is ridicu so know it seems ridiculous here's why Bob always gets the lowest rating so trust me you want Bob Bob is the reason that I'm able to promote you the Bob is a reason to be able to give you this heist so next time like next time you see Bob you give a little Bob a pat on the money there you keep keep rotting over here Bob not just smelling whatever um I mean to incentivize people to seek out driftwood or not and not get rid of them I mean it's it's it's it's so disheartening so deeply disheartening and again looking to a further than Microsoft to see this at its logical extreme but there are other companies that are not at that logical extreme but they are on that path it is it is absolutely the wrong path ranking is organizational cancer do not do it under any conditions and ranking leads a kind of variant of ranking is what my colleague mark cabbage calls the purple robes Club which I just can't help giggle at I'd say I have to visualize this so it's very tempting to kind of establish a select group of engineers usually with adjectives like distinguished or principled and nouns like architect or fellow and I would love to find a distinguished principal architectural fellow I'm sure they exist somewhere you can kind of mix and match these principal fellow distinguished architect and so on and we're gonna take the select group of engineers and and we're going to treat them specially they're the purple what they will Don the purple robes and they will engage and the secret rituals that no one understands it's got all the problems of ranking plus the added problem if you've actually technically empowered this group now you've actually created a pretty deep technical problem because you've got these technical decisions that are they're being handed down or ratified it by this select group of people that is not a meritocracy right at Sun these were des and you know I yeah I was a de at Sun but it was it's a it's something I would never actually Institute anywhere else because they the system I mean the system was so corrosive you had this Country Club that voted in new members and having seen the inside of that process people got in for all the wrong reasons and they were excluded for all the wrong reasons I mean god forbid you'd be a headstrong engineer who's pissed off the wrong person pissed off some some broken antibody somewhere in the system I mean god forbid because you would never actually be promoted so this is this is dangerous and again there's a gateway drug here architectural review boards it's a good idea to review architecture that's a good idea architectural review boards are a bad eye because they enshrine the idea that your job mr. senior distinguished principal architect is to go to meetings no one's job should be to go to meetings we build things that's what we do that's what engineers do meetings are a regretable artifact of needing to build things but what we do is build place um and so you know and at Sun we we had these architectural review boards and they they basically were when when they were powerful they were destructive very destructive finally they kind of became defanged and they were just kind of comical and it was by the time but yeah technology - called eat rice by the time dtrace came through the Architectural Review Board these are all kind of they've been largely defined we kind of knew how to game the architectural review board one of the things to do a classic trick these guys always look at the clock right you know the carpool is gonna leave at 3:30 so I got to make sure that I'm out of here so all you have to do is run out the clock and so what you do is you throw up issues that are like these little walnut size issues that everyone can wrap their tiny little brains around and just debate them incessantly so take some piece of your architecture you don't care about like yeah we just don't know whether to call this a foo or a bar it's like I think it should be foo I think it should be like and then make sure you're feeding both sides of the argument and and then they go I see we're out of time so I guess what that bump-bump bump-bump and I did fine and I will say that that one thing I did notice is that these things get got stacked with compiler people over time they're month there is some like weird thing where compiler people or if you have compiler people on your staff make sure you keep them in their little box stay but they've got a closet desire for intense bureaucracy that must be contained and and indeed when actually when dtrace came came before the architectural review board on the I weren't something very port like to power people they were very upset about he tries we're on TV this is you know this just reminds us a lot of awk I don't know my god that is high praise thank you so I that you know arcus such an inspiration to me I just you know I've I had Peter Weinberger signed my but and I died and um that actually hasn't happened but it only hasn't happened yet I actually would love him to sign my but um and then he was like oh wait a minute that was that was an insult you got oh my god you guys actually don't like talk believe it or not they don't like awk so we need a lot more awk and a lot less the architectural review boards and they don't like because it's way too you know way too effective I mean just kind of get things done like I can't do that get X just have a variable I got I got to clear it all right all right thing you love so another team motivator is non-technical management I think that folks that have not developed software can't resist channeling Frederick Winslow Taylor wins Oh Taylor is he did time studies in the 19th century where what he did is in revolutionary in a way where he went to what is basically blue-collar physical labor and would time how long it takes to do everything it's ok so it takes eight seconds to you know move you know this piece from this pallet to this pallet so I'm just gonna divide the number of hours a day out and like that's how many you should be doing in a day it's like wow ok and on the one hand it like it's a when you've got something that is rote and repeatable it makes sense to put that kind of structure in place software engineering is not rote and repeatable I mean it sometimes feels rote but the reality is if you are writing the same software over and over again you're not doing it correctly software engineering is not like building a parking garage software engineering you are you are fundamentally doing something new every time you cut code you're solving a new problem and if you're not solving a new problem then you're not applying existing software correctly um when you're solving a new problem there are unknown unknowns and there's some body of software where you've got a really good idea but how frequently does software take less time to write than you thought it would yah-ha-ha-ha-ha I would say that happens about really a 5% less of the time whenever that happens and whenever it's someone that has something on the team was anti-god problems could take me two weeks by God into it and like the whole thing just fell out it's beautiful I'm done it's like treasure that moment you gotta like lock that one in because it happened so rarely it does happen and it's so great when it happens the problem is it like you thought was good take you two weeks only took me picks you a day so you're excited with yourself that you spend the remainder of the two weeks building an elaborate monument to yourself in the office you think it'd look better like this or should I be pointing to the horizon like what do you like better um because I'm so awesome just because it happens very very rarely what much more happens like this should be easy I'll be done in an afternoon and it goes tail - f on your ass and it's a year later and you're and you're giving a lightning talk at surge so the software is there are there are always unknown unknowns in software yeah and the more non-trivial software gets the more unknown unknowns that are are that is say we don't even know what we don't know Thank You Don ROM self um no one technical management does not understand this fundamental I can't understand us in never develop software they don't understand the most fundamental truism sell of software a functionality quality and schedule you get to pick only two by the way if you're going to pick functionality and schedule I'm out of here and so is every other high quality engineer because I don't want deliver so it's kind of like cap you know it's like yeah you know I think ok no we don't have any partitions like oh yeah funny story you have to have partitions it's like oh that sucks yeah so so you got to have quality so you're actually picking between functionality and schedule and schedule is important it's important to get people visibility in terms of when you're doing things but you can really only pick schedule once you've got once you have stopped on functionality once you're and the unknown unknowns are all off the table then you can really drill down and in my experience it's very hard to do that with any accuracy very hard to do that and it's not an excuse for not doing it at all we're not trying to do it but but you need to clarify what what the unknown unknowns are and do not commit to a date until you know you can hit it and you're not gonna be able to do that until you've got a lot of these things off the table so non-technical management in my experience it's a recipe for date driven death marches a death march being one where everybody knows that the date is everyone knows this is wrong we're gonna miss the date but we're all kind of living this this little fiction we're still pretending that dad has a job just because he gets on a suit every day and he drives to the park and weeps um so you you need to not get into these death marches they're very very corrosive on a huge demotivator um so that that's non technical management now we go to the other extreme X technical management formerly technical management this is very very very dangerous because you retain engineers are managed a duality between arrogance and humility and you I thought was very interesting to watch Dean Krantz I mean how arrogant do you have to be to put a man on the moon very arrogant very Airy hey God I you got anyone arrogant monkey be like oh yeah oh yeah well I'm building a rocket ship you didn't see that one coming did you boom but but but but but but but the flipside of that is the humility because the arrogance is what gets you to do it and the humility is what gets you through we have to be humble these systems keep punching us in the mouth over and over and over and over and over again I was very interesting to hear I mean I'm so impressed with with Gene Kranz yesterday that profound sense of humility and what the mercury pad fire meant for them in terms of instilling that humility across the team that's what I heard from that it's like we needed to be a lot more humble we needed to understand everything and it is managing that that balancing at humility with that arrogant sloshed confidence the problem is that when engineers get promoted in the management it's like a humility all right arrogance here we come and next thing you know everything is easy because they've forgotten that nothing is easy they forgotten that and that even the simplest things are really complicated and the these sentences usually begin with guys guys guys guys you can just gotta fix this it's like dude I'm gonna punch you in the mouth do you remember we used to make fun of people like you back when you were one of us so ex technical management can be very very dangerous I had um III never again want to hear about the floating-point emulation on MIPS because I had a vice-president who implemented it once many many eons ago and time I had a technical problem he would he would tell me the story of implementing the floating-point exception handling on nips which is a story with no point other than to say that that I don't know what he was trying to say it was just he would drone on about it's like don't do this um the we need you need to be supportive but I think that the the challenge is that when you when you are actually running an organization you need to stay technical and how do you do this it would it's has to be done extremely carefully because you can't be on the critical path for an important project you can't be because you're having to go out to surge or what have you it's like it so you need to be careful about how you you use yourself but you need to find a way to technical speaking for me personally I try to do that by helping out with the bugging whenever there's a problem I try to help jump in and just help the bug because if someone helps you to bug something it's like that's never an right - and and it for me personally it teaches me a lot about the system there's still many aspects of our system that that you not no one of us understands all these complicated events but it's always educational for me to jump in and and help debug other people will do different things but I try to work on stuff that is important and valuable but maybe someone answering kind of the tale - fu well um but and I have found that for me personally it definitely has kept me definitely get that humility in place I mean I like that you don't actually quite this much humility like I can get my self-confidence up a little bit please BAM okay okay I got it so it's important I think you've only to maintain empathy another team o daters the inability to focus when you have a team with only Sproat of engineers all the world's problems become tempting and it can be very difficult to maintain focus focus is not saying yes focus is saying no if you haven't seen it you've got to see Steve Jobs as WWDC Worldwide Developer Conference 1987 keynote it is unbelievable it is a healthy strong jobs I means actually you forget how physically he how much he had eroded a big physical guy explaining all the projects at Apple that they're gonna kill to become focused and focus is saying no and it can't be rhetoric it can't be rhetoric at both individual organizational levels you've got empower and have the ability to say no or we're gonna do it later um I'll talk or see I don't want to dwell on I mean it's clearly like we all hate to Talat Arianism I actually because I said so I actually lifted that out of a bug comment we had someone new to the company who wanted something implanted and because I said so Mike Wow Wow that doesn't work that doesn't work at all that doesn't even or come close to working because I said something like are you serious um no because I said so and if any variants never at work do not present engineers with with solutions resent them with problems even if the problems our organization or economic we've got a problem here our biggest customer is really upset because this legacy bit of functionality has gone off the rails and we need help that's fine but like I need you to go debug this thing from eight versions ago because I said so that does not help and with and when you're short with a problem in my experience you can always almost always get to consensus based decisions among where I think he engineers with some except some exceptions and another demotivator is what I call shilly-shallying actually the dictionary great word 18th century word because sometimes decision-making become can be come to consensus oriented you become HP um and and you have to be consensus oriented about everything I'm now in my experience white thinking engineers failed to achieve consensus for one or two reasons one is it's a very small issue and boils down to personal style when giant erupts into civil war I am here to tell you that it is white space that did it it is the when we pick up arms brother against brother and brother against sister it is hard tabs versus spaces versus the guys in Seattle they do like this mixed hard tab spaces thing it's like that's like being bisexual man nobody likes you it's like I mean the gays think you're a traitor and the straights think you're on it's like no come on you got to pick one I mean you got um so so and and actually you can do because I said so with the what we've gone to a states rights model where every repo must be self-consistent and there must be a style tool that checks it but you can have I mean this is like not or not a rule that I can simply or anyone can simply oppose impose so you need to kind of either comment different styles to make a decision there can be in insufficient data in which case hopefully you can get you can get consensus on gathering more data or punting and doing the thing that forecloses on the least number of options it's like okay you say do this and we say do that but actually we're not at the decision point on either of these things we're kind of having a future land argument let's just let's defer this and let's not have endless meetings that's what what to avoid so that's kind of your collection D motivators here I just want to wrap up with kind of how you you structure and then importantly how you find top talent because one observation I would have is the best software engineers at every level of experience from your most junior to your to your most senior across all personality types introverts and extroverts are natural leaders and one of the things that I think I think people too often conflate personality with leadership and some of the best leaders I've ever known and ever had the pleasure to work with our deep introverts and people were like wow really it's like a really really really you should see the number of issues that that engineer exerted leadership on terrific leadership great leadership so the the don't can play personality type and in my experience top engineers always have natural leadership the good news is you don't actually need to put a whole bunch of formalized structure on top of them I'm we're flat mennenga near reports to me I break the organizational layout software it's terrific at one point we we had to actually invent middlemen that didn't exist just so we could actually like lay the thing out now that's because that reporting stork is actually meaningless it does actually mean anything that hierarchical structure does is is for us it's not meaningful what is meaningful is leadership and everybody I've got leaders all throughout the organization we have leaders all throughout the organization and again everyone leads and is lead every day so you don't actually need to have middle management pop management I can't scale can't scale well you mean there's a point to that maybe and you know you need to be kind of team focused and as you scale out you need to be careful when you do that but I would say err on the side of flatness not erring on the side of middle management structure because middle managers with nothing to do are very very dangerous that's when ranks and performance reviews and all the follow because they have had something to do every day because god knows is not going to build anything so in terms of finding engineers and this is now we get to the big issue bike okay yeah I get it of course of course everyone wants to build an organization with superlative engineers how do I find them I can't find any of them well the first place is the rolodex and this is probably this is the most reliable place all reliable the engineers that worked with you in the past that you've worked with and you know them to be good because those engineers and we've all talked to many people who seem so promising we didn't know they seemed very promising and you bring them into the organization and as it turns out like they just can't operate for probably complicated reasons and they probably need to get get on a couch and talk about their mother or something um but there are also people like this the people that you have worked with in the past those are the best people to find but rolodexes are finite now in some cases you've got that you could have the good fortune of having come from a vessel that is sinking you can just have to row up with a lifeboat and just kind of catch them they which is which is great so you know if you have people that are kind of coming from mendacious companies that's particularly easy but that isn't always the case great talent often is very happy doing what it's doing so you need to find those folks and really make the make the pitch about make the pitch around the why lead with the why we do the why follow quickly with the team and the problem um the second thing is is talented promising University graduates this is a this is a challenge I spent all I did a lot of time University recruiting it will erode your faith in the education system I already wrote it no not yet it's not it will be though you do a lot of really recruiting and it's it's tough yeah it is really tough I was banned for life from the University of Wisconsin that's a great story but I won't tell it now the but um you can't find these folks and I the Brooks way all either interested for trouble but but but many of the younger engineers that I have pleasure of working with our folks that we found at the university level so there's a very very important Avenue to pursue and the third one and one that may seem slightly less obvious but maybe not maybe you are already doing this is individuals that are in the community that are working on the open-source projects that you lead oh but I don't need any open source projects oh you should because that will allow you when you lead an open source project and you throw it out there there's a natural kind of magnetism and this is your farm system this this is your minor leagues and like that guy that's constantly submitting pull requests it's like and we in fact are that with the the the two of the guys we hired recently have been terrific terrific folks one came from from smart host community another from the node community and we all looked at the quality of their work I mean literally with one of them Josh Guillot ported KVM to AMD like my interview consisted of I needed just be sure that you did this work we I need like a three-minute exam to make sure that you didn't that you haven't like murdered somebody and taken their work but assuming that that's not like you're hired you are absolutely hired because that work is so hard and did it with EJ Fontaine and we hired recently and and TJ was working for a Lumber Company in Ohio and TJ is a terrific engineer with incredible promise but he was he was not in he wasn't in the Silicon Valley set he wasn't he wasn't attending an Ivy League university but what he was doing was he was contributing the node and he was doing some really interesting things in the node community and we look at it like wow this guy seems really good and he's terrific he's great those and I mean it's been a huge inspiration to work with both of those guys we've got a lot there are a lot more people like that so I would say really use open source if you've got an organization that is contemplating open source this is a major major reason to use it because the everyone agrees that finding talent is hard let the talent find you and open-source is definitely the way to do that so in terms of terms of further reading I'm about out of time the sorry the projector is kinda I guess it's the acro Reid has actually Acrobat has screwed up this is not just read like a ransom note by the way in terms of the font soul of a new machine if you haven't read at the top of your queue right now soul of a new machine have you read how many of you read soul oh my god soul is a it's a transcendent experience this is the the story of the development of data general of the the equipped system codenamed eagle unbelievable story beautifully written by Tracy Kidder on the recover effect with Hyman rickover I'm an admiral nuclear Navy on the flight Chris Kraft Gene Kranz mentioned Chris Kraft yesterday skunk works by been rich and I personally love empires of light Edison Tesla Westinghouse and Rachel around the world I love that because when Bill joy is called the Thomas Edison to the internet I say damn right he's the Thomas Anderson because he's not the George Westinghouse the Internet that's the guy who would actually build it so in terms like if you lionized Thomas Edison you need to read that book but the all of these books are about superlative people with unbelievable focused motivated by a mission and a team doing incredible things and I think we can be inspired by them and we can use that to build organizations that consists of only top performers thank you very much yep yep so I've got time for just a couple of questions yes well so I think that and that's kind of interesting about the open source pen is because when you have someone yes are there the question is what do you having people who are remote and in my experience I still like to work arm-in-arm with folks but we definitely do have remote folks at join who are incredibly productive top performers and they have more than the others kind of come out of the open-source communities because you've got someone's commune - open source community it's like we get you know their self motivator you know they know how to communicate electronically I mean you know a bunch of they're kind of like a bunch of unknowns that are off the table you did what oh yeah I mean I get a bunch of folks entered that are that are in that category um so III think that the my preference is still to work with people physically but you know that doesn't work everybody and open-source is a great way to find folks who can work remote yes yeah that's a great question about about mentoring and this is something that frankly we probably like a lot of organizations don't do enough of that we because the way the mentoring happens is well I should say we don't want to formally what I kind of believe in um sitting a junior engineer next to a terrific engineer that's a couple few years down the track and just having them absorb and sometimes that works and sometimes that doesn't and it kind of depends on personality type I think it also depends on the way you layout your office one of the things that that we have done in all of our offices is that we have got you we've got the stations where you've got a panel that's about this high and the height is very important because it allows you have your headphones on and they have your head down physically down when you're actually working but it also allows you to kind of engage in conversation easily the other thing we do is we're thinking a lot of people we use jabber incessantly and all aural actually are all of our office humor is actually online which actually thing is were important office humor is one of those things that you that you only really kind of get physically on we as an example we long ago had a chief security officer who was batshit insane and would say these absolutely ridiculous things so we have a bot that comes into the channel periodically and offers one of his words of wisdom which has been it's great I think you know you're having a tough day it's you know you're the maintenance window is being extended by another hour sometimes you need like just need to be reminded that a co is being a lot crazier if not a lot worse um so things like that I mean I I think that there are other ways you can kind of build that interconnectedness yes last question last question yes yes the question is like how do you reinstall in how you to rekindle absolutely and I think that one of the things that I took me a while to kind of figure out what I I think you truly can't over communicate and so one that we have a daily stand-up that's online on jabber it's called scrum for historical purposes but I hate a gel agile with the fiery white-hot passion of ten thousand suns the so we've got that daily cadence which is extremely important you get everyone in the room 9:30 every day everyone talking about what they did what they're doing today what their obstacles are and then we have a lot of discussion that's very important as that kind of metronome but that's not quite enough and so what we do I've got a weekly call with everybody I keep that as short as I can I tend to be about 26 minutes I always record it so people can go listen to it afterwards and that's an hour opportunity I think kind of like okay let's go take a little bit of a step back and let's remind ourselves why we're doing this it's a it's kind of a weekly cadence where you can kind of reinstall those values um and and talk about other issues that people may have a lot of people chance to vent or ask questions and so on and again I don't think we're doing it perfectly by any means but that's I I definitely see it as something that that needs to be done and I think that you can't over communicate that kind of stuff super thank you so much pressure

Channel: OmniTI

Views: 21,197

Rating: 4.9487181 out of 5

Keywords: Surge 2013, OmniTI, Bryan Cantrill

Id: 1KeYzjILqDo

Length: 63min 37sec (3817 seconds)

Published: Wed Oct 30 2013