SSAC19: The People Advantage: Culture Analytics

we're gonna get started welcome everyone thank you for coming once again welcome to the 2019 Sloan Sports analytics conference my name is JB Kalyan I am a first year MBA student at MIT Sloan and it is my pleasure to introduce our panel the people advantage culture analytics our panelists are Jessica Gelman CEO craft analytics group daryl morey general manager of the Houston Rockets and Nate Silver sata2 statistician author and founder of 538 our panel today will be moderated by Adam grant professor the Wharton School of the University Pennsylvania so our model air excuse me our panel today will last about 45 minutes and at the end we'll save some time for Q&A if you'd like to submit a question please do so via twitter using the hashtag culture data and with that I'll hand it over to Adam all right so we are about at the 10-year anniversary of the Michael Lewis article on the no stats all-star on Shane Battier obviously a big cultural force both on your team Darrell but also in this orbit so I texted Shane earlier today to find out what his ten-year reflections were and he said please thank Daryl for me for trading me when my wife was on bed rest we've discussed yeah he told me the one thing he wanted was me did not trade him in his last year but he's so smart he should have known we needed a trader we were we were transitioning from Yao Ming and Tracy McGrady to James Harden and what really bothers him is not that Heidi his wife was on bed rest what bothers him is that he was traded for Hasheem Thabeet actually true he's and I couldn't no matter how many times I explained there was a draft pick involved he didn't he didn't want to hear it so it's the question though was that good from a people I mean are you still friends I think I would say for sure so we took a lot of heat in that era this is maybe the more interesting part that we had a lot of player turnover and I would agree that setting the culture correctly during that period was extremely challenging I even told my friends hey get a jersey with your own name on the back because that's safer at that time and and the reason for that is you know we were not going to take which is the to be frank the more reliable more high probability path of getting a very high draft pick in the NBA so if you're not going to do that there's really only one way to still get yourself back into contention and that's through trade so even though we didn't like what it did for our culture in that period there was a greater good which was to get back to being a contender cool so one of the things I want to talk about this afternoon is how do you actually get inside people's heads so yeah I think we've all looked up to players like Shane and you know in my terms he was clearly a giver not a taker right who was constantly trying to elevate his team you have to respect the humility there and I think what's interesting about this panel is that you all have expertise on getting inside people's heads so Nate you do that with motors you did that a long time ago with with players as well Darryl you do that with athletes as well as with that with staff Jessica you do a ton of that with fans and customers and so I think one of the hardest things about all of our lives is trying to figure out well who is this person before I've actually gotten the chance to get to know them and so I want to try to figure out what we can all learn from each of your expertise on that Jessica I'd love to start with you so can you tell us from all the work you've done at craft analytics and throughout your career how is it that you go about understanding what fans really want and why are the Patriots so beloved I don't I don't think outside of Boston they're beloved I'm in here right because the rest of us do hit yes for good right because yet well I mean you should feel pretty good though Eagles won last year so Wow who let someone in there I know it's it's a great a great question and kind of the the whole background of I study psychology in college and always was very interested in trying to figure out how customers think and what is of most interest to them and I happen to start working in sports at a time when there was a lot more data that was available we you know we've had a lot of discussions today about that but the most significant is we could track information specifically to a person through email really to start and then it's it's grown attendance you can see when people are coming to games so there's all of this data about what our customers are doing and the other part which I think is often lost is what are they saying and so the and by the way you have to have both of those pieces of information one of the two or both can give you some insights a deeper insight that you might then go do more analysis on and I think the the qualitative stuff I would say when I was you know running various business operations for craft sports and entertainment after the first game of the season I would get the survey that we had sent out and it was often 40 pages of feedback and I would just read it and kind of say what was the experience that the fans had and it was just it was a lot of information obviously and but to happen I think fans in particular they want to know that someone's listening and so providing that information back but then from there if we heard that people that I mean this is a very specific example but that they couldn't hear the speaker was off in a specific part of the section we would fix it and you know we would do that type of work and then you can go much deeper into are there bigger issues from an analytic perspective that we need to be looking at to look for trends but for from that perspective always kind of starting with what are the customers saying and then what are their actual behaviors I think one of the the challenges that that we all face is that but in the software world would be called eating your own dog food okay saying okay I have all this expertise do I actually apply it to my own job so Nate tell us when you hire people what are the interview questions you ask and how do you figure out whether somebody is actually a person you'd want to work with you I mean you try to make sure they're not an [ __ ] how by having a conversation with them like I think the actual like but we exist at 5:38 in a universe of like a very small amount of people by the way I'm not someone who says that all behavior can be quantified or that you always want to use like the metric what I'm saying is like if you want to use metrics I can tell you how to do a good job with doing that to measure what you want to measure right we're hiring people up 538 you know what we are trying to measure it's you know we wanna know roughly speaking like how much good journalism does someone do and like there's not necessarily a great metric for that you can look at traffic but if you write an article about like whether LeBron James or Michael Jordan is the best player of all time that's gonna get a lot more traffic than writing about like crime statistics in Newark New Jersey or something for example and so you kind of you know in some ways basketball baseball other sports have very unique in that like you know you're measuring about 80% of what you want to measure or 90% I don't know right people complain about the oh the 10% that you can't measure you can't make your defense that way all right you can't measure spacing that way although you can increasingly right but like you're still like a lot further down the road toward what you want to measure than you are are in most fields look I think there is evidence that if you have a structured interview process that that tends to help and so making sure that you kind of have the same people conducting the interview to kind of ask people the same questions for the most part I think looking at kind of performance of people as they're progressing in the organization you know one thing it's a challenge working for a big organization like ABC News which is a which owns 538 week is owned by The Walt Disney Company is that there sometimes isn't enough flexibility with what you can do with high performers so we'll face situations where we're gonna take someone who's clearly very good you learn a lot more about someone you know in this first six months to a year of them doing work for you then you would learn in any interview setting right or even from like like you get references references we found are not always that reliable necessarily and doesn't always translate from one context to the next but we certainly do learn a lot about people in their first six months to a year and sometimes I don't have enough tools to enact that right you know someone might literally be underpaid by some by 2x or something right well it's hard to enact that this is somewhere you know the average raise is 3% or someone you know something might be overpaid that's a different type of problem someone might be someone who is not succeeding in like it's actually very hard to move them out at a company it's big and bureaucratic and so in some ways like we can't be as flexible as we would want and that probably makes us maybe a little more conservative some ways and our in our hiring decisions what do you think you know I start with like people shouldn't be [ __ ] I mean like one thing you learn is like in a kind of workplace setting someone who is detrimental to the culture they do not recoup that no matter how good their work is in the technical sense right it just wanted picking like so much bandwidth from management it lowers morale so much you know we work in an office where people are are pretty young it's a lot of Millennials for example and so you know Millennials all talk to one another about how they feel about their work experience I talked to wonder about how they feel about their their salary and compensation the kind of what they're asked to do and are they giving a fair shake so you have to you have to assume that you actually have to have to lead by by creating a good transparent and like honest culture and you can't manage an organization of Millennials at least by saying okay well you have this one conversation with one person another one with another person people all it's it's journalists - right they all find out everything that you say and so operating the critical that people are gonna be aware of everything and be smart about everything and we'll you know do reporting on like rumors and stuff about like involving their lives in their company and just kind of proceed from there so let's let's dive into that how do you screen out an [ __ ] question I think it's an important one Darryl Jessica when you think about drafting when you think about hiring what are the questions you asked what are the clues that you look for how do you get references to be more honest I'll take castles really they're good enough yeah where do you draw the line this guy no he also traded Shane what is no I think I mean generally I always have a no last [ __ ] rule but we are in a sport where an elite talent can overcome things so I would say you know talking to my peers in the industry like pretty much all the top top players I wouldn't say honestly they're [ __ ] but they all have their own set of know their own set of issues basically and you know our job is often to sort of work with those but then you try and put a very complementary team both on the court and off around them to create a to create a positive culture alert and I would say for drafting we spend the reason why everyone should know that everyone's bad at hiring I still have yet to see anyone who's good at it as we spend millions and millions of dollars on people who we get to see them actually perform their job prior to hiring them and it's still like 70% something around their error and 30% and so like if we spent that much time and multiple people in a structured process and we get to watch them perform their job and we still are this bad then it just shows you that you know we have a lot to learn still and I know Google's done a lot as well and it just feels like we're not moving fast enough so this is this is I mean this is my world right so as an organizational psychologist who the kinds of things that we study and I think that one of the reasons that so many of us are bad at hiring is we're just not using the right tools that have been studied for decades all right so let's let's throw out a couple observe something love love the structured interview idea is one example but I think most leaders don't go far enough so yes you want to have a set of valid questions that you ask everybody but you also want to go to your stars and your average and poor performers and ask them you interview questions and then build an answer key and then you can begin to compare okay what are the people who have really accepted this job say consistently or what do they do consistently that's different from the people who are not quite as good and that helps a little right it's kind of like when I give students a test if I don't have an answer key it's really hard to know who got the answers right on the same people though that's good maybe I'm homogeneity like maybe okay but I think it's it's probably better for screening out people who are gonna destroy your culture than it is saying here's who I should hire another thing we do is in reference checks we'll say okay look every reference it's due positive right so instead of of just asking you know tell me about this candidate and their strengths and weaknesses we give a force choice between two negatives and we say what's more likely that this person is a little bit too selfish or a little bit too much of a pushover what's more likely that this person beats himself up too much or herself up too much or doesn't care enough about mistakes and references don't know what the right answer is and so they often tend to give us the honest answer what I want what I asked went on the reference is is it how help me coach them better what what what can you know they're coming to work for me how can i what can I learn or what can you tell me that I should and I'm not saying be out like look look on be on the lookout for but you know that's how way too hard yeah that is that is that the tell-tale sign of an [ __ ] when a reference says someone's too competitive no I so your first question was what is the first question that someone asked so my first question is I'm usually the last person to interview the person and I asked them what they heard today and I'm not actually really listening what they say I'm listening if they know the names of the people who interview them so I do sometimes feel like maybe there's too much weight put on there are certain people who are very skilled at like winning the interview process or like the job acquisition process and I think sometimes like we actually probably we made hires I didn't feel like they worked out by the way we assumed that like I don't know where I came up this metric but it's been kind of empirically true we still look like 15% of hires are just not going to work out right just kind of from the start or miscast of course you don't know what those 15 percent are and you'd be surprised you know some of the people who I thought would for sure work out we're in that 15 percent probably but do you think like not reverting enough to your priors about like has this person done high-quality work in a similar setting it's very basic right you know so if they're a little bit more charming in an interview or they cobble together like really good references I mean still like again we're hiring for a very specialized skill set not as specialized it's like be able to shoot a three-pointer or something but fairly specialized and so if there is proof that you have done that work in another setting then that counts for a lot and if there's not a lot of tests where we can simulate that environment by giving like an editor for example an edit test they have a finite amount of time to edit the story right that that should be weighted heavily I think maybe we don't even weight it heavily enough because someone's oh they're a nice person in there and they're funny it can match getting a beer with that person right I mean you're not trying to like you know on the one hand an [ __ ] can be very disruptive but you're also not trying to select people to be your friends when you're hiring your we want people to be successful so your business can grow and that all the other people at work feel like they're getting have a good teammate now to work with and so I do think like you know maybe if you went to the extreme of not really hiring with an interview process at all maybe you could even think about that potentially so the the other thing is I kind of a unique perspective in that I was working for a team and then now I'm working for you know a technology company and the types of people who are attracted to the team and in you're inundated with people who want to go work at a team not that you're not on when you're working doing tech with it with a an organization that works across lots of teams but you're looking for specialized skill sets and the most obviously what we all want are people who are referring to us that from someone you know who knows the quality of their work but then as you're at a pace where you're in many cases at the forefront of change and trying to make new things happen the types of skills that you need are not gonna probably be coming from people or better refer that you know maybe and that that to me was the biggest shift that I had to so I think one of the big challenges that we all face here is you're trying to look at individual skills a lot but in sports you care a lot about the impact of those skills on the team right and you want players who are not just gonna try to advance their own success but try to do what's best for the team and so I'd love to see if we can figure out how do you know whether somebody's gonna be a good team player and maybe we can start by actually figuring out if you all are good team players so here here's my favorite question that comes out of the research on kind of this broad give or take or dynamic the question is let's think about a selfish behavior like stealing from a team or a company and the question is what percent of employees in the United States do you think steal at least ten dollars from their company in a typical month so from what Joe Lincoln on cash Tiller so wound quick cash IPA materials and merchandise like posted short that helps you Darryl okay now like I'm trying to picture what we're stealing that's all okay anything just $10 net per month what percent of employees doing what if you parking like the bosses spot is that worth $10 question okay penny your bosses okay you want me to start not the statistician I see first a 40 percent it seems like it would be a super low like I'm naive I guess like five percent uh uh yeah I was gonna go with fifteen fifteen all right so what's the real answer I have no idea with the human things you just took the average I do care though a lot about what you think the answer is because the data show the higher your estimate that other people are thieves the greater the chances Nate that you're a thief Darryl I'd check your wallet right now since he's I think the the psychology this is really fun because when a lot of people make these forecasts about others behavior they start by asking themselves what would I do or what have I done and then they project that on others so like the most selfish takers will say let's see what percent of people would steal ten dollars from a company because takers always talk like that last week I stole $384 gotta be pretty common 94 percent and meanwhile the more generous givers are saying like how do you even steal ten dollars like how many pens do you have to take home others say it's like a couple hundred post-it notes right this resonates with me you know I've had a bugger I had a player in the past that was always accusing other players of faking injuries meanwhile he's the biggest faker he was the biggest faker so like you know we never accused players on cuz they really don't like it's a really hard sport no but I don't think anyone really fix with this this player legit was like accusing our other players of faking injuries which is detrimental so it didn't make any sense but then we figured out he was probably doing it so that that does seem to be I mean that is one of the ways people justify selfish behavior right it's it's not me I think all of you people are like that and so I've just got to be that way in order to protect myself well I thought the fact that you posed that question though affects my answer right now so because we may hold on you're a statistician you're supposed to be completely sealed it against any bias that I can introduce right it no because you chose that question it's a biased question you chose that question have all the false questions you can ask right like if I say if I ask you he's been my definition that would be true but God okay so if I asked you what's the capital of Georgia the state of Georgia right like what you think will the availability bias would say Atlanta I see where you're going I'm asking that you like it's making or some [ __ ] no we think it must be a surprisingly high answered I think you overthought it is what you're trying to tell us anything now I will say though that I always give people the chance to explain their answer right so how did you come up with the 40% because I thought it was a weird question because I remember I remember many jobs ago you know where where you filed expense reports right and I wasn't make a lot of money back then and like [Laughter] shut off the camera if you work past 8:00 p.m. with more than 10 hours today then you were allowed to expense a cab home it was like overtime but what if he decided to take the train home having run a retail business I was looking at how much theft usually shrinkage come up with a new sure shrinkage and that was an and I was like okay it's normal it depends but industry across most industries it is around five to ten percent and I was like okay so maybe it's higher for employees that was my number that's totally fair and you have good data do another one I want I like this so there was there's a great study that I love that just came out that Emily Grijalva led where she measured NBA player narcissism by coding weight so there is a number of the 100 contrary to your previous point there is variance right they code it based on tweets and that like the most narcissistic players one of the tweets they captured was the tweet said I look in the mirror and then it's a picture of the player looking in the mirror and what I see staring back is greatness and some players don't usually they have hashtag grinding right there you go yes yes worship at the altar of grit and hustle but it turns out that if you score high on the narcissism scale by your Twitter profile that the team's mean narcissism and also maximum narcissism predicts fewer games one and also a poorer margin when you look at victory margin and that's mediated or driven by having fewer assists which seems to be like a proxy for bad coordination and also the longer those players play together the teams with more narcissists are less likely to get better whereas the teams with more humble players show more improvement over time so these narcissism actress here I'm being specialist so we would define narcissism as a sense of superiority and entitlement so I believe I'm better than others and I deserve more than others this is that predictive it's not like a necessary condition almost to be an NBA player you're the GM repeated feedback their whole life that they are superior to every person they encounter the grandiosity in it though here right say like I am I'm the greatest human being that ever lived is what we might be company they have overwhelming evidence they're correct but not relative to their peers in the NBA once they get to the league but like you think like they're all their formative years they're literally dominant over every thing they encounter so is it that is it their fault that they're no it's your phone cuz you're reinforcing it you're saying it's fine they should all be narcissists it's the costly web written in their Twitter or what they write in their Twitter like their description of themselves or if they're actual tweets and how narcissistic those tweets are and then you can also code their pictures for how much more attractive they look in their Twitter profile photo relative to their normal shop because the narcissist really want to put that best foot forward right so most people do that that might be true but again there's variance right so I'm curious about how you all look at these qualities it sounds derelict you're not bothered by it just a goodnight like do you do you think about is this person is their ego too big and how do I either screen that out or how do I change it once I work with them so well the first the my bigger concern is arrogance and being part of a team I think you need to give of yourself and understand that you don't know everything and be open to asking questions learning from other people and you know the ways that I usually try to screen for that is like kind of asking people as their that what they have done that they've committed a lot of time to and how they went about you know becoming better and kind of how they answer that question I find to be incredibly insightful you know are they saying it's me are they talking about the people who helped and taught them so I mean I think like a key qualifier for me is I don't care what someone has spent a lot of time doing to it that they are passionate about it I just want something it could be a musical instrument it could be you know they they were really into coding I mean it could be whatever it could be a sport I don't care what it is but as long as they dedicated themselves and tried to become better but also use the resources around them to become better so love it and so Daryl if you're gonna assume that you have relatively big egos coming into your team's a true and your paper that they're more narcissists than the MBA it has to be there wasn't a comparative analysis with other sports or other great I can't say okay but it wouldn't surprise me well yeah for me like I'm just about winning so if narcissism is actually a thing that will keep me from winning I'm gonna care a lot about I'm scary it seems to be so I mean I challenge I'd have to see the study it seems like nearly impossible they could have controlled for the variables you need like we would have to control for to that that narcissism they've isolated that variable is what makes them lose I think it maybe is highly correlated with something negative or something but I can't live in correlation I have to live in like that this that getting seven narcissists versus nine is gonna tip me over some scale or something that would maybe be something I've worried about all right so wait let's uh let's complicate that a little so it looks like narcissism hurts more if the point guard is the narcissist okay no no what I'm saying here is that what are you saying what I'm saying is I give up I'm done yeah okay I I think that at least if you take the premise that the point guard drives a lot of the offense you know it's easier for the narcissist to kind of take control and maybe hurt the the team's ability to get the right role player in the right spot and so would you actually use that kind of information when looking at to point cards with relatively equivalent skill I would look at anything that helps me predict whether they're gonna help me win paper this is a new variable that maybe we maybe we could look at I would say that if there's an assumption that someone who is more offensively minded at the point guard spot is negative I know that there's more studies that that's actually a positive thing so if somehow being more offensive minded makes you a narcissist then I would say probably there's a positive correlation to being in our says this would be my guess but I haven't seen this study so that's a modern MBA thing like if you don't have five threats on the floor he used to be the point guard could be an on thread this was in Jessica's era when she was the point guard awesome yes you could be in the Ivy League you could be a nine Fred you didn't play that's true I know I was terrible you could be a nun you could be an on threat at the point guard spot and do all the other point guard things and still be good but now defenses are too good you need if you if you have non threats on the floor that they can scheme for you to well I'm sorry I'm just have to read one of their text from Shane here he says people call Daryl the vitally put a pink of the pickup games straight bruiser Fowler no other skills that's correct I never sent cracks discussed no is it really true yeah I mean Daryl and I have a lot of things that we disagree on a lot of things we agree on but one is when we have played one-on-one is I don't think it's appropriate that he just backs me down he's much taller than me to score he should play me like straight up yeah I said I have a statistically better out of winning if I back you down as your skill level goes down your thuggery [Laughter] and it's not my fault she doesn't have referees oh you're the Bob Probert of pickup we've got it all right good so I want to talk about so we talked a bunch about kind of assessing individual people's psychology and maybe some of the different traps we run into and ways to get around those I think one of the things that's missing for me and sports analytics is culture assessment and I know that's we think the just individual psychology is hard right cultures even fuzzy or even messier even harder to pin down how do you think about gauging whether a culture is high functioning productive collaborative what are the kinds of indicators you would pay attention to that might show up early before you see even the lagging results one of the things that that I try to do each year is I actually do like a one-page questionnaire to everyone within whatever organization I might be overseeing and cater I mean I've been doing it for a long time so it's not like a net new thing but it but uh hearing from in asking people what is it how do you describe your job you know what do you how do you describe what we do as a company what are your aspirations things like that to me those responses are them I mean it's kind of the feedback qualitative stuff I mentioned earlier but that that's where I feel like I get the best pulse of not only how are people feeling about their job and is it clear what their job actually is if someone if someone is writing in and it's not clear I think that I I look at that and say oh that's a concern for me they should it should be clearly communicated what they're responsible for but then you know more broadly I want to understand what their aspirations are and that they are actually thinking about having aspirations at in this case kegger because that means that they like the culture and they want to stay right so one tool others yeah I mean obviously you want to you know turnover is very costly so you want to minimize involuntary turnover I think you also want to look at like when someone leaves where are they going right you know we have a lot of young journalists a high turnover industry but like the people who who end up leaving like get really awesome jobs elsewhere and so we understand their motivation for why they might want to leave that can give you some insight onto what you can do better to make to make their culture better but look there's it's like anything else there's like a wide variety of people and some people are like I'm gonna be very much about like I could part minimize my work and I do a very good job I'm very professional and then I go home and I'm not someone who is constantly checking an email right people who like their life is their work I'm like I think you have to be accommodating to both of those you know it's weird that we like wind up having a like a disproportionate number of people that work at 5:38 come from the Midwest which is Rick's we're a company based in New York so and I'm from the Midwest or from Michigan so maybe there's some like unconscious bias or whatever in that in that hiring process but like but the kind of Midwestern values of like humility and like conscientiousness unpretentious nasai you know those are things that I think make for better coworkers and so it's a weird little pattern I noticed there's a Peter Renfro study that Maps the 50 states on an average personality traits in the Midwest does come out pretty good on those metrics so not that we're biased Ohio yeah we're all from the Midwest I guess we are supervise yes sir I think the nice thing about our sport and why psychologists like to study us as you can keep score so I would say the the highest performing culture would be the highest performing team and the Spurs have been that and their culture is almost unquestionably the best in the NBA for many many years obviously set by great leaders like Popovich create leaders like Tim Duncan and I mean and they're like they're leaps and bounds we're second they're leaps and bounds of us we're like way down and so they and whenever I meet them I'm always amazed by the foundations they put in place and try to learn from them one of the things that that I think is interesting is is how do you transmit a culture from one team or one coach or one set of players to another and you all live in I think in the world where you have to deal with regular turnover and then say okay it's one thing to set a culture it's another thing to sustain it and there's a fun study that Samir cerebus java and his colleagues did recently where they looked at whether new people were fitting into the team and they found that one of the best indicators of cultural adaptation was whether you as a newcomer swear in emails at the same rate as your teammates and so if you if you swear too often or not often enough relative to the existing people it's a sign that you're not picking up the culture it's sort of giving the cues and so I'm curious about whether you all pay attention to those kinds of signals are there things you notice that are kind of micro clues about whether somebody is kind of buying into the culture or whether they might be be likely to reject it I like swearing Midwest no I the way I can pay attention and I'll shift away from the basketball team just to the team that is on the personnel and scouting side and the coaching side and I the people who don't fit generally are not talking so I'm like always encouraging those who aren't talking to talk and I get really frustrated people don't talk to the point where it's actually bad that I'm pushing people in the day so they're uncomfortable so I don't really want to do but I think if people aren't contributing and aren't any part of the back-and-forth then you then there's something off it might not be there culturally not a fit it could be just they're having issues at home they're having some things going on so I feel like you know as the leader it's my job to sort of understand better what's happening I'm yet for me it's very much about our people hanging out we have a you know beer Thursday kind of are people hanging out and talking to each other I mean it's not any more complicated or at lunches are people going in eating together and I mean I think if you are then that's good that means that you're kind of integrating in some form or fashion one of the other things I was curious to hear all of your takes on is the question of how to use data in a rapidly evolving world and I think this is probably harder in the people in culture realm than it is in most others because I think that the moment you you discover something or released a pattern the pattern itself changes right and they're all these feedback loops which is I think one of the curses of having influence in a platform but once that stationarity assumption is violated how do you think about when to rely on data of the past to predict the patterns of the future and when you throw out the the data that you've even gotten in the last couple of years I mean I know if the supply is to like analyzing the staff at 5:38 or our business per se but like in other realms I mean look what you ideally want is a data set that like spans multiple regimes I call them right we're like you know so one thing for example that's like quite robust in American politics is that the president's party tends to do poorly at the midterms they tend to lose seats in Congress has happened at least in the house in a big way last year as well in fact that was true in the 1920s and the 2010s and the 1950s whenever else makes it a lot more robust potentially so kind of seeing which findings are contingent on a particular set of assumptions in particular place and time and which ones are more universal I mean that's a lot of like what the art quote-unquote of fiscal analysis is and I think people of anything are like are too willing to throw out older data and because they think the system has changed well maybe it has changed but the whole problem is like it's gonna change again at some point and so having tendencies that survive these big kind of spectacular changes in the environment in the past will make them more robust go forward I would say saying cork incredibly hard an unsolvable problem it's like literally the art of the job completely in terms of us using data to make decisions in that you always have to be asking yourself is the fundamental assumptions with this model or prediction is when built on still apply and then you then have that sounds great but then you immediately have his issue which is as soon as you say okay I think it maybe doesn't apply then you're like okay well maybe I'm probably wrong and there's really there it's it's still usually I should study this stuff because this is this is unsolved and super core we don't know what window of data to use sometimes it's like oh well they change this rule does that throw it every draft pick prior to you know the perimeter touch rule or whatever and the reality is we tend towards Nate I think which is we leave more in then take it out because you know because of you know black swuan type stuff like it generally more data just more than not reminds you that you're clueless like it reminds you that things are really most of the mistakes when you get really good data saying you had a great prediction 90% they're done right at that moment where things have shifted on you and you know it plays out and and we have reliable day this is a crazy thing like his data is worse than mine my date is better but still bad and if you're just [ __ ] people users even worse than hers so like for example this year in the NFL and Ticketmaster opened up this open distribution which basically is now you people can see who's buying tickets that's new it's all net new information how many times someone is buying them secondary who the person is - the person not that not that the secondary ticket when you buy their own you might own a primary ticket Scott and then you're selling it who's buying it who's actually coming to the game that's net new information and so what you don't necessarily have known last year that this ticket or this person you think tended eight of eight of ten games for example but you learned this year that they actually only came to four I mean just as an example so does that fundamentally alter how you think about renewal well the reality is you can't actually predict that this year because it's all net new information you have to wait another year so I mean it's exactly what you guys are saying but that's the challenge so the question is you have all this great new information which by the way we're getting more information every single day 2.5 quintillion bytes per person so I think so that the challenge is how do you continue to think about renewal do you still use some of that information yes you have to because you still want to predict renewal but now you have all this other stuff that's so much more valuable how can you dig deeper into that new data to get some kernels of truth that you can think about applying this year even though you don't know what the actual results gonna be so then we there in another complication which is the question of how much does the individual actually the star especially shape the culture versus under the right culture the individual flourishes so that I think the Patriots example is a perfect one here so is Bill Belichick great because he had Tom Brady or is Tom Brady great because he had Bill Belichick Jessica can you settle this debate wonderful yeah so it's the craft so no it's a great question and it's certainly one that I've been asked before and I've thought a lot about it and I think from a statistical perspective or a data perspective not statistical the the way that you can potentially quantify it is if you look at the product sales so Brady's jerseys versus Belichick everyone knows he has that cutoff sweatshirt no but I mean so you could in theory look at those two which are the ones that if you from a fan perspective what are people you know acclimating towards what are they buying more of and you know as an example when when the original kind of Belichick hoodie was a Reebok one and when Nike became the new you know people who are who are making the the jerseys we bought like 8,000 jerseys thinking it would last us like a while and we sold out in like a year and so and I don't know if it's because people knew they were exclusive and they wouldn't come but so there's that and then Brady of course has been the number one jersey seller for many many years for the Patriots and they're more broadly so my answer is that I don't have enough data can you guys help us with this I mean I think this has been studied and that it's known that it's incredibly hard for a very successful culture to survive like the key person leaving like you just look at the longitudinal survival of major companies and it looks like it's incredibly rare for the culture to survive the original founding core person I mean yeah someone's just happy to get like a good like five-year run or something right yeah that's a long time I mean business like to have a competitive edge for five years is like a long time if you had two or three times in a row and you see the Spurs the Patriots be the two most obvious examples of where you survive a regime change and are still very good yeah I know you know with respect to that narrow question I think I think there it's easier to find the impact of quarterbacks in the NFL than coaches all the coaches matter a lot in the NFL so I would think that I would think Tom Brady was the more important part of that pairing and how would you help us quantify that what's what's the perfect analysis I haven't done a ton work in football the work I've read suggests that quarterbacks are incredibly important and then if I think quarterbacks are underappreciated underpaid relative to their peers in the NFL so that's my understanding of the research that others have done know that all right let's uh let's take some audience questions here ya go I have a question for you because you're studying culture so who is the person who's creating the culture is it the coach or is it the quarterback I would assume that I mean I wasn't there studying and coaching you're watching coaches like you know what are you observing when when you see how the culture is created can't turn the tables on me I just did it is your conference I think I think a lot of it I'd like to have better data on this right so I mean my my dream would be to to get a bunch of teams to agree to a coaching swap we do have evidence of Belichick not doing well in another environment we do not have evidence of Brady doing poorly any anywhere else I'm not saying that's that's true but we also have evidence of Brady not doing nearly as well at Michigan as he ultimately did with the Patriots I'm in the fact that that teams and all sports change coaches or managers every three years or so on average is probably a sign of that at the very least they don't really know kind of what makes a coach good right I mean obviously you do have turnover in quarterbacks the NFL but it's because the quarterbacks won't tell you ever get a better deal somewhere else or salary cap things right and so the amount of turnovers just that maybe it's a lot of hocus-pocus coach that's that way I want to say exactly but that like the the hard skills can't be that hard if the context of like this guy you thought was the best coach for your team two years ago and now he's fired because some things that probably aren't his fault you know it's just like there's not there's not that much differentiation maybe yeah I do think I mean the the only study I can think of that that speaks to this and at least a slightly controlled way was in the NBA it's a it's a quartet Cornell study of what happens when a team for basically moves to a coach who has no no professional playing experience versus played in the NBA versus was an all-star and they do find that there's an advantage of NBA experience and then there's also an additional advantage of having a coach who was an all-star what I would love to know is is there a selection bias in those data so that only the talented coaches who are players make it right but then also is there a respect thing that they're not actually better coaches their players are just more willing to go with them and so the culture is more naturally and easily set by the coach under those circumstances jury's still out so should we take some audience questions yes all right let's do it I see no audience questions audience and ask some questions please we will ask them even otherwise sweet they're our audience questions all right this is a good one how do you determine what motivates prospective employees or hires or fans and Darrell in particular can you tell which players want to win versus those who are just in it for the cash I don't care you really don't care no as long as it as long as they're helping us win a title I'm fine whatever I prefer there and I do think like longitudinally having a big love of the game has been anecdotally a very positive thing they put more work there but ultimately like I am mostly focused on can they increase our chance of winning the title so I would say it's like a secondary factor okay another question Adam silver I mentioned earlier today that social media entices athletes to disconnect from their teammates and I've actually heard from two NBA coaches this year that they're having another layer a problem with this which is at halftime players are going on Twitter and they're seeing really nasty comments and they're getting discouraged and it actually hurts their game so how do you manage this and prevent players from from getting sucked into the evil world of social media when it is evil Jessica this is something you see a lot on the fan side Wow how do you prevent the players from from from getting distracted or getting discouraged or getting angry that's something that's not really about the team or the game so the question is is how do we prevent the fans from writing mean tweets potentially Wow I don't know if there's an answer to that yeah I don't have a good answer I mean your players have to have the life tools to handle a whole set of things they've been yelled at if they played at Duke or something I've had the most nasty things screamed at them they they handle way more negative feedback than you could ever imagine a bad performance so it's pretty rare in my opinion that they're they're letting the land random Twitter troll affect their if they are then they won't make it because they're not helping me win so glad I said clear if anything we maybe give them the tools like we do with the media we give media training we give them some basic tools on how to handle situations but beyond that they better have the life skills by the time they get to the pros for anyone question is if you make a hire that later doesn't pan out how do you have a productive conversation with that person if you don't want to let them go and this could go for drafting as well I mean I think the question is always asking them what it is that they want to do the question that you asked Daryl to start the first question from the audience is like for me I'm always looking for someone who wants to learn because everything is evolving so quickly that they need to want to learn and I think in the case of someone who it sounds like the question is there a great cultural fit but they might not have the skill set I want to ask them you know what is it that that you aspire to do and can we redefine your role to make you more productive within our organization what about when it's the opposite that they have the skills but there may be toxic to the culture what do you say to them then to me that that one's about coaching understanding understanding the challenges that there have that they're having and then working with them I mean no one's perfect and kind of our job I think as managers is to understand what are the underlying factors and it might not have anything to do with work in it and it might have something to do with work but I mean I think of my job now today as being a coach and I need to help people get better and mature and grow and become better managers or better analysts or better leaders Darrell what do you say to a player who is capable of helping you win but not living up to the culture that you're trying to set honestly really depends how good they are if they're not very good then it's like a very like look get with the program or this isn't gonna work out for you and we have just pro-sports everyone understands that it's it's a high turnover business whether for cap reasons or whatever so I think player movement is sort of built in and so if they're a marginal player who's you know very bad for the culture you know creating negative things not following what is being asked then then those players last very short that's one thing we do well we move on quickly if they're an extremely good player that has those issues then now you're now you're trying to really help them understand how their behavior is hurting their overall everyone has the same goal that's the easiest thing we have it's what everyone's trying to win a championship in our sport so that's easy and so if they're very good but exhibiting those behaviors you're trying to coach them into understanding how those behaviors are actually hurting their goal and try and connect them to that I would say that has a lot of limited success though like like generally we get them farther along in their development that you know that maybe we're managing them 5% one way or the other but they they're gonna be who they are Nate would you ever advise Darryl to to draft or trade for a player who is an absolute star but might be a problem for possibly unquantifiable culture reasons I mean the NBA is weird in that like the value being like any elite like James Harden level stuff career level NBA player is just it's just so high right and it's also high relative to what the max salary is and so like that makes it maybe in some ways like a fairly bad example right where you can be a star for the youth some Rockets in a way that you can't be as like as a 538 journalists right or in any other sport really you know no baseball player in the history of baseball has ever been is worth as much as James Harden is to the Rockets right that's correct maybe some NFL quarterbacks have been I mean Barry Bonds was the closest maybe Barry Bonds maybe was with probably 12 or 13 wins a year yeah so half as much roughly right yeah so in wrap up what's the question each of you would most love to pose the panel looking to our right yeah being polite letting the lady go first mine's gonna be the neat I mean here's like knowing what you don't I mean I have no [ __ ] idea the single most likely person is Trump but yeah I mean but like there's no good would take field would you take field there's a bunch of new folks who are coming in right now in from on the Democratic side who do you look at and say they they have something that's special that you know maybe they're gonna be able to make a push I mean look I think Kamala Harris is a candidate where she has some of the markers of successful candidates from the past where she has a a diverse coalition behind her she has raised a fair amount of money so far you know apart from her I mean I think she's like kind of their frontrunner but a friend or in the sense of like in the NCAA tournament we're like the favorite probably has like a eighteen percent chance of winning or something and they're not really much of a favorite you know I don't know I mean beyond that like Bernie Sanders has I think a high floor he is very loyal supporters I'm not sure how high his ceiling is necessarily you know I think there are some candidates who are more moderate you know bet - Aurora core Amy Klobuchar you know Cory Booker potentially who you know do you think now that like the Democrat Party has drifted somewhat to the left but there still are a lot of moderates in the party so are there candidates those three I mentioned who could maybe unify the moderate wing and the more progressive wing of the party people I could win the Nam nation potentially but the thing is nomination processes are like kind of literally chaotic and I mean literally the sense that like one small thing right like in the Republican nomination process four years ago when when Chris Christie undercut Marco Rubio in that debate after Iowa that was a moment when Rubio had been rising in the polls if it was third and Iowa was like supposedly had momentum right so that like that one moment might have changed the whole trajectory of the GOP primary when you have like 17 candidates you might have 20 Democrats then it is like intrinsically unpredictable beyond a certain level it's so the speaking of bad data the data in your field is just awful Thanks yeah this is terrible thanks for inviting me by the way yeah oh it's really important data but it's awful so like how your data suck - what's that your data suck - I did a sexy I agree everyone use is worse not you personalized controlled experiments here I can tell you what causes what yeah no just but there's a good correlation by your premise there's so much correlation like how do you how do you like I just reject so many studies like just offhand I'm like I here am I go like there's no way they control for everything I'm not gonna read it how do you how do you know like this is a good one I think I think you should be a little bit more open okay insofar as maybe he'll give you a competitive yeah my study told me no I don't know I have a few indicators the first one is I look at obviously whether I think the researchers did the study or statistically and methodological rigorous second is did it did it meet the bar of a top Journal and then the third is did they actually pin down both the mechanisms and the boundary conditions and so I think there's a lot of crappy correlational research but when I start to see a bunch of evidence of like how they thought to track whether assists actually drove the changes in performance okay now they're diving into how this this narcissism dynamic begins to affect coordination and we know coordination is relevant for a team to win okay now they're they're a little more sophisticated right and then okay did you look at how this plays out at different periods in NBA history or did you look at how look at how this plays out as a function of how long the team has been together as soon as you start to say we're looking at when the effect is stronger and weaker we're looking at when it's positive and negative I think you just you basically show that you've got people who are foxes not hedgehogs and I trust their ability to run studies and much more what's the study we should all leave is we all read to wrap up what's one that everyone would be better in our jobs if we read it everyone here would be better than most people if you read one study yeah like you you have like a dossier of studies it's impressive you're like that's all-time literally the only thing I'm good at I don't know it's hard to pick one study I think for an audience that the cares about sports analytics I think one really cool study that I'm excited about at the moment is a study that that Frank Flynn worked on a few years ago with with professional golfers and he found that if you were if you were golfing in a major tournament with somebody who had a better average than you you actually did better and if you're golfing against somebody who like the rest did Greyhound effect it seems like it and I think that that really often the thing we're most afraid of is to be right next to somebody who's better than us we're afraid we're gonna get psyched out or we're gonna choke or you know I just don't belong in this league I think the reality is I just worked out a podcast episode about this so I'm biased but I think most of the time our rivals actually make us better we learn things from them they raised a bar for us and you see this too with NBA players when they get traded away from their their team they actually played better against their former team when they threw so I just I think we could chance we can embrace our rivals a little bit more and that's something we should all dive into do they control for which golfer goes first because if he's a good golfer right and you're like okay that guy tried that and like that worked so therefore I could try that to look it up Manitoulin and flynn org science and i cannot tell you whether that was a control variable thank you so much you you

Channel: 42 Analytics

Views: 3,752

Keywords:

Id: qqsLtCzQ_-4

Length: 60min 16sec (3616 seconds)

Published: Thu Mar 07 2019