Thanks. Near the end of his presidency, one of Thomas
Jefferson's friends wrote him a letter. He said, hey, I'm paraphrasing, hey, I'm thinking
about� [ Laughter ]
Becoming a newspaper publisher. Could you� do you have any advice for me? And Jefferson was pretty skeptical of newspapers
at the time. And he said, maybe one thing you could do
is you could do your thing in four sections. The first one would be truth, the second one
probabilities, the third one possibilities and the fourth section would be lies. And he said the first section, truth, would
be very short. And so today when I want to talk about is
how we have room for the second section, the newspaper that I work with hasn't followed
this model. We use travel and sports and national to divide
own content. But we do sometimes say explicitly that we
don't know. It's become part of our playbook for really
terrible things that we have a section, you know, here is what we know about something
that happened and here is with a we don't know. It's sort of interesting to me that we almost
exclusively do this for terrorist attacks and bombings and fires. We do it sometimes in the Data Viz world. The election, you'll see there's a doubtful
category. Abbreviated DFL. I think we should bring that back. It would be fun. It turns�us in this instance the doubtful
wasn't quite wide enough. There are at least three mistakes on this
one, I think Minnesota and Idaho and New Hampshire are the wrong color. Doubtful, they should have used it a little
bit more. This is not the type of not knowing that I'm
interested in. I'm less interested when it's black and white
than when it's a little bit more murky or a little bit more fuzzy. So there are examples of people doing that. When I think of murky and fuzzy, saying I
don't know, I think of these guys. And these are the sea monsters that people
used to draw on old maps. Maps in the 1500s and the 1600s. And like data viz, people were trying to pull
things off with sea monsters. Sometimes they were showing they were fancy,
really expensive. You had to hire a special guy to do it. Sometimes they were trying to fill some space. But my favorite ones, the ones I like the
most are ones where they were trying to indicate there's something here and I don't know it. And it might be dangerous. And so there's something in between. They're saying, like, is something weird happens
there. I don't know. Some of you can probably guess the reason
that I have been thinking about uncertainty and how we say things that we don't know. And that's this guy. Which was an election night offering. And this is how the upshot interpreted results
on election night. So there's a few things going on. We've got three main numbers up top. We've got the chance that we think Clinton
wins the presidency and then the popular vote margin. What that will be, and then what the electoral
votes will be. We have shaded bands. Those are essentially like 95% sort of confidence
intervalish type of things. And then we've got this needle that moves
back and forth and it stays in the middle 50% of the simulations. And it's using Perlin noise, noise that's
used to make marble and clouds and fire. And Gregor, my colleague, complemented that. And I think it's soft and graceful, but many
people disagree. There's two camps of people. Those who needle helps you feel the uncertainty
that if you just show a shaded band. I think a lot of data viz is about hierarchy. And the needle forces you to not be able to
ignore that these are estimates. That we're uncertain about them. We don't feel more confident that one particular
number is truer than the others. And there's others who feel deeply uncomfortable
by a moving needle. Who feel it's not helpful for them in understanding
what is confusing about the world. And so still, you know, there's lots of things
going on on this page. There's, you know, the state estimates, the
same sort of game. And they shrink over time. And then we're also keeping track of how our
estimates changed over time in a couple of different ways. One, just summarize the uncertainty. And two, we show electoral votes with some
bands. And I'm interested now when people are angry
on Twitter sometimes, they tweet this image to say like who are you to say anything? But like I read it as, like, who are you to
update your expectations in the face of new information? [ Laughter ]
30 minutes faster than the betting markets and three hours faster than the networks. But I think that's too many characters for
Twitter. So they just use the� they just use the
image. No one uses this one. This one is only understood by people who
don't need to be aggressive on Twitter. [ Laughter ]
And so, you know, lots of the work that we've done, I accept the fact that people interpret
it in different ways. Right? And, of course, that has to be true, that
people take their whole life experience to your work and their knowledge about things
and how, you know, everything that encompasses their life. But then you never felt more than with this
graphic about how people feel things differently. How it doesn't feel the same to everyone. And that shouldn't be surprising to me at
all. I think of this guy. Which is a chart� an experiment� where
someone asked a couple dozen male officers, tell me what you think about these words and
it's probable. It's probable that� give me a number what
that means. And there's little chance that� give me
a number what that means. And it's stunning that these are naval officers. They're all Englishspeaking as their first
language. And probably� if I said probably� some
of you interpret it like 40%, and some of you say 90%. So any time we're ever trying to communicate
anything, whether it's through pictures or through words, there's this like gap between
what you say or what you think you are saying and what people hear. This is older. They're special, someone on the Internet and
on Reddit redid it with just people on the Internet. And like here you get people who are a little
sillier on the Internet. Highly likely. But the shocking thing is, we basically get
the same answer. Whenever we say anything, there's this wide
interpretation of any word that we use of any image that we use. And so I think that this� oh, this is fun. Because in some ways when I think about the
Trump/Clinton odds, there is some work that says maybe it would be better if you always
present those in terms of the adverse outcome. And whether you're thinking the adverse outcome
is Trump or Clinton and personalize it. And if you are the person who might be highly
likely and that's 70%. Up that net verbal descriptions. Of course, that's not going to work. Because thinking, it turns out, is different
than feeling. And we know that from the psychology experiments
where people try to figure out how do we feel probability. And I can know in my head that 50% is 50%. It's like a coin flip. But I don't feel it in my hard. I tend to overestimate the smaller possibilities
and underestimate the big ones. And it's also possible I want to be lied to
some of the time. Here's my favorite chapter from Nate Silver's
book. He talks about weather forecasts. This is the national weather forecast. When they say 20%, they really mean 20%. That's what it means for the gray dots to
follow the black line. And this is the weather channel. It's possible they're cheating a little bit
on the low end. This makes sense. If I don't bring my umbrella and it rains
than if I do, and I have to carry it around all day. This is a local TV station in Kansas. And so I'm pretty sure that I don't want that. But I'm not certain that it's� I want the
one that's actually perfectly true. It's possible that in some context I want
to actually be lied to. And I think that that's the answer about why
some of this is so difficult and why we're never going to have a blanket answer. It depends on context and depends deeply on
the norms of the discipline. No one was upset when we played with a little
jitter on primary night. At least that I heard about. But the jitter on election night upset some
people. In the same way we think of this as a normal
way to represent uncertainty when we discover earthsized planets. Or when NASA discovers them. And at the Times how to signal this is not
quite real. Call it a rendering. Some could have surface water. But there's a beautiful article in the Atlantic
that talks about what they're looking at when they develop these things. The first picture that the rendering comes
off of this data. Like blows my mind in some ways. And the most delightful ways. This is the data that we have, and if you
know how to read it has to be consistent with certain types of things. The thing that the rendering loses is it loses
the arguments they have. Does that green indicate that there could
be plants there in a way that we don't want to? So the thing I'm interested in is what are
the forms that we use to say there could be plants there, but maybe not. I don't know. The classical one is this one. This is the government tomorrow. I think it will be an important day for the
government in federal revenue day. Maybe. Who knows. And this is a chart that they release every
time the federal government releases a budget. And I love it. I think it's one of the most delightful things. In part because if you know how to read the
scale, this is like the depth of the great recession, and this is like better than it's
ever been before. So essentially this chart says, like, I am
pretty confident it's not going to be like the great depression five years from now. It's not enemy favorite government uncertainty
chart, though. That's probably this one. Comes from the economic report of the President
in 2016. And this is about how low the unemployment
rate can go before it sparks inflation. And my favorite part� I mean, I think your
laughter suggests you get it. But the footnote at the bottom says a 50%
band is used because increasingly higher levels of confidence increase confidence bands, approach
unboundedness. It's even better than it appears. Essentially says I have no idea what this
thing is. But that's the point. That's why they use some sort of an uncertainty
band. And I'm kind of convinced that that's our
norm. We only do it when we're forced to. In the archives of the Times print graphics,
since roughly the last 25 years, I think there's about somewhere on the order of 4,000 times
when we have said either estimate or forecast or prediction. Like some word in that family. And I can find only eight when we've formally
expressed some type of confidence interval. And I think most of them are when, like, we
felt like we had to. We felt like we were forced into a corner
like this one where we wanted to say that childhood obesity rate is not rising, but
if you just showed like the point estimates, it was like, it kind of is. And so basically convinced that we only do
that� we only show it in a chart in some way when we're forced to. When we want to conclude the opposite. This is one small exception in that, in that
I'm not counting the times that we say, like, a margin of sampling area in a poll graphic. I'm convinced in my heart that does more harm
than good. I don't want to claim that introducing more
uncertainty into the world and jitter into your life is universally a good thing. And the example I would claim for is this
guy. One of my proudest moments of upshot stuff
is when we pull off data stunts. And this is an example of a data stunt where
we conducted a poll in Florida. And then we gave the data� the raw data�
the individual responses� to four other people. And we just said, tell me what I should have
said. Tell me what the answer is in the top line
number. If you're not going to read any cross tabs
or go deep what this means. Give me the answer of the poll. And it was more delightful than I dared to
dream. Get four of them and we get four answers back. Ranging from Trump plus one in Florida, which
ended up being the final result to Clinton plus four. And so this is not accounting for sampling
error at all. This is not� because we're all working off
the same data. It's all exactly the same data. And so this error, this difference, is about
analysis and about how you actually read the data. It's totally on top of the normal margin of
sampling error. And so I think for is indicative of sometimes
problems in data too where we solve the problem that we think we can. Like we know mathematically we can write down
what sampling area is. This kind of stuff, you have no idea. You don't know the turnout error. We focus all the time on sampling error and
put it in the footnotes and think it's enough. Turns out it's not enough. In the same way that you plot the data that
you have, it's the data you have and what am I supposed to do? Have a closed form or not closed form for
it. This is what we do. there are other examples in the Times archive
where we talk about uncertainty but we just don't use the word. So when I search the archive, I don't find
them. The classic one, of course, is a hurricane
chart of some sort where our style now� and I didn't know this before this talk�
is there's no indication of what that cone is at all. It's just like� it's the norm for like hurricane. It's like a hurricane cone. You know? And until people started talking about uncertainty
recently, something I didn't realize is that's like a twothirds of� the cone is made by
like going out points. So like six hours, 12 hours, 24 hours�and
you're figuring out empirically how off have forecasts been for the last five years. So there's something delightful about that. The cones are always the same size, no matter
what. They're different if they're in the Atlantic
or the Pacific. And there's some part of it that I think is
fun that� pretend you got omniscient. Pretend this was probability. And the cones would be increasingly wrong
for the next five years. And we didn't change the way we play the hurricane
game. But if you're omniscient you have better things
to do anyway. We are comfortable with this kind of idea,
with this kind of a cone. But I think it feels totally different sometimes
when you see the actual path of the hurricane. This is the same hurricane that we were looking
at in that map. And if you watch it, it's a double loop, rights? Which is totally crazy. My favorite, the Twitter comment, the first
one on this, is utterly ridiculous, yet somehow not out of the realm of possibility. That is a feeling in data viz that we can
use. Reduce it to a cone and a hierarchy. And gray and soft, and in the back, it says
please don't worry about me, except maybe worry about me if you care. This graphic� the like implied sort of craziness
of it� the model run for the same hurricane Matthew, makes me feel totally different than
the cone. It's design decisions, decisions about aggregating,
and decisions about how soft the weirdness is. So whether there are other design decisions
that affect that. I think of something like this guy. This is the only time I think in my New York
Times history when he uses squiggly axis. And there are a certain set of people who
think sketchiness is the response to uncertainty in data viz. If we started drawing in crayon it would be
clear immediately. I'm not sure I agree with those people. But there is something about form. I think about Ezra Klein when he talks on
the podcast, it feels like a looser medium. It feels like a place to try on ideas. And it's clear that he's not certain that
what he's saying all the time is true. There's some probability that some of this
stuff is true. And so this is an example of a chart where
we ask people to draw it within a few of these. And one of the first ones� and we said so
the X axis is your parents� income percentile. And there's a hundred equal bins. And the chance that you go to college. Not a fancy Massachusetts college, just any
college at all. A trade school. And not even graduate, just enroll at some
point. And so I thought, you know, maybe it might
look� maybe it looks something like this. You know, maybe it's an Sshape that, you know,
once your parents make $300,000 it doesn't really matter that much anymore. Except maybe there's a little trust fund dip
here like at the end. [ Laughter ]
And so maybe� maybe� maybe it looks something like that. But the truth is, actually the truth is it's
like a super, super, super straight line. And the straightness of the line I think of
as shocking in terms of what it says about class in America. That like you go up one percentage point�
or one income brink, and that changes your chances of going to college by the same amount,
about, no matter where you are on the income. It's exactly the same for poor kids and rich
kids. And sometimes, you know, I like to give this�
talk about this example in talks. I was reminded yesterday when I saw Noah Veltman
who will talk this afternoons about his reaction to this. Maybe inequality in America looks like�
looks like the Fraser skyline. Or Erik Hinton who is like I'm pretty sure
you drew that from the little prince. But when we think about uncertainty, when
I talk about it in certain forms, people who really understand data are confused about
the fact that the gray dots are real data. You think of it as model. They're too perfect. There is something in us that can understand
that that's not what real data is supposed to the look like. There's something broken about its straightness. The other reason I think I'm interested in
it today is I do think that our avoidance of uncertainty has real policy consequences
in the world. So, for example, right now the federal government,
one of its best tools for thinking about whether this chart is a problem or not is Pell Grants. But Pell Grants happen super late in college. So maybe by the time you understand that you
qualify to get a Pell Grant it's too late for you to do anything about your high school
transcripts or to prepare different. And so there's researchers who say, look,
we could do a pretty good job� a very good job, actually� of predicting who needs a
Pell Grant by looking at who needs a free lunch in the eighth grade. There's some uncertainty to it. There's 10% or something of kids who we went
with this policy in the eighth grade, we go, congratulations. We will give you a Pell Grant to go to college. Think about your high school career a little
bit differently. Who, you know, wouldn't by the time they end
up enrolling in college end up qualifying. There's lots of reasons in this country we
don't think of this as good policy. And a small one is we're uncomfortable with
being a little bit wrong even if it makes something so much better. So when I think about the election chart,
the reason that we can be faster than the networks, is we're saying I'm comfortable
with this uncertainty. I'm comfortable saying I might have to turn
around at some point. If I say there's a 95% chance this is going
to happen, one out of 20 times I should be wrong. Assuming I'm not calibrated like the weather
channel. I'm not cheating in some way to lie to you
because it's better. So I want to close� I think if we got more
uncomfortable with uncertainty, if we got more uncomfortable with the fact we don't
know the feature but we can have educated guesses about things, I think there are real
world implications and policy consequences to that. So I want to close with a little bit of at
least like a hopeful note. Two months ago the federal reserve started
producing these things called fan charts. When they say here is what I think the interest
rates are going to look like. And the fun part about the story for me is
that the Fed releases the minutes of their meetings with a five-year lag. So you can go in and see that ten years ago
Janet Yellen, now the most powerful woman in America, said, like, hey, I think we should
put some uncertainty on our charts. And you can see how important it was. It was like page 255 of 255. And you can also see, though, you know, they
went with 70% intervals. So by the end of 2008 they thought the federal
funds rate would be this. In between 4 and 6%. It was actually zero by that time. But you know that. You know in a 70% interval it should be outside
of it some of the time. And so my hope is that you believe that if
the most powerful woman in America is willing to fight for this for ten years, that maybe
some of us should too. Thanks. nytgraphics. [ Applause ]
I like questions more than yesterday. I don't have if we have time. But I'm game for questions if there are any. Yes? >> [ Away from microphone ]
>> Yeah, so the question for you who can't hear it is, how do you talk to your colleagues,
especially those more focused on words, about when you put more uncertainty either into
words or into graphics? And there is� I think there's this tricky
balance some of the time, because you don't want to entitle up saying nothing, right? Like there's� there's this class of words
that I think of as weasel words. Which are like could or maybe or perhaps. And if you stick a perhaps in something, like
the amount of things that you can say like opens way up, right? Like not in a helpful way. Not in a way with a ton of rigor. And so I think lots of the balance, especially
in words, is that you want to stay in the area where you can say something that is true. Like you want the Jeffersonian like first
section. But that takes a bunch of stuff off the table. There's a thought in journalism that journalists
are far more interesting than what they see in the stories. The things in the story are 100% true, but
at the bar tonight, it's like the Ezra Klein podcast. I'm going to back down from that a little
bit. I can say it and it's 97% true. I think with words we don't have a ton of
tools to treat uncertainty with rigor. The guy who made the naval chart who was behind
some of the research, used to fight there were like poets and mathematicians and the
poets would always screw it up and that's why we couldn't have nice things. I think it's a balance about wanting to stay
true, but also wanting to say something. And that's the inherent tension. >> Do you think that presenting probability�
a chart with uncertainty in it to a general audience is something that can be done straight
up like that? Or does probably have to be [inaudible]�
so they don't like say, oh, this is within the range of probability. I like that. So we believe this is true. >> Yeah. So the question is, when you show some sort
of probability estimate to a general audience, do you have to explain it, or can you just
throw it down there? >> Yeah. >> And I think that's another really tricky
question. There have been examples where, you know,
I� part of looking at the archives about how infrequently we do explain it in a formal
way made me sad. And then it was like how on earth am I giving
people a chance to understand what we do when we do it if you only see it like eight times
in 25 years. It's not a repeated exposure example. But on the other hand, there are times when,
you know, there are stories about� especially certain types of statisticians becoming like
Bayesians when think try to explain a confidence interval. I will be sent photographs about people trying
to explain what a Pvalue is. Don't do it, please. I think there is a sense, like your intuition
is right even if it's not 100% technically accurate about, like, I get a sense that I
don't know. For example, I've never been bothered by the
hurricane cones. I just trust that that's like a hurricane
cone, right? But at the same time, it like blew my mind
when I understood those were just empirical. They shift every five years and they're the
same every single hurricane. That's crazy. So I think there's levels of understanding. And you can understand it in a sort of intuitive
sort of I kind of get it way without needing to know a formal definition about what it
is. And the other thing is usually these things
are where they come from, that comes with a whole other set of assumptions. The hurricanes where we looked at the five
years of hurricanes and looked at how many were outside the band. Usually from a model or something more complicated. There's another set of assumptions that you
take to be true and then this is true. That's an extra level of complication in the
modeling. >> AUDIENCE MEMBER: Thank you very much for
the talk and your question and your work. Two questions. The first one is that I love the example of
the needles for explaining to my students uncertainty. So I saw that you had like a whole rerun of
the thing in there, is there any way we can� and the second one is given the response to
the [inaudible]� do you do it in a different way? >> Yeah. So I'm happy to share with you the rerun. And that's like the one I made for today. It's out like� I think it's at 120 speed. So that election plays in like two minutes. I left the needle jittering at the same right
so it doesn't feel frenetic. When people close their years, it feels more
comfortable� it was soft and gentle. And the second question about what will we
do in the future? And I don't know the answer. I don't know the answer. There's some probability, right? So if I say like probably and some of you
interpret that as 40% and some of you interpret that as like 90%, it would be fair. >> Going off that question. I have a� for a piece, that election night
graphic is� I'll never forget where I was and who I was with and who I was watching
that. Was the intention to build such a viscerally
moving graphic? >> Yeah, I think, obviously, this was teamwork. A lot of credit to Gregor and Nick and lots
of people. In the conversations about this, we wanted
to help you feel the answer, right? Like I don't think we thought it would be
as visceral as it actually is. But I think visceral data Viz is good. Things you can remember and feel. I'm uncertain about what uncertainty is supposed
to feel like. My colleague, Ken, argues this mostly just
raises people's blood pressure. Is raising blood pressure what uncertainty
is supposed to feel like? But I do think it gives you an answer in a
way that the maps don't. Like there's a point in the night where, you
know, a top level in my office being like, why is this so different than the map? It's like, because you don't know how to read
the map. Wisconsin being pink suggests the same thing
the needle suggests right now. I think there is a good� you know, there's
this tradeoff about how intellectual versus visceral do you want to see? And that's with the confidence intervals. These are intellectual things. I have to stop and think about it. And the needle� and they were there too. I like the way they shrink over the night. But the needle, I think you feel it in a better
way. >> AUDIENCE MEMBER: You mentioned wanting
to affect policy changes, but [inaudible] this will actually certainly have an effect. In a journalism setting, how do you manage
trying to affect policy change and trying to force people to understand�
>> So I don't think that we are actually� like our goal is not to change policy. Our goal is to inform people. But I do think that's related to policy like
in the Pell Grant example. You can say, like, this will help 90% of kids
and 10% of kids we will just like make a mistake on, right? Not mistake, like we gave Pell Grants to kids
who were like slightly more middle class than we should have. Horror of all horrors. And so the idea that I think it's really just
more of a general idea. That that's sort of things that we are comfortable
with is different depending on how complicated they are, or how much� or how we can understand
them. >> [ Away from microphone ]
>> Yeah. I do think� the question was, if we ever
did the needle again, what range would it be bounded in in the driftiness? Should it be more? I think we were too conservative in bounding
that to the 25th to 75th percentile. It would have been better if at the beginning
of the night it drifted into Trump range. The outcome was perfectly consistent with
what we thought we knew. But people didn't think that way. At the expense of people thinking that the
needle is crazier than they already think it is. That's because it's true. That is within what we do not know. And so, you know, those are arbitrary numbers,
but I think if I were to do it again, I would push for more drifting. >> AUDIENCE MEMBER: Have you seen audiences�
attitudes towards forecasts and uncertainty change over the past year? >> Have you seen audience�s attitudes towards
forecast of uncertainty change over the last year? I think, yes. They are more skeptical than they were in
the past. I think there's a giant group of people who
thought Nate Silver was a good and turns out he's a very smart man. [ Laughter ]
So, you know, how that� the betrayal that people feel about that. Even though he did� and he speaks eloquently
about 30% means 30%, right? And that's not because we don't have needles
in our hearts or something. We don't feel that way. And so I think there is a general� there's
a greater skepticism than there was a year ago. For better or worse maybe. You know, it's possible that expectations
are just calibrated more properly than they were in the past. >> AUDIENCE MEMBER: How do you A/B test for
the level of data that you're calling upon from your audiences to be able to understand
it? And how do you know when you want to go out
on a limb, for example, with peaks and valleys [inaudible]� the design? >> How do you A/B test for data literacy and
how do you know when you want to go out on a limb? We don't do a ton of A/B testing within individual
pieces. So I think it's more like as a body of work
you know� you know, I know that there are times in my career when I've sort of walked
off the edge. But I feel like you have to find the edge
so you know how not to walk off of it, right? Because a lot of the cycles that we work on
are pretty quick or we're just bad although being disciplined. We don't do a ton of A/B testing within individual
graphics. In part because it would be weird in a social
world. We don't control the distribution of these
things in the fact that lots of traffic comes from Facebook or Twitter. It's frustrating to see the screenshot and
not be able to achieve that screenshot. It's solvable, probably. How do you know when you have gone too far? I think it's when you feel like you can try
to mediate that with clear language and clear labels. I've sort of, you know, as I've grown up,
I've come to think when you need to know how to read a chart, that's probably gone too
far. Like if I have to have the key that says explicitly
here is your instruction manual, then I'm probably in trouble. But there are times when you feel passionately
about that. So it's just an unrigorous balance about what
is far enough. How much time do we have? Maybe one more? >> AUDIENCE MEMBER: How do you feel about
simulations and� versus the confidence interval? Like the dials in the election from�
>> What's the� how do you feel about simulations versus confidence intervals? I think I try to answer this a little bit. I think simulations feel more visceral in
a way that confidence intervals are good for my head. And simulations are better for my heart is
how I think that I feel about that. But that's an idea in progress. You know? The frustrating thing about simulations is
then it's like, well, I want to see the summary. I think the dials were trying to do both at
the same time. Trying to show you here's one that you can
ignore if you want. But if you want to think about and try to
remember the history of these simulations without trying to rewind in your head, or
stare at it for seven minutes and hope you get the impression of where it was. I think both at the same time can be effective. Okay. Thank you. [ Applause ]
