- [Justin] In the nation of blobs, there's a popular game
based around flipping coins. Each blob brings their own coin and they take turns flipping. When a coin comes up heads, the one who flipped it feels happy. And the other one feels sad. That's it. That's the game. It seems kind of simple, but
these blobs are a simple folk. There's a rumor going around
that some of the players are using trick coins that come up heads more
than half the time. And that's just not fair, so we would like to catch these cheaters. As a warmup, let's have each of these blobs
flip its coin five times. (playful electro-percussive music) Okay, you might be able to tell that this is an artificial sample. We have results ranging from zero heads, all the way up to five heads in a row. Which of these blobs, if any, would you accuse of being a cheater? If you'd like to try your hand at judging the blobs yourself there is an interactive version I introduced in the last video. Looking at the data from that game, when a blob got five
heads out of five flips, it turned out to be a cheater
only about 88% of the time. Because of the randomness, it's impossible to be completely sure whether a blob is a cheater. But some approaches
are better than others. During the rest of this video, we're gonna build up
one of the main methods of making decisions with limited data. If you like learning new vocabulary terms, you're in for a real treat. The name of this method is
frequentist hypothesis testing. We're gonna design a test
that this blob detective can use in its day-to-day
work searching for cheaters. We want three things from this test. First, if a player is using a fair coin, we want to have a low chance
of wrongly accusing them. Second, if a player is
cheating, using an unfair coin, we want to have a high
chance of catching them. And third, we want this test to use the smallest
number of coins possible. We only have one blob detective, and we want it to be able to
test as many players as it can. And it's also nice not to
bother the players too much. We're gonna design that test together, but if you're feeling up for it, it can be good for learning to
try things on your own first. So this is your chance to
pause and take a moment to think of a test that
might satisfy these goals. Okay, let's take it one flip at a time. It came up heads. The cheaters have heads
come up more often, so this blob must be a cheater. Well, no, we can't just call
them a cheater after one flip. I mean, we could, but with that policy, we'd wrongly accuse quite
a lot of fair players. After all, even if the player
is fair, there's a 50% chance that the first flip would come out heads. So let's see the second flip. Heads again! Cheater? Well, it's more suspicious
for sure, but again, we should think about how
likely it is for us to see this if the coin is actually fair. There are two possible
outcomes for the first flip and two possible outcomes
for the second flip. Two heads in a row is one
of four possible outcomes that are all equally likely. So the probability of two out of two heads is one fourth or 25%. Another way to get that number is to multiply the probability values of the two events together. You do have to be careful about multiplying probabilities,
since it only works if the two events are
independent of each other, and that is the case here,
because getting the first heads doesn't make the second
heads more or less likely. Anyway, with a one in four chance of falsely accusing an innocent blob, it still feels a bit too early to accuse the player of cheating. After another heads, this probability is divided by two again. I'm starting to get pretty suspicious, but we'd still accuse one
out of eight innocent blobs if we accused after three heads in a row. We want that rate of false accusations to be as low as we can get it but we're never gonna get
it all the way to zero. It'll always be possible
for an innocent blob to get an epic streak of
heads and look suspicious. So we have to make a decision
about what's good enough. The standard choice here is 5%, or one false accusation out
of every 20 fair players. We could choose a different
value if we wanted to, but we might as well start here. Okay, so at this point,
we've crossed the threshold. There's only a one in 32 or 3.125% chance of seeing five heads in
a row from a fair coin. So one possible test
we could use would be, if a player gets five out of five heads, accuse them of being a cheater. Otherwise, decide they're innocent. So let's see how this test performs. We're gonna want a lot of data. So let's make a set of 1000 players where half of them are cheaters. Before we see the results,
try making some predictions. How often will it wrongfully
accuse fair players? And what fraction of the cheaters
do you think it'll catch? Alright, we can divide these
blobs into four categories. Fair players the test decided are fair. Fair players we wrongly
accused of cheating. Cheaters who got away with it. And cheaters we caught. It looks like we achieved
goal one with flying colors. Not only did we accuse fewer
than 5% of the fair players, the test did even better than expected. When we use this test in the real world, we won't know how many
fair players there are, but seeing how the test
performed on this sample, combined with our analysis from before, it feels like we can be pretty confident that we would accuse
fewer than 5% of players. We didn't catch very many cheaters, but that's not too surprising. We haven't even thought about them yet, so I'm sure we could do better. Before we make the next
version of the test, I think it's worth mentioning
some fancy statistics terms. They aren't always necessary,
but you might see them around, and like any specialized words, they do make communication
easier in some contexts. If a test result says not to
accuse a blob of cheating, it's called a negative result,
since nothing was found. And when the test does say
to accuse a blob of cheating, it's called a positive result. Cheating is a bad thing,
so not very positive, but the term positive here is referring to the test saying "Yes, the thing I'm looking for is here." The same is true for medical tests. If the test finds what it's looking for, the result is called positive, even though it's usually a bad thing. So we have positive and
negative test results, but the results may
not agree with reality. When we test a blob
that's using a fair coin, the correct result would be negative. So if the test does come up negative, it's called a true negative. And if the test comes out
positive, that's wrong, so it's called a false positive. And when we test a cheater, the correct result would be positive, so if the test does come up positive, we call it a true positive. And if the test incorrectly
gives a negative result, that's a false negative. We can also rephrase that first
goal using another new term, the false positive rate. This can be a dangerously
confusing term though. It's easy to mix up what
the denominator should be. False positive rate
sounds like you're saying, out of all the positives, what fraction of those
positives are false. Or even, out of all the tests,
how many are false positives? But really, it's saying,
out of all the fair players, how many of them are
falsely labeled positive? I've known these words for quite a while, but my brain still
automatically interprets it the wrong way basically every time. So to keep things as clear
as possible for this video, we'll keep using the longer
wording for goal one. Okay, let's go back to designing the test. We still need to figure out a way to achieve goal number two. Let's start by making
the goal more precise. To do that, we need to pick a number for the minimum fraction of
cheaters we want to catch. Using the terms from before, we could also call this the
minimum true positive rate. But again, let's stick
with the plain language. And to throw even more words at you, this minimum is sometimes called the statistical power of the test. It's the power of the
test to detect a cheater. The standard target for
statistical power is 80%. Just like the 5% number in the first goal, we could pick any value we want here. But let's run with 80% for now, and we'll talk about
different choices later on. Now for calculating what we expect the true
positive rate to be. What's the probability that a cheater would
get five heads in a row? Take a moment to try that yourself. Okay, that was kind of a trick question. There's no way to calculate that number, since we haven't actually said anything about how often an unfair
coin comes up heads. In that trial we just did with 1000 blobs, the cheaters were using
coins that land heads 75% of the time. We don't know for sure if
that's what the real blobs do. So this 75% is an assumption. But we need some number here
to calculate the probabilities, so we gotta run with something. And yet another word, this
is called the effect size. In this case, it's the effect
of using an unfair coin. You might be getting annoyed that this is the third time I've said we should just run with
some arbitrary number. But what can I tell ya? Some things are uncertain
and some things are up to us. The important thing is to remember when we're making an
assumption or making a choice. That way we can note our assumptions when we make any conclusions,
and we can adjust the test for different choices if
we change our minds later. But now that we have a number,
let's do the calculation. If the probability of each heads is 0.75, the probability of five heads
in a row is 0.75 to the fifth, or about 24%. So our existing test should
catch about 24% of cheaters. And hey, that is pretty close
to what we saw in the trial, so everything seems to
be fitting together. But our goal is to catch 80% of cheaters. The current test is a little bit extreme. It requires 100% heads
for a positive result. This does make false positives
unlikely, which is good, but it also makes true positives
unlikely, which is bad. So we're gonna have to think about a test that allows for a mixture
of heads and tails. Calculating probabilities
for something like this can be a bit confusing though. For example, if we make a new test that requires a blob to
flip their coin 10 times, and accuses them of being a cheater if they get seven or more heads, the calculations in that situation are gonna be a lot harder. There are a bunch of ways for there to be seven
heads out of 10 flips. And we also have to think
about the possibilities of eight, nine, and 10 heads. To start making sense of this, let's go back to just two flips. With a fair coin, each of these four possible
outcomes is equally likely. So the probabilities are one out of four for getting zero heads, two out of four for
getting exactly one heads, and one out of four to get two heads. But with an unfair coin that favors heads, they're skewed toward
results with more heads. With three flips, there are
eight possibilities total, with four possible numbers of heads. As we add more and more flips, it quickly becomes quite a chore to list out all the possible outcomes and add up the probabilities. But there is a pattern to it, so thankfully there's a
formula for cases like this called the binomial distribution. It's not as scary as it looks, but still a full explanation
deserves its own video. I'll put some links about
this in the description, but for now just know that this formula is what we're using to
make these bar graphs, and it follows the same pattern we used for two flips and three flips. Now let's go back to our
test rule from before, where we accuse a player if they get five out of five heads. We can show the rule on these graphs by drawing a vertical line that separates the positive
test results on the right, from the negative test
results on the left. On the fair player graph, the bars to the left
represent the true negatives, or the innocent blobs we leave alone, and to the right are the false positives, the fair players we wrongfully accuse. And on the cheater graph,
the bars to the left represent the false negatives, the cheaters who evade our detection, and the bars to the right
are the true positives, the cheaters we catch. Just like before, we
can see that this test satisfies our first goal
of accusing less than 5% of the fair players we test on average. But it doesn't satisfy our second goal of catching at least 80%
of the cheaters we test, again, on average. But now that we have these graphs, we can see what happens when
we change the number of heads. If we lower the threshold to
four or more heads out of five, we don't meet either requirement. If we keep lowering the threshold, it can allow us to meet goal two, catching more than 80% of the cheaters, but then we accuse even more
fair blobs, so that won't work. Apparently, if we want to meet
both goals at the same time, we're gonna need more flips. If we put these graphs
right next to each other, we can see that the blue
and red distributions overlap quite a lot. So it's impossible to make a test that reliably separates
fair players from cheaters. But if we increase the
number of flips to, say, 100, now there's a big gap
between the distributions, so it's easy to find a
line that separates them. But we also have this third goal of using as few coin flips as possible, so we should try to find
a happy medium somehow. Since we already have the computer set up to run the numbers, we
can go back to five flips and just keep trying different thresholds with more and more flips until we find a test rule that works. It turns out that the smallest test that meets our first two goals has a blob flip its coin 23 times, and the blob is accused of being a cheater if they get 16 or more heads. That's more than I would've
guessed at the start, but it's not so, so huge, so, it'll do. Alright, let's use this
to test a few blobs. This blob got 17 heads. That fits our rule of 16 or more, so according to that test, we should call this blob a cheater. There is another term
worth mentioning here. Assuming this blob is innocent, the probability that
they'd get 17 or more heads is about 1.7%. We call this 1.7% the P
value for this test result. It's kind of like a false positive rate for a single test result. Kind of. 1.7% is below the 5% we
set as our threshold, so according to the test,
we call this one a cheater. And looking at it from
the other direction, if the blob is cheating, using a coin that comes
up heads 75% of the time, there's a 65% chance that
they'd get 17 or more heads. Another way to say it is
that they're in the top 65% of results we'd expect from cheaters. So if we wanna catch 80% of the cheaters we'd better call this one a cheater. Okay, let's try it with one more blob. This one got 13 heads. This is more than half of the 23 flips, so it's tempting to call it a cheater. But 13 is below the 16 heads the test requires for
a positive result, so, we call it a fair player. The P value of this result is about 34%. So if we accuse players
with results like this, we'd expect to wrongly
accuse about 34% of players. That's well beyond our 5% tolerance, so we can't call it a cheater. And looking at it from
the other direction, if it were a cheater, there
would be about a 99% chance that they'd get this many heads or more. We don't have to catch 99% of the cheaters to hit our 80% goal, so we
can still meet that goal if we let this one off the hook. Is the first one really a cheater? Is that second one really playing fair? We can't know for sure, but based on how we designed our test we should expect to catch at
least 80% of the cheaters, and falsely accuse less
than 5% of the fair players. So now let's see how this test does on another group of 1000 blobs. Like before, half the blobs in this group are using a trick coin that has a 75% probability
of landing heads. Okay, the results do look about right. We accused less than
5% of the fair players, and we caught more than
80% of the cheaters. 5% and 80% are the normal
numbers for historical reasons. So we could make different
decisions if we like. Maybe we decide that we really do not
want to bother the blobs who are playing fairly. So we wanna lower the
false positive rate to 1%. To achieve this with 23 flips, we'd have to raise the
heads threshold to 18 heads. This would lower the
fraction of cheaters we catch to about 47% though. If we don't want to increase
the number of flips, we could decide we're okay with that 47%, maybe we just want cheating to feel risky, so 47% is good enough. Or, if we still want to catch
80% of the cheaters we test, we could increase the number of flips until we find a test that
achieves both of those goals. We could also be super hardcore and go for a 99% true positive rate, and a 1% false positive rate. But we'd have to flip the coin 80 times to get to that level. We'll always be able to set
two of these goals in stone, but that'll limit how well
we can do on the third goal. How to set these goals
depends which trade-offs we're willing to make. For the rest of this video though, we're just gonna go with
the standard 5% and 80%. Now that we've settled on
the goals we're going for, and we have a test that seems
to be achieving those goals, let's test one more set of blobs. To pretend these are real blobs and not some artificial sample, I'm not going to tell you
anything about this group except that there are 1000 of them. How do you think this test will do on this more mysterious group? Will it manage to accuse fewer
than 5% of the fair players? And will it catch 80% of the cheaters? At this point in the video,
it would be easy to get lazy and not actually make the predictions. But if I'm asking you
these questions yet again, something must be about
to go wrong, right? Or, maybe I'm just pretending so you'll engage a little more. Who can say? But really, what do you think? Okay, so we labeled about a
fifth of them to be cheaters, which is a bit less than before. If this were the real world,
that's all you would get. You wouldn't get to see
who was really cheating and who was really innocent to get confirmation that the
test is working as expected. I mean, maybe you could, but
it would take more testing. You couldn't do it with this test alone. But because this is a computer simulation, I do know the full truth. This group was 90% cheaters. We still accused less than
5% of the fair players, but we only caught about
a quarter of the cheaters. Something went wrong. The problem is that we assumed that the cheater coins came
up heads 75% of the time. And that assumption was wrong. The real world cheaters were using coins that came up heads 60% of the time. If we knew that from the beginning, we still could have designed
a test to achieve our goals, but it would need 158 flips and require 90 heads to
reach those same thresholds, which is honestly way more coin
flips than I was expecting. But in hindsight, it's not that surprising that we need a lot of data to tease out that smaller difference. But we didn't design that test because we got the effect size wrong. I know, I know, I was the one
who said we should assume 75%. But be honest with yourself. Did you remember that assumption when making your prediction? It's very easy to forget that
assumptions are assumptions, and instead just treat them as facts. This concludes me tricking you
to try to teach you a lesson, but they really are easy
mistakes to make in real life. On the bright side, though,
our test did succeed at accusing less than
5% of the fair players. The framework we built up here isn't just good for catching unfair coins. It's the dominant framework used in actual scientific studies. To summarize we take a yes or no question. In this case, our question was, is this particular blob
using a biased coin? But it could be any question. Then we come up with a model for what kinds of results we'd
expect if the answer is yes, and if the answer is no. Then we come up with a test
that can do a decent job of telling those two situations apart, according to the models. The details are usually
a bit more complicated, since most real world
systems that we wanna study are more complicated than coin flips. But most scientific studies have this framework at their core. Like I mentioned at the beginning this is called frequentist
hypothesis testing. There's another method called
Bayesian hypothesis testing which we'll look at in the
next video in this series. See you then.
