This video should help you to gain an intuitive
understanding of ROC curves and Area Under the Curve, also known as AUC. An ROC curve is a commonly used way to visualize
the performance of a binary classifier, meaning a classifier with two possible output classes. For example, let's pretend you built a classifier
to predict whether a research paper will be admitted to a journal, based on a variety
of factors. The features might be the length of the paper, the number of authors, the number
of papers those authors have previously submitted to the journal, et cetera. The response (or
"output variable") would be whether or not the paper was admitted. Let's first take a look at the bottom portion
of this diagram, and ignore the everything except the blue and red distributions. We'll
pretend that every blue and red pixel represents a paper for which you want to predict the
admission status. This is your validation (or "hold-out") set, so you know the true
admission status of each paper. The 250 red pixels are the papers that were actually admitted,
and the 250 blue pixels are the papers that were not admitted. Since this is your validation set, you want
to judge how well your model is doing by comparing your model's predictions to the true admission
statuses of those 500 papers. We'll assume that you used a classification method such
as logistic regression that can not only make a prediction for each paper, but can also
output a predicted probability of admission for each paper. These blue and red distributions
are one way to visualize how those predicted probabilities compare to the true statuses. Let's examine this plot in detail. The x-axis
represents your predicted probabilities, and the y-axis represents a count of observations,
kind of like a histogram. Let's estimate that the height at 0.1 is 10 pixels. This plot
tells you that there were 10 papers for which you predicted an admission probability of
0.1, and the true status for all 10 papers was negative (meaning not admitted). There
were about 50 papers for which you predicted an admittance probability of 0.3, and none
of those 50 were admitted. There were about 20 papers for which you predicted a probability
of 0.5, and half of those were admitted and the other half were not. There were 50 papers
for which you predicted a probability of 0.7, and all of those were admitted. And so on. Based on this plot, you might say that your
classifier is doing quite well, since it did a good job of separating the classes. To actually
make your class predictions, you might set your "threshold" at 0.5, and classify everything
above 0.5 as admitted and everything below 0.5 as not admitted, which is what most classification
methods will do by default. With that threshold, your accuracy rate would be above 90%, which
is probably very good. Now let's pretend that your classifier didn't
do nearly as well and move the blue distribution. You can see that there is a lot more overlap
here, and regardless of where you set your threshold, your classification accuracy will
be much lower than before. Now let's talk about the ROC curve that you
see here in the upper left. So, what is an ROC curve? It is a plot of the True Positive
Rate (on the y-axis) versus the False Positive Rate (on the x-axis) for every possible classification
threshold. As a reminder, the True Positive Rate answers the question, "When the actual
classification is positive (meaning admitted), how often does the classfier predict positive?"
The False Positive Rate answers the question, "When the actual classification is negative
(meaning not admitted), how often does the classifier incorrectly predict positive?"
Both the True Positive Rate and the False Positive Rate range from 0 to 1. To see how the ROC curve is actually generated,
let's set some example thresholds for classifying a paper as admitted. A threshold of 0.8 would classify 50 papers
as admitted, and 450 papers as not admitted. The True Positive Rate would be the red pixels
to the right of the line divided by all red pixels, or 50 divided by 250, which is 0.2.
The False Positive Rate would be the blue pixels to the right of the line divided by
all blue pixels, or 0 divided by 250, which is 0. Thus, we would plot a point at 0 on
the x-axis, and 0.2 on the y-axis, which is right here. Let's set a different threshold of 0.5. That
would classify 360 papers as admitted, and 140 papers as not admitted. The True Positive
Rate would be 235 divided by 250, or 0.94. The False Positive Rate would be 125 divided
by 250, or 0.5. Thus, we would plot a point at 0.5 on the x-axis, and 0.94 on the y-axis,
which is right here. We've plotted two points, but to generate
the entire ROC curve, all we have to do is to plot the True Positive Rate versus the
False Positive Rate for all possible classification thresholds which range from 0 to 1. That is
a huge benefit of using an ROC curve to evaluate a classifier instead of a simpler metric such
as misclassification rate, in that an ROC curve visualizes all possible classification
thresholds, whereas misclassification rate only represents your error rate for a single
threshold. Note that you can't actually see the thresholds used to generate the ROC curve
anywhere on the curve itself. Now, let's move the blue distribution back
to where it was before. Because the classifier is doing a very good job of separating the
blues and the reds, I can set a threshold of 0.6, have a True Positive Rate of 0.8,
and still have a False Positive Rate of 0. Therefore, a classifier that does a very good
job separating the classes will have an ROC curve that hugs the upper left corner of the
plot. Conversely, a classifier that does a very poor job separating the classes will
have an ROC curve that is close to this black diagonal line. That line essentially represents
a classifier that does no better than random guessing. Naturally, you might want to use the ROC curve
to quantify the performance of a classifier, and give a higher score for this classifier
than this classifier. That is the purpose of AUC, which stands for Area Under the Curve.
AUC is literally just the percentage of this box that is under this curve. This classifier
has an AUC of around 0.8, a very poor classifier has an AUC of around 0.5, and this classifier
has an AUC of close to 1. There are two things I want to mention about
this diagram. First, this diagram shows a case where your classes are perfectly balanced,
which is why the size of the blue and the red distributions are identical. In most real-world
problems, that is not the case. For example, if only 10% of papers were admitted, the blue
distribution would be nine times larger than the red distribution. However, that doesn't
change how the ROC curve is generated. A second note about this diagram is that it
shows a case where your predicted probabilities have a very smooth shape, similar to a normal
distribution. That was just for demonstration purposes. The probabilities output by your
classifier will not necessarily follow any particular shape. To close, I want to add three other important
notes. The first note is that the ROC curve and AUC are insensitive to whether your predicted
probabilities are properly calibrated to actually represent probabilities of class membership.
In other words, the ROC curve and the AUC would be identical even if your predicted
probabilities ranged from 0.9 to 1 instead of 0 to 1, as long as the ordering of observations
by predicted probability remained the same. All the AUC metric cares about is how well
your classifier separated the two classes, and thus it is said to only be sensitive to
rank ordering. You can think of AUC as representing the probability that a classifier will rank
a randomly chosen positive observation higher than a randomly chosen negative observation,
and thus it is a useful metric even for datasets with highly unbalanced classes. The second note is that ROC curves can be
extended to classification problems with three or more classes using what is called a "one
versus all" approach. That means if you have three classes, you would create three ROC
curves. In the first curve, you would choose the first class as the positive class, and
group the other two classes together as the negative class. In the second curve, you would
choose the second class as the positive class, and group the other two classes together as
the negative class. And so on. Finally, you might be wondering how you should
set your classification threshold, once you are ready to use it to predict out-of-sample
data. That's actually more of a business decision, in that you have to decide whether you would
rather minimize your False Positive Rate or maximize your True Positive Rate. In our journal
example, it's not obvious what you should do. But let's say your classifier was being
used to predict whether a credit card transaction might be fraudulent and thus should
be reviewed by the credit card holder. The business decision might be to set the threshold
very low. That will result in a lot of false positives, but that might be considered acceptable
because it would maximize the true positive rate and thus minimize the number of cases
in which a real instance of fraud was not flagged for review. In the end, you will always have to choose
a classification threshold, but the ROC curve will help you to visually understand the impact
of that choice. Thanks very much to Navan for creating this
excellent vizualization. Below this video, I've linked to it as well as a very readable
paper that provides a much more in-depth treatment of ROC curves. I also welcome your questions
in the comments.
