>> On this special build
edition of the AI Show, we'll get to hear
from Scott Lundberg, Senior Researcher on the
Microsoft Research team. It is definitely
important to debug and explain your machine learning models. In this video, Scott will
explain the science behind SHAP values and how
they can be used to explain and debug your models.
Make sure you tune in. [MUSIC] >> Hi, my name is Scott Lundberg. I'm a Senior Researcher here
at Microsoft Research AI, and I look forward today to diving into the
background behind SHAP, which is a tool in research designed to use Shapley values from game theory to explain
machine learning models. So to understand how this works, let's start with a model
that we're going to explain. This model, let's imagine, works at a bank and takes information about
customers like John and outputs predictions
about their likelihood of having repayment problems if
the bank were to give him a loan. In this case, the repayment
problem is a bit high, so the banks unlikely
to give him a loan. As a data scientist responsible
for building this model, you may have used a whole variety of different packages in the
model development process. Anything from Scikit-learn,
or linear models, or trees, gradient
booster decision stuff, deep networks, all in pursuit of producing a good,
accurate, high-quality model. But in that process, a
lot of these things are very complicated and very opaque. Which means in order
to debug these things, you need to be able
to interpret them. So that's one huge motivation
for interpretability and explainability is the ability to
debug and understand your model. Not just for the data
scientists though, it's also for customers. There's even legal requirements
in the finance domain. But in many areas, you need to be able to communicate to a customer why a model is
making decision about them. Also for businesses, it would
depend on these models. Understanding how
they work and, hence, when they can break is
extremely important in order to manage the risk that is taken
in these businesses for models. All of these motivate how important it is to have
interpretability and explainability. So how does SHAP help with this? Well, if we go back to John, it's important to understand that whenever a model makes a prediction, it's always got some prior in mind, in the sense that there's
always some base rate that we would have predicted
if we knew nothing about John. In this case, it could
be our average or the training dataset of defaults, or could be some test dataset
that we have in mind, or some particular group of people, like all the people who got accepted. Whatever that background
prior knowledge is, that's actually where
we start when we don't know anything about prediction. But John didn't get
predicted the base rate, which in this case was 16
percent of our dataset. That's just the expected
value for our model's output. He got predicted 22 percent. So what SHAP does,
it says, "Hey look, we need to explain not
22 percent from zero, because zero is just
an arbitrary number. What we need to explain is how we got from the base rate where we knew nothing about John to the
current prediction for John, which is 22 percent." How do we go about doing this? Well, essentially, we can
look at this expectation of the model's prediction over our
training dataset in this case, and then we can fill out John's
application one field at a time. In this case, we're filling out
that his income is not verified. Now, what that does is it bumps up the expected value of the
model by 2.2 percent. So we can say that 2.2 percent must be attributable to the fact that John didn't
have his income verified. So relative [inaudible] and training dataset this increases his risk. If we do the same thing for
his debt-to-income ratio, we see that that is at 30, which bumps him up to 21 percent. Then we see that he had a
delinquent payment 10 months ago, which further increases his
risk up to 22.5 percent. Again, we're filling out his
application one entry at a time. Then we throw out the fact that he
had no recent account openings, and this drops his risk significantly because not applying for
credit is a good sign. But then finally, we fill in the fact that he has 46
years of credit history, which you would think would
be a really good thing, but ironically in this case, it turns out that that hurts him significantly and bumps
his risk up to 22 percent. So now what we've done is we've
filled out his entire application and we've arrived at the
prediction of the model, but we've done it
piece by piece so that we can attribute each
piece to each feature. Hence, explain how we
got from when we knew nothing about John to the
model's final prediction. Now, let's back up and
see how this works for a simple linear regression model
from scikit-learn, for example. So here's a model trained on this lending dataset
where it's a linear model, and I'm showing you
a straight line that represents the partial dependence
plot of that linear model. What we can see also on x-axis we
have the feature I'm explaining, which in this case happens
to be annual income. Then on the y-axis, we just have the axis for
partial dependence plot, which happens to be the
expected value that models output when we change one feature, which for a linear model
is a straight line. What I want to highlight
here is how easy it is for a linear model to read off the SHAP value from a
partial dependence plot. So the gray line here is just
the average output of the model. But that's the prior base
rate we were talking about. Then what we can see is that the
SHAP value is just the difference between that average and the partial dependence plot for the value of the feature
we're interested in. For a linear model, we can simply look and say, "John has made $140,000. That puts his partial dependence
plot at a certain point." Then we can just measure that
height from the mean value, which if it's higher than the average, it's just
going to be positive. If it's lower than the average,
it's going to be negative. We can do the exact same thing
for more complicated models like generalized additive models
such as whether an EVM package. What happens there is now you don't
have a straight line anymore. You can have a much more flexible
partial dependence plot. But again, the SHAP values, because there's no
interaction effects going on, we still have an additive model, the SHAP values are again, just exactly the difference between the height of the
partial dependence plot and the expected value of the model. So if you were to
plot the SHAP values for many different individuals, you would get essentially a line, and that line would be exactly the mean centered
partial dependence plot. So more complicated models though, or of course, where people are most interested in this kind of stuff. In that case, you can't
just use a single order. You can't just introduce
features one at a time, because it turns out that the orders you introduced features matters. If there's an and function
or an or function, the first or second one you
introduce will get all the credit. So here's an example on
a real dataset where we say no recent account openings
in 46 years of credit history, we've filled out account openings
first and then credit history. What if, in filling out
this application we first fill out credit history
and then account openings? Turns out, it makes
a huge difference. What that means is there's
a strong interaction effect between credit history
and account openings. That's where SHAP comes
in to try and fairly distribute the effects that are going on in high level interactions. You can say, "How on earth
are we going to do this?" Well, it turns out we
can go back in the 1950s and rely on some very solid
theory and game theory that is all about how
to do this fairly in complicated games with lots of interacting players that have
higher-order interaction effects. How can we share those interaction
effects fairly among all of the players such that a set of
basic axioms are satisfied? It turns out there's
only one way to do it. It came from values that are now called the Shapley values
after Lloyd Shapley. Lloyd Shapley did a
lot of great work in game theory and allocation
and things like this, and actually got a
Nobel Prize in 2012. So this is based on some solid math. So going back to our data scientist, you can say, "That's great. I'm really convinced by this. I think I should use these values. How do I compute
them?" Well, it turns out that result from averaging, just we did talk about before, using a single ordering, but we
have to do it over all orderings. Because it's computationally
intractable, and it's even worse because it's NP-hard, if you know what that means. So that's where the real
challenge in these values lie, it's how to compute these
things efficiently. I'm not going to go into the
algorithms that allow us to do that, but that's at the heart of what is in the SHAP package and
the research behind it. It is designed to
enable us to compute these very well-justified values efficiently and effectively
on real datasets. If we do that for extra boost, we can actually solve it in polynomial
time very quickly, exactly. Now we see that the SHAP values
no longer exactly matched the partial dependence plot because they're accounting for
these interaction effects. Because when you look at a
partial dependence plot, you're losing all the higher-order
interacting information about ands and ors that
your model may be doing. But the SHAP values
account for that and then drop that credit
down onto each feature. So you'll see vertical
dispersion when you plot many people's SHAP
values for a feature. So let's do this for a particular feature to dive
into this credit history. Because remember, it was a bit
surprising that credit history hurt John's credit score. So if we plot credit history versus the SHAP value
for that credit history, we get a dot for every person. Again, a little bit of
vertical dispersion from the interaction effects. Then if we look at John, we'll see he's in this tail here at the end where
he's got a really, really long credit history. It doesn't take too long before you realize that debugging
was super important because this model was actually identifying retirement
age individuals based on their long
credit histories and increase the risk of
default for them. This is a big problem because
age is a protected class. So this essentially found that credit history with
a complicated model was able to pull out credit history
and use it as a proxy for age. So it's really important to
explain and debug your models. So we've talked a little bit about just one example of explainable AI in practice and debugging
and at model exploration. But I'd like to highlight the fact that there are so many other ways to use these types of
interpretability in your workflow. You can monitor models by
explaining their error over time. You can encode prior beliefs
about models and then use explanations to actually
control your model train process. You can talk about
customer retention by supporting call centers by explaining
why a churn model was done. We've applied this in decision
support for medical settings. There's a lot of places where
human risk oversight of machine learning models is
enhanced with explanations. In regulatory compliance,
there's a lot of need for these type of transparency
for consumer explanations. It can help you better understand anti-discrimination as we
just showed an example of, and of course, risk-management, where you're understanding
what your model will do when economic conditions change. All the more important right now. Even in scientific discovery, you can find that explanations can help you better do
population subtyping, extremely helpful for
pattern discovery and even signal recovery of things
inside, things like DNA. All of these things are just tons of downstream applications
of interpretable ML that's supported by these
types of tools and research, and I hope that this insight
has given you a bit of a taste and excitement for what
can be done here. Thanks. [MUSIC]
