PETER SZOLOVITS: OK. Today's topic is
differential diagnosis. And so I'm just
quoting Wikipedia here. Diagnosis is the identification
of the nature and cause of a certain phenomenon. And differential diagnosis
is the distinguishing of a particular
disease or condition from others that present
similar clinical features. So doctors typically talk
about differential diagnosis when they're faced
with a patient and they make list of what
are the things that might be wrong with this patient. And then they go
through the process of trying to figure out
which one it actually is. So that's what we're
going to focus on today. Now, just to scare you,
here's a lovely model of human circulatory physiology. So this is from Guyton's
textbook of cardiology. And I'm not going to hold
you responsible for all of the details of this
model, but it's interesting, because this is, at least
as of maybe 20 years ago, the state of the art of how
people understood what happens in the circulatory system. And it has various
control inputs that determine things like
how your hormone levels change various aspects of the
cardiovascular system and how the interactions
between different components of the cardiovascular
system affect each other. And so in principle, if I
could tune this model to me, then I could make all kinds
of pretty good predictions that say if I increase my
systemic vascular resistance, then here's what's
going to happen as the rest of the
system adjusts. And if I get a blockage
in a coronary artery, then here's what's going to
happen to my cardiac output and various other things. So this would be terrific. And if we had this
kind of model for not just the cardiovascular system,
but the entire body, then we'd say, OK, we've solved medicine. Well, we don't have this kind
of model for most systems. And also, there's
this minor problem that if I give you this
model and say, "How does this relate to a
particular patient?", how would you figure that out? This has hundreds of
differential equations that are being represented
by this diagram. And they have many
hundreds of parameters. And so we were joking when we
started working with this model that you'd really have to
kill the patient in order to do enough measurements to
be able to tune this model to their particular physiology. And of course, that's probably
not a good practical approach. We're getting a little better
by developing more non-invasive ways of measuring these things. But that's moving
along very slowly. And I don't expect that I
or maybe even any of you will live long enough that
sort of this approach to doing medical reasoning
and medical diagnosis is actually going to happen. So what we're going
to look at today is what simpler models are
there for diagnostic reasoning. And I'm going to take
the liberty of inflicting a bit of history on you, because
I think it's interesting where a lot of these ideas came from. So the first idea was
to build flowcharts. Oh, and by the way,
the signs and symptoms, I've forgotten if we've talked
about that in the class. So a sign is something
that a doctor sees, and a symptom is something
that the patient experiences. So a sign is objective. It's something that can
be told outside your body. A symptom is something
that you feel. So if you're feeling
dizzy, then that's a symptom, because it's not
obvious to somebody outside you that you're dizzy, or that you
have a pain, or such things. Normally, we talk about
manifestations or findings, which is sort of a super
category of all the things that are determinable
about a patient. So we'll talk about
flowcharts, models based on associations
between diseases and these manifestations. Then there are some issues
about whether you're trying to diagnose a single
disease or a multiplicity of diseases, which makes the
models much more complicated whether you're trying to
do probabilistic diagnosis or definitive or categorical. And then we'll talk about some
utility theoretic methods. And I'll just mention some
rule-based and pattern-matching kinds of approaches. So this is kind of cute. This is from 1973. And if you were a woman and
walked into the MIT Health Center and complained
of potentially a urinary tract infection,
they would take out this sheet of paper, which
was nicely color-coded, and they would check
a bunch of boxes. And if you hit a red box,
that represented a conclusion. And otherwise, it
gave you suggestions about what further tests to do. And this was essentially
a triage instrument. It said, does this woman
have a problem that requires immediate attention? And so we should either
call an ambulance and take them to a hospital,
or is it something where we can just tell them to
come back the next day and see a doctor,
or is it in fact some self-limited thing where
we say, take two aspirin, and it'll go away. So that was the attempt here. Now, interestingly, if
you look at the history of this project between the
Beth Israel Hospital and Lincoln Laboratories, it started
off as a computer aid. So they were building
a computer system that was supposed to do this. And then in-- but you can
imagine, in the late 1960s, early 1970s, computers
were pretty clunky. PCs hadn't been invented yet. So this was like mainframe
kinds of operations. It was very hard to use. And so they said, well, this
is a small enough program that we can reduce it to
about 20 flow sheets-- 20 sheets like this, which
they proceeded to print up. And I was amused,
because in the-- around 1980, I was working
in my office one night. And I got this
splitting headache. And I went over to MIT medical. And sure enough,
the nurse pulled out one of these sheets
for headaches and went through it
with me and decided that a couple of
Tylenols should fix me. But it was interesting. So this was really
in use for a while. Now, the difficulty with
approaches like this, of which there have
been many, many, many in the medical world,
is that they're very fragile. They're very specific. They don't take account
of unusual cases. And there's a lot of effort
in coming to consensus to build these things. And then they're not necessarily
useful for a long time. So MIT actually stopped
using them shortly after my headache experience. But if you go over
to a hospital and you look on the bookshelf
of a junior doctor, you will still find
manuals that look kind of like this
that say, how do we deal with tropical diseases? So you ask a bunch of
questions, and then depending on the branching
logic of the flowchart, it'll tell you whether
this is serious or not. And the reason is because if
you do your medical training in Boston, you're
not going to see very many tropical diseases. And so you don't have
a base of experience on the basis of which
you can learn and become an expert at doing it. And so they use this as
a kind of cheat sheet. I mentioned that the association
between diseases and symptoms is another important
way of doing diagnosis. And I swear to you, there was
a paper in the 1960s, I think, that actually proposed this. So if any of you have hung
around ancient libraries, libraries used to have
card catalogs that were physical pieces
of paper, cardboard. And one of the things they
did with these was each card would be a book. And then around the edges
were a bunch of holes, and depending on
categorizations of the book along various dimensions,
like its Dewey decimal number, or the top digits of
its Library of Congress number or something, they would
punch out holes in the borders. And this allowed
you to do a kind of easy sorting of these books. So if you've got
a bunch of cards together when people were
returning their books and you pulled a bunch of cards. And you wanted to find
all the math books. So what you would do is you'd
stick a needle through the hole that represented math books,
and then you shake the pile, and all the math
books would fall out because they had punched. So somebody seriously proposed
this as a diagnostic algorithm. And in fact, implemented it. And was trying to
even make money on it. I think this was an attempt at
a commercial venture, where they were going to provide doctors
with these library cards that represented diseases. And the holes now
represented not mathematics versus literature,
but they represented shortness of breath versus
pain in the left ankle versus whatever. And again, as people
came in and complained about some condition,
you'd stick a needle through that condition
and you'd shake, and up would come the cards that
had that condition in common. So one of the obvious
problems with this approach is that if you had two
things wrong with you, then you would wind up with no cards
very quickly, because nothing would fall out of the pile. So this didn't go anywhere. But interestingly,
even in the late 1980s, I remember being asked by the
board of directors of the New England Journal of Medicine
to come to a meeting where they had gotten a pitch
from somebody who was proposing essentially exactly
this diagnostic model, except implemented in a computer
now and not in these library cards. And they wanted to know
whether this was something that they ought to get
behind and invest in. And I and a bunch
of my colleagues assured them that this was
probably not a great idea and they should stay away
from it, which they did. Well, a more sophisticated model
is something like a Naive Bayes model that says if
you have a disease-- where is my cursor? If you have a disease, and you
have a bunch of manifestations that can be caused
by the disease, we can make some
simplifying assumptions that say that you
will only ever have one disease at a
time, which means that the values of that node D
form an exhaustive and mutually exclusive set of values. And we can assume that
the manifestations are conditionally independent
observables that depend only on the disease that you
have, but not on each other or not on any other factors. And if you make that
assumption, then you can apply good old
Thomas Bayes's rule. This, by the way, is
the Reverend Bayes. Do you guys know his history? So he was a nonconformist
minister in England. And he was not a
mathematician, except I mean, he was an amateur mathematician. But he decided that
he wanted to prove to people that God existed. And so he developed
Bayesian reasoning in order to make this proof. And so his argument
was, well, suppose you're completely in doubt. So you have 50/50
odds that God exists. And then you say,
let's look at miracles. And let's ask, what's the
likelihood of this miracle having occurred if God exists
versus if God doesn't exist? And so by racking up
a bunch of miracles, you can convince people more
and more that God must exist, because otherwise
all these miracles couldn't have happened. So he never publish this in his
lifetime, but after his death one of his colleagues
actually presented this as a paper at the Royal
Society in the UK. And so Bayes became
famous as the originator of this notion of how to do
probabilistic reasoning about at least fairly
simple situations, like in his case, the existence
or nonexistence of God. Or in our case, the
cause of some disease, the nature of some disease. And so you can draw these trees. And Bayes's rule is very simple. I'm sure you've all seen it. One thing that, again,
makes contact with medicine is that a lot of times,
you're not just interested in the impact of one
observable on your probability distribution, but you're
interested in the impact of a sequence of observations. And so one thing you
can do is you can say, well, here is my
general population. So let's say disease 2 has
37% prevalence and disease 1 has 12%, et cetera. And now I make some observation. I apply Bayes's rule. And I revise my
probability distribution. So this is the equivalent of
finding a smaller population of patients who have
all had whatever answer I got for symptom 1. And then I just keep doing that. And so this is the sequential
application of Bayes's rule. And of course, it does depend
on the conditional independence of all these symptoms. But in medicine,
people don't like to do math, even
arithmetic much. And they prefer doing addition
rather than multiplication, because it's easier. And so what they've
done is they said, well, instead of
representing all this data in a probabilistic framework,
let's represent it as odds. And if you represent it
as odds, then the odds of some disease given
a bunch of symptoms, given the independence
assumption, is just the prior
odds of the disease times the conditional
odds, the likelihood ratio of each of the symptoms
that you've observed. So you've just got to
multiply these together. And then because they like
adding more than multiplying, they said, let's take
the log of both sides. And then you can just add them. And so if you remember when I
was talking about medical data, there are things
like the Glasgow Coma score, or the APACHE
score, or various measures of how badly or well a patient
is doing that often involve adding up numbers corresponding
to different conditions that they have. And what they're
doing is exactly this. They're applying
sequentially Bayes's rule with these independence
assumptions in the form of logs rather than
multiplications, log odds, and that's how they're doing it. Very quickly. Somebody in a previous lecture
was wondering about receiver operator characteristic curves. And I just wanted to give you a
little bit of insight on those. So if you do a test on two
populations of patients-- the red ones are sick patients. The blue ones are
not sick patients. You do some test. What you expect is that
the result of that test will be some continuous
number, and it'll be distributed something
like the blue distribution for the well patients
and something like the red distribution
for the ill patients. And typically, we
choose some threshold. And we say, well,
if you choose this to be the threshold between
a prediction of sick or well, then what
you're going to get is that the part of the
blue distribution that lies to the right is
the false positives and the part of the
red distribution that lies to the left is
the false negatives. And often people will choose the
lowest point at which these two curves intersect as the
threshold, but that, of course, isn't necessarily the case. Now, if I give
you a better test, one like this, that's terrific,
because there is essentially no overlap. Very small false negative
and false positive rates. And as I said, you
can choose to put the threshold in
different places, depending on how you
want to trade off sensitivity and specificity. And we measure this by
this receiver operator characteristics curve,
which has the general form that if you get a
curve like this, that means that there's an
exact trade-off for sensitivity and specificity, which is the
case if you're flipping coins. So it's random. And of course, if you manage
to hit the top corner up there, that means that there
would be no overlap whatsoever between
the two distributions, and you would get
a perfect result. And so typically you get
something in between. And so normally,
if you do a study and your AUC, the area
under this receiver operator characteristics curve,
is barely over a half, you're pretty
close to worthless, whereas if it's
close to 1, then you have a really good
method for distinguishing these categories of patients. Next topic. What does it mean
to be rational? I should have a
philosophy course here. AUDIENCE: Are you
talking about pi? PETER SZOLOVITS: Sorry. AUDIENCE: Are you
talking about pi? Pi is-- PETER SZOLOVITS:
Pi is irrational, but that's not what
I'm talking about. Well, so there is this
principle of rationality that says that what you want
to do is to act in such a way as to maximize your
expected utility. So for example, if
you're a gambler and you have a choice of various
ways of betting in some poker game or something, then if
you were a perfect calculator of the odds of getting a
queen on your next draw, then you could make
some rational decision about whether to
bet more or less, but you'd also have to take into
account things like, "How could I convince my opponent that I am
not bluffing if I am bluffing?" and "How could I convince them
that I'm bluffing if I'm not bluffing?" and so on. So there is a
complicated model there. But nevertheless,
the idea is that you should behave in a
way that will give you the best expected outcome. And so people joke that
this is Homo economicus, because economists make
the assumption that this is how people behave. And we now know that that's
not really how people behave. But it's a pretty common
model of their behavior, because it's easy
to compute, and it has some appropriate
characteristics. So as I mentioned,
every action has a cost. And utility measures the
value or the goodness of some outcome, which is the
amount of money you've won, or whether you live or die, or
quality adjusted life years, or various other
measures of utility-- how much it costs for
your hospitalization. So let me give you an example. This actually comes from a
decision analysis service at New England
Medical Center Tufts Hospital in the late 1970s. So this was an elderly
Chinese gentleman whose foot had gangrene. So gangrene is an infection
that usually people who have bad circulation
can get these. And what he was
facing was a choice of whether to amputate
his foot or to try to treat him medically. To treat him medically
means injecting antibiotics into his system and hoping that
his circulation is good enough to get them to the
infected areas. And so the choice becomes
a little more complicated, because if the medical treatment
fails, then, of course, the patient may
die, a bad outcome. Or you may have to now
amputate the whole leg, because the gangrene has spread
from his foot up the foot, and now you're
cutting off his leg. So what should you do? And how should you
reason about this? So Pauker's staff came up
with this decision tree. By the way, decision tree in
this literature means something different from decision
tree in like C4.5. So your choices here
are to amputate the foot or start with medical care. And if you amputate
the foot, let's say there is a 99% chance
that the patient will live. There's a 1% chance that maybe
the anesthesia will kill him. And if we treat
him medically, they estimated that there is a
70% chance of full recovery, a 25% chance that he'd
get worse, a 5% chance that he would die. If he got worse, you're now
faced with another decision, which is, do we
amputate the whole leg or continue pushing medicine? And again, there are various
outcomes with various estimated probabilities. Now, the critical thing here
that this group was pushing was the idea that
these decisions shouldn't be based on what the
doctor thinks is good for you. They should be based on what
you think is good for you. And so they worked
very hard to try to elicit individualized
utilities from patients. So for example, this guy said
that having your foot amputated was worth 850 points on a scale
of 1,000 where being healthy was 1 and being dead was 0. Now, you could imagine
that that number would be very different
for different individuals. If you asked LeBron
James how bad would it be to have your
foot amputated, he might think that it's
much worse than I would, because it would be a pain
to have my foot amputated, but I could still do
most of the things that I do professionally,
whereas he probably couldn't as a star basketball player. So how do you solve
a problem like this? Well, you say, OK,
at every chance node I can calculate the expected
value of what happens here. So here at it's 0.6
times 995, 0.4 times 0. That gets me a value
for this decision. Do the same thing here. I compare the values here
and choose the best one. That gives me a value
for this decision. And so I fold back
this decision tree. And my next slide should have-- yeah, so these are the
numbers that you get. And what you discover
is that the utility of trying medical treatment
is somewhat higher than the utility of
immediately amputating the foot if you believe these
numbers and those utilities, these probabilities
and those utilities. Now, the difficulty is that
these numbers are fickle. And so you'd like to do some
sort of sensitivity analysis. And you say, for example,
what if this gentleman valued his living
with an amputated foot at 900 rather than 850. And now you discover
that amputating the foot looks like a slightly better
decision than the other. So this is actually applied
in clinical medicine. And there are now
thousands of doctors who have been trained
in these techniques and really try to work through
this with individual patients. Of course, it's used much more
on an epidemiological basis when people look at
large populations. AUDIENCE: I have a question. PETER SZOLOVITS: Yeah. AUDIENCE: How are the
probabilities assessed? PETER SZOLOVITS:
So the service that did this study would
read the literature, and they would
look in databases. And they would try to
estimate those probabilities. We can do a lot better today
than they could at that time, because we have a lot more
data that you can look in. But you could say,
OK, for people-- men of this age who have
gangrenous feet, what fraction of them have
the following experience? And that's how
these are estimated. Some of it feels like 5%. OK. So I just said this. And then the
question of where do you get these utilities
is a tricky one. So one way is to do the
standard gamble, which says, OK, Mr. Szolovits,
we're going to play this game. We're going to roll a fair die
or something that will come up with some continuous
number between 0 and 1, and then I'm going to play the
game where either I chop off your foot, or I roll
this die and if it exceeds some threshold,
then I kill you. Nice game. So now if you find the point
at which I'm indifferent, if I say, well, 0.8, that's
a 20% chance of dying. It seems like a lot. But maybe I'll go
for 0.9, right? Now you've said,
OK, well, that means that you value
living without a foot at 0.9 of the value
of being healthy. So this is a way of doing it. And this is typically done. Unfortunately, of course,
it's difficult to ascertain the problem. And it's also not stable. So people have done experiments
where they get somebody to give them this kind of
number as a hypothetical, and then when that
person winds up actually faced with such a
decision, they no longer will abide with that number. So they've changed their mind
when the situation is real. AUDIENCE: But it's nice, because
there are two feet, right? So you could run this
experiment and see. PETER SZOLOVITS: They
didn't actually do it. It was hypothetical. OK. Next program I want to
tell you about, again, the technique for this was
developed as a PhD thesis here at MIT in 1967. So this is hot off the presses. But it's still used,
this type of idea. And so this was a program that
was published in the American Journal of Medicine, which is
a high impact medical journal. I think this was
actually the first sort of computational program
that journal had ever published as a medical journal. And it was addressed
at the problem of the diagnosis of acute
oliguric renal failure. Oliguric means you're
not peeing enough. Renal is your kidney. So this is something's gone
wrong with your kidney, and you're not
producing enough urine. Now, this is a good problem to
address with these techniques, because if something
happens to you suddenly, it's very likely that
there is one cause for it. If you are 85 years old
and you have a little heart disease and a little
kidney disease and a little liver disease
and a little lung disease, there's no guarantee that there
was one thing that went wrong with you that caused all these. But if you were OK yesterday
and then you stopped peeing, it's pretty likely that there's
one thing that's gone wrong. So it's a good
application of this model. So what they said is there
are 14 potential causes. And these are exhaustive
and mutually exclusive. There are 27 tests or
questions or observations that are relevant
to the differential. These are cheap
tests, so they didn't involve doing anything
either expensive or dangerous to the patient. It was measuring
something in the lab or asking questions
of the patient. But they didn't want to
have to ask all of them, because that's pretty tedious. And so they were
trying to minimize the amount of
information that they needed to gather
in order to come up with an appropriate decision. Now, the real problem, there
were three invasive tests that are dangerous
and expensive, and then eight
different treatments that could be applied. And I'm only going to tell
you about the first part of this problem. This 1973 article shows you
what the program looked like. It was a computer terminal
where it gave you choices, and you would type in an answer. And so that was the state
of the art at the time. But what I'm going to
do is, god willing, I'm going to demonstrate
a reconstruction that I made of this program. So these guys are the potential
causes of stopping to pee-- acute tubular necrosis,
functional acute renal failure, urinary tract obstruction, acute
glomerulonephritis, et cetera. And these are the
prior probabilities. Now, I have to warn
you, these numbers were, in fact, estimated by
people sticking their finger in the air and figuring out
which way the wind was blowing, because in 1973, there were not
great databases that you could turn to. And then these are
the questions that were available to be asked. And what you see in
the first column, at least if you're sitting
close to the screen, is the expected entropy of
the probability distribution if you answered this question. So this is basically saying, if
I ask this question, how likely is each of the possible answers,
given my disease distribution probabilities? And then for each
of those answers, I do a Bayesian revision,
then I weight the entropy of that resulting distribution
by the probability of getting that answer. And that gets me
the expected entropy for asking that question. And the idea is that the
lower the expected entropy, the more valuable the question. Makes sense. So if we look, for example,
the most valuable question is, what was the blood pressure
at the onset of oliguria? And I can click on this
and say it was, let's say, moderately elevated. And what this little
colorful graph is showing you is that if you
look at the initial probability distribution, acute tubular
necrosis was about 25%, and has gone down to
a very small amount, whereas some of these
others have grown in importance considerably. So we can answer more
questions, we can say-- let's see. What is the degree-- is there proteinuria? Is there protein in the urine? And we say, no, there isn't. I think we say, no, there isn't. 0. And that revises the
probability distribution. And then it says the next most
important thing is kidney size. And we say-- let's say
the kidney size is normal. So now all of a sudden
functional acute renal failure, which, by the way, is one
of these funny medical care categories that says
it doesn't work well, doesn't explain to why
it doesn't work well, but it's sort of
a generic thing. And sure enough. We can keep answering
questions about, are you producing
less than 50 ccs of urine, which is a tiny
amount, or somewhere between 50 and 400? Remember, this is for people
who are not producing enough. So normally you'd be over 400. So these are the only choices. So let's say it's moderate. And so you see the probability
distribution keeps changing. And what happened in
the original program is they had an
arbitrary threshold that said when the probability of one
of these causes of the disease reaches 95%, then we
switch to a different mode, where now we're actually
willing to contemplate doing the expensive tests and
doing the expensive treatments. And we build the
decision tree, as we saw in the case of the
gangrenous foot, that figures out which of those
is the optimal approach. So the idea here was
because building a decision tree with 27 potential questions
becomes enormously bushy, we're using a
heuristic that says information maximization
or entropy reduction is a reasonable way of
focusing in on what's wrong with this patient. And then once we focused
in pretty well, then we can begin to do
more detailed analysis on the remaining more
consequential and more costly tests that are available. Now, this program didn't
work terribly well, because the numbers
were badly estimated, and also because of the
utility model that they had for the decision analytic
part was particularly terrible. It didn't really reflect
anything in the real world. They had an incremental utility
model that said the patient either got better, or stayed
the same, or got worse. And obviously in that
order of utilities, but they didn't correspond
to how much better he got or how much worse he got. And so it wasn't
terribly useful. So nevertheless, in the 1990s,
I was teaching a tutorial at a Medical
Informatics conference, and there were a bunch of
doctors in the audience. And I showed them this program. And one of the doctors came
up afterwards and said, wow, it thinks just the way I do. And I said, really? I don't think so. But clearly, it
was doing something that corresponded
to the way that he thought about these cases. So I thought that
was a good thing. All right. Well, what happens
if we can't assume that there's just a single
disease underlying the person's problems? If there are
multiple diseases, we can build this kind
of bipartite model that says we have
a list of diseases and we have a list
of manifestations. And some subset of
the diseases can cause some subset
of the symptoms, of the manifestations. And so the manifestations
depend only on the diseases that are
present, not on each other. And therefore, we have
conditional independence. And this is a type of
Bayesian network, which can't be solved exactly
because of the computational complexity. So a program I'll
show you in a minute had 400 or 500 diseases and
thousands of manifestations. And the computational complexity
of exact solution techniques for these networks tends
to go exponentially with the number of undirected
cycles in the network. And of course, there are
plenty of undirected cycles in a network like that. So there was a program
developed originally in the early 1970s
called Dialog. And then they got sued, because
somebody owned that name. And then they
called it Internist, and they got sued because
somebody owned that name. And then they
called it QMR, which stands for Quick
Medical Reference, and nobody owned that name. So around 1982, this program
had about 500 diseases, which they estimated
represented about 70% to 75% of major diagnoses in
internal medicine, about 3,500 manifestations. And it took about 15 man
years of manual effort to sit there and read medical
textbooks and journal articles and look at records of
patients in their hospital. The effort was led by
a computer scientist at the University of Pittsburgh
and the chief of medicine at UPMC, the University of
Pittsburgh Medical Center, who was just a fanatic. And he got all the
medical school trainees to spend hours and hours
coming up with these databases. By 1997, they had
commercialized it through a company that had
bought the rights to it. And they had-- that
company had expanded it to about 750 diagnoses and
about 5,500 manifestations. So they made it
considerably larger. Details are-- I've tried to put
references on all the slides. So here's what data
in QMR looks like. For each diagnosis,
there is a list of associated
manifestations with evoking strengths and frequencies. So I'll explain
that in a minute. On average, there are about
75 manifestations per disease. And for each disease-- for each manifestation
in addition to the data you see here, there is
also an important measure that says how critical is it to
explain this particular symptom or sign or lab value
in the final diagnosis. So for example, if
you have a headache, that could be
incidental and it's not that important to explain it. If you're bleeding from your
gastrointestinal system, that's really
important to explain. And you wouldn't
expect a diagnosis of that patient that
doesn't explain to you why they have that symptom. And then here is an example
of alcoholic hepatitis. And the two numbers here are
a so-called evoking strength and a frequency. These are both on scales-- well, evoking strength
is on a scale of 0 to 5, and frequency is on
a scale of 1 to 5. And I'll show you what
those are supposed to mean. And so, for example,
what this says is that if you're
anorexic, that should not make you think about alcoholic
hepatitis as a disease. But you should expect
that if somebody has alcoholic hepatitis, they're
very likely to have anorexia. So that's the frequency number. This is the evoking
strength number. And you see that there
is a variety of those. So much of that many,
many years of effort went into coming
up with these lists and coming up with
those numbers. Here are the scales. So the evoking strength-- 0 means nonspecific. 5 means its pathognomonic. In other words, just
seeing the symptom is enough to convince
you that the patient must have this disease. Similarly, frequency 1
means it occurs rarely, and 5 means that it occurs
in essentially all cases with scaled values in between. And these are kind
of like odds ratios. And they add them kind of as
if they were log likelihood ratios. And so there's been
a big literature on trying to figure out exactly
what these numbers mean, because there's no formal
definition in terms of you count the number of this and
divide by the number of that, and that gives you
the right answer. These were sort of the
impressionistic kinds of numbers. So the logic in the system
was that you would come to it and give it a list of the
manifestations of a case. And to their credit, they went
after very complicated cases. So they took clinical
pathologic conference cases from The New England
Journal of Medicine. These are cases selected to be
difficult enough that doctors are willing to read these. And they're typically presented
at Grand Rounds at MGH by somebody who is often
stumped by the case. So it's an opportunity to watch
people reason interactively about these things. And so you evoke
the diagnoses that have a high evoking strength
from the giving manifestations. And then you do a
scoring calculation based on those numbers. The details of this
are probably all wrong, but that's the way
they went about it. And then you form a
differential around the highest scoring diagnosis. Now, this is actually
an interesting idea. It's a heuristic idea, but it's
one that worked pretty well. So suppose I have two diseases. D1 can cause
manifestations 1 through 4. And D2 can cause 3 through 6. So are these competing to
explain the same case or could they be complementary? Well, until we know what
symptoms the patient actually has, we don't know. But let's trace through this. So suppose I tell you that
the patient has manifestations 3 and 4. OK. Well, you would say,
there is no reason to think that the patient
may have both diseases, because either of them can
explain those manifestations, right? So you would consider
them to be competitors. What about if I add M1? So here, it's getting
a little dicier. Now you're more likely
to think that it's D1. But if it's D1, that could
explain all the manifestations, and D2 is still viewable
as a competitor. On the other hand,
if I also add M6, now neither disease can
explain all the manifestations. And so it's more likely,
somewhat more likely, that there may be
two diseases present. So what Internist had was
this interesting heuristic, which said that when you get
that complementary situation, you form a differential around
the top ranked hypothesis. In other words, you retain
all those diseases that compete with that hypothesis. And that defines
a subproblem that looks like the acute
renal failure problem, because now you have
one set of factors that you're trying to
explain by one disease. And you set aside all of
the other manifestations and all of the
other diseases that are potentially complementary. And you don't worry about
them for the moment. Just focus on this
cluster of things that are competitors
to explain some subset of the manifestations. And then there are different
questioning strategies. So depending on the scores
within these things, if one of those diseases
has a very high score and the others have
relatively low scores, you would choose a pursue
strategy that says, OK, I'm interested
in asking questions that will more
likely convince me of the correctness of
that leading hypothesis. So you look for the things
that it predicts strongly. If you have a very large
list in the differential, you might say, I'm
going to try to reduce the size of the differential
by looking for things that are likely in some of the
less likely hypotheses so that I can rule them out
if that thing is not present. So different strategies. And I'll come back to
that in a few minutes. So their test, of course,
based on their own evaluation was terrific. It did wonderfully well. The paper got published
in The New England Journal of Medicine, which was an
unbelievable breakthrough to have an AI program that
the editors of The New England Journal considered interesting. Now, unfortunately, it
didn't hold up very well. And so there was this
paper by Eta Berner and her colleagues in
1994 where they evaluated QMR and three other programs. DXplain is very similar
in structure to QMR. Iliad and Meditel
are Bayesian network, or almost naive
Bayesian types of models developed by other groups. And they looked for
results, which is coverage. So what fraction of the real
diagnoses in these 105 cases that they chose to test on could
any of these programs actually diagnose? So if the program didn't
know about a certain disease, then obviously it wasn't
going to get it right. And then they said, OK, of
the program's diagnoses, what fraction were considered
correct by the experts? What was the rank order
of that correct diagnosis among the list of diagnoses
that the program gave? The experts were asked to list
all the plausible diagnoses from these cases. What fraction of those showed
up in the program's top 20? And then did the program have
any value added by coming up with things that the experts
had not thought about, but that they
agreed when they saw them were reasonable
explanations for this case? So here are the results. And what you see is that the
diagnoses in these 105 test cases, 91% of them appeared
in the DXplain program, but, for example, only 73%
of them in the QMR program. So that means that
right off the bat it's missing about a quarter
of the possible cases. And then if you look
at correct diagnosis, you're seeing numbers like
0.69, 0.61, 0.71, et cetera. So these are-- it's like the
dog who sings, but badly, right? It's remarkable that
it can sing at all, but it's not something
you want to listen to. And then rank of the correct
diagnosis in the program is at like 12 or
10 or 13 or so on. So it is in the top 20, but it's
not at the top of the top 20. So the results were
a bit disappointing. And depending on where
you put the cut off, you get the proportion of cases
where a correct diagnosis is within the top end. And you see that at 20,
you're up at a little over 0.5 for most of these programs. And it gets better if you extend
the list to longer and longer. Of course, if you
extended the list to 100, then you reach 100%, but it
wouldn't be practically very useful. AUDIENCE: Why didn't
they somehow compare it to the human decision? PETER SZOLOVITS:
Well, so first of all, they assumed that their
experts were perfect. So they were the gold standard. So they were comparing
it to a human in a way. AUDIENCE: Yeah. PETER SZOLOVITS: OK. So the bottom line is that
although the sensitivity and specificity
were not impressive, the programs were
potentially useful, because they had interactive
displays of signs and symptoms associated with diseases. They could give you
the relative likelihood of various diagnoses. And they concluded
that they needed to study the effects of
whether a program like this actually helped a doctor
perform medicine better. So just here's an example. I did a reconstruction
of this program. This is the kind of
exploration you could say. So if you click on
angina pectoris, here are the findings that
are associated with it. So you can browse
through its database. You can type in an example
case, or select an example case. So this is one of those
clinical pathological conference cases, and then
the manifestations that are present and
absent, and then you can get an
interpretation that says, OK, this is our differential. And these are the
complementary hypotheses. And therefore these
are the manifestations that we set aside, whereas
these are the ones explained by that set of diseases. And so you could watch how the
program does its reasoning. Well, then a group at
Stanford came along when belief networks
or Bayesian networks were created, and
said, hey, why don't we treat this database as if
it were a Bayesian network and see if we can
evaluate things that way? So they had to fill
in a lot of details. They wound up using
the QMR database with a binary interpretation. So a disease was
present or absent. The manifestation was
present or absent. They used causal independence,
or a leaky noisy-OR, which I think you've
seen in other contexts. So this just says if there are
multiple independent causes of something, how likely is it
to happen depending on which of those is present or not. And there is a simplified way
of doing that calculation, which corresponds to sort
of causal independence and is computationally
reasonably fast to do. And then they also estimated
priors on the various diagnoses from national health statistics,
because the original data did not have prior data-- priors. They wound up not using
the evoking strengths, because they were doing a
pretty straight Bayesian model where all you need is the
priors and the conditionals. They took the frequency as a
kind of scaled conditional, and then built a
system based on that. And I'll just show
you the results. So they took a bunch of
Scientific American medicine cases and said, what are the
ranks assigned to the reference diagnoses of these 23 cases? And you see that like
in case number one, QMR ranked the correct
solution as number six, but their two methods,
TB and iterative TB ranked it as number one. And then these are attempts to
do a kind of ablation analysis to see how well the program
works if you take away various of its clever features. But what you see is that
it works reasonably well, except for a few cases. So case number 23, all variants
of the program did badly. And then they excused
themselves and said, well, there's actually
a generalization of the disease that was in the
Scientific American medicine conclusion, which the
programs did find, and so that would have been
number one across the board. So they can sort of make a
kind of handwavy argument that it really got
that one right. And so these were pretty good. And so this validated the
idea of using this model in that way. Now, today you can go out and
go to your favorite Google App store or Apple's app store
or anybody's app store and download tons and tons
and tons of symptom checkers. So I wanted to give you a demo
of one of these if it works. OK. So I was playing earlier
with having abdominal pain and headache. So let's start a new one. So type in how
you're feeling today. Should we have a cough, or runny
nose, abdominal pain, fever, sore throat, headache, back
pain, fatigue, diarrhea, or phlegm? Phlegm? Phlegm is the winner. Phlegm is like coughing
up crap in your throat. AUDIENCE: Oh, luckily,
they visualize it. PETER SZOLOVITS: Right. So tell me about your phlegm. When did it start? AUDIENCE: Last week. PETER SZOLOVITS: Last week? OK. I signed in as Paul,
because I didn't want to be associated
with any of this data. So was the phlegm bloody
or pus-like or watery or none of the above? AUDIENCE: None of the above. PETER SZOLOVITS:
None of the above. So what was it like? AUDIENCE: I don't know. Paul? PETER SZOLOVITS: Is it
any of these colors? AUDIENCE: Green. PETER SZOLOVITS: I think
I'll make it yellow. Next. Does it happen in the morning,
midday, evening, nighttime, or a specific time of year? AUDIENCE: Specific time of year. AUDIENCE: Yeah. Specific time of year. PETER SZOLOVITS:
Specific time of year. And does lying down or physical
activity make it worse? AUDIENCE: Well, it's
generally not worse. So that's physical activity. PETER SZOLOVITS:
Physical activity. How often is this a problem? I don't know. A couple times a week maybe. Did eating suspect food
trigger your phlegm? AUDIENCE: No. PETER SZOLOVITS: I don't know. I don't know what
a suspect food is. AUDIENCE: [INAUDIBLE] food. PETER SZOLOVITS: Yeah. This is going to
kill most of my time. AUDIENCE: Is it getting better? PETER SZOLOVITS:
Is it improving? Sure, it's improving. Can I think of another
related symptom? No. I'm comparing your case
to men aged 66 to 72. A number of similar
cases gets more refined. Do I have shortness of breath? No. That's good. All right. Do I have a runny nose? Yeah, sure. I have a runny nose. It's-- I don't know--
a watery, runny nose. AUDIENCE: Does it say you've
got to call [INAUDIBLE]?? PETER SZOLOVITS: Well,
I'm going to stop, because it will just take-- it takes too long to go through
this, but you get the idea. So what this is
doing is actually running an algorithm
that is a cousin of the acute renal failure
algorithm that I showed you. So it's trying to optimize the
questions that it's asking, and it's trying to come up
with a diagnostic conclusion. Now, in order not
to get in trouble with things like
the FDA, it winds up wimping out at the
end, and it says, if you're feeling really
bad, go see a doctor. But nevertheless, these kinds
of things are now becoming real, and they're getting
better because they're based on more and more data. Yeah. AUDIENCE: [INAUDIBLE] PETER SZOLOVITS: Well,
I can't get to the end, because we're only at 36%. [INTERPOSING VOICES] Yeah. Here. All right. Somebody-- AUDIENCE: Oh, I think
I need your finger. PETER SZOLOVITS: Oh. OK. Just don't drain
my bank account. So The British Medical
Journal did a test of a bunch of symptom checkers,
of 23 symptom checkers like this about four years ago. And they said, well, can it
on 45 standardized patient vignettes can it find at least
the right level of urgency to recommend whether you should
go to the emergency room, get other kinds of care, or
just take care of yourself. And then the goals were
that if the diagnosis is given by the program, it should
be in the top 20 of the list that it gives you. And if triage is
given, then it should be the right level of urgency. The correct diagnosis was
first in 34% of the cases. It was within the top
20 in 58% of the cases. And the correct triage
was 57% accurate. But notice it was more accurate
in the emergent cases, which is good, because those are the
ones where you really care. So we have-- OK. So based on what
he said about me, I have an upper respiratory
infection with 50% likelihood. And I can ask what to do next. Watch for symptoms like
sore throat and fever. Physicians often
perform a physical exam, explore other treatment
options, and recovery for most cases like this is
a matter of days to weeks. And I can go back and
say, I might have the flu, or I might have
allergic rhinitis. So that's actually reasonable. I don't know exactly
what you put in about me. AUDIENCE: What is
the less than 50? PETER SZOLOVITS: What is what? AUDIENCE: The less than 50. [INTERPOSING VOICES] AUDIENCE: Patients have to
be the same demographics. PETER SZOLOVITS: Yeah. I don't know what the less
than 50 is supposed to mean. AUDIENCE: It started
with 200,000 or so. PETER SZOLOVITS:
Oh, so this is based on a small number of patients. So what happens, of course,
is as you slice and dice a population, it gets
smaller and smaller. So that's what we're seeing. OK. Thank you. OK. So two more topics I'm
going to rush through. One is that-- as I mentioned in
one of the much earlier slides, every action has a cost. It at least takes time. And sometimes it induces
potentially bad things to happen to a patient. And so people began studying
a long time ago what does it mean to be rational
under resource constraints rather than rational just in
this Home economicus model. And so Eric Horvitz, who's
now a big cheese guy, he's head of Microsoft
Research, but used to be just a lowly graduate
student at Stanford when he started doing this work. He said, well,
utility comes not only from what happens
to the patient, but also from the
reasoning process from the computational
process itself. And so consider-- do
you guys watch MacGyver? This is way out of date. So if MacGyver is defusing
some bomb that's ticking down to zero and he runs out of
time, then his utilities take a very sharp
drop at that point. So that's what this work is
really about, saying, well, what can we do when we don't
have all the time in the world to do the computation as well
as having to try to maximize utility to the patient? And Daniel Kahneman, who won
the Nobel Prize a few years ago in economics for this notion
of bounded rationality that says that the way we
would like to be rational is not actually the
way we behave, and he wrote this popular
book that I really like called Thinking,
Fast and Slow that says that if you're trying to
figure out which house to buy, you have a lot of time to do it,
so you can deliberate and list all the advantages and
disadvantages and costs and so on of different houses and take
your time making a decision. If you see a car barreling
toward you as you are crossing in a crosswalk, you
don't stop and say, well, let me figure out the pluses and
minuses of moving to the left or moving to the right, because
by the time you figure it out, you're dead. And so he claims that
human beings have evolved in a way where we have a kind of
instinctual very fast response, and that the deliberative
process is only invoked relatively rarely. Now, he bemoans this
fact, because he claims that people make too
many decisions that they ought to be deliberative about based
on these sort of gut instincts. For example, our
current president. But never mind. So what Eric and his
colleagues were doing was trying really to look at
how this kind of meta level reasoning about how
much reasoning and what kind of reasoning is worth doing
plays into the decision making process. So the expected
value of computation as a fundamental
component of reflection about alternative
inference strategies. So for example, I
mentioned that QMR had these alternative
questioning methods depending on the length
of the differential that it was working on. So that's an example of a kind
of meta level reasoning that says that it may be
more effective to do one kind of question asking
strategy than another. The degree of
refinement, people talk about things like
just-in-time algorithms, where if you run out of time
to think more deliberately, you can just take the best
answer that's available to you now. And so taking the
value of information, the value of computation, and
the value of experimentation into account in doing
this meta level reasoning is important to come up with
the most effective strategies. So he gives an example
of a time pressure decision problem where you have
a patient, a 75-year-old woman in the ICU, and she develops
sudden breathing difficulties. So what do you do? Well, it's a challenge, right? You could be very
deliberative, but the problem is that she may die because
she's not breathing well, or you could
impulsively say, well, let's put her on a
mechanical ventilator, because we know that
that will prevent her from dying in the
short term, but that may be the wrong decision,
because that has bad side effects. She may get an infection, get
pneumonia, and die that way. And you certainly don't want
to subject her to that risk if she didn't need
to take that risk. So they designed an
architecture that says, well, this is the decision
that you're trying to make, which they're modeling
by an influence diagram. So this is a Bayesian network
with the addition of decision nodes and value nodes. But you use Bayesian network
techniques to calculate optimal decisions here. And then this is kind of
the background knowledge of what we understand
about the relationships among different things in
the intensive care unit. And this is a representation
of the meta reasoning that says, which utility
model should we use? Which reasoning
technique should we use? And so on. And they built an
architecture that integrates these various approaches. And then in my last
2 minutes, I just want to tell you about
an interesting-- this is a modern view,
not historical. So this was a paper presented at
the last NeurIPS meeting, which said the kinds of problems
that we've been talking about, like the acute renal
failure problem or like any of these others,
we can reformulate this as a reinforcement
learning problem. So the idea is that if you
treat all activities, including putting somebody on a
ventilator or concluding a diagnostic conclusion or
asking a question or any of the other things
that we've contemplated, if you treat those
all in a uniform way and say these are
actions, we then model the universe as a
Markov decision process, where every time that you
take one of these actions, it changes the state
of the patient, or the state of our
knowledge about the patient. And then you do
reinforcement learning to figure out what
is the optimal policy to apply under all
possible states in order to maximize
the expected outcome. So that's exactly the
approach that they're taking. The state space is the set of
positive and negative findings. The action space is
to ask about a finding or conclude a diagnosis. The reward is the correct or
incorrect single diagnosis. So once you reach a
diagnosis, the process stops, and you get your reward. It's finite horizon
because they impose a limit on the number of questions. If you don't get an
answer by then, you lose. You get a minus one reward. There is a discount factor so
that the further away a reward is, the less value it has to you
at any point, which encourages shorter question sequences. And they use a pretty standard Q
learning framework, or at least a modern Q learning framework
using a double deep neural network strategy. And then there are two
pieces of magic sauce that make this work better. And one of them
is that they want to encourage asking
questions that are likely to have
positive answers rather than negative answers. And the reason is
because in their world, there are hundreds and
hundreds of questions. And of course,
most patients don't have most of those findings. And so you don't want to ask
a whole bunch of questions to which the answer is no, no,
no, no, no, no, no, no, no, because that doesn't give
you very much guidance. You want to ask questions
where the answer is yes, because that helps you clue
in on what's really going on. So they actually
have a nice proof that they do this thing
they call reward shaping, which basically adds
some incremental reward for asking questions that
will have a positive answer. But they can prove that
an optimal policy learned from that reward
function is also optimal for the reward function
that would not include it. So that's kind of cool. And then the other
thing they do is to try to identify a
reduced space of findings by what they call
feature rebuilding. And this is essentially a
dimension reduction technique where they're co-training. In this dual network
architecture, they're co-training
the policy model. It's, of course, the
neural network model, this being that 2010s. And so they're generating a
sequence, a deep layered set of neural networks that
generate an output, which is the m questions and the n
conclusions that can be made. And I think there's
a soft max over these to come up with the right policy
for any particular situation. But at the same time, they
co-train it in order to predict a number of-- all of the manifestations from
what they've observed before. So it's using-- it's learning
a probabilistic model that says if you've answered
the following questions in the following ways, here
are the likely answers that you would give to the
remaining manifestations. And the reason they
can do that, of course, is because they really
are not independent. They're very often co-varying. And so they learn
that covariance, and therefore can predict which
answers are going to get yes answers, which questions are
going to get yes answers. And therefore, they can bias
the learning toward doing that. So last slide. So this system is called REFUEL. It's been tested
on a simulated data set of 650 diseases
and 375 symptoms. And what they show is that the
red line is their algorithm. The yellow line uses only
this reward reshaping. And the blue line is just
a straight reinforcement learning approach. And you can see
that they're doing much better after many
fewer epochs of training in doing this. Now, take this with a grain of
salt. This is all fake data. So they didn't have real
data sets to test this on. They got statistics on
what diseases are common and what symptoms are
common in those diseases. And then they had
a generative model that generated this fake data. And then they learned from
that generative model. So of course it would be
really important to redo the study with real data,
but they've not done that. This was just published
a few months ago. So that's sort of where we
are at the moment in diagnosis and in differential diagnosis. And I wanted to
start by introducing these ideas in a kind
of historical framework. But it means that there are a
tremendous number of papers, as you can imagine, that have
been written since the 1990s and '80s that I was
showing you that are essentially elaborations
on the same themes. And it's only in the
past decade of the advent of these neural network
models that people have changed strategy,
so that instead of learning explicit
probabilities, for example, like you do in a Bayesian
network, you just say, well, this is simply
a prediction task. And so we'll predict
the way we predict everything else with neural
network models, which is we build a CNN, or an RNN,
or some combination of things, or some attention
model, or something. And we throw that at it. And it does typically
a slightly better job than any of the previous
learning methods that we've used
typically, but not always. OK. Peace.
