ROC Curves

ok so now we're going to talk about receiver operator curves and what these do is they tell you how good a test is that is how good a test can distinguish between two things such as which patients have disease and which patients don't and better tests can definitively say that patients do or don't have disease whereas worst tests have a little bit more difficult and distinguish between those two but before we get there let's go back to our sensitivity and specificity things that were talking about in the previous video you'll remember these histograms we do these distributions we drew from the previous video and so this blue one represents patients who don't have disease and this green one represents the number of patients who do have disease and this down here is the value of a test or just saying for this test goes between zero and a hundred so most people who do have disease score about a let's age sixty seven or something on the test and those who don't most of them on average will score about a thirty on this test and so lift pick a place where we're gonna set a cut-off above which we're gonna call the test positive and below which we're gonna call the test negative so this area underneath the patients who have disease who test positive we call the true positives and of the patients who don't have the disease who were above the positive line so who tested positive this area would be called are false positives now on the negative side of the test the area under this distribution here this this proportion of the patients who don't have disease who test negative we would call our true negatives and the patients who who do have the disease who fell in the negative test area we would call our false negatives and you'll remember that sensitivity and specificity are measures of how often this test performed correctly so the proportion the patients who do have disease that were identified correctly that is true positives his sensitivity and similarly the proportion of patients who don't have disease that were correctly identified that is they had a negative test we call this our specificity and you'll remember from the video on the trade-offs between sensitivity and specificity if we move this cutoff line this way to increase sensitivity then what we have done by increasing sensitivity is we have decreased the specificity now we could do just the opposite and instead we can move this cutoff line this way and what we can see here happens as we increase the specificity but we've decreased the sensitivity so as we move this cutoff point at which we declare our test as either positive or negative on one side we are going to change our sensitivity or specificity now we could graph these with sensitivity and specificity being on these axes like this and we know that they they do have at some sort of inverse relationship as one goes up the other goes down you look at something like this or you could see that as the specify specificity is lower the sensitivity is high but as we increase the specificity the sensitivity goes low so this is a graph of that but traditionally this is not how we graph it instead of putting specificity on here we put 1 minus specificity which then creates a graph that looks like this so if if specificity is dropping and specificity is a number between 0 and 1 then 1 minus specificity is gonna go up what specificity goes down and when specificity goes down 1 minus specificity goes up so remember that inverse relationship we have now it's not inverse now they rise and fall together and they take some some kind of shape like this and so this is what we call a receiver operating curve to better understand this let's look at four different tests so here we have four different tests let's say to measure whether someone has a particular disease or not and these are the way the distributions are for each one the tests are the green ones a patient has no disease in this case and the blue one is that they do have disease and what you'll notice here is that this one has a very good distinguishing properties if you put the the divider line right here in the middle you could easily tell the difference between which patients have disease and which ones don't now this one here has a little bit of overlap and so this overlap makes it tricky and so this one is not as good as that but it's still pretty good it's only these small overlap areas where you could be wrong so you'd have a small portion here but you'd have a false negative and a small portion here well you'd have a false positive now this test over here has much more overlap and so it is a it's a it's a little bit harder to distinguish between positives and negatives and what you'll see here is there's a greater degree greater area of false negatives in a greater area of false positive and now this extreme example here look up the curves completely overlap there is no way to distinguish using this test whether someone has or doesn't have the disease because for someone with the disease of 7 1908 curves for each one of these so for the test that performs very well you could see that this this curve is very much fills up this entire space here can we call this space the ROC space and this particular ROC curve takes up about 90% of that or 0.9 now let's look at this test which performs ok you can see here that it doesn't have as dramatic a shape of this it doesn't fill up the entire space as much but it fills up a good deal of it I mean yes there's some space up here that's left unfilled but maybe the area under this curve takes up about let's say 80% of the space so the area under the curve is at 0.8 assuming that we're saying that the whole space the whole ro spec ROC space is 1 the area under the curve is 0.8 or you could say that the area of the curve fills in 80% now look at this test this test is maybe at best fair maybe bordering and poor you know it doesn't fill up a lot of the space let's say that it fills up 70% now as for this test which really we said we said doesn't distinguish anything at all you look at its ROC curve and it's basically a flat line it's a flat line that basically bisects the space in half so it takes up about 50% and so this number here this area under that curve is a measure of how good the distinguishing property of a test has of this one this particular test here has an area under the curve of 0.9 that's a pretty darn good test but this one here at the bottom that has an area under the curve of 0.5 that's a crappy test that doesn't do anything and we could see here that this one's a little bit better and this one's better still of course this one's up is the best and here are some generally accepted interpretations of the area under the curve values of 0.9 to 1 are considered excellent you could see at 0.7 2.8 sphere around point 5 is a failure and in between you can read the distinguishing properties of a test ability to distinguish between positive and negative based on its area under the curve so you might be asking yourself why are we using 1 minus specificity that seems like a weird choice well let me explain that you recall that this area is the true positive rate and the proportion of this area is also what we call the sensitivity and then on the other distribution over here but this area is what we call the specificity and the true negative rate so what is one - specificity where does that come in if the area under this curve would be 1 or you know that if we call that 1 then we said that specificity is of this portion so 1 minus that is whatever is left so that's this area over here or the false positive area so the sensitivity could be called really the true positive rate and then the 1-7 specificity or the false positive here this area could be called the false positive rate so now we're really just looking at positives here and so we could tell that as we move this cutoff line this way we're gonna be cutting into the true positive rate as well as the false positive rate and as we move it this way we're going to be increasing the true positive rate and the false positive rate now the reason that we're looking only at the positives false and true is also the reason where the name receiver operator curve comes from and this comes from world war ii and radar signal detection theory now the people who would operate the radar receivers were called receiver operators obviously and what their job was is they would see a blip on the radar screen and they needed to distinguish whether this blimp was an enemy or was this some sort of noise in other words was this a true positive or a false positive so these guys who were operating the radar receivers they had to tell whether that blip was a real or not and so you could measure that you can measure it for a given receiver operator you could measure their true positive rate and their false positive rate and then you could graph these things their true positive rate against their false positive rate and get a characteristic of this guy here a characteristic of the receiver operator and then these curves that you would get those would be your receiver operator characteristic curve hence ROC curves and so that's where we get this from signal detection theory and then it wasn't until the 1970s that we started applying this signal detection Theory ROC curve to medicine and interpreting medical test results so there you have it that is ROC curves and how you use them in order to judge the quality of a test and you can use it to compare two tests as well by comparing their a you sees thanks I'll talk to you later

Channel: Rahul Patwari

Views: 170,441

Rating: 4.9598618 out of 5

Keywords: EBM, DoodlecastPro

Id: 21Igj5Pr6u4

Length: 11min 46sec (706 seconds)

Published: Thu Jun 20 2013