23 Generalizability

[Music] all right it's now time to talk about a concept called generalizability you may remember from many many lectures ago when i posed this issue of what is the question right what are you studying are you studying the right thing well generalizability is somewhat related in the case of generalizability we're assuming you've got the right question and you're collecting the data in the right way but the people you're asking the population might not be the right population so let me walk you through some of the goals today we're going to discuss the concept of generalizability and how it relates to sampling we're going to explain how generalizability problems come in several flavors and remind you that just because the study does not generalize does not mean the findings aren't more broadly true so to give a brief example of this let's say that i want to figure out a good a good book to read this summer i want to know you know something that's going to be really entertaining that a lot of people like so i'm going to go out on the street and i'm going to ask people what they've been reading we're here on the lovely yale campus it's a beautiful spring day we're trying to figure out what the hottest book of the summer is going to be we are taking a sample of the students around here to find out what those exciting page turning beach reads are going to be i'm here with kendall kendall what book are you reading right now native son by richard wright totally so what book are you reading right now i i read nerd stuff like uh reinforcement learning and introduction is what i'm reading at the moment that that's a classic what book are you reading right now uh montaigne's essays that's a good one i am reading on deep history and the human brain by who is this by oh daniel smile dawn by octavia butler that i've actually heard of that book i think that's the first time this has happened what book are you reading right now um i'm not reading a book right now but i just finished homegoing by yan agassi uh i'm reading a book called brain on fire by susannah kalin right now i don't know what that is what book are you reading right now so i'm actually currently reading how democracies die very depressing um i started reading joseph campos the hero with thousand faces classic i'm reading difficult women by roxanne gay a collection of short stories oh that sounds like a lot of fun what book are you reading right now chronicle of a death foretold by gabriel garcia marquez good old ggm yeah okay so what was the problem there um we asked a good question right what are you reading i wanted to to get some book recommendations for my beach read this summer but i asked the wrong people my sample was not representative of the population i was interested in the population i was interested in was like regular people in the united states but we ended up walking around campus here and and talking to a bunch of college students who are reading things that really don't sound that interesting so good question data accurately collected but not really generalizable it doesn't uh the the sample that we obtained is not representative of the sample i'm interested in so this is a generalizability problem the results i got are accurate but not terribly useful so let's talk about some some broad generalizability issues in the context of a few different studies so the first one the study population is too specific uh when you need a more general population so here's this study effects of oral versus transdermal estrogen therapy on sexual function in early post menopause so this was a study asking if you could give hormone replacement therapy like estrogen just after menopause to increase sexual function in women so an interesting and important question um you might be uh you might think that this is something that would have have broad applicability to a large number of people in the united states but if you dig in and you look at the inclusion and exclusion criteria women were only included if they were 42 to 58 years old they were six months to 36 months from their last menstrual period their blood level of follicle stimulating hormone was at least 35 they hadn't had a hysterectomy they hadn't had an abnormal mammogram they had no untreated psychiatric illness no history of cardiovascular disease no smoking no obesity and so on and so on and so on this highly selected population so they're asking a really interesting question and one that's important but they're asking it of this really specific group and i challenge you to find a woman in your own life that would meet all of the inclusion and exclusion criteria for this study so how do we interpret that data it's accurate and so far as a woman matches up with these characteristics but if there are so few women that match these characteristics what are we really doing with that this study does not generalize well but it's important to know that just because a study doesn't generalize well doesn't mean you can always throw it in the waste paper basket a lot of times studies start out with very specific populations the initial studies in an area for example and then they get broader and broader and broader over time so a great example of this is a study published all the way back in 1954 we're in mad men era now okay looking at the mortality of doctors in relation to their smoking habits this is a very specific population as a doctor myself i can tell you that we're very strange and unusual people nevertheless what they found is that the more doctors smoked and this sort of dose response mechanism here the more doctors smoke the higher the incidence of lung cancer was this was one of the the first studies in uh in america that uh suggested that there might actually be a link between smoking and lung cancer now if i was looking at this back then i might say oh those are just doctors this doesn't generalize to the general population this is only true for doctors for some reason but if you think about it you know why what what makes doctors that different aside from the fact that we're just strange people so important study turned out to be true generally even though this study itself was quite specific i also want to point out one of the authors of this study was bradford hill you remember in our causality lecture we talked about the bradford hill criteria for causality so this guy was uh important turns out okay what if we study the wrong population entirely not just a specific subset of the population we're interested in but just the wrong people all together so this happens sometimes too so here's an article called stimulation of the prefrontal cortex reduces intentions to commit aggression so you read that okay what you need to know the prefrontal cortex is part of the brain and aggression is aggression okay so just from the title alone this is a study looking at whether you know you can sort of stimulate this part of the brain and i'm not sure whether it's with electricity or magnetism or something like stimulate this part of the brain and it's going to reduce aggressive impulses so that's kind of cool it's like a little scary i guess maybe a little creepy what population do you want to test this in well the population of interest is probably people with aggressive impulses you might even argue it should be maybe like violent offenders people in prison for violent crimes or something like that that's the interesting population now what they studied what you can see here is 81 healthy subjects and indeed they showed that this stimulation it reduced to their intent to commit physical assault and intent to commit sexual assault which of course leads you to that question which is like who are these 81 healthy volunteers who at baseline are like well i'd like to commit a little sexual assault so i'm not entirely sure who these people are but it's not the right population okay so this is an interesting study it's suggestive but if you were reading this you would say okay that's great but i'm not really interested in giving electric shocks to healthy people to help them control their latent urges to physically assault someone really if i'm going to do this kind of strange thing at all i'm going to do it to someone that has a documented history of having these urges wrong population the most extreme example of wrong population happens quite often in the news media so here's a study called eradication of spontaneous malignancy by local immunotherapy that sounds good spontaneous malignancy that's cancer local immunotherapy that sounds like they're injecting someone near the something near the cancer to sort of boost the immune system and have it fight the cancer really cool studies like this uh often show up in the news media as like the latest cure for cancer which you should always be skeptical about um and in fact the conclusion here was that a cancer vaccine as it was described eliminated 97 of distant metastases so really exciting data in mice okay that's the biggest generalizability problem of all mice aren't humans the population you want to cure cancer in is humans unless unless you're maybe a veterinarian for mice so be aware we want to study humans whenever possible if we're going to make conclusions about how best to treat humans so that is generalizability your take home points today whether study generalizes is somewhat subjective you want to ask who is the population you're interested in and who is the population they studied remember just because a study doesn't generalize doesn't mean you should completely throw it away it doesn't mean it won't generalize in the future and often early studies start in specific populations you start in healthy controls before you move on to the sicker population and maybe the most important thing mice aren't human so always look very skeptically at studies that show dramatic results in mice they rarely generalize to the human population thanks i'll see you next time

Channel: YaleCourses

Views: 831

Rating: 5 out of 5

Keywords:

Id: 5tkxba8ZDE4

Length: 10min 9sec (609 seconds)

Published: Fri Jul 31 2020