Teaching Bayesian and frequentist methods side by side

it's my pleasure to introduce dr. John crush key who is our morning speaker this morning he's a Provost professor in the department of psychological and brain sciences at Indiana University his PhD is in cognitive psychology from UC Berkeley he's well known for advocating the use of Bayesian methods of data analysis in the psychological sciences and he's written a very well-received and well-known textbook in Bayesian data analysis and especially with the emphasis on evaluating evidence that's the theme of this conference I thought it would be great to have the Bayesian perspective presented and I think it's great to have someone who's not in a statistics or math department to give us that perspective as well so I was delighted when John accepted the invitation to to present and I'm looking forward to his presentation so please welcome John kreski well thank you all very much I'm deeply appreciate the invitation to be here and I'm glad to see all these people first thing on a Saturday morning you could be sleeping in but you're here to hear about Bayes I'm curious to know because this is the first time I've been to u.s. cots meeting I'm curious to know how many of you have actually tried to teach some Bayesian in your courses a subset but a reasonable number interesting how many of you really don't know much about Bayesian but are here anyway ok great ok thank you so in the beginning of the talk what I'd like to do is in the process tell you a little bit about what got me so interested in Bayesian approaches and so I'm going to start with just this question why teach Bayesian and I'm going to show you a series of premises and the conclusion that it's worthwhile to teach Bayesian and then I'll unpack each of those premises a little bit more so first Bayesian is valuable in real applications and it's sought sought after by researchers second Bayesian is easy to teach and in my opinion easier than frequentist and Bayesian clarifies frequentist ideas especially if you set them side-by-side and you see the Bayesian information it really clarifies oh that's what the frequentist is telling me it's different than this Bayesian stuff and these together lead me to this conclusion that it's at least reasonable that Bayesian should be in the curriculum so now let me unpack these premises a bit so Bayesian is valuable in real applications for me this has been the clincher what I do what I really do is I'm not primarily a Bayesian educator I'm primarily a researcher and I have just found over and over and over that the Bayesian analyses let me do what I need to do when I'm analyzing complex datasets so just some examples from personal experience Wednesday just as I was finalizing this I haven't changed anything in this talk since Wednesday I got an email from some colleagues saying you know we're doing this study in mate choice and we have these different strategies people can use in mate choice oh this is mate choice in humans by the way we have these strategies people can use and we look at the proportion of strategies as a function of age and we see this kind of u-shaped curve in the proportion as a function of age and what we're trying to do what our reviewers have told us we need to do is demonstrate that the dip in the U is actually between the extremes it really does go down and then go up so I thought oh well there are lots of ways you could try to demonstrate that but I'll you know here are several from this behaves you know approach like one thing you can do is in the quadratic you're going to fit a quadratic curve find where the nadir of parabola is and look at the credible interval on that nadir and show it's clearly between the two endpoints of your X values or better yet if you're really going to get in there show that the posterior predictive interval on the what you're going to predict for the proportions on the one end on the both ends those are clearly above the posterior predictive interval for what's at the nadir okay and I'll get back to you two weeks after I go to this conference some work that I've published recently in the journal law and human behavior with a amazing student named Brad Celestine we've looked at civilian perceptions to police use of force what we do is we present people with mini vignettes the civilian did this action and then the police officer did this other reaction you have to rate on a slider how appropriate or how acceptable is this police reaction to that civilian action and it's a slide or it assumes you were 200 now our primary interest is not just ratings of appropriateness we're trying to estimate the latent severity of these actions just how severe is this civilian action how severe is that police reaction and so we're simultaneously estimating regression coefficients and item latent scale values but not only that as I've suggested at the end of this last line these are end sensor data that is the sliders very often will be pushed all the way to one end or all the way to the other in these ratings so all we know is that the respondent wanted to push the slider somewhere over there or somewhere over here and so we have this nice pile of responses on one end a smooth distribution in the middle and then a pile of responses at the other end that's not very normal but in a base in Bayesian software it's it's completely straightforward to treat those as sensor data we know the data are somewhere over there somewhere over here so many examples where I need models for unbalanced designs designs with empty cells heterogeneous variances outliers so I need heavy tail distributions I can put in all sorts of customized hierarchical structures once ready I've done just recently collaborated with has to do with the Neo philic behavior of raccoons neo philic behavior sorry these polysyllabic jargon it just means raccoons are curious critters right they get into stuff and so they had these um boxes that they put out into the woodlands tied to trees and then they have motion-sensitive cameras so when any critter comes within the radius of the motion sensor of the camera the camera starts rolling and you know you see what animals their question does it contact this box is it inquisitive and so they look at the proportion of contact for all these different species that happen to come come up in different seasons and as a function of different baiting that is is it baited or not and so on now in this situation there are lots of empty cells because sometimes you don't see a skunk at an unboxing winter but you'll see deer and so it's a very unbalanced in empty cell design and if you try to do that in traditional approaches it would just crash but in a bayesian world it's no problem and this goes on and on and on I can other situations where you look at your data and you realize oh I need this kind of trend with this sort of kind of decay function for different items and it's completely straightforward to put any model you need that actually describes your data in useful ways in a Bayesian framework so as I said all of these are straightforward and Bayesian and for me that's what really compels my continued use and fascination with Bayesian methods but you don't have to take my word for it Bayesian is sought by researchers and again I'll just try to argue for this point through personal experience over recent years I've done 45 workshops with audiences or working professionals and graduate students from academia in every department across the Sun from economics and education and climate science and biological sciences and cognitive sciences and you know keep going through the alphabet they're all there business and industry representatives show up at these workshops people studying retail sales charitable giving they're trying to model they're trying to model the characteristics of who's going to give and who not food production people showing up they're interested for example in how to make better bacteria for cheese so that the cheese is more stable and so on in government I've given workshops at the FAA where they're studying human factors and at the FDA where among other things they're starting survival analysis and they're all interested in these Bayesian methods presumably because it's really useful laziness easy to teach and I would say easier than frequentist in fact I was initially driven to Bayesian by aversion to teaching frequentist for years and years I've taught frequentist stats and I try so hard to explain the p-value and the sampling distributions and then we'd get to the section in ANOVA where I have to justify Corrections for multiple tests you know and then I have to talk about things like the Tukey HSD and have you done the Tukey HSD show of hands HST to Kia what is H s D stand for honestly significantly different okay you know if honest you know be honest do you really want to do all this comparisons come on you know you do you know you really want to do this don't you okay and I've read so many manuscripts but his manuscripts submitted to scientific journals in which in which the authors go through these kind of gymnastic contortions to try to justify why they only want to do these three tests and curiously the p-values of those three are locked point oh four you know and it's like it's obvious really you should be looking at a lot of other tests but no no they're gonna rationalize wait no no just these three or a lot take what's clearly one experiment design and they'll split it into two experiments this part of the design is experiment one this part of the design that's experiment two and therefore when we're doing our Corrections for multiple tests we only have to correct over these Corrections that are for these tests that are an experiment one and not Factory and all these other tests that are in experiment 2 so it's having to teach things like that that just bothered me and also that you know you know so well this kind of inverse logic of the of the p-value and I've taught frequentist in Bayesian courses separately for years so let me tell you a tale of two courses these are courses I teach at the introductory graduate student level first year graduate students from across the university okay both courses begin the same way I cover fundamental concepts of data and models you're going to use a model to describe your data both courses conclude the last half or more is the generalized linear model so I cover you know the various dependent variable types under metric dependent variable as we go through multiple linear regression and all the various ANOVA style models we look at logistic regression and then under nominal nominal dependent variables we look at of course multinomial regression or conditional logistic regression and would go on to ordered probit models and then for count data we look at poisson or negative binomial models and so on so that's the same in both courses what's different well all the framework you're going to use to consider all those models so of course in the frequentist class so I have to cover sampling distributions and p-values and confidence intervals and the testing intentions and stopping intentions why did your n come out to be 122 did you really plan to have a hundred and twenty-two in advance and in the bayesian world on the Bayesian class I teach the Bayesian reallocation of credibility across parameter values I'll tell you more about that soon and then I cover the Markov chain Monte Carlo methods for representing those distributions and just from personal experience the right column is so much smoother and students just get it the left column I've worked on and I think the students actually get it but even after they get it they look at me like really that's how science works okay all right and then this other premise Bayesian clarifies frequentist ideas this has actually come up in conversation spontaneously and I was so glad to have this slide in the talk because it echoed exactly what I've heard from other people how often have you told students this is a quote from an instructor the p-value is not the probability of the null hypothesis and then the student says oh well then what is and you have to provide a Bayesian answer there's a Bayesian approach to putting actual probabilities on null and alternative hypotheses and so you can point to the Bayesian answer and then the student will get this AHA by pointing out the distinction they say oh how the p-value is about a distribution of imaginary data and it really focuses them on that distinction or the instructor can say the 95 percent confidence interval is not the range of most possible values of most probable values and the student goes really well well what is and you provide a Bayesian answer a Bayesian posterior distribution that shows you the distribution of most probable values and the student goes oh and then through contrast realizes oh the confidence interval is about not rejected values so putting them side by so it really helps clarify the information so just to recap this series of premises and the suggested implication that Bayesian should be in the curriculum so then we get to the question how how can we include Bayesian in our courses small font sorry but there's a lot of little options which get rejected option number one let's replace the standalone frequent discourse right let's get the frequent discourse out of there shook it well no the frequentist methods are entrenched and they really do address the issue of error rates they really do address this issue of error rates if you want to control error you're you got to go with the frequentist route okay well what about adding an optional stand-alone bayesian course well no students won't take it and instructors won't prep it okay how many of you would just go out there and prep a brand new course and Bayesian even the enthusiasts find that very daunting and I say students won't take it if it's optional and they're you know it's all their schedules already full they're not going to take it now I have done this okay and I kever a year I get 40 students in that room from all across campus and so in that sense it works and I'm you know I'm a real enthusiast so I've prepped the course but if you look at that course versus all the frequencies courses taken my course is just a tiny sliver of what students are doing so just having an optional standalone course won't really do the job uh-huh but our required standalone Bayesian course I think that would be terrific but no that's it's not going to be required at anytime soon so that's unrealistic well then any separate course should you just have a separate Bayesian course separate from frequentist and No maybe that's just not the best way to go maybe the juxtaposition inside one course actually can help can actually help clarify both so I'm gonna take that and run with it a bit let's inject Bazin and frequentist into existing courses so how how to inject Bazin in frequentist into existing courses well one option is what we need is a module a module that is self-contained so it minimizes teacher prep yay all in favor of minimizing teacher prep hey a module it has a complete tutorial explaining Bazin and frequent analyses so the instructor doesn't have to go do a lot of background self education has interactive software that's browser-based you don't to install software on your computer that's some interactive exercises and it has clear learning objectives and assessment be nice to have that and when Alan invited me to give this talk I was thinking you know something like this has been on my mind for a long time so I'm going to take the plunge I'm going to take today as my deadline for getting something like this at least unveiled so I've been working really hard well on a new shiny app to at least get this started and here I say it's accomplished that might be a little bit of an over selling it's it's there it's ready for revision so what I'd like to do is give you an overview of the shiney app it's available on my website I think the hardest thing about learning this particular branch of Bayesian statistics is learning how to spell that guy's last name it's Chris yeah and you try to type it into Google and it gives you like former leaders of the Soviet Union and anyway it's on my web page you can get you it there so here's the screen layout of the app the basic structure of the screen layout notice on the left side there's where you specify data and then there are two columns there's a frequentist column in a bayesian column okay so side by side yay but then there are also two rows in the app there's a row for an estimate with uncertainty and then there's a row for a null hypothesis chest so then you might be asking hmm why do you have those two rows in there why teach hypothesis testing and estimation with uncertainty well they have different goals hypothesis testing seeks a decision regarding a specific hypothesis that's what it's about can you decide about this hypothesis but estimation with uncertainty seeks a description of data with some degree of precision quantified precision on that description different goals different information hypothesis testing is traditional and ubiquitous so we can't just kind of pretend it's not there but of course it does have issues the cognitive trap of black and white thinking that was described yesterday as bright line thinking and estimation with uncertainty is encouraged by best practices as we've heard a lot now from yesterday's speakers the the a sa statement and the American statistician special issue going beyond point oh five so that means that both of these approaches are perhaps important but why teach them well again the juxtaposition can clarify both and estimation with uncertainty is again in my opinion more intuitive and easier to teach indeed the apps default view has no hypothesis tests yay so the app you know you pull up the app you invoke the app it's got the two columns what shows up is the estimation row there's nothing in the hypothesis testing row and to me that alone was a little mini revelation as I was setting up the app by default I had everything there and then I realized oh wait a minute to do a hypothesis test you have to specify what hypotheses you're testing and by default you're not you're not testing anything oh why default I should leave that that row blank and then it was just that's like it's weight off my shoulders okay nevertheless though in the app you have results from all four of those cells simultaneously displayed and so in particular you can see side by side what does the frequentist analysis provide and what does the Bayesian analysis provide either in the row for estimation or in the row for hypothesis testing and of course you can look at the comparison between the two rows Oh what is the information that estimation provides what is the information that hypothesis testing provides and just having them all available at once it's a busy array but it gives it the opportunity to really clearly see the distinction between the information provided okay in the app there are lots of controls lots of sliders there are controls of course in the data you've got to specify where your data are there are sliders that specify it like in the lower left here what are your null hypotheses what's the value of you know value for MU of the mean for example over in the Bayesian column what are your assumptions about the prior distributions so are all sorts of sliders for setting up your assumptions to do the analysis now what's really neat then is you can observe all the influences simultaneously so you change your data you can see that what that does to all the results simultaneously you change your null hypothesis tests you can see what that does to all the output simultaneously you know you change your priors in the Bayesian what does that do to all this information but what's also neat about this kind of setup is the sliders point you to exactly the information you need from the real world when you're trying to do an analysis okay so you set up you know there's some story problem some exercise or even some real-world analysis where the story problem or the real world provides this vast array of information and then you say to the student okay go analyze it go what they what they need to do is extract from the world the relevant bits of information to set up the analysis and the sliders basically focus their attention on exactly what's needed oh ok if I'm gonna do this frequentist analysis oh yeah I've got to think about what is the null value and and so on what are my what are my stopping intentions and all that and the structure of the app and this applies not only to this app but to all the many apps that many of you have created it sets up a really natural structure to think about the intended learning outcomes so one category of learning outcomes is for as I say here's for in-depth understanding of frequentist and Bayesian analyses and their interaction or sorry inter relation and it's achieved by interactively manipulating the sliders and watching what happens and figuring out why and there's a finite although it's a large number there's a finite number of these arrows from sliders to what happens in the analysis and each of those arrows is a potential assessment question as well and then the other category of learning outcomes is the connection of the sliders to the outside world if and as it says here you should have an in-depth understanding of how to apply to real situations and that's achieved interactively by translating situation to a setting of the sliders and the app makes explicit what needs to be found so then how do you assess those learning outcomes well you should be able to predict the qualitative effect of every slider and button on the reason on the results in every cell of the table and explain why and how do you assess that other category of outcome we should be able to set the sliders appropriately to reflect real-world scenarios and explain why so that's your overview in my remaining time I'd like to give you some more details there is with this app an extensive online tutorial it's big and growing it has a dynamic table of contents on the left so as you read through the tutorial the table of contents shows you where you are you can click in the table of contents to go directly to any point in the tutorial the tutorial is replete with triage exercises which basically has you explore you know put the sliders here and look at what happens over there so a tour a tour of this shiny app that as you might have guessed there's an enormous amount of information in the shiny app I don't have time to go through the whole thing I prepared slides to go through the whole thing there's no way I can possibly get through it but what I will do is go through kind of the beginning beginning of it and then jump to the end so the ordering of topics Oh first off this is the this is the display of the app with all of its bells and whistles okay there's a lot there but if you step through it scaffold through it it's actually not too bad so in the tutorial you start off with data that's why we're here at all it all starts with data so we start with the data we move over to the analysis which basically is the model what kind of description of data are we going to use then we move over to frequentist estimation from there we go to Bayesian estimation and then down to Bayesian hypothesis testing then over to frequentist hypothesis testing and then back up to finally frequentist confidence intervals as I thought about it this I think is actually the uniquely best way to scaffold through the topics if you want to talk to me about that please do but there's actually I think very strong reasons to do it in this order so let me just now go through these so first the data section the way I've set up the app here is rather than the user entering their own data I've set up so there's a kind of constrained possibilities of the data so there's one slider for where the mean is one slider for setting up the standard deviation and one slider for the sample size let's zoom in a bit just on that data column and the first triad exercise simply says Oh move those sliders around and look at what happens to the histogram of the data and the obvious things happen you know as you shift the mean up the histogram goes higher you know and they make the standard deviation smaller and the histogram gets narrower increase n and the histogram smooths out okay but that's the the first triad exercise then we go on to the model and for these data the model I'm going to use as just a normal distribution with parameters mu and Sigma now this notion of model is utterly fundamental to a Bayesian approach and I would argue to statistics statistical analysis in general once you get up into applied research and so I take some pains to give my students and in the tutorial at least some basic intuition regarding what the heck a model is and as it says here a model is a data generating machine with control knobs called parameters and perhaps the simplest example I can think of is a bathroom showerhead okay a bathroom showerhead is a data generating machine with control knobs now what are the data well you know you can think of these water droplets spewing out of the showerhead as little data points and you know they fall on the floor and now you've got this pile of data on the floor okay and you've got control knobs you know you've got a little swivel what you can call mu because you know we like creeks and so hey mu that's the swivel and there's a dispersion control we'll call that Sigma and it's very intuitive to say oh yeah we've got this data generating machine okay and it has these two control knobs and you know in statistics instead of calling them control knobs they call them parameters okay you could also say oh there's a data generating machine it's a licorice factory it's a licorice factory and this licorice factory has control knobs the knobs control you know how long the piece of licorice is and you know how thick the piece of licorice is and so it's actually a very intuitive concept and then oh yeah by the way we can put a mathematical formula on the shape of this data distribution generated by the machine but the math isn't the important part the math is that you understand sorry what's important is that there's this thing called a distribution it's a model of data it has a certain shape and we have control knobs called parameters once you understand that then you realize oh I'm gonna understand that data in terms of a machine that mimics the data and in terms of the settings of the control knobs that mimic the data I've spent a few minutes on that because the Bayesian approach relies on this notion of model and I'm just trying to illustrate that you can explain the notion of model pretty much math free or at least with math strongly de-emphasized so they don't move on to a frequentist point estimation and here I just have some pictures we got the the normal distribution that's the machine that's the shape of data that would be generated by the machine the gold histogram those are our observed data and we're just gonna change the settings of the parameters change the settings of those control knobs until the Machine kind of mimics the data right that's how we're going to estimate these parameters and there's a triad exercise you change the nature of the data and then look at the estimates of the parameters and find oh yeah those when the parameters are said over there it kind of describes the data reasonably well okay I'm gonna skip through that quickly because I know this is what you're all waiting for no I know you're really waiting for 9:30 to come but I can't make that happen faster so let me take a few moments to talk about Bayesian estimation it may have been a while since many of you have gone through Bayesian ideas some of you perhaps maybe haven't ever even been exposed to Bayesian ideas I don't know so at the last moment I decided gee I really should insert a quick intro to Bayesian estimation so I grabbed a few slides from other presentations and stuck them in here here we go there are two foundational ideas of Bayesian reasoning you can explain Bayesian reasoning on the elevator between floors by saying these two things Bayesian reasoning is reallocation of credibility across possibilities and in data analysis the possibilities are parameter values in a mathematical model of data okay any questions well if you unpack these two things then it's a terrific summary so let me unpack them first Bayesian reasoning is reallocation of credibility across possibilities this is completely intuitive Sherlock Holmes use this kind of reasoning all the time so here for example is a quote from a Sherlock how often have I said to you that when you have eliminated the impossible whatever remains however improbable must be the truth well what is Sherlock doing here he's going into an investigation of a crime say and what I've got here in this graph across the horizontal axis are four suspects cleverly named a B C and D okay and these are the four suspects for this crime what's being plotted on the vertical axis is the credibility of the hypothesis that each suspect did it so here we're assuming the suspects are independent there are no conspiracies of suspects so the vertical blue bar over suspect a that's the credibility of this notion that a did it that's relatively large in this case cuz you know Sherlock things you know a was nearby a hit a motive you know could have been a then there's suspect D over on the right and suspect D is possible but there's a lower initial credibility and that hypothesis I'm using the word credibility because it's an English sort of everyday synonym for the meaning of probability in a bayesian world so that vertical axis is actually probability now the top of that graph is marked prior because that's what Sherlock believes going into the investigation those are the prior beliefs so suppose Sherlock does some detecting and finds that suspect a is impossible suspect a has an airtight alibi what do you then do you reallocate credibility to the remaining possibilities you reallocate credibility across possibilities that that becomes your prior distribution for subsequent investigation so now suppose Sherlock finds that suspect B is impossible okay you reallocate credibility to the remaining possibilities and suppose now you find o suspect C is also impossible well whatever remains no matter how improbable it was at the beginning must be the truth okay so that's Sherlock being Bayesian what does that have to do with data analysis and data analysis the possibilities are parameter values in a mathematical model of data so I'm going to show you just a really really simple model of data to map it on to what Sherlock did so let's consider the tendency of a coin to come up heads and we'll consider data values here or why that's the thing you're trying to be doing a dependent variable Y is one for heads and it's zero for tails now how many of you in here care about heads and tails chopin's okay so obviously you know we use this in place of something people do you care about any kind of dichotomous data maybe you're looking at the the proportion of females and a species of birds you know you're an ornithologist or the proportion of left-handers in a group of people or the the probability of getting a certain question correct on a statistics examination but we're just gonna go with heads and tails now the tendency for heads is the value of a parameter we're going to call that parameter theta and mathematically you can write it down this way to make mathematic mathematically obscure a very simple idea theta is the probability of heads how do you write that down you say the probability that y equals 1 given the value of theta is the value of theta so if theta is 0.7 the probability of heads is 0.7 okay so we're gonna be Sherlock now we're gonna collect data that is we're going to flip the coin we're gonna collect data and we're now going to infer what theta done it okay what theta done it so we're gonna go into our investigation with a collection of possible theta values here just to make the analogy clearer I'm putting on putting up 11 candidate data values theta could be zero point one point two point three all the way up to 0.9 and 1.0 of course theta has to be between 0 and 1 and here just to be uncontroversial I'm putting equal credibility on all the 11 possibilities okay so those are the 11 suspects for this crime and we're gonna go and collect some data collect some evidence suppose we flip the coin and we find heads so this annotation there is y equals 1 which means so far we have one head and 0 tails we reallocate credibility across the possible values of theta and if you go through the math it looks like that intuitively what's happened well we got a head and therefore higher theta values should be more credible higher theta values are consistent with getting a head notice in particular they look at 0 a theta of 0 the credibility of zero has gone to zero it's now impossible because if theta were zero you'd never get a head we did get a head okay that becomes the prior for subsequent investigation suppose we flip the coin again oh now we get tails so for this flip y is zero we now have a total of one head and one tail we've now reallocated credibility across the possibilities and intuitive intuitively what's going on there oh the most credible value of theta is 0.5 because we've seen 50% heads but notice also here a theater of 1 has been eliminated theta couldn't be one because we've observed a tail so we're just reallocating credibility across the possibilities and the you know we keep going ok this becomes the prior for subsequent data we collect a third flip perhaps now we've got another heads we have two heads in one tail and you can see the peak of the distribution is now centering around two thirds and that's Bayesian estimation that's Bayesian analysis right there that's 90 percent of Bayesian analysis the rest is just details everything you want to know you just read off that posterior distribution so you want to know what are the most credible values oh there they are you just look you can summarize those distributions you can summarize where's the mode what's the most credible value you can summarize the spread how far do you have to go to cover 95% of that posterior distribution and so on and of course in this whole approach I never mentioned a sampling distribution okay I never mentioned some hypothetical value from which we're gonna draw simulated samples and come up with a sampling distribution it's just not there okay so that was my little digression into Bayesian estimation back to the app the app is not flipping coins the app is looking it's using a normal distribution so in that Bayesian column there is a section for specifying the prior and then a section that shows you the posterior zooming in on the prior here's where we set the prior on mu the mean of the normal and you can adjust the center of that prior distribution you can adjust how spread out it is it defaults to a very spread out prior again because that's uncontroversial you're not importing any prior biases into your analysis but you can play with it right you can make that prior very specific and see what that does here's the graph of the posterior distribution it's annotated and of course the tutorial explains all the notation there so I don't assume people you know students know o P of mu given D what is all this crazy notation that that's all explained but here what's marked is the the mode of the distribution that is the most probable the most credible parameter value and the distribution that the spread of the distribution is summarized by the 95% HDI what's that that's the highest density interval HDI it contains the 95% most probable parameter values it's just a summary of the distribution it's simply saying ah everything inside that interval has higher probability than anything outside the interval and that interval covers 95% of the distribution it marks the 95% most credible values it's just a summary and so there are lots of triad exercises you can manipulate the sliders for the data see what that does to the posterior distribution you notice all the intuitively appropriate things for example the the posterior tracts the data so you move them the mean of the data up the posterior on mu goes up and so on you also notice that as you increase the sample size the posterior get narrower you get a more precise estimate you are more certain about the most credible value of the parameter and of course there are triad exercises where you manipulate the prior to see what effect that has on the posterior and one of the cool cool things you notice is that Chi most of the time the prior doesn't have much of an effect on the posterior as long as the prior is reasonably vague the posterior is virtually unchanged and it's only when you have very strongly informed priors that it has much of an influence on the posterior so I know a lot of people might be really concerned about capricious priors like oh it's Saturday morning it is Saturday morning isn't it it's Saturday morning I'm feeling pretty good I've had enough coffee I'm gonna set a really good prior you know I mean you you can't do that in scientific research you can't just throw in some capricious prior your prior has to be justified to the audience of the of the analysis and what this shows you is that if you're going to have a justifiable skeptical prior it's fine don't worry okay I'm coming close to the end here so I just want to tell you one more thing before I conclude as I mentioned when the app is invoked it doesn't show you any hypothesis tests look at that lower row it's blank there's a wonderful book that was written well first came out in the 90s edited by Lisa Harlow and others it was called what if there were no significance tests hence the title across the slide what if there were no significance tests what would the world look like it would look like this it's a beautiful world it just it's what should you believe about your parameter values there it is oh but I know I know sometimes you want to do a hypothesis test and so what you do is you click where I've highlighted in red you click a button for which hypothesis test you want to do and then it populates that section of the app and it goes through Bayesian hypothesis testing which I don't have time to go through it moves on to frequentist hypothesis testing there's a whole attempt to describe in the tutorial what the heck sampling distribution is by starting with a picture of a null hypothesis and illustrating all the you know recent samples and then you know making a sampling distribution inside you know go through the effort to explain that and then this is what gets displayed in the frequentist world but in the frequent to sell of the app it shows the p-value and because p-values despite what we heard yesterday about you know don't use significance that term that's the way P values are you so if you're trying to teach students I think you still have to teach them what that means even while you're telling them not to do that so the Alpha is on display one of the things you'll notice is here I'm doing two tests I'm testing both mu and Sigma and so the Alpha is corrected for these two tests and the P values are marked with the star or NS you know depending on if they fall below alpha and so on so that's all highlighted and explained there are lots of triad exercises you can manipulate the data watch the effect on the p-values you can select which tests you're going to do and look at the effect on alpha and the pretty star or the ugly ns here's something that I've stuck in which expresses it expresses a pet peeve I've had all these years and that is the assumption of fixed in every sampling distribution under the Sun assumes fixed in okay like the one I just showed you a minute ago this one you've got your hypothesis and you generate samples and every sample has the same sample size I have almost never done research with a fixed n in advance and there are some some people say oh the only way you can do proper research is by fixing n in advance well why is that because it's good research no because if you want to look up a p-value based on fixed in your data collection has to have fixed in how do I collect data oh there are lots of different ways so for example you can put a survey on crowd-sourced data collection like Amazon Mechanical Turk so you say okay I'm gonna put out 200 200 opportunities to take this survey and I know I've got like them so I put out these 200 surveys and I get 200 people filtered out well some of those people are BOTS some of those people are just clicking at random to get through it as fast as they can and you know get their dollar 50 and so you've got to point on a lot of catch questions a lot of catch questions that say oh for this question move the slider all the way to the right or on this question please check the box for somewhat disagree and then you have to go through the data and the people who have failed those check questions you exclude so how many what's your in well n is you know 148 it's a random number it's a random number so we really should be sampling to mimic the way the data are actually collected that's what this is for what would have happened if the null hypothesis were true and I collected data the way I actually collected my data that's what a null hypothesis is if that sort of sampling distribution is it's a distribution of data I would have collected the way I collected my data if the null hypothesis were true okay so what I do is I've got this version of random n it's a Poisson distribution on n and it computes a p-value had a Poisson distribution on n and it's a different p-value okay so that's in there for your your pleasure okay so just to recap because I'm out of time here although it is a busy display I think it can be really useful both as a learner and as a teacher because you get to compare the information side-by-side you really get to see oh the Bayesian information is different than the frequentist information oh the hypothesis testing information really is different than the than the estimation information so you get to compare this thing side-by-side and not only do you get to see what the sliders influenced you get to see what the sliders don't influence and you see oh wait a minute when I change these these which tests I'm doing that changes my alpha value but it doesn't change anything over on the Bayesian side and when you change the priors oh that doesn't change anything on the frequentist side and so learning what the sliders don't do is also really valuable I think to students and then finally just to reprise the learning outcomes these internal arrows say you should be able to predict the qualitative effect of every slider and button on the results of every cell and explain why and then those external arrows be able to set the sliders appropriately to reflect real-world scenarios and explain why so as I said this is my first kind of version 0.9 and what I'd like to hear from you in the breakout session among other things is how to improve the app so would it help to have guidelines for where and how to insert this into existing classes that's the goal that you could just take this and put it into a class would it help to have guidelines how about sub modules with videos yeah a videos would it help to have some specific exercise sets and quiz banks should there be a simpler one parameter version so you don't have all these sliders from you and Sigma the peril of a one parameter version I guess I can't put in corrections from multiple tests but maybe it would help what about software that allows inserting user data there are pros and cons of doing that so I hope you can discuss that at the breakout session and then the sixty-four million dollar question has there been inflation is it more than that now the the national debt size question how to get teachers to adopt the app well do I need more arguments in favor of Bayesian demos of how intuitive and easy it is to use examples of how the juxtaposition clarifies both more arguments against frequentist and everybody loves that know everybody hates that so I won't do that evidence of efficaciousness would it help to go to a study and show well when you teach people this they tend to learn more than when you don't teach it to them I know it's but would it actually helped if study was done right to show their certifications I don't know how about endorsement by agency societies and leading instructors I'd love to have leading instructors endorse this so we can discuss that so again the tutorial and the app are online at that weird spelling website I hope you'll at least check it out get a reaction to it one way or the other and whether you like it or don't let me know what you think I really appreciate it again I really appreciate your coming here early thanks thank you very much how about one quick question we have a question they went dying to ask a question you have a chance at the breakout session obviously to question and discuss yes please right no great question so this app is for just a particular set of data that is if you will one group of metric data okay and so we're going to describe those data with a normal distribution that's what this app starts with so if you have different kind of data dichotomous data and you'd like to describe the proportion well that's different data a different model different app what would be great is to have a whole catalogue of these things so here's the app for proportions here's the app for one group of metric values here's the app for you know linear regression of course the more parameters you put in this gets incredibly densely packed right cut off the questions there thank you again very much [Applause]

Channel: CAUSEweb

Views: 1,904

Rating: 5 out of 5

Keywords:

Id: kG43ZKBEqNw

Length: 62min 18sec (3738 seconds)

Published: Tue Jun 04 2019