Introduction to meta-analysis, Joshua R. Polanin

you can have to go thank you doesn't get old keep it going alright thank you all for being here at my session specifically but this is maybe the first time that I've spoken with some of you and I just want to say thanks for coming to the conference in general it's been a blast putting it together over the last year I'm part of the local organizing team I should say that here at Loyola University of Chicago so it's been a blast putting it together the last year I'm never going to do it again so don't ask me but I hope you're enjoying yourself and it's going to be a fun evening on a fun afternoon so so what I want to start out by saying is at the end of the presentation or the second half of the presentation decided to give a little demo on software okay so I see a lot of laptops right now computer savvy people if you'd like to follow along step-by-step here in this room you could go to these two websites right here I'll give you a minute or two to download this it's a very easy process you click on the download button and you can download it or these are streaming obviously we'll have them up on the web soon this will all be made available to you so that you could download these programs and theoretically put in all the numbers and conduct a meta-analysis so but if you wanted to follow along with me step by step feel free the first one up top they're both of the so the the one on top comprehensive meta-analysis it's a free trial so you could you could download it right now and use it for free you just type in an email address and it'll zip it to you rev man we'll talk about more about it in a minute it was it's been built by the Cochrane Collaboration our sister organization that deals more with medical reviews and it is completely free so you can download it now and and open it up and use it for the rest of your life would anybody else should I wait 30 seconds so you can get down the email or the addresses or everybody have it who wants it yep for you Dave I will wait 30 seconds yeah that's right so while we're waiting this is I was going to mention this after my first act oh by the way I'm using a Prezi sobre familiar with Prezi I haven't caused motion sickness with any of my audiences yet but you if you are feeling a little bit you know queasy my mom has to sit in the front seat when we're driving maybe you should come sit up in the front row I don't know if that'll help or I don't know it's excuse yourself for a second and I can send you the PDF version of this Dave you got it go on with the show okay so I'd like to start off by saying that systematic reviews and we'll talk about what that means in a second but this is my view of systematic reviews system error views are like this huge mountain here okay so it's obviously up on the screen it looks like we've captured all of the information possible using a systematic review but when we look beneath the surface we find something that's actually quite large right and when we get to the bottom of the iceberg to get to the bottom of the iceberg we need a quantitative synthesis or a meta analysis okay so what do I mean by that a systematic review well let's start at the bottom I was just talking to Lisa about this and she reminded me one of my cot one of my co-authors and colleagues Lisa Taylor rude wave your hand yep I embarrassed her good if we start at the bottom and maybe you guys have been talking about this a little bit you guys have heard of narrative reviews right sort of the in formal literature review that's sort of the in Campbell speak our meta-analysis speak review speak that's sort of the lowest tier review that is possible right you just go out there and you type in some search terms and you get whatever comes back and you say this is what the research world looks like sort of the the step up or it's a very large step up is what we'd call a systematic review and a systematic review like I mentioned it there are great they pull together the obviously a systematic search process and everything that you guys have been learning about they pull together all of the information within those documents but they often leave out any quantitative information in other words they don't calculate and effect size they don't talk to you about they don't they don't give you the information that David Wilson just provided you this morning and so when we talk about a meta analysis quantitative synthesis another word for it that's when we calculate our effect sizes and then go ahead and pool them which is what I'm going to talk to you today so when I say that systematic reviews are great and they look like they give you the whole picture what I mean is that if we dig underneath the surface area we actually quantify some information you can tell them quant guy already and we quantify some information we might be able to explain the picture a little bit more clearly okay so that's the angle that I'm that's my bias for the qualitative people who in the room that's my advice right there is leaning towards the quantitative information okay I haven't made anybody sick yet right so let's just review the steps one more time I realize that this is the fifth session is anybody is everybody let me ask you this has anybody been to all five sessions have you all been to all five sessions Wow okay so we're really gonna we're going to try to keep this thing choppy as you guys are about to pass out from information I'm guessing but let's just one more time for my benefit and for yours let's just overview the previous steps that you all learned about in the last thirty-six hours a lot of information okay so of course our on talk to you uh that's a little bit miss centered aren't talked to you yesterday my way back at 10:30 yesterday morning about problem formulation right and it really comes down to the purpose of your meta-analysis just like in primary research I use that term to mean general research studies non ie not meta analyses so primary research is what I'm calling just a regular old study regular old study very difficult regular old study you have to start with a purpose right so just like a prior research and meta-analysis you have to start with the purpose you have to have some inspiration you can see it up there so that's the keys to having a clear set of research questions and expectations for your research right then of course you create a protocol while that's really miss centered I wonder if it's at the screen maybe but we can make we can figure out what that means we create a protocol by creating inclusion exclusion criteria right for which studies we're going to select we want to identify the outcomes and we come up with some timeline and we and hopefully we distribute the responsibilities evenly I'm guessing are and talk to you a little bit about making sure that you don't do one of these things by yourself is that correct has that been thrown at you is that phrase been thrown at you at all okay well I'll say it for the first time do not start a meta-analysis or sis matter group by yourself it's a lot of work that will that you'll see and you've probably seen over the last few day and a half so make sure that when you're coming up with this protocol aside from just doing all the things you wouldn't a primary research where you identify your outcomes and talk about your research questions that you identify who's responsible for which tasks and create a timeline okay so next of course you do the systematic search and carry on talked to you guys yesterday about that she's great and I'm sure she gave you could give you a lot more information that then even I am aware of so make sure you follow her advice and of course start with a standardized search tool and locate central articles and then what's very important which I am reminded of every time I do a systematic review it's essential to contact those primary authors especially something we are not even going to talk about today if you have missing data or if you can't calculate an effect size even through David Wilson's great effect size calculator you can still want to contact those primary authors to make sure you could find all the essential information ok then of course Sandra Joe Wilson talk to you about extracting information or coding we like to call coding create a codebook you train the coders double code it hopefully and then you extract the effects sizes and then you get to the fun part which is what I could talk to you about the day what I like to do the most and that's when you're close to synthesizing so you've got all your studies in one maybe Excel spreadsheet maybe access SPSS whatever you're going to use you have all the effect size information retrieved it's very tempting to repeatedly synthesize your effect size and and get a pooled estimates but I would HIGHLY caution you about doing repeating your meta-analysis over and over I would wait to do the step so I'm about to talk to you about here in a minute until after you've collected all of your information and you've calculated your effect sizes first and so you want to make sure that you're almost completely done with that process before you start doing any of the fun fancy statistics that I'm about to show you and you should be able to convince yourself if not others at least yourself that you've collected every study every report every piece of gray literature every effect size that's available we're looking for the population of effect sizes okay that's the only way really that meta analyses can be valid they can be valid I guess if you don't have every one of them but you're going to bias your effects you're going to abayas your meta-analysis if you don't have every effect size collected so you have to make at least every effort and convince yourself and hopefully others that you collected all the studies that you need okay that's not hard to do right those those four steps you got this okay so once you have your spreadsheet so I've just gone over in eight minutes everything we've talked about in the last day and a half okay it's you know a couple months maybe probably more like a year's worth of work that we've skimmed over but you're finally done you've got all your effect sizes you're ready to to finally do the meta analytic synthesis and I should say this I say this whenever I teach introduction to statistics or really any statistics course I'm going to throw I don't know maybe six fancy looking formulas up on the board they are not difficult you can do this everybody in this room can do statistics and some form this is this is pooling effect sizes is a relatively straightforward process if you can calculate an average we're going to take it one step further and calculate a weighted average and I'll explain what that term means but it's a relatively straightforward process so sometimes like my mother-in-law just got done with her intro to stats course for her EDD I'm sorry Rhonda if you see this ever and every every few weeks she would ask me you know for help because I'm her statistician son in law and she would have a you know a slight you know she could oh you know can I do this and I have to say you can do this it's fine take it one step at a time convince yourself you can and we're going to do that same process here all right are we all on board okay good that's that's the kind of energy that we need right there okay so ah a few steps to combining effect sizes across size so of course we're going to compute the effect size in the variance within each study then we create a set of independent effect sizes and then we get to the fun part where we compute that weighted mean and variance of the effect size and we're going to compute that 95% confidence interval and conduct the z-test associated with the fxs okay so of course see David Wilson's presentation on how to calculate these things we won't talk about the effect size since since he since he talked about that a little bit now creating a set of independent effect sizes okay this is a topic of much discussion my dissertation focused almost solely on this slide right here all 200 pages of it feel free to download it and take a look at it but it's it's still a debatable question what do we do when we have a study that has multiple outcomes multiple time points multiple treatment groups multiple sub scales there's a number of different things that we can do the key to meta-analysis one of the assumptions of these statistics is that you have an independent set of data points so there's no just like in if you're familiar with multiple regression analyses or ANOVA right every point every person can't be represented can only be represented once right you can't have one person have multiple rows in your data set that's the same idea in meta-analysis you can't have more than one study represented in your data set okay essentially so how do you get that well this is what I talked about excuse me I'm sweating a little bit this is what I talked about quite a bit in my dissertation there's a you could split up your effect sizes into different syntheses maybe one for each outcome that's something that's Lisa and I I'm going to keep talking about our our project since it's on my mind right now that's something we've been talking about where we have a different outcome we have a different synthesis for each outcome and that way you have a completely independent set some people do what's called lumping effect sizes where you actually take an average of the effect size within a study first so if there's two outcomes listed in a study you would actually average those first and then you've got one effect size per study no matter how you do it and of course there's a lot more to be talked about here that we just don't have time for today no matter how you do it you have to make sure that you have an independent set of effect sizes now I say all that I've spent a couple minutes talking about it and down at the bottom there's one caveat there's a there's a little bit of research the methods assistant editor Emily Tanner Smith has been working on this a little bit there's a technique called the robust variance estimation technique which I will not talk about at all other than to say what it is right now and to say that there's there's a chance within certainly in my lifetime I think probably the next few years maybe in the next decade that's this whole create an independent set of effect sizes might go away and you could actually use all the information within one study so that's just to say that at the moment when you're doing it I would try as hard as I could or are as far as you could to make sure there's an independent set of effect sizes that's the the most likely way to do it but there's some more research coming that you might not have - okay so you're also saying they're like we've spent two twenty five minutes and we haven't even seen a an equation so here we go we're about to about to get to it so now we've talked about how we get the studies a little bit and we've talked about creating an independent set of effect sizes now what we need to do first before we get into any type of synthesis and pooling so we're going to do is we're going to create a weight for each effect size now what is a weight well a weight just quantifies the magnitude that the effect size will have on the pooled effect set the overall pooled affects us what I mean by that is we want the studies that have the largest sample sizes most often to have the most impact on our weighted average okay as remember we're going after our last equation that I'll show you in this set of slides here is a weighted average of all those studies into one effect size and so it doesn't seem and so the theory is that the smaller studies have a larger variance they there's a larger distribution of basically chance findings and so we want the studies that have the largest sample size and the small in turn the smallest standard error to have the most weight in our calculation of our pooled effect size okay and so I was thinking about this slide when I was looking at my slides earlier today and I thought that I would just show you what I mean so that maybe you could get a better sense of this so I just went back and I'm guessing David Wilson talked to you about how to calculate the standard errors of each effect size okay and what we're going to end up doing is taking the inverse of the standard error of each effect size and that's how we calculate the weight I'll show you that in a second but I want what I want to do so I can plant the seed in your head is show you how you just one more time going to write it down the calculation of the standard error we'll just go with Cohen's D okay so you can see how much of an impact sample size has on the standard error so if we look at the calculation of the standard error of D it's I know this to look at it n1 plus n2 so the sample size of the first group plus the sample size of the second group over in one times in two sample size the first group times the second size and the second group plus and I run out of room plus and now here's where you've got the D so this is your actual effect size of study I over 2 times sample size of the first group times the sample size of the second group and this is the standard error and this is the thing this this the output of this equation the standard error is the way that we calculate the weights okay I'm show you in a second so you can see as you increase your sample sizes you're going to decrease that standard error that's what happens and so the studies with the smallest standard error what ends up happening get the largest weight in the effect size okay and I'll just give you I'll throw one more at you if we were if you were going to synthesize correlation coefficients it's completely dependent on sample size and actually its standard error of r is 1 over the square root of the total n minus 3 and that's the sample size and that's the standard eh okay oops I'm going too quick there we go ok so I just want to plant that seed in your head ok let's see I have all this yes it does so the key is so so once again before we get to the next slide we want to use that way to mean because the larger studies have smaller variance and it makes life a little bit more difficult now before we get into how do you calculate the weights or let me show you will I show you the equation for calculating the weight there's one more thing that we have to and that's decide which model we're going to use before we actually synthesize anything and this is a fun way of saying that statisticians like to make your life more difficult they make you choose a model before you even do anything okay and it's key that you choose a model now what do I mean by choosing a model okay we have developed two different ways of calculating the weights essentially you can calculate using the fixed effects model or the random effects model now there's literally an entire we last year we spent an entire hour and a half discussing what's the difference between a fixed effects model and a random effects model so I'm going to spend 30 seconds on it right now and really confuse you all but the idea is do you think you're going to estimate with your pooled effect sizes one single parameter where the effect sizes only differ by the sampling technique sampling error or do you think that you'll be actually estimating multiple parameters where the studies differ by sampling here and by some underlying just characteristic that isn't explained just by taking the average of them and so if you think you're only estimating one this is what I say here if you think you only estimating one single population parameter you choose the fixed effects model if you think you're estimating more than one you're going to choose the random effects model and so what does that actually do it changes the weights slightly in a fixed effects model when we calculate the weight all we're doing as you can see in the bottom of the left-hand column there all we're doing is taking one over the variance component which is the square root of the standard error where this okay so we use a fixed effects model we calculate the standard error we get the variance and we and we take the inverse of it and that's our weight for each study okay if we're going to use a random effects model we have this this term in here tau squared and tau square it is a little bit of a it's a little bit of a bugger because it's difficult to calculate and it's a long equation and it's something that's a little bit outside of the scope of an introduction to meta-analysis class but the point is if you're going to conduct a meta-analysis you're going to have to make a choice between a fixed effects model and a random effects model and so these are the differences here now all that being said the next few slides I'm going to use a fixed effect model and it's the easy mainly because it's easiest but also because no not but also because because it's the easiest but I will say that in Campbell reviews we often pressure or push review authors the methods group does to using a random effects model it's a little bit more conservative the standard errors are the confidence intervals a little bit bigger and it's just because you have you're calculating the way it's a little bit different okay okay so whoo there we go we're finally here at 145 grades this is the all-important equation so you've gone through you collected all your studies you've you've calculated all of your effect sizes you've calculated the weights for each effect size remember that one over the variance or one over the square root of the standard error and now you're ready to synthesize your effect sizes into one pooled effect size and this is the calculation this is the fancy calculation now a lot of symbols a lot of numbers up there but all this is saying is for each study okay for each single study that you have you're going to multiply the weight of that study times its effect size okay and you're going to add all that that big number up okay so if you've got three studies you're going to calculate the weight calculate the effect size multiply that for each of the three studies add that number up and you're going to divide it by the sum of all of the weights just the weights themselves so mathematically if you remember back to algebra maybe back to high school or whenever mathematically this is actually no different this is equivalent to just the regular old average because the sum of the weights terms will cancel out on the top and the bottom but for us for our purposes it gives it provides the weighted effect size term and so if we look you think well Josh I don't believe you I need to I need to see some proof I don't just trust that you know you're doing I've done a little bit of a practical example here can everybody see this at least a little bit so I have three studies that would be row two three and four and I've chosen to use Terry Pickett's last name my name and Emily Tanner Smiths the co-chair of the methods group and each one of those studies let's say we are looking to increased mathematic test scores using some program okay the most basic most basic example that we could come up with so you go into those studies and you calculate an effect size for each study this is exactly what your spreadsheet could look like you calculate an effect size for each study it looks like for mine and for Emily's we're getting positive effects as you can see and Terry study is failing miserably the control control group students are doing better than the treatment new students okay and now so what I've done here is I've actually calculated the weight so I have putting the different statistics you'll have to assume that I know what I'm doing and calculating the weights and so that third column the middle column that weight column is the weight for each study so that's that one over the variance component the square root of the standard error okay so my study was the largest so I have the largest the biggest number in that weight category so my so the the plannin row is going to get the most weight when we calculate that average effect size okay so then the last column over is the effect size of each study times the weight of each study okay and so this is the one that matters here okay so we sum this term up and we get seven point four four six so if we're following along with this this summation right here is the we'll call it the top part of the equation just in case we don't remember what they're called is the top part so seven point four four six and then of course the point three nine one is the sum nope it's not I don't have the sum of the weights on there yeah this is silly of me well we we've got the sum of the weights Oh some of the weights nineteen there it is right there okay I know what I'm doing so we've got so if we take the seven point four four 6/19 do a little bit of math we come with our weighted average of 0.39 one okay now you say well that that's all well and good what would happen if we just took the average of those three studies so this would be the unweighted average this is sort of the non meta analytic approach and you get an effect size of 0.1 zero seven okay so you can see this is obviously an illustrative example because there's quite a big difference between the weighted and unweighted effect size and the weighted one is is pulled up by the large effect size and the large weight of my study okay and that's it that's meta-analysis okay we all go home let's just let's just take it's just take five that's the that's but you know we kid but that that's the essential part there's no tricks here this is the equation that you're gonna that's that the computer is going to calculate and that we're trying to derive so that's the practical example okay so but once you get this there's a whole lot more we can do with it okay so the next thing is some policymaker or some pro se some researcher or some astute policymaker is going to say yeah but what about the confidence interval around that effect size okay so to do that we have to calculate the standard error now of the effect size so now we've had standard errors for each individual effect size now we're going to calculate the standard error for the pooled effect size the weighted average and to do that it's just it's simply the square root of one over the sum of the weights so if we go back down here oh this is much it's a better zoom there the sum of the weights again is nineteen and the one over the sum of the weights point zero five two and we take the square root it's 0.229 okay so that's all well and good what do we do with it once we calculate the standard error that pooled effect size well we can calculate the 95% confidence interval and we can also conduct a Z test just like we would need any any other analysis so let's look at the 95% confidence interval first just like the 95% confidence intervals that you would calculate for a t-test or any other confidence interval you're going to first calculate the lower limit of the effect size you do that by taking the effect size - that we usually is one point nine six times the standard error of the pooled effect size and then of course you do the same thing for the upper limit so if we look here for our example the lower limit is point three nine one minus 1.96 times two two point two two nine and that's point zero five eight why don't I think that math is wrong I might have to check that for some reason I think that 1.96 is okay that's alright in the upper limit we're just going to assume it is and the upper limit is point three nine one that our weighted effect size again plus point one nine six times point two two nine and that's point eight four one so that would be the 95 percent confidence interval or round the pooled effect size if we want to do the Z test this is this statistical significance test of whether or not that effect size is different from zero we can do that as well just like a normal Z test it's the effect size pooled effect size over the standard error of its of that effect size and for that it's one three nine one divided by point two two nine and that's one point seven zero seven we go to a Z table and look for the p-value and we get a p-value of 0.08 eight we generally use the p-value of 0.05 as a cut-off my dissertation focused on a little bit about this - and I'm I'll tell you the conclusion which was I don't think we should be using a p-value of 0.05 as a cut-off we generally conduct about 50 tests of statistical significance / meta-analysis so type 1 error is quite inflated if we use this point oh five cutoff but at present myself and only a few other people are talking about this so you feel free to use the 0.05 as as you would for anything else you know maybe in 20 years when we're talking again then I'll have something else to say but I'm the only one doing it so okay so we've covered a lot here I want to stop just for a second before we go into the sort of the next step once we've calculated the pool effect those are there any questions are there any questions about this very complicated procedure yeah how can we ignore the your study after you have work four years old you believe it you took all this analysis to bring it up you want to use again and yet you backed up down fall off tests so just what can we do you realize hey the biases due to review testing and apply to us as well as the people on well statistically there are some some ways that we can prevent bias we can use multiplicity Corrections or we could simply stop using significance testing is that where you're getting about research in Haiti and so how can we solve this problem all those statistical tests surely I agree we could stop conducting statistical significance tests no no I don't mean I don't mean stop sistas as a whole this right here is a statistical significance a hypothesis test that we don't I've argued in my dissertation in we've started to talk about that we don't maybe even need these in meta-analysis we already have an effect size no I I think what we could rely on a little bit more is this thing this 95% confidence interval yeah I think there's a I think there's a this is a discussion for a different maybe for a different time and I'd love to hear your thoughts more on it but I think there's a little bit of a different underlying maybe some share maybe underlying idea of using a 95% confidence interval one and two I think if if we could get people to use these yes you're still sort of backdooring a significance test but you're getting a heck of a lot more information than you are just from a p-value we should be published in confidence interval I think I would agree with you but it's not what people summarized they say oh it had an effect on this plant so that was you from the conservative position of expressing the confidence interval depth to actual investment offices have been coming up and making a positive statement about that versus not as you don't make that conclusion no one will understand what we were saying yes and you have no policy you can't communicate it right I agree with all of that I think we this might be a topic for a different time but I certainly agree that we that we need to start thinking about that and why I've been talking about and why we've been talking about it quite a bit yeah yeah that is what I suggested that I in my dissertation I suggested using the false discovery rate direction yeah so that was the conclusion that I came up with exactly so yeah let's talk after this session about yeah yeah yeah I agree I agree and and yeah well let's let's talk after that week I can keep going on this for you know a few hours but let's let's keep going with the talk and we'll talk afterwards oh I do want is there any other questions about the actual nuts and bolts of of waiting and effects sighs perfect okay so you have a your weighted effect size and everybody's very happy and policy makers and politicians are calling you and and congratulating you on your hard work but you have to tell them that you're not quite done because you need to assess for heterogeneity among those effect sizes okay and so when we assess for heterogeneity we're essentially asking is are we estimating some common parameter or are the effect sizes different enough that there could be some underlying difference causing them to be different using difference three times in the same setting the idea so we wanted we want to test we're going to use a visual test and also use statistics to ask whether or not there's a common parameter or if there's if there's some differences in those effect sizes okay so to do that we're going to use we can use a simple graphical test using a forest plot and I'll show you a forest plot maybe for the first time for some of you we calculate the cue statistic and it's another hypothesis test unfortunately and we could also calculate the I squared statistic okay so the forest plot just shows each study excuse me each study the effect size of each study the standard error of each study and then that 95% confidence are all around each study and so it's helpful not only for assessing heterogeneity but just also as a visual to sort of interpret how our effect sizes look sort of laid out and one nice clear visual man okay so let's look at it so this is a forest plot that I've calculated from comprehensive meta-analysis they build this as publication ready so you could potentially pull this out and drop this into a publication I have done so in a publication of mine so I'm not saying it's a guarantee but school psychology review was very happy with how this looked so you could potentially use it in yours and so like I was saying study listed there so again the three studies each row I've got the hedges gee the effect size of course the standard error around that effect size and a bunch of other statistics and then each one of the studies is represented by a line up there and that straight up and down line the vertical line is the effect size and then the bar around it is the 95 percent confidence interval now this big diamond down at the bottom that is the pool the weighted effect size the pooled affects us okay so where the diamond where the center of the diamond is is the pooled effect size and then the edges represent the 95% confidence interval of it okay now the way to test for variation graphically is simply to draw a straight line try to draw a straight line vertically through all the 95% confidence intervals if you can sort of visually draw a line that connects every one of those 95% confidence or volts then you would say that the effect sizes are homogeneous there's no variation between the effect sizes but if you can't you're saying that there's one or two or maybe a group of studies that are completely different from the average group now in ours it looks like we don't have a pointer I don't have a pointer and ours it looks like we might be able to draw a straight line through it's hard to tell the first and second study where those two confidence intervals end but we could at the very least we're going to say we're very close to having a heterogeneous set of effect sizes okay so that's why we can't just do this in visually we are not single case researchers sorry to the single case researchers in that and any single case researchers in the group good all right yeah we're not single case researchers we don't use visual statistics to catch rfx sizes so we will calculate the Q statistic okay I have this long thought experiment which I don't really want to try to explain to you at the moment but the idea is do our effect sizes differ more than just sampling variation okay and that's what the Q statistic is getting at that's what assessing heterogeneity is getting at and that's what this this link fee paragraph gets at which I don't really want to get into at the moment because we're running a little bit short of time but these are the two equations for calculating the Q statistic on the left is the original form on the right is the computational form just in case you would like to calculate the Q statistic by hand fortunately we don't have to do that anymore but it really is if you think back to your nova days for those of you familiar with that it's the same principled idea are the effect size different is one effect size different from the average and far enough away that you can say it'd be different from differed more than sampling variation would indicate that it should okay and so the hypothesis here hypothesis test here is that all the effect sizes are essentially equal okay but there's no difference between the effects so that's art that would be our null hypothesis here okay and we test the Q statistic against degrees of freedom of k minus 1 so the number of studies minus one okay so if we go back to our simple example here we get the effect size of the weights if X this time the weight and then I have calculated each one of those the left the original form okay so each one of these numbers represent each study so we've got the weights of study I of Terry study times the the average pooled effect size minus the effect size of Terry study squared and then you sum that up and compare that summation 7.0 1 2 to 2 degrees of freedom on a chi-square table and you get a p-value of 0.03 and so our once again our null hypothesis was that all the effect sizes are the same or represent the same underlying distribution so because that p-value is less than 0.05 we're saying that they're actually a heterogeneous set of effect sizes that one of them probably differs from the rest of them ok now some there has been discussion and some people have used some researchers have used the Q statistic to justify whether or not they should use a fixed or random effects model remember I talked about fixed versus random effects models about half an hour ago I'm telling you you should not do that in fact most people will say that to you now the Q statistic is heavily dependent on the number of studies that you have if you have a large number of studies in your meta-analysis it's you have a lot of power to reject the null hypothesis therefore you're very likely to say that your set of studies is heterogeneous and so it's quite unfair to select a quite unfair to your model to select a model based on this Q statistic now you're going to go out there and you're probably going to read people doing it people then it's still used today but it's a much you have a much stronger argument if at the beginning you can theoretically justify whether or not you've used a fixed or a random effects model rather than just plopping your data into a computer and shooting out a Q statistic okay so I just want to I want to make that clear okay so and of course we usually use p-value less than 0.05 and if we could get into that discussion one more time because there's a lot of these going on as well and you have to think very carefully about what p-value are going to use so there's one more way to assess whether or not we have a heterogeneous set of effect sizes and that's the I squared statistic and the I squared statistic uses the q-value that we've calculated that we just calculated a minute ago and the question in asked it asks a slightly different question it says what's the proportion of the observed variance that's that reflects real differences in the effect size so we're now we're looking at proportions or as the cue statistic just asked is there differences among effect sizes the I squared says what's the proportion of variance that's reflected in those differences so the I squared takes what the cue statistic is calculated it does some fairly simple arithmetic and it shoots out of proportion to you this was created by Julian Higgins he's a British researcher it's a very recent statistic this has probably only been around I'm going to say 10 years and hope that none of you know the exact date I think about 10 years 8 to 10 years so it's it's it's used fairly infrequently in education in psychology and so we're just trying to get people to just to reviewers to use it but it is nice because it does give you this this percentage where you can say okay so if you have a zero to forty percent I squared it's the the heterogeneity is probably not that important this is his language by the way if you have thirty percent thirty to sixty percent don't ask me why the percentage is overlap I don't understand that I don't know what it means when you have 35 percent heterogeneity this is Julian Higgins mind at work right here I guess thirty to sixty percent though means moderate heterogeneity fifty to ninety percent means substantial if it's 75 to 100 means considerable so I guess that means if you have eighty percent on your I squared you get to choose whether or not is substantial are considerable I feel like I'm throwing Julian on the bus so it really never ceases you know sorry to him too but having a little fun as I squared it's a look it's fairly easy to calculate so our our three studies had what are we calling it neither substantial heterogeneity so we're in the sweet spot of substantial hetero infinity with 71.3% hope and so briefly I don't have a slide in here I don't have a slide in here so this is my other thing I have written down on my notes for the conference what does it mean when you have heterogeneity that's one last thing that I want to that I want to discuss before we get into software what time is the session end I should know this because I wrote it but 2:30 2:15 in stops check the hard program 2:33 what does it mean when you have a substantial heterogeneity what does it mean when you have a significant Q statistic well that usually just that usually means that it justifies conducting moderator analyses ok so the next thing you would do is say are there characteristics study characteristics that explain the differences that explain the heterogeneity okay I'm trying to think of a simple dichotomous I've got one so we used to be very concerned we're still concerned with publication bias so one of the moderators that we check is these where the publication derive from was a peer reviewed or was it not peer reviewed the grey literature or is it in a published manuscript so what you would do is you conduct a moderator analysis and you'd say okay what's the average effect size for the published studies and what's the average effect size for the unpublished studies and if there was a difference and a lot of the times just a few moderators will explain that heterogeneity and so you can say okay there's a lot of heterogeneity among the effect sizes in the group that we have but these study characteristics explain away those differences and it's due to publication status or the percent of females in the treatment group or whatever study characteristic that you've coded the point is when you have some substantial heterogeneity you have a significant Q statistic that sort of gives you the right to conduct these moderator analyses so that would be the next step and moderator analyses was Brian Williams presented on it yesterday at 10:30 and morning it's another one of those hour and a half long workshops that we talked to you about so we won't talk about here today but I wanted to make sure that we that we mentioned that okay yeah sure go ahead if you find that you have too much in eternity for your comfort you do how do you get into this problem of doing X too many number of tests oh gosh we're back here again yeah yeah it is it is a real question I'm going to promote my work I've proposed some guidelines on what you should do based on based on the number of tests that are common so I've just said how many moderator tests do researchers usually use and I've based my proposal on that to my knowledge there isn't a hard and fast rule it's still sort of up in the air in terms of how many you use but the caveat being always as always the more that you do the greater the likelihood of false positives the greater likely have type 1 errors different methodological things you could study there might also be lots of different interventions there are yeah our best best I had a study that I coded that conducted 950 significance tests one study one meta-analysis a remarkable number of pairwise comparisons between a number of these different moderators so you're exactly right I mean there's a lot of things that were interested in just but I think a lot of it comes down to in some sense when we conduct a primary research study we're faced with the same problem often and so part of its just research or constraint at some point I can propose all the guidelines I want but at some point we just have to you have to have a theoretical basis you know should we conduct 5 more tests or not I think that's still a debatable question though so it's not a question I'm willing to put my neck out for at the moment but you could read my dissertation yeah yeah talk about looking at these different studies and process controls and creating narratives foundation yeah I think that everything should start with those with that in mind with a with the questions in mind with a theoretical basis in mind and in Campbell reviews we excuse me if Lisa and I just dealt with this if we're we proposed a number of different moderators to test and one of the reviewers said you didn't provide enough theoretical justification for why you're going to conduct this moderator so we had to go back and say these are the reasons before we even this is just in the protocol stage this is before we even did anything so yeah there has to be an underlying theory but at the moment they're there it's still a growing area of research and meta-analysis is so young so I don't think we have the answers quite yet other than it needs to be theoretical ok so I want to move on because I'm pulling out about 12 minutes left so I think what I'm going to do is skip by my funny jokes there's there's a large supercomputer in what I think is the current I can't stop what I think is the kernel maybe this is how he came up with his secret recipe I'm pretty sure fortunately you don't have to use a computer like that anymore and I have another joke my camera what it is so we've only got 10 minutes so what I'm going to do is I'm going to just briefly walk through the it's really only 12 steps 11 steps to calculate this effect size using perhaps of meta-analysis and then I'm going to skip over rep man for now you've got the slides it'll be up on the web there's a number of different great resources it'll just take me more than ten minutes and I don't want to rush rush through it so you'll have all of this information you can plug it all in okay so cma was developed the first version was developed by Michael Bourn seen almost by himself in the late 90s early 2000s and then a team of meta analysts got together over the course of a few years Terry Pickett was one of them Sandra Joe Wilson David Wilson all the people who've been talking to you they all got in a room for a number of sessions they all sat there and they came up with this comprehensive meta-analysis thing and it was published in 2005 and it provides a number of different things just curiously anybody download this whether it's in here am I talking to oh sweet great okay perfect so if you when you do get it and you open up the software after you completely download it I'm not going to use the tutorial once if you open it up for the first time yourself if you guys use this there's a nice little tutorial that walks you through step by step how to plug in all the information and run all of the analyses I'm not going to do that because I think it's a little bit easier for me to just show you this way okay so you start you get a blank screen this is what it looks like just like a normal packaged like SPSS something like that and it's got a number of different little tools at the top we're going to mainly use that insert tool so the first thing you do you say insert column four and you put in study names and that gives you a little study name column there on the left this is my favorite part of the talk by the way I just look at the screen and talk to you about that's difficult packages the next thing we want to do is tell it give it a column for the effect size data all right and then there's there's a little boy who's looking at a computer making you feel bad because he can do meta-analysis and apparently we can't so he's just there to encourage you and also make you feel bad all the same time so there's a little there's a little button down there at the bottom says show all 100 formats make sure you click that you click next you get a screen there there is again you get a screen that says okay what type of studies were included in your analysis and we're going to say there were comparisons of two groups our time points or exposures there's also there's a little tab that says generic point estimates you can also put in proportions or rates there's a number of different effect sizes and just like David Wilson's effect size calculator which I'm guessing he talked to you about in the morning this will actually calculate the effect sizes for you as well if you just have the means and standard deviations but if you've already calculated the effect sizes you can use this is the same so you get a little thing that pops up two groups or correlations you click on the one that says continuous because we're using we're going to use continuous data we're going to use the means and then looking boy that's hard to see on the screen we're going to use the hedges G one where we already know what the standard error is and then he comes up again and says are you sure you know what you're doing and there we go yep and we hit finish and then you get this screen okay so a box comes up and says okay are you sure that you want to put the intervention and the control in these different columns and you say okay and you get a screen that looks like this so again this would be if you had calculated your effect sizes outside of comprehensive meta-analysis okay so you use David Wilson's of excess calculator you've calculated the effect sizes outside of CMA so you're going to put in this do I have a yep so you're going to put in the study names the effect sizes in that hedges G column column put in the standard air and then you can leave everything else blank except for saying if the effect was negative or positive and I'll just mention this one more time if you did have standard if you did have means and standard deviations or a number of different summary statistics F values if your calculate an odds ratio you can use proportions and you can use a whole number of different things CMA is set up to take that data so you can just you can type it in and it will calculate the effect size in the standard error almost automatically for you so you don't have to worry about calculating the effect size somewhere else okay once you have all the studies in there like I do there you go up to analyses and run analysis and after you do that there we go I don't know where that slides in there hey there we go the pooled effect size so now CMA doesn't believe in justifying your model three theoretically before calculating your pooled effect size so it gives you the fixed in random effects pooled effect size automatically for you so you have to be very conscious about which one you're going to choose beforehand because it will spit both of them out here for you but again you can see the each study with the edges G the standard error and then you've got a nice little force well a kind of nice for spot over there on the right if you click on the next table button which is up in the top left-hand corner you get a nice-looking output and this will give you all the information that you would want to put into a manuscript like the point estimate the standard error there's the Z value with the p value again and see my math was wrong it was negative point zero five eight for the lower limit and point eight forty one for the upper limit there's the heterogeneity statistics the Q value what exactly what we got seven points one two same p-value and in the IH squared of 71 point four seven seven this is the high resolution plot which I mentioned to you back a few slides ago you literally just one click away you can see the button that says high resolution plot there and it'll also give you nice so this is a funnel plot which I won't talk about more than to say this is what a funnel plot is this is how you you could assess for publication bias and it has a number of different analyses that you can use for this and then meta-regression and using moderator analysis is also a an option so now I'm worried about I want to go to my summary but I don't know how to oh just look away for can I do this so I'm going to skip over Redman since we've got about four minutes and we'll just do that okay so conclusion so system egg reviews provide part of the answer this is what I'm saying this is my my bias and meta-analysis quantify the effects okay that's just the terminology you'll probably hear people talk about them interchangeably if I were going to talk to you this is how this is how we would talk okay so first you calculate all those affects us in the variances you decide on how you're going to handle independent effect sizes you create the weight of each study from the inverse of the variance the square root of the standard error you synthesize those effect sizes using that very simple to use effect size calculation weighted effect size of fact calculation and then you can create the confidence intervals you can also test using the Z test then you can test for homogeneity assess for heterogeneity is way I've why I've coined it and then enjoy it and do everything else nope and this is just an email that I'll leave up here for Terry Emily and that's it any questions I've got about two and a half minutes if anybody has a burning to a half minute long question yeah all it'll be online we're obviously putting all the the the video online as well but the actual presentation itself will be up there as well so you can click through this is a Prezi it's called a Prezi but it comes out you can you can create a PDF version of it that goes just slide by slide doesn't look as cool but it'll give you slide by slide and maybe won't make you is sick sitting in front of your computer too so feel free to contact the methods group myself Terry Emily we're more than happy to help you out and let me know if there are further questions enjoy the rest of the conference thanks for coming to my talk

Channel: The Campbell Collaboration

Views: 45,135

Keywords:

Id: How7LW7WIFQ

Length: 68min 48sec (4128 seconds)

Published: Tue Jun 04 2013