4 Course Meta-Analyses VU: Calculating and pooling effect sizes

so the next step in doing meta-analysis is that you have to calculate and pull effect sizes step 4 in this step I will talk about how you can calculate effect sizes how you can pull the results the the pool pull the effect sizes of the different studies I will say something about software you can use for conducting meta analysis I will try to show you how you can interpret the forest plot in a meta analysis which is basically a summary of your meta analysis and I will say something about how you can interpret the pooled effect size but first how you can you calculate effect sizes it depends on the outcome you look at and psychological treatments you usually look at continuous outcomes levels of depression levels of anxiety quality of life and if you calculate effect sizes based on continuous outcomes you need the mean and standard deviation from the treatment and control group you can also do meta-analysis on the autonomous outcomes yes or no somebody is ill or is not ill as somebody is that or is not that you can do meta-analysis on other types of outcomes but that's beyond the scope of this course so I won't go into that if you calculate effect sizes based on continuous outcomes what you need is the mean and standard deviation and and the number of participants in each of the two groups you're comparing psychotherapy versus control group if you have these data mean standard deviation and and you can calculate cohan's D Cohen's D is the mean of the two groups that distracted from each other and divided by the pooled standard deviation of the two groups hsg is the same as go hence t only with a little another way of calculating the pools SD and it adjusts for a small sample values so if you have more studies with smaller sample sizes you rather use as just G instead of Cohen's D this is just what a effect size indicates you have a treatment group with a normal distribution and a control group with a normal distribution and the effect size indicates the difference between these two in terms of standard deviations often in trials the standard deviation is not reported so you have to use other statistics to make a to calculate them into the standard deviation so you could you if they report the 95 confidence interval around the means then you can convert that to the standard error and if they only use the report a standard error you can transform that to the standard deviations using these formulas if you have only exact p-values or T values or F values the F values should only indicate the difference between two groups then you can also use calculate effect sizes using software packages and of course you can also do that by hand but that's too specialized if you want to calculate effect sizes from baseline to post-test within one group you also need the correlation between the pretest score and a post-test score and that's often difficult because most studies do not report that so in the end you will get an excel file like this if you get the data from these studies so you have here you have three studies with three outcome instruments a B and C the studies are two studies report to outcome instruments one sorry only one and here you have the means standard deviations and four for the control group and a therapy group basically this is what you need to calculate effect sizes and to answer into the software package later so this I will just give an example how you read such a paper and get the data the relevant data out of it this is a paper one of my PhD students recently published in the Journal of medical internet research on a web-based guided self-help intervention for employees with depressive symptoms and so if you look at the PDF of the study you have here you see all the pages here is a table with all the relevant information but you have of course read the whole paper but you focus on that table where the data are and here you see all the data that are relevant for the for calculating effect sizes it's here at the CSD as a measure of depression and that's where you focus on in the meta-analysis and these are exactly the data you need to calculate the effect sizes the intervention group the control group and for each of those group you have the mean standard deviation and n of the conditions and this is enough to calculate the effect size one problem with effect sizes is that it's not very easy to interpret for example for patients or for therapists because if you want to explain to a patient's what the benefits of a treatment are you should say well if you get these treatments you score point three standard deviation better on depression than when you don't receive this treatment so no patients will understand what you're talking about and many clinicians don't understand that either so a very useful way of making that more comprehensible is to transform effect sizes into numbers needed to be treated there are several methods to do this the most straightforward way is developed by Kramer and Kupfer it's not the best method but it's the most straightforward and that has the advantage that each effect size directly corresponds with one value for numbers needed to be treated numbers need to be needed to be treated indicates the number of patients you have to treat with one treatments in order to have one more positive outcome than in the alternative condition so numbers needed so an effect size of let's say points:1 gifts indicates and numbers needed to be treated of 18 which means that you have to treat 18 patients in order to have one more positive outcome than no treatment of that of that group of patients you can also calculate effect sizes based on dichotomous outcomes but then usually you work with alter ratios or relative risks and here you see what exactly odds ratios and relative risks are and these odds ratios and relative risks can be directly answered into software for doing meta-analysis so if you look for example if you collect data based on dichotomous outcome this is a table you would you need in order to calculate the effect sizes the number the total number in the treatment group the number of events in the treatment group the total number in the control group and the number of events in the control group and this allows you to calculate relative risks or odds ratios or risk differences in these within these studies and also calculates a pooled relative risk or odds ratio so then you have collected these data for all these studies and you have calculated effect sizes for each of these studies how do you pool that how do you calculate pools effect sizes well pooling means that you combine or integrates the results the effect sizes in the individual studies into one overall estimate of the true effect size the that has several advantages over the individual effect sizes from the individual studies they're more precise you can detect smaller effects effect sizes and you can also examine different groups of studies - so do subgroup analysis or moderator analysis so you just it means that you calculate the mean of the effect sizes and basically the idea is that if you have a large study that should weigh more heavy than a small study and if you wouldn't if you would just calculate the mean of the effect sizes that would mean that a small study of 20 participants would be as important as a study of a thousand participants so that's not correct and what you do in a matter analysis is you just calculate the mean of the effect sizes while adjusting for the size of the study that's the basic idea there are two basic methods of pooling you can pull according to the random effects model and according to the fixed effects model if you pool according to the fixed effects model you pull you assume that all the studies are exact replications of each other so for example you have a medication which from which you access Youm that it works the same way in patients in North America Europe or China and it doesn't really matter whether those patients are poor or rich or whatever and you will assume that all studies are estimates of a from the same day all points in the same direction there are no differences between the studies in the random effects model you allow each study to introduce some of its own heterogeneity so you it's it's a little more conservative the 95 confidence intervals are somewhat broader than in there then in the fixed effects model but you allow that there are some kinds of differences small differences between studies and I think that in our fields in psychological treatments we should always use the random effects model in meta-analysis so if you think about choosing between the random or the fixed effects model I think for all complex interventions like psychological treatments or case management or other complex interventions there are always differences between studies saying that you should use the random effects model and I would say that you that you that you should always use that random effects model unless you're absolutely certain that there is no heterogeneity between those studies and I will show later that it's very difficult to show that there is indeed little heterogeneity and even when you find a statistical very little education for statistical heterogeneity but if the uncertainty around that is very high I still think that you should choose the random effects model because it's more conservative so what about software to do these meta-analysis you can you can do meta-analysis in all kinds of software packages you have Stata you have SPSS and SAS and there are specific macros available developed by David Wilson it was an expert in the field of meta analysis you can download them for free you can use the software developed by the Cochrane Collaboration which is called review manager it's also available for free but I still think that comprehensive meta analysis is the best software package at this moment available and it's the easiest to use you can download a free trial version from this website and the big advantage is that it's that it's it's it allows you just to copy and paste from a spreadsheet data from us you have collected in a spreadsheet just put it in cm a comprehensive meta-analysis and run your analysis with one push on the bottom on the on the button answer the advantage is that most not all but most sophisticated analysis are available in comprehensive meta analysis so I did don't pay me to say this but I do think that it's the best software package available at this moment so there are all kinds of tutorials available here you see the website again and you have all the tutorials in which the developer of this package explains how you enter data how you do your basic analysis and how you do all kinds of advanced analysis step by step by working through the software so you can you you can download that trial version I will show you briefly how it works here this is when you open comprehensive meta analysis you get this screen and I will show some brief very briefly how it works this is the opening screen if you look in the menu you click on insert then you get a new menu you first click on study names and then you do that again and click on outcome names you can this screen and there you can just fill in or copy the study names and the outcome names if you wanna answer data for calculating effect sizes you again click on insert and then add effect size data in the new menu and then you get a new screen like this you continue to the next screen and you get all the possibilities all the different ways of calculating effect sizes in the software and then you click on continuous measures and for most meta-analysis you can use mean standard deviations and n so you click on that click on finish you click OK and then you here you see the screen you'd then get and now you filled in the data which I gave in an earlier sheets with the example not really not real data but just for illustrative purposes just this is just with just what you get you can you can just answer mean standard deviation and and copy and paste it from a spreadsheet indicate the direction of the effect size do people improve or not improve and then basically you can run your analysis so what you you you have to further have to indicate a direction which can be you have to think about that so if you if people get better score better in the therapy than in a control group I I would say that's a positive effect and so you you you in the software you have to indicate that that's a positive effects and that if if the treatment group is worse off than the control group then it's a negative effect but you can also do it the other way around that doesn't really matter as long as you do it consequently what you also see in this screen is that if you have entered these data you get yet you get the values where the effects are and our standard errors so here your sense either standard difference in means which is Cohen's D and you CH is G with the standard error and then you are ready to run your basic analysis you click on this button run analysis and then you get to a new screen don't forget that you are in this and below in the screen you can choose for the model you want to work with and we always work with a random effects model so you just push the random button here and here you see the summary of the of the meta-analysis three studies with a pool outcome you can click on this button and then you get into a new screen which is more informative than this one so you here you see the outcomes for the fixed and effects effects model you see also see I squared I will come back to that later because that's an indication of heterogeneity and you can with the nice thing of CMA is that it's also possible to directly export a high-resolution plot which you can download in other software in words or powerpoints to put into your published meta-analysis you can also examine for example if you click on the analysis button you can look for publication bias here you see the three studies in a funnel plot and what you see here is that there are no missing studies I will come back later how you should interpret this but so as you see it's very easy to use just copy the data from your excel sheet to CMA push the run analysis button and you get your outcomes but how can you interpret the forest plot what exactly does that mean well basically it's an excellent summary of a meta-analysis it gives a summary of each of the individual study effects the effect size and the 95 confidence interval but also the pools outcome and you can examine outliers so this is a forest plot which I took from a recently published meta-analysis from Kathy Griffiths from Australia and world psychiatry and what she did here it's about personal stigma and social distance and what you what you see here is that she ordered the effect sizes from the highest to the lowest you can also you can or you can present it in any way you want most people present it's just an alphabetical order on the F the the name of the author but this is also very good so there's there's no reason to do not to do this what you see here is for this is this effect size indicates the study from Cairo Palace this is the 95 confidence interval around that effect size and here you see the pooled effect size this is also this first 30 is also an outlier and how can you see that well if you look at the pools effect size at the bottom and the effect size from Cairo polis they don't those 95 confidence intervals don't overlap from all the other studies you can see that the 95 confidence interval they overlap with those with the confidence interval of the pooled effect size so that's there's no significant difference between these studies and the pooled outcome but for Corrales there is that does differ significantly what you should do with an outlier like this is examine why this is an outlier so you have to very look look very carefully to that study and try to examine what the reasons are why this is an outlier but the nice thing of the forest plots is that you can see it directly in such a plot well this is this is another example no just to just to show you how these forest plots look like so how should you interpret the pooled effect size well cohan when he developed the Cohen's D the standardized mean difference he made a rough estimate of what a small moderate and large and he said while an effect size of point to a small 0.5 as moderates and point 8 is large there that is not based on empirical data a few years later Lipsey and Wilson by that time there were already several hundreds of meta-analysis and what they did is they took all those meta-analysis together and they looked they they looked how large are these effects and they had a more data based indication of what large small and moderate are and you can see these values here you have to remember that that an effect size is still only a statistical measure and that is not directly related to a clinical relevance outcome so an effect size is not more than a statistical outcome not a clinical outcome so for example if you would find an effect size of 0.1 and which indicates the difference between a treatment and control group in terms of years of survival then most clinicians would consider that to be a major breakthrough if you would have on the other hand an effect size of 0.1 on for example social skills or knowledge on anxiety then most people would not consider that to be clinically relevant so that same effect size of 0.1 can be highly important clinically important or it can be earning for and it's still that same effect size of 0.1 so the key points here cohan's D indicates the difference between treatments and a control group at post as in terms of a self standard deviations you can use many other statistics and basically anything with a standard error can be pulled in meta-analysis pooling means that you integrate the results of individual effect sizes into one overall effect size and if you do a meta-analysis than the forest plots of the individual studies and effect sizes it's an excellent summary of that meta-analysis and effect sizes are aesthetic statistical measure and you could say that's an effect source of point to its small 0.5 is moderate and 0.8 is large

Channel: Pim Cuijpers

Views: 12,186

Keywords: Meta-analyses, systematic reviews, mental health research, Pim Cuijpers, Vrije Universiteit Amsterdam, Psychological interventions

Id: VQla4idVnm0

Length: 25min 14sec (1514 seconds)

Published: Tue Nov 15 2016