3 Course Meta-Analyses VU: Selection of studies and retrievement

so the next step in doing meta-analysis is that you have to select the studies and retrieve the data from including studies so that's what I will tell you in this part of the course I will first show you how you can select studies from all those searches you have been done in those databases then I will show you how you can extract and which data you have to extract from the studies you have included in your from your searches the next thing you have to do is that you have assessed the method or the methodological quality of these studies and risk of bias in those studies so first the selection of studies as I explained in the earlier is that you have to save the results of your searches in files you do that specifically for each search in PubMed psych and for Cochrane Database you have to use bibliographic software like EndNote or reference manager it's beyond the scope of this course to show you how these software packages work but I do advise you to use it if you do a small matter analysis you can handle it by hand for example in Excel but if the number of hits from those databases is too large it's not possible to do this by hand and you have to use and notes or reference manager one of the important issues there is that if you search in different databases you will come up with studies that that are included in all three or in two of them so you have to remove the duplicates and the guidelines for doing meta-analysis which are summarized in the Prisma guidelines I will come back to that later they require that you report how many abstracts were removed from your research as you did and how many abstracts are we remain when you have removed the duplicates so it's very important to do that properly it's also very important if you do these searches that you documents everything you do so because you have to report this in your paper so you have to report you have to note the date on which you do you get your search the exact search strings you used the number of hits you had for each database the number of duplicates removed from those databases and the number of remaining abstracts so if you have done that you you you have put together the searches from those difference databases into one and notes or reference manager file and you you have to read all these the remaining abstracts where the duplicates removed the best thing to do is that if you have that list of abstracts you're gonna read for inclusion of studies is to do that by two independent researchers there is evidence that if you do not do that there there is a chance that you will miss certain studies so it's much better to do that by two independent researchers and compare the results of that if you have disagreements between those two searches researchers you should just discuss that and see when you whether you can find a solution if you do not find that you can ask a third or a senior researcher to look at this question you have to remember that if you read these abstracts the only thing you do at this stage is to see whether you will retrieve the full text of that paper so it's not exactly that you are making a decision about the inclusion in your meta-analysis you just decide whether you you you get a full text of that paper so if there's an doubt if you do not know for certain that your study may or may not include needs your inclusion criteria just get a full text and check the full text of that paper so you have you have it's easy for me to say it now because it's only one sheet further but you have been working for a couple of days and all the reading all those abstracts and titles and you have decided about whether or not you're gonna retrieve the full text of those studies so let's say you had 1500 abstracts you decided that you have 100 papers you wanna see the full text of you have to wait you have to go to your library maybe you have to get copies from those papers from other libraries because they're not in your own library but at some point you have those 100 PDFs of those of those studies that may be included in your systematic review and when you have those papers you can start the selection process for include including those studies in your meta-analysis or not again you have to make a very clear list of the inclusion and exclusion criteria and you have to read all those hundreds PDFs very carefully and systematically for all the inclusion and exclusion criteria the nice thing is that if you do that that you also get automatically a very nice overview of that field so you get a nice feel of what is going on what are the issues how good are the studies you already get that when you read those papers so that's a interesting process in itself again you do you you have to do that reading of those 100 papers by two independent researchers so that's very important because the decision of in an exclusion of these studies is of course very important and again with two researchers and you have to solve disagreements by discussion and if needed involve another researcher so suppose you have done you have read those 100 PDFs and you decide that you includes 22 of those papers so then you select those 22 papers and you can start with the data extraction of those 22 papers so then you go to the next phase the data extraction itself so what exactly do you want to retrieve which data do you have to get out of those studies well you have to get all kinds of characteristics of the study you have to do quality assessments or better risk of bias assessments and you have to data to calculate effect sizes and all this should again be done preferably by two independent reviewers so first the characteristics of the studies there's no gold rule for what you which characteristics should be included probably after reading those 100 PDFs you already have a very good idea of what the major characteristics of this group of studies is so but usually you again can follow the Pico acronym the characteristics of the participants the disorder the severity however they recruit it's the diagnosis the type of intervention that is used only CBT are also problem-solving therapy and how many sessions what is the treatment formats what are how it's a manual used how were their therapists trains how was the supervision of the therapist to organize things like that the comparison group so which control which characteristics of a control group you want to give in an overview and which outcomes do you do you want to get from those studies this is a specific part because these outcomes are used for the calculation of effect sizes well I will come back to that later in more detail but there are also all kinds of other characteristics of the studies that you can collect for example the country where it was done the year that that's that the study was done the sample size but again there are no definite rules for what you should and should not collect from these individual characteristics the second thing is that you have to assess the methodological quality and bias and I will I will I will give a somewhat more extended presentation of this because it's key to any meta-analysis so I will first say something about why this is important I will talk about lists you can use to assess quality and risk of bias I think at this moment the best way to assess risk of bias is the cocaine risk of bias assessment - and I will say something about if you have done that how you can how can you examine the influence of quality and risk of bias in your meta-analysis and I will give an example of why assessment of quality is so important and I will give the results of a paper on this we published a couple of years ago so there is a difference between quality and bias risk of bias quality is quite a fake concept it's not very clear what quality is if you ask individual researchers they all come up with all kinds of criteria which could be seen as an indicator of the quality of trial but it's we don't have something to where we can verify that with so it's not very clear what quality is and so for example if you look the quality assessments lists available they there they they all differ from each other so it the quality also depends on the type of studies you examine and there is no good definition of what quality of randomised trials in general is risk of bias is different what you doing risk of bias is that you look at the as where randomized trials can go wrong so where can randomized trials give an outcome which may not be the real true outcome and then you you you basically you look at the weak points of randomized trials where can they go wrong and you assess how in this trial that risk of bias was handled and then you can get an idea of how was is the risk of bias of this study is that high or is it low or do we just don't know and that makes it possible because if you do a meta-analysis as I said in the beginning the the results of a meta-analysis depends on the quality of the individual trials if there is a high risk of bias in all trials than the meta-analysis the the outcome may not be a certain as you would like if there's low risk of bias the outcome of your meta-analysis is good so the risk of bias and validity is a key concept in any matter analysis it defines how valuable your meta-analysis is so mohair in 1995 already identified five different scales and even more checklists for quality and most were based on what what what the author said on generally accepted criteria but there is no as I said no generally accepted list available and I think therefore that assessment of risk of bias is more straightforward and we can assess that much better in randomised trials for example these are some quality criteria that are used in randomised trials and they they vary from very methodological issues for example in psychological treatments a quality criteria would be how the intervention was delivered where the therapist trains where they supervised good enough it's a manual use and how well they told therapists adhere to the manual so quality is a relative thing but if you look at sources of bias that's much more straightforward these are the biases which are used in the cocaine risk of bias assessment tool and if you want to assess risk of bias my advice would be that you use the Cochrane risk of bias assessment tool and they they they just indicate where can i trial go wrong basically and the first thing is what what what is called selection bias so and that has to do with the how the randomization was done so if you do not randomize properly it may be that's the select that there is an error in the selection of participants to early intervention groups or to with the control group and if the randomization goes wrong at some points you may end up with a outcome of your trial which is caused by the selection bias and not by the intervention so what you what you do in what you should check in the published papers is whether the sequence generation of the randomization was done properly and how was the allocation sequence conceals from the from the participants of the trial and usually it is useful it is necessary that the allocation sequence concealment is done by an independent researcher who is not involved in a trial themselves himself or herself this is something which has not been done standards in psychotherapy research until about ten years ago when these issues came up and they entered psychotherapy and psychological treatment fields also and since then you see that more and more psychologists used these proper methods for the randomization that doesn't mean that they didn't do it properly before that only did it dated and reported in their papers so we we are uncertain whether they did it properly or not the second type of bias is performance bias so blinding thats related to blinding of study participants and personnel that's often not possible in psychotherapy research because patients know whether they receive a psychotherapy or are on a waiting list control group or they know whether they are in the therapy group or in the pharmacotherapy group so blinding is always a problem in psychotherapy research and are no good solutions for that then we have detection bias so the blinding of outcome Assessors so if the outcome Assessors after treatment know to which condition a patient's is assigned to then there we see from from research and we've seen that in psychotherapy research too that they are inclined to give a more positive outcome to patients who have received the treatments compared to those who receive the control treatment so it's very important that outcome Assessors do not know to which condition the patient they are assessing sign too then we have attrition bias the problem that some people do not finish the study so if you randomized 200 people 102 to treatment and 100 to the control group not all of those 200 people will will fill in the post-test assessments so maybe you have to drop out or attrition of about 10 20 30 percent but because the randomized it is important in the analysis that you analyze all the patients that are randomized because otherwise it would be possible that those who do not benefit from the treatments drop out from the treatments and that's they do not improve by the treatments and you only keep the people who benefit from the treatments in your trial so then you you if you only look at those who complete the intervention and the study that would give the impression that your treatment is very good well in fact the ones who didn't benefit from it dropped out and if you look at all the randomized patients the the effects are probably much smaller so what you have to do intention-to-treat analysis indicating that all people randomized in the beginning are included in the in the final analysis and that's you can check that in the in the PDFs then we have reporting bias so there are systematic differences between reported and unreported findings the problem there is that there are especially in earlier days that we have trials for example focusing on the treatment of depression and in these trials you use suppose you use three depression assessments outcome measures so you look at those three outcome measures and suppose you find only significant effects of the intervention on one of those outcome measures what some researchers are or were inclined to do is that they done only report results for that one outcome measure for which they they for which they found positive and significant outcomes and they do not report the fact that they did not find significant outcomes for the other two outcome measures so then if you use this in meta-analysis it seems that this intervention is very effective well it may be that if you look at if you calculate the effect sizes for all three outcome measures and pull them that the effects are not that not that good or not so even not significant and that's the problem of reporting bias it's very difficult to to assess that but nowadays what we can do many trials are have published protocols so you can check the protocol the published protocol of that study to see which outcome measures they have used and verify whether they report about that if there is no published protocol you can also look in trial registries because most trials nowadays are registers registered and trial registries and you can check in these registries so in the paper usually you have a number of a trial register you can verify that in the registry and see whether they actually report the outcomes for all the relevant outcome measures well there there is a category of other biases and I won't go into that you can check that for yourself in the cochran risk of noise assessment tool but I would say that in psychotherapy research we have another risk of bias which is not covered by this by the Cochran risk of bias assessment tool and that is researcher allegiance we know from several meta-analysis that there are that researchers may have a preference for a specific type of treatments they are inclined to hope that that treatment gives the best outcomes and we know from all kinds of meta-analysis that if that is the case that the effects Isis found for these studies with researcher allegiance that they are larger than the effect sizes found for studies with done by researchers who do not have that Allegiance so that that may be something to specifically look at in psychotherapy research so again these are the criteria for the cochran risk of bias assessment tool and if you do a meta-analysis I would advise to use this assessment tool these are the criteria isomer III explains what each of them meant earlier and if you assess it for each of the criterion you say there's a low risk of bias because it's explicitly explained in the paper how this was handled and it was done properly there is a high risk of bias there may be a high risk of bias so in the paper they explain how they're randomized but that's not true randomization for example and what often happens especially with all the studies is that you have an unclear risk of bias that paper just doesn't say anything about how this risk of bias was handled again this is very important to do that by two independent researchers and this is from a meta-analysis we just that's one of the PhD students in my department that did it you can summarize the risk of bias often meta-analysis in a graph like this and you can you can report it in different way so you can report the risk of bias separately for each study you can aggregate it for the whole all the studies together or you can get and give a graphical representation like we did here and you have to remember this is core to any meta-analysis the risk of bias is that the outcomes are only Fallot if there is a low risk of bias so how can you examine the risk of bias later when you do your meta-analysis well one thing you can do is you you can say well we only look at the studies with low risk of bias because they are the best studies and if we want to have if we want to have the best estimate of the true effect size we will only want the studies with a low risk of bias that's possible but usually what you find is that a number of studies done it's very small what you can what you can also do is you can look at each of these individual risk of biases and examine them in your meta-analysis I will show later how you can do that what you can also do is you can calculate differential effect sizes for each for the high quality studies and compare them with the effect sizes of the low quality studies and that's a very good way of getting an idea of how risk of bias affects the outcome of your meta-analysis what you can also do is you can you use also quality or risk of bias as a continuous variable which is not really the best way to look at it but you can examine this matter regression in a matter of regression analysis with the validity to risk of bias as a contain outcome and the effect size as the dependent variable I will show you later how you can do these effect sizes so I want to give an example of this so we we publish the paper about quality and size and psychotherapy in for adult depression at that point we had 115 controlled studies on psychotherapy for adult depression we looked at risk of bias and quality criteria which are more or less generally accepted in the psychotherapy field and although we found that one of those 115 trials only 11 met all our quality criteria which are not which are not so very strict just normal quality criteria or risk of bias criteria and what but what was worse is that the high quality studies there were 16 comparison so those 11 studies at 16 comparisons of psychotherapy with a control group the effect size D was only point 22 while the the other studies had an effect size of point 75 and you could say well that's because often waiting less control groups were used which are not usually not the best quality so if we exclude that those studies and LU only looked at studies in which psychotherapy was compared to care as usual we still found that very large difference between the high quality study and the other studies and that the same was true when we only looked at studies in which psychotherapy was compared to placebo so key points of this part is that you if you have selected the studies for your from the bibliographical databases you have to work through the abstracts and select the studies you retrieve full-text preferably by two researchers you have to make a clear overview of the in and exclusion criteria to guide this process after inclusion you have to assess the validity of the study the risk of bias of the included studies which is of vital importance for the value of the of your meta-analysis and if you do that you I advise you to use the Cochrane risk of bias assessment tool

Channel: Pim Cuijpers

Views: 3,577

Keywords: Meta-analyses, systematic reviews, mental health research, Pim Cuijpers, Vrije Universiteit Amsterdam, Psychological interventions

Id: PdXFybvP5Kw

Length: 28min 52sec (1732 seconds)

Published: Tue Nov 15 2016