Systematic Review and Meta analysis - All you ever need to know

welcome to another podcast from school of surgery I'm here today with Bret Dolman who I work with it's got a very very strong background in statistics and particularly systematic review and meta-analysis so welcome Brett thanks for coming hello thank you and so today mechanizing one of these things that taken as a last word in truth when it comes to research and and everybody looks at these and decides and changed their practice or quotes them and each other but I think they're probably quite poorly understood things and and it's important everybody knows how to look at a match analysis rather than just a loss engine see which side of the line it is he's actually be able to understand it properly in all the strengths and weaknesses so hopefully today we're going to we're going to talk around the eyes all right yep that's fine so yes as mr. Allen said we're going to discuss today systematic reviews and meta-analyses and we're going to start from basic concepts and try to explain all of the advantages and disadvantages of systematic reviews and meta-analyses which should help you in your future practice when appraising papers under exams and the exams of course so the learning objectives we're going to discuss the advantages and disadvantages of systematic reviews and meta-analyses and I think this is a key point because meta analyses are often considered at the top of the hierarchy of evidence however current thinking's shifting that slightly we're going to talk a bit about the the key concepts when you come to look at meta analyses in published journals and how you can appraise those papers we're going to look a bit at basic plots so forest plots funnel plots and bubble plots we'll discuss them and what they mean and what you can glean from from these plots finally we're going to discuss the stages of meta analysis so meta analyses are actually a good thing to get involved with if you want an introduction to research because they're very flexible around your clinical duties and if done with somebody who who understands them they're they're a good way to get an appreciation of how research works on how to appraise research so let's just look at some definitions first so referring to a systematic review a systematic review is a systematic search for all available studies to answer a particular clinical question now the important aspects of this statement is systematic so the search that you do need to be systematic and it needs to be reproducible and quite often you'll be expected to include search strategies including what journals you looked for studies under and it needs to be replicated by people in the future so it gives an example of that and so you see all these things and they have so what's what I already have a search strategy what would you what you write down when you ain't known a bit of paper first so what I would recommend first actually is most hospitals have a good library and a good clinical librarian who can introduce you to how search terms work so we've all heard of MEDLINE PubMed as a as a journal database and there's various ways you can search for studies and what you would do is you would base that on your clinical question so this is often expressed in the Pico format so you would first state your participants so who you'd want included in the studies the intervention you're looking at and the comparison as well and you also want to look at particular outcomes and that's how you would structure your clinical question and you would then use particular search terms based on that those clinton those Pico format so you start with a question we want to answer so let's take an example so you and I recently wrote together a meta-analysis or access review in fact wasn't it over pile of treatment pilonidal sinus yes so how do we do it we we said we wanted to know the effect of fibrin glue in pilonidal science anyway so what what tools we use so much and how do we do that so you would use the intervention there so fibrin glue was the intervention we looked at there so you could look at that both as a keyword and you can also expose particular terms that have other related terms but again I would add speech some clinical librarian on how to do that exactly and they can construct a very good search strategy for you and then once you understand in more detail what the particular terms mean and how they link to ever then that's something you can take on yourself but it's it's a good skill to get used to not just for systematic reviews but for when you want to research a particular clinical question that's related to your clinical practice and so I would recommend that yeah so you go think of easy called pilonidal sinus and you call it pilonidal cyst and all the other words that you can diagnosis and operation and surgery and intervention and treatment and have all these things in the librarian can help you string those together in barriers and knots and ORS and phrases in brackets and in inverted commas etc and that's important so you get you actually get what you want to see and then it generates a huge amount of well demands a few let suppose but the the the the initial data set of papers you want to start have a look at is that right yeah and for example if you're looking for fibrin glue what we had to do is make sure we captured brand names as well as the generic term for the drug because some studies might just quote a particular brand name if funded by that particular pharmaceutical company so you need to take all of that into account when looking all the various our tasks using wide stars exactly exactly see what comes back and you can always refine it a bit closer or again go back to the clinical librarian and say you want a bit more of a sensitive search or specific search based on that so next definition is for a meta-analysis so a meta-analysis is the statistical pooling of results from all of the included studies you've identified during your systematic review now the decision of whether to do a meta-analysis has to be taken on the basis of the studies you've got in front of you if the studies you have in front of you are too different then it might not be appropriate to pull these and I think the temptation is that everybody likes a nice forest plot and they'll try and pull together lots of studies performed in different ways in different populations that just aren't comparable and therefore this needs to be considered before you perform a meta-analysis so fundamentally different doing the systematic review on a mesh analysis is the statistical pooling so you get you do literature search sort out what the everything you want to include and exclude because the criteria previously agreed and then in a meta-analysis you'll do some stats which will come want to talk about after this can you only include randomized control trials in the meta-analysis I think it depends on what you're looking at so for example if you're looking at an intervention you want to know the most kind of robust study to assess that intervention you would ideally want to include randomized control trials however if you're looking at say a surgical procedure and you want to know outcomes such as long-term recurrence quite often the best studies in order to measure that are observational studies and in fact some of the the best systematic reviews and meta-analyses have actually pulled together both randomized control trials to assess short-term outcomes and then for longer-term outcomes such as long term recurrence use large observational studies and they can often work well together but ideally the most suitable study to assess an intervention would be a randomized control trial Thanks so advantages of systematic reviews and meta-analyses firstly by increasing the number of participants within a particular analysis you increase power and what that means is it increases your ability to detect a statistically significant difference between the groups if one should exist and we'll see quite a nice example of a forest plot where we've done the analysis and increase the power and that's come up with a conclusion that wasn't really apparent in the individual studies secondly meta analyses can look at studies performed in different clinical populations to see whether this consistency for example you might read a study in British general surgery which might show some nice results from a particular surgical procedure however that might just be one of a whole range of studies that have been done in perhaps different clinical populations and you might not see that effect in in other populations and a meta-analysis is a good way to evaluate whether there is consistency across clinical populations thirdly part of meta-analysis that's often overlooked and is actually poorly done is you can actually look at the studies you've included within the meta-analysis and you can investigate whether perhaps if they would the intervention was delivered in a different way and the classic example in medical studies would be dose of a medication and what you can then do is investigate whether the effects in each individual study differ based on the dose of the say analgesic there's you've given those studies and that's quite an important aspect of meta-analysis that's often overlooked and we'll talk about that later and finally systematic reviews by their very nature include all available evidence rather than cherry-picking of evidence so we've discussed a bit about if you find come across a paper in which stirring of surgery and it shows a nice significant effect on the intervention you're evaluating then that could just be because it was a novel interesting finding that was in one of the leading journals there may be other evidence out there that's in the lesser journals that shows negative or non beneficial results from an intervention so so I'm going to come on to the disadvantages about meta analyses in again these are often overlooked and often if you read traditional textbooks on clinical research you find that meta analyses of randomized control trials are at the peak of the hierarchy of evidence when in fact there's many disadvantages associated with meta analyses that actually I would question whether this is actually appropriate so there was an interesting study done in the past couple of months in the British Journal of anaesthesia and this looked at 16 interventions with over 100 endpoints and it looked at whether meta-analysis could predict statistically significant results in large randomized control trials and it found that meta analyses only had a sensitivity 42% and a specificity of 70% and for predicting significant results in large randomized controlled trials which essentially means only 42% of the statistically significant results meta-analyses were later found to be statistically significant in large randomized control trials and indeed there's been other research in more general fields which is showing that the agreement between meta-analysis and large randomized control trials is poor and this is often cited to be because of something called publication bias which we will discuss later and there's a noise inherent in smaller trials which you add up to make more noise and so I mean so the best thing really would be an RCT that's large enough to answer the question but of course the problem is is funding one RCT is large enough to do or preferably to that's what drug administration's often what early to large em RCTs and so it's probably a way of getting around you know not having to have 10,000 people in a particular trial but you have to that's important is that everyone thinks that because it's where it is and that's that's it that's the conversation over yeah I would certainly say the optimum method to assess any intervention is a large randomized control trial be enough exactly exactly and we'll discuss a bit later all of the disadvantages associated with with meta analyses and all of these kind of adult to quite unreliable method to predict results in these later large randomized control trials and certainly within the field of anesthesia there's a major push towards trying to get these large multicenter multinational trials that have have shown very different results to proceed in meta analyses and and and this is important because it could lead to interventions being used that are either harmful or not beneficial and the classic example of that was antiarrhythmics in myocardial infarction so everyone thought that antiarrhythmics were great and that they worked in acute myocardial infarction and meta-analyses had shown that there was a beneficial effect of antiarrhythmics in myocardial infarction but when they later did the large randomized control trials it was actually found to be harmful and again this was thought to be the various limitations of meta-analysis so how I'm going to focus on the limitations of meta-analyses is to think of them in terms of a grading system which is conveniently titled grade which is a criteria that's used in Cochrane reviews and it looks at five different limitations of meta-analyses and downgrades evidence based on this so the first assumption is that evidence derived from a meta-analysis of randomized control trials is high-quality evidence and based on the presence of any one of these limitations it can be downgraded to either moderate low or very low quality so I would suggest when looking at meta-analysis in the future and particularly Cochrane reviews if you look at the quality of evidence quoted you get a better idea of how alive all the results are from that particular meta-analysis what you see in the text when you're reading so the way it works in Cochrane it will say usually with the outcome it will have a usually it's by the actual result it will say high quality moderate quality low quality or very low quality evidence and this is usually within the abstract of Cochrane and that's part of the reporting standard so it should be within the abstract so you'll get an idea from from the actual abstract itself whether that evidence is high moderate or and in things that have gone to be Cochrane so meta analyses have been produced for just published in a journal and will it will to tend to have these because I haven't seen it too much of thinking non Cochrane one it's less often used so this has been developed over the last I think decade or so and it's starting to filter through so I've certainly seen some journals require you to use this gradient of evidence but I think it's it's essential and I think like most things it takes a bit of time to filter through to the academic community but I think this as time goes on I think more journals will want to adhere to Cochrane standards and they're looking at mesh analysis this is the quality marker it isn't if you can see the this grade criteria used in in the text then you can think well this is probably a better done petronella system one that doesn't yes certainly and within the text itself they often have and tables listen which will list each individual outcome and it will say what the quality is and it'll say why it was downgraded so the actual Cochrane reviews themselves have got quite clear tables quite often the the table for the major outcome will be very early on in the review itself so you can get a good idea of quality of evidence from that so the first thing I'm going to come to is risk of bias which bias is essentially any deviation away from the true underlying value and the the kind of really basic way to think about bias is the kind of lay term of a bias referee in football in that if you have a bias referee in football and they give lots of penalties to the team that they're favoring it might deviate the result away from what the troopers would have been and that's just a way to think of bias and and essentially this is because any meta-analysis is only ever as good as the trials it includes so if you've got a meta-analysis that includes lots of poorly conducted studies the results become a lot less reliable than one that includes maybe larger better perform studies that are free of bias now the gold standard in order to undertake this is to use something called the Cochran risk of bias tool and this has many advantages over traditionally used scales such as the j-dub scale and things I which I won't go into now but this has certainly been accepted widely by the academic community that this is the most appropriate tool to use and it essentially allocates different scores to each domain of bias so low risk of bias is often reporting in green and the risk of bias summary if there's unclear risk of bias this will appear as yellow now unclear risk of bias is where there's not enough information within the text of the study to be able to determine whether it's low or high risk of bias so for example they might not explain how they randomized participants they might just say we randomized 200 people - rather than explain the method for which they perform that randomization in that case you would have to say that that's unclear now clearly within the unclear element you can have low and high risk of bias but it's it's important to quantify it in that nature rather than consider all poorly reported trials as being at high risk which is what was traditionally done with the old scales and then finally high risk of bias would appear as red in the risk of bias summary now there's various domains of internal validity risk of bias which are assessed by the Cochran risk of bias tool the first one is randomization so randomization is where the patient's truly did they truly have an equal chance of being allocated to each group so for example some studies would be low risk of bias if they use computer-generated randomization whereas they would receive high risk if they allocated people on their date of birth or their Hospital number because that might not be truly the next one is allocation concealment which just means that can the people who are recruiting the participants for see which group they're going to be allocated to and the example I would give is if I'm recruiting patients for surgical study and I've got this exciting new surgical technique and I know that the next participant that walks through that door is going to be going in my exciting new surgical technique and a frail patient with dementia walks in who fits the inclusion criteria but I don't think it's going to perform very well within the trial then I might not recruit that participant and that creates a lot of bias within the process and this is actually a very under considered thing it's been shown in studies that this is responsible for major differences between study results in how how allocation was concealed and blinding will be a bit more familiar to to this nursing that it's whether participants or people who were conducting the study knew which group the participant was allocated to now this is actually quite difficult to do in surgical studies if you're studying very different techniques because quite often obviously the surgeons going to know which group they were in and but it's important to if that isn't possible that is to try and blind outcome assessment so whoever's assessing those outcomes postoperatively doesn't really know what group they're in but again this might be difficult if they've got dressings and surgical scars yeah so in the famous laparoscopic cholecystectomy randomized trial against open that all all the patients had dressings on some of which was sham dressings so that we put a big dressing in the right hypochondria man's that had a lab clearly also and talked a little bit of blood on in theatres so you deal all at the same to blind the assessor exactly and that I'm sure there's lots of inventive ways you could come up with two blind things but as much as possible it should always be it should always be attempted because it helps reduce bias and attrition bias is essentially participants who were lost to follow-up say for example if we randomized 200 people to one surgical technique versus another and then postoperatively we excluded maybe 20 percent of our intervention group for some reason that could be related to outcome then that would bias the results because we won't get a representative sample which is why a lot of randomized control trials will push for you to do something called intention-to-treat analysis where you you analyze all the participants who were initially randomized regardless of whether they receive the intervention or received less at the intervention and selective outcome reporting that's essentially it's similar to publication bias but sometimes investigators will set out to measure a particular set of outcomes and then because a couple of them aren't statistically significant they won't report this fully in the final manuscript which means it remains hidden and that contributes again to publication bias which we will talk about later and so all good trials will have their protocols published on on clinical trials.gov or a similar website so you can see what before they finish what they were trying to show what their endpoints were yes exactly and that the whole push towards trial registration if if people out there have conducted trials themselves they'll notice that there's a been a big push over the last decade to get randomized control trials registered and this is for two reasons first reason is because of selective outcome reporting it prevents people from changing their primary outcomes based on statistical significance rather than what their initial primary outcome was and the second reason is it allows people who conduct systematic reviews and meta-analyses to track down trials that might not have been published but have been completed so that they can use that data in order to reduce publication bias which again we'll talk about later and finally there's a category for other bias so this might be imbalances in baseline characteristics for example purely by chance one group might be twenty years younger than the other group and this particularly affects smaller trials so I've just got an example here with two risk of bias summaries from two meta analyses and we can see on the left-hand side that the studies are listed in the rows column and the columns here are the different domains of risk of bias which we've just discussed and the same on this side as well and we can get a feel for this meta-analysis on the left the included studies are a high risk of bias compared to the ones on the right-hand side so we can see a lot more high risk element here mainly because it involves the blinding of participants which was difficult within this surgical study and here we've got one looking at post-operative analgesics and we can see that generally there's a better quality of trials included here and therefore could be deemed a slightly more reliable than the the meta-analysis on the left so when you're looking at a source review meta-analysis seeing one of these tables is another quality indicator yeah exactly this will give you a summary of the quality of the trials that are included within that meta-analysis of the metals as well so if you see one of these that they're doing the right thing yeah they've been very very transparent about how got to the final result exactly here so the second thing to consider within the limitations of meta-analysis is directness of evidence and this is quite a simple concept and it just means can I apply the findings of this particular review to the patients I have in front of me and quite often this can be assessed from the characteristics of included studies which will be in most meta-analysis which will list both characteristics of the participants who are included for example what age they are what sex they are what particular condition they had as well as different bits of information about the intervention and the comparison and that will allow you to look at that group of trials and say can I apply that to the population of patients I have in front of me in a very simple way of thinking about that is if all studies were performed in men can you apply these results to women so I'm going to talk a bit about heterogeneity now now this is quite confusing topic and people do get confused between the different aspects of this I'm going to try and break it down into and quite simple language and hopefully make this understandable good define heterogeneity for us for us yes so this heterogeneity is not really Universal term and that it's broken down into two separate definitions so the first thing would be clinical heterogeneity so clinical heterogeneity just relates to differences in the individual trials themselves so again that would be similar to what we spoke about previously differences in participant differences in the way the intervention was delivered you know in surgical studies was it an open to some include open procedures - some include laparoscopic procedures so that can be determined from the characteristics of included studies the next term is statistical heterogeneity and this is explaining the differences between study in statistical terms and all it means is it means that do the studies differ by more than would be expected by chance so if we took a sample of different people from a particular population we would expect those results to vary by a certain degree simply because we are drawing different samples from the same population however there may be other differences between the studies such as clinical heterogeneity which we've discussed as well as different methodological limitations so we've spoke a bit about risk of bias and that can also create differences between the studies and statistical heterogeneity will quantify that and quite often an easier way to look for statistical heterogeneity is to look at the forest plot and we'll have a look at an example of that in a second now statistically this is often expressed by something called the I squared statistic so this is this gives the percentage of variability that's due to differences between the studies rather than sampling variants so we spoke about something variance so a high value in the I squared statistic which ranges from 1 to 100 tells you that there's a lot of statistical heterogeneity there so and that's always on that on there we'll have a little look over a minute and I know but always on next to the forest plot the I squared statistic and I think there's people just ignore that and don't look at it when do you start to have to exercise a lot more caution with it with the I square statistic when and when what's open what well what's a good one and what's a about all so people use different cutoff terms and the Cochrane manual uses quite broad cutoff terms for different degrees of heterogeneity but a lot of them overlap as well so quite often authors will often pick their own reasons for choosing a particular value so they might say that 50% is evidence of statistical heterogeneity but I would say with like most statistics you have to you have to take it in context now clearly anything above 50 is you know good evidence for statistical heterogeneity but even if you're about 30% you need to consider are these and studies quite a bit different from each other so traditionally I would say 50% is used as a cutoff but like most things it's it has to be taken in context so I'm looking at looking at emotion artists and this happens work in surgery and the the lozenge is quite clearly in one side line and then I should look at the I squared cystic and the narcissistic says 10% they well I can have a lot of confidence and that is exactly what's going on and I should have a lot of confidence in that she wouldn't be better than the other treatment whereas I look at it and it says the Icicle a statistic is 90% like I said well that's kind of okay but I'm not so sure that that was in that loss you can move around quite a lot if is a 90% statistical heterogeneity in there so there's a lot of bias inherent in that so people should have a look at lozenge and have a look at the I square sitting next all right yeah and in any meta-analysis that's got a high I squared value so if there's a large degree of statistical heterogeneity if they've used a random effects model which again we'll talk about later it will actually it will penalize that heterogeneity it will actually broaden the confidence intervals as well so that will be inherent within the model itself and essentially you have to say that if if there's a lot of heterogeneity there we lose a bit of confidence in the underlying results because it doesn't look like the studies or assessing one true underlying effect and we'll talk a bit more about varying apples and pears and a Ford Transit yes exactly here so what we're going to look at here is an example of a forest plot which I'm sure most people are familiar with and have seen with in previous publications or probably more likely within exams but what we've got here is on the left hand side this is the list of the individual studies included within the meta-analysis and here we have the intervention group in this case we're looking at when paracetamol administered and this was before surgery this is our intervention group and this will give you a breakdown of your statistics so you mean you standard deviation and the total number of participants within that study here we have our comparison group which again lists our opioid usage postoperatively in our post incision group and this will give the mean standard deviation in total the weight here is just the basically amount of information that that particular study is contributing to the overall result and it's often expressed as a percentage and that's essentially due to how many participants are in the study and the various elevate to greater the weight and yeah yep we'll talk a bit more about the models a bit later but how that weight is distributed slightly different between fixed effects and random effects models now we see this bit on the right-hand side this is our kind of graphical part of the forest plot and this that chart is called a forest plot yes and so here we've got the results from each individual study so the small green dot here that's the difference in means between the two groups within each study and the black line corresponds to the 95% confidence interval in this case which should be stated at the top of the forest plot which it is here now we can see from this forest plot that all of these 95% confidence intervals don't seem to overlap very well they're quite variable as are the difference in means in each of the individual studies and instantly we can look at that there and think there's probably quite a bit of statistical heterogeneity there because it seems like the studies differ by more than would be expected by chance and thinking of it by that way if if it was not varying by more than chance you would expect confidence intervals to overlap with each other in this case they don't and we can see at the bottom this is the pooled result here from all of the included studies now in actual fact the black diamonds there that the top and bottom of the black diamond correspond to the difference in means and the end of each part of the diamond corresponds for the 95% confidence interval and it might be a bit difficult to appreciate on this particular case but here we use in a random effects model and what that does is it penalizes this heterogeneity so in actual fact that confidence interval is actually quite broad for the pool in all of those results and that's because the random effects model will penalize that heterogeneity and we can see at the bottom there we have an I squared value of 79% which is bad yeah that would be regarded as high and we already had a bit of an idea that that was going to be the case when we looked at these studies and saw that the confidence intervals are quite scattered around and that's a good indication that there's likely to be a statistical heterogeneity so when you look at that black diamond that lozenge there then it's good to be as small as possible exactly and that would give you a more precise result and you can have more confidence that the result is somewhere around the result in our actual individual meta-analysis of all over 102 that done the x-axis 0 minus 1 minus 2 etc what do those numbers actually mean so they would correspond to the particular and we call these effect estimate that we would use within that particular analysis now this varies depending on what you measuring whether it's a kind of binary outcome or whether it's a continuous outcome such as means and standard deviations so in this particular example it's the standardized mean difference and what the standardized mean difference allows you to do is combine different measurements of different things into one analysis so what we're actually looking at here is a forest plot of different opioids so some of these studies have used morphine some of them have used tramadol and what a standardized mean difference allows us to do is to combine those all into one analyses because it standardizes and by dividing them by their standard deviation more commonly for continuous outcomes you would see a mean difference which is more correctly called a difference in means where you will just take the mean value of the same thing and then look at the difference between the means of both groups the binary outcomes this is often expressed by either an odds ratio or a risk ratio the more appropriate one would be a risk ratio and that would just be your ratio of risk in both groups which would just be your event divided by your total number of participants and that would give you risk and then you just look at the ratio between the two groups we can see an example here of the risk ratio so this would be a what we're measuring here is post-operative vomiting so this would be a yes/no outcome so this is expressed as a risk ratio and again we've got a similar setup with studies on the left hand side the events and total number of participant in our intervention and again in our comparison and here we've got the results from each individual study and what we can see here is that both the average effects and the 95% confidence intervals overlap quite nicely so there's quite a lot of overlap between all of them and I think we can appreciate that most of these results seem to be pretty similar and therefore we would expect that there to be low levels of statistical heterogeneity in these studies now I think another thing to appreciate here is as we discussed earlier the advantages with meta-analysis over individual trials now the bit I'll come to is here in the forest plot so this risk ratio of one if the confidence interval crosses that point that's equivalent to results being not statistically significant at the level of P equals naught point naught 5 so what we can see is that each individual trial apart from one all had results that were in statistically significant in the original studies themselves however it was likely that these studies were small enough that they were really powered to detect outcomes in a binary outcome such as post-operative vomiting but by performing a meta-analysis and getting our overall resultant at the bottom here then we can see that that's actually identified a statistically significant difference between the groups because the confidence interval doesn't overlap the a1 okay so I mean that's really interesting so they get CBI squared there and you can just to go over you can see that all those despite the the confidence intervals being wider or narrower that they all stack up pretty much on top of each other yeah like stacking plates and then the other thing of course is that the size of the square equates to the weight of the studies all right so that's correct here so that so you can see that represent rather than looking at the figures you can see that represented visually in the in the forest plot as well exactly and we can see that particular study has the characteristics we would expect of a study that would have more weight in that it's got 200 participants as opposed to the other ones that have got around 60 so we can see how the number of participants the larger trials will contribute more information than the smaller trials so this is the important part I come to with statistical heterogeneity and that is quite often meta-analysis people will leave their analysis at this point they'll say well I've identified statistical heterogeneity my jobs done that say it's fine but one thing I think there's a move towards over the last decade and something which I think's essential in any meta-analysis is to investigate this heterogeneity because an important clinical question is well why do these studies differ from each other because there might be an important reason we've discussed earlier dose of a particular medication might be important information within that that we want to identify now this is either performed using subgroup analysis or meta-regression analysis so subgroup analysis you might break your studies into subgroups of say one dose gave 300 milligrams in the other studies gave 600 you would put those into subgroups and you can then compare them each other and that will give you a p-value to say whether they are significantly different from each other but probably a more useful way of doing this is through something called meta regression analysis so this is similar to a traditional regression except instead of using the participant as the unit of analysis it uses the study so we would put the study into the the meta regression and we would say that this study had a dose of 100 this one of 200 this one of 600 and it would then do a regression and it would give you something called the r-squared analog which will tell you the percentage of heterogeneity explained by your model and you can actually put multiple independent variables within that so you could look at and dose of the medication whether it was given pre or post operatively look at various factors and you might find that you're able to explain high percentage of that heterogeneity and that can give you useful information about where the intervention may work better than and other situations and I think clinically that's that's the important thing to do although should be regarded as I'm so using your drug example you'll give us it'll tell us how important is the drug dose variation in the overall outcome exactly so if you've got results that vary by a lot and you then put drug dose into that as an independent variable if you've got a nice significant regression thing there you might be able to infer that actually using this medication a higher dosage is going to make it more effective than using a lower dosage without having that information from the individual trials themselves and I think these analysis should be regarded as as hypothesis generating and clearly should be pre stated in a protocol which again should accompany most reviews simply because what you don't want is people go in fishing for lots of different differences looking at ludacris variables and finding one that's statistics significant and you end up with a lot of false positives and it just creates a problem so when you're planning your review think carefully about things that could affect the results in each individual trial and pre state what you're going to do before so again when you're reading a match ours is a very very clear plan that's made before the actual analysis is done is another marker of quality of the of the the meta-analysis are you reading exactly we'll discuss that a bit later when we come to the stages of a systematic review and meta-analysis so what I've got here is an example of a meta regression plot it's sometimes called a bubble plot for a bit more obvious reasons than the forest plots so what we have here is along the x-axis is we have our independent variable so this is the thing we think that might change the results from each individual trial so the dose example use previously would go on this axis and then in this particular example we're looking at post-operative analgesics and we're looking at the amount of morphine used in the control group as a marker for how painful the procedure is along the y-axis here we have the mean difference or more correctly the difference in means from each individual trial so that's the difference in morphine consumption in the first 24 hours between the intervention and the control group and what we can see is quite a nice relationship between so the amount of morphine consumed by the control group in each individual study determined the degree of reduction in each individual trial so for a study where participants we use in about 90 milligrams of morphine there was a 40 milligram reduction whereas we can look a bit further up here and if they were using about 10 milligrams then they'd only get about a 5 reduction there and what that can reveal is quite important relationships or situations where that particular agent might be more effective this can reveal quite nice relationships it's not definitive but it's it's an extra thing you can get from meta-analysis and it's hypothesis-generating and then you may want to test these in further trials so the dosed example you could then do large clinical trials where you compare different dose subgroups and see whether that makes them any different and just to note the size of the bubbles on this plotter related to the percentage weight essentially they have within the meta-analysis so the largest studies will appear as larger bubbles right so we've touched a little bit about different models when it comes to heterogeneity and I just wanted to clear up a quite common misconception within meta analyses so the models essentially just explained the statistical assumptions of the pooling of results that you undertaken so there's two main types of models that you would use the first one being the fixed effects model now the fixed effects model assumes that you're assessing only one true underlying effect so if you had lots of studies that were conducted in exactly the same population exactly the same dose of the medication and outcomes are measured in the same way in all trials with inducted in the same way then you could use the fixed effects model but it's actually quite rare and meta-analyses in actual fact more commonly is the random effects model and that random effects model assumes that you're measuring different underlying effects in the trials you including and it's a more common situation and again the random effects model will penalize confidence intervals in the presence of high statistical heterogeneity try to touch on a bit of an example of that earlier so if you've got a lot of statistical heterogeneity then the random effects model will broaden the confidence intervals to reflect that uncertainty and quite often review authors will state that their choice of model was made on the basis of the I squared value but it should never be made on the basis of the I squared model so the thing you'll commonly see within publications is we found evidence of statistical heterogeneity I squared of seventy we therefore use the random effects model but in actual fact the use of the model depends on the underlying assumptions of the model which we've stated here rather than basing it purely on the I squared value so the next thing I want to talk about is precision so precision merely relates to the width of the confidence intervals now if you've got a meta-analysis of very few studies or particularly for dichotomous outcomes we've looked a post-operative vomiting that would be an example either yes or no then a lack of precision can create uncertainty within the results so for example if we had a meta-analysis that had a risk ratio of two and the confidence interval was from naught point four one to nine point seven one we can see that there's a broad range of results that could be encompassed within that risk ratio and therefore we can't really be that confident about the results we've got from our meta-analysis because those results range from a 60% reduction to a nine times increase which is a massive variation in the results and although results closer to a risk ratio of two or more likely it's still so spread out that we can't really be reliable now if we look at the opposing example if a confidence interval ranges from one point eight two point two with a risk ratio of two we can be reasonably confident that the results are close to our overall effect we found in the meta-analysis and just as a general rule if you've got more participants and less variants for continuous outcomes or lots of heterogeneity less heterogeneity then your results are going to be more precise hmm so if this goes across warm then the the effect can either you can either make it worse or better and so you're not going to have much confidence in that because it could have gone either way and so it's either need to comfort these the confidence that either or both ends going more than one or both hands even less than one then that will give you all you know everything's heading in the same direction so yeah exactly so we can see from the first example that that not 0.41 if something can either make things better or worse we don't really have much certainty in what that interventions going to do so you would treat that with with a lot of caution and remember what we said when we looked at the forest plot because that crosses one so anything that crosses a risk ratio of one would be equivalent to something that's not statistically significant so therefore we couldn't have much confidence okay so because gosh wanted just a quick eyeball if the confidence applause go across the number one yeah then we don't have a lot of confidence in that study yet direct ISM or any change yeah he's not making anything different so it would be one for risk ratios and odds ratios for continuous outcomes like mean difference in means it would be across the zero zero yeah that's right so finally I'm just going to talk a bit about publication bias so publication bias is due to the fact that studies are published based on their results being statistically significant and they're also published so they're more likely to be published and they're published faster than studies with negative results and what this does is it creates bias because you get a pool of studies that aren't representative of all of the studies performed within the area and this creates a lot of buyers there's been quite large studies done in the past that have shown that the main reason for differences between meta-analysis and large randomized control trials is because of publication bias now the best way to avoid this is when you're doing your search strategy is to not only look at your published databases like medline and n base and central and things like that you want to look for unpublished data and the best way to do this is a as we've alluded to previously is look at clinical trial databases conference proceedings are quite good because they'll often have studies that may be just published in abstract form or that haven't reached the final publication stage yet and you can also look at what are called grey literature databases which look at things like cc's and various other what's called grey literature and that that's the best way to try and deal with publication bias advice because you know these things aren't very interesting to read yeah and from a journal point of view the problem I'm going to get cited there's no difference between this and this it's probably not the exciting thing here's a new treatment for and it's titled isn't so with impact factor Drive except there's not a lot not a great deal in it for journals although exactly I think the reasons are quite multifaceted and they do come from both journals so again like you said journals are less keen to publish things that show negative results because they're not Cedar's as interesting or excitable as studies that have got exciting significant results and the other aspect is I think a lot of the time researchers you know they'll do their big study they'll find that there's no difference between the groups and don't be like I don't want to publish that and it's called the file drawer problem because researchers will often just chuck it in the file drawer and not bother to publish it rewarded for positive outcomes I mean I suppose doing a big thing of course it's quite controversially as pharmaceutical trials so what about farmers who do a companies don't want to publish the trial that says that their death what they thought was an innovative new drug is completely ineffective exactly and there's millions of dollars developing it and then suddenly something says it's you made a novice party and I think that's going to come out with there's Drive at the moment for I think it's Tamiflu so they're trying to push to get the pharmaceutical company to release all of the data regarding Tamiflu because there's a worry that because of publication bias the effects of Tamiflu have been a Resta mated and on that basis the government purchased hundreds of millions of drugs that may not be as effective as they originally thought to be and I think this is allegedly at the moment yeah it's likely would like to have that the words mothers and then as well as trying to avoid publication bias which is by far the best way of doing it and by far the least likely to be done within the meta-analysis and particularly looked at this recently looked at about 100 and different reviews and found that less than 10 percent look at clinical trial databases less than 10 percent look at conference proceedings and about 1 percent look at great literature sources so it's very poorly done out there so it's it but it's an important thing to bear in mind it's the only way you can actually deal with so again as a quality indicator you're reading a systematic review mesh analysis and they talk about examining conference proceedings and other and then gray-gray date or another and other areas so there's an exhaustive search yes that's correct but if you're actually conducting a meta-analysis yourself and you've tried to avoid publication bias as much as possible you can still try and identify publication bias by using something called funnel plots which we'll look at at the next slide there are also quantitative tests which will give you a p-value as well because everybody loves those things exactly so here we've got an example of a funnel plots so if we just explain what the different axes are along the bottom you've got your effect estimate in this case is our risk ratio and on this axis we've got essentially a measure of our precision we've got the standard error here but it's on a reverse scale so what that means is our larger more precise studies at the top of the funnel and a smaller less precise studies or at the bottom now the theory goes in that the smaller the trial you have the more likely you are to attain extreme results either side of the things and the more precise the study the closer it should converge on the mean effect which is our dotted line here now this is an example of a symmetrical funnel plot and it should demonstrate this shape here so you should see a nice inverted funnel like that now there is other causes for an asymmetric funnel plot which we will look at at the next slide but this is an example where you could be reasonably confident that there's no evidence of publication bias because you've got a nice serving vessel funnel exactly and here we've got another example of a funnel plant at the bottom here we have our effect estimate which is a different demeaned in this case but the y-axis is the same so again that's just a measure of how large the trial is what we can see what the problem is yes so we can see that the larger more precise studies tend to converge a bit more on this mean effect in the meta-analysis which again is this dotted blue line but we can see on this side we've got quite a few studies here that are smaller and less precise but on the other side we've seem to be missing quite a few studies now these studies would all fall into the negative category because their difference in means is above zero so what we could infer from this what we call asymmetric funnel plots is that there may be publication bias simply because these negative studies don't appear to exist on the other side of the axis so this is something to look for in meta-analysis and gives you a good indication of whether there may be publication bias present finally I just wanted to discuss a bit about the stages of a systematic review because like I've said for surgical trainees wanting to get involved in research I think it's a good Avenue to go down because it's it's very flexible and you can easily fit it along your clinical time and dip in and out of it as you please but it doesn't cost anything it doesn't cost anything next time exactly exactly and it's a good way to get involved in research for people who are interested how do you do a Cochrane and lots of training courses is our because he's obviously quite a complicated thing to get involved with yeah I think that so locally the University of Nottingham the I think the schizophrenia Cochrane group do a training session but I think lots of groups around the country ruin training sessions on how to actually perform a systematic review and they're definitely worth go into yeah we've already mentioned some of these stages but the first step is you have to decide what topic you want to do it on and what this might come from your own clinical practice you might identify a gap in knowledge or you might think oh I wonder what the evidence is on that and on that basis you would probably initially see if there's another systematic review metalness on the subject but if there isn't you would then carefully construct your clinical question around the Pico format so your participants your interventions your comparisons and the outcomes you want to look at you would then register the review on the prospero website which is a website run by the University of York where you can register you review and like we've discussed before this is important so you don't start changing outcomes and inclusion and exclusion criteria as you do in the review and this reduces bias in the review process you then want to conduct a systematic search of both published and unpublished studies and we've discussed the importance of searching for unpublished studies within that study search and like I said that's an area that doesn't tend to be done as well you then from your list of studies that you've identified from these sources want to decide which to include and exclude based on your inclusion and exclusion criteria from your Pico format question and ideally you want to do this with two people independently because that again reduces bias you then want to conduct your risk of bias ideally using the copy and risk of bias tool and again doing this independently with someone else reduces bias the analysis part so this is the bit that's initially a bit scary and where I would recommend teaming up with someone who does have experience of systematic reviews and meta-analyses because it's quite easy to think you can because the review manager software from Cochrane is actually freely available to download it so it's quite easy to obtain and but it's quite tempting to just chuck all your numbers in there and see what comes out and then try and write something on the basis of that but hopefully during this presentation we've discussed some of the pitfalls and problems and the fact that there needs to be a little bit more thought on how you're going to approach the analysis and how are you going to investigate publication bias and at regenitive we discussed via meta-regression so that's where I'd advise teaming up with someone until you're confident in that area and then the final stage would be to write it up and hopefully get published somewhere after all that work you'd hope so exactly that's the that's the most frustrating part so just a bit of a summary about what we've discussed so we've discussed about the advantages of meta analyses but I just wanted to highlight that there are many limitations and the in actual fact meta analyses are very poor predictors of results in large randomized control trials and a useful way to think about these limitations is by the grade system which we've discussed and which we've also mentioned should be present within Cochrane abstracts and is an important thing to take into account when you're looking at the results of of a meta analysis so we've looked at forest plots and these can be used to identify both heterogeneity by looking at whether the confidence intervals able app and also imprecision by the width of the confidence interval funnel plots can be used to identify publication bias but be aware that other causes of film apply asymmetry exist and finally bubble plots can be used to investigate heterogeneity which could generate new clinical questions and interesting findings that warrant further research okay well that's really really good practice it's very very clear I think some it's a really poorly understood topic just just the alien summary so they are you sitting there in an environment you know you're reading something all or in the exam and we don't need to make a decision about how good the meta analysis that you're reading is so just give us a quick step by step things that you would see us things to look for when you only look in that paper in new you got and then over 30 seconds just to have a quick look at it what would you look for I would look at the method section because that's going to tell you how the review was performed you want to make sure that like we said there's a good wide-ranging search strategy so lots of published databases as well as all of that and published data we looked at you then want to see how the review was conducted so you want to make sure that two people of formal risk of bias and that two people have independently assessed it is against inclusion and exclusion criteria and there's something in there about resolving differences yeah so if there's quite often if there's differences between two authors they might resolve it by discussion or they might consult a third or fourth author and sometimes even three authors get together and decide on these things and that helps to reduce by so that's another good quality marker you want to make sure that the studies assess publication bias so you want to see whether they've used formal plots and ideally quantitative test such as something like acres linear regression test you want to look at whether they've put in their analysis all of the limitations we've discussed about whether you can determine whether there's imprecision in the results and whether the results apply directly to the patients that they're saying it applies to essentially not just the patient's you've got you want to make sure that that marries up together and also if there's any evidence of heterogeneity have they investigated this have they looked for reasons why results differ from each other and if they've been unable to explain that then that again casts further doubt over the conclusion to the meta-analysis okay so that's good one and then we're looking at results have a look at how the other forest plot stacks up this would be pretty much on top of each other yeah how big the diamond the lozenges so the smaller the better and then also have a look at the I squared and again the smaller the better of the I square statistic exactly yeah brilliant well I think I understand them though well thank you very much chef dr. Brett Dorman for taking us through that thank you thank you for listening to another podcast brought to you by School of surgery remember you can follow us on Facebook at school of surgery on iTunes on podomatic at school of surgery podomatic calm and finally by searching School of surgery on YouTube thank you very much and see you next time

Channel: school of surgery

Views: 45,766

Keywords: meta analysis

Id: Wt79y0YutVc

Length: 62min 15sec (3735 seconds)

Published: Thu Dec 15 2016