Hi, this is Shady Attia, professor at Liege University in Belgium and I'm going to present to you today a Systematic review,
how to conduct them in the context of qualitative research methodologies. This is a presentation as part of a playlist
on qualitative research methodologies. And I invite you to subscribe, look at the previous videos
on qualitative research methods in general, in order to be able to contextualize the presentation. Before starting, I would like to introduce you the structure that I use when I start to group the different research methodologies. I classify research on the two main titles, quantitative research methodologies, where we are looking at modelling empirical research
or experimental research and qualitative research methodology. We are looking at systematic reviews, content analysis,
observations, in-depth interviews, focus group discussions, usability, case studies, sketching,
and so on and so forth. And in this presentation, I'm going to focus on systematic reviews as part or as a method that makes a part of qualitative research methodologies. Don't forget that some people tried to combine qualitative with quantitative
and in this case, we are talking about mixed research method. And in this sense, we need to combine some methods from the qualitative
approach with some methods from the quantitative approach. When we look at qualitative research methods,
we have two major groups of qualitative research methods. One is related to documents, and we call them desk methods
and some are related to people, and for those, we call them population-based research methods. And research reviews are part of the desk
or the document-based approaches. The audience of this presentation is mainly researchers
who are preparing a research or thesis dissertation or conference or journal publication, and they're looking to do a systematic review
in a specific area or a specific domain. The goal here is to enhance your capacity to conceptualize design,
and conduct a systematic literature review-based research. And I'm looking to answer the following questions:
how to conduct a systematic review? What are the methodologies of this particular approach?
What is the methodology to ensure the validity of my research? And what are the specific advices for each step to help reaching my objective? So, these are the objectives of the presentation.
I'm going to explore with you an introduction. What is the protocol for systematic reviews? How I select my studies?
How I extract data? ? How I can make sure that there is a quality and validation of my approach,
how can I synthesize my data, and summarize it? And finally, how to write my report and some learned lesson to conclude with? So, let's start with the introduction. What is systematic review? The systematic review covers long period trend with
a large geographical scope, it's time and cost effective, and it looks at high quality open access data. So, this is the general definition. And it is part of desk research, where we have different other methods
that are entitled as - Desk research under the qualitative research methodology. Systematic reviews appear as the first desk research method. Then, we have the content analysis, and we have historical research. So, these are methods or approaches that we use when we do desk research. For sure, they involve something in common all three of them,
that we need to do coding and classification, meaning, that we need to find concepts that are found
in our documents or manuscripts. And we need to code them, group them, classify them, categorize them
and try to write to provide a structure or an overall understanding of the different reviewed documents. And for sure, this is part of it when we look at desk research. Okay, when do we do systematic reviews?
And what are the characteristics of systematic review? Many empirical studies, when many empirical studies have been published,
then it's time to conduct a systematic review, meaning that I need to have before starting my systematic reviews,
a bunch or a group of documents that were published already in the area where I want to do a review. We need to answer a specific question. And we need to try to group these researchers
and try to get them answered through combining these papers or the results in these papers to answer a global question
that connects them all. And we need to define our research strategy for the articles. So, we need to have criteria of inclusion and exclusion to make sure
what will be the pool of our selection and the documents that we will process
and we need to submit a clear protocol so that it can be easily reproducible, and this is part of it. And we need to conduct a quality assessment of our studies
to make sure that the studies that we are selecting that they have a certain quality. And finally, we do the synthesis of our finding for the reviewed documents. This is a table that compares the classical literature review
with systematic review. And just to let you know, we have a presentation on my channel
on literature review, the classical literature review, but this presentation here today is focusing mainly on systematic reviews. And as you can see, there are serious differences. When it comes to the focus of the review, literature reviews introduce context
and current thinking often without a specific question, and it generally covers several aspects and topics. However, when we do systematic reviews, we use a precise question
to produce evidence to underpin a piece of research. So, it's a standalone piece of research, and it should be conducted prior
to understanding further research, particularly in higher degree thesis. So, this is the first difference when it comes on the focus, when it comes on the difference between the methods of literature review
and systematic review. Under the literature review; we find papers through a fairly random process
usually searching on a few databases, use of a grey literature, common, but not usually systematic.
So, we are not doing a systematic approach. When we do systematic reviews, we are searching
for several specified databases, using precise search terms. A similar systematic search for grey literature seems sometimes included,
depending on the question. And this is also important to distinguish both of them. When it comes to the methods of data extraction, for literature review, we are looking at papers that are read,
take-home messages used in the review. However, in systematic reviews, we are looking at data extraction tools
that are professional, that are used to identify precise pieces of information,
two or more researchers can undertake the data extraction. So, it's not only an individual work,
also a very important factor that distinguish literature review from systematic review are the number of papers included in the review. It's very easy to go up 250 papers or more when we do a literature review. However, when you do a systematic review, the systematic review
should be always less than 50 papers, even often fewer than 10. Because here I'm not going to review everything. I'm going to select very important and relevant and high-quality papers. 10 are even enough, 20 are fine. And I go from there to do a systematic review in the content of these papers. So, it's very important here to distinguish this difference,
many people misunderstand it. And when it comes to the methods for data analysis, literature review,
we look at writer interpretation, to the meaning of the results. While for systematic reviews, we have recognized referenced methods
for data analysis, that includes analysis of methods, the rigor of conduct of research,
strength of the evidence, and so on. So, this is an important part of it. So, this is the main thing, when we look at publication literature reviews. They can be more a form of reports, some public journal publication, while here we are most suitable for technical publication here,
those are more review papers. So, we look at review journals. While here we can publish it without having,
in general, core specialization journals. So, these are the major differences between
literature review and systematic reviews. And if you need to know more about literature review,
there is a video called literature review on my channel. What are the steps of a systematic review? First of all, we look at planning. So, we have to specify our research question, we have to develop a protocol. After that, we go with our systematic review execution.
We do search and select studies based on defined criteria. We extract the data, we start to assess the quality of our papers and the data. And then we start the synthesis. And finally, we come up to the review report. Or the systematic review report where we start to write a report
that summarizes our findings, and we can continuously update it. So, these are the steps for doing a systematic review. What are the protocols that should be used when I do a systematic review? We need to first of all, before starting to formulate the problem,
review the literature to make sure it has not been done before. I need to form a hypothesis and give the review a title. And in this sense, I need to describe what I'm going to do. Just to give you an idea, what is a protocol,
when it comes to systematic review? A protocol is something that describes the steps
that I will follow in a systematic review. And it looks at how and where studies are going to be found. What will be the criteria for the selection of the studies? What are the methods used for data analysis and validity assessment? What will be the time available to conduct it? And why I'm going to do this? So when we talk to why, the protocol should answer the question,
what it should provide information that guides the reviewer during his review, to explain why he's doing things, ensure the replicability,
so it's a formal approach, in general. And this is an example of a protocol that I would like to share with you where we can see that a researcher here wanted to focus on a specific topic. This researcher defined the purpose of his systematic review. He defined the research questions in the protocol setting,
this is a template for a protocol, the keywords are defined, the synonyms are defined,
the sources of libraries are identified. And as you can see, the search strings that will be searched
in the research engine are already defined, and there are inclusion criteria and there are exclusion criteria. So, this is a simple example of a protocol when I do a systematic review. And when you are publishing a paper that is on a systematic review,
you must share this protocol with the reviewer. So that you show more transparency regarding your approach
and while you are developing your methodology or your protocol, you should validate it with peer review through your peers
or through your supervisor. So, this is an example of a protocol for a systematic review. And as you can see, these all elements need to be answered. Now some tips when it comes to the sources and searching. Search terms used for literature should be clearly described
with information on the relevance to the research question. Make a reference list of reviews captured during your research. Mention searches with and without the language restriction
to estimate in general, the number of papers that will be excluded. So that when you start to say, I'm only going to review 10 papers,
15 papers, 20 papers, and do not forget, we are here in this systematic review lecture,
we are not in the literature review lecture. We are focusing only on a little number of high-quality papers
that are commonly published on a topic. And we are trying to find something common between them to confirm
or to answer a specific question. And therefore, we need to say,
Okay, what were our exclusion and inclusion criteria? How many papers did we exclude, and those will be much, much more
than the paper that we included in our review? Now, how can we select our studies? And how can I select my papers or the documents or the manuscripts
that I will use in my systematic review? You need to screen first the titles and the abstract. And you will be able to reject those documents,
which do not fulfill your inclusion criteria. So that's simple to start with. Your inclusion criteria can also be
about the type of publication preferred. So, you can from the beginning, exclude reports, you can exclude papers
that are not published in key one quarter or high-quality published papers, for example. You make a table of papers that you will read in detail,
and those that you will be rejecting. And in general, this is an example that you have to keep the record,
why it was included or rejected. So, in general, you can come up with a table like that,
you start to define the number, the publication, how many times it was cited, and some observations, reasons,
why did you include it or why you reject it? So, this is in the early phase, when you are in the process
of having several papers, and you are selecting among them. So, you just start to compile your table, and this time you add some comments,
you just read the title and the abstract. And later on, you can refine your table until you make your final selection. Assuming I have now selected my review documents,
how do I do data extraction? It's necessary to summarize, the findings of the reviewed studies,
in general, in a form of a table. So, you need to create a table and you start to look at the references,
the study parameter, what was the focus? What were the gaps? And what are the findings? And the data extraction is done by at least two reviewers
in order to establish inter-rater reliability. And this is very important here to talk about
because this is part of the quality of your systematic review. It's not only about one researcher, it's always recommended
to have a second reviewer or a second researcher. He or she should join you to make sure that while you are classifying
and grouping the selected documents in order to come up with an interpretation, this is done by two persons. So, there are four eyes looking at the same information,
trying to find a consensus, and make sure that interpretation and understanding are validated. This is an example of a literature review matrix. This is an example of the matrix we use in my lab. Normally, I ask reviewers or my students to list the reference. What are the study parameters that were investigated?
What was the focus of the study? What are the gaps in the study? And what are the findings in the studies? And based on that, we can proceed more and start to compare
the different documents in order to come up with a general understanding and start to answer our research question. Moving almost to the end, well, how to assure the quality
and validation of my systematic review? As I told you before, in different presentations, we always need to look when we do qualitative research to the external validity
and to the internal validity. So, we need to ask our question, do my research, think about reliability? Did I think about the validity in the same way
that I conducted for different researchers? Yes. How to make sure that I assure that you look at:
Are my results able to be generalized? Is it transferable? Is it dependent on other factors or not? So, this is the external validity. And when it comes to internal validity, I have to ask myself,
do I have credibility in my approach of selection? This is when it comes to having a second colleague,
or a secondary searcher looking with you with a different eye to assure that your results are credible. When you do a comparison when you look at different reviews
and confirm the validity. Here, I'm looking at the intention versus reality.
And this is called internal validity. So, we need to be always worried about the validity of our approach. And as I told you, again, systematic reviews are subjective.
But we try here to avoid bias. And in order to avoid bias, we look at internal validity and external validity. And I'm going to show you in the following slides some methods
and approaches to improve the validity of your systematic review. Well, how to validate my systematic review? The original manuscripts or data, are they created for a different purpose? So, here this makes a challenge because the documents
that you're reviewing were not made for the review. So, this is a challenge. And this might influence your data collection, your results,
and the access to the data is sometimes restricted. These are all some challenges. And data can also be quickly outdated, you can find now when you're doing
a review on a topic, and you selected 15 or 10 studies, but there are new studies coming out that are not confirming
or negating what was done in the previous one. So, this can also be influential. So, when we look at reliability, we look at data that can be influenced
during collection or coding, or processing. And you need to make sure that you are doing a systematic review
with high reliability. And your coding process is very rational and is done
with the second researcher who can bring up more reliability in the process. There are several methods evaluate the quality of your systematic review, the quality evaluation of review has an impact on the conclusion. So it is very important to avoid bias and during the process
of the systematic review, always look at quality evaluation. How to do that? There are different types of bias,
maybe I have a bias with my publication. I'm looking at publications that are not matching the content,
maybe I have a bias with location, and only selecting publications that are coming from a specific country,
or a region or a hospital, or a specific place. So, I have to make sure that I have diversity in the sense.
Maybe I have a citation bias. Maybe because this paper is cited very high, I think it's a good paper. But in reality, people found in this paper, a lot of trendy topics. So, they cited the paper. But in reality, when it comes to the seriousness
and the solidity, and the validity of the test and experiments done in the study, it's not so good.
Maybe I have a language bias. So, I'm explaining the studies because I'm only accessing
English speaking documents, and I neglected totally other words, like the French-speaking community,
or the Spanish speaking community, or a Chinese community, and so on. So, I can have also a bias with language. And I can also have outcome reporting bias like I have fixations
and I am driven from the beginning to believe that there is a common correlation or no correlation. So, these are different biases, I need to be aware of them. I need to investigate them and address them
while I'm developing my methodology. And I need to discuss how I avoided these biases. Now, how can I avoid the risk of bias in my systematic review? I need to first create a risk bias table. So, I put here elements where the study is at risk for bias
and I need to start to assess them and I have to show how these studies are performing
for each particular element. So, I can apply weight in a regression equation,
and I use visual methods for evaluation. So, I can show how studies are different and might affect the results. And I will show you an example of that later on. I can segregate studies in real building settings,
for example, and experimental settings. This is in my sector of architecture, engineering, and construction. And I can perform a sensitivity analysis. So, these are all things that I can do. And I will explain the sensitivity analysis because we can redo the analysis
by excluding studies with a higher risk of bias. So, this is a table where I'm talking about. When I talked about doing a risk bias table, this is how it looks. You can see I'm listing the different papers based on their main author
and the year of publication. And I'm starting to do a risk assessment based on series of criteria. What about the study design? Is it clear? Is it mentioned in this paper or not? What about the random sequence generation,
the blinding outcome assessment, the withdrawal and drop out, these are all criteria that I added in order to make sure
that they are relevant to the quality of the reviewed papers. And I start just to come up with the plus and the minus and the question mark,
in order to start to classify or qualitatively assess these papers. And for here with such a small paper table,
I can show that I did a risk assessment. I eliminated the studies with the most Red and Yellow.
And then I increase the quality and reduced my review pool of documents. However, I made sure it's a very high-quality selection of papers. So, this is an example of a risk bias table and I advise you all
when you are conducting a systematic review, to include your risk bias table in your methodology in order to show
that you're transparent and to make sure that you provide a better internal validity of your results. Now, what is sensitivity analysis? This is an additional measure of validation
and quality assurance for my systematic review, you first do the analysis with all the studies included
and then you withdraw studies based on some elements. So, how to do that? You can have questions that you can ask yourself like
what will be the results if I exclude studies at high risk of bias? What happens if I change inclusion criteria? What if unpublished works are included? And based on these scenarios, you start to test and tweak and go back visit
your assumptions and your overall understanding and finally, if the conclusion remains the same, no matter which studies fairly small number or each time you can modify add one or two or three is removed,
then the conclusion is robust. So, this is called sensitivity. How sensitive is my conclusion that I generated
for my systematic review influenced by adding or taking out studies to the pool of selected documents? And by that, I can also assure another measure
to make sure how far is robust my findings and how they can be generalized,
and here we are talking about external validity. So, these are examples. Now we are done with our analysis of how to synthesize all this information. Normally, we summarize the results of each study and bring them together. And there are two approaches depending on the question
and type of information. Either we do a narrative synthesis,
or we do a statistical synthesis with meta-analysis. Those are the two approaches to report our findings,
the general framework that we try to do with the narrative approach that we develop a synthesis of the finding included studies. And then we define the concept with a framework and we start. Okay, based on our systematic review of 10 papers or 12 or 15 papers,
we came up with that. And narratively, we go chronologically or we go,
narratively explaining what we found in common. And we explore some relationships were between studies,
and we answer our questions in a narrative way based on our findings. But if we do the other approach of meta-analysis, then the results of each study
are combined statistically to improve power and precision in estimating the intervention effect. So, I start to go statistically, and I start to take the samples
and take the results, combine them together and start to come up with an overall analytical statistical analysis
for all samples used to say - okay, if there is a correlation between a certain factor with another one,
with a certain food, with a certain sickness and so on. Methods used here, the meta-analysis are used to combine results
that depend on the type of outcome assessed. And two main statistical models are used here, either a fixed effect model
or a random effect model. And these studies are done a lot in the medical world
also in the building sector. They do it in thermal comfort that I go combine different results
from different countries. For example, the thermal comfort in Vietnam, the thermal comfort in Japan,
the thermal comfort in China. And I start to combine these samples and then do a meta-analysis
to see if the common findings that were found in those different countries are coherent,
giving the same direction or they are not or contradicting, for example. So, this is one of the types that they can conduct when I do systematic reviews. And I investigate the reasons for any inconsistency between my samples. After that, I start writing my report. And when it comes to writing my reports, it's a classical report,
I need to first do a report for my systematic review. And from this report, I can develop my publication for the journal. So, when you do systematic reports, systematic reviews,
you need to produce first, the report with a classical structure, the title, the content, the executive summary,
the IMRED structure of background, objective method, and results. The body; what are the review question,
what are the review methods that you used? What are the results of your review? The discussion, the conclusion, and so on. So, this is a report that needs to be produced
at the end of your systematic review. And from this report; if you want to publish,
you can create your publication based on your report. So, the report can be more comprehensive, and your publication
will be more specific focusing on your methodology and your outcomes. This ends it up. I'm now going to wrap up with some lessons learned
when it comes to systematic review. A systematic review is a detailed review of existing literature
on a precise topic to address a specific question, it is not a literature review, it's a selection of high-quality documents. And a protocol is always necessary to define the study design,
the goals, and the outcomes. And this needs to be shared by the reviewers. A systematic review can be validated without using
quality assurance techniques to avoid bias. And we talked to you about sensitivity analysis, the risk table,
and those are the ways that make you sure that you have a higher internal validity, and external validity of your review. And systematic reviews can be reported through a narrative analysis
or a meta-analysis, depending on your choices. Most systematic reviews are following a narrative analysis approach
where we just describe what we found in common or not in common? While in the meta-analysis, they involve that you take the information and go statistical step and actually go more into another approach,
which is a quantitative approach. Also, systematic reviews are very useful. Or you can learn a lot when you start to look at journal papers that did already
a systematic review and you learn from them. So, this will be a very interesting approach to make sure that you are following
best practices, especially if you do it for the first time. So that's it for the presentation. I would like to thank you. This was the presentation about the systematic review. Don't hesitate to like the video if you find it interesting
or comment or provide feedback. In general, this is one of the methods of qualitative research
and there will be the following videos on qualitative research methods. Thank you for your attention!
