Systematic Review Articles

What is a systematic review?

Given the continuous rise in the number of studies being published in the field of health sciences, it becomes quite difficult, if not impossible, for busy clinicians and researchers to stay updated with the vast amount of literature. Consequently, reviews that provide a concise summary of the results from different intervention trials serve as an incredibly effective approach to obtain a clear understanding of what interventions are effective and which ones are not.

Systematic reviews and traditional narrative reviews differ in several aspects. Narrative reviews are often descriptive in nature and lack a systematic search of the literature. They tend to focus on a selected subset of studies based on availability or author preference, which can introduce selection bias. Furthermore, narrative reviews can be confusing when similar studies yield different results and conclusions. On the other hand, systematic reviews involve a comprehensive plan and search strategy that is predetermined. Their objective is to minimize bias by identifying, evaluating, and synthesizing all relevant studies on a specific topic. Systematic reviews commonly incorporate meta-analysis, a statistical technique that combines data from multiple studies to produce a single quantitative estimate or summary effect size. Unlike traditional hypothesis testing, which primarily determines statistical significance (i.e., whether the intervention group differs from the control group), effect sizes provide information about the strength and magnitude of the relationship between variables, indicating the clinical significance or the size of the intervention effect. The specific type of effect size calculated depends on the outcome and intervention under examination, as well as the available data from published trials. Examples include odds ratios (OR), weighted/standardized mean differences (WMD, SMD), and relative risk or risk ratios (RR). Systematic reviews are typically published in academic forums, but there are also dedicated organizations and databases that promote and disseminate them.

Eight stages of a systematic review and meta-analysis

Researchers are suggested to follow some steps for conducting systematic reviews. Here is the list of stages:

1. Formulate the review question

The initial phase of conducting a systematic review involves several key steps. First, it is essential to define the review question, establish hypotheses, and formulate a concise and descriptive review title. A recommended format for the title is as follows: “Intervention for population with condition” (e.g., “Dialectical behavior therapy for adolescent females with borderline personality disorder”). Reviews published within the Cochrane Collaboration do not require specific identification in the title, but reviews published in other sources should indicate that they are systematic reviews and/or meta-analyses. If authors choose to conduct their review through the Cochrane Collaboration, they are required to register their title with the appropriate review group. This registration secures their topic and provides access to additional support from the Cochrane Collaboration, such as assistance in developing search strategies.

2. Define inclusion and exclusion criteria

The Cochrane Collaboration employs the PICO (or PICOC) acronym, which stands for population, intervention, comparison, outcomes (and context), as a useful tool to ensure that all essential components are determined before initiating a review. Authors should decide in advance on the specific population characteristics, age range, conditions, outcomes of interest, and types of interventions and control groups. It is crucial to provide operational definitions regarding the inclusion and exclusion criteria for study types (e.g., randomized controlled trials-RCTs only, RCTs and quasi-experimental designs, qualitative research), the minimum number of participants in each group, inclusion of published and unpublished studies, and any language restrictions. In the case of Cochrane Reviews, this information is organized, subjected to peer review, and published in a Protocol format initially. Once the review is completed, the Protocol is replaced by the full Review.

3. Develop search strategy and locate studies

During this stage, the assistance of a reference librarian can be highly beneficial in developing and conducting electronic searches. It is essential to create a comprehensive list of key terms, including MeSH (Medical Subject Headings) terms, related to each component of the PICO/PICOC framework. This ensures the identification of all relevant trials within the chosen topic area. For instance, if the target age range is 13–18-year-old females, search terms may encompass variations such as adolescents, teenagers, youth, female, women, girls, and so on. The goal in developing an optimal search strategy is to strike a balance between sensitivity (capturing a high proportion of relevant studies) and specificity (retrieving a low proportion of irrelevant studies). Typically, searches involve multiple relevant electronic databases. Additionally, checking the reference lists of articles, hand-searching key journals, posting requests on listservs, and reaching out to experts or key researchers in the field are also common approaches to supplement the search process.

4. Select studies

After retrieving and reviewing a comprehensive list of abstracts, studies that seem to meet the inclusion criteria are obtained and thoroughly reviewed in full. This reviewing process typically involves at least two reviewers to ensure inter-rater reliability. It is advisable for authors to maintain a log of all reviewed studies, documenting the reasons for their inclusion or exclusion. In some cases, it may be necessary to contact the authors of the studies to acquire missing information required for data pooling, such as means, standard deviations, or other relevant data. Translations of non-English studies may also be required if they meet the inclusion criteria.

5. Extract data

Creating and utilizing a straightforward data extraction form or table is a beneficial practice to systematically organize the information obtained from each reviewed study. This form or table can include details such as authors, publication year, number of participants, age range, study design, outcomes measured, and information regarding inclusion or exclusion criteria. It is crucial to have at least two reviewers involved in the data extraction process to ensure inter-rater reliability and minimize the risk of data entry errors. By following this approach, the systematic review maintains consistency and accuracy in extracting relevant information from the included studies.

6. Assess study quality

In recent years, there has been a growing emphasis on assessing the quality of individual randomized controlled trials (RCTs) included in systematic reviews. While there are brief checklists available, such as the 5-point Oxford Quality Rating Scale (Jadad et al., 1996), commonly used in Cochrane reviews, it primarily focuses on double-blinding, which is more relevant to drug trials than psychological or non-pharmacological interventions. To address this limitation, more comprehensive guidelines and standards have been recommended, such as the Consolidated Standards of Reporting Trials (CONSORT) statement (http://www.consort-statement.org/). CONSORT provides detailed guidelines for reporting RCTs and is widely recognized as a valuable resource for assessing the quality of RCTs. Additionally, there are articles available that offer recommendations for improving the quality of RCTs and meta-analyses specifically in the context of psychological interventions, such as the work by Uman et al. (2010). These resources contribute to enhancing the assessment of RCT quality and promoting rigorous research standards in the field of systematic reviews.

7. Analyze and interpret results

There are several statistical programs available for calculating effect sizes in meta-analyses, one of which is the Review Manager (RevMan) program endorsed by the Cochrane Collaboration. Effect sizes are typically reported with a 95% confidence interval (CI) range and are presented both numerically and graphically, often using forest plots. Forest plots visually represent each trial as a horizontal diamond shape, with the center of the diamond indicating the effect size (e.g., standardized mean difference, SMD), and the endpoints representing the upper and lower bounds of the confidence interval. These diamonds are plotted on a graph, with a center line denoting the zero mark. The left side of the graph (< zero) usually indicates favoring the treatment group, while the right side (> zero) favors the control group. At the bottom of the graph, there is a summary effect size diamond representing the pooled results of all individual studies. Ideally, the entire diamond (including the effect size and both anchors of the CI) should fall below zero, indicating that the intervention is more effective than the control.

Most statistical programs also calculate a measure of heterogeneity to assess the similarity of individual studies. Non-significant heterogeneity is preferred, indicating that the studies are sufficiently comparable for pooling. However, it is still possible to combine studies even in the presence of significant heterogeneity, although caution should be exercised in interpreting these results, and potential reasons for the heterogeneity should be explored.

The final step in the writing process involves summarizing the findings and providing recommendations for both clinical practice and further research. This includes identifying which interventions are efficacious, for whom they are effective, and under what conditions. Recommendations may also highlight areas, topics, or interventions that require additional research to fill existing knowledge gaps.

8. Disseminate findings

Reviews conducted through the Cochrane Collaboration are typically published in the Cochrane Database of Systematic Reviews, which tends to be comprehensive and detailed. However, it is also possible and encouraged to publish condensed versions of the review in other relevant academic journals, as long as it is clearly stated that they are abbreviated versions (e.g., Uman et al., 2008). In addition to academic publications, it is common to provide plain language summaries of the review for families and patients, making the findings accessible to a broader audience. There is an expectation that systematic reviews should be regularly updated to ensure they remain current and relevant. Participating in review updates or joining an established review team can be an excellent opportunity to engage in the systematic review process and contribute to ongoing research efforts.