Systematic Reviews and other Knowledge Syntheses

Introduction

A systematic review is a type of knowledge synthesis.  A knowledge synthesis is 'the contextualization and integration of research findings of individual research studies within the larger body of knowledge on the topic.  A synthesis must be reproducible and transparent in its methods, using quantitative and/or qualitative methods.' (Grimshaw, 2010)

Steps

  • Define the research question
  • Define the eligibility criteria for included studies
  • Identify (all) potentially eligible studies
  • Select the eligible studies
  • Assemble the data set
    • Data extraction
    • Quality appraisal (not all knowledge syntheses include appraisal)
  • Synthesize the data (quantitative and/or qualitative)
  • Report the results

Types of reviews

Name Definition Strength Weakness
Literature Review Generic term: published materials that provide examination of recent or current literature. Can cover wide range of subjects at various levels of completeness and comprehensiveness. May include research findings Seeks to identify what has been accomplished previously, allowing for consolidation, for building on previous work, for summation, for avoiding duplication and for identifying omissions or gaps Lacks an explicit intent to maximize scope or analyse data collected
Systematic Review Seeks  to systematically search for, appraise and synthesis research evidence, often adhering to guidelines on the conduct of a review Draw together all known knowledge on a topic area Restricting studies for inclusion to a single study design can limit this methodology to providing insights about effectiveness rather than seeking answers to more complex questions
Rapid Review Assessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing research Rigorous and explicit in method and thus systematic but make concessions to the breadth or depth of the process by limiting particular aspects of the systematic review process Limiting the time taken to search may result in publication bias, limiting appraisal or quality assessment may emphasize poorer quality research, lack of attention to synthesis may overlook inconsistencies or contradictions
Scoping Review Preliminary assessment of potential size and scope of available research literature. Aims to identify nature and extent of research evidence Systematic, transparent and replicable. Inform policymakers as to whether a full systematic review is needed Does not include a process of quality assessment. As such, findings cannot be used to recommend policy/practice
Integrative Review General review of the existing literature done as a 'systematic' process Inclusion of diverse data sources enhances a holistic understanding of the topic of interest Combining diverse data sources is complex and challenging
Realist Review Realist syntheses specifically attempt to provide ‘an explanatory analysis aimed at discerning what works for whom, in what circumstances, in what respects and how’ Tries to identify the underlying mechanisms, helps understand context of an intervention Requires expertise in program theory, not reproducible in the sense of a systematic review

References and Further Reading

  1. Grimshaw, J. (2010). CIHR. https://cihr-irsc.gc.ca/e/41382.html
  2. Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
  3. Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review - a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. https://doi.org/10.1258/1355819054308530
  4. Whittemore, R., & Knafl, K. (2005). The integrative review: Updated methodology. Journal of Advanced Nursing, 52(5), 546–553. https://doi.org/10.1111/j.1365-2648.2005.03621.x

What does it take to do a systematic review?

  • Time: On average, systematic reviews require 18 months of preparation.
  • A team: A systematic review can't be done alone! You need to work with subject experts to clarify issues related to the topic: librarians to develop comprehensive search strategies and identify appropriate databases; reviewers to screen abstracts and read the full text; a statistician who can assist with data analysis; and a project leader to coordinate and write the final report.
  • A clearly defined question: Clarify the key question(s) of you systematic review and the rationale for each question. Use the PICO, PECO or SPICE frameworks to identify key concepts or aspects of the question. Determine inclusion/exclusion criteria.
  • A written protocol: You need to write a protocol outlining the study methodology. The protocol should include the rationale for the systematic review, key questions broken into aspects or "PICO components", inclusion/exclusion criteria, literature searches for published/unpublished literature, data abstraction/data management, assessment of methodological quality of individual studies, data synthesis, and grading the evidence for each key question. Need help writing a protocol? 
  • A registered protocol: After you write the protocol, you should register it with PROSPERO, an International Prospective Register of Systematic Reviews. Registration is free and open to anyone undertaking systematic reviews of the effects of interventions and strategies to prevent, diagnose, treat, and monitor health conditions, for which there is a health related outcome.
  • Comprehensive literature searches: First, identify systematic reviews that may address your key questions. Then, identify appropriate databases and conduct comprehensive and detailed literature searches that can be documented and duplicated.
  • Citation management: You should have working knowledge of EndNote to help manage citations retrieved from literature searches.
  • Follow reporting guidelines: Use appropriate guidelines for reporting your review for publication.

Guides

  • Cochrane Collaboration Handbook
    The Cochrane Handbook for Systematic Reviews of Interventions is the official document that describes in detail the process of preparing and maintaining Cochrane systematic reviews on the effects of healthcare interventions. This is Version Version 6.1, 2020.
  • Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care
    Provides practical guidance for undertaking evidence synthesis based on a thorough understanding of systematic review methodology. Presents core principles of systematic reviews and highlights issues that are specific to reviews of clinical tests, public health interventions, adverse effects, and economic evaluations. The final chapter discusses incorporation of qualitative research in or alongside effectiveness reviews.
  • Finding What Works in Health Care: Standards for Systematic Reviews (IOM)
    IOM's (2011) standards address the entire systematic review process, from locating, screening, and selecting studies for the review, to synthesizing the findings and assessing the overall quality of the body of evidence, to producing the final review report. Includes a link to the IOM Standards for Systematic Reviews.
  • JBI Manual for Evidence Synthesis
  • Methods Guide for Effectiveness and Comparative Effectiveness Reviews (AHRQ) This guide was developed to improve the transparency, consistency, and scientific rigor of those working on Comparative Effectiveness Reviews.

Repositories

  • Systematic Review Data Repository (SRDR)
    Free; requires login. This systematic review repository also acts as a data extraction tool.
  • Campbell Collaboration
    Guidelines for producing a Campbell Systematic Review. The Campbell Collaboration is an international research network that produces systematic reviews of the effects of social interventions
  • 3IE Impact
    International Initiative for Impact Evaluation offers a database of systematic reviews on impact evaluations and has methods information for conducting your own.
  • The EPPI-Centre is committed to informing policy and professional practice with sound evidence
  • Cochrane Register of Studies
    Is meta-register or repository for Specialised Registers from all Cochrane Groups. It is a management tool specifically designed for storing, maintaining and developing registers.
  • Collaboration for Environmental Evidence

Tools

A review plan or protocol

A major cause of bias in a systematic review is answering a different question to that being originally asked. This is why it is important to develop a review plan or protocol.

The benefits of having a protocol before the beginning of a review:

  • relate to the validity and merit of a research process that reduces risk of bias
  • promotes a systematic rather than ad hoc approach to the review process
  • facilitates communication with others and promotes consistency between review team members
  • support the reliability and usefulness of reviews to health professionals

Steps for a protocol and conducting a systematic review

  • Check existing reviews/protocols - a decision needs to be made whether to abort the review or amend the question.
  • Formulate a specific question - PICO (for a quantitative review) - PICo (for a qualitative review)
  • Inclusion criteria – puts the question into operation (population, intervention or comparison, outcome, study design, language, publication)
  • Exclusion criteria for not including certain studies e.g. specific populations, language, setting
  • Eligibility criteria for including studies– qualitative and quantitative (study design, language, date of publication, duplicate data)
  • Search strategy - explicit and reproducible (must include grey literature)
  • Critical appraisal - Assess risk of bias in individual studies
  • Synthesis of results (interpret/analyse results/determining the applicability of results
  • Report - comprehensive report on all steps of the systematic review and presents results

Where to register your protocol?

Once you've developed your protocol, register it. This is a basic requirement for a trial or systematic review, to check if your study has already been done by someone else, and whether your proposed review is necessary.

  • PROSPERO: international prospective register of systematic review
  • OSF

Search

Search is a balance between sensitivity and specificity (or precision).

  • Sensitivity is the number of relevant results identified divided by the total number of relevant results in existence.
    • Capture most relevant results
    • Capture more irrelevant results
    • Spend more time filtering/screening
  • Specificity (or precision) is the number of relevant results identified divided by the total number of results identified.
    • Capture some relevant results
    • Capture fewer irrelevant results
    • Risk missing relevant results

For systematic and scoping reviews the goal is a comprehensive search, maximizing sensitivity.

Where to search

  • Databases by Subject
  • Grey Literature
  • "Hand searching" key journals - If there are key journals relevant to you topic you may want to search through them specifically (print or electronic)
  • Expert referrals for key studies

How to search

  • Techniques
  • Process
    • Developing a search strategy 
      • There are several frameworks that can help frame your research question and develop your search strategy.  They are based on the main ideas or concepts from your research question.
      • Create a list of search terms that describe the main ideas of your topic.  Don't forget to think of synonyms for your search terms.  Different disciplines may use different terminology for the same topic. If your topic is multidisciplinary make sure you consider search terms from all interested disciplines. 
      • Depending on the search tool(s) you may use, consider controlled vocabularies terms in addition to keywords for your search terms.
      • If you know of any key papers use these to help get started.  You can also do a preliminary search to find some relevant studies and create a list of search terms from these studies.   
    • Testing a search strategy
      • Establish a test list of articles known to be relevant.  Divide it in half and use one half to develop your search strategy.  Use the other half to test/validate  your search strategy to see if it finds all of these articles.
      • As you look at your initial list of results you may find new search terms to add to you list.  You can test these new search terms by adding them to your search strategy and seeing if they add new results.

Definition

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context. (Burls, 2009).  Not all knowledge syntheses include the appraisal step.

Criteria

Below are criteria that are commonly used for critical appraisal of studies.  The bracketed terms are often used for qualitative evidence and the non-bracketed terms are used for quantitative evidence (Hannes, 2011)

  • Internal validity (Credibility)
    • Internal validity refers to the accuracy of statements made about the causal relationship between two variables: the independent and dependent variables.
    • There are many threats to internal validity. These can include confounding, selection bias, attrition, detection bias, cross-over, among others.
    • Knowledge of study design is important for critical appraisa of internal validity.
  • External validity (Transferability)
    • External validity refers to the the generalizability or representativeness of experimental effects or program treatments.
    • Threats to external validity include non-representative samples, an artificial laboratory environment, and testing effects.
  • Reliability (Dependability)
    • Reliability refers to the consistency or repeatability of a study
  • Objectivity (Confirmability)
    • Objectivity refers to the extent to which research projects are undistorted by the biases of researchers.

Websites

Tools

There are many tools for critical appraisal.  Check the websites to find other tools for different study designs and methodologies.

References and Further Reading

  1. Burls, A. (2009). What is critical appraisal? http://www.bandolier.org.uk/painres/download/whatis/What_is_critical_appraisal.pdf
  2. Hannes K. Chapter 4: Critical appraisal of qualitative research. In: Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, Lockwood C (editors), Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Version 1 (updated August 2011). Cochrane Collaboration Qualitative Methods Group, 2011. Available from URL http://cqrmg.cochrane.org/supplemental-handbook-guidance  

 

Documenting the search process

  • Systematic review searches must be documented in sufficient detail to allow others to be able to assess the thoroughness of the search.
    You will need to keep track of:

    • database searched, together with the platform searched via (eg. Ebsco PsycINFO,)
    • date each search were conducted
    • Subject headings and keywords used - including whether terms were exploded, truncated, etc
    • Search history, including the combination of terms
    • Number of results retrieved for each search and combination
    • Total number of records
    • Duplicates identified
    • Numbers pre-screening and post-screening
    • In addition, all searches conducted via hand searching must identify the source (name of journal, conference proceedings, etc), plus the years.

Websites

  • MECIR (Methodological Expectations of Cochrane Intervention Reviews) Standards for the reporting of new Cochrane Intervention Reviews
  • PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Minimum set of items for reporting in systematic reviews and meta-analyses
  • MOOSE (Meta-Analysis of Observational Studies in Epidemiology) Reporting guidelines for meta-analyses of observational studies in epidemiology
  • CONSORT (Consolidated Standards of Reporting Trials) CONSORT Statement, is an evidence-based, minimum set of recommendations for reporting randomized trials.
  • STARD (STAndards for the Reporting of Diagnostic Accuracy studies) Standards for diagnostic accuracy studies.
  • EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Database of reporting guidelines for different study types

Tools

  • Rayyan
    Rayyan is a 100% FREE web application to help systematic review authors perform their job in a quick, easy and enjoyable fashion. Authors create systematic reviews, collaborate on them, maintain them over time and get suggestions for article inclusion.
  • Abstrackr
    From Brown University. Open and free to all. It’s set up to automatically pull in abstracts from NLM using PMIDs. One can also transfer abstracts over from EndNote. Allows for collaborative screening of abstracts.
  • Covidence
    Covidence is a core component of Cochrane’s review production toolkit. It is dedicated to improving the use of evidence in healthcare decision-making. You can have a free trial for 1 project and two reviewers.
  • EPPI-Reviewer 4
    EPPI-Reviewer 4 is software for all types of literature review, including systematic reviews, meta-analyses, 'narrative' reviews and meta-ethnographies. It is a ‘not for profit’ service with different subscription fees.  
  • DistillerSR
    DistillerSR is an online application designed specifically for the screening and data extraction phases of a systematic review. Subscription Fee. Additional technical information at: http://www.tectutorials.com/TERDistillerSR.html.
  • Review Manager (RevMan)
    Review Manager is the software used for preparing and maintaining Cochrane Reviews. It is most useful when you have formulated the question for the review, and allows you to prepare the text, build the tables showing the characteristics of studies and the comparisons in the review, and add study data. It can perform meta-analyses and present the results graphically.
  • CADIMA
    FREE tool for screening studies for systematic reviews, access to information and data from ongoing and future projects in biosafety research

Similar to other research projects, you should make a plan for how you will manage the vast amounts of data acquired during the course of your review project. Some of the types of data you will acquire/create are:

  • Exported files from each database
  • Article PDFs
  • Data extraction files (Excel, Word docs)
  • Notes (documenting each step of the process)

Why should you use a software program for your citations?

  • The nature of systematic searching is that you will find thousands of articles that you will need to manage
  • Systematic Review guidelines require you to track what you find from a search, what you discard & why, as well as what articles you keep
  • Citation management software is practically required in order to effectively accomplish these tasks
  • By creating folders, subfolders, and appropriate and clear naming rules, citation management software will keep you organized throughout the SR process

Data management tips

  1. It is good to have a central backup of all data from your project, regardless of which member of the review team is responsible for acquiring or creating that particular type of data. This ensures that the team can always access a copy of all data, in the case of a technology accident (laptop crash, lost USB) or if someone needs to be away for an extended period of time or a personnel change.
  2. Name your files appropriately. It is good to think about a naming convention for the files themselves, as well as the folders in which you will organize your backups. Remember that 12 months from now, you may not be able to remember what 1.ris means.
    • You may want to use a multi part name which includes aspects such as: the project name, step of the review process, database name, individuals name, date, etc.
    • For naming files exported from a database during the search stage, you might name your file ProjectName_NameofDatabase_DateofSearch_#ofRecords.
    • For naming the files that were used to pilot the screening criteria, you might name your file ProjectName_PilotScreening_#ofRecords_Date_InitialsofScreener
    • For naming each of the full-text article PDFs, you might consider naming your files  FirstAuthor_Year_FirstFourWordsOfTitle     
  3. After the search, citation management is necessary...
    • By citation management, we mean managing all the citations (or references) you retrieve from the bibliographic databases (PubMed, etc) or search engines. Typically you would use a bibliographic manager software application to manage the citations. Once you import the references into your application, you can review and weed out those that do not meet your inclusion criteria. You can also annotate the record, and attach or import the full-text PDF into the record.    
    • Citation management tools 

Reference management software for systematic reviews and meta-analyses: an exploration of usage and usability. BMC Med Res Methodol. 2013; 13: 141. doi: 10.1186/1471-2288-13-141

  • De-duplication tips
    • In Endnote, from the menu select References -> "Find Duplicates".  Select which duplicate record to keep by selecting "Keep This Record".
    • In Zotero, click on the "Duplicate Items" collection in your library. You can resolve duplicates by merging the files. 
    • In Mendeley, select your folder of interest. Go to your Tools menu and select "Check for Duplicates". Select the details that you would like to keep from each of the documents. Click merge to create one entry containing the complete document details.
Content last reviewed: October 7, 2020