Content last reviewed:
Carleton offers support for knowledge syntheses (including systematic and scoping reviews) for students, postdoctoral fellows, research assistants and faculty. If you wish to conduct a knowledge synthesis please refer to the different levels of service we can offer.
Before your first meeting with a library staff member, please complete the Carleton Synthesis Workplan document to the best of your ability and share it before your initial meeting.
Introduction
A systematic review is a type of knowledge synthesis. A knowledge synthesis is 'the contextualization and integration of research findings of individual research studies within the larger body of knowledge on the topic. A synthesis must be reproducible and transparent in its methods, using quantitative and/or qualitative methods.' (Grimshaw, 2010)
Types of reviews
| Name | Definition | Strength | Weakness |
|---|---|---|---|
| Literature Review | Generic term: published materials that provide examination of recent or current literature. Can cover wide range of subjects at various levels of completeness and comprehensiveness. May include research findings | Seeks to identify what has been accomplished previously, allowing for consolidation, for building on previous work, for summation, for avoiding duplication and for identifying omissions or gaps | Lacks an explicit intent to maximize scope or analyse data collected |
| Systematic Review | Seeks to systematically search for, appraise and synthesis research evidence, often adhering to guidelines on the conduct of a review | Draw together all known knowledge on a topic area | Restricting studies for inclusion to a single study design can limit this methodology to providing insights about effectiveness rather than seeking answers to more complex questions |
| Rapid Review | Assessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing research | Rigorous and explicit in method and thus systematic but make concessions to the breadth or depth of the process by limiting particular aspects of the systematic review process | Limiting the time taken to search may result in publication bias, limiting appraisal or quality assessment may emphasize poorer quality research, lack of attention to synthesis may overlook inconsistencies or contradictions |
| Scoping Review | Preliminary assessment of potential size and scope of available research literature. Aims to identify nature and extent of research evidence | Systematic, transparent and replicable. Inform policymakers as to whether a full systematic review is needed | Does not include a process of quality assessment. As such, findings cannot be used to recommend policy/practice |
| Integrative Review | General review of the existing literature done as a 'systematic' process | Inclusion of diverse data sources enhances a holistic understanding of the topic of interest | Combining diverse data sources is complex and challenging |
| Realist Review | Realist syntheses specifically attempt to provide ‘an explanatory analysis aimed at discerning what works for whom, in what circumstances, in what respects and how’ | Tries to identify the underlying mechanisms, helps understand context of an intervention | Requires expertise in program theory, not reproducible in the sense of a systematic review |
References and Further Reading
- Grimshaw, J. (2010). CIHR. https://cihr-irsc.gc.ca/e/41382.html
- Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), 91–108. https://onlinelibrary.wiley.com/doi/10.1111/j.1471-1842.2009.00848.x
- Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review - a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. http://journals.sagepub.com/doi/10.1258/1355819054308530
- Whittemore, R., & Knafl, K. (2005). The integrative review: Updated methodology. Journal of Advanced Nursing, 52(5), 546–553. https://onlinelibrary.wiley.com/doi/10.1111/j.1365-2648.2005.03621.x
What does it take to do a systematic review/scoping review?
- Time: A systematic review can take up to 12 months or more to conduct. Scoping reviews usually take less time. It all depends on your research question and the amount of literature published on the topic.
- A team: A systematic/scoping review usually requires a team. You need to work with subject experts to clarify issues related to the topic: librarians to develop comprehensive search strategies and identify appropriate databases; reviewers to screen abstracts and read the full text; a statistician who can assist with data analysis; and a project leader to coordinate and write the final report.
- A clearly defined question: Clarify the key question(s) of you systematic/scoping review and the rationale for each question. Use the PICO, PECO or PCC frameworks to identify key concepts or aspects of the question. Determine inclusion/exclusion criteria based on your question.
- A protocol: You should write a protocol outlining the study methodology. The protocol should include:
- the rationale for the systematic review
- key questions broken into key concepts or "PICO components"
- inclusion/exclusion criteria
- literature searches for published/unpublished literature
- data abstraction/data management plan
- plan for assessment of methodological quality of individual studies
- data synthesis plan
- Comprehensive literature searches:
- Identify a few relevant studies that answer your question
- Identify appropriate databases and search terms for your literature search
- Conduct comprehensive and detailed literature searches that can be documented and duplicated.
- Citation management: You should have working knowledge of a citation manager to help manage citations retrieved from literature searches.
- Follow reporting guidelines: Use appropriate guidelines for reporting your review for publication.
Steps for conducting a systematic review/scoping review
- Check existing reviews/protocols - a decision needs to be made whether to abort the review or amend the question if a similar review or protocol is identified
- Formulate a specific question - often based on a framework (see Search section for more details)
- Eligibility criteria - used to select relevant studies from the search results (selection or screening is often done in duplicate to reduce bias, piloting criteria helps improve consistency)
- Inclusion criteria usually operationalize the research question (eg: population, intervention or comparison, outcome)
- Exclusion criteria are used to exclude certain studies (eg: specific populations, language, study designs, publication types)
- Search strategy - explicit and reproducible (can include grey literature)
- Screening - Using eligibility criteria screen title/abstracts, then use expanded set of criteria to screen full text articles
- Critical appraisal - assess risk of bias in individual studies (not done for scoping reviews)
- Data extraction/Charting - identify and extract/chart data elements ("charting" is used for scoping reviews)
- Synthesis of results - interpret/analyse results (can be either quantitative or qualitative)
- Report - comprehensive report on all steps of the systematic review and presents results
A review plan or protocol
A major cause of bias in a review is answering a different question to that being originally asked. This is why it is important to develop a review plan or protocol.
The benefits of having a protocol before the beginning of a review:
- defines a research process that reduces risk of bias
- promotes a systematic rather than ad hoc approach to the review process
- facilitates communication with others and promotes consistency between review team members
- supports the reliability and usefulness of reviews
Where to register your protocol?
Once you've developed your protocol, register it. This is a basic requirement for a trial or systematic review, to check if your study has already been done by someone else, and whether your proposed review is necessary.
Review Guides
- Cochrane Collaboration Handbook
The Cochrane Handbook for Systematic Reviews of Interventions is the official document that describes in detail the process of preparing and maintaining Cochrane systematic reviews on the effects of healthcare interventions. This is Version Version 6.1, 2020. - Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care
Provides practical guidance for undertaking evidence synthesis based on a thorough understanding of systematic review methodology. Presents core principles of systematic reviews and highlights issues that are specific to reviews of clinical tests, public health interventions, adverse effects, and economic evaluations. The final chapter discusses incorporation of qualitative research in or alongside effectiveness reviews. - Finding What Works in Health Care: Standards for Systematic Reviews (IOM)
IOM's (2011) standards address the entire systematic review process, from locating, screening, and selecting studies for the review, to synthesizing the findings and assessing the overall quality of the body of evidence, to producing the final review report. Includes a link to the IOM Standards for Systematic Reviews. - JBI Manual for Evidence Synthesis
- Methods Guide for Effectiveness and Comparative Effectiveness Reviews (AHRQ) This guide was developed to improve the transparency, consistency, and scientific rigor of those working on Comparative Effectiveness Reviews.
Repositories
- Systematic Review Data Repository (SRDR)
Free; requires login. This systematic review repository also acts as a data extraction tool. - Campbell Collaboration
Guidelines for producing a Campbell Systematic Review. The Campbell Collaboration is an international research network that produces systematic reviews of the effects of social interventions - 3IE Impact
International Initiative for Impact Evaluation offers a database of systematic reviews on impact evaluations and has methods information for conducting your own. - The EPPI-Centre is committed to informing policy and professional practice with sound evidence
- Cochrane Register of Studies
Is meta-register or repository for Specialised Registers from all Cochrane Groups. It is a management tool specifically designed for storing, maintaining and developing registers. - Collaboration for Environmental Evidence
Tools
Search
Search is a balance between sensitivity and specificity (or precision).
- Sensitivity is the number of relevant results identified divided by the total number of relevant results in existence.
- Capture most relevant results
- Capture more irrelevant results
- Spend more time filtering/screening
- Specificity (or precision) is the number of relevant results identified divided by the total number of results identified.
- Capture some relevant results
- Capture fewer irrelevant results
- Risk missing relevant results
For systematic and scoping reviews the goal is a comprehensive search, maximizing sensitivity.
Where to search
- Databases by Subject
- Grey Literature
- Hand searching key journals
- References of included studies/relevant systematic reviews
- Expert referrals for key studies
How to search
- Techniques
- Process
- Developing a search strategy
- There are several frameworks that can help frame your research question and develop your search strategy. They are based on the main ideas or concepts from your research question.
- Create a list of search terms that describe the main ideas of your topic. You can do preliminary searching to discover search terms, check reference materials, or ask content experts for relevant search terms.
- Think of synonyms for your search terms. Different disciplines may use different terminology for the same topic. If your topic is multidisciplinary make sure you consider search terms from all interested disciplines.
- Depending on the search tool(s) you may use, consider controlled vocabularies terms in addition to keywords for your search terms.
- If you know of any key papers use these to help get started. You can also do a preliminary search to find some relevant studies and create a list of search terms from these studies.
- Testing a search strategy
- Establish a test list of articles known to be relevant. Divide it in half and use one half to develop your search strategy. Use the other half to test/validate your search strategy to see if it finds all of these articles.
- As you look at your initial list of results you may find new search terms to add to you list. You can test these new search terms by adding them to your search strategy and seeing if they add new results.
- Developing a search strategy
- Translating a Search
- Each database requires use of specialized search syntax, therefore knowledge synthesis search strategies must be 'translated' between databases.
- Tools to help with translation
- MEDLINE Transpose (translate PubMed to MEDLINE and vice versa)
-
Database syntax guide (for various databases)
- Reference Searching
- Once you have identified eligible studies for your knowledge synthesis by searching the appropriate resources above, it is best practice to then check the reference lists of these studies to identify any additional studies.
- You can also check the references of any relevant systematic reviews that have been conducted.
- You can use citation tracking databases to help do this.
- Hand Searching
- Hand-searching relevant journals page by page can be incredibly time-consuming but it may identify additional studies that have been missed by database search strategies or studies from journals that are not covered in electronic databases.
Definition
Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context. (Burls, 2009). Not all knowledge syntheses include the appraisal step.
Criteria
Below are criteria that are commonly used for critical appraisal of studies. The bracketed terms are often used for qualitative evidence and the non-bracketed terms are used for quantitative evidence (Hannes, 2011)
- Internal validity (Credibility)
- Internal validity refers to the accuracy of statements made about the causal relationship between two variables: the independent and dependent variables.
- There are many threats to internal validity. These can include confounding, selection bias, attrition, detection bias, cross-over, among others.
- Knowledge of study design is important for critical appraisa of internal validity.
- External validity (Transferability)
- External validity refers to the the generalizability or representativeness of experimental effects or program treatments.
- Threats to external validity include non-representative samples, an artificial laboratory environment, and testing effects.
- Reliability (Dependability)
- Reliability refers to the consistency or repeatability of a study
- Objectivity (Confirmability)
- Objectivity refers to the extent to which research projects are undistorted by the biases of researchers.
Websites
Tools
- EPOC Risk of Bias Tool
- OHAT Risk of Bias Tool (human and animal studies)
- SYRCLE Risk of Bias Tool (animal studies)
- CEECAT (from CEE)
- KCC (mechanistic data for carcinogens)
- GRADE
There are many tools for critical appraisal. Check the websites to find other tools for different study designs and methodologies.
References and Further Reading
- Burls, A. (2009). What is critical appraisal? http://www.bandolier.org.uk/painres/download/whatis/What_is_critical_appraisal.pdf
- Hannes K. Chapter 4: Critical appraisal of qualitative research. In: Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, Lockwood C (editors), Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Version 1 (updated August 2011). Cochrane Collaboration Qualitative Methods Group, 2011. Available from URL http://cqrmg.cochrane.org/supplemental-handbook-guidance
Documenting the search process
-
Systematic review searches must be documented in sufficient detail to allow others to be able to assess the thoroughness of the search.
You will need to keep track of:- database searched, together with the platform searched via (eg. Ebsco PsycINFO,)
- date each search were conducted
- Subject headings and keywords used - including whether terms were exploded, truncated, etc
- Search history, including the combination of terms
- Number of results retrieved for each search and combination
- Total number of records
- Duplicates identified
- Numbers pre-screening and post-screening
- In addition, all searches conducted via hand searching must identify the source (name of journal, conference proceedings, etc), plus the years.
Reporting Guidelines
- EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Database of reporting guidelines for different study types
- PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Minimum set of items for reporting in systematic reviews and meta-analyses
- MOOSE (Meta-Analysis of Observational Studies in Epidemiology) Reporting guidelines for meta-analyses of observational studies in epidemiology
- CONSORT (Consolidated Standards of Reporting Trials) CONSORT Statement, is an evidence-based, minimum set of recommendations for reporting randomized trials.
Screening Tools
- Covidence
Covidence is a web-based software platform that streamlines the production of knowledge syntheses. Carleton has an institutional subscription. - Rayyan
Rayyan is a 100% FREE web application to help systematic review authors perform their job in a quick, easy and enjoyable fashion. Authors create systematic reviews, collaborate on them, maintain them over time and get suggestions for article inclusion. - Abstrackr
From Brown University. Open and free to all. It’s set up to automatically pull in abstracts from NLM using PMIDs. One can also transfer abstracts over from EndNote. Allows for collaborative screening of abstracts. - EPPI-Reviewer 4
EPPI-Reviewer 4 is software for all types of literature review, including systematic reviews, meta-analyses, 'narrative' reviews and meta-ethnographies. It is a ‘not for profit’ service with different subscription fees. - DistillerSR
DistillerSR is an online application designed specifically for the screening and data extraction phases of a systematic review. Subscription Fee. Additional technical information at: http://www.tectutorials.com/TERDistillerSR.html. - Review Manager (RevMan)
Review Manager is the software used for preparing and maintaining Cochrane Reviews. It is most useful when you have formulated the question for the review, and allows you to prepare the text, build the tables showing the characteristics of studies and the comparisons in the review, and add study data. It can perform meta-analyses and present the results graphically. - CADIMA
FREE tool for screening studies for systematic reviews, access to information and data from ongoing and future projects in biosafety research
Similar to other research projects, you should make a plan for how you will manage the vast amounts of data acquired during the course of your review project. Some of the types of data you will acquire/create are:
- Exported files from each database
- Article PDFs
- Data extraction files (Excel, Word docs)
- Notes (documenting each step of the process)
Why should you use a software program for your citations?
- The nature of systematic searching is that you will find thousands of articles that you will need to manage
- Systematic Review guidelines require you to track what you find from a search, what you discard & why, as well as what articles you keep
- Citation management software is practically required in order to effectively accomplish these tasks
- By creating folders, subfolders, and appropriate and clear naming rules, citation management software will keep you organized throughout the SR process
Data management tips
- It is good to have a central backup of all data from your project, regardless of which member of the review team is responsible for acquiring or creating that particular type of data. This ensures that the team can always access a copy of all data, in the case of a technology accident (laptop crash, lost USB) or if someone needs to be away for an extended period of time or a personnel change.
- Name your files appropriately. It is good to think about a naming convention for the files themselves, as well as the folders in which you will organize your backups. Remember that 12 months from now, you may not be able to remember what 1.ris means.
- You may want to use a multi part name which includes aspects such as: the project name, step of the review process, database name, individuals name, date, etc.
- For naming files exported from a database during the search stage, you might name your file ProjectName_NameofDatabase_DateofSearch_#ofRecords.
- For naming the files that were used to pilot the screening criteria, you might name your file ProjectName_PilotScreening_#ofRecords_Date_InitialsofScreener
- For naming each of the full-text article PDFs, you might consider naming your files FirstAuthor_Year_FirstFourWordsOfTitle
- After the search, citation management is necessary...
- By citation management, we mean managing all the citations (or references) you retrieve from the bibliographic databases (PubMed, etc) or search engines. Typically you would use a bibliographic manager software application to manage the citations. Once you import the references into your application, you can review and weed out those that do not meet your inclusion criteria. You can also annotate the record, and attach or import the full-text PDF into the record.
-
Citation management tools
- Mendeley
- Zotero
- EndNote Basic (free online version)
Reference management software for systematic reviews and meta-analyses: an exploration of usage and usability. BMC Med Res Methodol. 2013; 13: 141. doi: 10.1186/1471-2288-13-141
- De-duplication tips
- In Endnote, from the menu select References -> "Find Duplicates". Select which duplicate record to keep by selecting "Keep This Record".
- In Zotero, click on the "Duplicate Items" collection in your library. You can resolve duplicates by merging the files.
- In Mendeley, select your folder of interest. Go to your Tools menu and select "Check for Duplicates". Select the details that you would like to keep from each of the documents. Click merge to create one entry containing the complete document details.
