Are you new to conducting systematic reviews of interventions? If the answer is yes, or you need a refresh, you may benefit from completing the e-learning package Cochrane Interactive Learning. It consists of eleven e-learning modules that have been designed by experts to improve your knowledge and skills in systematic review methodology. Although designed to support healthcare intervention reviews, many researchers conducing different types of systematic reviews are likely to find modules 1-3 useful.
Please note, you will need to use a custom link to register for a Cochrane Interactive Learning account. Please register and annually re-register using the link below.
You will need to re-register your account on an annual basis by visiting the custom link above. Please contact library@exeter.ac.uk if you have any questions.
The term grey literature is used to describe a wide range of different information that is produced outside of traditional publishing and distribution channels, and which is often not well represented in indexing databases. Grey literature includes reports, conferences proceedings, government publications, working papers and much more.
Find out more by clicking on the introduction guide.
There are different checklists for different study types, as each are prone to different biases.
The following tabs will give you an overview of some of the different study types you will come across, the sort of questions you will need to consider with each, and the checklists you can use.
Not sure what type of study you're looking at? See the Spotting the study design guide from the Centre for Evidence-Based Medicine for more help.
What is a systematic review?
A review of a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results of the included studies.
From Cochrane Glossary
Some questions to ask when critically appraising a systematic review:
From: Critical Appraisal Skills Programme (2018). CASP Systematic Review Checklist. [online] Available at: https://casp-uk.net/casp-tools-checklists/. Accessed: 22/08/2018
Checklists you can use to critically appraise a systematic review:
What is a randomised controlled trial?
An experiment in which two or more interventions, possibly including a control intervention or no intervention, are compared by being randomly allocated to participants. In most trials on intervention is assigned to each individual but sometimes assignment is to defined groups of individuals (for example, in a household) or interventions are assigned within individuals (for example, in different orders or to different parts of the body).
From Cochrane Glossary
Some questions to ask when critically appraising RCTs:
From: Critical Appraisal Skills Programme (2018). CASP Randomised Controlled Trial Checklist. [online] Available at: https://casp-uk.net/casp-tools-checklists/. Accessed: 22/08/2018
Checklists you can use to critically appraise an RCT:
What is a qualitative study?
Qualitative research is designed to explore the human elements of a given topic, where specific methods are used to examine how individuals see and experience the world...Qualitative methods are best for addressing many of the why questions that researchers have in mind when they develop their projects. Where quantitative approaches are appropriate for examining who has engaged in a behavior or what has happened and while experiments can test particular interventions, these techniques are not designed to explain why certain behaviors occur. Qualitative approaches are typically used to explore new phenomena and to capture individuals’ thoughts, feelings, or interpretations of meaning and process.
From Given, L. (2008) The SAGE Encyclopedia of Qualitative Research Methods. Sage: London.
Some questions to ask when critically appraising a qualitative study:
From: Critical Appraisal Skills Programme (2018). CASP Qualitative Checklist. [online] Available at: https://casp-uk.net/casp-tools-checklists/. Accessed: 22/08/2018
Checklists you can use to critically appraise a qualitative study:
Watch the video for an example of how to critically appraise a qualitative study using the CASP checklist:
What is a cohort study?
An observational study in which a defined group of people (the cohort) is followed over time. The outcomes of people in subsets of this cohort are compared, to examine people who were exposed or not exposed (or exposed at different levels) to a particular intervention or other factor of interest. A prospective cohort study assembles participants and follows them into the future. A retrospective (or historical) cohort study identifies subjects from past records and follows them from the time of those records to the present. Because subjects are not allocated by the investigator to different interventions or other exposures, adjusted analysis is usually required to minimise the influence of other factors (confounders).
From Cochrane Glossary
Some questions to ask when critically appraising a cohort study
From: Critical Appraisal Skills Programme (2018). CASP Cohort Study Checklist. [online] Available at: https://casp-uk.net/casp-tools-checklists/. Accessed: 22/08/2018
Checklists you can use to critically appraise a cohort study:
What is a case-control study?
A study that compares people with a specific disease or outcome of interest (cases) to people from the same population without that disease or outcome (controls), and which seeks to find associations between the outcome and prior exposure to particular risk factors. This design is particularly useful where the outcome is rare and past exposure can be reliably measured. Case-control studies are usually retrospective, but not always.
From Cochrane Glossary
Some questions to ask when critically appraising a case-control study:
From Critical Appraisal Skills Programme (2018). CASP Case Control Study Checklist. [online] Available at: https://casp-uk.net/casp-tools-checklists/. Accessed: 22/08/2018.
Checklists you can use to critically appraise a case-control study:
What is a case report?
A study reporting observations on a single individual.
Some questions to ask when critically appraising a case report:
Checklists you can use to critically appraise a case report:
What are in vivo animal studies?
In vivo animal studies are experiments carried out using animals as models. These studies are usually pre-clinical, often bridging the gap between in vitro experiments (using cell or microorganisms) and research with human participants.
Arrive guidelines provide suggested minimum reporting standards for in vivo experiments using animal models. You can use these to help you evaluate the quality and transparency of animal studies
Some questions to ask when critically appraising in vivo studies:
Adapted from: The ARRIVE guidelines 2.0: author checklist
Checklist
The ARRIVE guidelines 2.0: author checklist
While this checklist has been designed for authors to help while writing their studies, you can use the checklist to help you identify whether or not a study reports all of the required elements effectively.
SciRAP : evaluation of in vivo toxicity studies tool
The SciRAP method for evaluating reliability of in vivo toxicity studies consists of criteria for for evaluating both the reporting quality and methodological quality of studies, separately. You can switch between evaluation of reporting and methodological quality.
Further guidance
Hooijmans CR, Rovers MM, de Vries RB, Leenaars M, Ritskes-Hoitinga M, Langendam MW. SYRCLE's risk of bias tool for animal studies. BMC Med Res Methodol. 2014 Mar 26;14:43. doi: 10.1186/1471-2288-14-43. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4230647/
Kilkenny C, et al, Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoSBiol2010;8:e1000412. doi:10.1371/journal.pbio.100041220613859: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1000412
Moermond CT, Kase R, Korkaric M, Ågerstrand M. CRED: Criteria for reporting and evaluating ecotoxicity data. Environ Toxicol Chem. 2016 May;35(5):1297-309. doi: 10.1002/etc.3259.
What are in vitro studies?
In vitro studies involve tests carried out outside of a living organism, usually involving tissues, organs or cells.
Some questions to ask when critically appraising in vitro studies:
Adapted from: https://echa.europa.eu/support/registration/how-to-avoid-unnecessary-testing-on-animals/in-vitro-methods
Guidance and checklists
SciRAP : evaluation of in vitro toxicity studies tool
The SciRAP method for evaluating reliability of in vitro toxicity studies consists of criteria for for evaluating both the reporting quality and methodological quality of studies, separately. You can switch between evaluation of reporting and methodological quality.
Checklist designed to support the evaluation of in vitro dentistry studies, although can be used to assess risk of bias in other types of in vitro studies.
What is grey literature?
The term grey literature is used to describe a wide range of different information that is produced outside of traditional publishing and distribution channels, and which is often not well represented in indexing databases.
A widely accepted definition in the scholarly community for grey literature is
"information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing" ie. where publishing is not the primary activity of the producing body."
From: Third International Conference on Grey Literature in 1997 (ICGL Luxembourg definition, 1997 - Expanded in New York, 2004).
You can find out more about grey literature and how to track it down here.
Some questions to ask when critically appraising grey literature:
Checklist
The AACODS checklist has been designed to support the evaluation and critical appraisal of grey literature.