Critical appraisal for medical and health sciences: 3. Checklists

What is a systematic review?

A review of a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results of the included studies.

From Cochrane Glossary

Some questions to ask when critically appraising a systematic review:

• Do you think all the important, relevant studies were included?
• Did the review’s authors do enough to assess quality of the included studies?
• If the results of the review have been combined, was it reasonable to do so?

From: Critical Appraisal Skills Programme (2018). CASP Systematic Review Checklist. [online] Available at: https://casp-uk.net/casp-tools-checklists/. Accessed: 22/08/2018

Checklists you can use to critically appraise a systematic review:

What is a randomised controlled trial?

An experiment in which two or more interventions, possibly including a control intervention or no intervention, are compared by being randomly allocated to participants. In most trials on intervention is assigned to each individual but sometimes assignment is to defined groups of individuals (for example, in a household) or interventions are assigned within individuals (for example, in different orders or to different parts of the body).

From Cochrane Glossary

Some questions to ask when critically appraising RCTs:

• Was the assignment of patients to treatments randomised?
• Were patients, health workers and study personnel ‘blind’ to treatment? i.e. could they tell who was in each group?
• Were all of the patients who entered the trial properly accounted for at its conclusion? 
• Were all participants analysed in the groups in which they were randomised, i.e. was a Intention to treat analysis undertaken?

From: Critical Appraisal Skills Programme (2018). CASP Randomised Controlled Trial Checklist. [online] Available at: https://casp-uk.net/casp-tools-checklists/. Accessed: 22/08/2018

Checklists you can use to critically appraise an RCT:

What is a qualitative study?

Qualitative research is designed to explore the human elements of a given topic, where specific methods are used to examine how individuals see and experience the world...Qualitative methods are best for addressing many of the why questions that researchers have in mind when they develop their projects. Where quantitative approaches are appropriate for examining who has engaged in a behavior or what has happened and while experiments can test particular interventions, these techniques are not designed to explain why certain behaviors occur. Qualitative approaches are typically used to explore new phenomena and to capture individuals’ thoughts, feelings, or interpretations of meaning and process.

From Given, L. (2008) The SAGE Encyclopedia of Qualitative Research Methods. Sage: London.

Some questions to ask when critically appraising a qualitative study: 

• What was the selection process and was it appropriate? 
• Were potential ethical issues addressed, such as the potential impact of the researcher on the participants? Has anything been done to limit the effects of this?
• Was the data analysis done using explicit, rigorous, and justified methods?

From: Critical Appraisal Skills Programme (2018). CASP Qualitative Checklist. [online] Available at: https://casp-uk.net/casp-tools-checklists/. Accessed: 22/08/2018

Checklists you can use to critically appraise a qualitative study:

Watch the video for an example of how to critically appraise a qualitative study using the CASP checklist:

What is a cohort study?

An observational study in which a defined group of people (the cohort) is followed over time. The outcomes of people in subsets of this cohort are compared, to examine people who were exposed or not exposed (or exposed at different levels) to a particular intervention or other factor of interest. A prospective cohort study assembles participants and follows them into the future. A retrospective (or historical) cohort study identifies subjects from past records and follows them from the time of those records to the present. Because subjects are not allocated by the investigator to different interventions or other exposures, adjusted analysis is usually required to minimise the influence of other factors (confounders).

From Cochrane Glossary

Some questions to ask when critically appraising a cohort study

• Have there been any attempts to limit selection bias or other types of bias?
• Have the authors identified any confounding factors?
• Are the results precise and reliable?

From: Critical Appraisal Skills Programme (2018). CASP Cohort Study Checklist. [online] Available at: https://casp-uk.net/casp-tools-checklists/. Accessed: 22/08/2018

Checklists you can use to critically appraise a cohort study:

What is a case-control study?

A study that compares people with a specific disease or outcome of interest (cases) to people from the same population without that disease or outcome (controls), and which seeks to find associations between the outcome and prior exposure to particular risk factors. This design is particularly useful where the outcome is rare and past exposure can be reliably measured. Case-control studies are usually retrospective, but not always.

From Cochrane Glossary

Some questions to ask when critically appraising a case-control study:

• Was the recruitment process appropriate? Is there any evidence of selection bias?
• Have all confounding factors been accounted for?
• How precise is the estimate of the effect? Were confidence intervals given?
• Do you believe the results?

From Critical Appraisal Skills Programme (2018). CASP Case Control Study Checklist. [online] Available at: https://casp-uk.net/casp-tools-checklists/. Accessed: 22/08/2018. 

Checklists you can use to critically appraise a case-control study:

What is a case report?

A study reporting observations on a single individual.

Some questions to ask when critically appraising a case report:

• Is the researcher’s perspective clearly described and taken into account?
• Are the methods for collecting data clearly described?
• Are the methods for analysing the data likely to be valid and reliable?
• Are quality control measures used?
• Was the analysis repeated by more than one researcher to ensure reliability?
• Are the results credible, and if so, are they relevant for practice? Are the results easy to understand?
• Are the conclusions drawn justified by the results?
• Are the findings of the study transferable to other settings?

From: Roever and Reis (2015), 'Critical Appraisal of a Case Report', Evidence Based Medicine and Practice Vol. 1 (1) 

Checklists you can use to critically appraise a case report:

What are in vivo animal studies?

In vivo animal studies are experiments carried out using animals as models. These studies are usually pre-clinical, often bridging the gap between in vitro experiments (using cell or microorganisms) and research with human participants.

Arrive guidelines provide suggested minimum reporting standards for in vivo experiments using animal models. You can use these to help you evaluate the quality and transparency of animal studies

Some questions to ask when critically appraising in vivo studies:

• Is the study/experimental design explained clearly?
• Was the sample size clearly stated, with information about how sample size was decided?
• Was randomisation used?
• Who was aware of group allocation at each stage of the experiment?
• Were outcomes measures clearly defined and assessed?
• Were the statistical methods used clearly explained?
• Were all relevant details about animals used in the experiment clearly outlined (species, strain and substrain, sex, age or developmental stage, and, if relevant, weight)
• Were experimental procedures explained in enough detail for them to be replicated?
• Were the results clear, with relevant statistics included?

Adapted from: The ARRIVE guidelines 2.0: author checklist

Checklist

The ARRIVE guidelines 2.0: author checklist

While this checklist has been designed for authors to help while writing their studies, you can use the checklist to help you identify whether or not a study reports all of the required elements effectively.

SciRAP : evaluation of in vivo toxicity studies tool

The SciRAP method for evaluating reliability of in vivo toxicity studies consists of criteria for for evaluating both the reporting quality and methodological quality of studies, separately. You can switch between evaluation of reporting and methodological quality.

Further guidance 

Hooijmans CR, Rovers MM, de Vries RB, Leenaars M, Ritskes-Hoitinga M, Langendam MW. SYRCLE's risk of bias tool for animal studies. BMC Med Res Methodol. 2014 Mar 26;14:43. doi: 10.1186/1471-2288-14-43. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4230647/

Kilkenny C, et al, Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoSBiol2010;8:e1000412. doi:10.1371/journal.pbio.100041220613859: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1000412

Moermond CT, Kase R, Korkaric M, Ågerstrand M. CRED: Criteria for reporting and evaluating ecotoxicity data. Environ Toxicol Chem. 2016 May;35(5):1297-309. doi: 10.1002/etc.3259.

What are in vitro studies?

In vitro studies involve tests carried out outside of a living organism, usually involving tissues, organs or cells.

Some questions to ask when critically appraising in vitro studies:

SciRAP : evaluation of in vitro toxicity studies tool

The SciRAP method for evaluating reliability of in vitro toxicity studies consists of criteria for for evaluating both the reporting quality and methodological quality of studies, separately. You can switch between evaluation of reporting and methodological quality.

Development and validation of a risk-of-bias tool for assessing in vitro studies conducted in dentistry: The QUIN

Checklist designed to support the evaluation of in vitro dentistry studies, although can be used to assess risk of bias in other types of in vitro studies.

What is grey literature?

The term grey literature is used to describe a wide range of different information that is produced outside of traditional publishing and distribution channels, and which is often not well represented in indexing databases.

A widely accepted definition in the scholarly community for grey literature is

"information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing" ie. where publishing is not the primary activity of the producing body." 

From: Third International Conference on Grey Literature in 1997 (ICGL Luxembourg definition, 1997  - Expanded in New York, 2004).

You can find out more about grey literature and how to track it down here.

Some questions to ask when critically appraising grey literature:

• Who is the author? Are they credible and do they have appropriate qualifications to speak to the subject?
• Does the source have a clearly stated aim or brief, and does it meet this?
• Does the source reference credible and authoritative sources?
• Is any data collection valid and appropriate for it's purpose?
• Are any limits stated? e.g. missing data, information outside of scope or resource of project.
• Is the source objective, or does the source support a viewpoint that could be biased?
• Does the source have a identifiable date?
• Is the source appropriate and relevant to the research area you have chosen?

Adapted from AACOS checklist

Checklist

AACODS Checklist

The AACODS checklist has been designed to support the evaluation and critical appraisal of grey literature.