Journal de biométrie et biostatistique

Quality Control Tool for Screening Titles and Abstracts by seco nd Reviewer: QCTSTAR

Abstract

Nevis IF, Sikich N, Ye C and Kabali C

Background: Systematic reviews (SRs) remain the core of evidence based medicine. Routinely, two reviewers are required to screen titles and abstracts in SRs. Nonetheless, many organizations may use a single reviewer due to restricted time and resources. In situations where there is only a single reviewer, we propose a sampling method and assessed its performance in a simulation study. Methods: We described the sampling process guided by a set of instructions. For validation, we generated 20,000 citations from a skewed normal distribution and assigned a score of raters’ agreement. From these, we randomly selected a fixed number of citations, whose probability of selection was determined by a uniform distribution, and repeated the iteration 1000 times. In each iteration set, the sample size was fixed at 50, 100, and 200. Results: We evaluated the sampling performance and proposed the appropriate sample size formula. Of the 20,000 citations, 86.7% fell into the category of “both reviewers have an agreement”. The sampling performance was optimal. On average, the percent of agreement for samples of size 50, 100, and 200 were 86.7% (95% CI 76% to 96%), 86.7% (95% CI 79% to 93%), and 86.8% (95% CI 81.5% to 91.5%) respectively. When comparing the performance of sample size formula with simulations, we obtained identical results. Conclusions: We propose a reliable and valid sampling methodology for screening titles and abstracts This method may be used in resource constrained environments conducting SRs.