Introduction

  • The high sensitivity of searches used in systematic literature reviews (SLRs) often leads to a large screening burden for the reviewer.
  • We have developed a prototype screening tool to filter text and to guide reviewers to relevant passages with the aim of improving the speed of screening while leaving inclusion/exclusion decisions in the hands of the reviewer, ensuring that the rigour of reviewing is maintained.
  • The tool identifies biomedical concepts using Layar, a deep-learning data fabric built by Vyasa Analytics (Boston, MA, USA).

Objective

  • The aim was to assess the efficiency (time savings and accuracy) of using the prototype, artificial intelligence (AI)-assisted screening tool against fully manual reference screening.

Research design and methods

Tool development
  • The named entity models developed by Vyasa Analytics were trained and tested on a collection of Beginning-Inside-Outside (BIO)-labelled data sets across 24 life science-related entity classes.
Screening
  • A set of 300 references from a previously completed SLR was screened by four reviewers; 150 were screened manually and 150 were screenedwith assistance from the AI tool (Figure 1).
  • To mitigate for interpersonal variability and increased screening speed over time, the allocation of references was chosen randomly, and their order switched for each reviewer (Table 1).
  • The time required to screen all references was recorded and compared by reviewer and by screening method.
  • Accuracy was determined for each reviewer by comparing included references against those that were selected in the double-blind screening of the previously completed SLR.
Figure 1: Study design.
Figure 1
Table 1. Allocation and screening order of references.
Screening order
Reviewer 1 AI assisted
References 1–150
Manual
References 151–300
Reviewer 2 Manual
References 1–150
AI assisted
References 151–300
Reviewer 3 AI assisted
References 151–300
Manual
References 1–150
Reviewer 4 Manual
References 151–300
AI assisted
References 1–150

Results

Time
  • The median time taken to screen 150 papers manually was 51 (interquartile range [IQR]: 44–56) minutes, whereas the median time taken using the AI-assisted screening tool was 38 (IQR: 33–47) minutes (Figure 2 and Figure 3).
Accuracy
  • Accuracy was similar in manual and AI-assisted approaches (median: 82% and 80%, respectively; Figure 4).
Figure 2. Time required for AI-assisted and manual screening.
Box and whisker plot to compare data distribution. Black circles represent data points, solid black lines represent medians, white diamonds represent means.
Figure 3. Time spent on screening by each reviewer and method used.
Figure 4. Accuracy of the decisions by reviewer and method used.

Conclusions

  • Citation screening may be more efficient when assisted by AI. These results suggest screening is faster without loss of accuracy, as inclusion/exclusion decisions remain in the hands of the reviewer ensuring that the rigour of reviewing is maintained.
  • These findings support further development of the AI-assisted screening tool and our next step is to increase the statistical power of our test, via a follow-up study that will include approximately 1000 references and eight reviewers.
Code availability: Codes for named entity model training, inference and feature extraction are proprietary and contained within the Layar software.
Keywords: AI, artificial intelligence, information extraction, named entity recognition, natural language processing, systematic review, text analytics.
Poster number: 15
To be presented at the ISMPP 2022 European Meeting (25–26 January 2022)
/* --------------------------------------- Pie Chart ------------------------------------------ */