Skip to main content
Effective Health Care Program
Home » Products » Comparing Evidence Synthesis Search Strategies Developed With and Without Text Mining Tools » A Prospective Comparison of Evidence Synthesis Search Strategies Developed With and Without Text-Mining Tools

A Prospective Comparison of Evidence Synthesis Search Strategies Developed With and Without Text-Mining Tools

Research Report Draft

This report is available in PDF only (Draft Methods Research Report, 474 KB). For additional assistance, please contact us.

Purpose of Study

The objectives of this study were to compare the benefits and trade-offs of searches with and without the use of text-mining tools (TMTs) for evidence synthesis products in real world settings. Specific questions included: (1) Do TMTs decrease the time spent developing search strategies? (2) How do TMTs affect the sensitivity and yield of searches? (3) Do TMTs identify groups of records that can be safely excluded in the search evaluation step? (4) Does the complexity of a systematic review topic affect TMTs performance? In addition to quantitative data, we collected librarians' comments on their experiences using TMTs to explore when and how these new tools may be useful in systematic review search creation.

Key Messages

  • TMTs the time required to develop keyword and subject terms compared to usual practice (UP) search development strategies used in 6 out of 7 reports.
  • TMTs searches were less sensitive than UP searches in all but one project.
  • Number-needed-to-read (NNR) results were mixed; NNR was lower using TMTs compared to UP in 4 out of 7 reports.
  • TMTs neither affected search evaluation time nor improved identification of exclusion (false positive) concepts that can be safely removed from the search set.
  • Across "simple" review topics (e.g., single indication-single drug) TMTs yielded no unique additional relevant studies while missing only one relevant study in 75% of the reports and reduced time spent on creating searches compared to UP. Thus, TMTs may be useful in rapid review search strategy development when timeliness is prioritized over comprehensiveness.
  • Across "complex" review topics (e.g., multicomponent interventions) TMTs identified some unique includable studies and reduced time spent in search strategy development but missed some relevant studies compared to UP. TMTs may be useful as an adjunct to usual practice for complex evidence synthesis reviews (e.g. evidence maps, scoping reviews, systematic reviews, health technology assessments, and update reviews, etc.) when comprehensiveness is prioritized over timeliness.

Structured Abstract

Background: In an era of explosive growth in biomedical evidence, improving systematic review (SR) search processes is increasingly critical. Text-mining tools (TMTs) are a potentially powerful resource to improve and streamline search strategy development.

Objectives: The objectives of this study were to compare the benefits and trade-offs of searches with and without the use of TMTs for evidence synthesis products in real world settings. Specific questions included: (1) Do TMTs decrease the time spent developing search strategies? (2) How do TMTs affect the sensitivity and yield of searches? (3) Do TMTs identify groups of records that can be safely excluded in the search evaluation step? (4) Does the complexity of a systematic review topic affect TMTs performance? In addition to quantitative data, we collected librarians' comments on their experiences using TMTs to explore when and how these new tools may be useful in systematic review search creation.

Methods: In this prospective comparative study, we included nine SR projects, and classified them into simple or complex topics. The project librarian used conventional "usual practice" (UP) methods to create the MEDLINE search strategy, while a paired TMT librarian simultaneously and independently created a search strategy using a variety of TMTs. All text-mining searches were created using freely available TMTs, so librarians across diverse settings can replicate the process. We collected results from each MEDLINE search (with and without TMTs), coded every citation's origin (UP or TMT respectively), deduplicated them, then sent the citation library to the review team for screening. When the draft report was submitted, we used the final list of included studies to calculate the sensitivity, precision, and number-needed-to-read for each search (with and without TMTs). Separately, we tracked the time spent on various aspects of search creation by each librarian. Simple and complex topics were analyzed separately to provide insight into whether TMTs performed better on one or the other.

Results: Across all reviews, we found that TMT searches missed more relevant articles than they added. Thus, the UP searches were more sensitive (92 percent (95 percent CI 85 percent to 99 percent) than TMT searches (84.9 percent (95 percent CI 74.4 percent to 95.4 percent). The mean number-needed-to-read was 83 (SD 34) for UP and 90 (SD 68) for TMT. Keyword and subject term development using TMTs generally took less time than those developed using UP alone. The average number of total hours to create the complete model search by UP librarians was 12 hours (SD 8) and for the TMT librarians 5 hours (SD 2). TMTs neither affected search evaluation time nor improved identification of exclusion (false positive) concepts that can be safely removed from the search set.

Conclusion: Across all reviews but one, TMT searches were less sensitive than UP searches as they missed more relevant studies than they found. For simple SR topics (e.g., single indication-single drug), TMT searches were slightly less sensitive, but reduced time spent in search design. TMTs may be useful for rapid review searches when timeliness is more important than comprehensiveness. For complex SR topics (e.g., multicomponent interventions), TMT searches were less sensitive than UP searches, however at the same time they identified includable studies not found by the UP search. TMT searches also reduced time spent in search strategy development. TMT searches may be useful as an adjunct to UP for evidence synthesis types, such as evidence maps, scoping reviews, systematic reviews, health technology assessments, and update reviews because they can identify unique includable studies.