This is a chapter from AHRQ's Methods Guide for Medical Test Reviews.

Abstract

This chapter discusses appropriate techniques for developing search strategies for systematic reviews of medical tests. It offers general advice for searching for systematic reviews and also addresses issues specific to systematic reviews of medical tests. Diagnostic search filters are currently not sufficiently developed for use when searching for systematic reviews. Instead, authors should construct a highly sensitive search strategy that uses both controlled vocabulary and text words. A comprehensive search should include multiple databases and sources of grey literature. A list of subject-specific databases is provided.

Introduction

Locating all published studies relevant to the key questions is a goal of all systematic reviews. Inevitably, systematic reviewers encounter variation in whether or how a study is published and in how the elements of a study are reported in the literature or indexed by organizations such as the National Library of Medicine. A systematic search must attempt to overcome these issues in order to identify all relevant studies, taking into account the usual constraints on time and resources.

Although I have written this chapter of the Methods Guide for Medical Test Reviews (also referred to as the Medical Test Methods Guide) as guidance for Evidence-based Practice Centers (EPCs), I hope it will also serve as a useful resource for other investigators interested in conducting systematic reviews on medical tests and in particular for the librarian or information specialist conducting the search. Searching for genetic tests and prognostic studies is covered in chapters 11 and 12 of this Medical Test Methods Guide.

While this chapter will discuss issues specific to systematic reviews of medical tests, (including screening, diagnostic, and prognostic tests), it is important to remember that general guidance on searching for systematic reviews¹ also applies. Literature searches will always seek a balance between recall (how much of the relevant literature is located) and precision (how much of the retrieved literature is relevant). The optimal balance depends on context. Within the context of comparative effectiveness research, the goal is to have a comprehensive (if not exhaustive) search while still trying to minimize the resources necessary for review of the retrieved citations.

In general, bibliographic searches for systematic reviews in health care should always include MEDLINE® and the Cochrane Central Register of Controlled Trials. Additional databases that are often useful to search include Embase®, CINAHL® and PsychINFO®. When constructing the searches in these bibliographic databases, it is important to use both controlled and uncontrolled vocabulary and to tailor the search for each individual database. Limits such as age and language should not be used unless a specific case can be made for their use.

Working closely with the research team and reviewing the analytic framework and inclusion and exclusion criteria will help to develop the search strategy. Reading the references of all included studies is a useful technique to identify additional studies, as is using a citation database such as Scopus® or Web of Science® to find articles that have cited key articles that have already been retrieved. In addition to published literature, a comprehensive search will include looking for unpublished or “grey literature.” In the context of comparative effectiveness research, regulatory information, clinical trial registries, and conference proceedings/abstracts are the most useful sources for identifying data.

Common Challenges

Systematic reviews of test strategies for a given condition require a search for each of the relevant test strategies under consideration. In conducting the search, systematic reviewers may use one of two approaches. Either the reviewers may search on all possible tests used to evaluate the given disease, which requires knowing all the possible test strategies available, or they may search on the disease or condition and then focus on medical test evaluation for that disease.

When a review focuses on specific named tests, searching is relatively straightforward. The names of the tests can be used to locate studies, and a specific search for the concept of diagnosis, screening or prognosis may not be necessary.^{2, 3} But because testing strategies are constantly evolving, using the strategy of relying on specific named tests may risk missing emerging approaches. Tests that measure a gene product may be associated with multiple diseases, so searching by test name alone may be insufficient. It is often advisable to search for the target illness in addition to known test names, or for the target illness alone if specific tests are unknown. However, searches for a disease or condition are broader searches and greatly increase the burden of work in filtering down to the relevant studies on medical test evaluation.

Principles for Addressing the Challenges

Principle 1: Do not rely on search filters alone.

Several search filters (sometimes called “hedges”), which are pre-prepared and tested searches that can be combined with searches on a particular disease or condition, have been developed to aid systematic reviewers evaluating medical tests. Most of these filters have been developed for MEDLINE.^2–6 One filter in particular⁷ is used in the PubMed® Clinical Queries for diagnosis (Table 4–1). Search filters have also been developed specifically for diagnostic imaging⁸ and for Embase.^9,10

Unfortunately, although these search filters are useful for the casual searcher who simply needs some good articles on diagnosis, they are inappropriate for use in systematic reviews of clinical effectiveness. Several researchers^6,11–14 have reported that using these filters for systematic reviews may result in relevant studies being missed. Vincent found that most of the available filters perform better when they are being evaluated than when they are used in the context of an actual systematic review;¹³ this finding is particularly true for studies published before 1990 because of nonstandardized reporting and indexing of medical test studies.

In recent years, improved reporting and indexing of randomized controlled trials (RCTs) have made such trials much easier to find. There is reason to believe that reporting and indexing of medical test studies will similarly improve in the future.¹² In fact, Kastner and colleagues¹⁵ recently reviewed 22 systematic reviews of diagnostic accuracy published in 2006 to determine whether the PubMed Clinical Queries Filter for diagnosis would be sufficient to locate all the primary studies that the 22 systematic reviews had identified through traditional search strategies. Using these filters in MEDLINE and Embase, the authors found 99 percent of the articles in the systematic reviews they examined, and they determined that the missed articles would not have altered the conclusions of the systematic reviews. The authors therefore concluded that filters may be appropriate when searching for systematic reviews of medical test accuracy. However, until more evidence of their effectiveness is found, we recommend that searchers not rely on them exclusively.

Principle 2: Do not rely on controlled vocabulary (subject headings) alone.

When searching, it is important to use all known variants of the test name such as abbreviations, generic and proprietary names, as well as international terms and spellings, and these may not all be controlled vocabulary terms. Because reporting and indexing of studies of medical tests is so variable, one cannot rely on controlled vocabulary terms alone.³

Using textwords for particular medical tests will help to identify medical test articles that have not yet been indexed or that have not been indexed properly.² Filters may suggest the sort of textwords that may be appropriate. Michel16 discusses appropriate MeSH headings and other terminology useful for searching for medical tests.

Principle 3: Search in multiple locations.

Always—but in particular with searches for studies of medical tests—we advise systematic reviewers to search more than one database and to tailor search strategies to each individual database.¹⁷ Because there can be little overlap between many databases,^18–20 failure to search additional databases carries a risk of bias.^21–23 For more information on potentially appropriate databases to use see Table 4–2.

Until reporting and indexing are improved and standardized, a combination of highly sensitive searches and brute-force article screening will remain the best approach for systematically searching the medical test literature.^{6, 11–13} However, this approach is still likely to miss relevant articles; therefore, authors should search additional sources of information. Tracking citations, reading references of relevant articles, and identifying articles that cite key studies are important ways to find additional citations.²⁴ Table 4–3 lists databases that are appropriate for tracking citations.

In addition to bibliographic databases and citation analysis, regulatory documents are another potential source of information for systematic reviews of medical reviews. The FDA regulates many medical tests as devices. The regulatory documents for diagnostic tests are available on the FDA’s Device Web site: www.accessdata.fda.gov/scripts/cdrh/devicesatfda/.

Illustration: Contrasting Search Strategies

Two contrasting search strategies may help illustrate these principles. In the AHRQ report, Testing for BNP and NT-proBNP in the Diagnosis and Prognosis of Heart Failure,25 the medical tests in question were known. Therefore, the search consisted of all possible variations on the names of these tests and did not need to include a search string to capture the diagnostic testing concept. By contrast, in the AHRQ report, Effectiveness of Noninvasive Diagnostic Tests for Breast Abnormalities,²⁶ all possible diagnostic tests were not known. For this reason, the search strategy included a search string meant to capture the diagnostic testing concept, and this relied heavily on textwords. The actual search strategy used in PubMed to capture the concept of diagnostic tests was as follows: diagnosis OR diagnose OR diagnostic OR di[sh] OR “gold standard” OR “ROC” OR “receiver operating characteristic” OR sensitivity and specificity[mh] OR likelihood OR “false positive” OR “false negative” OR “true positive” OR “true negative” OR “predictive value” OR accuracy OR precision.

Summary

Key points to keep in mind when developing a search strategy for medical test reviews:

• Diagnostic search filters—or, more specifically, the reporting and indexing of medical test studies upon which these filters rely—are not sufficiently well developed to be depended upon exclusively for systematic reviews.
• If the full range of tests is known, one may not need to search for the concept of diagnostic testing; searching for the specific test using all possible variant names may be sufficient.
• Combining highly sensitive searches utilizing textwords with hand searching and acquisition and review of cited references in relevant papers is currently the best way to identify all or most relevant studies for a systematic review.
• Do not rely on controlled vocabulary alone.
• Check Devices@FDA (www.accessdata.fda.gov/scripts/cdrh/devicesatfda/).

References

1. Relevo R, Balshem H. Finding evidence for comparing medical interventions: Agency for Healthcare Research and Quality (AHRQ) and the Effective Health Care program. J Clin Epidemiol. 2011;64(11):1168-77.
2. Deville WL, Bezemer PD, BouterLM. Publications on diagnostic test evaluation in family medicine journals: an optimal search strategy. J Clin Epidemiol. 2000. 53(1): 65-9.
3. van der Weijden T, Ijzermans CJ, Dinant GJ, et al. Identifying relevant diagnostic studies in MEDLINE. The diagnostic value of the erythrocyte sedimentation rate (ESR) and dipstick as an example. Family Practice. 1997;14(3):204-8.
4. Bachmann LM, Coray R, Estermann P, et al. Identifying diagnostic studies in MEDLINE: reducing the number needed to read. Journal of the American Medical Informatics Association. 2002; 9(6):653-8.
5. Haynes RB et al. Developing optimal search strategies for detecting clinically sound studies in MEDLINE. Journal of the American Medical Informatics Association 1994. 1(6):447-58.
6. Ritchie G, Glanville J, Lefebvre C. Do published search filters to identify diagnostic test accuracy studies perform adequately? Health Information and Libraries Journal. 2007; 24(3):188-92.
7. Haynes RB, Wilczynski NL. Optimal search strategies for retrieving scientifically strong studies of diagnosis from Medline: analytical survey. BMJ. 2004;328(7447):1040.
8. Astin MP, Brazzelli MG, Fraser CM, et al. Developing a sensitive search strategy in MEDLINE to retrieve studies on assessment of the diagnostic performance of imaging techniques. Radiology. 2008. 247(2):365-73.
9. Bachmann LM, Estermann P, Kronenberg C, et al. Identifying diagnostic accuracy studies in EMBASE. Journal of the Medical Library Association. 2003;91(3):341-6.
10. Wilczynski NL, Haynes RB. EMBASE search strategies for identifying methodologically sound diagnostic studies for use by clinicians and researchers. BMC Medicine. 2005;3:7.
11. Leeflang MM, Scholten RJ, Rutjes AW, et al. Use of methodological search filters to identify diagnostic accuracy studies can lead to the omission of relevant studies. J Clin Epidemiol. 2006;59(3):234-40.
12. Doust JA, Pietrzak E, Sanders S, et al. Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract. J Clin Epidemiol. 2005;58(5):444-9.
13. Vincent S, Greenley S, Beaven O. Clinical Evidence diagnosis: Developing a sensitive search strategy to retrieve diagnostic studies on deep vein thrombosis: a pragmatic approach. Health Information and Libraries Journal. 2003;20(3):150-9.
14. Whiting P, Westwood M, Beynon R, et al. Inclusion of methodological filters in searches for diagnostic test accuracy studies misses relevant studies. J Clinical Epidemiol. 2011;64(6):602-7.
15. Kastner M, Wilczynski NL, McKibbon AK, et al. Diagnostic test systematic reviews: Bibliographic search filters ("clinical queries") for diagnostic accuracy studies perform well. J Clinical Epidemiol. 2009;62(9):974-81.
16. Michel P, Mouillet E, Salmi LR. Comparison of Medical Subject Headings and standard terminology regarding performance of diagnostic tests. J Med Libr Assoc. 2006;94(2):221-3.
17. Honest H, Bachmann LM, and Khan K. Electronic searching of the literature for systematic reviews of screening and diagnostic tests for preterm birth. European Journal of Obstetrics & Gynecology and Reproductive Biology. 2003;107(1):19-23.
18. Conn VS, Isaramalai SA, Rath S, et al. Beyond MEDLINE for literature searches. J Nurs Scholarsh., 2003;35(2):177-82.
19. Suarez-Almazor ME, Belsek E, Homick J, et al. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials. 2000;21(5):476-487.
20. Betrán AP, Say L, Gülmezoglu AM, et al. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol. 2005;5(1):6.
21. Sampson M, Barrowman NJ, Moher D, et al., Should meta-analysts search Embase in addition to Medline? [see comment]. J Clin Epidemiol. 2003;56(10):943-55.
22. Zheng MH, Zhang X, Ye Q, et al. Searching additional databases except PubMed are necessary for a systematic review. Stroke, 2008. 39(8):e139; author reply e140. [Comment on Stroke 2008;39(6):1911-9.]
23. Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complement Ther Med. 2004;12(4):228-32.
24. Whiting P, Westwood M, Burke M, et al. Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol. 2008;61(4):357-364.
25. Balion C, Santaguida PL, Hill S, et al. Testing for BNP and NT-proBNP in the Diagnosis and Prognosis of Heart Failure. Evidence Report/Technology Assessment No. 142. (Prepared by the McMaster University Evidence-based Practice Center under Contract No. 290-02-0020). AHRQ Publication No. 06-E014. Rockville, MD: Agency for Healthcare Research and Quality; September 2006. https://www.ahrq.gov/prevention/guidelines/index.html. Accessed August 7, 2011.
26. Bruening W, Launders J, Pinkney N, et al. Effectiveness of Noninvasive Diagnostic Tests for Breast Abnormalities. Comparative Effectiveness Review No. 2. (Prepared by ECRI Evidence-based Practice Center under Contract No. 290-02-0019.) Rockville, MD: Agency for Healthcare Research and Quality; February 2006. /sites/default/files/brcadx-final-report.pdf. Accessed August 7, 2011.

Funding

Funding: Funded by the Agency for Health Care Research and Quality (AHRQ) under the Effective Health Care Program.

Disclaimer: The findings and conclusions expressed here are those of the authors and do not necessarily represent the views of AHRQ. Therefore, no statement should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

Public domain notice: This document is in the public domain and may be used and reprinted without permission except those copyrighted materials that are clearly noted in the document. Further reproduction of those copyrighted materials is prohibited without the specific permission of copyright holders.

Accessibility: Persons using assistive technology may not be able to fully access information in this report. For assistance contact EPC@ahrq.hhs.gov.

Conflicts of interest: The author has no affiliations or financial involvements that conflict with the information presented in this chapter.

Corresponding author: Rose Relevo, M.L.I.S., M.S. Oregon Health & Science University, 3181 SW Sam Jackson Park Road, Portland, OR 97217. Phone: 503–220–8262 ext. 51318. Fax: 503–346–6815. Email: relevo@ohsu.edu.

Suggested citation: Relevo R. Effective Search Strategies for Systematic Reviews of Medical Tests. AHRQ Publicaton No. 12-EHC076-EF. Chapter 4 of Methods Guide for Medical Test Reviews (AHRQ Publication No. 12-EHC017). Rockville, MD: Agency for Healthcare Research and Quality; June 2012. Also published in a special supplement to the Journal of General Internal Medicine, July 2012.