Background

Information biases, including publication bias, time-lag bias, selective outcome reporting bias, selective analysis bias, and fraud are major threats to the validity of systematic reviews. Systematic reviewers have pursued two methods approaches for dealing with information bias: 1) detecting (and correcting results for) information bias using only the identified studies (e.g., using funnel-plot based methods^1-4 or various selection models^5-7) and 2) examining trial registries, surveying researchers, and perusing the grey literature to identify unpublished study results or ongoing studies. Arguably the best way to obtain empirical data on the prevalence and impact of information bias (and perhaps to mitigate its impact) is through prospective clinical trial registries that include prospective registration of full study protocols, as well as summarized results (e.g., the National Library of Medicine ClinicalTrials.gov registry and registry networks such as International Clinical Trials Registry Platform [ICTRP]). Empirical analyses of prospective registry data can inform on the time between study completion and publication, the number of unpublished studies, the fidelity of studies to registered protocols, and the congruence of study results between result registries and publications.^8-11

Despite efforts to spur pediatric research, children remain "therapeutic orphans"¹² for whom a paucity of pediatric-specific research data is available to guide clinical decision making. Searching the grey literature improves the identification of evidence not found in the peer-reviewed literature and may prove particularly valuable for pediatric research synthesis. Empirical evidence suggests that FDA regulated and/or industry sponsored research are more likely to be found in trial registries, but compliance with mandated ClinicalTrials.gov requirements remains poor, including low rates of timely registration and posting of results.¹³ Despite efforts to incentivize pediatric research, only modest impact has been made to increase available data on pediatric drugs and devices.¹⁴ Studies conducted outside the United States may not be registered ClinicalTrials.gov but may be found in a local registry (e.g., ICTRP)¹⁵. For these reasons, we propose to search and evaluate studies from both sources.

Objectives

The objective of this methodology report is to examine the feasibility and additional utility--in terms of impact on risk of bias and strength of evidence assessments--of comprehensive searches of the ClinicalTrials.gov and ICTRP registries to supplement the evidence identified in an ongoing systematic review on tymapanostomy tubes conducted by the Brown Evidence-based Practice Center (EPC).¹⁶ Our findings will support the development of search methods, data collection, and evaluation techniques to optimize the use of trial registries in the context of systematic reviews.

Methods

Overview

We will use a systematic review that is currently being conducted by our EPC on the relationship between tympanostomy tubes and a variety of outcomes, including hearing, balance, developmental and quality of life, adverse events, and otorrhea. Our ongoing systematic review (hereafter referred to as "original report") is being conducted in accordance to IOM standards and AHRQ guidance.

We will search two clinical trial registries, ClinicalTrials.gov and ICTRP, in parallel with the original report to identify additional eligible data, comprising 1) additional studies that were not identified in the published-literature search and 2) additional information on the design or results of studies included in the original review. For newly found studies, we will record additional data and assess their risk of bias. For studies identified in the original review, we will also assess the congruence of any additional information on design or results with that in publications included in the original review, and whether the additional information would change study-level risk of bias assessments. At the level of the evidence-base, and for each pertinent exposure-outcome relationship, we will assess whether the additional information changes our overall risk of bias and strength of evidence assessments, or our conclusions.

Terminology

We use the term study to refer to the conducted research. Information about the design or results of studies may be reported in publications or in registry records. It is possible that studies identified through the registry search have no associated publications; and that studies identified in the original review have no records in ClinicalTrials.gov or ICTRP.

Registry searches

Because the registry databases are not indexed, queries can only include text words. Thus, it is necessary to translate the search of the original review, which includes text words, as well as controlled-vocabulary (MeSH) terms, to a semantically equivalent query using the ClinicalTrials.gov and ICTRP interfaces. The ClinicalTrials.gov search interface allows only for queries with a limited number of characters, and documentation on advanced searching options, such as truncation and adjacency searching, is sparse.^17,19 Glanville et al. recommend searching for intervention terms only.¹⁷ We will therefore issue a query that corresponds to the scope of the intended search in both ClinicalTrials.gov and ICTRP. Appendix A includes the literature searches from the original report and the specific search strategies to be used in ClinicalTrials.gov and ICTRP.

Screening Criteria and Evidence Map

The same eligibility criteria established for the original report will be employed to screen registry records for inclusion (Appendix B contains the original report's eligibility criteria). Initial screening will be performed by a single investigator who will peruse the title, intervention(s), and outcome(s) within each record. Records screened in during the initial phase will be included in an evidence map, which will parallel the evidence map created for the original report. This spreadsheet will capture basic intervention, outcome, study design, sample size, and whether results have been reported (but not the actual results data or risk of bias assessment). A researcher other than the one who initially screened the record in will reassess study eligibility and will extract the basic information for the evidence map.

When the evidence map is completed, the still-eligible records will be assessed to determine whether they meet the specific criteria used to determine eligibility for inclusion in the original report. We will work with the team creating the original report to harmonize samples-size and study-design criteria.

Data Extraction and Management

As noted above, all potentially relevant study records identified in registry searches will be incorporated into the original report's evidence map to include data on study design, intervention type and duration, population, outcomes, and sample size.

For relevant ClinicalTrials.gov/ICTRP citations that include results and that meet full eligibility criteria for inclusion in the original report, limited data will be extracted into the same customized forms developed and utilized in the original report in the Systematic Review Data Repository (SRDR) online system (https://srdr.ahrq.gov). Specifically, we will capture basic information about the study design, study population, intervention details (i.e., n-3 FA type, dose, and duration), reported outcomes, and results (that were not captured by articles included in the original report).

Analysis

We will provide descriptive statistics on the registry search yield and identify records/publications found exclusively in the original report, in a registry database, or in both. We will characterize registry records and associated publications that have been discontinued or are in progress/ongoing at the time of this study by detailing study initiation date and rationale for discontinuation or delay. We will, thus, categorize studies as 1) included in the original review but not found in the registry, 2) included in the original review and found in a registry but with no new results data, 3) included in the original review and found in a registry with new data, and 4) identified via the registry but not found in the original review. We will focus on the value of results data identified via registry searches, and thus highlight the congruence, or lack thereof, among data identified via the registry and found in the original report in light of additional study data identified via registry searches.

Analyses of studies included in the original review that also have a registry record

For these studies, the additional information in the registry records pertains to their design (if the registry record includes protocol information) or their findings (if the record includes results).

Information found in records can be examined against information obtained from publications to judge whether important changes in the analysis plan occurred. We will make such comparisons only with respect to 1) general design items used to inform risk of bias assessments and 2) the analysis plan of the eligible exposure-outcome relationships. The risk of bias of each study result in the original review will be evaluated based on predefined questions. We will assess whether the additional information in the registry records changes the risk of bias assessments in the original review. In the assessment for changes in the analysis plan, we will look for changes in the estimand (determined by the population to which the analysis refers [e.g., all assigned to an interventions, all receiving the intervention], the effect measure [e.g., difference in means, odds ratios for specific categorizations of continuous outcomes], and follow-up [the maximum follow up recorded]); the estimation procedure (the prescribed statistical learning procedure [e.g., taking unadjusted differences of means, adjusting in regressions and for which factors, or via stratified analysis]); and the plan for handling missing values. Deviations from the protocol's analysis plan may be suggestive of selective analysis reporting.

When results are reported in registry records, we will describe whether registry records and publications describe the same outcome concepts, and if yes, whether the results agree qualitatively (are in the same direction). We will also describe which outcome-instantiations are reported in the registry record, the publication, or both. For outcome instantiations that are reported in both, we will record whether the quantitative results are the same (within rounding error) or not.

Analyses of studies that were not included in the original review

Registry records of newly identified studies will be summarized in narrative form and added to the original report's evidence map. We will apply the same risk of bias assessments as in the original review.

Risk of bias for the evidence base and Strength of Evidence

For outcomes with new data from the registries, we will reassess the risk of bias of the evidence base and the strength of evidence using the same methodology used for the original report. We will evaluate if any additional data are likely to impact the findings of the study included in the original report. We quantify such impact as a potential increase in total study population sample size (>20%), a change in the magnitude of outcome measures (20% change in estimate or a change in direction; by meta-analysis), or a change in statistical significance (by meta-analysis). If meta-analyses are not conducted, we will assess whether the new studies fall within the range of the similar studies from the original report. If none of these conditions are met, the additional data are unlikely to directly impact the strength of evidence or the assessment of risk of bias for the evidence-base. We will describe and explain any changes to strength of evidence for any intervention and outcome relationship.

References

1. Duval S, Tweedie R. Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000 Jun;56(2):455-63. PMID: 10877304.
2. Egger M, Davey Smith G, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997 Sep 13;315(7109):629-34. PMID: 9310563.
3. Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Statistics in medicine. 2006 Oct 30;25(20):3443-57. PMID: 16345038.
4. Rucker G, Carpenter JR, Schwarzer G. Detecting and adjusting for small-study effects in meta-analysis. Biometrical journal Biometrische Zeitschrift. 2011 Mar;53(2):351-68. PMID: 21374698.
5. Copas J, Shi JQ. Meta-analysis, funnel plots and sensitivity analysis. Biostatistics. 2000 Sep;1(3):247-62. PMID: 12933507.
6. Copas JB, Shi JQ. A sensitivity analysis for publication bias in systematic reviews. Statistical methods in medical research. 2001 Aug;10(4):251-65. PMID: 11491412.
7. Hedges LV, Vevea JL. Estimating effect size under publication bias: small sample properties and robustness of a random effects selection model. Journal of Educational and Behavioral Statistics. 1996;21(4):299-332.
8. Chan AW, Hrobjartsson A, Haahr MT, et al. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. Jama. 2004 May 26;291(20):2457-65. PMID: 15161896.
9. Roest AM, de Jonge P, Williams CD, et al. Reporting Bias in Clinical Trials Investigating the Efficacy of Second-Generation Antidepressants in the Treatment of Anxiety Disorders: A Report of 2 Meta-analyses. JAMA psychiatry. 2015 May 1;72(5):500-10. PMID: 25806940.
10. Vedula SS, Bero L, Scherer RW, et al. Outcome reporting in industry-sponsored trials of gabapentin for off-label use. The New England journal of medicine. 2009 Nov 12;361(20):1963-71. PMID: 19907043.
11. Vedula SS, Li T, Dickersin K. Differences in reporting of analyses in internal company documents versus published trial reports: comparisons in industry-sponsored trials in off-label uses of gabapentin. PLoS medicine. 2013;10(1):e1001378. PMID: 23382656.
12. Shirkey H. Therapeutic orphans. Pediatrics. 1999 Sep;104(3 Pt 2):583-4. PMID: 10469793.
13. Anderson ML, Chiswell K, Peterson ED, et al. Compliance with results reporting at ClinicalTrials.gov. The New England journal of medicine. 2015 Mar 12;372(11):1031-9. PMID: 25760355.
14. Milne CP, Davis J. The pediatric studies initiative: after 15 years have we reached the limits of the law? Clinical therapeutics. 2014 Feb 1;36(2):156-62. PMID: 24529287.
15. Chan AW, Krleza-Jeric K, Schmid I, et al. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne. 2004 Sep 28;171(7):735-40. PMID: 15451835.
16. Steele D. Tympanostomy Tubes in Children Protocol. 2015. Avaialable at: https://effectivehealthcare.ahrq.gov/products/tympanostomy-tubes/
17. Glanville JM, Duffy S, McCool R, et al. Searching ClinicalTrials.gov and the International Clinical Trials Registry Platform to inform systematic reviews: what are the optimal search approaches? Journal of the Medical Library Association : JMLA. 2014 Jul;102(3):177-83. PMID: 25031558.
18. De Angelis C, Drazen JM, Frizelle FA, et al. Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors. New England Journal of Medicine. 2004;351(12):1250-1. PMID: 15356289.
19. Tse T. JS, Ripple A. Update: Expanded ClinicalTrials.gov Search Capabilities. NLM Technical Bulletin. 2002 Jul.-Aug.;327(35).

Definition of Terms

Not applicable.

Summary of Protocol Amendments

No protocol amendments have been made.

EPC Team Disclosures

Our research team has no disclosures of potential conflicts of interest.

Role of the Funder

This project was funded under Contract No. HHSA290-2015-00002I from the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services. The Task Order Officer reviewed contract deliverables for adherence to contract requirements and quality. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.

Appendixes

Appendix A. Search Strategies

Registry Searches

ClinicalTrials.gov and ICTRP
tympanostomy OR grommet OR ear tube OR pressure equalization tube OR PE tube OR myringotomy OR ventilating OR ventilation OR otitis surgery OR Middle Ear Ventilation OR tympanic tube OR Otologic Surgical Procedures OR T-tube or tubulation

Original Report

MEDLINE (5/26/15 6553 citations)

(otitis) OR ("glue ear") OR "Otitis Media with Effusion"[Mesh] OR "Otitis Media, Suppurative"[Mesh] OR "Ear, Middle/secretion"[Mesh] OR (middle and ear and (effusion* or infect* or inflame* or disease*)) OR ((OME OR SOM or AOM) AND (otitis OR ear)) OR ((mucoid* AND middle AND ear) OR (mucous AND middle AND ear) OR (seromuc* AND middle AND ear))
AND
tympanostomy OR grommet* OR ((ear or "pressure equalization" or PE or myringotomy or ventilating or ventilation) and (tube or tubes)) OR "Otitis Media with Effusion/surgery"[mesh] OR "Middle Ear Ventilation"[Mesh] OR ((middle AND (ear OR tympanic)) AND (tube or tubes)) OR "Otologic Surgical Procedures"[Mesh] OR T-tube or tubulation

Cochrane: (7/13/15 393 citations)
(otitis) OR ("glue ear") OR [mh "Otitis Media with Effusion"] OR [mh "Otitis Media, Suppurative"] OR [mh "Ear, Middle/secretion"] OR (middle and ear and (effusion* or infect* or inflame* or disease*)) OR ((OME OR SOM or AOM) AND (otitis OR ear)) OR ((mucoid* AND middle AND ear) OR (mucous AND middle AND ear) OR (seromuc* AND middle AND ear))
AND
tympanostomy OR grommet* OR ((ear or "pressure equalization" or PE or myringotomy or ventilating or ventilation) and (tube or tubes)) OR [mh "Otitis Media with Effusion/surgery"] OR [mh "Middle Ear Ventilation"] OR ((middle AND (ear OR tympanic)) AND (tube or tubes)) OR [mh "Otologic Surgical Procedures"] OR T-tube or tubulation

CINAHL (7/13/15 852 citations)
(MH "Otitis") OR (MH "Otitis Media with Effusion") OR (MH "Otitis Media") OR otitis OR ("glue ear") OR (MH "Ear, Middle") OR (middle and ear and (effusion* or infect* or inflame* or disease*)) OR ((OME OR SOM or AOM) AND (otitis OR ear)) OR ((mucoid* AND middle AND ear) OR (mucous AND middle AND ear) OR (seromuc* AND middle AND ear))
AND
tympanostomy or myringotomy OR (MH "Middle Ear Ventilation") OR grommet* OR ((ear or "pressure equalization" or PE or myringotomy or ventilating or ventilation) and (tube or tubes)) OR ((middle AND (ear OR tympanic)) AND (tube or tubes)) OR (MH "Ear Surgery") OR T-tube or tabulation

EMBASE (7/14/15 5556)
Otitis OR 'otitis media'/exp OR glue ear OR (middle and ear and (effusion* or infect* or inflame* or disease*)) OR ((OME OR SOM or AOM) AND (otitis OR ear)) OR ((mucoid* AND middle AND ear) OR (mucous AND middle AND ear) OR (seromuc* AND middle AND ear))
AND
tympanostomy OR 'tympanostomy tube'/exp OR 'myringotomy'/exp OR 'middle ear ventilation'/exp OR grommet* OR ((ear or "pressure equalization" or PE or myringotomy or ventilating or ventilation) and (tube or tubes)) OR ((middle AND (ear OR tympanic)) AND (tube or tubes)) OR T-tube or tabulation

Appendix B. Key Questions and Eligibility Criterial of the Tympanostomy Tubes Report

The Key Questions

Question 1

What is the effectiveness of tympanostomy tubes, compared to watchful waiting, in children with chronic otitis media with effusion on hearing and vestibular outcomes, quality of life and patient-centered outcomes, and intermediate outcomes?

1. What factors (such as age, age of onset, duration of effusion, comorbidities, and sociodemographic risk factors) predict which children are likely to benefit most from the intervention?
2. Does obtaining a hearing test help identify which children are more likely to benefit from the intervention?

Question 2

What is the effectiveness of tympanostomy tubes, compared to watchful waiting with episodic or prophylactic antibiotic therapy in children with recurrent acute otitis media on hearing and vestibular outcomes, quality of life and patient-centered outcomes, and intermediate outcomes?

1. What factors (such as age, age of onset, number of recurrences, presence of persistent middle ear effusion, comorbidities, and sociodemographic risk factors, history of complications of acute otitis media, antibiotic allergy or intolerance) identify children who are most likely to benefit from the intervention?
2. Does obtaining a hearing test help identify which children are more likely to benefit from the intervention?

Question 3

What adverse effects and complications are associated with inserting tympanostomy tubes in children with either chronic otitis media with effusion or recurrent acute otitis media?

Question 4

Do water precautions reduce the incidence of tympanostomy tube otorrhea, affect quality of life, or alter the pathogens cultured from otorrhea?

Question 5

In children with tympanostomy tube otorrhea, what is the comparative effectiveness of topical antibiotic drops versus systemic antibiotics or watchful waiting on duration of otorrhea, quality of life, or need for tube removal?

Eligibility Criteria

For all KQs, the Eligibility Criteria used will be:

Populations

• All KQs: Ages: infant (28 days to 12 months), toddler (13 months to 2 years), early childhood (2 to 5 years), middle childhood (6 to 11 years), early adolescence (12 to 18 years).
• All KQs: Subpopulations:
  • Trisomy 21, cleft palate, other craniofacial anomalies, primary ciliary dyskinesia
  • High-risk children: preexisting hearing loss, speech/language problems, or developmental disorders.
  • Sociodemographic risk factors
• KQ 1: Children with chronic OME (allow study-specific definitions of "chronic" but use as a standard definition effusion that persists for 3 months or longer¹)
• KQ 2: Children with recurrent AOM (allow study-specific definitions of "recurrent" but use as a standard definition Three or more well-documented and separate AOM episodes in the past 6 months or at least four well-documented and separate AOM episodes in the past 12 months with at least one in the past 6 months¹)
  • With middle ear effusion
  • Without middle ear effusion
• KQ 3, 4: Children with tympanostomy tubes placed for OME or AOM
• KQ 5: Children with tympanostomy tube otorrhea
  • Postoperative
  • Symptomatic or asymptomatic

Interventions/Exposures

• KQ 1, 2, 3: Myringotomy with tympanostomy tube placement with or without adenoidectomy
  • Short-term (grommet-type)
  • Medium-term
  • Long-term (T-tubes)
  • Emerging technologies
    • Laser fenestration
• KQ 4: Water precautions
  • Avoidance of high-risk activities
  • Ear plugs, headbands, other canal occlusion methods
  • Ototopical antibiotic prophylaxis
• KQ 5: Ototopical preparations
  • Antibiotics (common examples listed)
    • Ciprofloxacin 0.2%
    • Ofloxacin otic 0.3%
  • Combination products (common examples listed)
    • Ciprofloxacin 0.3% + dexamethasone 0.1%
    • Ciprofloxacin 0.2% + hydrocortisone 1%
  • Other – non FDA approved such as:
    • Hydrocortisone + bacitracin + colistin
    • Hydrocortisone + oxytetracycline + polymyxin B
    • Neomycin sulfate + polymyxin B sulfate +hydrocortisone

Comparators

• KQ 1:
  • Watchful waiting
  • With and without adenoidectomy
• KQ 2:
  • Systemic antibiotics for recurrent episodes of AOM
  • Prophylactic antibiotics
• KQ 3: No comparator
• KQ 4:
  • No water precautions
  • Ear plugs
  • Prophylactic ear drops after water exposure
  • Avoidance of higher risk activities
• KQ 5:
  • Watchful waiting
  • Oral (systemic) antibiotics

Outcomes

• KQ 1, 2: Indications for tympanostomy tubes
  • Hearing and vestibular outcomes
    1. Improved hearing levels (audibility)
    2. Tests of auditory perception and discrimination (clarity)
    3. Balance and coordination (vestibular function)
  • Quality of life and patient-centered outcomes
    1. Global and otitis-specific child and parental quality of life
    2. Speech and language outcomes
    3. Educational achievement
    4. Behavioral outcomes such as disobedience, enuresis, or tantrums
  • Intermediate outcomes
    1. Prevalence of middle ear effusion (per ear, per patient)
    2. Recurrent AOM/otorrhea (KQ 2)
    3. Need for replacement of tympanostomy tubes
• KQ 3: Adverse effects of intervention(s):
  1. Intraoperative and immediate postoperative anesthetic and surgical adverse events
  2. Otorrhea
  3. Blockage of the tube lumen
  4. Granulation tissue
  5. Premature extrusion
  6. Tympanostomy tube displacement into the middle ear
  7. Persistent perforation of the tympanic membrane, possibly requiring surgical closure
  8. Myringosclerosis
  9. Tympanic membrane atrophy, atelectasis and retraction pockets
  10. Worsened hearing thresholds
• KQ 4: Water precautions:
  • Final health or patient-centered outcomes
    1. Child and parental quality of life
  • Intermediate outcomes
    1. Otorrhea prevalence
    2. Pathogens cultured from otorrhea
• KQ 5 Treatment of otorrhea:
  • Final health or patient-centered outcomes
    1. Global and otitis-specific child and parental quality of life
  • Intermediate outcomes
    1. Duration of otorrhea
    2. Need for removal of tympanostomy tube

Timing

Any duration of followup

Setting

Primary and specialty care

Study Design

• Randomized controlled trials
• Nonrandomized comparative studies, prospective or retrospective
• Observational cohorts, longitudinal, prospective or retrospective (KQ 1a and 1b, KQ 2a and 1b, KQ 3)

Comments about the Eligibility Criteria

The preliminary literature search identified a large number of observational studies of various types. There is interest in comparative effectiveness of tympanostomy tubes in high-risk and at-risk populations, and in defining harms. Children at high risk of chronic otitis media or recurrent acute otitis media have been excluded from randomized controlled trials. Many randomized trials are relatively small, limiting their ability to define risks of less common harms. Further guidance from the systematic review's Technical Expert Panel (TEP) members and full-text review will be needed regarding criteria for inclusion of observational studies. Inclusion of nonrandomized comparative trials will likely be necessary primarily for KQs 1b and 2b (hearing testing) and in special populations. With input from the TEP, study-design and sample-size filters will be implemented, as needed, to contain the scope of the review to a manageable size without compromising its validity. For example, we may examine excluding papers reporting small surgical case series or case reports without control groups from the KQs that address comparative effectiveness.