Powered by the Evidence-based Practice Centers
Evidence Reports All of EHC
Evidence Reports All of EHC



Comparison of Effects as Evidence Evolves From Single Trials to High-Quality Bodies of Evidence

White Paper Mar 31, 2015
Download PDF files for this report here.

Page Contents

People using assistive technology may not be able to fully access information in these files. For additional assistance, please contact us.

Structured Abstract


The objective of our methods project was to use a diverse sample of medical interventions to assess empirically whether first trials rendered substantially different treatment effect estimates than reliable, high-quality bodies of evidence.


We employed a meta-epidemiological study design using 100 bodies of evidence from Cochrane reports that had been graded as high quality of evidence. To determine the concordance of effect estimates between first and subsequent trials, we applied both quantitative and qualitative approaches. For quantitative assessment, we used Lin's concordance correlation and calculated z-scores; to determine the magnitude of differences of treatment effects, we calculated standardized mean differences (SMDs) and ratios of relative risks. We determined qualitative concordance based on a 2-tiered approach incorporating changes in statistical significance and magnitude of effect.


First trials both over- and under-estimated the true treatment effects in no discernible pattern. Nevertheless, depending on the definition of concordance, effect estimates of first trials were concordant with pooled subsequent studies in at least 33 percent but up to 50 percent of comparisons. The pooled magnitude of change as bodies of evidence advanced from single trials to high-quality bodies of evidence was 0.16 SMD (95% confidence interval [CI], 0.12 to 0.21). In 80 percent of comparisons the difference in effect estimates was smaller than 0.5 SMDs. In first trials with large treatment effects (>0.5 SMD), however, estimates of effect substantially changed as new evidence accrued (mean change 0.68 SMD, 95% CI, .50 to 0.86).


Results of first trials often change but the magnitude of change, on average, is small. Exceptions are first trials that present large treatment effects which often dissipate as new evidence accrues.

Project Timeline

Testing the Predictive Validity of Strength of Evidence Grades

Oct 29, 2013
Topic Initiated
Oct 29, 2013
Mar 31, 2015
White Paper
Sep 14, 2015
Page last reviewed November 2017
Page originally created November 2017

Internet Citation: White Paper: Comparison of Effects as Evidence Evolves From Single Trials to High-Quality Bodies of Evidence. Content last reviewed November 2017. Effective Health Care Program, Agency for Healthcare Research and Quality, Rockville, MD.

Select to copy citation