Skip to main content
Effective Health Care Program

Observational Studies: Empirical Evidence of Their Contributions to Comparative Effectiveness Reviews

White Paper

People using assistive technology may not be able to fully access information in these files. For additional assistance, please contact us.

Structured Abstract


Although observational studies are increasingly being used to address gaps in the evidence from randomized controlled trials, the effect they have on the results and conclusions of systematic reviews is unclear. Our objectives were to evaluate: (1) how often observational studies are searched for and included in comparative effectiveness reviews (CERs); (2) the rationale for including or excluding observational studies; (3) how data from observational studies are appraised, analyzed, and graded; and (4) the impact of observational studies on the strength of evidence (SOE) and overall conclusions.


In June 2013 we searched the Effective Health Care Program Web site for final reports of CERs. One reviewer screened titles, abstracts, and Key Questions for CERs that examined a therapeutic or preventive intervention provided at an individual patient level. We selected a 25 percent sample of the most recent eligible CERs. Data were extracted by one reviewer and verified by a second reviewer. We extracted the number and type of study designs included and the approaches to quality assessment, presentation of results, and grading the SOE. We identified all comparisons for which both trials and observational studies provided data, and evaluated whether observational studies had an impact on the SOE and conclusions. We applied an RCT filter to the searches to determine the impact on search yield.


From 129 records we identified 88 eligible CERs. Our final sample included 23 CERs published since November 2012. EPCs searched for observational studies in 20 CERs, of which 18 included a median of 11 (interquartile range: 2, 31) studies. Sixteen CERs incorporated the observational studies in their SOE assessments. We identified 78 comparisons from 12 CERs for which both trials and observational studies provided evidence; observational studies had an impact on SOE and conclusions for 19 (24 percent) of the comparisons. There was considerable diversity across the CERs regarding decisions to include or exclude observational studies, the study designs considered, and the approaches used to appraise, synthesize, and grade the SOE. Applying an RCT filter reduced the search yield by 65 percent (range 39 to 92 percent).


Reporting guidelines and methods guidance relating to observational studies are needed in order to ensure clarity and consistency across Evidence-based Practice Centers. It was not always clear that the inclusion of observational studies added value in light of the additional resources needed to search for, select, appraise, and analyze such studies.