Skip to main content
Effective Health Care Program

Methods for Evaluating Natural Experiments in Obesity: A Systematic Evidence Review

Systematic ReviewDraft

Open for comment through Oct. 25, 2017

This report is available in PDF only (4.7 MB). People using assistive technology may not be able to fully access information in this file. For additional assistance, please contact us.

Purpose of Review

To characterize studies of programs and policies in obesity prevention and control in terms of their data sources, data linkages, measures reported, study designs, and analytic approaches, and identify needed methodological advances.

Key Messages

  • 136 natural experiments, 107 experimental studies, and 18 other studies evaluated relevant programs, policies, or built environment changes.
  • 90 data sources met criteria for a data system (source exists, available for research, sharable, outcomes of interest).
  • 33% of US data systems were linked to secondary data.
  • Outcome measures included dietary activity (135 studies), physical activity (132 studies), childhood weight (77 studies), and adult weight (30 studies).
  • Natural experiments most commonly used regression models comparing exposed and unexposed groups at one time.
  • Natural experiments generally had moderate risk of selection bias, and high risk of bias for losses to follow-up.
  • Research could be advanced by more use of data dictionaries, reporting standards on data linkage, long-term obesity-related outcomes, and study designs with multiple pre- and post-exposure time points.

Structured Abstract

Objectives

Obesity is an enormous public health problem among adults and children. Our objective was to systematically review studies evaluating programs and policies addressing obesity prevention and control in terms of their population-based data sources, use of data linkages, measures reported, study designs, and analytic approaches. The overarching goal of the review was to identify methodological advances that could strengthen research that uses natural experiments to evaluate the effectiveness of policies and programs to prevent and control obesity.

Data sources

We systematically searched PubMed, CINAHL, PsycINFO, and EconLit from 2000 to November 4, 2016 to identify all US and non-US studies of programs or policies targeting obesity prevention and control in people of all ages and in any setting.

Review methods

Two independent reviewers screened abstracts and full-text articles. We required articles to be in English, address a program, policy or built environment change, include 100 or more study subjects, and have a defined comparison or unexposed group. We used the Effective Public Health Practice Project (EPHPP) tool to rate studies for their risk of bias. This tool rates studies for their ability to draw causal inferences about program effectiveness.

Results

The search identified 19,037 unique citations. Of the 261 studies (reported in 276 articles) eligible for inclusion (171 US and 90 non-US), 136 (52 percent) were natural experiment studies, 107 (41 percent) were experimental studies (randomized or non-randomized controlled trials), and 18 (7 percent ) had other study designs that did not fall into either of the other categories.

Data Sources: We identified 126 secondary data sources and 19 sharable primary data sources, totaling 99 sharable data sources after duplicates were removed. Ninety (63 US and 27 non-US) data sources met criteria for a data system (data source exists, is available for research, is sharable, and contains outcomes of interest). Fifty-nine percent of the 63 US data systems contained at least one of the main measures for weight or body mass index (BMI) in adults or children, or dietary or physical activity behaviors. Fifty-one percent of the 63 US data systems included at least one outcome related to the food-environment, physical activity environment, commuting behavior, or purchasing behavior, or included information about a relevant exposure in a policy, program, or built environment change. These 63 US data systems often reported more than one outcome. One third of the 63 US data systems were linked with a secondary data source or system other than the primary data source. Most studies that linked their data systems with external data systems used an individual-level key or a geographic allocation.

Outcomes/Measures: Of the 261 included studies, we identified 77 studies with childhood weight measures, 30 studies with adult weight measures, 132 studies with physical activity measures, and 135 studies with dietary measures. Twenty-nine of 261 studies reported on outcomes related to the food environment, physical activity environment, commuting behavior, or purchasing behavior.

Study Design and Methods: Natural experiment studies most commonly used regression models that compared exposed and unexposed groups at a single time point (n=47). Difference-in-differences approaches that compared exposed and unexposed groups before and after an exposure were used in 41 studies (30%), while 37 studies (27%) used pre/post designs that compared one group before and after an exposure. Most natural experiment studies were rated as having a “weak” global rating (i.e., high risk of bias), with 62% having a weak rating for handling of withdrawals and dropouts, 40% having a weak rating for study design, 42% having a weak rating for confounding, and 27% having a weak rating for data. Experimental studies were rated as “strong” (low risk of bias) in study design, control of confounding, and data collection methods, but were weaker with regard to blinding and selection bias. We identified a number of methodological and analytic advances that would help to strengthen efforts to estimate the effect of programs, policies, or built environment changes on obesity prevention and control, such as consistent use of data dictionaries, reporting standards on linkage methods of data sources, data sources with long term public health surveillance of obesity and health behavioral outcomes, and use of study designs with multiple pre- and post- exposure time points.

Conclusions

Our systematic review identified a large number of natural experiment studies (n=136) and data sources (n=99) that have been used to estimate the effect of programs, policies, or built environment changes on obesity prevention and control. The studies used a wide variety of outcome measures and analytic methods, often with substantial risk of bias. The findings reinforce the need for methodological and analytic advances that would strengthen efforts to improve obesity prevention and control.