Powered by the Evidence-based Practice Centers
SHARE:
FacebookTwitterFacebookPrintShare

Analysis of Requirements for Coverage with Evidence Development (CED) – Topic Refinement

Key Questions Sep 7, 2022
A person reviewing the document

Documents

Background

When the Centers for Medicare & Medicaid Services (CMS) decides that there is insufficient evidence to conclude definitively that an item or service is “reasonable and necessary,”1 it may issue a Coverage with Evidence Development (CED) decision.2 CED enables the Medicare program to cover items and services on the condition that they are furnished in the context of approved clinical studies or with the collection of additional clinical data. A CED decision is a National Coverage Determination (NCD) that allows patients to access these select medical items and services, with coverage, on the condition that there is prospective collection of agreed upon clinical data.

In brief, the CED process was designed in 2005. The stated goals were to generate data so that CMS could verify the appropriateness of the use of an item or service, consider future changes in coverage for an item or service, and generate clinical information to improve the evidence base for or against the use of an item or service.3 With its update in 2006, CMS outlined two subtypes of CED. The first was coverage with appropriateness determination, or CAD. Here, CMS agrees that an item or service is reasonable and necessary but requests clinical data that are not generally available in claims to ensure appropriate use. The second is coverage with study participation, or CSP. In this case, CMS would ask that additional data be generated in the context of research.4, 5 CMS no longer differentiates between these two activities, as described in the most recent guidance document.6

In April 2012, the Obama Administration stated that CMS should better define “the parameters and guidance for [CED] so it can be used more widely and effectively as a driver of innovation.”7 Soon thereafter, in November 2012, CMS released revised guidance clarifying that CED should be carried out via prospective studies and that a CED cycle is completed when CMS has sufficient evidence to reconsider the coverage decision. The updated guidance document of 2014 describes the updates to the program that came about following input from the Medicare Evidence Development and Coverage Advisory Committee (MEDCAC) and describes related, but distinct, programs like the coverage that is part of the Investigational Device Exemption (IDE) program.6

The 2014 guidance acknowledges that CMS is increasingly challenged by requests for coverage of items and services when the expectations of interested parties are not adequately supported by the existing evidence base. With the CED program, CMS can provide support for items and services that are likely to benefit the Medicare population, but where the available evidence base is insufficient to support coverage. The goal of every CED is to provide coverage of promising technologies while evidence is collected to determine if the technology is reasonable and necessary for the stated indications/outcomes. This process is intended to expedite beneficiary access to innovative items and services while assuring that the technology is provided to clinically appropriate patients, meaning those who are likely to benefit. The process includes protections to reduce the risks inherent to new technologies, or to new applications of existing technologies. 

When a CED NCD is issued, CMS publicly posts the Decision Summary that describes what evidence CMS will accept during the CED process and what questions, at a minimum, must be addressed for coverage of the technology or drug within the context of a trial or other prospective data collection. For example, the recent CED NCD on monoclonal antibodies directed against amyloid for the treatment of Alzheimer’s disease indicates that, for coverage, the investigators must answer the questions: Does the antiamyloid monoclonal antibody meaningfully improve health outcomes for patients in broad community practice? Do benefits, and harms such as brain hemorrhage and edema, associated with use of the antiamyloid monoclonal antibody, depend on characteristics of patients, treating clinicians, and settings? How do the benefits and harms change over time?8 This specific Decision Summary specifies that the product may be covered in a randomized controlled trial conducted under an investigational new drug (IND) application when a surrogate endpoint is used, or may be covered in CMS-approved prospective comparative studies, with data collected in a registry, when the outcomes of interest are those of direct clinical importance to a patient.

Each Decision Summary also includes the list of specific study design requirements that should be met to assure study integrity.2 The current 13 requirements are listed in item VI, Requirements for CED under Section 1862(a)(1)(E).2 (Medicare Coverage Document) CMS is confident that the CED NCD process is sound; this report describes our response to the request from CMS for recommendations about updates to the CED study design requirements.

Purpose

The objective of this report is to describe our process and resulting recommendations to CMS for an update to the CED study design requirements. We aimed to refine the study design requirements so that investigators are efficient in completing studies that contribute to an evidence base with the goal of ending the CED process when there is 1) sufficient evidence for a coverage NCD; 2) sufficient evidence for a non-coverage NCD; or 3) a decision to defer the coverage decision to a Medicare Administrative Contractor (MAC). Our goal for the set of requirements is that they will guide investigators to collect and use data generated in the care of patients to produce strong evidence about the health outcomes from use of products by Medicare beneficiaries, with integrity in the scientific process and transparency at all stages.

Questions

Guiding Questions

  • Guiding Question 1: What are the strengths and limitations of the current CED criteria (that we now refer to as “requirements”)?
  • Guiding Question 2: What criteria (“requirements”) are used by similar decision-making bodies?

Key Questions (KQ)

  • KQ1: What revisions to the CED criteria (“requirements”) may best address the limitations while preserving the strengths?
  • KQ2: How might the revised criteria (“requirements”) be evaluated in the future?

Methods

We generated revised requirements using the following process: searching for and reviewing relevant literature, drafting revised requirements based on the literature review, gathering input from Key Informants (KIs), revising the requirements, and delivering the revised requirements to CMS for presentation to the MEDCAC. The details of the process follow:

Literature Search

  1. We started by conducting a targeted search of the English-language literature using PubMed and search terms for CED. The search included literature from 1978 through July 1, 2022. We reviewed the reference lists to include any articles that were relevant to our questions about CED. We also performed an expanded search adding terms as shown in Appendix 1.
  2. The expanded search added many citations; therefore, we reviewed the abstracts of a random sample of 100 citations to estimate the incremental yield for identifying additional reports about study design recommendations (for a CED) or describing CED policies outside of the U.S. We excluded abstracts if they focused on aspects of the CED process unrelated to study design and conduct, and if they were evaluations of costs of therapies, analyses of cost-effectiveness, or were about economic or econometric valuation methods, as these are less relevant to the CMS process.
  3. We looked for guidance documents about the production of real-world evidence, which we thought to be highly relevant because the data used to answer CED questions will often, although not always, be generated in the usual care of patients. Most of these documents were cited in the articles found in the initial literature search described above, and others were recommended by team members and our team’s advisors, including those from key professional societies and national and international organizations.
  4. We identified and reviewed grey literature describing the CED polices of other countries, limited to documents published in English. We first identified candidate countries from three international review articles of CED schemes.9-11 The countries were Australia, Belgium, Canada, England, France, Germany, the Netherlands, Spain, Sweden, and Switzerland. We then searched English-language government websites for health technology assessment bodies located in these countries to identify documentation of CED policies. We supplemented this government website search by asking colleagues in the health technology assessment (HTA) field (based in Canada, England, the Netherlands, Sweden, and Switzerland) to direct us to any documentation of CED policies in their respective countries.

Generating Candidate Requirements

  1. We reviewed the 13 requirements in the existing CED guidance (listed as a.– m.) and assigned labels to these requirements (e.g., data, protocol) to indicate the goals of each, expecting that each requirement contributes to assuring that the submitted study has scientific integrity.
  2. The team members divided the identified literature based upon each person’s expertise. Each then extracted recommendations that are intended to lead to the production of a strong body of evidence, as well as recommendations for evidence generation that are used in international settings in the context of coverage decisions. As described above, we focused on extracting recommendations for generating strong real-world evidence. Where needed, we used the following definitions for real-world data and real-world evidence as defined by the Food and Drug Administration (FDA).
    • “Real-world data are the data relating to patient health status and/or the delivery of health care that are routinely collected.”12
    • “Real-world evidence is the clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of real-world data.”12

    We did not extract recommendations for the conduct of traditional randomized controlled trials, such as those that are conducted for regulatory marketing approval, as we expect that these requirements are well-known to sponsors.

  3. We then labelled the extracted recommendations, adding additional labels as needed.
  4. The recommendations were aggregated and sorted by their labels. We then crafted one or more requirements to correspond to each of the labels based on the language of the recommendations and the perceived intent in the source documents.
  5. The co-investigators and advisors reviewed the draft requirements and made suggestions that were iteratively discussed and incorporated to assure that there was not duplication of the requirements.
  6. We mapped the new CED requirements onto the existing requirements and noted the rationale for changes.

Key Informant (KI) Process

Within the AHRQ Evidence-based Practice Center (EPC) Program, the Key Informant (KI) role is to provide stakeholder perspectives and input. The EPC solicits input from KIs when developing questions for systematic review or when identifying high priority research gaps and needed new research. KIs are not involved in analyzing the evidence or writing the report.

  1. The team members, advisors, CMS, and AHRQ generated a list of key stakeholders with varied interests whose collective knowledge would be valuable to the process. The group generated candidate names of experts who could represent these stakeholders. Federal content experts were drawn from the FDA, the National Institute on Aging, and the National Institute on Minority Health and Health Disparities within the Department of Health and Human Services. Non-federal stakeholders were sought to provide perspectives and expertise such as patient/consumer advocacy, health registries, similar international efforts, the use of real-world data, medical specialty societies, health technology assessment agencies, commercial health plans, and health policy. The list of Key Informants who participated in this project will be listed in the final report. Representatives from the life sciences industries were not included at the KI stage, but will have the opportunity to provide input during the public review period.
  2. In preparation for virtual KI Panel meetings, we sent the KIs a feedback form via Qualtrics to introduce them to the proposed new requirements and seek their feedback. The introduction to the task indicated that we sought feedback as to whether each of the proposed new requirements should be required for a study done for the purpose of CED. The Qualtrics tool asked the KIs to assess each of the new requirements on a scale of 1 to 3 where 1 is essential, 2 is important but not essential, and 3 is not needed. The KIs were also asked if each requirement was: 1 –clear as written, or 2- in need of textual revision. They were given space to comment on each item. KIs were also given a text box to convey comments on the body of requirements and their sufficiency for meeting the goal of CMS, which is to receive studies that contribute to the evidence for a decision.
  3. For each KI meeting, we gave the KIs a summary of their collective feedback.
  4. The principal investigator moderated the KI call and highlighted the comments from the KIs that required further discussion. The call was recorded and automatically transcribed as a reference for the revision of the requirements.
  5. The discussion points from the KI meeting were used to further revise the requirements and a second feedback form, using Qualtrics, was sent to the KIs to confirm that the revisions were consistent with their recommendations.
  6. The results of this second feedback were used to determine the degree of consensus about the importance of the proposed requirements and identify points of disagreement that might be appropriate for discussion with the MEDCAC.

Results

Results of Literature Search and Recommendations Extraction

From our initial literature search and the review of reference lists in relevant articles, we found 27 articles that were appropriate for data extraction. We also identified 8014 articles with our expanded search strategy. The abstract review of the random subset did not identify any additional articles for inclusion. The majority of articles were excluded because they did not have sufficiently granular recommendations about the process of conducting studies within a CED. Other articles were excluded because they primarily addressed the evaluation of costs of therapies, cost-effectiveness, or other economic or econometric valuation methods.

The identified literature included the existing guidance documents from CMS.2 We also included the framework and guidance documents from the National Evaluation System for health Technology Coordinating Center (NESTcc), which was a collaboration between FDA and medical device manufacturers.13, 14 We extracted recommendations from three reports about generation of real-world evidence that were joint publications from the International Society of Pharmacoepidemiology (ISPE) and the International Society of Pharmaceutical Outcomes Research (ISPOR).15-17 We also extracted guidance statements from three publications from the National Academies of Medicine18-21 and multiple guidance documents from the FDA.12, 22-28 We reviewed the Grace principles 5.1 document about registry design recommendations,29, 30 as well as a framework for use of evidence for coverage decisions by Pearson and colleagues in 201831 and a framework about regulatory use of evidence from the Margolis Center for Health Policy Research at Duke University.32 Other key documents were the 21st Century Cures Act33 and the transcript of the MEDCAC meeting in 2012 when the CED process was last discussed.34

The most relevant international publications were the work by Drummond and colleagues, which was part of an initiative by EU Horizon 2020 COMED (Pushing the Boundaries of Cost and Outcome Analysis of MEDical Technologies);35 and the guidelines describing the United Kingdom’s (UK) National Institute for Health and Care Excellence (NICE) Cancer Drugs Fund36 and the Innovative Medicines Fund.37 The EU Horizon 2020 COMED project included development of theory regarding CED, a systematic review, and 25 semi-structured interviews with decision makers and economists, and thus provided rich context for this present work.11, 38

We found little literature describing CED policies in countries outside of the US that were accessible in English. Indeed, two review articles relied on expert interviews to identify CED policies given the absence of written policies.9, 11

The recommendations ranged from 8 to more than 50 discrete recommendations within a given publication.(Appendix 2) We found that the recommendations addressed the following topics, which we used as labels: context, data, data registry, data source, data sufficiency, data validation, design, bias, blinding, censoring, controls, definitions, exposures, outcomes, monitoring, population, precision, randomization, sensitivity analyses, dissemination, experts, generalizability, governance, interpretation, investigators, protocol, reporting, reproducibility, and transparency. The team had 172 recommendations to distill into a parsimonious set of proposed CED requirements.

The proposed requirements (Table 1) were contextualized as being requirements for a study that is designed to address one or more of the question(s) stated in a CED NCD. The order of the requirements is consistent with the order in which an investigator would approach the problem of framing the questions and generating evidence. The proposed requirements reflect best practices for both experimental and observational designs to efficiently generate evidence that contributes to the key decision as to whether CMS should: 1) end the CED due to sufficient evidence for a coverage NCD; 2) end the CED due to sufficient evidence for a non-coverage NCD, or 3) defer the coverage decision to a MAC. These are framed as requirements although CMS has the discretion to adjust or delete any requirement in a specific situation.

Table 1. Proposed Requirements for CED Studies for Presentation to Key Informants (KIs)

 

Tag

Requirement

A

Team

The study is sponsored by investigators with the resources and skills to complete it successfully.

B

Communication

A written plan describes scheduled communication by the investigators with CMS throughout the evidence generation period for review of study milestones.

C

Governance

The information governance and data protection requirements are established in writing and included in the study protocol.

D

Context

The rationale for the study is supported by scientific and medical evidence and its results are expected to fill a knowledge gap.

E

Context

CMS and investigators agree upon the evidentiary threshold for the stated question. This reflects the clinically relevant difference in the key outcome(s) relative to the chosen comparator and the targeted precision. 

F

Outcome(s)

The key outcome(s) for study are those that are clinically important to patients and durable.  A surrogate outcome that reliably predicts key clinical outcomes might be appropriate for some questions. 

G

Protocol

A protocol describing the data source(s), key outcome(s), and key elements of design, at a minimum, is publicly posted on the CMS website

H

Population

The studied population reflects the intended users of the product and also the racial, gender, and socio-economic diversity of the Medicare beneficiary population including older adults, individuals on dialysis, and disabled younger persons when relevant to the questions.

I

Consent

The investigators obtain meaningful informed consent from patients regarding the risks associated with the study items and/or services, and the use and eventual disposition of the collected data, unless an institutional review board deems it to not be human subjects research or eligible for waiver or alteration of consent.

J

Data source

When feasible and appropriate for answering the CED question, data for the study should come from the real-world practice of medicine including from practitioners diverse in experience and diverse sites of care delivery.

K

Data quality

The data are of sufficient size, completeness, continuity, and accuracy to assess participant eligibility, key prognostic and predictive factors, exposure to therapy (including a unique device identifier, if relevant), and key outcomes.

L

Data use

The investigators validate algorithms for the measurement of key exposures and outcomes. When infeasible, the investigators assess the performance of the operational definition of the variable or cite relevant validation exercises.

M

Design

The study design is selected to efficiently generate the needed evidence. Expected designs include pragmatic trials with randomization and blinding when feasible, single arm intervention studies with contemporaneous comparator groups, prospective cohort studies with contemporaneous comparison groups, self-controlled designs where appropriate, or retrospective cohort studies with contemporaneous comparators nested within registries.

N

Analysis

The investigators minimize the impact of confounding and biases on inferences by using rigorous design and statistical techniques.

O

Design: Heterogeneity of treatment effect

The investigators pre-specify subpopulations for study if they expect that key outcomes in response to treatment will be meaningfully different in those subgroups compared with the majority population. Otherwise, investigators will explore for heterogeneity of treatment effect if there are not a priori hypotheses.

P

Design: registry

When relevant, investigators follow best practices for establishing and maintaining a registry.

Q

Reproducibility

The investigators demonstrate reproducibility of results from the study by conducting alternative and sensitivity analyses, and/or using other data sources.

R

Reporting

The results and analytic code are submitted for peer review using a reporting guideline appropriate for the design.

S

Replication

The reporting is structured to enable replication by a regulator, payor, or another research team.

 T

Sharing

The investigators commit to sharing data, methods, and analytic code with CMS. Other sharing is to follow the rules of the funder and the institutional review boards.

Regulation

The study is not designed to exclusively test toxicity or disease pathophysiology in healthy individuals. Such studies may meet this requirement only if the disease or condition being studied is life threatening as defined in 21 CFR §312.81(a) and the patient has no other viable treatment options.

V

Regulation

The research study complies with all applicable Federal regulations concerning the protection of human subjects found in the Code of Federal Regulations (CFR) at 45 CFR Part 46. If a study is regulated by the Food and Drug Administration (FDA), it is also in compliance with 21 CFR Parts 50 and 56.  

Table 2 illustrates our comparison of the existing requirements and the proposed requirements showing that we moved from 13 requirements to 22 requirements, including the two requirements that cite specific regulations (U and V). The increase in the count of requirements was partially due to our decomposing the content of some of the existing requirements so that each requirement reflected a single concept with the goal of improved clarity. Additionally, we included recommendations that more completely reflect contemporary best practices regarding transparency and reproducibility.

Table 2. Comparison between Existing and Proposed Requirements as Presented to Key Informants and Rationale for Changes

Tag for Existing Requirement

Existing Requirement

Proposed Requirement

Rationale

Context

a. The principal purpose of the study is to test whether the item or service meaningfully improves health outcomes of affected beneficiaries who are represented by the enrolled subjects.  

E.CMS and investigators agree upon the evidentiary threshold for the stated question. This reflects the clinically relevant difference in the key outcome(s) relative to the chosen comparator and the targeted precision. 

Aimed to specifically require that the population is representative by promoting to a separate requirement.

Population

a. The principal purpose of the study is to test whether the item or service meaningfully improves health outcomes of affected beneficiaries who are represented by the enrolled subjects.  

 

H. The studied population reflects the intended users of the product and also the racial, gender, and socio-economic diversity of the Medicare beneficiary population including older adults, individuals on dialysis, and disabled younger persons when relevant to the questions.

Split for clarity.

Context

b. The rationale for the study is well supported by available scientific and medical evidence.

D. The rationale for the study is supported by scientific and medical evidence and its results are expected to fill a knowledge gap.

b. and c. are combined as information that should be included in the background of a protocol to establish the need for the study

Context

c. The study results are not anticipated to unjustifiably duplicate existing knowledge.

D. (as above)

(as above)

Design

d. The study design is methodologically appropriate and the anticipated number of enrolled subjects is sufficient to answer the research question(s) being asked in the National Coverage Determination.

M. The study design selected is to efficiently generate the needed evidence. Expected designs include pragmatic trials with randomization and blinding when feasible, single arm intervention studies with contemporaneous comparator groups, prospective cohort studies with contemporaneous comparison groups, self-controlled designs where appropriate, or retrospective cohort studies with contemporaneous comparators nested within registries.

The sufficiency of the subjects is included above in context (as precision); focus here is on efficiency and study design.

Team

e. The study is sponsored by an organization or individual capable of completing it successfully.

A. The study is sponsored by investigators with the resources and skills to complete it successfully.

Minor change

Legal

f. The research study is in compliance with all applicable Federal regulations concerning the protection of human subjects found in the Code of Federal Regulations (CFR) at 45 CFR Part 46. If a study is regulated by the Food and Drug Administration (FDA), it is also in compliance with 21 CFR Parts 50 and 56.  In addition, to further enhance the protection of human subjects in studies conducted under CED, the study must provide and obtain meaningful informed consent from patients regarding the risks associated with the study items and/or services, and the use and eventual disposition of the collected data.

V. The research study complies with all applicable Federal regulations concerning the protection of human subjects found in the Code of Federal Regulations (CFR) at 45 CFR Part 46. If a study is regulated by the Food and Drug Administration (FDA), it is also in compliance with 21 CFR Parts 50 and 56.  In addition, to further enhance the protection of human subjects in studies conducted under CED, the study must provide and obtain meaningful informed consent from patients regarding the risks associated with the study items and/or services, and the use and eventual disposition of the collected data.

Unchanged

Consent

f.  The research study is in compliance with all applicable Federal regulations concerning the protection of human subjects found in the Code of Federal Regulations (CFR) at 45 CFR Part 46. If a study is regulated by the Food and Drug Administration (FDA), it is also in compliance with 21 CFR Parts 50 and 56.  In addition, to further enhance the protection of human subjects in studies conducted under CED, the study must provide and obtain meaningful informed consent from patients regarding the risks associated with the study items and/or services, and the use and eventual disposition of the collected data.

I. The investigators obtain meaningful informed consent from patients regarding the risks associated with the study items and/or services, and the use and eventual disposition of the collected data, unless an institutional review board deems it to not be human subjects research or eligible for waiver or alteration of consent.

The new requirement more explicitly calls out informed consent for data collection. The existing recommendation included more than one concept.

Analysis

g. All aspects of the study are conducted according to appropriate standards of scientific integrity.

N The investigators minimize the impact of confounding and biases on inferences by using rigorous design and appropriate statistical techniques.

All of the proposed requirements contribute to scientific integrity; this now focuses more narrowly on bias and confounding.

Protocol

h. The study has a written protocol that clearly demonstrates adherence to the standards listed here as Medicare requirements.

G. A protocol describing the data source(s), key outcome(s), and key elements of design, at a minimum, is publicly posted on the CMS website

This newly calls for public posting of the protocol. The decision about posting to Clinicaltrials.gov or elsewhere will be at the discretion of the investigator.

Legal

i. The study is not designed to exclusively test toxicity or disease pathophysiology in healthy individuals. Such studies may meet this requirement only if the disease or condition being studied is life threatening as defined in 21 CFR §312.81(a) and the patient has no other viable treatment options.

U. The study is not designed to exclusively test toxicity or disease pathophysiology in healthy individuals. Such studies may meet this requirement only if the disease or condition being studied is life threatening as defined in 21 CFR §312.81(a) and the patient has no other viable treatment options.

Unchanged

Protocol

j. The clinical research studies and registries are registered on the www.ClinicalTrials.gov website by the principal sponsor/investigator prior to the enrollment of the first study subject.  Registries are also registered in the Agency for Healthcare Quality (AHRQ) Registry of Patient Registries (RoPR).

G. (as above)

The existing recommendation was split as it included more than one concept.

Design registry

j. The clinical research studies and registries are registered on the www.ClinicalTrials.gov website by the principal sponsor/investigator prior to the enrollment of the first study subject.  Registries are also registered in the Agency for Healthcare Quality (AHRQ) Registry of Patient Registries (RoPR).

P. When relevant, investigators follow best practices for establishing and maintaining a registry.

More explicit recommendation to follow best registry practices.

Reporting

k. The research study protocol specifies the method and timing of public release of all prespecified outcomes to be measured including release of outcomes if outcomes are negative or study is terminated early.  The results must be made public within 12 months of the study’s primary completion date, which is the date the final subject had final data collection for the primary endpoint, even if the trial does not achieve its primary aim.  The results must include number started/completed, summary results for primary and secondary outcome measures, statistical analyses, and adverse events. Final results must be reported in a publicly accessibly manner; either in a peer-reviewed scientific journal (in print or on-line),, in an on-line publicly accessible registry dedicated to the dissemination of clinical trial information such as ClinicalTrials.gov, or in journals willing to publish in abbreviated format (e.g., for studies with negative or incomplete results).

R. The results and analytic code are submitted for peer review using a reporting guideline appropriate for the design.

Minimized text by recommending use of an existing reporting guideline and stressed submitting of outcomes results for peer review.

Sharing

None, although some of the intent is captured in the Reporting requirement.

T. The investigators commit to sharing data, methods, and analytic code with CMS. Other sharing is to follow the rules of the funder and the institutional review boards.

Additional requirement; not explicitly in existing requirements

Design (subpopulations)

l. The study protocol must explicitly discuss beneficiary subpopulations affected by the item or service under investigation, particularly traditionally underrepresented groups in clinical studies, how the inclusion and exclusion criteria effect enrollment of these populations, and a plan for the retention and reporting of said populations in the trial.  If the inclusion and exclusion criteria are expected to have a negative effect on the recruitment or retention of underrepresented populations, the protocol must discuss why these criteria are necessary.

O.  The investigators pre-specify subpopulations for study if they expect that key outcomes in response to treatment will be meaningfully different in those subgroups compared with the majority population. Otherwise, investigators will explore for heterogeneity of treatment effect if there are not a priori hypotheses.

This requirement is to assure that the study appropriately explores or tests for heterogeneity of treatment effect that might inform a coverage decision

Generalizable

m. The study protocol explicitly discusses how the results are or are not expected to be generalizable to affected beneficiary subpopulations. Separate discussions in the protocol may be necessary for populations eligible for Medicare due to age, disability or Medicaid eligibility.

J. When feasible and appropriate for the CED question, data for the study should come from the real-world practice of medicine including from practitioners diverse in experience and diverse sites of care delivery.

This requirement focuses on the data for the study expecting that use of real-world data will better reflect the experience of diverse Medicare beneficiaries.

Design outcome

 

F. The key outcome(s) for study are those that are clinically important to patients and durable.  A surrogate outcome that reliably predicts key clinical outcomes might be appropriate for some questions.

Additional requirement; not explicitly in existing requirements

Governance

 

C. The information governance and data protection requirements are established in writing.

Additional requirement; not explicitly in existing requirements

Data quality

 

K. The selected data are of sufficient size, completeness, continuity, and accuracy to assess participant eligibility, key prognostic and predictive factors, exposure to therapy (including a unique device identifier, if relevant), and key outcomes.

Additional requirement; not explicitly in existing requirements

Data use

 

L. The investigators validate algorithms for the measurement of key exposures and outcomes. When infeasible, the investigators assess the performance of the operational definition of the variable or cite relevant validation exercises.

Additional requirement; not explicitly in existing requirements

Reproducibility

 

Q. The investigators demonstrate reproducibility of results from the study by conducting alternative and sensitivity analyses, and/or using other data sources.

Additional requirement; not explicitly in existing requirements

Replication

 

S. The reporting is structured to enable replication by a regulator, payor, or another research team.

Additional requirement; not explicitly in existing requirements

Results of the Key Informants Call

Eleven KIs provided rich comments about the proposed requirements. The rating of the proposed requirements, which ranged from essential (2 points) to important (1 point) to not important (0 points), indicated that all were considered important or essential [Appendix 3]. (Table 3)

Table 3. Rating of Importance of Proposed Requirements by the Key Informants (2 = essential; 1 = important; 0 = not important)

Proposed Requirements

Mean Rating of Importance*

Number of Zeros

 D. The rationale for the study is supported by scientific and medical evidence and its results are expected to fill a knowledge gap.

2.0

0

 K. The data are of sufficient size, completeness, continuity, and accuracy to assess participant eligibility, key prognostic and predictive factors, exposure to therapy (including a unique device identifier, if relevant), and key outcomes.

2.0

0

 A. The study is sponsored by investigators with the resources and skills to complete it successfully.

1.9

0

 C. The information governance and data protection requirements are established in writing and included in the study protocol.

1.9

0

 E. CMS and investigators agree upon the evidentiary threshold for the stated question. This reflects the clinically relevant difference in the key outcome(s) relative to the chosen comparator and the targeted precision.

1.9

0

 F. The key outcome(s) for study are those that are clinically important to patients and durable.  A surrogate outcome that reliably predicts key clinical outcomes might be appropriate for some questions.

1.9

0

 S. The reporting is structured to enable replication by a regulator, payor, or another research team.

1.9

0

 G. A protocol describing the data source(s), key outcome(s), and key elements of design, at a minimum, is publicly posted on the CMS website.

1.8

0

 I. The investigators obtain meaningful informed consent from patients regarding the risks associated with the study items and/or services, and the use and eventual disposition of the collected data, unless an institutional review board deems it to not be human subjects research or eligible for waiver or alteration of consent.

1.8

1

 T. The investigators commit to sharing data, methods, and analytic code with CMS. Other sharing is to follow the rules of the funder and the institutional review boards.

1.8

0

 N. The investigators minimize the impact of confounding and biases on inferences by using rigorous design and statistical techniques.

1.7

1

 R. The results and analytic code are submitted for peer review using a reporting guideline appropriate for the design.

1.7

0

U. The study is not designed to exclusively test toxicity or disease pathophysiology in healthy individuals. Such studies may meet this requirement only if the disease or condition being studied is life threatening as defined in 21 CFR §312.81(a) and the patient has no other viable treatment options.

1.6

1

 O. The investigators pre-specify subpopulations for study if they expect that key outcomes in response to treatment will be meaningfully different in those subgroups compared with the majority population. Otherwise, investigators will explore for heterogeneity of treatment effect if there are not a priori hypotheses.

1.6

0

 J. When feasible and appropriate for answering the CED question, data for the study should come from the real-world practice of medicine including from practitioners diverse in experience and diverse sites of care delivery.

1.5

1

 L. The investigators validate algorithms for the measurement of key exposures and outcomes. When infeasible, the investigators assess the performance of the operational definition of the variable or cite relevant validation exercises.

1.5

0

 M. The study design is selected to efficiently generate the needed evidence. Expected designs include pragmatic trials with randomization and blinding when feasible, single arm intervention studies with contemporaneous comparator groups, prospective cohort studies with contemporaneous comparison groups, self-controlled designs where appropriate, or retrospective cohort studies with contemporaneous comparators nested within registries.

1.5

2

 P. When relevant, investigators follow best practices for establishing and maintaining a registry.

1.5

1

 H. The studied population reflects the intended users of the product and also the racial, gender, and socio-economic diversity of the Medicare beneficiary population including older adults, individuals on dialysis, and disabled younger persons when relevant to the questions.

1.4

2

 Q. The investigators demonstrate reproducibility of results from the study by conducting alternative and sensitivity analyses, and/or using other data sources.

1.4

1

 B.  A written plan describes scheduled communication by the investigators with CMS throughout the evidence generation period for review of study milestones.

1.3

1

*11 KIs voting

In the discussion during the KI meetings, the KIs generally agreed that the proposed requirements as written did not seem to be excessively burdensome. Most of the KIs favored splitting multipart requirements into separate requirements rather than bundling several recommendations into a single requirement. It was felt that more granular requirements may be easier to act upon. On the basis of the discussion with the KIs, the proposed requirements were amended and re-ordered, and the rationales for the amendments are included in Table 4.

Upon review of the amended requirements (Table 4), nine responding KIs remained supportive of all of the requirements. The mean ratings of the amended requirements ranged from 1.48 (for J) to 2.00 (for A, C, L, and S). Based upon the limited comments of the KIs upon their re-survey, we made minor alterations to the wording of the requirements; the final amended requirements are in Table 5 along with the mean ratings [Appendix 3]. None of the KIs rated any of these requirements as zeros. One KI strongly suggests that examination of demographic subpopulations be required for every CED. However, there remained disagreement regarding whether demographic subpopulations should be studied in every CED, the focus should simply be on clinically relevant subpopulations, or whether this should remain flexible based upon prior evidence from studies of related interventions or earlier studies of the given intervention.

Table 4. Amendments Requirements Based on the Recommendations of the Key Informants

 

Proposed Requirements

Key Informant Feedback

Amended Requirements for Public Posting with New Lettering

Rationale

A

The study is sponsored by investigators with the resources and skills to complete it successfully.

Recommendation to change word sponsored to conducted.

  1. The study is conducted by investigators with the resources and skills to complete it successfully.

Responded to suggestions.

B

A written plan describes scheduled communication by the investigators with CMS throughout the evidence generation period for review of study milestones.

Suggestion to focus on the milestones and less on communication with CMS; CMS can decide how it will monitor the milestone completion.

  1. A written plan describes the schedule for completion of key study milestones.

Responded to suggestions.

D

The rationale for the study is supported by scientific and medical evidence and its results are expected to fill a knowledge gap.

There are lots of potential sources of uncertainty and it's important to be clear on what uncertainty the study is trying to reduce.

  1. The rationale for the study is supported by scientific evidence and study results are expected to fill the specified knowledge gap.

Responded to suggestions by inclusion of the word specified and made the wording more concise by removing “and medical”

E

CMS and investigators agree upon the evidentiary threshold for the stated question. This reflects the clinically relevant difference in the key outcome(s) relative to the chosen comparator and the targeted precision. 

Varied written comments but little discussion on the call.

  1. CMS and investigators agree on an evidentiary threshold for the study as needed to demonstrate clinically meaningful differences in key outcome(s) with adequate precision.

Responded to suggestions by re-writing this as one sentence and removing attention to comparators. Precision refers to sufficient sample size for statistically significant comparisons.

G

A protocol describing the data source(s), key outcome(s), and key elements of design, at a minimum, is publicly posted on the CMS website

Some disagreement about whether the protocol needs to be published or whether posting on the CMS site is most appropriate, but general agreement and agreement that these are key elements (at a minimum).

  1. The study’s protocol is publicly posted on the CMS websiteminimum, the data source(s), key outcome(s), and study design.

The ordering of the sentence was changed; the content was not. The details of these three elements are described in other requirements.

C

The information governance and data protection requirements are established in writing and included in the study protocol.

No feedback

  1. The protocol describes the information governance and data protection requirements that have been established.

The ordering of the sentence was changed; the content was not.

K

The data are of sufficient size, completeness, continuity, and accuracy to assess participant eligibility, key prognostic and predictive factors, exposure to therapy (including a unique device identifier, if relevant), and key outcomes.

This prompted many comments for clarification. One person suggested that the investigator needs to justify the choice of data.

  1. The data are generated or selected with attention to completeness, accuracy, sufficiency of duration of observation, and sample size as required by the question.

Responded to suggestions.  Because item E states that the data is described in the protocol, it is expected that these details will be described in the protocol.

J

When feasible and appropriate for answering the CED question, data for the study should come from the real-world practice of medicine including from practitioners diverse in experience and diverse sites of care delivery.

This prompted discussion about the term “real-world” and recognition that the evaluation of devices differs from evaluation of drugs with agreement that evaluation may be optimal in diverse settings.

  1. Data for the study comes from patients treated in the usual sites of care delivery for the product.

The usual site of care delivery may be a specialized clinical facility (e.g. “center of excellence”) when the product is newly in use and may include more diverse sites of care as usage expands. 

F

The key outcome(s) for study are those that are clinically important to patients and durable.  A surrogate outcome that reliably predicts key clinical outcomes might be appropriate for some questions. 

Little interest in the durability of the outcomes on the first call but greater interest in this on the second call. Agreement that patient relevance is important and that surrogate outcomes are sometimes appropriate.

  1. The key outcome(s) for the study are those that are important to patients. A surrogate outcome that reliably predicts these outcomes may be appropriate for some questions. 

Modifications reflect that there is often existing information about what is important to patients. If there is not, this information may need to be generated. Because item E states that outcomes are described in the protocol, it is expected that this will be described in the protocol.

H

The studied population reflects the intended users of the product and also the racial, gender, and socio-economic diversity of the Medicare beneficiary population including older adults, individuals on dialysis, and disabled younger persons when relevant to the questions.

This was thought to be unclear and that adherence would make studies inefficient given excessively broad enrollment criteria.

  1. The study population reflects the demographic and clinical complexity among the Medicare beneficiaries who are the intended users of the product.

This was rephrased but the intent remains the same; the study enrolls the intended users but with attention to the inclusion of diverse users of the product.

I

The investigators obtain meaningful informed consent from patients regarding the risks associated with the study items and/or services, and the use and eventual disposition of the collected data, unless an institutional review board deems it to not be human subjects research or eligible for waiver or alteration of consent.

There was uncertainty as to whether this is necessary given requirement V below. It seems likely that the informed consent process will be a requirement of the IRB.

DELETED

Responded to suggestions

L

The investigators validate algorithms for the measurement of key exposures and outcomes. When infeasible, the investigators assess the performance of the operational definition of the variable or cite relevant validation exercises.

Provided suggestions regarding wording for clarity.

  1. When using secondary data, investigators provide information about the performance of the algorithms used for measurement of key exposures and outcomes.

Responded to suggestions although did not include a requirement to use validated algorithms for confounders.  Used the word “secondary data” to indicate data from electronic health records, claims, etc.

M

The study design is selected to efficiently generate the needed evidence. Expected designs include pragmatic trials with randomization and blinding when feasible, single arm intervention studies with contemporaneous comparator groups, prospective cohort studies with contemporaneous comparison groups, self-controlled designs where appropriate, or retrospective cohort studies with contemporaneous comparators nested within registries.

Discussion that the detailed list of possible study designs was both unnecessary and restrictive. Agreement that the word efficient is valuable.  Recommendation to focus on the need for a design that generates valid evidence. Discussion that a comparator is not always necessary in these settings.

  1. The study design is selected to efficiently generate valid evidence. If a contemporaneous comparison group is not included, this choice must be justified.

Responded to suggestions. Revision reflects that efficiency is NOT being prioritized over validity.

N

The investigators minimize the impact of confounding and biases on inferences by using rigorous design and statistical techniques.

Note overlap with the requirement about choosing a study design that generates valid evidence.

  1. The investigators minimize the impact of confounding and biases on inferences with appropriate statistical techniques, in addition to rigorous design.

Reordered the elements.

O

The investigators pre-specify subpopulations for study if they expect that key outcomes in response to treatment will be meaningfully different in those subgroups compared with the majority population. Otherwise, investigators will explore for heterogeneity of treatment effect if there are not a priori hypotheses.

Urged to avoid any suggestion that investigators need only evaluate social class and race/ethnicity when the data indicate a difference. Respondent emphasizes that a set of fundamental factors should always be measured in a standardized way and considered as having an effect on outcomes until proven otherwise.

  1. In the protocol, the investigators describe considerations for analyzing demographic subpopulations as well as clinically-relevant subgroups as motivated by existing evidence.

Modified to reflect that existing evidence (such as from phase II/III studies, related products or class effects) should inform the pre-specification of clinically-relevant subgroups, while all studies should include analysis of demographic subpopulations.

P

When relevant, investigators follow best practices for establishing and maintaining a registry.

Unclear why there was a separate requirement about registries; uncertainty about whether this refers to establishing a registry to meet a CED requirement or conducting a “registry study”

DELETE

It is understood that establishing a registry does not generate evidence without an accompanying study design. The conduct of a study nested in a registry may generate valid evidence. The proposed requirement is not needed because other requirements address the need to justify the study design.

Q

The investigators demonstrate reproducibility of results from the study by conducting alternative and sensitivity analyses, and/or using other data sources.

Comments about robustness rather than reproducibility. 

  1. The investigators demonstrate robustness of results by conducting alternative analyses, and/or using other data sources.

This wording was clarified.

R

The results and analytic code are submitted for peer review using a reporting guideline appropriate for the design.

Comment that there could be a requirement for public posting on a website. Thought that this might be merged with the other requirement about reporting.

  1. The results and analytic code are submitted for peer review using a reporting guideline appropriate for the study design and structured to enable replication.

We favor peer review for vetting rather than just public posting, although both might be appropriate. This now reflects a merging of two requirements.

S

The reporting is structured to enable replication by a regulator, payor, or another research team.

Comment that this could be more concise.

DELETE - MERGED with R

Responded to suggestions

 T

The investigators commit to sharing data, methods, and analytic code with CMS. Other sharing is to follow the rules of the funder and the institutional review boards.

Discussion about whether patients will be reluctant to enroll if their personal data will be shared with the government. No one knows, really. Discussion that anonymized data may be comfortable to patients.  Discussion that this may not contribute importantly to transparency.

Q. The investigators commit to sharing de-identified data, methods, and analytic code with CMS or with a trusted third party. Other sharing is to follow the rules of the funder and the institutional review boards.

Inserted “or with a trusted third party” to allow the investigators to share data elsewhere if they learn that sharing with CMS impacts study enrollment. Rationale for sharing is so that CMS has the opportunity to verify results and possibly do additional learning.

The study is not designed to exclusively test toxicity or disease pathophysiology in healthy individuals. Such studies may meet this requirement only if the disease or condition being studied is life threatening as defined in 21 CFR §312.81(a) and the patient has no other viable treatment options.

Uncertain meaning given that healthy individuals do not have disease pathophysiology or life threatening conditions. The intent of the initial requirement is unknown.

R. The study is not designed to exclusively test toxicity unless the disease or condition being studied is life threatening as defined in 21 CFR §312.81(a) and the patient has no other viable treatment options.

A study evaluating disease pathophysiology is unlikely to be brought forward for CED. A study of toxicity of a product seems potentially appropriate if used in an individual with few options.

V

The research study complies with all applicable Federal regulations concerning the protection of human subjects found in the Code of Federal Regulations (CFR) at 45 CFR Part 46. If a study is regulated by the Food and Drug Administration (FDA), it is also in compliance with 21 CFR Parts 50 and 56.  

No comments

S. The research study complies with all applicable Federal regulations concerning the protection of human subjects found in the Code of Federal Regulations (CFR) at 45 CFR Part 46. If a study is regulated by the FDA, it is also in compliance with 21 CFR Parts 50 and 56.  

No changes

CED, coverage with evidence development; CFR code of federal regulations, CMS, Centers for Medicare & Medicaid Services; FDA Food and Drug Administration; IRB, institutional review board; KI key informant

Table 5. Amended Requirements Based on the Recommendations of the Key Informants (2 = essential; 1 = important; 0 = not important)

Final Amended Requirements for Public Posting

Mean Rating of Importance*

 A. The study is conducted by investigators with the resources and skills to complete it successfully.

2.0

B. A written plan describes the schedule for completion of key study milestones.

1.6

 C. The rationale for the study is supported by scientific evidence and study results are expected to fill the specified knowledge gap.

2.0

D. CMS and investigators agree on an evidentiary threshold for the study as needed to demonstrate clinically meaningful differences in key outcome(s) with adequate precision.

1.9

 E. The study’s protocol is publicly posted on the CMS website and describes, at a minimum, the data source(s), key outcome(s), and study design.

1.8

F. The protocol describes the information governance and data protection requirements that have been established.

1.9

G. The data are generated or selected with attention to completeness, accuracy, sufficiency of duration of observation, and sample size as required by the question.

1.9

H. Data for the study comes from patients treated in the usual sites of care delivery for the product.

1.4

I. The key outcome(s) for the study are those that are important to patients. A surrogate outcome that reliably predicts these outcomes may be appropriate for some questions.

1.8

J. The study population reflects the demographic and clinical diversity among the Medicare beneficiaries who are the intended users of the intervention.

1.3

K. When using secondary data, investigators provide information about the performance of the algorithms used for measurement of key exposures and outcomes.

1.7

 L. The study design is selected to efficiently generate valid evidence. If a contemporaneous comparison group is not included, this choice must be justified.

2.0

M. The investigators minimize the impact of confounding and biases on inferences with appropriate statistical techniques, in addition to rigorous design.

1.8

N. In the protocol, the investigators describe considerations for analyzing demographic subpopulations as well as clinically-relevant subgroups as motivated by existing evidence.

1.3

O. The investigators demonstrate robustness of results by conducting alternative analyses and/or using other data sources.

1.7

 P. The results and analytic code are submitted for peer review using a reporting guideline appropriate for the study design and structured to enable replication.

1.7

Q. The investigators commit to sharing de-identified data, methods, and analytic code with CMS or with a trusted third party. Other sharing is to follow the rules of the funder and the institutional review board.

1.7

R. The study is not designed to exclusively test toxicity unless the disease or condition being studied is life threatening as defined in 21 CFR §312.81(a) and the patient has no other viable treatment options.

1.4

S. The research study complies with all applicable Federal regulations concerning the protection of human subjects found in the Code of Federal Regulations (CFR) at 45 CFR Part 46. If a study is regulated by the Food and Drug Administration, it is also in compliance with 21 CFR Parts 50 and 56.

2.0

*9 KIs voting; CED, coverage with evidence development; CFR code of federal regulations, CMS, Centers for Medicare & Medicaid Services; FDA Food and Drug Administration; IRB, institutional review board; KI key informant

Guiding Questions

In approaching this task, we looked to the guiding questions as we developed a strategy to generate this new set of requirements. We considered the strengths and limitations of the existing requirement and we sought to learn what requirements are used by similar decision making bodies. The existing requirements have not been formally evaluated making it challenging to comment objectively on their strengths and limitations. Our review of the documentation of completed studies for a CED or studies underway does not allow for comprehensive assessment of adherence to the requirements. There has not been a requirement for public posting of protocols, and we have not seen peer reviewed and published protocols, although they may exist. Peer reviewed results are often available and the study’s methods section provides information about the design and conduct. Of the 23 CEDs for which registries and/or trials were used, 16 (62%) had some publicly available results, including 6 in which results were posted on ClinicalTrials.gov.39 We suggest that immediately valuable work would be a review, similar to that conducted by the EU Horizon 2020 COMED, to assess the historical adherence of investigators to the existing requirements. This would then inform the implementation and evaluation of amended requirements.

The evaluation of the impact of the existing set of requirements requires assessment of whether a decision was made based on the generated evidence, as well as what the decision was. In the recent review by Zeitler and colleagues, an action based on the results of CED studies was infrequent. In only 5 of the 26 CEDs has there been an outcome.39 For 3 CEDs, the CED was retired when the evidence was deemed to be adequate: carotid artery stenting, implantable cardioverter defibrillator (ICD) for primary prevention of sudden cardiac death, and magnetic resonance angiography/magnetic resonance imaging in patients with a cardiac implantable electronic device. Two CEDs, artificial hearts and home oxygen for cluster headaches, progressed to retirement of the CED and deferral of coverage decisions to MACs. The remaining CEDs are considered ongoing, suggesting that the studies have not yet generated the evidence needed for a coverage decision other than that provided by the CED.  Whether this is due to the existing study requirements or due to other elements of the CED program cannot be easily determined.

The guiding question about requirements used by similar decision making bodies was valuable in pushing us to search for international publications on the topic as well as domestic publications from organizations that conduct health technology assessment. We found extensive literature describing processes, but very little that is granular enough to be considered recommendations or study criteria. Given that CMS is fairly unique among decision-making bodies in that its evaluation does not take into account the costs of the items or services, much of the literature is only peripherally pertinent. For example, the literature about managed-entry agreements and risk-sharing was not on-target with our needs. We also learned that with decision-making bodies having greater access to data from diverse sources over the past decade, and the expansion of methods for drawing inferences from observational data, older literature about study design and conduct was less valuable to our revision of the requirements. That being said, many of the principles, including transparency and reproducibility of results, are evergreen.

Led by the guiding questions, we then addressed the key questions that were posed by AHRQ on behalf of CMS.

Key Questions

KQ1: What Revisions to the CED Criteria (“Requirements”) May Best Address the Limitations While Preserving the Strengths

We suggest that the proposed requirements, although lengthier, have more explicit expectations for the studies that are designed to generate the needed evidence for CMS and should be easier to act upon by sponsors. Many of the existing requirements are important and were retained. We suggest that the process of separating some of the requirements, which included multiple goals, into more discrete requirements improves the clarity. The inclusion of additional requirements reflects our understanding of the limitations of the existing requirements from our review of the literature. The existing requirements did not address the need for a governance plan, the quality of the data, validation of exposures and outcomes in the data, reproducibility of inferences, and sharing of code to allow replication of results. Most of the proposed requirements are applicable across study designs and across varied sources of data.

Our suggestion about the use of real-world data when feasible is reflected in amended requirement H, which describes the inclusion of patients in their usual clinical settings. The focus on real-world data to generate real-world evidence was intentional; this is often the appropriate evidence for a coverage decision (in contrast to a regulatory decision).40, 41 Additionally, the focus on use of data generated in the usual care of patients may help assure the inclusion of a population generalizable to all Medicare beneficiaries who may be impacted by the coverage decision, and may help with the inclusion of sufficient beneficiaries representing subpopulations of interest.  

Although real-world evidence is often sought for coverage decisions, for many CED, we expect that there will continue to be the need for more traditional trials. This largely arises because the therapies recommended for CED are often devices or diagnostics, rather than drugs or biologics, or are therapies being used for novel indications, without FDA-approval for marketing for these indications. In these situations, there may not be the extensive clinical trial record that is generated during regulatory approval of pharmaceuticals. Even Class III devices may be released from FDA’s pre-market approval process if the sponsor successfully petitions for reassignment of the device to allow for the 501(k) process, which does not require the generation of extensive clinical evidence of efficacy or safety. Therefore, decision-makers at CMS may require the generation of new evidence to inform the coverage decision and this may require a more traditional clinical trial. These trials can still be expected to follow the criteria presented here.

KQ2: How Might the Amended Criteria be Evaluated in the Future

We are unaware of any previous evaluation of the existing criteria so what we propose here is unique. The amended criteria might be evaluated with attention to both process and outcome metrics. If protocols that are developed by sponsors of the product, or by other investigators, are posted to the CMS site, it will facilitate external evaluation. This is consistent with what was recommended in an Organisation for Economic Co-operative and Development (OECD) Health Working Paper by Wenzl and Chapman42 that argues “that as many features of [CED-like] schemes as possible should be in the public domain, apart from confidential items such as the details of any financial settlement made following the scheme (e.g., on the price of the device). Features of schemes that could be made public are the study design and methodology, the new evidence generated by the scheme, and any policy recommendations that were made following the scheme.” CMS and/or AHRQ might review the protocols for their attention to the amended criteria. The sponsors might be encouraged to use a checklist to confirm attention to the criteria at the time of protocol preparation. It is possible that not every criterion is appropriate to every study, and this will need to be documented when embarking upon a study. The sponsors should explain why a criterion is considered to be inappropriate in that setting. Upon completion of the proposed work, the published results might again be reviewed by CMS for adherence to the criteria.

The impact of the criteria on outcomes can be evaluated by an assessment of the value of the evidence that is produced. Does the evidence generated in a study or series of studies allow CMS to end a CED with a coverage or non-coverage decision or with deferral to a MAC? If the evidence is insufficient for these decisions, this would be considered to be a poor outcome as the studies should have been designed and adequately powered to generate the needed evidence. The quality and strength of the evidence generated is the ultimate test of the effectiveness of the criteria as this will allow for a decision.

Conclusion

We reviewed published literature about best practices for generating evidence as is appropriate for a coverage decision. We found 27 articles to be relevant to the update, yielding 172 recommendations to distill into a parsimonious set of revised requirements. We circulated 22 revised requirements to the 11 key informants. After incorporating their feedback, we had 19 amended requirements that were posted for public comment. The new additions included requirements for a milestone driven process, public posting of a protocol, improved clarity regarding data selection and data governance, attention to clinically important outcomes and to the diversity of Medicare beneficiaries, demonstration of robustness of results, and sharing of de-identified data. The amended requirements make explicit the expectations for studies that are designed to generate needed evidence for CMS. The requirements pertain to observational studies and traditional trials which may be sources of evidence for future CED decisions, depending on the clinical context. We propose that the impact of these requirements might be evaluated by assessment of the value of evidence produced. Does the evidence generated in a study or series of studies allow CMS efficiently to end a CED with a coverage or non-coverage decision? This is the test of the effectiveness of the requirements.  In Table 5, we present a full set of proposed new criteria that have been amended to incorporate input from a distinguished panel of experts in the field.

References

  1. Medicare Program; Medicare Coverage of Innovative Technology (MCIT) and Definition of ‘‘Reasonable and Necessary’’(CMS–3372–F). In: Centers for Medicare & Medicaid, editor: Federal Register; 2021. p. 2987-3010.
  2. Guidance for the public, industry, and CMS staff: coverage with evidence development. Baltimore, MD: Centers for Medicare & Medicaid Services; 2014. Accessed on June 24, 2022.
  3. Medicare Announces Draft Guidance for National Coverage Determinations with Evidence Development. Baltimore, MD: U.S. Centers for Medicare & Medicaid Services; 2005. Accessed on July 1, 2022.
  4. Straube BM. How changes in the Medicare coverage process have facilitated the spread of new technologies. Health Aff (Millwood). 2005 Jan-Jun;Suppl Web Exclusives:W5-314-w5-6. doi: 10.1377/hlthaff.w5.314. PMID: 15975989.
  5. Tunis SR, Pearson SD. Coverage options for promising technologies: Medicare's 'coverage with evidence development'. Health Aff (Millwood). 2006 Sep-Oct;25(5):1218-30. doi: 10.1377/hlthaff.25.5.1218. PMID: 16966717.
  6. Guidance for the Public, Industry, and CMS Staff: Coverage with Evidence Development Baltimore, MD: U.S. Centers for Medicare & Medicaid Services; 2014. Accessed on July 1, 2022.
  7. National Bioeconomy Blueprint. White House.  2012.
  8. Monoclonal Antibodies Directed Against Amyloid for the Treatment of Alzheimer’s Disease. CAG-00460N. Baltimore, MD: U.S. Centers for Medicare & Medicaid Services; 2022. Accessed on July 1, 2022.
  9. Brügger U. A Review of Coverage with Evidence Development (CED) in Different Countries: What Works and What Doesn’t. 2014. Accessed on July 12, 2022.
  10. Diaby V, Goeree R. Coverage with Evidence Development in Comparative Effectiveness Research. Comp Eff Res Heal Serv. 2015doi: 10.1007/978-1-4899-7586-7.
  11. Federici C, Reckers-Droog V, Ciani O, et al. Coverage with evidence development schemes for medical devices in Europe: characteristics and challenges. Eur J Health Econ. 2021 Nov;22(8):1253-73. doi: 10.1007/s10198-021-01334-9. PMID: 34117987.
  12. Framework for FDA's Real-World Evidence Program US Food & Drug Administration. Silver Spring, MD: 2018.
  13. National Evaluation System for health Technology Coordinating Center (NESTcc) Data Quality Framework: A Report of the Data Quality Subcommittee of the NEST Coordinating Center - An initiative of MDIC NEST Coordinating Center.  2020.
  14. National Evaluation System for health Technology Coordinating Center (NESTcc) Methods Framework: A Report of the Methods Subcommittee of the NEST Coordinating Center - An initiative of MDIC NEST Coordinating Center.  2020. 
  15. Berger ML, Sox H, Willke RJ, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol Drug Saf. 2017 Sep;26(9):1033-9. doi: 10.1002/pds.4297. PMID: 28913966.
  16. Wang SV, Schneeweiss S, Berger ML, et al. Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0. Pharmacoepidemiol Drug Saf. 2017 Sep;26(9):1018-32. doi: 10.1002/pds.4295. PMID: 28913963.
  17. RWE for Regulatory Decision Making. Bethesda, MD: International Society for Pharmacoepidemiology; 2022. Accessed on June 22, 2022.
  18. National Academies of Sciences E, Medicine, Health, et al. The National Academies Collection: Reports funded by National Institutes of Health.  Real-World Evidence Generation and Evaluation of Therapeutics: Proceedings of a Workshop. Washington (DC): National Academies Press (US). Copyright 2017 by the National Academy of Sciences. All rights reserved.; 2017.
  19. National Academies of Sciences E, Medicine, Health, et al. The National Academies Collection: Reports funded by National Institutes of Health.  Examining the Impact of Real-World Evidence on Medical Product Development: II. Practical Approaches: Proceedings of a Workshop—in Brief. Washington (DC): National Academies Press (US). Copyright 2018 by the National Academy of Sciences. All rights reserved.; 2018.
  20. National Academies of Sciences E, Medicine, Health, et al. The National Academies Collection: Reports funded by National Institutes of Health. In: Shore C, Gee AW, Kahn B, Forstag EH, eds. Examining the Impact of Real-World Evidence on Medical Product Development: Proceedings of a Workshop Series. Washington (DC): National Academies Press (US). Copyright 2019 by the National Academy of Sciences. All rights reserved.; 2019.
  21. Towse A, Chalkidou K, Firth I, et al. How Should the World Pay for a Coronavirus Disease (COVID-19) Vaccine? Value Health. 2021 May;24(5):625-31. doi: 10.1016/j.jval.2020.12.008. PMID: 33933230.
  22. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices US Food & Drug Administration.  Silver Spring, MD: 2017.
  23. Use of Electronic Health Record Data in Clinical Investigations Guidance for Industry US Food & Drug Administration.  Silver Spring, MD: 2018.
  24. Submitting Documents Using Real-World Data and Real-World Evidence to FDA for Drugs and Biologics Guidance for Industry US Food & Drug Administration.  Silver Spring, MD: 2019.
  25. Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products US Food & Drug Administration.  Silver Spring, MD: 2021.
  26. Real-World Data: Assessing Registries to Support Regulatory Decision-Making for Drug and Biological Products Guidance for Industry US Food & Drug Administration.  Silver Spring, MD: 2021.
  27. Considerations for the Use of Real-World Data and Real-World Evidence To Support Regulatory Decision-Making for Drug and Biological Products US Food & Drug Administration.  Silver Spring, MD: 2021.
  28. Data Standards for Drug and Biological Product Submissions Containing Real-World Data US Food & Drug Administration.  Silver Spring, MD: 2021.
  29. GRACE Principles: A validated checklist for evaluating the quality of observational cohort studies for decision-making support GRACE Initiative.
  30. Dreyer NA, Bryant A, Velentgas P. The GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness. J Manag Care Spec Pharm. 2016 Oct;22(10):1107-13. doi: 10.18553/jmcp.2016.22.10.1107. PMID: 27668559.
  31. Pearson SD, Dreitlein WB, Towse A, et al. A framework to guide the optimal development and use of real-world evidence for drug coverage and formulary decisions. J Comp Eff Res. 2018 Dec;7(12):1145-52. doi: 10.2217/cer-2018-0059. PMID: 30427724.
  32. Berger M, Daniel G, Frank K, et al. A Framework for Regulatory Use of Real-World Evidence Duke Margolis Center for Health Policy.  2017.
  33. 21st Century Cures Act, Public Law 114–255. In: United States Congress, editor. Washington, DC: U.S. Government Publishing Office; 2016.
  34. Meeting Minutes of the Center for Medicare and Medicaid Services Medicare Evidence Development and Coverage Advisory Committee U.S. Centers for Medicare and Medicaid Services.  Baltimore, MD: 2012.
  35. Drummond M, Federici C, Reckers-Droog V, et al. Coverage with evidence development for medical devices in Europe: Can practice meet theory? Health Econ. 2022 Feb 26doi: 10.1002/hec.4478. PMID: 35220644.
  36. Specification for Cancer Drugs Fund data collection arrangements. National Institute for Health and Care Excellence. Accessed on June 22, 2022.
  37. The Innovative Medicines Fund Principles. National Health Service; 2022. Accessed on June 22, 2022.
  38. Reckers-Droog V, Federici C, Brouwer W, et al. Challenges with coverage with evidence development schemes for medical devices: A systematic review. Health Policy and Technology. 2020;9:146-56. doi: 10.1016/j.hlpt.2020.02.006.
  39. Zeitler EP, Gilstrap LG, Colylewright M, et al. Coverage with evidence development: where are we now? AJMC. 2022;28(8):In Press.
  40. Garrison LP, Jr., Neumann PJ, Erickson P, et al. Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report. Value Health. 2007 Sep-Oct;10(5):326-35. doi: 10.1111/j.1524-4733.2007.00186.x. PMID: 17888097.
  41. Hampson G, Towse A, Dreitlein WB, et al. Real-world evidence for coverage decisions: opportunities and challenges. J Comp Eff Res. 2018 Dec;7(12):1133-43. doi: 10.2217/cer-2018-0066. PMID: 30411972.
  42. Wenzl M, Chapman S. Performance-based managed entry agreements for new medicines in OECD countries and EU member states: How they work and possible improvements going forward OECD Publishing.  Paris: 2019.

Project Timeline

Analysis of Requirements for Coverage with Evidence Development (CED)

Sep 6, 2022
Topic Initiated
Sep 7, 2022
Key Questions
Sep 7, 2022 - Sep 28, 2022
Page last reviewed September 2022
Page originally created May 2022

Internet Citation: Key Questions: Analysis of Requirements for Coverage with Evidence Development (CED) – Topic Refinement. Content last reviewed September 2022. Effective Health Care Program, Agency for Healthcare Research and Quality, Rockville, MD.
https://effectivehealthcare.ahrq.gov/products/coverage-evidence-development/draft-comment

Select to copy citation