Skip Navigation
AHRQ--Agency for Healthcare Research and Quality: Advancing Excellence in Health Care

Back to "About This Webcast"

Slide 1

Integrating stakeholder preferences in comparative effectiveness research using multi-criteria decision analysis (MCDA) and Conjoint Analysis (CA)

Maarten J. IJzerman, PhD
Professor and chair, department Health Technology & Services Research

Slide 2

Multiple endpoints in comparative effectiveness research

  • In CER we wish to make an informed decision based on available clinical evidence for multiple endpoints.
  • Primary endpoints chosen in clinical trials may not be the most relevant endpoints for patients and other stakeholders.
  • Also, process related factors contribute to the actual use by stakeholders and are usually neglected in decision making.
  • Three approaches for including stakeholder preferences
    • Stakeholder representatives in appraisal committees
    • Elicitation of preferences to guide deliberative process
    • Integration of preferences in quantitative framework

Slide 3

All MCDA and CA methods decompose a decision problem into a set of criteria (attributes and levels):

  • I want the best treatment:
  Clinical Outcome Adverse Events Out-of-Pocket Costs
Drug A symptom relief high blood pressure 5 US$
Drug B partial relief high blood pressure 3 US$
Drug C partial relief No adverse events 5 US$

Slide 4

And they use a linear additive value function to determine the relative preference for the alternatives:

  • I want the best treatment:


  Clinical Outcome
C1 = 0.5
Adverse Events
C2 = 0.33
Out-of-Pocket Costs
C3 = 0.17
Drug A symptom relief
L1.1   = 0.5
high blood pressure
L2,1   = 0.25
5 US$
L3,1   = 0.25
Drug B partial relief
L1,2   = 0.25
high blood pressure
L2,2   = 0.25
3 US$
L3,2   = 0.5
Drug C partial relief
L1,3   = 0.25
No adverse events
L2,3   = 0.5
5 US$
L3,3  = 0.25

drug A: (C1*L1,1 + C2*L2,1 + C3*L3,1) = 0.375

Slide 5

Multi-criteria decision analysis and Conjoint Analysis

  • Multi-criteria Decision analysis (MCDA) is a subdiscipline of operations research that explicitly considers multiple criteria in decision making. MCDA methods enable the evaluation of many alternatives by explicit ranking, rating or pairwise comparison of criteria and alternatives. (See: Belton & Steward, 2002)
    • The Analytic Hierarchy Process (AHP) is one of the most widely used MCDA techniques available (Saaty, 1989)
  • Conjoint Analysis (CA) methods offer subjects a series of choices among two or more product profiles. The pattern of choices reveals the implicit decision weights patients attach to therapeutic benefits, harms, processes and costs that describe the treatment profiles (ISPOR taskforce, 2012)

Slide 6

MCDA and CA: an example

  • Suppose a study in which we will select a good restaurant for dinner. We have defined 3 criteria (Ci) and 3 levels (Li)
    • Cooking style: Italian, Greek or Thai
    • Travel distance: need a car, walking or cycling
    • Price: 15$, 20$, 30$
  • Conjoint analysis will generate scenarios based on the levels (Li). In this case, 27 (33) scenario’s describe all possible combinations in the decision space.
  • MCDA will ask you which criteria and levels (Ci and Li) are important using ranking, rating or pairwise comparisons. 12 comparisons can describe the complete decision space.


Slide 7

Conjoint analysis

  • Each participant receives N choice sets.
  • Based on their response it is possible to estimate the part-worth utility of the levels and to estimate attribute weights
  • E.g. if Thai food is presented in X scenarios and if a respondent picks all these scenarios we can assume Thai food is an important factor

Please select the most preferred restaurant.

Italian food,
Need a car,
20 $
Thai food
Need a car
30 $.
Checkbox: not checked Checkbox: checked

Slide 8

Analytic Hierarchy Process (MCDA)

(Flowchart:  Hierarchical relationships from left to right, top to bottom)

  • Choose the best restaurant
    • Style (0,5)
      • Thai (0,5)
      • Italian (0,3)
      • Greek (0,2)
    • Price (0,3)
      • 15$ (0,7)
      • 20$(0,2)
      • 25$ (0,1)
    • Travel (0,2)
      • Car (0,3)
      • Bicycle (0,1)
      • Foot (0,6)

Restaurant 1 (0.52): Includes arrows that point upward toward the boxes Thai, 20$, and bicycle

Restaurant 2 ( 0.23): Includes arrows that point upward toward the boxes Greek, 15$, and car

Restaurant 3 ( 0.25): Includes arrows that point upward toward the boxes Italian, 25$, and foot

Slide 9

MCDA vs. Conjoint analysis methods

MCDA (e.g. AHP)

  • Helps to improve judgment
  • Decomposed
  • Hierarchical
  • Low cognitive stress
  • Less realistic
  • Flexible
  • Not really used in marketing
  • Decision sciences / OR
  • Possible to use in N=1 or as consensus building tool (group-decision)


CA methods

  • Imitates consumers’ judgment
  • Holistic evaluation
  • Matrix structure
  • Cognitive stress
  • Realistic task
  • Less flexible (e.g. design)
  • Use in marketing (i.e. creates a hypothetical market)
  • Large datasets required


Slide 10

MCDA methods: the Analytic Hierarchy Process

Picture: Hierarchical flowchart of the analytic hierarchy process. The top object labeled Decision objective (treatment) is linked to objects below labeled Criterion 1, Criterion 2 and Criterion 3 with the corresponding equation C1+C2+C3 = 1.

Criterion 3 is broken down into Criterion 3.1 and Criterion 3.2 below the main Criterion 3 box with the corresponding equation C3.1+C3.2 = 1. 

Below these boxes are objects labeled Alternative 1 and Alternative 2 that are linked by lines to all the objects above. These have the corresponding equation A1+A2 = 1

Slide 11

Picture: screenshot of a program for comparing the clinical benefit of a drug to its impact of treatment based on the Verbal scale of importance in pairwise comparisons.  A decision matrix is highlighted that includes: Clinical benefit, Impact of treatment, Side effects, additional costs to patients.
Verbal scale of importance in pairwise comparisons

  • Equal
  • Equal to moderate
  • Moderate
  • Moderate to Strong
  • Strong
  • Strong to very strong
  • Very strong
  • Very strong to extreme
  • Extreme

Slide 12

Eigenvalue to obtain criteria (Ci) and level (Li) weights (1)

  Clinical benefit Adverse event Treatment impact Co-payment
Clinical benefit 1 3 5
Matrix with pairwise comparisons (4 criteria, 6 comparisons)
Adverse event 1/3 1 1/3 1/5
Treatment impact 5
Reciprocal scores of pairwise comparisons
3 1 3
Co-payment 1/7 1/5 1/3 1
Sum score: 6.48 7.20 6.67 11.20

Slide 13

Eigenvalue to obtain criteria (Ci) and level (Li) weights (2)

  Clinical benefit Adverse event Treatment impact Co-payment Average
Clinical benefit 0.15 0.42 0.75 0.63 0.49
Adverse event 0.05 0.14 0.05 0.02 0.06
Treatment impact 0.77 0.42 0.15 0.27 0.40
Co-payment 0.02 0.03 0.05 0.09 0.05
  1. Obtain normalized scores for each cell using the sum score
  2. The average for each row gives you the “priority scores”


Slide 14

Group decision support systems (GDSS)

Picture: screenshot of the GDSS program for comparing the clinical benefit of a drug to its impact of treatment.

Slide 15

Decision structure in case of anti-depressants

Heirarchical flowchart:

This chart links Prioritize endpoints to:

  • Efficacy
  • Adverse events
  • Disease specific QoL .

The flowchart shows the following hierarchical paths from left to right:

  • Efficacy points to:
  • Response
  • Remission
  • No relapse
  • Adverse events  points to:
  • Serious adverse events, which points to :
    • Suicide and attempted suicide
    • Other serious adverse events
  • Adverse events, which points to:
    • Sexual dysfunction
    • Other adverse events
  • Disease specific QoL points to:
    • Social function
    • Anxiety
    • Pain
    • Cognitive function

Slide 16

How patients and experts value patient relevant endpoints

Picture: Bar chart measuring response, remission and relapse for psychiatrists and patients for Effectiveness, Adverse Events and Quality of life. Data points not provided.

Danner, Hummel et al. Int. J. Techn. Assessm. Healthcare. 2011.

Slide 17

Weighted preference for three antidepressants

Picture: Line graph measuring weighted preferences for Venlafaxine, SSRI, Duloxetine. points not provided.

Slide 18

General remarks on use of MCDA and CA methods

  • MCDA or stated preference techniques do support decision making. They do not make the decision.
  • Stated preference techniques are used to obtain stakeholder preferences over a range of treatment options. MCDA techniques, like AHP, can also support the process of decision making. 
  • The value of MCDA is not the decision algorithm itself but the process of making the decision problem more transparent and explicit. 
  • Preference data is to be used in conjunction with clinical data. They cannot replace the original clinical evidence.