- Webcast: Multi-Criteria Decision Analysis Techniques
- Integrating Stakeholder Preferences in Text Format
Integrating stakeholder preferences in comparative effectiveness research using multi-criteria decision analysis (MCDA) and Conjoint Analysis (CA)
Maarten J. IJzerman, PhD
Professor and chair, department Health Technology & Services Research
Multiple endpoints in comparative effectiveness research
- In CER we wish to make an informed decision based on available clinical evidence for multiple endpoints.
- Primary endpoints chosen in clinical trials may not be the most relevant endpoints for patients and other stakeholders.
- Also, process related factors contribute to the actual use by stakeholders and are usually neglected in decision making.
- Three approaches for including stakeholder preferences
- Stakeholder representatives in appraisal committees
- Elicitation of preferences to guide deliberative process
- Integration of preferences in quantitative framework
All MCDA and CA methods decompose a decision problem into a set of criteria (attributes and levels):
- I want the best treatment:
|Clinical Outcome||Adverse Events||Out-of-Pocket Costs|
|Drug A||symptom relief||high blood pressure||5 US$|
|Drug B||partial relief||high blood pressure||3 US$|
|Drug C||partial relief||No adverse events||5 US$|
And they use a linear additive value function to determine the relative preference for the alternatives:
- I want the best treatment:
C1 = 0.5
C2 = 0.33
C3 = 0.17
|Drug A||symptom relief
L1.1 = 0.5
|high blood pressure
L2,1 = 0.25
L3,1 = 0.25
|Drug B||partial relief
L1,2 = 0.25
|high blood pressure
L2,2 = 0.25
L3,2 = 0.5
|Drug C||partial relief
L1,3 = 0.25
|No adverse events
L2,3 = 0.5
L3,3 = 0.25
drug A: (C1*L1,1 + C2*L2,1 + C3*L3,1) = 0.375
Multi-criteria decision analysis and Conjoint Analysis
- Multi-criteria Decision analysis (MCDA) is a subdiscipline of operations research that explicitly considers multiple criteria in decision making. MCDA methods enable the evaluation of many alternatives by explicit ranking, rating or pairwise comparison of criteria and alternatives. (See: Belton & Steward, 2002)
- The Analytic Hierarchy Process (AHP) is one of the most widely used MCDA techniques available (Saaty, 1989)
- Conjoint Analysis (CA) methods offer subjects a series of choices among two or more product profiles. The pattern of choices reveals the implicit decision weights patients attach to therapeutic benefits, harms, processes and costs that describe the treatment profiles (ISPOR taskforce, 2012)
MCDA and CA: an example
- Suppose a study in which we will select a good restaurant for dinner. We have defined 3 criteria (Ci) and 3 levels (Li)
- Cooking style: Italian, Greek or Thai
- Travel distance: need a car, walking or cycling
- Price: 15$, 20$, 30$
- Conjoint analysis will generate scenarios based on the levels (Li). In this case, 27 (33) scenario’s describe all possible combinations in the decision space.
- MCDA will ask you which criteria and levels (Ci and Li) are important using ranking, rating or pairwise comparisons. 12 comparisons can describe the complete decision space.
- Each participant receives N choice sets.
- Based on their response it is possible to estimate the part-worth utility of the levels and to estimate attribute weights
- E.g. if Thai food is presented in X scenarios and if a respondent picks all these scenarios we can assume Thai food is an important factor
Please select the most preferred restaurant.
Need a car,
Need a car
|Checkbox: not checked||Checkbox: checked|
Analytic Hierarchy Process (MCDA)
(Flowchart: Hierarchical relationships from left to right, top to bottom)
- Choose the best restaurant
- Style (0,5)
- Thai (0,5)
- Italian (0,3)
- Greek (0,2)
- Price (0,3)
- 15$ (0,7)
- 25$ (0,1)
- Travel (0,2)
- Car (0,3)
- Bicycle (0,1)
- Foot (0,6)
- Style (0,5)
Restaurant 1 (0.52): Includes arrows that point upward toward the boxes Thai, 20$, and bicycle
Restaurant 2 ( 0.23): Includes arrows that point upward toward the boxes Greek, 15$, and car
Restaurant 3 ( 0.25): Includes arrows that point upward toward the boxes Italian, 25$, and foot
MCDA vs. Conjoint analysis methods
MCDA (e.g. AHP)
- Helps to improve judgment
- Low cognitive stress
- Less realistic
- Not really used in marketing
- Decision sciences / OR
- Possible to use in N=1 or as consensus building tool (group-decision)
- Imitates consumers’ judgment
- Holistic evaluation
- Matrix structure
- Cognitive stress
- Realistic task
- Less flexible (e.g. design)
- Use in marketing (i.e. creates a hypothetical market)
- Large datasets required
MCDA methods: the Analytic Hierarchy Process
Picture: Hierarchical flowchart of the analytic hierarchy process. The top object labeled Decision objective (treatment) is linked to objects below labeled Criterion 1, Criterion 2 and Criterion 3 with the corresponding equation C1+C2+C3 = 1.
Criterion 3 is broken down into Criterion 3.1 and Criterion 3.2 below the main Criterion 3 box with the corresponding equation C3.1+C3.2 = 1.
Below these boxes are objects labeled Alternative 1 and Alternative 2 that are linked by lines to all the objects above. These have the corresponding equation A1+A2 = 1
Picture: screenshot of a program for comparing the clinical benefit of a drug to its impact of treatment based on the Verbal scale of importance in pairwise comparisons. A decision matrix is highlighted that includes: Clinical benefit, Impact of treatment, Side effects, additional costs to patients.
Verbal scale of importance in pairwise comparisons
- Equal to moderate
- Moderate to Strong
- Strong to very strong
- Very strong
- Very strong to extreme
Eigenvalue to obtain criteria (Ci) and level (Li) weights (1)
|Clinical benefit||Adverse event||Treatment impact||Co-payment|
Matrix with pairwise comparisons (4 criteria, 6 comparisons)
Reciprocal scores of pairwise comparisons
Eigenvalue to obtain criteria (Ci) and level (Li) weights (2)
|Clinical benefit||Adverse event||Treatment impact||Co-payment||Average|
- Obtain normalized scores for each cell using the sum score
- The average for each row gives you the “priority scores”
Group decision support systems (GDSS)
Picture: screenshot of the GDSS program for comparing the clinical benefit of a drug to its impact of treatment.
Decision structure in case of anti-depressants
This chart links Prioritize endpoints to:
- Adverse events
- Disease specific QoL .
The flowchart shows the following hierarchical paths from left to right:
- Efficacy points to:
- No relapse
- Adverse events points to:
- Serious adverse events, which points to :
- Suicide and attempted suicide
- Other serious adverse events
- Adverse events, which points to:
- Sexual dysfunction
- Other adverse events
- Disease specific QoL points to:
- Social function
- Cognitive function
How patients and experts value patient relevant endpoints
Picture: Bar chart measuring response, remission and relapse for psychiatrists and patients for Effectiveness, Adverse Events and Quality of life. Data points not provided.
Danner, Hummel et al. Int. J. Techn. Assessm. Healthcare. 2011.
Weighted preference for three antidepressants
Picture: Line graph measuring weighted preferences for Venlafaxine, SSRI, Duloxetine. points not provided.
General remarks on use of MCDA and CA methods
- MCDA or stated preference techniques do support decision making. They do not make the decision.
- Stated preference techniques are used to obtain stakeholder preferences over a range of treatment options. MCDA techniques, like AHP, can also support the process of decision making.
- The value of MCDA is not the decision algorithm itself but the process of making the decision problem more transparent and explicit.
- Preference data is to be used in conjunction with clinical data. They cannot replace the original clinical evidence.