Discrete choice experiments in practice

An introduction using teacher pension preferences

James Zuccollo

April 29, 2025

Introduction

What is a discrete choice experiment?

  • A method to understand preferences by analyzing choices between alternatives
  • Respondents choose their preferred option from sets of alternatives
  • Each alternative is described by different attributes and levels
  • Used to estimate the value of different attributes and predict choices

When to use DCEs?

  • Reveal preferences not captured in administrative datasets
  • Quantify policy trade-offs before implementation
  • Measure willingness to pay for education interventions
  • Predict take-up rates for new policies or reforms

The teacher pension research question

  • Teacher retention crisis + falling pay (-14% since 2010)
  • Pensions are generous (DB, 38.6% total contrib.) but inflexible
  • Research question: How do teachers value different compensation attributes?
  • Policy goal: Identify compensation structures that improve recruitment/retention

Designing a DCE

The main components of a DCE

  1. Attributes: The characteristics that define the alternatives (e.g., salary, pension type)
  2. Levels: The possible values of each attribute (e.g., current salary, +5%, +10%)
  3. Choice sets: Groups of alternatives presented to respondents
  4. Experimental design: The systematic arrangement of attributes and levels
  5. Statistical analysis: Models that estimate the impact of attributes on choices

Our pension DCE design

  • 3 attributes selected based on policy relevance:
    • Current salary: -10%, -5%, same, +5%, +10% (vs. own salary)
    • Retirement income: -20%, -10%, same, +10%, +20% (vs. own pension)
    • Certainty: Guaranteed (DB) vs. Stock market (DC)
  • 5 choice sets per respondent
  • Binary choices (Option A vs. Option B)
  • Sample: 5,750 teachers via Teacher Tapp panel

Sample size planning

  • Guidelines for sample size:
    • Rule of thumb: (500 × # of levels)/(# of alternatives × # of tasks)
    • Larger samples needed for subgroup analysis
  • Our pension study:
    • 3 attributes with 5,5,2 levels (12 total)
    • 2 alternatives per task × 5 tasks per teacher
    • Minimum sample: 500 × 12 ÷ (2 × 5) = 600 teachers
    • Actual sample: 5,750 teachers (allows detailed subgroup analysis)

Example choice set

Each teacher faced choices like this.

Teachers selected their preferred option in each of 5 sets.

Sets were randomized.

Statistical framework

The random utility model

  • Based on the idea that choice depends on utility derived from alternatives
  • Total utility has two components:
    • \(U_{ij} = V_{ij} + \epsilon_{ij}\)
    • Systematic component (V): The part we can measure
    • Random component (ε): The part we can’t observe
  • The probability that person \(i\) chooses option \(j\):
    • \(P(j) = P(U_{ij} > U_{ik})\) for all other options \(k\)

Basic statistical analysis

  • Outcome variable: Whether alternative was chosen (1) or not (0)
  • Predictors: Attribute variables
  • Coefficients: Effect of attributes on log-odds of selection
# Basic model structure
model <- glm(
  selected ~ choice_salary + choice_pension + choice_pensiontype,
  family = binomial(link = "logit"),
  data = analysisdata
)

Handling clustered data

  • Each respondent provides multiple observations (not independent)
  • Need to account for clustering in standard errors
# Implementation for our pension study
svy_design <- svydesign(
  ids = ~respondent,  # Clustering variable
  weights = ~1,       # Equal weights
  data = analysisdata
)

svy_model <- svyglm(
  selected ~ choice_salary + choice_pension + choice_pensiontype,
  family = quasibinomial(),
  design = svy_design
)

Interpreting results

Average marginal component effects (AMCEs)

  • Convert log-odds to probability scale for easier interpretation
  • Average change in probability when attribute level changes
  • Relative to a reference level (usually status quo)
  • Range from -1 to 1 (or -100% to +100%)
# Calculate AMCEs for our pension study
amces <- marginaleffects::avg_comparisons(svy_model)

Visualizing our pension AMCEs

Calculating willingness to pay

  • Measures how much of one attribute respondents will trade for another
  • Formula: WTP = -AMCE(attribute) ÷ AMCE(monetary attribute)
# For our pension study:
amce_dc <- -0.22      # AMCE for DC pension vs DB
amce_salary_10pct <- 0.09  # AMCE for 10% salary increase

# Teachers would give up 24.4% salary to keep DB pension
wtp_pension_type <- (amce_dc / amce_salary_10pct) * 10

Subgroup analysis

Analyzing preference heterogeneity

  • Different groups may have different preferences, so add interaction terms between attributes and demographics
  • Example: Do younger teachers value salary differently?
# Interaction model for our pension study
interaction_model <- svyglm(
  selected ~ choice_salary * demog_age + 
             choice_pension * demog_age + 
             choice_pensiontype * demog_age,
  family = quasibinomial(),
  design = svy_design
)

# Extract AMCEs for interaction model
amces_interaction <- marginaleffects::avg_comparisons(interaction_model)

Teacher pension preferences by age

Policy applications

Simulating policy scenarios

  • Use model to predict choices between policy options
  • Calculate preference shares through share of utility
# For our pension study:
policy_options <- tribble(
  ~choice_salary, ~choice_pension, ~choice_pensiontype,
  "10% higher",   "20% lower",     "Defined contribution",  # Policy A
  "Same",         "Same",          "Defined benefit"        # Status quo
)

# Calculate preference shares by age group
policy_sim <- marginaleffects::predictions(
  interaction_model, 
  newdata = policy_options,
  type = "link"
  ) |>
  group_by(demog_age) |>
  mutate(probability = exp(estimate) / sum(exp(estimate)))

Pension policy simulation results

  • Shows predicted choices between options
  • Key findings:
    • 15% of all teachers would choose salary over pension
    • Younger teachers more likely to switch (19%)
    • Most teachers still prefer DB pension

Practical outputs for policymakers

  • Willingness to pay:
    • “Teachers would give up 24% of salary for a DB pension”
    • “Younger teachers would sacrifice less for pension security”
  • Policy simulations:
    • “If we increase salary by 10% but reduce pension by 20%, 15% of teachers would choose this option”
    • “A flexible pension system would appeal to about 1 in 5 teachers under 30”

Implementation considerations

Common challenges in DCE implementation

  • Design complexity: Too many attributes increases cognitive burden
  • Solution: Focus on most important attributes, use blocking
  • Hypothetical bias: Stated preferences may differ from actual behavior
  • Solution: Make scenarios realistic, reference familiar experiences
  • Inattentive respondents: Some may not carefully consider choices
  • Solution: Include attention checks, look for dominance violations

Quality checks we used

  • Response time analysis (excluded top/bottom 2.5%)
  • Dominance tests (checked for irrational choices)
  • Pattern checking (identified straightlining)
  • All analyses robust to these exclusions

Software tools for DCE implementation

  • Analysis: Hand coding in R using {survey}, {marginaleffects}, {brms}. Could have used an all-in-one package like Apollo in R or Biogeme in Python.
  • Our workflow: R for analysis, Teacher Tapp for data collection

What would we do differently?

  • Sample more widely: Include graduates and non-teachers as comparators.
  • Use a hierarchical model: Partial pooling allows greater efficiency and a more saturated model. Also allows teacher-level predictions for shares.
  • Use a more complex design: Use a D-efficient design to reduce the number of choice sets needed. This would allow us to include more attributes and levels. Teacher Tapp doesn’t do this.
  • Improve the robustness testing: Use a likelihood test to check for inattentive respondents.

The End

Go forth and experiment!