When to Transplant: Value Iteration and Optimal Stopping

Thu, 08 Dec 2011 00:00:00 +0000

Notes from ISE 789, the healthcare-optimization seminar I am taking this term at NC State. We keep circling a small, stylized model of a hard question: if you are on a transplant waiting list, when should you accept an organ rather than hold out for a better moment? The model is a toy, but the machinery behind it, value iteration on a Markov decision process, is the same dynamic programming that scales up to real sequential-decision problems. The math below is solved live, not baked in.

The model

The patient occupies one of five states. Three describe pre-transplant health, from Good to Fair to Poor. Two are absorbing: Transplanted (the organ has been accepted) and Death.

Each period the patient chooses one of two actions:

Wait. Stay on the list and live another period. Reward $r = 17$. Health may hold, decline, or, at worst, end in death.
Transplant now. Accept an organ. The reward is larger the healthier the patient, $100$, $75$, or $60$ from Good, Fair, and Poor. The process then ends.

The decision states are Good, Fair, and Poor. Waiting (gray) keeps the patient alive but risks decline or death; transplanting (green) ends the process with a health-dependent payoff. Exact transition probabilities appear below.

Waiting moves the patient between health states, or to death, with these probabilities:

From \ to	Good	Fair	Poor	Death
Good	0.80	0.10	0.05	0.05
Fair	0.10	0.20	0.40	0.30
Poor	0.05	0.05	0.10	0.80

Transplanting sends any health state to Transplanted with probability 1. Future rewards are discounted by $\lambda = 0.95$ per period.

Solving it by value iteration

The optimal value of a state satisfies the Bellman optimality equation: the best action is the one whose immediate reward plus discounted expected future value is largest.

$$V(s) = \max_{a \in \{\text{wait},\, \text{transplant}\}} \left[\, r(s, a) + \lambda \sum_{s'} P(s' \mid s, a)\, V(s') \,\right]$$

Value iteration turns this equation into an algorithm. Start from $V_0 = 0$ and apply the right-hand side as an update, $V_{k+1} = \mathcal{T} V_k$. Because $\lambda < 1$, the update $\mathcal{T}$ is a contraction, so the $V_k$ converge to the unique fixed point $V^\star$ regardless of where they start. The optimal policy reads off greedily: in each state, pick the action attaining the maximum.

The result: the healthiest patient waits

Value iteration converges to

$$V^\star = (\,112.40,\ 75,\ 60,\ 0,\ 0\,), \qquad \pi^\star = (\,\text{wait},\ \text{transplant},\ \text{transplant}\,).$$

The Fair and Poor states transplant immediately: their values equal the transplant payoffs $75$ and $60$ exactly, because acting now is best and the process ends. The interesting state is Good. Its value, $112.40$, is larger than the $100$ it would earn by transplanting today:

$$\begin{aligned} Q(\text{Good}, \text{wait}) &= 17 + 0.95\big(0.8 \cdot 112.40 + 0.1 \cdot 75 + 0.05 \cdot 60 + 0.05 \cdot 0\big) = 112.40, \\ Q(\text{Good}, \text{transplant}) &= 100, \qquad\text{so waiting wins.} \end{aligned}$$

That gap is the option value of waiting. A healthy patient is likely to stay healthy, collecting the $17$ per-period reward and keeping the choice open, so holding the organ in reserve beats spending it now. This is a control-limit policy: wait while you are above a health threshold, act once you cross it. Such thresholds show up throughout optimal stopping, from equipment replacement to exercising an option.

Play it yourself

The numbers make the case, but the tradeoff is easier to feel than to read. You are the patient, starting in Good health. Each period you choose: wait for a better moment, or accept the organ in front of you. The panel shows the odds you face and what the optimal policy would do; your job is to beat its long-run average of 112.40. Wait too long in a declining state and you will learn firsthand why the threshold exists.

What the discount factor decides

The whole story hinges on how much the future is worth. The discount factor $\lambda$ trades immediate payoff against the option value of waiting. Drag it and watch the policy move: a patient future keeps Good on the list, while a heavily discounted future tips every state toward transplanting now.