Beyond Calibration: The Rise of Prediction Stability

While calibration corrects the long-standing misconception that discrimination alone is sufficient, it does not answer a deeper structural question:

“Can I trust the model to give the same predictions if the training sample had been slightly different?”

This is the domain of Prediction Stability—a newly recognized, methodologically essential property of Clinical Prediction Models (CPMs), highlighted in modern CPM development frameworks, including "The 9-step CPM roadmap".

1. Stability: The Missing Third Pillar After ROC and Calibration

Traditionally, CPM performance was judged on two axes:

1. Discrimination (ROC / AUROC)

How well the model separates future cases from non-cases.

2. Calibration

How closely predicted probabilities match observed event rates.

These answer:

“Does the model rank patients correctly?”
“Are the absolute risks trustworthy?”

But neither evaluates whether the model itself is robust to sampling variability. A model can show excellent AUROC and near-perfect calibration on the derivation dataset while remaining fragile, overfitted, and unreliable when rebuilt on another sample from the same population.

Stability fills this gap.

2. What Stability Really Means

Building on the CPM logic:

Prediction stability is the condition in which the predictions for any given patient remain similar even if the model is derived from different samples of the same size from the same population.

This aligns to the principle that the objective of prediction modeling is not inference but accurate predictions for unseen individuals.

A stable CPM ensures:

Predictor coefficients do not fluctuate wildly across resamples.
Predictions for a patient with fixed characteristics remain consistent.
Apparent performance ≈ test performance (low optimism/overfitting).
Internal validation (bootstrap/cross-validation) reveals minimal drift.

Stability is therefore a property of the derivation process, not just the resulting model.

3. Why Stability Is Now Clinically Essential

1. Unstable models fail implementation

Clinicians cannot adopt a CPM whose predictions “move” depending on the training subset. Stability ensures trustworthiness and reproducibility—essential for clinical decision support.

2. Stability is tightly linked to sample size

Modern CPM methodology rejects the old “10 events per variable” rule. Instead, it recommends explicitly sizing samples to ensure unbiased and stable predictions, consistent with CECS Step 4: modern sample-size logic for CPMs .

3. Stability is the first safeguard against overfitting

Overfitted models capture noise, leading to:

inflated AUROC (apparent performance)
poor generalization
unstable coefficients
high variance in predictions

Stability evaluation exposes this immediately.

4. Stability predicts external validation success

A stable model internally will more likely maintain calibration and discrimination during geographic or domain validation, where performance differences are typically largest.

4. How Stability Is Evaluated

CECS documentation lists the full suite of stability diagnostics used in Step 9 of model evaluation :

1. Mean Average Probability Error (MAPE) Instability Plot

Assesses how predicted probabilities vary across bootstrap samples.

2. Prediction Instability Plot

Shows the spread of predicted risks per patient across resamples.

3. Calibration Instability Plot

Visualizes how the calibration curve fluctuates across resamples.

4. Classification Instability Plot

For threshold-based rules (e.g., “high-risk ≥ 10%”), it shows how often a patient is reclassified into a different risk category.

Interpretation Logic:

Narrow bands → stable
Wide, erratic bands → unstable (often due to overfitting, sparse data, or too many candidate predictors)

5. Stability as the Bridge to Generalizability

The CPM development continuum is:

Development → Internal Validation → External Validation → Implementation

Stability matters most in the first two stages.

If a model is internally unstable:

Bootstrapped optimism is large.
Corrected calibration slope < 1.0.
Coefficients fluctuate substantially.
External validation is likely to collapse.

If a model is internally stable:

Calibration slope is near 1.0.
Predictor contributions are consistent.
External drift is easier to interpret (true population differences vs. modeling artifacts).
Implementation becomes realistic.

Stability is therefore a prerequisite for external validation—not a substitute.

6. Stability and the Philosophy of Prediction

Stability represents the philosophical shift emphasized in CECS CPM guidance:

In prediction modeling, the goal is not to test hypotheses or identify “significant” predictors; the goal is stable and unbiased prediction for future patients.

Stability enforces:

Fully pre-specified predictor sets
Avoidance of data-driven selection unless using penalized methods
Preference for penalization (ridge, lasso, elastic net) when many predictors exist
Proper predictor handling (splines instead of categorization)
Adequate sample size planning

This aligns with the modern shift away from classical regression-inference thinking toward predictive modeling logic.

7. Practical Implications: What Clinicians Should Demand

When reviewing a CPM manuscript or developing your own model, insist on:

1. Stability diagnostics (plots + summary metrics)

Not just AUROC and calibration.

2. Clear sample size justification based on stability targets

Not events-per-variable.

3. Internal validation using bootstrapping

Preferably ≥ 200 repetitions.

4. Optimism-corrected performance reporting

Slope-adjusted calibration, optimism-adjusted AUROC.

5. Penalized regression when appropriate

Especially with high-dimensional or correlated predictors.

6. External validation only after stability is demonstrated

Premature external validation misleads the field.

8. Conclusion: Stability as the Third Dimension of Prediction Quality

To evaluate a model properly, clinicians and researchers must move beyond the two-dimensional mindset of ROC and calibration:

ROC/AUROC tells us whether the model separates cases from non-cases.
Calibration tells us whether the risk estimates are correct.
Stability tells us whether the model is trustworthy and reproducible.

A clinically deployable CPM is one that:

Discriminates well
Is well-calibrated
Has stable predictions

Without stability, discrimination and calibration are illusions of a single sample.With stability, the model becomes a reliable clinical tool.

Key Takeaways

ROC is not enough—it ignores absolute risk and says nothing about model fragility.
Calibration improves trust, but only within one dataset.
Stability is the structural guarantee that predictions are robust to sampling variability.
Stability is central to CPM development (Step 9) and modern prediction methodology.
Only models that are stable internally deserve external validation and eventual clinical implementation.

Stability: The Key to Trustworthy Clinical Prediction Models