Introduction

Once the relatedness between the development and validation populations has been established (Step 1), the next task in the Debray framework is to rigorously assess how well the original prediction model performs in the new validation sample. This step focuses on core predictive performance metrics—calibration and discrimination—accompanied by essential visual assessments. Together, these provide a comprehensive picture of predictive accuracy and potential model misfit in the validation setting.

---

1. Calibration: Does the Model Predict the Right Absolute Risks?

Calibration examines the agreement between predicted probabilities and observed outcomes. A well-calibrated model produces predictions that closely match the true event rate among patients with similar predicted risk.

Debray et al. emphasize two summary metrics :

A. Calibration-in-the-Large (Intercept α)

Estimated using the recalibration model:

• Ideal value: α = 0
• Interpretation:
  • α < 0 → model overestimates risk / model overpredicts risk
  • α > 0 → model underestimates risk / model underpredicts risk

Since calibration-in-the-large reflects differences in overall outcome incidence between development and validation datasets, it often mirrors differences in LP mean identified in Step 1.

B. Calibration Slope (β)

Estimated from:

• Ideal value: β = 1  → perfect spread
• β < 1 → predictions too extreme "model overfitting" Predictions are too extreme, often from overfitting in development
• β > 1 → predictions too modest "model underfitting" Predictions are not extreme enough, suggesting weaker predictor effects in the validation sample

Calibration slope is central for determining whether the strength of predictor-outcome relationships is stable across populations. Key insight:

• CITL = mean shift
• Slope = spread distortion

These two metrics diagnose different forms of miscalibration. Below are four clear clinical examples showing

---

2. Discrimination: Can the Model Rank-Order Patients Correctly?

Discrimination describes the model’s ability to distinguish individuals with the outcome from those without it.

The main metric is:

Concordance (c) Statistic / AUROC

• Range: 0.5 (no discrimination) to 1.0 (perfect discrimination)
• Reflects: How well the model orders predicted risks across patients
• Influenced by: Case-mix heterogeneity
  • Validation samples with wider LP spread (higher LP SD) naturally yield higher c-statistics

Thus, discrimination must always be interpreted in light of Step 1.

---

3. Visual Calibration Assessment: Essential for Detecting Local Miscalibration

Debray et al. strongly advocate calibration plots, as summary metrics may hide region-specific errors.

Calibration Plot

• Patients are grouped into quantiles of predicted risk
• Observed outcome proportions per group are plotted against predicted probabilities
• Perfect performance lies along the 45° diagonal

Visual inspection identifies:

• Nonuniform miscalibration (e.g., correct at low risk but wrong at high risk)
• Threshold-specific problems important for clinical decision-making
• Prediction drift across the risk spectrum

The DVT empirical example used seven quantile groups, showing not only global miscalibration but also localized deviations across risk intervals.

---

4. Empirical Findings from Debray’s DVT Validation Studies

Using four datasets assessing a diagnostic model for deep venous thrombosis (DVT), Debray’s Step 2 results demonstrated how calibration and discrimination behaviors differ across populations with varying case mix.

Validation Study 1

• Calibration slope ≈ 0.90 → predictions slightly too extreme
• Calibration-in-the-large: −0.52 → model systematically overpredicted risk
• c ≈ 0.76 → small decrease from development sample
• Interpretation:
  • Sample very similar to development (Step 1) → assessing reproducibility
  • Minor miscalibration correctable via intercept update

Validation Study 2

• Improved c-statistic (≈ 0.82)
• Reflects greater case-mix heterogeneity (larger LP SD from Step 1)
• Calibration slope ~0.88, calibration-in-the-large nearly optimal
• Interpretation:
  • Despite population differences, discrimination improves due to diverse risk profiles
  • Model remains acceptably calibrated

Validation Study 3

• Strong discrimination (c ≈ 0.85)
• Calibration slope >1 (≈ 1.12)
  • Indicates predicted risks are not extreme enough
• Systematic underprediction at higher risks is visible in the calibration plot
• Interpretation:
  • Represents transportability assessment with pronounced case-mix differences
  • Miscalibration requires slope adjustment + intercept update

---

Summary of Step 2 Insights

• Predictive performance must be evaluated using both numeric metrics and visual calibration.
• Calibration-in-the-large diagnoses baseline risk differences.
• Calibration slope diagnoses scaling or overfitting issues.
• Discrimination depends strongly on LP variability, not just model quality.
• Calibration plots reveal miscalibration patterns that summary metrics cannot capture.
• Interpretation is inseparable from Step 1 (relatedness of samples).