Calibration in the Large & the Calibration Intercept
On this page
Abstract
A clinical prediction model can separate high-risk from low-risk patients beautifully and still be wrong about the actual risk level. Calibration in the large is the coarsest, most important calibration check: does the mean predicted risk equal the observed event rate? When a model is transported to a population where the disease is more, or less, common than in the development sample, it typically mis-states the overall level — the calibration curve sits parallel to, but off, the 45° line. This article defines calibration in the large, introduces the calibration intercept $A$ that measures it, and shows how an intercept-only recalibration, $\text{logit}(p)=A+\text{LP}$, restores the correct level without disturbing how the model ranks patients. An interactive plot and a line-by-line R/Python terminal let you watch the calibration curve lift back onto the diagonal as $A$ changes. This is Part 1 of a three-part series on calibration and recalibration.
Part 1 of 3 · Calibration & Recalibration. Next: the calibration slope [[calibration-slope]], then recalibration in practice [[recalibration-in-practice]].
Discrimination is not calibration
Two questions live inside every risk model. Discrimination asks: does the model give higher risks to the patients who go on to have the event? Calibration asks a different question: are the predicted probabilities numerically right — when the model says 20%, do about 20 in 100 such patients have the event? A model can rank patients perfectly (high AUC) and yet quote risks that are systematically too high or too low. Decisions made on the number — treat above 10%, reassure below 5% — depend on calibration, not ranking.
What "calibration in the large" means
Calibration in the large is the simplest calibration summary: it compares just two numbers — the mean predicted risk across everyone and the observed event rate.
$$\bar{p}=\frac{1}{n}\sum_{i=1}^{n}\hat p_i \qquad\text{vs.}\qquad \bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_i$$
If the model predicts an average risk of 23% but only 12% of patients actually had the event, predictions are, on average, roughly twice too high. On a calibration plot — observed risk against predicted risk by tenths of predicted risk — this shows up as a curve that runs below the diagonal everywhere: at every level the observed rate is lower than predicted. The most common cause is a different outcome prevalence: the model was developed where the disease was more common, and moving it to a lower-prevalence setting leaves its baseline risk too high.
The calibration intercept $A$
We quantify the level error with a single number, the calibration intercept $A$. Take the model's linear predictor for each patient,
$$\text{LP}=\hat\beta_0+\hat\beta_1x_1+\cdots+\hat\beta_kx_k,$$
hold it fixed as an offset, and fit only an intercept against the observed outcome:
$$\text{logit}\{P(Y=1)\}=A+\text{LP}.$$
The ideal value is $A=0$. A negative $A$ means the model over-predicts (too much baseline risk); a positive $A$ means it under-predicts. Because $A$ shifts every patient's log-odds by the same amount, it changes the overall level of risk but never the ordering of patients — discrimination is untouched.
Drag $A$ below and watch the calibration curve move. Press Fit A to jump to the value that best matches the data, and Reset to return to the frozen model.
Intercept-only recalibration
When the slope of the calibration line is already close to 1 — the predictor effects still hold, only the baseline differs — the fix is to update the intercept alone:
$$\text{logit}(p_{\text{updated}})=A+\text{LP},\qquad p_{\text{updated}}=\frac{1}{1+e^{-(A+\text{LP})}}.$$
This is exactly the situation of transporting a model to a population with a different outcome prevalence but similar predictor–outcome relationships. It is the lightest possible model update: one number, estimated from the new data, re-levels every prediction. If the calibration slope is also wrong — predictions too extreme or too flat — an intercept alone is not enough, and you move to the slope (Part 2) or to a joint update (Part 3).
A caution that Part 3 develops: you cannot honestly estimate $A$ by applying the model back to its own development data — there the apparent intercept is ~0 by construction. An honest correction needs a validation sample or resampling (bootstrap / cross-validation).
Run it yourself — R and Python
The terminal below runs the real analysis one line at a time on a simulated cohort of 3,000 patients: apply a frozen, over-predicting model, measure calibration in the large, estimate $A$, and watch the calibration plot on the Stage panel lift onto the diagonal after the intercept fix. Type the highlighted line (or press Enter to autofill and run), press Tab to accept the ghost, and switch the R / Python tabs for the same analysis in either language. Nothing runs in your browser; every value and plot was computed beforehand on synthetic data.
โมเดลทำนายที่ จำแนก ผู้ป่วยได้ดี ยังอาจให้ระดับความเสี่ยง โดยรวม ผิดได้ เมื่อนำไปใช้ในประชากรที่โรคพบบ่อยหรือน้อยกว่าเดิม เทอร์มินัลนี้รันการวิเคราะห์จริงทีละบรรทัดบนข้อมูล จำลอง 3,000 คน โมเดลเดิมที่ถูก "แช่แข็ง" ทำนายเฉลี่ย ~23% ทั้งที่เกิดเหตุการณ์จริงเพียง ~12% นี่คือปัญหา calibration in the large เราวัดด้วย calibration intercept (A) แล้วแก้แบบ intercept-only จนเส้น calibration ขยับกลับมาแนบเส้นทแยง คุณเป็นคนพิมพ์เอง กด Enter เพื่อเติมและรัน กด Tab เพื่อรับคำที่ระบบเดา แผง Stage ด้านขวาจะโชว์ค่าและ กราฟ calibration ที่เปลี่ยนไป สลับแท็บ R / Python ได้ ไม่มีการคำนวณในเบราว์เซอร์ ทุกค่าและกราฟถูกคำนวณไว้ก่อนด้วย R และ Python
A prediction model that ranks patients well can still get the overall risk level wrong when it is moved to a population where the disease is more, or less, common. This terminal runs the real analysis one line at a time on a simulated cohort of 3,000 patients. A frozen original model predicts an average risk of ~23% when only ~12% actually had the event — a calibration-in-the-large problem. We measure it with the calibration intercept (A) and apply an intercept-only recalibration until the calibration curve lifts back onto the 45° line. You drive it: press Enter to autofill and run, Tab to accept the autocomplete ghost, and the Stage on the right shows the numbers and the calibration plot changing. Switch between the R and Python tabs for the same analysis in either language. Nothing runs in your browser; every value and plot was computed in R and Python beforehand.
Key takeaways
- Calibration in the large compares mean predicted risk with the observed event rate — the overall risk level.
- The calibration intercept $A$ measures it: $A=0$ is perfect, $A<0$ over-prediction, $A>0$ under-prediction.
- Intercept-only recalibration $\text{logit}(p)=A+\text{LP}$ re-levels predictions without changing patient ranking — the right tool when the calibration slope is already near 1.
- Estimate $A$ from validation data or by resampling, never by re-using the raw development data.
Continue to Part 2 — the calibration slope [[calibration-slope]] for what to do when predictions are too extreme, and Part 3 — recalibration in practice [[recalibration-in-practice]] for updating the intercept and slope together with honest validation.