Beyond AUC: Understanding Net Reclassification Improvement (NRI) and Integrated Discrimination Improvement (IDI)
On this page
Introduction
When evaluating a new diagnostic test or biomarker, we often compare it against existing methods using the Area Under the ROC Curve (AUC). While AUC is helpful, it doesn’t always reveal how much a test improves decision-making. That’s where reclassification analysis comes in. Two modern tools—Net Reclassification Improvement (NRI) and Integrated Discrimination Improvement (IDI)—can quantify how much a new test changes clinical decisions and improves diagnostic precision.
Why Go Beyond the ROC Curve?
The AUC measures a model’s ability to discriminate between those with and without disease. But:
- AUC increases may be small even when a test is clinically meaningful.
- AUC doesn't directly show how many patients are classified better (or worse).
In many clinical situations, especially where decision thresholds guide treatment, we need more actionable measures.
Concept of Diagnostic Added Value
Imagine you already use a standard test (Test A) to estimate disease probability, which gives an AUC of 0.70. Now, suppose adding a new test (Test B) increases the combined AUC to 0.80.
The diagnostic added value of Test B is the improvement beyond what Test A already provides:
- Added value = AUC(Test A + Test B) − AUC(Test A)
- In this case: 0.80 − 0.70 = 0.10
This difference reflects enhanced discrimination. However, it does not explain how individual patients shift across decision thresholds. That’s where NRI and IDI become valuable.
Reclassification Tables: Understanding Patient Movement
When a new test is added to a prediction model, some patients are reclassified across a clinical decision threshold (e.g., 25% predicted risk of disease). A reclassification table records how many individuals move:
- Upward (from below to above the threshold)
- Downward (from above to below the threshold)
These shifts are then assessed separately for:
- Patients with the disease (cases)
- Patients without the disease (non-cases)
Example: Deep Vein Thrombosis (DVT)
- 416 patients had confirmed DVT.
- 123 moved upward with the new model.
- 26 moved downward.
- 1670 patients did not have DVT.
- 116 moved upward.
- 227 moved downward.
Net Reclassification Improvement (NRI)
Formula
NRI =[P(up | D = 1) − P(down | D = 1)] +[P(down | D = 0) − P(up | D = 0)]
Where:
- P(up | D = 1): Proportion of true cases moved up
- P(down | D = 1): Proportion of true cases moved down
- P(up | D = 0): Proportion of non-cases moved up (bad)
- P(down | D = 0): Proportion of non-cases moved down (good)
Calculation in DVT example
- For cases: (123 / 416) − (26 / 416) = 0.30 − 0.06 = 0.24
- For non-cases: (227 / 1670) − (116 / 1670) = 0.14 − 0.07 = 0.07
- NRI = 0.24 + 0.07 = 0.31
Interpretation: A net of 31% more patients were correctly reclassified using the new model.
Limitations of NRI
NRI depends on the choice of the probability cutoff. A model might perform differently if a 10% or 50% threshold is used instead of 25%. To avoid this subjectivity, another metric—IDI—can be used.
Integrated Discrimination Improvement (IDI)
The IDI compares how much the average predicted probability for each group (diseased and non-diseased) changes between the two models.
Formula
IDI =[(Average P in cases: extended model − base model)] −[(Average P in non-cases: extended model − base model)]
Example
Let’s say:
- Mean predicted probability for cases increases from 0.13 → 0.49
- Mean predicted probability for non-cases increases from 0.18 → 0.28
IDI = (0.49 − 0.13) − (0.28 − 0.18) = 0.36 − 0.10 = 0.26
Interpretation: The extended model improves discrimination between cases and non-cases by 26%.
Summary of Concepts
| Measure | Captures | Requires Cutoffs? | Clinical Use |
| AUC | Overall discrimination | No | General model comparison |
| NRI | Direction of movement across decision thresholds | Yes | Useful when treatment decisions hinge on specific risk levels |
| IDI | Average separation of predicted probabilities | No | Provides overall improvement regardless of cutoffs |
Conclusion
Evaluating a diagnostic test should go beyond just checking the AUC. Tools like NRI and IDI offer deeper insight into how a new biomarker or test changes individual patient classification—exactly what clinicians need for real-world decisions. These measures bring us closer to personalized, data-driven diagnosis and treatment planning.
Let me know when you're ready for the next concept or if you’d like a figure-style layout for the reclassification table and formulas.