Beyond AUC: Understanding Net Reclassification Improvement (NRI) and Integrated Discrimination Improvement (IDI)

Mayta
Jul 24
3 min read

Introduction

When evaluating a new diagnostic test or biomarker, we often compare it against existing methods using the Area Under the ROC Curve (AUC). While AUC is helpful, it doesn’t always reveal how much a test improves decision-making. That’s where reclassification analysis comes in. Two modern tools—Net Reclassification Improvement (NRI) and Integrated Discrimination Improvement (IDI)—can quantify how much a new test changes clinical decisions and improves diagnostic precision.

Why Go Beyond the ROC Curve?

The AUC measures a model’s ability to discriminate between those with and without disease. But:

AUC increases may be small even when a test is clinically meaningful.
AUC doesn't directly show how many patients are classified better (or worse).

In many clinical situations, especially where decision thresholds guide treatment, we need more actionable measures.

Concept of Diagnostic Added Value

Imagine you already use a standard test (Test A) to estimate disease probability, which gives an AUC of 0.70. Now, suppose adding a new test (Test B) increases the combined AUC to 0.80.

The diagnostic added value of Test B is the improvement beyond what Test A already provides:

Added value = AUC(Test A + Test B) − AUC(Test A)
In this case: 0.80 − 0.70 = 0.10

This difference reflects enhanced discrimination. However, it does not explain how individual patients shift across decision thresholds. That’s where NRI and IDI become valuable.

Reclassification Tables: Understanding Patient Movement

When a new test is added to a prediction model, some patients are reclassified across a clinical decision threshold (e.g., 25% predicted risk of disease). A reclassification table records how many individuals move:

Upward (from below to above the threshold)
Downward (from above to below the threshold)

These shifts are then assessed separately for:

Patients with the disease (cases)
Patients without the disease (non-cases)

Example: Deep Vein Thrombosis (DVT)

416 patients had confirmed DVT.
- 123 moved upward with the new model.
- 26 moved downward.
1670 patients did not have DVT.
- 116 moved upward.
- 227 moved downward.

Net Reclassification Improvement (NRI)

Formula

NRI =[P(up | D = 1) − P(down | D = 1)] +[P(down | D = 0) − P(up | D = 0)]

Where:

P(up | D = 1): Proportion of true cases moved up
P(down | D = 1): Proportion of true cases moved down
P(up | D = 0): Proportion of non-cases moved up (bad)
P(down | D = 0): Proportion of non-cases moved down (good)

Calculation in DVT example

For cases: (123 / 416) − (26 / 416) = 0.30 − 0.06 = 0.24
For non-cases: (227 / 1670) − (116 / 1670) = 0.14 − 0.07 = 0.07
NRI = 0.24 + 0.07 = 0.31

Interpretation: A net of 31% more patients were correctly reclassified using the new model.

Limitations of NRI

NRI depends on the choice of the probability cutoff. A model might perform differently if a 10% or 50% threshold is used instead of 25%. To avoid this subjectivity, another metric—IDI—can be used.

Integrated Discrimination Improvement (IDI)

The IDI compares how much the average predicted probability for each group (diseased and non-diseased) changes between the two models.

Formula

IDI =[(Average P in cases: extended model − base model)] −[(Average P in non-cases: extended model − base model)]

Example

Let’s say:

Mean predicted probability for cases increases from 0.13 → 0.49
Mean predicted probability for non-cases increases from 0.18 → 0.28

IDI = (0.49 − 0.13) − (0.28 − 0.18) = 0.36 − 0.10 = 0.26

Interpretation: The extended model improves discrimination between cases and non-cases by 26%.

Summary of Concepts

Measure	Captures	Requires Cutoffs?	Clinical Use
AUC	Overall discrimination	No	General model comparison
NRI	Direction of movement across decision thresholds	Yes	Useful when treatment decisions hinge on specific risk levels
IDI	Average separation of predicted probabilities	No	Provides overall improvement regardless of cutoffs

Conclusion

Evaluating a diagnostic test should go beyond just checking the AUC. Tools like NRI and IDI offer deeper insight into how a new biomarker or test changes individual patient classification—exactly what clinicians need for real-world decisions. These measures bring us closer to personalized, data-driven diagnosis and treatment planning.

Let me know when you're ready for the next concept or if you’d like a figure-style layout for the reclassification table and formulas.