Classic MAPE: Mean Absolute Prediction Error and Bootstrap Internal Validation

Apparent Performance and Internal Validation Using Bootstrap

---

1. Introduction

In clinical prediction models, performance is commonly evaluated using metrics such as:

• AUROC → discrimination (ranking ability)

• Calibration slope & intercept → agreement between predicted and observed risk

• Brier score → overall accuracy

However, an additional intuitive metric is:

Mean Absolute Prediction Error (MAPE)

This metric directly quantifies how far predicted probabilities are from actual outcomes.

---

2. Definition of MAPE (classic form)

Here MAPE means the mean absolute error between predicted probabilities and observed binary outcomes—on the probability scale (0–1), not the textbook “percentage error” formula often also called MAPE in other fields.

Where:

• p^i = predicted probability (0–1)

• yi = observed outcome (0 or 1)

👉 This is mathematically equivalent to Mean Absolute Error (MAE) applied to predicted probabilities.

---

3. Interpretation of MAPE

MAPE represents:

Average absolute difference between predicted risk and actual outcome

Example:

• Patient 1: predicted = 0.65, outcome = 1 → error = 0.35

• Patient 2: predicted = 0.20, outcome = 0 → error = 0.20

If average across all patients = 0.25 → On average, predictions are 25% away from the truth

---

4. Apparent MAPE

Definition

Apparent MAPE is calculated using:

• The final model

• Evaluated on the same dataset used for model development

---

Key Insight (Important)

Even on training data:

MAPE ≠ 0

Why?

• Logistic regression predicts probabilities, not exact outcomes

• Patients with similar predictors may have different outcomes

• The model estimates average risk, not individual truth

👉 Therefore, error always exists, even in training data

---

Interpretation

• Apparent MAPE is optimistically low

• Because the model is evaluated on data it has already “seen”

---

5. Internal Validation Using Bootstrap

To correct for optimism, bootstrap resampling is used.

Algorithm (e.g., 500 iterations)

For each bootstrap iteration (b):

---

Step 1 – Resample

• Draw a bootstrap sample (with replacement) from original data

---

Step 2 – Fit model

• Fit logistic regression on bootstrap sample

---

Step 3 – Apparent performance (app_b)

• Predict on bootstrap sample

• Compute:

(This is optimistic)

---

Step 4 – Test performance (test_b)

• Use same bootstrap model

• Predict on original dataset

• Compute:

(This is more realistic)

---

Step 5 – Optimism

Because training error is lower:

👉 optimism is typically negative

---

6. Optimism-Corrected MAPE

After all bootstrap iterations:

Then:

---

Key Property

Since:

• apparent MAPE < test MAPE

• optimism < 0

Then:

Corrected MAPE > Apparent MAPE

---

Interpretation

Corrected MAPE represents:

Expected prediction error in new patients from the same population

---

7. Comparison with AUROC

---

Important Insight

A model can have:

• High AUROC → good ranking

• High MAPE → poor probability accuracy

👉 Therefore, MAPE adds complementary information

---

8. Role of MAPE in Clinical Research

MAPE is useful when:

• You care about the accuracy of predicted probabilities

• You want a simple, interpretable error metric

However, it should be supplementary, not primary.

Recommended reporting:

• AUROC

• Calibration slope & intercept

• Brier score

• MAPE (optional but informative)

---

9. Key Takeaways

• MAPE = mean absolute difference between predicted probability and outcome

• Apparent MAPE is optimistically low

• Bootstrap estimates and corrects this optimism

• Corrected MAPE reflects real-world expected error

• MAPE complements (but does not replace) AUROC