Classic MAPE: Mean Absolute Prediction Error and Bootstrap Internal Validation

Clinical Epidemiology ResearchUniqcret doctor knowledgesData Analytics or Statistics
Classic MAPE: Mean Absolute Prediction Error and Bootstrap Internal Validation
On this page

Apparent Performance and Internal Validation Using Bootstrap


1. Introduction

In clinical prediction models, performance is commonly evaluated using metrics such as:

  • AUROC → discrimination (ranking ability)
  • Calibration slope & intercept → agreement between predicted and observed risk
  • Brier score → overall accuracy

However, an additional intuitive metric is:

Mean Absolute Prediction Error (MAPE)

This metric directly quantifies how far predicted probabilities are from actual outcomes.


2. Definition of MAPE (classic form)

Here MAPE means the mean absolute error between predicted probabilities and observed binary outcomes—on the probability scale (0–1), not the textbook “percentage error” formula often also called MAPE in other fields.

MAPE=1n∑i=1n|p^i-yi|

Where:

  • p^i = predicted probability (0–1)
  • yi = observed outcome (0 or 1)

👉 This is mathematically equivalent to Mean Absolute Error (MAE) applied to predicted probabilities.


3. Interpretation of MAPE

MAPE represents:

Average absolute difference between predicted risk and actual outcome

Example:

  • Patient 1: predicted = 0.65, outcome = 1 → error = 0.35
  • Patient 2: predicted = 0.20, outcome = 0 → error = 0.20

If average across all patients = 0.25 → On average, predictions are 25% away from the truth


4. Apparent MAPE

Definition

Apparent MAPE is calculated using:

  • The final model
  • Evaluated on the same dataset used for model development
MAPEapparent=1n∑|p^imodel-yi |

Key Insight (Important)

Even on training data:

MAPE ≠ 0

Why?

  • Logistic regression predicts probabilities, not exact outcomes
  • Patients with similar predictors may have different outcomes
  • The model estimates average risk, not individual truth

👉 Therefore, error always exists, even in training data


Interpretation

  • Apparent MAPE is optimistically low
  • Because the model is evaluated on data it has already “seen”

5. Internal Validation Using Bootstrap

To correct for optimism, bootstrap resampling is used.

Algorithm (e.g., 500 iterations)

For each bootstrap iteration (b):


Step 1 – Resample

  • Draw a bootstrap sample (with replacement) from original data

Step 2 – Fit model

  • Fit logistic regression on bootstrap sample

Step 3 – Apparent performance (app_b)

  • Predict on bootstrap sample
  • Compute:
MAPEapp(b)

(This is optimistic)


Step 4 – Test performance (test_b)

  • Use same bootstrap model
  • Predict on original dataset
  • Compute:
MAPEtest(b)

(This is more realistic)


Step 5 – Optimism

optimism(b)=MAPEapp(b)-MAPEtest(b)

Because training error is lower:

👉 optimism is typically negative


6. Optimism-Corrected MAPE

After all bootstrap iterations:

Meanoptimism=1B∑optimism(b)

Then:

MAPEcorrected=MAPEapparent-Meanoptimism

Key Property

Since:

  • apparent MAPE < test MAPE
  • optimism < 0

Then:

Corrected MAPE > Apparent MAPE


Interpretation

Corrected MAPE represents:

Expected prediction error in new patients from the same population


7. Comparison with AUROC

Metric Measures Clinical meaning
AUROC Discrimination Can the model rank high vs low risk?
MAPE Absolute error How close predicted risks are to truth

Important Insight

A model can have:

  • High AUROC → good ranking
  • High MAPE → poor probability accuracy

👉 Therefore, MAPE adds complementary information


8. Role of MAPE in Clinical Research

MAPE is useful when:

  • You care about the accuracy of predicted probabilities
  • You want a simple, interpretable error metric

However, it should be supplementary, not primary.

  • AUROC
  • Calibration slope & intercept
  • Brier score
  • MAPE (optional but informative)

9. Key Takeaways

  • MAPE = mean absolute difference between predicted probability and outcome
  • Apparent MAPE is optimistically low
  • Bootstrap estimates and corrects this optimism
  • Corrected MAPE reflects real-world expected error
  • MAPE complements (but does not replace) AUROC
0
Message for International and Thai ReadersUnderstanding My Medical Context in ThailandRead more →Message for International and Thai ReadersUnderstanding My Broader Content Beyond MedicineRead more →

Comments

No comments yet. Be the first to share your thoughts.

Sign in to comment