Why the Observed Outcome Belongs in the Imputation Model, and When It Becomes Leakage
On this page
อ่านฉบับภาษาไทย (Thai version)
Abstract
When a baseline predictor is partly missing in an outcome regression, the observed outcome usually belongs in the multiple-imputation model, because the outcome carries information about the missing predictor and leaving it out weakens the predictor-outcome association the regression is meant to estimate. The analysis model explains the outcome; the imputation model only reconstructs the missing predictor, and the two purposes must not be confused. Using an observed outcome is not automatically data leakage. Leakage is about availability: in an explanatory study and during prediction-model development the outcome is known and may inform the imputation; validation outcomes must not contaminate how performance is judged; and for a new patient the outcome does not yet exist, so it cannot fill a missing predictor. Including the outcome does not make data missing completely at random, does not turn data missing not at random into missing at random, and does not remove the need for several imputations. This article works through a five-patient example and states the rule for each setting.
A biomarker, a death register and a missing value
A research team wants to know whether a baseline biomarker, measured at hospital admission, is associated with death during the admission. Age and illness severity are recorded for every patient, and so is death. The biomarker is not: for some patients the sample was never sent, or the result never reached the record.
The team plans to fill the gaps by multiple imputation. A colleague trained in machine learning objects to putting death into the imputation model: the outcome, she says, must never enter predictor preprocessing, because that is data leakage. The objection mixes up two different jobs, and the rest of this article separates them.
The problem: a missing predictor in an outcome regression
Suppose the scientific model is
$$Y = \beta_0 + \beta_1 X + \beta_2 Z + \varepsilon$$
where $Y$ is the observed outcome, $X$ is an important baseline predictor that is partly missing, and $Z$ stands for the other baseline covariates. The aim is to estimate the relationship between $X$ and $Y$, but some values of $X$ are missing.
A common solution is multiple imputation (MI): each missing value is filled several times with plausible values drawn from a model, each completed dataset is analysed, and the results are combined. The important question is:
When imputing missing $X$, should the observed outcome $Y$ be included in the imputation model?
Usually, yes. $Y$ contains information about $X$, and excluding $Y$ can weaken the $X$-$Y$ relationship that the final regression is intended to estimate. That is the logic behind the short answer: the outcome enters the imputation model to preserve predictor-outcome associations and improve prediction, in a sense of "prediction" that a later section makes precise.
Analysis model versus imputation model
These are two different models.
Scientific (substantive) model
This is the model that answers the research question, $Y = f(X, Z)$. In the example:
$$\text{Mortality} = f(\text{Biomarker}, \text{Age}, \text{Severity})$$
Imputation model
This model is used only to reconstruct plausible values of a missing variable. If the biomarker $X$ is missing:
$$X_{\text{missing}} = f(Y, \text{Age}, \text{Severity}, \text{other useful variables})$$
A useful general principle is that the imputation model should be sufficiently rich and compatible with the analysis model. Variables used in the analysis, along with variables that help predict the missing values or the chance of being missing, are generally candidates for inclusion [1].
Why does the outcome help impute a missing predictor?
| Patient | Biomarker $X$ | Death $Y$ |
|---|---|---|
| A | 2 | 0 |
| B | 3 | 0 |
| C | ? (missing) | 1 |
| D | 9 | 1 |
| E | 10 | 1 |
Reading patient C's missing value from the other four
-
Look at the recorded pairs
Patients A and B have biomarker values 2 and 3 and survived ($Y = 0$); patients D and E have 9 and 10 and died ($Y = 1$). Among patients with a recorded biomarker, larger $X$ tends to occur among those who die.
-
Conclude that the outcome carries information about the predictor
Because $X$ and $Y$ move together, knowing a patient's outcome says something about that patient's likely biomarker value: $Y$ contains information about $X$.
-
Use it for patient C
Patient C's biomarker is missing, but C died ($Y = 1$). Knowing that $Y = 1$ helps generate plausible values of the missing $X$.
-
Do not plug in a single value
Multiple imputation does not simply say "The patient died, therefore set $X = 9$." It uses the joint relationships among the variables and incorporates uncertainty to generate several plausible values.
-
Draw several values, analyse, combine
\[ X_C^{(1)} = 7.8, \quad X_C^{(2)} = 9.1, \quad X_C^{(3)} = 8.4, \quad \dots \]
Each draw goes into its own completed dataset, and so on for as many datasets as are created. Those datasets are then analysed and the results combined.
Result: Because patient C died, C's missing biomarker is drawn as several plausible values, such as 7.8, 9.1 and 8.4, rather than fixed at 9; the completed datasets are then analysed and combined.
What happens if the outcome is left out?
Suppose the missing biomarker is instead imputed as
$$X_{\text{missing}} = f(\text{Age}, \text{Severity})$$
while $Y$ is ignored. The imputation then does not fully use the observed information that links $X$ to $Y$.
This does not literally mean that the procedure assumes $X \perp Y$ (that $X$ and $Y$ are independent), because other variables, such as severity, might indirectly preserve some of that association. However, leaving out an informative outcome can weaken the reconstructed predictor-outcome relationship. For example:
True relationship
Low X → outcome uncommon
High X → outcome commonAfter an imputation model that ignores an informative Y
Low X ↘
outcomes become more mixed
High X ↗The resulting regression association may therefore be attenuated, that is, pulled toward zero. The classic missing-data guidance gives exactly this reasoning: when a predictor is missing, the outcome carries information about its plausible values, and excluding the outcome from imputation can falsely weaken the predictor-outcome association [2, 3].
Does this cause data leakage? The most important distinction
This is the most important distinction in the whole topic. Data leakage means letting information that would not legitimately be available at the point where a prediction is made shape a model or the judgement of its performance.
Situation A: explanatory, causal or outcome regression
Suppose the research goal is:
What is the association between biomarker $X$ and mortality $Y$?
Mortality has already been observed in the study participants. The purpose of multiple imputation is to reconstruct the missing $X$ values before estimating
$$Y = f(X)$$
Using the observed $Y$ in the imputation model is therefore appropriate, because $Y$ is legitimate information about the missing $X$.
This is not the usual meaning of prediction leakage.
Why it feels like leakage
In machine learning, a common rule is:
Never let the outcome enter predictor preprocessing.
That rule applies when information from the answer is allowed to influence predictions for patients whose outcome should be unknown. Imagine a deployed mortality model,
$$P(\text{Death}) = f(\text{Age}, \text{Biomarker})$$
and a new patient who arrives with:
Age = 68
Biomarker = missing
Death = ???At the moment of prediction, $Y = \text{Death}$ is unknown. Nobody can deploy
$$\text{Biomarker}_{\text{missing}} = f(\text{Death})$$
because the answer the model is meant to predict is not available. That would be impossible in real clinical use.
Prediction-model development is more nuanced
The statement
"In a clinical prediction model, never include the outcome in the imputation model"
is too strong. During model development, the outcomes of the development participants are observed. Many legitimate clinical prediction-model studies have included the outcome in the multiple-imputation models for missing predictors, and methodological work in prediction research found that using the outcome to impute missing predictor values was preferred [2]. Multiple imputation is also generally recommended over relying automatically on complete-case analysis when predictors are missing [3].
The real distinction is between three stages.
Model development
Observed outcome information may legitimately help estimate relationships and impute predictors, provided the statistical procedure is appropriately designed.
Model validation
Validation checks how well the model predicts for patients whose outcomes it was not fitted to. Preprocessing must not allow information from the validation outcomes to contaminate the fitted prediction process in a way that would be unavailable at prediction time.
Clinical deployment
A missing predictor must be handled using information that will actually be available for the future patient.
Training, validation and deployment
| Setting | Outcome known? | Can the outcome inform an MI strategy? | Key issue |
|---|---|---|---|
| Explanatory regression dataset | Yes | Yes, usually | Preserve the $X$-$Y$ association |
| Prediction-model development | Yes | Yes, can be appropriate | Correct development and validation procedure |
| Internal validation or test partition | Known to the researcher but conceptually hidden | Caution: leakage must be prevented | Evaluation must mimic unseen data |
| External validation | Known after follow-up | Caution: depends on the validation strategy | Must evaluate the realistic intended use |
| New patient at deployment | No | No: cannot depend on a future $Y$ | $Y$ does not yet exist or is unknown |
Multiple imputation and MCAR
Write $R_X$ for the indicator that marks whether $X$ was recorded ($R_X = 1$) or is missing ($R_X = 0$). Data are missing completely at random (MCAR) when $P(R_X = 0)$ does not systematically depend on observed or unobserved data. Informally, whether $X$ is missing is essentially unrelated to patient characteristics or to the missing value itself.
Under strong MCAR conditions, complete-case analysis can remain unbiased for many analyses, although it loses observations and therefore often loses precision. Multiple imputation can retain more information [3].
Multiple imputation and MAR
Data are missing at random (MAR) when, conditional on the observed information, missingness no longer depends on the unseen value itself. Schematically:
$$P(R_X \mid X_{\text{mis}}, X_{\text{obs}}, Y, Z) = P(R_X \mid X_{\text{obs}}, Y, Z)$$
For example, the biomarker $X$ may be missing more often among patients with severe observed outcomes. If that outcome $Y$ is observed, including it in the imputation model allows the model to use information related to the missingness process.
Including relevant observed variables can therefore make the MAR assumption more plausible, conditional on the information used.
Multiple imputation and MNAR
Data are missing not at random (MNAR) when, even after conditioning on the observed information, missingness still depends on the unobserved value itself. For example, extremely high biomarker values may be particularly likely to go unrecorded precisely because they are extremely high. Schematically,
$$R_X \leftarrow X_{\text{missing}}$$
even after the observed variables are accounted for. Adding $Y$ and other informative variables may explain some of the missingness and improve the imputation model.
Correct missing-data interpretation
| Mechanism | Main idea | Role of the observed outcome |
|---|---|---|
| MCAR | Missingness unrelated to the relevant data | $Y$ may improve efficiency and the imputation, but it is not what makes MCAR valid |
| MAR | Missingness explainable conditional on observed data | Including an informative $Y$ can make the imputation model more appropriate and MAR more plausible |
| MNAR | Missingness still depends on unseen values after conditioning | $Y$ may help, but cannot guarantee removal of MNAR |
The imputation model should be compatible with the scientific model
Suppose the final analysis model is
$$Y = \beta_0 + \beta_1 X + \beta_2\,\text{Age} + \beta_3\,\text{Sex}$$
A reasonable imputation model for the missing $X$ should generally use the variables that carry the relationships relevant to that analysis:
$$X_{\text{mis}} = f(Y, \text{Age}, \text{Sex}, \text{useful auxiliary variables})$$
Auxiliary variables are variables outside the analysis model that help predict the missing values or the chance of being missing. If the scientific model contains important non-linear terms or interactions, the imputation procedure may also need to respect them.
This idea is often described as compatibility or congeniality between the imputation and analysis models [5]. Current methodological guidance recommends that relevant variables from the analysis model, the outcome, treatment indicators where appropriate, interactions and useful auxiliary variables all be considered in imputation modelling [1].
Why multiple imputation instead of fitted-mean imputation?
A tempting shortcut is to make every imputed value equal to its fitted mean. That is not the purpose of multiple imputation. Suppose
$$E(X \mid Y, Z) = 8$$
Multiple imputation does not simply assign $X = 8$ to every such patient. Instead, it reflects uncertainty by generating plausible values across several completed datasets, for example
$$7.1,\ 8.5,\ 9.0,\ 7.8,\ \dots$$
This preserves the uncertainty created by the missing information. The separate analyses are then combined appropriately, so the spread between the completed datasets is carried into the final confidence interval.
Three things including the outcome cannot do
-
Including the outcome guarantees that the data are missing completely at random.
Whether data are MCAR, MAR or MNAR is a property of the missing-data process. Adding variables to an imputation model cannot retroactively guarantee MCAR.
Fix: Ask what drove the missingness, and treat MCAR, MAR or MNAR as an assumption about that process, never as something the imputation model produces.
-
With the outcome in the imputation model, multiple imputations are no longer needed.
Including $Y$ improves the information available to the imputation model, but it does not eliminate the uncertainty about $X_{\text{missing}}$.
Fix: Still create several plausible completed datasets, analyse each one and combine the results.
-
Including the outcome prevents model misspecification.
No statistical method guarantees that. The imputation model can still be misspecified through an incorrect functional form, omitted interactions, poor treatment of non-linear relationships, incorrect distributional assumptions or inappropriate missingness assumptions.
Fix: Make the imputation model compatible with the analysis model, including its non-linear terms and interactions, and state the missingness assumption it relies on.
Applying the logic: what "improve prediction" means
Put the problem in one sentence: a baseline predictor is partly missing and will be used in an outcome regression; why should the observed outcome usually enter the imputation model? Two phrases carry the answer: baseline predictor missing and outcome regression.
The scientific objective is to estimate the relationship
$$X \leftrightarrow Y$$
Since the observed $Y$ contains information about the missing $X$, excluding $Y$ can weaken that relationship. The outcome is therefore included to preserve predictor-outcome associations and improve prediction.
Here "improve prediction" primarily refers to improving the prediction of the missing predictor values within the imputation process, not to cheating by using $Y$ to predict itself.
The data-leakage rule to remember
Do not memorise "outcome in imputation = leakage". Instead, ask one question every time the outcome comes near a missing predictor:
The same question in three settings
For explanatory inference,
$$Y_{\text{observed}} \rightarrow X_{\text{missing}} \rightarrow \text{preserve the } X\text{-}Y \text{ association}$$
is appropriate. For a future patient,
$$Y_{\text{future}} \rightarrow X_{\text{missing}}$$
is impossible, because $Y_{\text{future}}$ is not yet known. For prediction-model validation,
$$Y_{\text{validation}} \rightarrow \text{preprocessing that contaminates prediction evaluation}$$
can create leakage or optimistic performance estimates.
The issue is therefore the availability and separation of information, not simply whether the word "outcome" appears in the imputation model.
Final memory map
Explanatory or outcome regression
Missing $X$ + observed $Y$ $\rightarrow$ include $Y$ in the MI model.
Why? To preserve the $X$-$Y$ association.
Prediction development
The outcome may legitimately enter development-stage MI, provided development and validation are handled correctly.
Prediction for a new patient
The future $Y$ is unavailable $\Rightarrow$ $Y$ cannot be used to impute $X$.
Missingness mechanism
Including $Y$ $\neq$ guaranteeing MCAR.
Including $Y$ $\neq$ automatically converting MNAR into MAR.
Glossary
- analysis model
- The model that answers the research question, such as mortality on a biomarker, age and severity; also called the scientific or substantive model.
- imputation model
- A working model used only to reconstruct plausible values of a variable where it is missing.
- multiple imputation (MI)
- Filling each missing value several times with plausible values drawn from an imputation model, analysing each completed dataset and combining the results.
- auxiliary variable
- A variable outside the analysis model that helps predict the missing values or the chance of being missing.
- compatibility (congeniality)
- Agreement between the imputation model and the analysis model, so that the imputation keeps the relationships, non-linear terms and interactions the analysis relies on.
- complete-case analysis
- An analysis restricted to patients with no missing values in the model's variables.
- attenuation
- A pull of an estimated association toward zero.
- data leakage
- Use of information during model fitting, preprocessing or performance evaluation that would not legitimately be available at the point where the prediction is made.
- internal validation
- Estimating how well a model predicts for new patients from the same source, for example in a held-out test partition whose outcomes are treated as unseen.
- external validation
- Testing a finished model in a separate sample, such as another hospital or a later period.
- deployment
- Using a finished model to make predictions for real patients, whose outcome is not yet known.
- MCAR
- Missing completely at random: missingness unrelated to any observed or unobserved data.
- MAR
- Missing at random: given the observed information, missingness no longer depends on the unseen value itself.
- MNAR
- Missing not at random: missingness still depends on the unseen value after conditioning on the observed information.
References
- White IR, Royston P, Wood AM. Multiple imputation using chained equations: issues and guidance for practice. Stat Med. 2011;30(4):377-399. https://doi.org/10.1002/sim.4067
- Moons KGM, Donders RART, Stijnen T, Harrell FE Jr. Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol. 2006;59(10):1092-1101. https://doi.org/10.1016/j.jclinepi.2006.01.009
- Sterne JAC, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, Wood AM, Carpenter JR. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393. https://doi.org/10.1136/bmj.b2393
- Hoogland J, van Barreveld M, Debray TPA, et al. Handling missing predictor values when validating and applying a prediction model to new patients. Stat Med. 2020;39(25):3591-3607. https://doi.org/10.1002/sim.8682
- Meng XL. Multiple-imputation inferences with uncongenial sources of input. Stat Sci. 1994;9(4):538-558. https://doi.org/10.1214/ss/1177010269
Key takeaways
- When a predictor is missing in an outcome regression, include the observed outcome in the imputation model: it carries information about the missing predictor and preserves the predictor-outcome association.
- The analysis model explains the outcome; the imputation model only reconstructs the missing predictor. The two purposes must not be confused.
- Leaving the outcome out does not literally assume independence, but it mixes outcomes across low and high values of the predictor and can attenuate the association.
- Leakage is about availability: the observed outcome is legitimate in an explanatory analysis and may enter development-stage imputation; validation outcomes must not contaminate evaluation; a new patient's outcome does not exist yet.
- Including the outcome does not guarantee MCAR, does not turn MNAR into MAR, does not remove the need for several imputations and does not prevent misspecification.
Related in the wiki: [[missing-data-mechanisms]] [[multiple-imputation-clinical-research]]