What Is Feature Importance in Random Forest? Gini vs Permutation Explained
Clinical Epidemiology ResearchUniqcret doctor knowledgesMethodology and Research DesignMethodologic [Methodology]
On this page
What is Feature Importance?
Feature importance answers the question:
“Which predictors contribute most to the model’s predictions?”
Importantly:
- Feature importance does not change model performance
- It is used for interpretation, especially in clinical research
Two main methods are used:
- Impurity-based importance (Gini importance)
- Permutation-based importance
Method 1: Impurity-Based Importance (Gini Importance)
Core Idea
Each time a feature is used to split a node, it reduces impurity (e.g., Gini).The total importance of a feature is:
Sum of all impurity reductions across all trees
How It Works
Across the forest:
| Feature | Total impurity reduction | Normalized importance |
| Age | High cumulative reduction | Highest importance |
| GCS | Moderate reduction | Moderate importance |
| SBP | Moderate reduction | Moderate importance |
| HR | Small reduction | Low importance |
| RR | Very small reduction | Lowest importance |
Interpretation
| Property | Behavior |
| Computation timing | During model training |
| Mechanism | Tracks impurity reduction |
| Speed | Fast |
| Output meaning | “How often and how effectively a feature was used for splitting” |
Limitation: Systematic Bias
Bias toward high-cardinality features
| Feature type | Behavior in impurity importance |
| Many categories (e.g., hospital ID) | Artificially high importance |
| Continuous variables | Favored |
| Binary variables | Underestimated |
Why this happens
- Features with more possible split points→ more opportunities to reduce impurity→ higher accumulated importance
Even if the feature is not clinically meaningful
Example Interpretation
| Scenario | Result |
| Hospital ID (many categories) | Appears highly important |
| Age (continuous) | Appears important |
| True clinical predictors | May be underestimated |
This leads to misleading conclusions in interpretation.
Method 2: Permutation-Based Importance
Core Idea
Instead of tracking splits, this method asks:
“If I destroy this feature’s information, how much worse does the model perform?”
How It Works
Step-by-step logic
- Compute baseline model performance(e.g., AUROC = 0.850)
- Randomly shuffle one feature (break its relationship with outcome)
- Recompute performance
- Importance = performance drop
Example Results
| Feature | Baseline AUROC | Shuffled AUROC | Performance drop | Importance |
| Age | 0.850 | 0.720 | 0.130 | High |
| GCS | 0.850 | 0.740 | 0.110 | High |
| SBP | 0.850 | 0.810 | 0.040 | Moderate |
| HR | 0.850 | 0.830 | 0.020 | Low |
| RR | 0.850 | 0.845 | 0.005 | Very low |
Interpretation
| Property | Behavior |
| Computation timing | After model training |
| Mechanism | Measures performance degradation |
| Speed | Slower |
| Output meaning | “How much this feature contributes to prediction accuracy” |
Why Permutation Importance is Unbiased
Example 1: Many categories vs true signal
| Feature | True predictive value | Impurity importance | Permutation importance |
| Hospital ID | None | High (biased) | Near zero (correct) |
| Age | Strong | Moderate | High (correct) |
Example 2: Continuous vs binary
| Feature | Type | True effect | Impurity result | Permutation result |
| Age | Continuous | Moderate | High (biased) | Moderate |
| Sex | Binary | Moderate | Low (biased) | Moderate |
Key Insight
Permutation importance reflects:
Actual contribution to predictive performance
not:
Opportunity to create splits
Side-by-Side Comparison
| Aspect | Impurity-Based Importance | Permutation-Based Importance |
| When computed | During training | After training |
| Mechanism | Sum of impurity reductions | Performance drop after shuffling |
| Speed | Fast | Slower |
| Bias | Biased (toward many categories/continuous) | Unbiased |
| Interpretation | Model usage frequency | True predictive contribution |
| Use case | Quick exploration | Clinical reporting and publication |
Clinical Interpretation Perspective
In clinical prediction modeling:
- The goal is not:
- “Which variable is used often?”
- The goal is:
- “Which variable actually improves prediction?”
Permutation importance aligns with this goal.
This is consistent with prediction modeling principles:
- Focus on predictive contribution, not structural artifacts
- Avoid misleading interpretations due to model mechanics
Practical Recommendation for Clinical Papers
- Do not rely on impurity-based importance for interpretation
- Use permutation-based importance for reporting
Reason:
- It reflects real impact on model performance
- It is methodologically defensible
- It aligns with clinical reasoning
Conceptual Summary
Feature importance methods differ in what they measure:
- Impurity-based importance → how often a feature is used
- Permutation-based importance → how much a feature matters
Key Takeaways
- Feature importance does not affect model performance, only interpretation
- Impurity-based importance is fast but biased toward certain variable types
- Permutation-based importance measures true contribution to prediction
- In clinical research, permutation importance is preferred for validity
- Misinterpreting feature importance can lead to incorrect clinical conclusions
0