SHAP Values

Preface

A fitted model can predict well and still be silent about why. Two patients get different risk scores, a customer is flagged and another is not, a cell is classified one way rather than another, and the model that produced these outputs is usually happy to keep producing more of them without ever saying which of its inputs actually drove a particular answer.

SHAP takes a fitted model exactly as it is, without retraining it or simplifying it, and for one observation at a time, splits the gap between a baseline prediction and that observation’s actual prediction among the model’s features. The same machinery, applied to many observations, also produces a global picture of which features a model relies on and how.

To follow the chapter you need to be comfortable with a fitted predictive model, features and predictions, and basic probability. Chapter 3’s derivation uses undergraduate algebra and expectation notation. Readers who want a refresher on gradient boosting and XGBoost’s own gain-based importance before starting should read the companion XGBoost chapter on this site first, since Chapter 2 assumes XGBoost’s basics and picks up specifically at explaining individual predictions.

Start with SHAP.