Conformal Prediction

Preface

A fitted model that predicts a patient’s recovery time as 14 days has not told you how much to trust that number. A classifier that assigns an image to “cat” with probability 0.51 has not told you that “dog” was a close second. Ordinary predictive models are built to produce their single best guess, and most of them are silent about how far that guess might be from the truth.

Conformal prediction takes a fitted model exactly as it is, without retraining it or requiring it to report its own uncertainty, and wraps it with a procedure that turns its predictions into intervals or sets with a guaranteed error rate. A regression model becomes a model that also reports a range of plausible values. A classifier becomes a model that also reports a set of plausible labels, which can contain one class, several, or in principle all of them, depending on how confidently the data support a single answer.

The procedure asks for very little from the underlying model. It does not need to be linear, it does not need to be Bayesian, and it does not need to expose anything about its own internal uncertainty. It only needs to produce predictions that can be scored against held-out data. This makes conformal prediction applicable to linear models, random forests, gradient boosting models, and neural networks alike.

To follow the chapter you need to be comfortable with a fitted supervised model, features and predictions, and basic probability. Chapter 3’s derivation uses undergraduate algebra, sums, and expectation notation, and builds up every result from a small worked numerical example rather than assuming a prior statistics course.

Start with understanding what problem conformal prediction solves.