Package initialization

Calibration analysis of probabilistic models in Julia

David Widmann (@devmotion )

Uppsala University, Sweden

JuliaCon, July 2021

Probabilistic predictive models

A probabilistic predictive model predicts a probability distribution over a set of targets for a given feature.

One can express the uncertainty in the prediction, which might be inherent to the prediction task or caused by insufficient knowledge of the underlying relation between feature and target.

Example: Prediction of penguin species

Gorman KB, Williams TD, Fraser WR (2014). Ecological sexual dimorphism and environmental variability within a community of Antarctic penguins (genus Pygoscelis). PLoS ONE 9(3):e90081

We split the Palmer penguin dataset randomly into a training (70%) and validation (30%) dataset.

We learn a Gaussian naive Bayes classifier that is able to predict the probability of the penguin species from the bill length and the flipper length.

We denote the features by $X$ and the target by $Y$.

The predictions are distributions of the penguin species.

We let $P_X$ denote the distribution predicted by a model $P$ for features $X$.

There exist many different probabilistic predictive models for this task.

Ideally, we would like that

$$P_X=\mathrm{law}(Y\hspace{0.17em}\mid \hspace{0.17em}X)$$

almost surely.

Of course, usually it is not possible to achieve this in practice.

Calibration

A probabilistic predictive model $P_X$ of the conditional distributions $\mathrm{law}(Y|X)$ is calibrated if

$$P_X=\mathrm{law}(Y|P_X)$$

almost surely.

It is not relevant how the model was obtained. In particular it does not matter if one uses a maximum likelihood approach or performs Bayesian inference.

Often weaker conditions are investigated, corresponding to less informative models.

E.g., "confidence calibration" only considers the most-confident class: This corresponds to a binary classification model

Other target spaces can be considered as well.

Calibrated models

Bröcker, J. (2009). Reliability, sufficiency, and the decomposition of proper scores. Q.J.R. Meteorol. Soc., 135: 1512-1519.

Any model of the form

$$P_X:=\mathrm{law}(Y\hspace{0.17em}|\hspace{0.17em}\varphi (X))\text{almost surely},$$

where $\varphi$ is some measurable function, is calibrated.

Function $\varphi$ corresponds to the amount/loss of information about $X$:

• Identity function yields the ideal model $P_X:=\mathrm{law}(Y\hspace{0.17em}|\hspace{0.17em}X)$

• Every constant function yields the baseline model $P_X:=\mathrm{law}(Y)$ ("climatology")

Reliability diagrams

Murphy, A., & Winkler, R. (1977). Reliability of Subjective Probability Forecasts of Precipitation and Temperature. Journal of the Royal Statistical Society. Series C (Applied Statistics), 26(1), 41-47.

Bröcker, J., & Smith, L. A. (2007). Increasing the reliability of reliability diagrams. Weather and forecasting, 22(3), 651-661.

Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J. & Schön, T. B. (2019). Evaluating model calibration in classification. Proceedings of Machine Learning Research, in Proceedings of Machine Learning Research 89:3459-3467 (AISTATS 2019).

Reliability diagrams are used to visually inspect calibration of binary classification models. They show binned averages of empirical frequencies versus confidence.

Visualizations with ReliabilityDiagrams.jl

• Supports Plots.jl and Makie.jl

• Allows to plot deviation, i.e., empirical frequency minus confidence.

• Includes consistency bars

Bins: equal massequal size

10

Deviation:

Consistency bars:

Expected calibration error

The expected calibration error (ECE)

$${\mathrm{ECE}}_d:={\mathbb{E}}_{P_X}d(P_X,\mathrm{law}(Y|P_X))$$

measures the expected deviation of the predictions $P_X$ and empirical frequencies $\mathrm{law}(Y|P_X)$ with respect to distance measure $d$.

• For (multi-class) classification models, common choices for $d$ are (semi-)metrics on the probability simplex such as Euclidean or squared Euclidean distance

• For general probabilistic models statistical divergences can be chosen for $d$, e.g.,

  • $f$-divergences such as the Kullback-Leibler divergence,

  • Wasserstein distance,

  • maximum mean discrepancy (MMD).

Estimation with CalibrationErrors.jl

Supports

• different distance measures $d$ (default: total variation distance),

• different binning algorithms (bins of uniform size and bins that minimize variance within bins)

Distance: EuclideanCityblockTotal variationSquared Euclidean Number of bins:

10

Why scoring rules are not sufficient

Bröcker, J. (2009). Reliability, sufficiency, and the decomposition of proper scores. Q.J.R. Meteorol. Soc., 135: 1512-1519.

Probabilistic predictive models can be evaluated by the expected score

$${\mathbb{E}}_{P_X,Y}s(P_X,Y)$$

where scoring rule $s(P,\omega )$ is the reward of prediction $P$ if the true outcome is $\omega$.

Examples: Brier score, logarithmic score

Proper scoring rules can be decomposed as

$$\begin{array}{rl}{\mathbb{E}}_{P_X,Y}s(P_X,Y)=& \underset{\text{resolution}}{\underbrace{{\mathbb{E}}_{P_X}d(\mathrm{law}(Y),\mathrm{law}(Y|P_X))}}\\ & -\underset{\text{ECE}}{\underbrace{{\mathbb{E}}_{P_X}d(P_X,\mathrm{law}(Y|P_X))}}-\underset{\text{uncertainty of }Y}{\underbrace{S(\mathrm{law}(Y),\mathrm{law}(Y))}}\end{array}$$

where $S(P,Q):={\int }_{\mathrm{\Omega }}s(P,\omega )Q(\mathrm{d}\omega )$ is the expected score of $P$ under $Q$ and $d(P,Q):=S(Q,Q)-S(P,P)$ is the score divergence.

• resolution: minimized for uninformative models with $\mathrm{law}(Y|P_X)=\mathrm{law}(Y)$ such as constant models

• ECE: minimized for calibrated models

• uncertainty of $Y$: does not dependend on the model

Alternatives to ECE

Widmann, D., Lindsten, F., & Zachariah, D. (2021). Calibration tests beyond classification. ICLR 2021.

A probabilistic predictive model is calibrated if

$$(P_X,Y)\stackrel{d}{=}(P_X,Z_X),$$

where $Z_X|P_X\sim P_X$.

• No explicit conditional distributions $\mathrm{law}(Y|P_X)$

• Suggests discrepancy of $\mathrm{law}(P_X,Y)$ and $\mathrm{law}(P_X,Z_X)$ as calibration measure

Kernel calibration error (KCE)

Widmann, D., Lindsten, F., & Zachariah, D. (2019). Calibration tests in multi-class classification: A unifying framework. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (pp. 12257–12267).

Widmann, D., Lindsten, F., & Zachariah, D. (2021). Calibration tests beyond classification. ICLR 2021.

Integral probability metrics can be used to define a general class of probability metrics with minimal assumptions about the involved distributions. The kernel calibration error (KCE) is an example of this class:

The kernel calibration error (KCE) with respect to a real-valued kernel $k$ is defined as

$${\mathrm{KCE}}_k:={\mathrm{MMD}}_k(\mathrm{law}(P_X,Y),\mathrm{law}(P_X,Z_X)),$$

where ${\mathrm{MMD}}_k$ is the maximum mean discrepancy with respect to $k$.

• Applies to all probabilistic predictive models

• Existing (un)biased and consistent estimators of the MMD without challenging estimation of $\mathrm{law}(Y|P_X)$

• Variance of estimators can be reduced by marginalizing out $Z_X$

Estimation with CalibrationErrors.jl

Supports

• biased and unbiased estimators

• estimators with quadratic and subquadratic sample complexity

• kernels from KernelFunctions.jl

Here we choose a tensor product kernel of the form

$$k((p,y),(\tilde{p},\tilde{y})):=k_1(p,\tilde{p})\delta (y-\tilde{y}).$$

Kernel $k_1$: ExponentialSquared exponentialMatérn 3/2Matérn 5/2 Length scale:

1

Estimator: Unbiased (quadratic)Unbiased (subquadratic)Unbiased (linear)Biased

Estimation with CalibrationErrorsDistributions.jl

Closed-form expressions for predictions of Gaussian and Laplace distributions in Distributions.jl

Example: Gaussian process

1. We sample from a Gaussian process (GP) with zero mean and a squared exponential kernel at 40 random locations in the interval $[0,10]$.

2. We split the data randomly in a training (75%) and validation (25%) dataset, and compute the GP posterior from the training data.

3. We compute the predicted normal distributions on the validation data.

4. We estimate the squared KCE with a tensor product kernel of the form

   $$k((\mu ,y),(\tilde{\mu },\tilde{y})):=\exp (-W_2(\mu ,\tilde{\mu }))\exp (-\frac{(y-\tilde{y}{)}^2}{2{\ell }^2}),$$

   where $W_2(\mu ,\tilde{\mu })$ is the 2-Wasserstein distance of the Gaussian distributions $\mu$ and $\tilde{\mu }$.

Length scale $\ell$:

1

Estimator: Unbiased (quadratic)Unbiased (subquadratic)Unbiased (linear)Biased

Calibration tests

Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J. & Schön, T. B. (2019). Evaluating model calibration in classification. Proceedings of Machine Learning Research, in Proceedings of Machine Learning Research 89:3459-3467 (AISTATS 2019).

Widmann, D., Lindsten, F., & Zachariah, D. (2019). Calibration tests in multi-class classification: A unifying framework. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (pp. 12257–12267).

Widmann, D., Lindsten, F., & Zachariah, D. (2021). Calibration tests beyond classification. ICLR 2021.

• Calibration errors have no meaningful unit or scale.

• Different calibration errors rank models differently.

• Estimators of calibration errors are random variables.

Perform a statistical test of the null hypothesis $H_0:=\text{"model is calibrated"}$.

• Hypothesis testing of calibration is a special two-sample problem

• Applies to all probabilistic predictive models

• Existing two-sample tests based on the MMD can be improved by marginalizing out $Z_X$

Calibration tests with CalibrationTests.jl

Supports

• tests based on consistency resampling

• tests based on distribution-free bounds and asymptotic properties of KCE

• tests with quadratic and subquadratic sample complexity

• interface of HypothesisTests.jl

Test: AsymptoticAsymptotic (subquadratic)Asymptotic (linear)Distribution-free (biased)Distribution-free (unbiased)

pycalibration

• Python interface for CalibrationErrors, CalibrationErrorsDistributions, and CalibrationTests

• Inspired by diffeqpy

• Uses PyJulia interface

Usage

• Load package and install Julia dependencies:

  >>> import pycalibration
  >>> pycalibration.install()

• Define estimator of the SKCE with kernel

  $$k((\mu ,y),(\hat{\mu },\hat{y}))=\exp (-\parallel \mu -\hat{\mu }\parallel )\delta (y-\hat{y}):$$

  >>> from pycalibration import calerrors as ce
  >>> skce = ce.UnbiasedSKCE(ce.tensor(ce.ExponentialKernel(), ce.WhiteKernel()))

• Estimate the SKCE for some random predictions and outcomes:

  >>> import numpy as np
  >>> from pycalibration import calerrors as ce
  >>> rng = np.random.default_rng(1234)
  >>> predictions = rng.random(100)
  >>> outcomes = rng.choice([True, False], 100)
  >>> skce(predictions, outcomes)
  0.03320398246523166

• Perform a calibration test with some random predictions and outcomes:

  >>> from pycalibration import caltests as ct
  >>> import numpy as np
  >>> rng = np.random.default_rng(1234)
  >>> predictions = rng.dirichlet((3, 2, 5), 100)
  >>> outcomes = rng.integers(low=1, high=4, size=100)
  >>> kernel = ct.tensor(ct.ExponentialKernel(metric=ct.TotalVariation()), ct.WhiteKernel())
  >>> test = ct.AsymptoticSKCETest(kernel, predictions, outcomes)
  >>> print(test)
  <PyCall.jlwrap Asymptotic SKCE test
  --------------------
  Population details:
      parameter of interest:   SKCE
      value under h_0:         0.0
      point estimate:          6.07887e-5
  
  Test summary:
      outcome with 95% confidence: fail to reject h_0
      one-sided p-value:           0.4330
  
  Details:
      test statistic: -4.955380469272125
  >>> ct.pvalue(test)
  0.435

More examples can be found in the documentation.

rcalibration

• R interface for CalibrationErrors, CalibrationErrorsDistributions, and CalibrationTests

• Inspired by diffeqr

• Based on JuliaCall

Usage

• Load package and install Julia dependencies:

  > library(rcalibration)
  > rcalibration::install()

• Define estimator of the SKCE with kernel

  $$k((\mu ,y),(\hat{\mu },\hat{y}))=\exp (-\parallel \mu -\hat{\mu }\parallel )\delta (y-\hat{y}):$$

  > ce <- calerrors()
  > skce <- ce$UnbiasedSKCE(ce$tensor(ce$ExponentialKernel(), ce$WhiteKernel()))

• Estimate the SKCE for some random predictions and outcomes:

  > ce <- calerrors()
  > set.seed(1234)
  > predictions <- runif(100)
  > outcomes <- sample(c(TRUE, FALSE), 100, replace=TRUE)
  > skce$.(predictions, outcomes)
  [1] 0.01518318

• Perform a calibration test with some random predictions and outcomes:

  > library(extraDistr)
  > ct <- caltests()
  > set.seed(1234)
  > predictions <- rdirichlet(100, c(3, 2, 5))
  > outcomes <- sample(1:3, 100, replace=TRUE)
  > kernel <- ct$tensor(ct$ExponentialKernel(metric=ct$TotalVariation()), ct$WhiteKernel())
  > test <- ct$AsymptoticSKCETest(kernel, ce$RowVecs(predictions), outcomes)
  > print(test)
  Julia Object of type AsymptoticSKCETest{KernelTensorProduct{Tuple{ExponentialKernel{TotalVariation}, WhiteKernel}}, Float64, Float64, Matrix{Float64}}.
  Asymptotic SKCE test
  --------------------
  Population details:
      parameter of interest:   SKCE
      value under h_0:         0.0
      point estimate:          0.0259434
  
  Test summary:
      outcome with 95% confidence: reject h_0
      one-sided p-value:           0.0100
  
  Details:
      test statistic: -0.007291403994633658
  > ct$pvalue(test)
  [1] 0.004

More examples can be found in the documentation.

Take home messages

• Calibration is an important aspect of probabilistic predictive models

• Reliability diagrams with consistency bars help to visually inspect calibration

• There exist alternatives to the ECE such as the KCE with favourable theoretical properties

• Calibration tests can be used to deal with the randomness of calibration error estimates

• Python and R interfaces if you do not use Julia