Uncertainty Quantification (UQ) in Machine Learning Models

A. Bayesian Neural Networks (BNNs)

In BNNs, model weights are not fixed parameters but probability distributions:

Predictions integrate over all possible weights:

BNNs yield uncertainty naturally but are computationally expensive. Approximate inference (e.g., Variational Inference, MCMC) is used in practice.

B. Monte Carlo Dropout (MC Dropout)

A practical approximation of BNNs proposed by Gal & Ghahramani (2016).

• Idea: Use dropout at inference time, not just training
• Each forward pass samples a different network → creates a predictive distribution

Predictive mean and variance are computed across T stochastic passes.

C. Ensemble and Bootstrap Methods

Train multiple models on bootstrapped samples. Uncertainty is approximated by the variance in their predictions.

These are easy to deploy in enterprise MLOps pipelines.

🧠 A. Bayesian Linear Regression using PyMC3

import pymc3 as pm
import numpy as np
import matplotlib.pyplot as plt

# Simulate data
np.random.seed(42)
X = np.linspace(0, 10, 50)
y = 2.5 * X + np.random.normal(0, 1.5, len(X))

with pm.Model() as model:
    alpha = pm.Normal('alpha', mu=0, sigma=10)
    beta = pm.Normal('beta', mu=0, sigma=10)
    sigma = pm.HalfCauchy('sigma', beta=5)
    mu = alpha + beta * X

    Y_obs = pm.Normal('Y_obs', mu=mu, sigma=sigma, observed=y)
    trace = pm.sample(1000, tune=1000, cores=2, target_accept=0.95)

pm.plot_posterior(trace, var_names=["alpha", "beta", "sigma"])
plt.show()

This produces posterior distributions for parameters — giving not just the best-fit line but a range of plausible models, each weighted by probability.

🧩 B. Monte Carlo Dropout in Neural Networks (Keras/TensorFlow)

import tensorflow as tf
import numpy as np

# Sample regression data
X = np.linspace(-3, 3, 200).reshape(-1, 1)
y = np.sin(X) + np.random.normal(0, 0.1, X.shape)

# Define model with dropout
def create_model():
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(64, activation='relu', input_shape=(1,)),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(64, activation='relu'),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(1)
    ])
    model.compile(optimizer='adam', loss='mse')
    return model

model = create_model()
model.fit(X, y, epochs=200, verbose=0)

# Monte Carlo sampling at inference
T = 100
preds = np.array([model(X, training=True).numpy().flatten() for _ in range(T)])
mean_preds = preds.mean(axis=0)
std_preds = preds.std(axis=0)

import matplotlib.pyplot as plt
plt.figure(figsize=(8,5))
plt.plot(X, y, 'k.', alpha=0.3, label='Data')
plt.plot(X, mean_preds, 'b-', label='Mean Prediction')
plt.fill_between(X.flatten(),
                 mean_preds - 2*std_preds,
                 mean_preds + 2*std_preds,
                 color='lightblue', alpha=0.4, label='Uncertainty Band')
plt.legend(); plt.title("Monte Carlo Dropout: Predictive Uncertainty")
plt.show()

• Each forward pass gives a slightly different prediction — the spread of predictions = uncertainty
• The shaded region represents 95% confidence intervals

🧮 C. Quantile Regression for Predictive Intervals (LightGBM)

import lightgbm as lgb
import pandas as pd
from sklearn.model_selection import train_test_split

# Generate synthetic insurance claims data
np.random.seed(42)
X = pd.DataFrame({
    'age': np.random.randint(20, 80, 1000),
    'policy_years': np.random.randint(1, 10, 1000)
})
y = 2000 + 100*X['age'] - 150*X['policy_years'] + np.random.normal(0, 500, 1000)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Train two quantile models
params = {'objective': 'quantile', 'alpha': 0.1, 'min_data_in_leaf': 10}
lower = lgb.train(params, lgb.Dataset(X_train, label=y_train))
params['alpha'] = 0.9
upper = lgb.train(params, lgb.Dataset(X_train, label=y_train))

pred_lower = lower.predict(X_test)
pred_upper = upper.predict(X_test)

The intervals [pred_lower, pred_upper] quantify uncertainty for each prediction — ideal for risk forecasts (e.g., "Expected claim = $5000 ± $1200").