Regression models — pricing, severity and frequency¶

Three insurance models, three shapes of continuous target:

Model	Target	Metric that fits
Premium pricing	naira per policy	rmse / mae
Claims severity	naira per claim, heavily skewed	mape
Claims frequency	claim count, mostly zero	poisson_deviance

Regression changes more than the metric. Demographic parity has no analogue — there is no "selected" class to count — so four different fairness checks replace it. And the most important of them asks a question that only makes sense in insurance: is one group charged a higher margin over its own expected loss?

New to the library? 01_binary_classification_sklearn.ipynb covers contexts, reports and verdicts. This notebook assumes those and focuses on what regression changes.

In [1]:

# %pip install -q "bdp-model-gate[structured]"
import logging

import numpy as np
import pandas as pd

import bdp_model_gate

logging.basicConfig(level=logging.INFO, format="%(levelname)-8s %(name)s: %(message)s")
logging.getLogger("bdp_model_gate").setLevel(logging.WARNING)  # quieter; we log per-section
pd.set_option("display.width", 130)

print("bdp-model-gate", bdp_model_gate.__version__)

# %pip install -q "bdp-model-gate[structured]" import logging import numpy as np import pandas as pd import bdp_model_gate logging.basicConfig(level=logging.INFO, format="%(levelname)-8s %(name)s: %(message)s") logging.getLogger("bdp_model_gate").setLevel(logging.WARNING) # quieter; we log per-section pd.set_option("display.width", 130) print("bdp-model-gate", bdp_model_gate.__version__)

bdp-model-gate 0.4.2

1. A motor book with a margin problem¶

The premium follows risk, which is exactly as it should be — a higher-risk segment ought to pay more. What we plant instead is subtler and genuinely unfair: one region is charged a higher margin over its own expected loss. Their premium is not just higher, it is higher relative to what they cost.

For the model to actually reproduce that — rather than just the historical prices containing it — the book also carries distance_to_branch_km, a geographic rating factor that is nearly a relabelling of region. That is how a loading applied by region survives into a model that never sees region, and it is the realistic version of this problem.

The loss-ratio check exists to find exactly this, and no raw-price comparison can see it.

In [2]:

rng = np.random.default_rng(31)
N = 1600

region = rng.choice(["Lagos", "Kano", "Abuja"], N, p=[0.45, 0.35, 0.20])
gender = rng.choice(["F", "M"], N, p=[0.47, 0.53])

X = pd.DataFrame(
    {
        "driver_age": rng.integers(21, 68, N).astype(float),
        "vehicle_age": rng.integers(0, 18, N).astype(float),
        "sum_insured_ngn": rng.lognormal(15.4, 0.40, N).round(0),
        "claims_last_3y": rng.poisson(0.7, N).astype(float),
        "annual_mileage_km": rng.normal(18_000, 6_000, N).clip(2_000).round(0),
        # A geographic rating factor that is very nearly a relabelling of
        # region — which is how the regional loading below becomes learnable
        # even though region itself is never a feature.
        "distance_to_branch_km": (
            pd.Series(region).map({"Lagos": 2.0, "Abuja": 5.0, "Kano": 16.0}).to_numpy()
            + rng.normal(0, 1.0, N)
        ).round(2),
    }
)

# True expected loss — genuinely risk-driven, no regional term.
expected_loss = (
    0.010 * X["sum_insured_ngn"]
    + 5_500 * X["claims_last_3y"]
    + 0.28 * X["annual_mileage_km"]
    - 220 * (X["driver_age"] - 21)
).clip(20_000).round(0)

# The planted unfairness: Kano is loaded 42% over expected loss, everyone
# else 12%. Same risk, different margin.
loading = pd.Series(region).map({"Lagos": 1.12, "Abuja": 1.12, "Kano": 1.42}).to_numpy()
quoted_premium = (expected_loss * loading).round(0)

# Realised loss for the period — expected loss plus noise.
realised_loss = (expected_loss * rng.gamma(shape=8.0, scale=1 / 8.0, size=N)).round(0)

protected_df = pd.DataFrame({"region": region, "gender": gender})

print("mean figures by region:")
display(
    pd.DataFrame(
        {
            "expected_loss": pd.Series(expected_loss).groupby(region).mean(),
            "quoted_premium": pd.Series(quoted_premium).groupby(region).mean(),
            "loss_ratio": pd.Series(quoted_premium / expected_loss).groupby(region).mean(),
        }
    ).round(2)
)

rng = np.random.default_rng(31) N = 1600 region = rng.choice(["Lagos", "Kano", "Abuja"], N, p=[0.45, 0.35, 0.20]) gender = rng.choice(["F", "M"], N, p=[0.47, 0.53]) X = pd.DataFrame( { "driver_age": rng.integers(21, 68, N).astype(float), "vehicle_age": rng.integers(0, 18, N).astype(float), "sum_insured_ngn": rng.lognormal(15.4, 0.40, N).round(0), "claims_last_3y": rng.poisson(0.7, N).astype(float), "annual_mileage_km": rng.normal(18_000, 6_000, N).clip(2_000).round(0), # A geographic rating factor that is very nearly a relabelling of # region — which is how the regional loading below becomes learnable # even though region itself is never a feature. "distance_to_branch_km": ( pd.Series(region).map({"Lagos": 2.0, "Abuja": 5.0, "Kano": 16.0}).to_numpy() + rng.normal(0, 1.0, N) ).round(2), } ) # True expected loss — genuinely risk-driven, no regional term. expected_loss = ( 0.010 * X["sum_insured_ngn"] + 5_500 * X["claims_last_3y"] + 0.28 * X["annual_mileage_km"] - 220 * (X["driver_age"] - 21) ).clip(20_000).round(0) # The planted unfairness: Kano is loaded 42% over expected loss, everyone # else 12%. Same risk, different margin. loading = pd.Series(region).map({"Lagos": 1.12, "Abuja": 1.12, "Kano": 1.42}).to_numpy() quoted_premium = (expected_loss * loading).round(0) # Realised loss for the period — expected loss plus noise. realised_loss = (expected_loss * rng.gamma(shape=8.0, scale=1 / 8.0, size=N)).round(0) protected_df = pd.DataFrame({"region": region, "gender": gender}) print("mean figures by region:") display( pd.DataFrame( { "expected_loss": pd.Series(expected_loss).groupby(region).mean(), "quoted_premium": pd.Series(quoted_premium).groupby(region).mean(), "loss_ratio": pd.Series(quoted_premium / expected_loss).groupby(region).mean(), } ).round(2) )

mean figures by region:

	expected_loss	quoted_premium	loss_ratio
Abuja	55474.40	62131.34	1.12
Kano	56088.76	79646.03	1.42
Lagos	56401.67	63169.88	1.12

Look at the two right-hand columns. Kano's premium is higher — but so is its expected loss, so a price comparison alone is ambiguous. The loss ratio is the number that gives it away: 1.42 against 1.12.

Whether the model reproduces that skew is the question the gate answers, and it is not the same question — a model that fails to learn the loading would price fairly despite unfair history.

In [3]:

from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split

(
    X_train, X_val,
    y_train, y_val,
    el_train, el_val,
    prot_train, prot_val,
) = train_test_split(
    X, quoted_premium, expected_loss, protected_df, test_size=0.35, random_state=31
)
realised_val = realised_loss[X_val.index]

# The pricing model learns the quoted premium, loading and all — region is
# not a feature, but the loading is learnable from correlated columns.
pricer = GradientBoostingRegressor(random_state=31, n_estimators=250, max_depth=3)
pricer.fit(X_train, y_train)
y_pred = pricer.predict(X_val)

print(f"train {len(X_train)} | validation {len(X_val)}")
print(f"predicted premium: mean {y_pred.mean():,.0f}  range {y_pred.min():,.0f}-{y_pred.max():,.0f}")

from sklearn.ensemble import GradientBoostingRegressor from sklearn.model_selection import train_test_split ( X_train, X_val, y_train, y_val, el_train, el_val, prot_train, prot_val, ) = train_test_split( X, quoted_premium, expected_loss, protected_df, test_size=0.35, random_state=31 ) realised_val = realised_loss[X_val.index] # The pricing model learns the quoted premium, loading and all — region is # not a feature, but the loading is learnable from correlated columns. pricer = GradientBoostingRegressor(random_state=31, n_estimators=250, max_depth=3) pricer.fit(X_train, y_train) y_pred = pricer.predict(X_val) print(f"train {len(X_train)} | validation {len(X_val)}") print(f"predicted premium: mean {y_pred.mean():,.0f} range {y_pred.min():,.0f}-{y_pred.max():,.0f}")

train 1040 | validation 560
predicted premium: mean 67,680  range 20,838-205,250

2. The context¶

Two regression-specific fields: task="regression" and expected_loss. The latter is what unlocks the margin check — supply a per-row expected loss, technical premium or pure premium.

In [4]:

from bdp_model_gate import GateConfig, ModelGate, StructuredGateContext
from bdp_model_gate.structured import default_structured_checks

model_card = {
    "model_name": "motor-pricing-gbm",
    "version": "3.0.2",
    "use_case": "pricing",                        # high-risk -> DPIA required
    "legal_basis": "Contractual necessity (NDPA 2023, s.25(1)(b))",
    "data_minimization_justification": "Rating factors only; no device or browsing data.",
    "training_data_source": "Motor book 2021-2025",
    "dpia_completed": True,
    "influences_decision_about_person": True,
    "explainability_method": "SHAP contributions retained per quote",
}

context = StructuredGateContext(
    model=pricer,
    X=X_val,
    y_true=realised_val,          # what actually happened
    y_pred=y_pred,                # what we charged
    protected_df=prot_val,
    expected_loss=el_val,         # enables LossRatioParityCheck
    model_card=model_card,
    latencies_ms=rng.gamma(9.0, 6.0, 400),
    cost_per_inference=0.0005,
    task="regression",
)

from bdp_model_gate import GateConfig, ModelGate, StructuredGateContext from bdp_model_gate.structured import default_structured_checks model_card = { "model_name": "motor-pricing-gbm", "version": "3.0.2", "use_case": "pricing", # high-risk -> DPIA required "legal_basis": "Contractual necessity (NDPA 2023, s.25(1)(b))", "data_minimization_justification": "Rating factors only; no device or browsing data.", "training_data_source": "Motor book 2021-2025", "dpia_completed": True, "influences_decision_about_person": True, "explainability_method": "SHAP contributions retained per quote", } context = StructuredGateContext( model=pricer, X=X_val, y_true=realised_val, # what actually happened y_pred=y_pred, # what we charged protected_df=prot_val, expected_loss=el_val, # enables LossRatioParityCheck model_card=model_card, latencies_ms=rng.gamma(9.0, 6.0, 400), cost_per_inference=0.0005, task="regression", )

3. Metrics, and which way they point¶

The big change: error metrics are lower-is-better, so they are gated with max_error rather than min_score. Each metric declares its own direction and the report says which comparison ran.

There is deliberately no default max_error — a sensible ceiling depends entirely on whether the target is naira or claim counts — so configuring an error metric without one is an error rather than a silent pass.

In [5]:

from bdp_model_gate import PerformanceConfig
from bdp_model_gate.metrics import BUILTIN_METRICS
from bdp_model_gate.structured.performance import PerformanceThresholdCheck

rows = []
for name, spec in sorted(BUILTIN_METRICS.items()):
    if "regression" not in spec.tasks:
        continue
    cfg = (
        PerformanceConfig(metric=name, min_score=-1e12)
        if spec.greater_is_better
        else PerformanceConfig(metric=name, max_error=1e12)
    )
    result = PerformanceThresholdCheck(cfg).run(context)[0]
    rows.append(
        {
            "metric": name,
            "value": result.metadata["value"],
            "direction": "higher is better" if spec.greater_is_better else "lower is better",
            "gated with": result.metadata["threshold_field"],
        }
    )
display(pd.DataFrame(rows).set_index("metric"))

from bdp_model_gate import PerformanceConfig from bdp_model_gate.metrics import BUILTIN_METRICS from bdp_model_gate.structured.performance import PerformanceThresholdCheck rows = [] for name, spec in sorted(BUILTIN_METRICS.items()): if "regression" not in spec.tasks: continue cfg = ( PerformanceConfig(metric=name, min_score=-1e12) if spec.greater_is_better else PerformanceConfig(metric=name, max_error=1e12) ) result = PerformanceThresholdCheck(cfg).run(context)[0] rows.append( { "metric": name, "value": result.metadata["value"], "direction": "higher is better" if spec.greater_is_better else "lower is better", "gated with": result.metadata["threshold_field"], } ) display(pd.DataFrame(rows).set_index("metric"))

	value	direction	gated with
metric
mae	20035.8003	lower is better	max_error
mape	0.4748	lower is better	max_error
poisson_deviance	9669.2056	lower is better	max_error
r2	0.3056	higher is better	min_score
rmse	25582.2245	lower is better	max_error

All five are implemented in numpy, so they work on a core install without scikit-learn. metric="auto" resolves to r2 for regression — an rmse default threshold would be meaningless without knowing the scale of the target.

In [6]:

from bdp_model_gate.exceptions import GateConfigurationError

# The guard: an error metric with nothing to compare against.
try:
    PerformanceThresholdCheck(PerformanceConfig(metric="rmse")).run(context)
except GateConfigurationError as exc:
    print("GateConfigurationError:", exc)

from bdp_model_gate.exceptions import GateConfigurationError # The guard: an error metric with nothing to compare against. try: PerformanceThresholdCheck(PerformanceConfig(metric="rmse")).run(context) except GateConfigurationError as exc: print("GateConfigurationError:", exc)

GateConfigurationError: performance.metric='rmse' is an error metric (lower is better), so it is gated with performance.max_error — which is unset. There is no sensible default: a ceiling depends on the scale of your target. Set max_error, or choose a higher-is-better metric such as 'r2'.

In [7]:

# A realistic pricing gate.
strict = PerformanceConfig(metric="rmse", max_error=45_000.0, max_latency_ms_p95=150.0)
for r in PerformanceThresholdCheck(strict).run(context):
    print(f"{r.flag:18} {r.detail}")

# A realistic pricing gate. strict = PerformanceConfig(metric="rmse", max_error=45_000.0, max_latency_ms_p95=150.0) for r in PerformanceThresholdCheck(strict).run(context): print(f"{r.flag:18} {r.detail}")

OK                 rmse=25582.2245 (max 45000.0) [computed without scikit-learn]
OK                 p95 latency=82.43ms (max 150.0ms)
OK                 cost/inference=0.00050 (max 0.002)

Claims severity — why MAPE¶

Severity is money and heavily right-skewed: a handful of large claims dominate squared error, so rmse ends up measuring the tail rather than typical accuracy. mape scores relative error, which is usually what a reserving team cares about.

In [8]:

severity_mask = realised_val > 0
severity_context = StructuredGateContext(
    model=pricer,
    X=X_val[severity_mask],
    y_true=realised_val[severity_mask],
    y_pred=y_pred[severity_mask],
    protected_df=prot_val[severity_mask],
    task="regression",
)
for name, cfg in (
    ("rmse", PerformanceConfig(metric="rmse", max_error=1e12)),
    ("mape", PerformanceConfig(metric="mape", max_error=1e12)),
):
    md = PerformanceThresholdCheck(cfg).run(severity_context)[0].metadata
    print(f"{name:6} {md['value']:>14,.4f}")
print("\nrmse is in naira; mape is a fraction — 0.25 means 25% average relative error.")

severity_mask = realised_val > 0 severity_context = StructuredGateContext( model=pricer, X=X_val[severity_mask], y_true=realised_val[severity_mask], y_pred=y_pred[severity_mask], protected_df=prot_val[severity_mask], task="regression", ) for name, cfg in ( ("rmse", PerformanceConfig(metric="rmse", max_error=1e12)), ("mape", PerformanceConfig(metric="mape", max_error=1e12)), ): md = PerformanceThresholdCheck(cfg).run(severity_context)[0].metadata print(f"{name:6} {md['value']:>14,.4f}") print("\nrmse is in naira; mape is a fraction — 0.25 means 25% average relative error.")

rmse      25,582.2245
mape           0.4748

rmse is in naira; mape is a fraction — 0.25 means 25% average relative error.

Claims frequency — why Poisson deviance¶

Frequency is a count, mostly zero. Squared error understates over-dispersion and treats "predicted 0.1 claims, saw 3" far too gently. poisson_deviance is built for this. It needs strictly positive predictions, since it takes their log.

In [9]:

freq_rng = np.random.default_rng(7)
true_rate = 0.05 + 0.02 * X_val["claims_last_3y"].to_numpy()
observed_claims = freq_rng.poisson(true_rate).astype(float)
predicted_rate = np.clip(true_rate * freq_rng.uniform(0.8, 1.25, len(true_rate)), 1e-4, None)

freq_context = StructuredGateContext(
    X=X_val, y_true=observed_claims, y_pred=predicted_rate,
    predict_fn=lambda df: np.full(len(df), true_rate.mean()),
    protected_df=prot_val, task="regression",
)
result = PerformanceThresholdCheck(
    PerformanceConfig(metric="poisson_deviance", max_error=0.5)
).run(freq_context)[0]
print(f"{result.flag:18} {result.detail}")
print(f"\nzero-claim policies: {(observed_claims == 0).mean():.1%} — deviance handles them; "
      "MAPE would be undefined there.")

freq_rng = np.random.default_rng(7) true_rate = 0.05 + 0.02 * X_val["claims_last_3y"].to_numpy() observed_claims = freq_rng.poisson(true_rate).astype(float) predicted_rate = np.clip(true_rate * freq_rng.uniform(0.8, 1.25, len(true_rate)), 1e-4, None) freq_context = StructuredGateContext( X=X_val, y_true=observed_claims, y_pred=predicted_rate, predict_fn=lambda df: np.full(len(df), true_rate.mean()), protected_df=prot_val, task="regression", ) result = PerformanceThresholdCheck( PerformanceConfig(metric="poisson_deviance", max_error=0.5) ).run(freq_context)[0] print(f"{result.flag:18} {result.detail}") print(f"\nzero-claim policies: {(observed_claims == 0).mean():.1%} — deviance handles them; " "MAPE would be undefined there.")

OK                 poisson_deviance=0.3227 (max 0.5) [computed without scikit-learn]

zero-claim policies: 94.5% — deviance handles them; MAPE would be undefined there.

In [10]:

# The guard for non-positive predictions.
try:
    PerformanceThresholdCheck(
        PerformanceConfig(metric="poisson_deviance", max_error=1.0)
    ).run(
        StructuredGateContext(
            X=X_val, y_true=observed_claims, y_pred=np.zeros(len(X_val)),
            predict_fn=lambda df: np.zeros(len(df)), task="regression",
        )
    )
except GateConfigurationError as exc:
    print("GateConfigurationError:", exc)

# The guard for non-positive predictions. try: PerformanceThresholdCheck( PerformanceConfig(metric="poisson_deviance", max_error=1.0) ).run( StructuredGateContext( X=X_val, y_true=observed_claims, y_pred=np.zeros(len(X_val)), predict_fn=lambda df: np.zeros(len(df)), task="regression", ) ) except GateConfigurationError as exc: print("GateConfigurationError:", exc)

GateConfigurationError: poisson_deviance requires strictly positive predictions (it takes their log); got a prediction <= 0

4. Fairness without a "selected" class¶

Four checks replace demographic parity, each answering something the others cannot:

Check	Question	Needs
LossRatioParityCheck	higher margin over own expected loss?	expected_loss
GroupMeanGapCheck	systematically higher predictions?	—
ErrorParityCheck	model materially less accurate for a group?	y_true
CalibrationParityCheck	predictions over- or under-shoot reality?	y_true

All gaps are measured relative to the overall figure, so one threshold works whether the target is naira or claim counts. Groups smaller than FairnessConfig.min_group_size (default 30) are reported but not scored — a three-policy segment otherwise produces a wild ratio that reads as a finding.

In [11]:

from bdp_model_gate.structured.regression_fairness import (
    CalibrationParityCheck,
    ErrorParityCheck,
    GroupMeanGapCheck,
    LossRatioParityCheck,
)

for check in (LossRatioParityCheck(), GroupMeanGapCheck(), ErrorParityCheck(),
              CalibrationParityCheck()):
    print(f"--- {check.name} ---")
    for r in check.run(context):
        print(f"  {r.flag:20} {r.detail}")
    print()

from bdp_model_gate.structured.regression_fairness import ( CalibrationParityCheck, ErrorParityCheck, GroupMeanGapCheck, LossRatioParityCheck, ) for check in (LossRatioParityCheck(), GroupMeanGapCheck(), ErrorParityCheck(), CalibrationParityCheck()): print(f"--- {check.name} ---") for r in check.run(context): print(f" {r.flag:20} {r.detail}") print()

--- loss_ratio_parity ---
  LOSS_RATIO_RISK      region: premium-to-expected-loss ratio spans 1.123 (Lagos) to 1.412 (Kano) — 23.6% of the overall ratio 1.225; Kano carries the higher margin over its own expected cost
  OK                   gender: premium-to-expected-loss ratio spans 1.220 (M) to 1.231 (F) — 0.9% of the overall ratio 1.225; F carries the higher margin over its own expected cost

--- group_mean_gap ---
  MEAN_GAP_RISK        region: mean prediction spans 62,023.24 (Lagos) to 77,356.29 (Kano) — 22.7% of the overall mean 67,679.61
  OK                   gender: mean prediction spans 65,769.34 (M) to 69,915.66 (F) — 6.1% of the overall mean 67,679.61

--- error_parity ---
  ERROR_PARITY_RISK    region: mean absolute error spans 16,683.46 to 25,629.29 (worst: Kano) — 44.6% of the overall MAE 20,035.80
  OK                   gender: mean absolute error spans 19,531.61 to 20,625.98 (worst: F) — 5.5% of the overall MAE 20,035.80

--- calibration_parity ---
  CALIBRATION_RISK     region: prediction bias spans 4,146.29 (Abuja, under-predicted) to 21,777.86 (Kano, over-predicted) — 31.4% of the overall actual mean 56,119.96
  OK                   gender: prediction bias spans 11,247.67 (M, under-predicted) to 11,924.83 (F, over-predicted) — 1.2% of the overall actual mean 56,119.96

The one that matters: loss-ratio parity¶

GroupMeanGapCheck sees that Kano pays more. On its own that is not evidence of unfairness — risk-based pricing is supposed to produce different premiums, and flagging it would make the gate noisy enough to ignore.

LossRatioParityCheck is the one that isolates the actual problem, by dividing each group's premium by its own expected cost.

In [12]:

result = LossRatioParityCheck().run(context)[0]
print(result.flag, "\n ", result.detail, "\n")
for key, value in result.metadata.items():
    print(f"  {key:24} {value}")

result = LossRatioParityCheck().run(context)[0] print(result.flag, "\n ", result.detail, "\n") for key, value in result.metadata.items(): print(f" {key:24} {value}")

LOSS_RATIO_RISK 
  region: premium-to-expected-loss ratio spans 1.123 (Lagos) to 1.412 (Kano) — 23.6% of the overall ratio 1.225; Kano carries the higher margin over its own expected cost 

  protected_attr           region
  relative_gap             0.2362
  threshold                0.1
  group_loss_ratio         {'Lagos': 1.1228, 'Kano': 1.4122, 'Abuja': 1.125}
  highest_margin_group     Kano
  lowest_margin_group      Lagos
  rows_ignored             0

It has to be told what "cost" means. Without expected_loss it reports NOT_APPLICABLE rather than falling back to a raw-price comparison, which would answer a different question under the same name.

In [13]:

no_loss = StructuredGateContext(
    model=pricer, X=X_val, y_true=realised_val, y_pred=y_pred,
    protected_df=prot_val, task="regression",
)
r = LossRatioParityCheck().run(no_loss)[0]
print(r.flag, "-", r.detail)

no_loss = StructuredGateContext( model=pricer, X=X_val, y_true=realised_val, y_pred=y_pred, protected_df=prot_val, task="regression", ) r = LossRatioParityCheck().run(no_loss)[0] print(r.flag, "-", r.detail)

NOT_APPLICABLE - no expected_loss supplied — margin parity needs a per-row expected loss (or technical premium) to compare the prediction against

Error parity — a quality-of-service question¶

A group the model simply predicts worse for is being under-served, however fair the average price looks. Below, predictions are degraded for one region only.

In [14]:

degraded = y_pred.copy()
kano = (prot_val["region"] == "Kano").to_numpy()
degraded[kano] *= 1.9

result = ErrorParityCheck().run(
    StructuredGateContext(
        model=pricer, X=X_val, y_true=realised_val, y_pred=degraded,
        protected_df=prot_val, task="regression",
    )
)[0]
print(result.flag, "\n ", result.detail)
print("\n  worst-served group:", result.metadata["worst_served_group"])

degraded = y_pred.copy() kano = (prot_val["region"] == "Kano").to_numpy() degraded[kano] *= 1.9 result = ErrorParityCheck().run( StructuredGateContext( model=pricer, X=X_val, y_true=realised_val, y_pred=degraded, protected_df=prot_val, task="regression", ) )[0] print(result.flag, "\n ", result.detail) print("\n worst-served group:", result.metadata["worst_served_group"])

ERROR_PARITY_RISK 
  region: mean absolute error spans 16,683.46 to 91,398.52 (worst: Kano) — 173.1% of the overall MAE 43,172.48

  worst-served group: Kano

ShapSubgroupCheck on a money target¶

ShapSubgroupCheck is task-agnostic and runs here too. SHAP contributions inherit the target's scale, so on a premium model they are measured in thousands of naira rather than fractions of a probability.

Since 0.4.2 the threshold is relative to the mean absolute contribution, so the same default works on either scale — a gap of 0.50 means "worth half a typical contribution". Before that it was absolute, and flagged essentially every feature on a money target.

In [15]:

from bdp_model_gate import FairnessConfig
from bdp_model_gate.structured.fairness import ShapSubgroupCheck

results = ShapSubgroupCheck().run(context)
flags = [r for r in results if not r.is_ok]
print(f"default threshold {FairnessConfig().shap_gap_threshold}: {len(flags)} flag(s)\n")
for r in flags:
    print("   ", r.detail)

# Only flagged features carry a result of their own; the check emits one
# summary result when nothing is above threshold.
print("\nrelative gap, per flagged feature:")
for r in results:
    gap = r.metadata.get("relative_gap")
    if gap is not None:
        print(f"   {r.metadata['feature']:24} {r.metadata['protected_attr']:8} {gap:6.3f}")

from bdp_model_gate import FairnessConfig from bdp_model_gate.structured.fairness import ShapSubgroupCheck results = ShapSubgroupCheck().run(context) flags = [r for r in results if not r.is_ok] print(f"default threshold {FairnessConfig().shap_gap_threshold}: {len(flags)} flag(s)\n") for r in flags: print(" ", r.detail) # Only flagged features carry a result of their own; the check emits one # summary result when nothing is above threshold. print("\nrelative gap, per flagged feature:") for r in results: gap = r.metadata.get("relative_gap") if gap is not None: print(f" {r.metadata['feature']:24} {r.metadata['protected_attr']:8} {gap:6.3f}")

default threshold 0.5: 2 flag(s)

    distance_to_branch_km SHAP contribution gap across region=15,961.291 — 268% of the mean absolute contribution 5,962.598
    sum_insured_ngn SHAP contribution gap across gender=3,423.240 — 57% of the mean absolute contribution 5,962.598

relative gap, per flagged feature:
   distance_to_branch_km    region    2.677
   sum_insured_ngn          gender    0.574

The planted geographic proxy stands well clear of the noise. That separation is what makes the threshold portable: on a probability-scale classifier the real driver lands in the same range, because both are measured against their own typical contribution rather than an absolute number of naira.

5. Robustness on a continuous output¶

A "prediction flip" is meaningless when the output is continuous — every perturbation moves it, so a flip rate would be ~1.0 and every regression model would be permanently blocked. Regression measures the mean relative prediction shift instead, against SecurityConfig.adversarial_max_relative_shift.

In [16]:

from bdp_model_gate.structured.security import AdversarialRobustnessCheck

result = AdversarialRobustnessCheck().run(context)[0]
print(result.flag, "-", result.detail)
print()
for key, value in result.metadata.items():
    print(f"  {key:22} {value}")

from bdp_model_gate.structured.security import AdversarialRobustnessCheck result = AdversarialRobustnessCheck().run(context)[0] print(result.flag, "-", result.detail) print() for key, value in result.metadata.items(): print(f" {key:22} {value}")

OK - mean relative prediction shift under random perturbation=0.0208 (max 0.1); inputs moved by epsilon=0.02

  relative_shift         0.0208
  threshold              0.1
  method                 random
  task                   regression
  epsilon                0.02

Note sum_insured_ngn is in the millions while claims_last_3y is a single digit. Each feature is perturbed relative to its own magnitude — a single global scale would be dominated by the largest column and would shove the small ones by orders of magnitude.

6. The whole gate¶

Classification-only checks report NOT_APPLICABLE, so the report records what was skipped and why.

In [17]:

config = GateConfig()
config.performance.metric = "rmse"
config.performance.max_error = 60_000.0
config.fairness.loss_ratio_threshold = 0.10
config.fairness.mean_gap_threshold = 0.15
# shap_gap_threshold needs no rescaling since 0.4.2 — it is relative.

report = ModelGate(checks=default_structured_checks(config, include_plugins=False)).run(context)
print(report.summary())

config = GateConfig() config.performance.metric = "rmse" config.performance.max_error = 60_000.0 config.fairness.loss_ratio_threshold = 0.10 config.fairness.mean_gap_threshold = 0.15 # shap_gap_threshold needs no rescaling since 0.4.2 — it is relative. report = ModelGate(checks=default_structured_checks(config, include_plugins=False)).run(context) print(report.summary())

Gate status: NEEDS_REVIEW (105ms, regression)
  rmse: 25582.2245
  performance: 0 flag(s)
  compliance: 0 flag(s)
  security: 0 flag(s)
  fairness: 7 flag(s)

In [18]:

display(
    pd.DataFrame(
        [
            {
                "category": r.category,
                "check": r.check_name,
                "flag": r.flag,
                "blocking": r.blocking,
                "detail": (r.detail[:60] + "...") if len(r.detail) > 60 else r.detail,
            }
            for r in report.results
        ]
    )
)

display( pd.DataFrame( [ { "category": r.category, "check": r.check_name, "flag": r.flag, "blocking": r.blocking, "detail": (r.detail[:60] + "...") if len(r.detail) > 60 else r.detail, } for r in report.results ] ) )

	category	check	flag	blocking	detail
0	fairness	proxy_correlation	PROXY_RISK	False	distance_to_branch_km correlates with region (...
1	fairness	disparate_impact	NOT_APPLICABLE	False	check does not apply to a regression task (sup...
2	fairness	shap_subgroup_gap	SUBGROUP_IMPACT_RISK	False	distance_to_branch_km SHAP contribution gap ac...
3	fairness	shap_subgroup_gap	SUBGROUP_IMPACT_RISK	False	sum_insured_ngn SHAP contribution gap across g...
4	fairness	counterfactual_flip	NOT_APPLICABLE	False	check does not apply to a regression task (sup...
5	fairness	group_mean_gap	MEAN_GAP_RISK	False	region: mean prediction spans 62,023.24 (Lagos...
6	fairness	group_mean_gap	OK	False	gender: mean prediction spans 65,769.34 (M) to...
7	fairness	error_parity	ERROR_PARITY_RISK	False	region: mean absolute error spans 16,683.46 to...
8	fairness	error_parity	OK	False	gender: mean absolute error spans 19,531.61 to...
9	fairness	calibration_parity	CALIBRATION_RISK	False	region: prediction bias spans 4,146.29 (Abuja,...
10	fairness	calibration_parity	OK	False	gender: prediction bias spans 11,247.67 (M, un...
11	fairness	loss_ratio_parity	LOSS_RATIO_RISK	False	region: premium-to-expected-loss ratio spans 1...
12	fairness	loss_ratio_parity	OK	False	gender: premium-to-expected-loss ratio spans 1...
13	performance	performance_thresholds	OK	True	rmse=25582.2245 (max 60000.0) [computed withou...
14	performance	performance_thresholds	OK	True	p95 latency=82.43ms (max 200.0ms)
15	performance	performance_thresholds	OK	True	cost/inference=0.00050 (max 0.002)
16	compliance	compliance_mapping	OK	True	model_card.legal_basis present
17	compliance	compliance_mapping	OK	True	model_card.data_minimization_justification pre...
18	compliance	compliance_mapping	OK	True	model_card.training_data_source present
19	compliance	compliance_mapping	OK	True	DPIA completed
20	compliance	compliance_mapping	OK	True	explainability method documented
21	security	adversarial_robustness	OK	True	mean relative prediction shift under random pe...
22	security	pii_leakage	OK	True	no string columns to scan
23	security	prompt_injection	NOT_APPLICABLE	True	no generative component supplied

In [19]:

print(f"verdict: {report.gate_status}   task: {report.task}   "
      f"{report.model_metric}={report.model_score:,.2f}\n")
for r in report.flags:
    print(f"[{'BLOCK ' if r.blocking else 'review'}] {r.check_name}")
    print(f"          {r.detail}\n")

print(f"verdict: {report.gate_status} task: {report.task} " f"{report.model_metric}={report.model_score:,.2f}\n") for r in report.flags: print(f"[{'BLOCK ' if r.blocking else 'review'}] {r.check_name}") print(f" {r.detail}\n")

verdict: NEEDS_REVIEW   task: regression   rmse=25,582.22

[review] proxy_correlation
          distance_to_branch_km correlates with region (eta^2=0.977)

[review] shap_subgroup_gap
          distance_to_branch_km SHAP contribution gap across region=15,961.291 — 268% of the mean absolute contribution 5,962.598

[review] shap_subgroup_gap
          sum_insured_ngn SHAP contribution gap across gender=3,423.240 — 57% of the mean absolute contribution 5,962.598

[review] group_mean_gap
          region: mean prediction spans 62,023.24 (Lagos) to 77,356.29 (Kano) — 22.7% of the overall mean 67,679.61

[review] error_parity
          region: mean absolute error spans 16,683.46 to 25,629.29 (worst: Kano) — 44.6% of the overall MAE 20,035.80

[review] calibration_parity
          region: prediction bias spans 4,146.29 (Abuja, under-predicted) to 21,777.86 (Kano, over-predicted) — 31.4% of the overall actual mean 56,119.96

[review] loss_ratio_parity
          region: premium-to-expected-loss ratio spans 1.123 (Lagos) to 1.412 (Kano) — 23.6% of the overall ratio 1.225; Kano carries the higher margin over its own expected cost

7. From the CLI¶

--expected-loss-col names a column in the data file, so the margin check works from a plain CSV.

In [20]:

import json
import subprocess
import sys
from pathlib import Path

import joblib

workdir = Path("cli_regression")
workdir.mkdir(exist_ok=True)

frame = X_val.copy()
frame["realised_loss"] = realised_val
frame["technical_premium"] = el_val
frame.to_csv(workdir / "validation.csv", index=False)
prot_val.to_csv(workdir / "protected.csv", index=False)
(workdir / "model_card.json").write_text(json.dumps(model_card, indent=2))
joblib.dump(pricer, workdir / "model.joblib")

proc = subprocess.run(
    [
        sys.executable, "-m", "bdp_model_gate.cli",
        "--model", str(workdir / "model.joblib"),
        "--data", str(workdir / "validation.csv"),
        "--target-col", "realised_loss",
        "--protected", str(workdir / "protected.csv"),
        "--model-card", str(workdir / "model_card.json"),
        "--task", "regression",
        "--expected-loss-col", "technical_premium",
        "--metric", "rmse",
        "--max-error", "60000",
        "--output", str(workdir / "gate_report.json"),
    ],
    capture_output=True, text=True,
)
print(proc.stdout or proc.stderr)
print("exit code", proc.returncode, "->",
      {0: "PASS", 1: "BLOCKED", 2: "NEEDS_REVIEW"}.get(proc.returncode))

import json import subprocess import sys from pathlib import Path import joblib workdir = Path("cli_regression") workdir.mkdir(exist_ok=True) frame = X_val.copy() frame["realised_loss"] = realised_val frame["technical_premium"] = el_val frame.to_csv(workdir / "validation.csv", index=False) prot_val.to_csv(workdir / "protected.csv", index=False) (workdir / "model_card.json").write_text(json.dumps(model_card, indent=2)) joblib.dump(pricer, workdir / "model.joblib") proc = subprocess.run( [ sys.executable, "-m", "bdp_model_gate.cli", "--model", str(workdir / "model.joblib"), "--data", str(workdir / "validation.csv"), "--target-col", "realised_loss", "--protected", str(workdir / "protected.csv"), "--model-card", str(workdir / "model_card.json"), "--task", "regression", "--expected-loss-col", "technical_premium", "--metric", "rmse", "--max-error", "60000", "--output", str(workdir / "gate_report.json"), ], capture_output=True, text=True, ) print(proc.stdout or proc.stderr) print("exit code", proc.returncode, "->", {0: "PASS", 1: "BLOCKED", 2: "NEEDS_REVIEW"}.get(proc.returncode))

Gate status: NEEDS_REVIEW (840ms, regression)
  rmse: 25582.2245
  performance: 0 flag(s)
  compliance: 0 flag(s)
  security: 0 flag(s)
  fairness: 7 flag(s)
Full report written to cli_regression/gate_report.json

exit code 2 -> NEEDS_REVIEW

In [21]:

saved = json.loads((workdir / "gate_report.json").read_text())
ratios = [
    r for r in saved["results_by_category"]["fairness"]
    if r["check_name"] == "loss_ratio_parity"
]
print(json.dumps(ratios, indent=2))

saved = json.loads((workdir / "gate_report.json").read_text()) ratios = [ r for r in saved["results_by_category"]["fairness"] if r["check_name"] == "loss_ratio_parity" ] print(json.dumps(ratios, indent=2))

[
  {
    "check_name": "loss_ratio_parity",
    "flag": "LOSS_RATIO_RISK",
    "detail": "region: premium-to-expected-loss ratio spans 1.123 (Lagos) to 1.412 (Kano) \u2014 23.6% of the overall ratio 1.225; Kano carries the higher margin over its own expected cost",
    "blocking": false,
    "metadata": {
      "protected_attr": "region",
      "relative_gap": 0.2362,
      "threshold": 0.1,
      "group_loss_ratio": {
        "Lagos": 1.1228,
        "Kano": 1.4122,
        "Abuja": 1.125
      },
      "highest_margin_group": "Kano",
      "lowest_margin_group": "Lagos",
      "rows_ignored": 0
    },
    "duration_ms": 0.63
  },
  {
    "check_name": "loss_ratio_parity",
    "flag": "OK",
    "detail": "gender: premium-to-expected-loss ratio spans 1.220 (M) to 1.231 (F) \u2014 0.9% of the overall ratio 1.225; F carries the higher margin over its own expected cost",
    "blocking": false,
    "metadata": {
      "protected_attr": "gender",
      "relative_gap": 0.0087,
      "threshold": 0.1,
      "group_loss_ratio": {
        "M": 1.2202,
        "F": 1.2309
      },
      "highest_margin_group": "F",
      "lowest_margin_group": "M",
      "rows_ignored": 0
    },
    "duration_ms": 0.63
  }
]

In [22]:

import shutil

shutil.rmtree(workdir, ignore_errors=True)
print("cleaned up")

import shutil shutil.rmtree(workdir, ignore_errors=True) print("cleaned up")

cleaned up

Summary¶

Doing regression	Do this
Any continuous target	task="regression"
Premium / severity	metric="rmse" or "mae"; "mape" for skewed money
Claim counts	metric="poisson_deviance" (needs positive predictions)
Error metrics	gate with max_error, not min_score — no default exists
Margin fairness	supply expected_loss to enable LossRatioParityCheck

Three things worth carrying away:

1. A higher premium is not unfairness. Risk-based pricing produces different prices by design. LossRatioParityCheck compares each group's premium to its own expected cost, which is the question that separates discrimination from actuarially justified variation.
2. Direction is explicit. rmse gated with min_score would pass every terrible model, so error metrics require max_error and the report names which comparison ran.
3. Pick the metric for the target's shape. rmse on skewed severity measures the tail; mape is undefined on zero claims; poisson_deviance handles counts. The gate will score whatever you ask for — choosing badly is not something it can catch for you.

Next: 02_multiclass_ordinal_sklearn.ipynb for underwriting decisions, or 04_any_framework_classification.ipynb for PyTorch and other non-sklearn models.