Regression models — pricing, severity and frequency¶

Three insurance models, three shapes of continuous target:

Model	Target	Metric that fits
Premium pricing	naira per policy	rmse / mae
Claims severity	naira per claim, heavily skewed	mape
Claims frequency	claim count, mostly zero	poisson_deviance

Regression changes more than the metric. Demographic parity has no analogue — there is no "selected" class to count — so four different fairness checks replace it. And the most important of them asks a question that only makes sense in insurance: is one group charged a higher margin over its own expected loss?

New to the library? 01_binary_classification_sklearn.ipynb covers contexts, reports and verdicts. This notebook assumes those and focuses on what regression changes.

In [1]:

# %pip install -q "bdp-model-gate[structured]"
import logging

import numpy as np
import pandas as pd

import bdp_model_gate

logging.basicConfig(level=logging.INFO, format="%(levelname)-8s %(name)s: %(message)s")
logging.getLogger("bdp_model_gate").setLevel(logging.WARNING)  # quieter; we log per-section
pd.set_option("display.width", 130)

print("bdp-model-gate", bdp_model_gate.__version__)

# %pip install -q "bdp-model-gate[structured]" import logging import numpy as np import pandas as pd import bdp_model_gate logging.basicConfig(level=logging.INFO, format="%(levelname)-8s %(name)s: %(message)s") logging.getLogger("bdp_model_gate").setLevel(logging.WARNING) # quieter; we log per-section pd.set_option("display.width", 130) print("bdp-model-gate", bdp_model_gate.__version__)

bdp-model-gate 0.7.0

1. A motor book with a margin problem¶

The premium follows risk, which is exactly as it should be — a higher-risk segment ought to pay more. What we plant instead is subtler and genuinely unfair: one region is charged a higher margin over its own expected loss. Their premium is not just higher, it is higher relative to what they cost.

For the model to actually reproduce that — rather than just the historical prices containing it — the book also carries distance_to_branch_km, a geographic rating factor that is nearly a relabelling of region. That is how a loading applied by region survives into a model that never sees region, and it is the realistic version of this problem.

The loss-ratio check exists to find exactly this, and no raw-price comparison can see it.

In [2]:

rng = np.random.default_rng(31)
N = 1600

region = rng.choice(["Lagos", "Kano", "Abuja"], N, p=[0.45, 0.35, 0.20])
gender = rng.choice(["F", "M"], N, p=[0.47, 0.53])

X = pd.DataFrame(
    {
        "driver_age": rng.integers(21, 68, N).astype(float),
        "vehicle_age": rng.integers(0, 18, N).astype(float),
        "sum_insured_ngn": rng.lognormal(15.4, 0.40, N).round(0),
        "claims_last_3y": rng.poisson(0.7, N).astype(float),
        "annual_mileage_km": rng.normal(18_000, 6_000, N).clip(2_000).round(0),
        # A geographic rating factor that is very nearly a relabelling of
        # region — which is how the regional loading below becomes learnable
        # even though region itself is never a feature.
        "distance_to_branch_km": (
            pd.Series(region).map({"Lagos": 2.0, "Abuja": 5.0, "Kano": 16.0}).to_numpy()
            + rng.normal(0, 1.0, N)
        ).round(2),
    }
)

# When each policy was written, 0-35 months ago. Not a feature — a pricing
# model must not learn "policies written in March are dearer" — but it is what
# the out-of-time split below is built on.
written_month = rng.integers(0, 36, N)

# True expected loss — genuinely risk-driven, no regional term.
expected_loss = (
    0.010 * X["sum_insured_ngn"]
    + 5_500 * X["claims_last_3y"]
    + 0.28 * X["annual_mileage_km"]
    - 220 * (X["driver_age"] - 21)
).clip(20_000)

# Claims inflation over the period. Everything scales together, so the loss
# *ratio* is untouched — but the level drifts, and a random split would spread
# that drift evenly across both sides and hide it. An out-of-time split will
# not, which is the whole argument for one.
expected_loss = (expected_loss * (1 + 0.006 * written_month)).round(0)

# The planted unfairness: Kano is loaded 42% over expected loss, everyone
# else 12%. Same risk, different margin.
loading = pd.Series(region).map({"Lagos": 1.12, "Abuja": 1.12, "Kano": 1.42}).to_numpy()
quoted_premium = (expected_loss * loading).round(0)

# Realised loss for the period — expected loss plus noise.
realised_loss = (expected_loss * rng.gamma(shape=8.0, scale=1 / 8.0, size=N)).round(0)

protected_df = pd.DataFrame({"region": region, "gender": gender})

print("mean figures by region:")
display(
    pd.DataFrame(
        {
            "expected_loss": pd.Series(expected_loss).groupby(region).mean(),
            "quoted_premium": pd.Series(quoted_premium).groupby(region).mean(),
            "loss_ratio": pd.Series(quoted_premium / expected_loss).groupby(region).mean(),
        }
    ).round(2)
)

rng = np.random.default_rng(31) N = 1600 region = rng.choice(["Lagos", "Kano", "Abuja"], N, p=[0.45, 0.35, 0.20]) gender = rng.choice(["F", "M"], N, p=[0.47, 0.53]) X = pd.DataFrame( { "driver_age": rng.integers(21, 68, N).astype(float), "vehicle_age": rng.integers(0, 18, N).astype(float), "sum_insured_ngn": rng.lognormal(15.4, 0.40, N).round(0), "claims_last_3y": rng.poisson(0.7, N).astype(float), "annual_mileage_km": rng.normal(18_000, 6_000, N).clip(2_000).round(0), # A geographic rating factor that is very nearly a relabelling of # region — which is how the regional loading below becomes learnable # even though region itself is never a feature. "distance_to_branch_km": ( pd.Series(region).map({"Lagos": 2.0, "Abuja": 5.0, "Kano": 16.0}).to_numpy() + rng.normal(0, 1.0, N) ).round(2), } ) # When each policy was written, 0-35 months ago. Not a feature — a pricing # model must not learn "policies written in March are dearer" — but it is what # the out-of-time split below is built on. written_month = rng.integers(0, 36, N) # True expected loss — genuinely risk-driven, no regional term. expected_loss = ( 0.010 * X["sum_insured_ngn"] + 5_500 * X["claims_last_3y"] + 0.28 * X["annual_mileage_km"] - 220 * (X["driver_age"] - 21) ).clip(20_000) # Claims inflation over the period. Everything scales together, so the loss # *ratio* is untouched — but the level drifts, and a random split would spread # that drift evenly across both sides and hide it. An out-of-time split will # not, which is the whole argument for one. expected_loss = (expected_loss * (1 + 0.006 * written_month)).round(0) # The planted unfairness: Kano is loaded 42% over expected loss, everyone # else 12%. Same risk, different margin. loading = pd.Series(region).map({"Lagos": 1.12, "Abuja": 1.12, "Kano": 1.42}).to_numpy() quoted_premium = (expected_loss * loading).round(0) # Realised loss for the period — expected loss plus noise. realised_loss = (expected_loss * rng.gamma(shape=8.0, scale=1 / 8.0, size=N)).round(0) protected_df = pd.DataFrame({"region": region, "gender": gender}) print("mean figures by region:") display( pd.DataFrame( { "expected_loss": pd.Series(expected_loss).groupby(region).mean(), "quoted_premium": pd.Series(quoted_premium).groupby(region).mean(), "loss_ratio": pd.Series(quoted_premium / expected_loss).groupby(region).mean(), } ).round(2) )

mean figures by region:

	expected_loss	quoted_premium	loss_ratio
Abuja	61365.65	68729.56	1.12
Kano	61882.48	87873.11	1.42
Lagos	62059.39	69506.53	1.12

Look at the two right-hand columns. Kano's premium is higher — but so is its expected loss, so a price comparison alone is ambiguous. The loss ratio is the number that gives it away: 1.42 against 1.12.

Whether the model reproduces that skew is the question the gate answers, and it is not the same question — a model that fails to learn the loading would price fairly despite unfair history.

In [3]:

from sklearn.ensemble import GradientBoostingRegressor

# An out-of-time split, not a random one. A random split asks "can the model
# price a policy it has not seen?" — but the model will be applied to *next
# quarter's* business, and that is a different question. Inflation, seasonality
# and mix all leak backwards through a random split, so it flatters the model.
#
# The gate now requires this for high-risk use cases, pricing among them.
CUTOFF = 24  # months: the first 24 train, the last 12 validate
is_train = written_month < CUTOFF

X_train, X_val = X[is_train], X[~is_train].reset_index(drop=True)
y_train = quoted_premium[is_train]
el_train, el_val = expected_loss[is_train], expected_loss[~is_train].reset_index(drop=True)
realised_val = realised_loss[~is_train].reset_index(drop=True)
prot_train, prot_val = protected_df[is_train], protected_df[~is_train].reset_index(drop=True)

# The pricing model learns the quoted premium, loading and all — region is
# not a feature, but the loading is learnable from correlated columns.
pricer = GradientBoostingRegressor(random_state=31, n_estimators=250, max_depth=3)
pricer.fit(X_train, y_train)
y_pred = pricer.predict(X_val)

print(f"train {len(X_train)} policies (months 0-{CUTOFF - 1}) | "
      f"validation {len(X_val)} (months {CUTOFF}-35)")
print(f"predicted premium: mean {y_pred.mean():,.0f}  "
      f"range {y_pred.min():,.0f}-{y_pred.max():,.0f}")

from sklearn.ensemble import GradientBoostingRegressor # An out-of-time split, not a random one. A random split asks "can the model # price a policy it has not seen?" — but the model will be applied to *next # quarter's* business, and that is a different question. Inflation, seasonality # and mix all leak backwards through a random split, so it flatters the model. # # The gate now requires this for high-risk use cases, pricing among them. CUTOFF = 24 # months: the first 24 train, the last 12 validate is_train = written_month < CUTOFF X_train, X_val = X[is_train], X[~is_train].reset_index(drop=True) y_train = quoted_premium[is_train] el_train, el_val = expected_loss[is_train], expected_loss[~is_train].reset_index(drop=True) realised_val = realised_loss[~is_train].reset_index(drop=True) prot_train, prot_val = protected_df[is_train], protected_df[~is_train].reset_index(drop=True) # The pricing model learns the quoted premium, loading and all — region is # not a feature, but the loading is learnable from correlated columns. pricer = GradientBoostingRegressor(random_state=31, n_estimators=250, max_depth=3) pricer.fit(X_train, y_train) y_pred = pricer.predict(X_val) print(f"train {len(X_train)} policies (months 0-{CUTOFF - 1}) | " f"validation {len(X_val)} (months {CUTOFF}-35)") print(f"predicted premium: mean {y_pred.mean():,.0f} " f"range {y_pred.min():,.0f}-{y_pred.max():,.0f}")

train 1086 policies (months 0-23) | validation 514 (months 24-35)
predicted premium: mean 72,239  range 22,182-211,780

2. The context¶

Two regression-specific fields: task="regression" and expected_loss. The latter is what unlocks the margin check — supply a per-row expected loss, technical premium or pure premium.

In [4]:

from bdp_model_gate import GateConfig, ModelGate, StructuredGateContext
from bdp_model_gate.structured import default_structured_checks

model_card = {
    "model_name": "motor-pricing-gbm",
    "version": "3.0.2",
    "use_case": "pricing",                        # high-risk -> DPIA required
    "legal_basis": "Contractual necessity (NDPA 2023, s.25(1)(b))",
    "data_minimization_justification": "Rating factors only; no device or browsing data.",
    "training_data_source": "Motor book 2021-2025",
    "validation_strategy": "out_of_time",         # required, and checked
    "dpia_completed": True,
    "influences_decision_about_person": True,
    "explainability_method": "SHAP contributions retained per quote",
}

context = StructuredGateContext(
    model=pricer,
    X=X_val,
    y_true=realised_val,          # what actually happened
    y_pred=y_pred,                # what we charged
    protected_df=prot_val,
    X_train=X_train,              # enables the split-overlap and drift checks
    expected_loss=el_val,         # enables LossRatioParityCheck
    model_card=model_card,
    latencies_ms=rng.gamma(9.0, 6.0, 400),
    cost_per_inference=0.0005,
    task="regression",
)

from bdp_model_gate import GateConfig, ModelGate, StructuredGateContext from bdp_model_gate.structured import default_structured_checks model_card = { "model_name": "motor-pricing-gbm", "version": "3.0.2", "use_case": "pricing", # high-risk -> DPIA required "legal_basis": "Contractual necessity (NDPA 2023, s.25(1)(b))", "data_minimization_justification": "Rating factors only; no device or browsing data.", "training_data_source": "Motor book 2021-2025", "validation_strategy": "out_of_time", # required, and checked "dpia_completed": True, "influences_decision_about_person": True, "explainability_method": "SHAP contributions retained per quote", } context = StructuredGateContext( model=pricer, X=X_val, y_true=realised_val, # what actually happened y_pred=y_pred, # what we charged protected_df=prot_val, X_train=X_train, # enables the split-overlap and drift checks expected_loss=el_val, # enables LossRatioParityCheck model_card=model_card, latencies_ms=rng.gamma(9.0, 6.0, 400), cost_per_inference=0.0005, task="regression", )

3. Metrics, and which way they point¶

The big change: error metrics are lower-is-better, so they are gated with max_error rather than min_score. Each metric declares its own direction and the report says which comparison ran.

There is deliberately no default max_error — a sensible ceiling depends entirely on whether the target is naira or claim counts — so configuring an error metric without one is an error rather than a silent pass.

In [5]:

from bdp_model_gate import PerformanceConfig
from bdp_model_gate.metrics import BUILTIN_METRICS
from bdp_model_gate.structured.performance import PerformanceThresholdCheck

rows = []
for name, spec in sorted(BUILTIN_METRICS.items()):
    if "regression" not in spec.tasks:
        continue
    cfg = (
        PerformanceConfig(metric=name, min_score=-1e12)
        if spec.greater_is_better
        else PerformanceConfig(metric=name, max_error=1e12)
    )
    result = PerformanceThresholdCheck(cfg).run(context)[0]
    rows.append(
        {
            "metric": name,
            "value": result.metadata["value"],
            "direction": "higher is better" if spec.greater_is_better else "lower is better",
            "gated with": result.metadata["threshold_field"],
        }
    )
display(pd.DataFrame(rows).set_index("metric"))

from bdp_model_gate import PerformanceConfig from bdp_model_gate.metrics import BUILTIN_METRICS from bdp_model_gate.structured.performance import PerformanceThresholdCheck rows = [] for name, spec in sorted(BUILTIN_METRICS.items()): if "regression" not in spec.tasks: continue cfg = ( PerformanceConfig(metric=name, min_score=-1e12) if spec.greater_is_better else PerformanceConfig(metric=name, max_error=1e12) ) result = PerformanceThresholdCheck(cfg).run(context)[0] rows.append( { "metric": name, "value": result.metadata["value"], "direction": "higher is better" if spec.greater_is_better else "lower is better", "gated with": result.metadata["threshold_field"], } ) display(pd.DataFrame(rows).set_index("metric"))

	value	direction	gated with
metric
lorenz_gini	0.2116	higher is better	min_score
mae	20028.7245	lower is better	max_error
mape	0.4183	lower is better	max_error
poisson_deviance	8948.3929	lower is better	max_error
r2	0.4431	higher is better	min_score
rmse	25570.2945	lower is better	max_error

All five are implemented in numpy, so they work on a core install without scikit-learn. metric="auto" resolves to r2 for regression — an rmse default threshold would be meaningless without knowing the scale of the target.

In [6]:

from bdp_model_gate.exceptions import GateConfigurationError

# The guard: an error metric with nothing to compare against.
try:
    PerformanceThresholdCheck(PerformanceConfig(metric="rmse")).run(context)
except GateConfigurationError as exc:
    print("GateConfigurationError:", exc)

from bdp_model_gate.exceptions import GateConfigurationError # The guard: an error metric with nothing to compare against. try: PerformanceThresholdCheck(PerformanceConfig(metric="rmse")).run(context) except GateConfigurationError as exc: print("GateConfigurationError:", exc)

GateConfigurationError: performance.metric='rmse' is an error metric (lower is better), so it is gated with performance.max_error — which is unset. There is no sensible default: a ceiling depends on the scale of your target. Set max_error, or choose a higher-is-better metric such as 'r2'.

In [7]:

# A realistic pricing gate.
strict = PerformanceConfig(metric="rmse", max_error=45_000.0, max_latency_ms_p95=150.0)
for r in PerformanceThresholdCheck(strict).run(context):
    print(f"{r.flag:18} {r.detail}")

# A realistic pricing gate. strict = PerformanceConfig(metric="rmse", max_error=45_000.0, max_latency_ms_p95=150.0) for r in PerformanceThresholdCheck(strict).run(context): print(f"{r.flag:18} {r.detail}")

OK                 rmse=25570.2945 [23835.256, 27249.879] (max 45000.0) — the whole interval sits below 45000.000
OK                 p95 latency=86.65ms [82.92, 91.75] (max 150.0ms) — the whole interval sits below 150.000
OK                 cost/inference=0.00050 (max 0.002)

Claims severity — why MAPE¶

Severity is money and heavily right-skewed: a handful of large claims dominate squared error, so rmse ends up measuring the tail rather than typical accuracy. mape scores relative error, which is usually what a reserving team cares about.

In [8]:

severity_mask = realised_val > 0
severity_context = StructuredGateContext(
    model=pricer,
    X=X_val[severity_mask],
    y_true=realised_val[severity_mask],
    y_pred=y_pred[severity_mask],
    protected_df=prot_val[severity_mask],
    task="regression",
)
for name, cfg in (
    ("rmse", PerformanceConfig(metric="rmse", max_error=1e12)),
    ("mape", PerformanceConfig(metric="mape", max_error=1e12)),
):
    md = PerformanceThresholdCheck(cfg).run(severity_context)[0].metadata
    print(f"{name:6} {md['value']:>14,.4f}")
print("\nrmse is in naira; mape is a fraction — 0.25 means 25% average relative error.")

severity_mask = realised_val > 0 severity_context = StructuredGateContext( model=pricer, X=X_val[severity_mask], y_true=realised_val[severity_mask], y_pred=y_pred[severity_mask], protected_df=prot_val[severity_mask], task="regression", ) for name, cfg in ( ("rmse", PerformanceConfig(metric="rmse", max_error=1e12)), ("mape", PerformanceConfig(metric="mape", max_error=1e12)), ): md = PerformanceThresholdCheck(cfg).run(severity_context)[0].metadata print(f"{name:6} {md['value']:>14,.4f}") print("\nrmse is in naira; mape is a fraction — 0.25 means 25% average relative error.")

rmse      25,570.2945
mape           0.4183

rmse is in naira; mape is a fraction — 0.25 means 25% average relative error.

Claims frequency — why Poisson deviance¶

Frequency is a count, mostly zero. Squared error understates over-dispersion and treats "predicted 0.1 claims, saw 3" far too gently. poisson_deviance is built for this. It needs strictly positive predictions, since it takes their log.

In [9]:

freq_rng = np.random.default_rng(7)
true_rate = 0.05 + 0.02 * X_val["claims_last_3y"].to_numpy()
observed_claims = freq_rng.poisson(true_rate).astype(float)
predicted_rate = np.clip(true_rate * freq_rng.uniform(0.8, 1.25, len(true_rate)), 1e-4, None)

freq_context = StructuredGateContext(
    X=X_val, y_true=observed_claims, y_pred=predicted_rate,
    predict_fn=lambda df: np.full(len(df), true_rate.mean()),
    protected_df=prot_val, task="regression",
)
result = PerformanceThresholdCheck(
    PerformanceConfig(metric="poisson_deviance", max_error=0.5)
).run(freq_context)[0]
print(f"{result.flag:18} {result.detail}")
print(f"\nzero-claim policies: {(observed_claims == 0).mean():.1%} — deviance handles them; "
      "MAPE would be undefined there.")

freq_rng = np.random.default_rng(7) true_rate = 0.05 + 0.02 * X_val["claims_last_3y"].to_numpy() observed_claims = freq_rng.poisson(true_rate).astype(float) predicted_rate = np.clip(true_rate * freq_rng.uniform(0.8, 1.25, len(true_rate)), 1e-4, None) freq_context = StructuredGateContext( X=X_val, y_true=observed_claims, y_pred=predicted_rate, predict_fn=lambda df: np.full(len(df), true_rate.mean()), protected_df=prot_val, task="regression", ) result = PerformanceThresholdCheck( PerformanceConfig(metric="poisson_deviance", max_error=0.5) ).run(freq_context)[0] print(f"{result.flag:18} {result.detail}") print(f"\nzero-claim policies: {(observed_claims == 0).mean():.1%} — deviance handles them; " "MAPE would be undefined there.")

OK                 poisson_deviance=0.3289 [0.258, 0.408] (max 0.5) — the whole interval sits below 0.500

zero-claim policies: 94.4% — deviance handles them; MAPE would be undefined there.

In [10]:

# The guard for non-positive predictions.
try:
    PerformanceThresholdCheck(
        PerformanceConfig(metric="poisson_deviance", max_error=1.0)
    ).run(
        StructuredGateContext(
            X=X_val, y_true=observed_claims, y_pred=np.zeros(len(X_val)),
            predict_fn=lambda df: np.zeros(len(df)), task="regression",
        )
    )
except GateConfigurationError as exc:
    print("GateConfigurationError:", exc)

# The guard for non-positive predictions. try: PerformanceThresholdCheck( PerformanceConfig(metric="poisson_deviance", max_error=1.0) ).run( StructuredGateContext( X=X_val, y_true=observed_claims, y_pred=np.zeros(len(X_val)), predict_fn=lambda df: np.zeros(len(df)), task="regression", ) ) except GateConfigurationError as exc: print("GateConfigurationError:", exc)

GateConfigurationError: poisson_deviance requires strictly positive predictions (it takes their log); got a prediction <= 0

4. Fairness without a "selected" class¶

Four checks replace demographic parity, each answering something the others cannot:

Check	Question	Needs
LossRatioParityCheck	higher margin over own expected loss?	expected_loss
GroupMeanGapCheck	systematically higher predictions?	—
ErrorParityCheck	model materially less accurate for a group?	y_true
CalibrationParityCheck	predictions over- or under-shoot reality?	y_true

All gaps are measured relative to the overall figure, so one threshold works whether the target is naira or claim counts. Groups smaller than FairnessConfig.min_group_size (default 30) are reported but not scored — a three-policy segment otherwise produces a wild ratio that reads as a finding.

In [11]:

from bdp_model_gate.structured.regression_fairness import (
    CalibrationParityCheck,
    ErrorParityCheck,
    GroupMeanGapCheck,
    LossRatioParityCheck,
)

for check in (LossRatioParityCheck(), GroupMeanGapCheck(), ErrorParityCheck(),
              CalibrationParityCheck()):
    print(f"--- {check.name} ---")
    for r in check.run(context):
        print(f"  {r.flag:20} {r.detail}")
    print()

from bdp_model_gate.structured.regression_fairness import ( CalibrationParityCheck, ErrorParityCheck, GroupMeanGapCheck, LossRatioParityCheck, ) for check in (LossRatioParityCheck(), GroupMeanGapCheck(), ErrorParityCheck(), CalibrationParityCheck()): print(f"--- {check.name} ---") for r in check.run(context): print(f" {r.flag:20} {r.detail}") print()

--- loss_ratio_parity ---

  LOSS_RATIO_RISK      region: premium-to-expected-loss ratio spans 1.021 (Lagos) to 1.282 (Kano) — 23.3% of the overall ratio 1.118; Kano carries the higher margin over its own expected cost — the whole 95% interval sits above 0.100
  OK                   gender: premium-to-expected-loss ratio spans 1.112 (M) to 1.126 (F) — 1.2% of the overall ratio 1.118; F carries the higher margin over its own expected cost — the whole interval sits below 0.100

--- group_mean_gap ---

  MEAN_GAP_RISK        region: mean prediction spans 63,898.46 (Lagos) to 84,990.99 (Kano) — 29.2% of the overall mean 72,238.80 — the whole 95% interval sits above 0.100
  UNCERTAIN            gender: mean prediction spans 71,090.28 (M) to 73,798.26 (F) — 3.7% of the overall mean 72,238.80 — the estimate stays the right side of 0.100 but the interval reaches 0.113, so this sample cannot rule out a breach

--- error_parity ---

  UNCERTAIN            region: mean absolute error spans 18,193.09 to 23,186.77 (worst: Kano) — 24.9% of the overall MAE 20,028.72 — the estimate breaches 0.200 but the interval reaches back to 0.132, so this sample cannot confirm a breach
  OK                   gender: mean absolute error spans 20,016.93 to 20,044.74 (worst: F) — 0.1% of the overall MAE 20,028.72 — the whole interval sits below 0.200

--- calibration_parity ---

  UNCERTAIN            region: prediction bias spans 4,224.02 (Abuja, under-predicted) to 13,681.74 (Kano, over-predicted) — 14.8% of the overall actual mean 63,975.19 — the estimate breaches 0.100 but the interval reaches back to 0.079, so this sample cannot confirm a breach
  UNCERTAIN            gender: prediction bias spans 7,141.16 (M, under-predicted) to 9,787.66 (F, over-predicted) — 4.1% of the overall actual mean 63,975.19 — the estimate stays the right side of 0.100 but the interval reaches 0.113, so this sample cannot rule out a breach

The one that matters: loss-ratio parity¶

GroupMeanGapCheck sees that Kano pays more. On its own that is not evidence of unfairness — risk-based pricing is supposed to produce different premiums, and flagging it would make the gate noisy enough to ignore.

LossRatioParityCheck is the one that isolates the actual problem, by dividing each group's premium by its own expected cost.

In [12]:

result = LossRatioParityCheck().run(context)[0]
print(result.flag, "\n ", result.detail, "\n")
for key, value in result.metadata.items():
    print(f"  {key:24} {value}")

result = LossRatioParityCheck().run(context)[0] print(result.flag, "\n ", result.detail, "\n") for key, value in result.metadata.items(): print(f" {key:24} {value}")

LOSS_RATIO_RISK 
  region: premium-to-expected-loss ratio spans 1.021 (Lagos) to 1.282 (Kano) — 23.3% of the overall ratio 1.118; Kano carries the higher margin over its own expected cost — the whole 95% interval sits above 0.100 

  protected_attr           region
  relative_gap             0.2331
  threshold                0.1
  exposure_weighted        False
  group_loss_ratio         {'Lagos': 1.021, 'Kano': 1.2817, 'Abuja': 1.0277}
  highest_margin_group     Kano
  lowest_margin_group      Lagos
  rows_ignored             0
  on_uncertain             review
  point                    0.2331
  ci_low                   0.2251
  ci_high                  0.2415
  ci_level                 0.95
  bootstrap_samples        1000

It has to be told what "cost" means. Without expected_loss it reports NOT_APPLICABLE rather than falling back to a raw-price comparison, which would answer a different question under the same name.

In [13]:

no_loss = StructuredGateContext(
    model=pricer, X=X_val, y_true=realised_val, y_pred=y_pred,
    protected_df=prot_val, task="regression",
)
r = LossRatioParityCheck().run(no_loss)[0]
print(r.flag, "-", r.detail)

no_loss = StructuredGateContext( model=pricer, X=X_val, y_true=realised_val, y_pred=y_pred, protected_df=prot_val, task="regression", ) r = LossRatioParityCheck().run(no_loss)[0] print(r.flag, "-", r.detail)

NOT_APPLICABLE - no expected_loss supplied — margin parity needs a per-row expected loss (or technical premium) to compare the prediction against

Error parity — a quality-of-service question¶

A group the model simply predicts worse for is being under-served, however fair the average price looks. Below, predictions are degraded for one region only.

In [14]:

degraded = y_pred.copy()
kano = (prot_val["region"] == "Kano").to_numpy()
degraded[kano] *= 1.9

result = ErrorParityCheck().run(
    StructuredGateContext(
        model=pricer, X=X_val, y_true=realised_val, y_pred=degraded,
        protected_df=prot_val, task="regression",
    )
)[0]
print(result.flag, "\n ", result.detail)
print("\n  worst-served group:", result.metadata["worst_served_group"])

degraded = y_pred.copy() kano = (prot_val["region"] == "Kano").to_numpy() degraded[kano] *= 1.9 result = ErrorParityCheck().run( StructuredGateContext( model=pricer, X=X_val, y_true=realised_val, y_pred=degraded, protected_df=prot_val, task="regression", ) )[0] print(result.flag, "\n ", result.detail) print("\n worst-served group:", result.metadata["worst_served_group"])

ERROR_PARITY_RISK 
  region: mean absolute error spans 18,193.09 to 90,218.71 (worst: Kano) — 161.7% of the overall MAE 44,546.24 — the whole 95% interval sits above 0.200

  worst-served group: Kano

ShapSubgroupCheck on a money target¶

ShapSubgroupCheck is task-agnostic and runs here too. SHAP contributions inherit the target's scale, so on a premium model they are measured in thousands of naira rather than fractions of a probability.

Since 0.4.2 the threshold is relative to the mean absolute contribution, so the same default works on either scale — a gap of 0.50 means "worth half a typical contribution". Before that it was absolute, and flagged essentially every feature on a money target.

In [15]:

from bdp_model_gate import FairnessConfig
from bdp_model_gate.structured.fairness import ShapSubgroupCheck

results = ShapSubgroupCheck().run(context)
flags = [r for r in results if not r.is_ok]
print(f"default threshold {FairnessConfig().shap_gap_threshold}: {len(flags)} flag(s)\n")
for r in flags:
    print("   ", r.detail)

# Only flagged features carry a result of their own; the check emits one
# summary result when nothing is above threshold.
print("\nrelative gap, per flagged feature:")
for r in results:
    gap = r.metadata.get("relative_gap")
    if gap is not None:
        print(f"   {r.metadata['feature']:24} {r.metadata['protected_attr']:8} {gap:6.3f}")

from bdp_model_gate import FairnessConfig from bdp_model_gate.structured.fairness import ShapSubgroupCheck results = ShapSubgroupCheck().run(context) flags = [r for r in results if not r.is_ok] print(f"default threshold {FairnessConfig().shap_gap_threshold}: {len(flags)} flag(s)\n") for r in flags: print(" ", r.detail) # Only flagged features carry a result of their own; the check emits one # summary result when nothing is above threshold. print("\nrelative gap, per flagged feature:") for r in results: gap = r.metadata.get("relative_gap") if gap is not None: print(f" {r.metadata['feature']:24} {r.metadata['protected_attr']:8} {gap:6.3f}")

/private/tmp/claude-501/-Users-vangelis-Documents-Bluechip-2026-Applications-model-gate/631c3443-469b-4452-8b45-f2ce39407b48/scratchpad/devvenv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

default threshold 0.5: 3 flag(s)

    sum_insured_ngn SHAP contribution gap across region=3,923.278 — 0.628 [0.182, 1.617] at 95% of the mean absolute contribution 6,242.353 (max 50%) — the estimate breaches 0.500 but the interval reaches back to 0.182, so this sample cannot confirm a breach
    distance_to_branch_km SHAP contribution gap across region=16,809.671 — 2.693 [2.643, 2.761] at 95% of the mean absolute contribution 6,242.353 (max 50%) — the whole 95% interval sits above 0.500
    sum_insured_ngn SHAP contribution gap across gender=1,237.666 — 0.198 [0.008, 0.978] at 95% of the mean absolute contribution 6,242.353 (max 50%) — the estimate stays the right side of 0.500 but the interval reaches 0.978, so this sample cannot rule out a breach

relative gap, per flagged feature:
   sum_insured_ngn          region    0.628
   distance_to_branch_km    region    2.693
   sum_insured_ngn          gender    0.198

The planted geographic proxy stands well clear of the noise. That separation is what makes the threshold portable: on a probability-scale classifier the real driver lands in the same range, because both are measured against their own typical contribution rather than an absolute number of naira.

5. Robustness on a continuous output¶

A "prediction flip" is meaningless when the output is continuous — every perturbation moves it, so a flip rate would be ~1.0 and every regression model would be permanently blocked. Regression measures the mean relative prediction shift instead, against SecurityConfig.adversarial_max_relative_shift.

In [16]:

from bdp_model_gate.structured.security import AdversarialRobustnessCheck

result = AdversarialRobustnessCheck().run(context)[0]
print(result.flag, "-", result.detail)
print()
for key, value in result.metadata.items():
    print(f"  {key:22} {value}")

from bdp_model_gate.structured.security import AdversarialRobustnessCheck result = AdversarialRobustnessCheck().run(context)[0] print(result.flag, "-", result.detail) print() for key, value in result.metadata.items(): print(f" {key:22} {value}")

OK - mean relative prediction shift under random perturbation=0.0241 (max 0.1); inputs moved by epsilon=0.02

  relative_shift         0.0241
  threshold              0.1
  method                 random
  task                   regression
  epsilon                0.02

Note sum_insured_ngn is in the millions while claims_last_3y is a single digit. Each feature is perturbed relative to its own magnitude — a single global scale would be dominated by the largest column and would shove the small ones by orders of magnitude.

6. The whole gate¶

Classification-only checks report NOT_APPLICABLE, so the report records what was skipped and why.

In [17]:

config = GateConfig()
config.performance.metric = "rmse"
config.performance.max_error = 60_000.0
config.fairness.loss_ratio_threshold = 0.10
config.fairness.mean_gap_threshold = 0.15
# shap_gap_threshold needs no rescaling since 0.4.2 — it is relative.

report = ModelGate(checks=default_structured_checks(config, include_plugins=False)).run(context)
print(report.summary())

config = GateConfig() config.performance.metric = "rmse" config.performance.max_error = 60_000.0 config.fairness.loss_ratio_threshold = 0.10 config.fairness.mean_gap_threshold = 0.15 # shap_gap_threshold needs no rescaling since 0.4.2 — it is relative. report = ModelGate(checks=default_structured_checks(config, include_plugins=False)).run(context) print(report.summary())

Gate status: BLOCKED (6730ms, regression)
  rmse: 25570.2945
  validation: 0 flag(s)
  performance: 2 flag(s)
  compliance: 0 flag(s)
  security: 0 flag(s)
  fairness: 9 flag(s)

In [18]:

display(
    pd.DataFrame(
        [
            {
                "category": r.category,
                "check": r.check_name,
                "flag": r.flag,
                "blocking": r.blocking,
                "detail": (r.detail[:60] + "...") if len(r.detail) > 60 else r.detail,
            }
            for r in report.results
        ]
    )
)

display( pd.DataFrame( [ { "category": r.category, "check": r.check_name, "flag": r.flag, "blocking": r.blocking, "detail": (r.detail[:60] + "...") if len(r.detail) > 60 else r.detail, } for r in report.results ] ) )

	category	check	flag	blocking	detail
0	validation	target_leakage	OK	True	no single feature approaches the model's own p...
1	validation	split_overlap	OK	True	0 duplicate row(s) within the validation set (...
2	validation	split_overlap	OK	True	0 of 514 validation rows (0.0%, max 1%) also a...
3	validation	validation_strategy	OK	True	validated with a 'out_of_time' split (out-of-t...
4	validation	feature_contract	OK	True	X matches the 6 feature(s) recorded in model.f...
5	validation	feature_drift	OK	False	no drift above threshold across 6 shared featu...
6	comparison	comparison_integrity	NOT_APPLICABLE	True	no context.champion — the comparison checks me...
7	comparison	challenger_performance	NOT_APPLICABLE	True	no context.champion — the comparison checks me...
8	comparison	challenger_guardrails	NOT_APPLICABLE	True	no context.champion — the comparison checks me...
9	comparison	subgroup_regression	NOT_APPLICABLE	False	no context.champion — the comparison checks me...
10	comparison	fairness_delta	NOT_APPLICABLE	False	no context.champion — the comparison checks me...
11	comparison	decision_churn	NOT_APPLICABLE	False	check does not apply to a regression task (sup...
12	comparison	operational_delta	NOT_APPLICABLE	False	no context.champion — the comparison checks me...
13	fairness	proxy_correlation	PROXY_RISK	False	distance_to_branch_km correlates with region (...
14	fairness	disparate_impact	NOT_APPLICABLE	False	check does not apply to a regression task (sup...
15	fairness	shap_subgroup_gap	UNCERTAIN	False	sum_insured_ngn SHAP contribution gap across r...
16	fairness	shap_subgroup_gap	SUBGROUP_IMPACT_RISK	False	distance_to_branch_km SHAP contribution gap ac...
17	fairness	shap_subgroup_gap	UNCERTAIN	False	sum_insured_ngn SHAP contribution gap across g...
18	fairness	counterfactual_flip	NOT_APPLICABLE	False	check does not apply to a regression task (sup...
19	fairness	equalised_odds	NOT_APPLICABLE	False	check does not apply to a regression task (sup...
20	fairness	subgroup_calibration	NOT_APPLICABLE	False	check does not apply to a regression task (sup...
21	fairness	group_mean_gap	MEAN_GAP_RISK	False	region: mean prediction spans 63,898.46 (Lagos...
22	fairness	group_mean_gap	OK	False	gender: mean prediction spans 71,090.28 (M) to...
23	fairness	error_parity	UNCERTAIN	False	region: mean absolute error spans 18,193.09 to...
24	fairness	error_parity	OK	False	gender: mean absolute error spans 20,016.93 to...
25	fairness	calibration_parity	UNCERTAIN	False	region: prediction bias spans 4,224.02 (Abuja,...
26	fairness	calibration_parity	UNCERTAIN	False	gender: prediction bias spans 7,141.16 (M, und...
27	fairness	loss_ratio_parity	LOSS_RATIO_RISK	False	region: premium-to-expected-loss ratio spans 1...
28	fairness	loss_ratio_parity	OK	False	gender: premium-to-expected-loss ratio spans 1...
29	performance	performance_thresholds	OK	True	rmse=25570.2945 [23835.256, 27249.879] (max 60...
30	performance	performance_thresholds	OK	True	p95 latency=86.65ms [82.92, 91.75] (max 200.0m...
31	performance	performance_thresholds	OK	True	cost/inference=0.00050 (max 0.002)
32	performance	calibration	NOT_APPLICABLE	True	check does not apply to a regression task (sup...
33	performance	actual_vs_expected	AE_LEVEL_RISK	True	book A/E = 0.886 (11.4% from break-even, over-...
34	performance	actual_vs_expected	AE_BAND_RISK	True	worst band is 5 of 10 (60,098–66,152): A/E = 0...
35	performance	risk_discrimination	OK	True	Gini = 0.212, 74% of the 0.288 this book allow...
36	compliance	compliance_mapping	OK	True	model_card.legal_basis present
37	compliance	compliance_mapping	OK	True	model_card.data_minimization_justification pre...
38	compliance	compliance_mapping	OK	True	model_card.training_data_source present
39	compliance	compliance_mapping	OK	True	DPIA completed
40	compliance	compliance_mapping	OK	True	explainability method documented
41	compliance	monotonicity	NOT_APPLICABLE	True	no actuarial.monotonic_features declared — a m...
42	compliance	prediction_dislocation	NOT_APPLICABLE	False	no baseline_pred or champion supplied — disloc...
43	security	adversarial_robustness	OK	True	mean relative prediction shift under random pe...
44	security	pii_leakage	OK	True	no string columns to scan
45	security	prompt_injection	NOT_APPLICABLE	True	no generative component supplied — set context...
46	security	report_injection	OK	False	no instruction-shaped text in the 26 string(s)...

In [19]:

print(f"verdict: {report.gate_status}   task: {report.task}   "
      f"{report.model_metric}={report.model_score:,.2f}\n")
for r in report.flags:
    print(f"[{'BLOCK ' if r.blocking else 'review'}] {r.check_name}")
    print(f"          {r.detail}\n")

print(f"verdict: {report.gate_status} task: {report.task} " f"{report.model_metric}={report.model_score:,.2f}\n") for r in report.flags: print(f"[{'BLOCK ' if r.blocking else 'review'}] {r.check_name}") print(f" {r.detail}\n")

verdict: BLOCKED   task: regression   rmse=25,570.29

[review] proxy_correlation
          distance_to_branch_km correlates with region (eta^2=0.977, q=0.012 across 12 comparison(s))

[review] shap_subgroup_gap
          sum_insured_ngn SHAP contribution gap across region=3,923.278 — 0.628 [0.182, 1.617] at 95% of the mean absolute contribution 6,242.353 (max 50%) — the estimate breaches 0.500 but the interval reaches back to 0.182, so this sample cannot confirm a breach

[review] shap_subgroup_gap
          distance_to_branch_km SHAP contribution gap across region=16,809.671 — 2.693 [2.643, 2.761] at 95% of the mean absolute contribution 6,242.353 (max 50%) — the whole 95% interval sits above 0.500

[review] shap_subgroup_gap
          sum_insured_ngn SHAP contribution gap across gender=1,237.666 — 0.198 [0.008, 0.978] at 95% of the mean absolute contribution 6,242.353 (max 50%) — the estimate stays the right side of 0.500 but the interval reaches 0.978, so this sample cannot rule out a breach

[review] group_mean_gap
          region: mean prediction spans 63,898.46 (Lagos) to 84,990.99 (Kano) — 29.2% of the overall mean 72,238.80 — the whole 95% interval sits above 0.150

[review] error_parity
          region: mean absolute error spans 18,193.09 to 23,186.77 (worst: Kano) — 24.9% of the overall MAE 20,028.72 — the estimate breaches 0.200 but the interval reaches back to 0.132, so this sample cannot confirm a breach

[review] calibration_parity
          region: prediction bias spans 4,224.02 (Abuja, under-predicted) to 13,681.74 (Kano, over-predicted) — 14.8% of the overall actual mean 63,975.19 — the estimate breaches 0.100 but the interval reaches back to 0.079, so this sample cannot confirm a breach

[review] calibration_parity
          gender: prediction bias spans 7,141.16 (M, under-predicted) to 9,787.66 (F, over-predicted) — 4.1% of the overall actual mean 63,975.19 — the estimate stays the right side of 0.100 but the interval reaches 0.113, so this sample cannot rule out a breach

[review] loss_ratio_parity
          region: premium-to-expected-loss ratio spans 1.021 (Lagos) to 1.282 (Kano) — 23.3% of the overall ratio 1.118; Kano carries the higher margin over its own expected cost — the whole 95% interval sits above 0.100

[BLOCK ] actual_vs_expected
          book A/E = 0.886 (11.4% from break-even, over-priced; tolerance 5.0%)

[BLOCK ] actual_vs_expected
          worst band is 5 of 10 (60,098–66,152): A/E = 0.805 over 51 rows — over-priced by 19.5% (tolerance 10.0%)

7. Is the evidence sound?¶

Logically this section comes first. Every number above rests on one assumption — that the validation set is a fair test — and until 0.5.2 nothing checked it.

The distinction the gate now draws:

A performance finding says the model is not good enough. A validation finding says you do not yet know whether it is.

Five checks, all blocking except drift, all reported before anything else. Here they pass, because the split above was built properly. So let us break it three ways on purpose.

In [20]:

# StructuredGateContext is a dataclass, so `replace` gives a variant of it
# with one field swapped — handy for showing what a broken setup looks like.
from dataclasses import replace

for r in report.by_category("validation"):
    print(f"{r.flag:<26} {r.check_name:<20} {r.detail[:88]}")

# StructuredGateContext is a dataclass, so `replace` gives a variant of it # with one field swapped — handy for showing what a broken setup looks like. from dataclasses import replace for r in report.by_category("validation"): print(f"{r.flag:<26} {r.check_name:<20} {r.detail[:88]}")

OK                         target_leakage       no single feature approaches the model's own predictive power (0.720)
OK                         split_overlap        0 duplicate row(s) within the validation set (0.0%, max 5%) — duplicates weight the same
OK                         split_overlap        0 of 514 validation rows (0.0%, max 1%) also appear in X_train, compared over 6 shared c
OK                         validation_strategy  validated with a 'out_of_time' split (out-of-time)
OK                         feature_contract     X matches the 6 feature(s) recorded in model.feature_names_in_, in order
OK                         feature_drift        no drift above threshold across 6 shared feature(s)

Break it 1: the training set as the validation set¶

The oldest mistake, and the one that used to sail straight through. The model has memorised these rows; it will score beautifully.

In [21]:

from bdp_model_gate.structured.validation_checks import SplitOverlapCheck

leaky_split = replace(context, X=X_train, X_train=X_train)
for r in SplitOverlapCheck().run(leaky_split):
    print(f"{r.flag:<24} {r.detail[:100]}")

from bdp_model_gate.structured.validation_checks import SplitOverlapCheck leaky_split = replace(context, X=X_train, X_train=X_train) for r in SplitOverlapCheck().run(leaky_split): print(f"{r.flag:<24} {r.detail[:100]}")

OK                       0 duplicate row(s) within the validation set (0.0%, max 5%) — duplicates weight the same observation
SPLIT_OVERLAP_RISK       1086 of 1086 validation rows (100.0%, max 1%) also appear in X_train, compared over 6 shared column(

Matching is by row content, not index — so resetting an index cannot hide an overlap, and shuffling cannot invent one.

Break it 2: a leaked target¶

settled_amount is the kind of column that gets joined back in by accident: populated only after the claim closes, so it is unavailable at scoring time and nearly determines the answer during training.

In [22]:

from bdp_model_gate.structured.validation_checks import LeakageCheck

X_leaky = X_val.assign(settled_amount=realised_val * 0.98 + rng.normal(0, 500, len(X_val)))
leaked = replace(context, X=X_leaky, X_train=None)

for r in LeakageCheck().run(leaked):
    print(f"{r.flag:<16} {r.detail[:120]}")

from bdp_model_gate.structured.validation_checks import LeakageCheck X_leaky = X_val.assign(settled_amount=realised_val * 0.98 + rng.normal(0, 500, len(X_val))) leaked = replace(context, X=X_leaky, X_train=None) for r in LeakageCheck().run(leaked): print(f"{r.flag:<16} {r.detail[:120]}")

LEAKAGE_RISK     settled_amount alone reaches 1.000 of the target on its own, 139% of the whole model's 0.720 — check it is populated bef

Two conditions had to hold for that to fire: settled_amount reaches leakage_ratio (0.95) of the whole model's own predictive power, and clears leakage_min_power (0.80) in absolute terms.

The second condition is what keeps the check usable. Against a weak model every feature reaches ~100% of its power, and ratio alone would flag the entire frame.

Break it 3: a random split on a pricing model¶

The subtlest of the three, and the one no other tool will say out loud. The data is fine, the split is genuinely random, nothing is duplicated — and it is still the wrong test, because this model will be applied to business written after everything it has seen.

In [23]:

from bdp_model_gate.structured.validation_checks import ValidationStrategyCheck

for strategy in ("out_of_time", "random_split", "holdout", None):
    card = {**model_card}
    if strategy is None:
        card.pop("validation_strategy")
    else:
        card["validation_strategy"] = strategy
    (r,) = ValidationStrategyCheck().run(replace(context, model_card=card))
    print(f"{str(strategy):<14} {r.flag:<28} {r.detail[:82]}")

from bdp_model_gate.structured.validation_checks import ValidationStrategyCheck for strategy in ("out_of_time", "random_split", "holdout", None): card = {**model_card} if strategy is None: card.pop("validation_strategy") else: card["validation_strategy"] = strategy (r,) = ValidationStrategyCheck().run(replace(context, model_card=card)) print(f"{str(strategy):<14} {r.flag:<28} {r.detail[:82]}")

out_of_time    OK                           validated with a 'out_of_time' split (out-of-time)
random_split   VALIDATION_STRATEGY_RISK     pricing is high-risk and was validated with a 'random_split' split — a random spli
holdout        VALIDATION_STRATEGY_RISK     model_card.validation_strategy='holdout' is not a recognised strategy — use one of
None           VALIDATION_STRATEGY_RISK     model_card.validation_strategy missing — state how the holdout was produced, one o

holdout is rejected rather than accepted, which is the point worth pausing on: it sounds like a claim and is not one. "Held out how?" has two very different answers and only one of them survives contact with next quarter.

This is also the one check here that cannot be verified from the data. It records a claim — which is what a model card is for. An untrue answer is a signed statement, not an oversight.

And the fourth thing, which is not a break¶

feature_drift is non-blocking, and deliberately so. An out-of-time holdout should differ a little from training — that is the point of one. A gate that hard-failed on every seasonal shift would be switched off within a month.

In [24]:

from bdp_model_gate.structured.validation_checks import FeatureDriftCheck

for r in FeatureDriftCheck().run(context):
    print(f"{r.flag:<12} blocking={r.blocking}  {r.detail[:96]}")

from bdp_model_gate.structured.validation_checks import FeatureDriftCheck for r in FeatureDriftCheck().run(context): print(f"{r.flag:<12} blocking={r.blocking} {r.detail[:96]}")

OK           blocking=False  no drift above threshold across 6 shared feature(s)

8. From the CLI¶

--expected-loss-col names a column in the data file, so the margin check works from a plain CSV.

In [25]:

import json
import subprocess
import sys
from pathlib import Path

import joblib

workdir = Path("cli_regression")
workdir.mkdir(exist_ok=True)

frame = X_val.copy()
frame["realised_loss"] = realised_val
frame["technical_premium"] = el_val
frame.to_csv(workdir / "validation.csv", index=False)
prot_val.to_csv(workdir / "protected.csv", index=False)
(workdir / "model_card.json").write_text(json.dumps(model_card, indent=2))
joblib.dump(pricer, workdir / "model.joblib")

proc = subprocess.run(
    [
        sys.executable, "-m", "bdp_model_gate.cli",
        "--model", str(workdir / "model.joblib"),
        "--data", str(workdir / "validation.csv"),
        "--target-col", "realised_loss",
        "--protected", str(workdir / "protected.csv"),
        "--model-card", str(workdir / "model_card.json"),
        "--task", "regression",
        "--expected-loss-col", "technical_premium",
        "--metric", "rmse",
        "--max-error", "60000",
        "--output", str(workdir / "gate_report.json"),
    ],
    capture_output=True, text=True,
)
print(proc.stdout or proc.stderr)
print("exit code", proc.returncode, "->",
      {0: "PASS", 1: "BLOCKED", 2: "NEEDS_REVIEW"}.get(proc.returncode))

import json import subprocess import sys from pathlib import Path import joblib workdir = Path("cli_regression") workdir.mkdir(exist_ok=True) frame = X_val.copy() frame["realised_loss"] = realised_val frame["technical_premium"] = el_val frame.to_csv(workdir / "validation.csv", index=False) prot_val.to_csv(workdir / "protected.csv", index=False) (workdir / "model_card.json").write_text(json.dumps(model_card, indent=2)) joblib.dump(pricer, workdir / "model.joblib") proc = subprocess.run( [ sys.executable, "-m", "bdp_model_gate.cli", "--model", str(workdir / "model.joblib"), "--data", str(workdir / "validation.csv"), "--target-col", "realised_loss", "--protected", str(workdir / "protected.csv"), "--model-card", str(workdir / "model_card.json"), "--task", "regression", "--expected-loss-col", "technical_premium", "--metric", "rmse", "--max-error", "60000", "--output", str(workdir / "gate_report.json"), ], capture_output=True, text=True, ) print(proc.stdout or proc.stderr) print("exit code", proc.returncode, "->", {0: "PASS", 1: "BLOCKED", 2: "NEEDS_REVIEW"}.get(proc.returncode))

Gate status: BLOCKED (9253ms, regression)
  rmse: 25570.2945
  validation: 0 flag(s)
  performance: 2 flag(s)
  compliance: 0 flag(s)
  security: 0 flag(s)
  fairness: 10 flag(s)
Full report written to cli_regression/gate_report.json

exit code 1 -> BLOCKED

In [26]:

saved = json.loads((workdir / "gate_report.json").read_text())
ratios = [
    r for r in saved["results_by_category"]["fairness"]
    if r["check_name"] == "loss_ratio_parity"
]
print(json.dumps(ratios, indent=2))

saved = json.loads((workdir / "gate_report.json").read_text()) ratios = [ r for r in saved["results_by_category"]["fairness"] if r["check_name"] == "loss_ratio_parity" ] print(json.dumps(ratios, indent=2))

[
  {
    "check_name": "loss_ratio_parity",
    "flag": "LOSS_RATIO_RISK",
    "detail": "region: premium-to-expected-loss ratio spans 1.021 (Lagos) to 1.282 (Kano) \u2014 23.3% of the overall ratio 1.118; Kano carries the higher margin over its own expected cost \u2014 the whole 95% interval sits above 0.100",
    "blocking": false,
    "metadata": {
      "protected_attr": "region",
      "relative_gap": 0.2331,
      "threshold": 0.1,
      "exposure_weighted": false,
      "group_loss_ratio": {
        "Lagos": 1.021,
        "Kano": 1.2817,
        "Abuja": 1.0277
      },
      "highest_margin_group": "Kano",
      "lowest_margin_group": "Lagos",
      "rows_ignored": 0,
      "on_uncertain": "review",
      "point": 0.2331,
      "ci_low": 0.2251,
      "ci_high": 0.2415,
      "ci_level": 0.95,
      "bootstrap_samples": 1000
    },
    "duration_ms": 185.64
  },
  {
    "check_name": "loss_ratio_parity",
    "flag": "OK",
    "detail": "gender: premium-to-expected-loss ratio spans 1.112 (M) to 1.126 (F) \u2014 1.2% of the overall ratio 1.118; F carries the higher margin over its own expected cost \u2014 the whole interval sits below 0.100",
    "blocking": false,
    "metadata": {
      "protected_attr": "gender",
      "relative_gap": 0.0122,
      "threshold": 0.1,
      "exposure_weighted": false,
      "group_loss_ratio": {
        "M": 1.1121,
        "F": 1.1258
      },
      "highest_margin_group": "F",
      "lowest_margin_group": "M",
      "rows_ignored": 0,
      "on_uncertain": "review",
      "point": 0.0122,
      "ci_low": 0.0009,
      "ci_high": 0.0322,
      "ci_level": 0.95,
      "bootstrap_samples": 1000
    },
    "duration_ms": 185.64
  }
]

In [27]:

import shutil

shutil.rmtree(workdir, ignore_errors=True)
print("cleaned up")

import shutil shutil.rmtree(workdir, ignore_errors=True) print("cleaned up")

cleaned up

Summary¶

Doing regression	Do this
Any continuous target	task="regression"
Premium / severity	metric="rmse" or "mae"; "mape" for skewed money
Claim counts	metric="poisson_deviance" (needs positive predictions)
Error metrics	gate with max_error, not min_score — no default exists
Margin fairness	supply expected_loss to enable LossRatioParityCheck
A model going to production	split out of time, declare it in the model card, and pass X_train

Four things worth carrying away:

1. A higher premium is not unfairness. Risk-based pricing produces different prices by design. LossRatioParityCheck compares each group's premium to its own expected cost, which is the question that separates discrimination from actuarially justified variation.
2. Direction is explicit. rmse gated with min_score would pass every terrible model, so error metrics require max_error and the report names which comparison ran.
3. A random split is the wrong test for a pricing model. It asks whether the model can price a policy it has not seen; the real question is whether it can price next quarter. Inflation, seasonality and mix all leak backwards through a random split, and the gate now requires an out-of-time holdout for high-risk use cases.
4. Pick the metric for the target's shape. rmse on skewed severity measures the tail; mape is undefined on zero claims; poisson_deviance handles counts. The gate will score whatever you ask for — choosing badly is not something it can catch for you.

Next: 02_multiclass_ordinal_sklearn.ipynb for underwriting decisions, or 04_any_framework_classification.ipynb for PyTorch and other non-sklearn models.