Skip to content

Data profiling and preprocessing

Pricing preprocessing should make a portfolio easier to reason about without hiding the learned tariff structure. Azoic separates cheap feature screening from fitted binning and grouping so an actuary can inspect every mapping.

Profile before fitting

profile_features returns one row per column. screen_features adds a rule-based recommendation; it does not silently drop anything.

from azoic.profile import profile_features, screen_features

profile = profile_features(portfolio)
screening = screen_features(
    profile,
    max_groups=10,
    max_numeric_unique=20,
    missing_drop=0.5,
)

print(screening[["column", "action", "reason"]].to_string(index=False))

The actions are keep, bin, group, and drop. Treat them as a review queue: business meaning, availability at quote time, and leakage risk still override a cardinality heuristic.

Boolean and nullable-boolean columns are discrete features: their profiles include missingness and cardinality, without continuous numeric statistics.

Learn transparent mappings

AutoBinner handles numeric features. Quantile binning balances row count or exposure; tree binning learns target-aware cut points. Credibility floors can merge bins with insufficient exposure or claim count. Intervals include their finite lower edge and exclude their upper edge: with edges 30 and 60, the bins are (-inf, 30.0), [30.0, 60.0), and [60.0, inf). A value of exactly 30 belongs to the second bin; exactly 60 belongs to the third. With no edges, all observed values belong to (-inf, inf); missing values still use Missing. Monotonic pooling uses only occupied bins. With monotonic pooling enabled, constant predictors produce a single observed-value interval.

AutoGrouper handles categorical features. The rare strategy collapses thin levels; similarity stably sorts nominal levels by aggregate pure premium (sum(claim_amount) / sum(exposure)). Ordered categoricals retain their declared order. Groups below an exposure or claim-count floor merge with their closest adjacent risk. The closest adjacent pair then merges until max_groups is met, recomputing aggregate risks after each merge and breaking distance ties to the left. For equal exposures, rates 100, 101, and 10000 with max_groups=2 pool 100 and 101, leaving 10000 separate. If the whole portfolio cannot meet a floor, only one group remains.

from sklearn.pipeline import Pipeline

from azoic.preprocessing import AutoBinner, AutoGrouper

preprocessor = Pipeline(
    [
        (
            "binner",
            AutoBinner(
                cols=["driver_age", "vehicle_age"],
                strategy="tree",
                max_bins=6,
                min_exposure=50,
                min_claims=5,
                exposure_col="exposure",
                claim_count_col="claim_count",
                target_col="claim_amount",
                random_state=42,
            ),
        ),
        (
            "grouper",
            AutoGrouper(
                cols=["region", "vehicle_brand"],
                strategy="similarity",
                max_groups=6,
                min_exposure=50,
                min_claims=5,
                exposure_col="exposure",
                claim_count_col="claim_count",
                target_col="claim_amount",
            ),
        ),
    ]
)

prepared = preprocessor.fit_transform(portfolio)
print(preprocessor.named_steps["binner"].mapping_)
print(preprocessor.named_steps["grouper"].mapping_)

Configured exposure and claim-count columns must exist during fitting and contain finite numeric values. Each exposure must be at least one day (1/366, with a 0.1% day-count tolerance); claim counts must be non-negative. Omitting exposure_col explicitly selects unweighted binning or unit row weights for grouping. A missing target_col falls back to y: rates stay rates for tree binning and are multiplied by exposure before similarity grouping pools them. Supplied outcomes must remain finite after numeric and unit conversion. NaN, infinity, and entirely missing outcomes raise an error before mappings are learned, even when the corresponding predictor is missing. Unknown outcomes cannot count as zero claims.

Transform input must contain every fitted mapping column. DataFrames use their column names; arrays use x0, x1, and so on. Switching between named DataFrames and arrays raises ValueError when those names do not match. Keep the fitted column names when scoring so each learned mapping is applied.

Both transformers expose mapping_ and accept set_mapping(...) after fitting. An override replaces the entire mapping; omitted columns pass through unchanged. Keys must name currently binned or grouped columns. Bin edges must be finite, one-dimensional, and strictly increasing; an empty edge vector selects one bin. Invalid overrides leave mappings and category vocabularies unchanged. Missing numeric values get a stable Missing bin. Unknown categorical levels map to other_label during general preprocessing; tariff application is stricter and rejects them.

Fit preprocessing inside the training boundary

Tree binning and similarity grouping use outcomes. Fit them only on training data. In run_experiment they live inside each model pipeline, and outcome columns are removed before the final model sees features.

Configure the same steps in YAML

preprocessing:
  binner:
    cols: [driver_age, vehicle_age]
    strategy: tree
    max_bins: 6
    min_exposure: 50
    min_claims: 5
  grouper:
    cols: [region, vehicle_brand]
    strategy: similarity
    max_groups: 6
    min_exposure: 50
    min_claims: 5

Do not repeat exposure_col, claim_count_col, or target_col in YAML. run_experiment derives them from DatasetSpec.

Choose and fit models See every preprocessing setting