Skip to content

Data and preparation API

Use these APIs to validate a Parquet data contract, review feature quality, learn inspectable mappings, and create leakage-safe split labels. The data profiling and preprocessing guide shows the task flow before the generated signatures below.

Module Public API Purpose
azoic.data DatasetSpec, load_data Special-column contract; Parquet loading with contract validation
azoic.profile profile_features, screen_features Per-column screening stats; recommended keep/drop/group actions
azoic.preprocessing AutoBinner, AutoGrouper Inspectable numeric binning and categorical grouping transformers
azoic.validation make_strata, temporal_split Exposure-balanced stratum labels; leakage-safe temporal split

Dataset specifications and loaders.

pandas / pyarrow at boundaries. S3 paths work out of the box via fsspec when the optional aws extra is installed (uv sync --extra aws).

EXPOSURE_FLOOR = MIN_EXPOSURE * (1.0 - 0.001) module-attribute

Validation threshold: one day minus a 0.1% relative tolerance for day-count representations (freMTPL2 stores one-day policies ~163 ulps below 1/366).

MIN_EXPOSURE = 1.0 / 366.0 module-attribute

Minimum per-row exposure: one day in year fractions (leap-year denominator).

Sub-day rows are data errors for pure-premium modelling: a tiny exposure inflates claim_amount / exposure rates and destabilizes GLM fits.

DatasetSpec

Bases: BaseModel

Names of the special (non-feature) columns in a pricing dataset.

Feature columns are everything else. .target and .exposure are required; the rest are optional. Unknown fields are rejected.

load_data(path, spec=None, **kwargs)

Read a Parquet dataset (local or s3://) into a pandas DataFrame.

When spec is given, required columns must be present or ValueError. Extra kwargs forward to pandas.read_parquet.

Feature profiling and screening (actuary-in-the-loop, before modelling).

profile_features returns one row per column with the cheap stats needed for screening (missingness, cardinality, zero-variance, dtype, basic numerics). Per-level actuarial stats live on the binning/grouping transformers' mapping_, not here -- profiling is for the column-level keep/drop/review decision.

screen_features turns a profile into per-column actions: keep / bin (numeric high cardinality) / group (categorical high cardinality) / drop (zero variance or excessive missingness).

profile_features(df)

One row per column with screening-relevant stats.

screen_features(profile, *, max_groups=10, max_numeric_unique=20, missing_drop=0.5)

Per-column recommended action with a one-line reason.

Rules

zero_variance -> drop missing_rate > missing_drop -> drop numeric & n_unique > max_numeric_unique -> bin categorical & n_unique > max_groups -> group else -> keep

Auto-binning and auto-grouping transformers for tariff preprocessing.

numeric columns -> actuarially-credible interval bins

("quantile" equal-exposure edges, or "tree" decision-tree edges on y).

AutoGrouper: categorical columns -> credibility-stable level groups ("rare" floor-based, or "similarity" greedy 1D merging on pure premium).

Both follow the scikit-learn transformer API and survive sklearn.utils.estimator_checks.parametrize_with_checks. Special columns (exposure, claim_count, target) travel inside X per AGENTS.md rule 8. mapping_ is overridable via set_mapping; transform honours the override.

AutoBinner

Bases: TransformerMixin, BaseEstimator

Bin numeric columns into actuarially-credible tariff bins.

Binned columns become ordered interval categoricals (DataFrame input) or integer bin codes (ndarray input). Non-target columns pass through unchanged. Finite bin edges are lower-inclusive and upper-exclusive. NaN is assigned to a reserved "Missing" category. Transform input must contain every fitted mapping column; arrays use x0, x1, etc. Unmatched column names raise ValueError.

Parameters

cols : list[str] | None Columns to bin. None -> all numeric columns found in fit. strategy : {"quantile", "tree"} "quantile" = (exposure-)weighted equal-frequency edges. "tree" = DecisionTreeRegressor(max_leaf_nodes=max_bins) on the target. max_bins : int Maximum number of bins per column (>=2). min_exposure : float | None Minimum total exposure per bin; smaller bins merged into neighbours. Requires a real exposure_col column and must be >= 1/366 (one day); raises otherwise (a count-unit fallback would change units). min_claims : float | None Minimum aggregate claim count per bin; smaller bins merge into neighbours. exposure_col, claim_count_col, target_col : str | None Configured exposure and claim-count columns must exist at fit and contain finite values; exposures must be at least one day (1/366, with day-count tolerance), and claim counts must be non-negative. Without exposure_col, binning is unweighted. For tree binning, the target is target_col / exposure_col (pure premium) when both are set, else target_col, else y. Supplied outcomes must be finite after numeric and unit conversion; missing outcomes are never treated as zero. monotonic : False | True | "increasing" | "decreasing" When set, bin means are smoothed to be monotonic in the target via isotonic_regression and adjacent bins with the same smoothed mean merge. No-op without a target. False = off, True / "increasing" = non-decreasing bin means, "decreasing" = non-increasing. Applied after the strategy and before the min_exposure small-bin merge. random_state : int Tree strategy reproducibility.

Attributes

mapping_ : dict[str, np.ndarray] Bin edges per binned column (length n_bins - 1). category_dtypes_ : dict[str, pd.CategoricalDtype] Stable ordered output vocabularies for DataFrame inputs. bin_cols_ : list[str] n_features_in_ : int feature_names_in_ : list[str] | None

set_mapping(mapping)

Replace fitted bin mappings with finite, strictly increasing edge vectors.

Keys must name currently binned columns. Omitted columns pass through. Invalid overrides leave the fitted mappings and vocabularies intact.

AutoGrouper

Bases: TransformerMixin, BaseEstimator

Group categorical levels into credibility-stable groups.

Strategies

"rare" Levels below min_exposure / min_claims -> other_label. min_exposure requires a real exposure_col column and must be >= 1/366 (one day). "similarity" Nominal levels are stably risk-sorted; ordered categoricals retain their declared order. Under-credible groups merge with their nearest-risk neighbour, then the closest adjacent pairs merge until max_groups is met. Risks are aggregate claims / exposure, recomputed after each merge; ties go left.

mapping_ is {col: {original_level: group_label}}. Unknown levels at transform time map to other_label. Transform input must contain every fitted mapping column; arrays use x0, x1, etc. Unmatched column names raise ValueError.

Configured exposure and claim-count columns must exist at fit and contain finite values; exposures must be at least one day (1/366, with day-count tolerance), and claim counts must be non-negative. Without exposure_col, grouping uses unit row weights. target_col contains aggregate claims; when absent, y supplies rates, multiplied by exposure before pooling. Supplied outcomes must be finite after numeric and unit conversion; missing outcomes are never treated as zero.

Attributes

mapping_, category_dtypes_, group_cols_, n_features_in_, feature_names_in_.

set_mapping(mapping)

Replace fitted groups: {col: {level: group_label}}.

Keys must name currently grouped columns. Omitted columns pass through. Invalid overrides leave the fitted mappings and vocabularies intact.

Stratum labels for sklearn stratifiers and a tie-safe temporal holdout.

make_strata(y, sample_weight=None, *, n_strata=10)

Integer stratum labels for a continuous y, for sklearn's stratifiers.

sklearn's StratifiedKFold / StratifiedGroupKFold stratify on discrete classes; pure-premium / claim-count targets are continuous. Discretize into n_strata quantiles of y -- exposure-weighted when sample_weight is given (each fold then carries similar portfolio adequacy, the actuarial CV convention) -- and feed the codes back to the splitter as the stratification target. Missing values and their exposure do not affect weighted boundaries. Weights must be finite and non-negative; zero total observed weight assigns observed rows to group 0.

Returns

ndarray[int] Integer codes (-1 for NaN rows); distinct positive codes count <= n_strata when y has fewer unique values. The unweighted path inherits pandas qcut behaviour of skipping the indices of dropped duplicate edges, so the label set need not be 0..k-1 contiguous -- sklearn stratifiers only need distinct labels. Drop -1 rows or remap to 0 before passing to a splitter that rejects negative labels. Constant unweighted observations belong to group 0; missing rows still receive -1. Weighted tie semantics are unchanged.

temporal_split(df, time_col, *, test_size=None, cutoff=None)

Return positional (train_idx, test_idx) for a single temporal holdout.

Rows are sorted by time_col (ascending) regardless of input order, so the train block precedes the test block in time -- no leakage. Pass exactly one of:

test_size Fraction in (0, 1) or an int selects the latest rows for test. If the boundary falls inside a timestamp tie, the whole tied block goes to test. cutoff Timestamp / scalar comparable to df[time_col]. Rows with time <= cutoff go to train; time > cutoff go to test.

Returns

(ndarray[int], ndarray[int]) Positional row indices into the input df (use df.iloc[...]).

returns positional indices only -- a single holdout does not need

a sklearn splitter; for walk-forward OOT use TimeSeriesSplit.