Data and preparation API
Use these APIs to validate a Parquet data contract, review feature quality, learn inspectable mappings, and create leakage-safe split labels. The data profiling and preprocessing guide shows the task flow before the generated signatures below.
| Module | Public API | Purpose |
|---|---|---|
azoic.data |
DatasetSpec, load_data |
Special-column contract; Parquet loading with contract validation |
azoic.profile |
profile_features, screen_features |
Per-column screening stats; recommended keep/drop/group actions |
azoic.preprocessing |
AutoBinner, AutoGrouper |
Inspectable numeric binning and categorical grouping transformers |
azoic.validation |
make_strata, temporal_split |
Exposure-balanced stratum labels; leakage-safe temporal split |
Dataset specifications and loaders.
pandas / pyarrow at boundaries. S3 paths work out of the box via fsspec when
the optional aws extra is installed (uv sync --extra aws).
EXPOSURE_FLOOR = MIN_EXPOSURE * (1.0 - 0.001)
module-attribute
Validation threshold: one day minus a 0.1% relative tolerance for day-count representations (freMTPL2 stores one-day policies ~163 ulps below 1/366).
MIN_EXPOSURE = 1.0 / 366.0
module-attribute
Minimum per-row exposure: one day in year fractions (leap-year denominator).
Sub-day rows are data errors for pure-premium modelling: a tiny exposure
inflates claim_amount / exposure rates and destabilizes GLM fits.
DatasetSpec
Bases: BaseModel
Names of the special (non-feature) columns in a pricing dataset.
Feature columns are everything else. .target and .exposure are required;
the rest are optional. Unknown fields are rejected.
load_data(path, spec=None, **kwargs)
Read a Parquet dataset (local or s3://) into a pandas DataFrame.
When spec is given, required columns must be present or ValueError.
Extra kwargs forward to pandas.read_parquet.
Feature profiling and screening (actuary-in-the-loop, before modelling).
profile_features returns one row per column with the cheap stats needed
for screening (missingness, cardinality, zero-variance, dtype, basic numerics).
Per-level actuarial stats live on the binning/grouping transformers' mapping_,
not here -- profiling is for the column-level keep/drop/review decision.
screen_features turns a profile into per-column actions:
keep / bin (numeric high cardinality) / group (categorical high
cardinality) / drop (zero variance or excessive missingness).
profile_features(df)
One row per column with screening-relevant stats.
screen_features(profile, *, max_groups=10, max_numeric_unique=20, missing_drop=0.5)
Per-column recommended action with a one-line reason.
Rules
zero_variance -> drop missing_rate > missing_drop -> drop numeric & n_unique > max_numeric_unique -> bin categorical & n_unique > max_groups -> group else -> keep
Auto-binning and auto-grouping transformers for tariff preprocessing.
numeric columns -> actuarially-credible interval bins
("quantile" equal-exposure edges, or "tree" decision-tree edges on y).
AutoGrouper: categorical columns -> credibility-stable level groups ("rare" floor-based, or "similarity" greedy 1D merging on pure premium).
Both follow the scikit-learn transformer API and survive
sklearn.utils.estimator_checks.parametrize_with_checks. Special columns
(exposure, claim_count, target) travel inside X per AGENTS.md rule 8.
mapping_ is overridable via set_mapping; transform honours the override.
AutoBinner
Bases: TransformerMixin, BaseEstimator
Bin numeric columns into actuarially-credible tariff bins.
Binned columns become ordered interval categoricals (DataFrame input) or
integer bin codes (ndarray input). Non-target columns pass through
unchanged. Finite bin edges are lower-inclusive and upper-exclusive.
NaN is assigned to a reserved "Missing" category.
Transform input must contain every fitted mapping column; arrays use
x0, x1, etc. Unmatched column names raise ValueError.
Parameters
cols : list[str] | None
Columns to bin. None -> all numeric columns found in fit.
strategy : {"quantile", "tree"}
"quantile" = (exposure-)weighted equal-frequency edges.
"tree" = DecisionTreeRegressor(max_leaf_nodes=max_bins) on the target.
max_bins : int
Maximum number of bins per column (>=2).
min_exposure : float | None
Minimum total exposure per bin; smaller bins merged into neighbours.
Requires a real exposure_col column and must be >= 1/366 (one
day); raises otherwise (a count-unit fallback would change units).
min_claims : float | None
Minimum aggregate claim count per bin; smaller bins merge into neighbours.
exposure_col, claim_count_col, target_col : str | None
Configured exposure and claim-count columns must exist at fit and
contain finite values; exposures must be at least one day (1/366,
with day-count tolerance), and claim counts must be non-negative.
Without exposure_col, binning is unweighted. For tree binning, the target is
target_col / exposure_col (pure premium) when both are set, else
target_col, else y. Supplied outcomes must be finite after
numeric and unit conversion; missing outcomes are never treated as zero.
monotonic : False | True | "increasing" | "decreasing"
When set, bin means are smoothed to be monotonic in the target via
isotonic_regression and adjacent bins with the same smoothed mean
merge. No-op without a target. False = off, True / "increasing" =
non-decreasing bin means, "decreasing" = non-increasing. Applied
after the strategy and before the min_exposure small-bin merge.
random_state : int
Tree strategy reproducibility.
Attributes
mapping_ : dict[str, np.ndarray] Bin edges per binned column (length n_bins - 1). category_dtypes_ : dict[str, pd.CategoricalDtype] Stable ordered output vocabularies for DataFrame inputs. bin_cols_ : list[str] n_features_in_ : int feature_names_in_ : list[str] | None
set_mapping(mapping)
Replace fitted bin mappings with finite, strictly increasing edge vectors.
Keys must name currently binned columns. Omitted columns pass through. Invalid overrides leave the fitted mappings and vocabularies intact.
AutoGrouper
Bases: TransformerMixin, BaseEstimator
Group categorical levels into credibility-stable groups.
Strategies
"rare" Levels below min_exposure / min_claims -> other_label.
min_exposure requires a real exposure_col column and
must be >= 1/366 (one day).
"similarity" Nominal levels are stably risk-sorted; ordered categoricals
retain their declared order. Under-credible groups merge
with their nearest-risk neighbour, then the closest adjacent
pairs merge until max_groups is met. Risks are aggregate
claims / exposure, recomputed after each merge; ties go left.
mapping_ is {col: {original_level: group_label}}. Unknown levels at
transform time map to other_label. Transform input must contain every
fitted mapping column; arrays use x0, x1, etc. Unmatched column names
raise ValueError.
Configured exposure and claim-count columns must exist at fit and contain
finite values; exposures must be at least one day (1/366, with day-count
tolerance), and claim counts must be non-negative. Without exposure_col,
grouping uses unit row weights. target_col contains aggregate claims;
when absent, y supplies rates, multiplied by exposure before pooling.
Supplied outcomes must be finite after numeric and unit conversion;
missing outcomes are never treated as zero.
Attributes
mapping_, category_dtypes_, group_cols_, n_features_in_, feature_names_in_.
set_mapping(mapping)
Replace fitted groups: {col: {level: group_label}}.
Keys must name currently grouped columns. Omitted columns pass through. Invalid overrides leave the fitted mappings and vocabularies intact.
Stratum labels for sklearn stratifiers and a tie-safe temporal holdout.
make_strata(y, sample_weight=None, *, n_strata=10)
Integer stratum labels for a continuous y, for sklearn's stratifiers.
sklearn's StratifiedKFold / StratifiedGroupKFold stratify on
discrete classes; pure-premium / claim-count targets are continuous.
Discretize into n_strata quantiles of y -- exposure-weighted when
sample_weight is given (each fold then carries similar portfolio
adequacy, the actuarial CV convention) -- and feed the codes back to the
splitter as the stratification target. Missing values and their exposure
do not affect weighted boundaries. Weights must be finite and non-negative;
zero total observed weight assigns observed rows to group 0.
Returns
ndarray[int]
Integer codes (-1 for NaN rows); distinct positive codes count
<= n_strata when y has fewer unique values. The unweighted
path inherits pandas qcut behaviour of skipping the indices of
dropped duplicate edges, so the label set need not be 0..k-1
contiguous -- sklearn stratifiers only need distinct labels. Drop
-1 rows or remap to 0 before passing to a splitter that
rejects negative labels.
Constant unweighted observations belong to group 0; missing rows
still receive -1. Weighted tie semantics are unchanged.
temporal_split(df, time_col, *, test_size=None, cutoff=None)
Return positional (train_idx, test_idx) for a single temporal holdout.
Rows are sorted by time_col (ascending) regardless of input order, so
the train block precedes the test block in time -- no leakage. Pass exactly
one of:
test_size
Fraction in (0, 1) or an int selects the latest rows for test. If the
boundary falls inside a timestamp tie, the whole tied block goes to test.
cutoff
Timestamp / scalar comparable to df[time_col]. Rows with
time <= cutoff go to train; time > cutoff go to test.
Returns
(ndarray[int], ndarray[int])
Positional row indices into the input df (use df.iloc[...]).
returns positional indices only -- a single holdout does not need
a sklearn splitter; for walk-forward OOT use TimeSeriesSplit.