import yaml
from azoic.mlops import log_run
portfolio = load_portfolio()
glm_config, gbm_config = make_experiment_configs(str(PORTFOLIO_PATH))
glm_run = run_experiment(glm_config)
gbm_run = run_experiment(gbm_config)
runs = [glm_run, gbm_run]
metrics = comparison_table(runs)6 Reporting, MLflow, and the CLI
Each experiment gets a Markdown model card. A shared standalone HTML comparison dashboard and cross-config comparison table collect the two main fitted models from both experiments. Reports and MLflow use raw run predictions, as in the prediction-scale convention.
This optional operations recipe refits its own candidates. It creates every report it logs; no workbook or fitted model from another chapter is required. For a brief handoff, generate the cards and comparison below and stop there.
glm_card_path = ARTIFACT_DIR / "model_card_tariff_structured.md"
gbm_card_path = ARTIFACT_DIR / "model_card_raw_feature.md"
dashboard_path = ARTIFACT_DIR / "comparison_dashboard.html"
glm_card_path.write_text(model_card(glm_run), encoding="utf-8")
gbm_card_path.write_text(model_card(gbm_run), encoding="utf-8")
dashboard_path.write_text(comparison_dashboard(runs), encoding="utf-8")4860582Verify report artifacts and display raw metrics
assert metrics["model"].nunique() == 2
assert glm_card_path.stat().st_size > 0
assert gbm_card_path.stat().st_size > 0
assert dashboard_path.stat().st_size > 0
print("Held-out comparison — raw rates")
display(metrics[["config", "model", "gini_test", "op_ratio_test", "deviance_test"]])Held-out comparison — raw rates| config | model | gini_test | op_ratio_test | deviance_test | |
|---|---|---|---|---|---|
| 0 | fremtpl2-tariff-structured | direct-tweedie-glm | 0.309850 | 1.022067 | 75.338331 |
| 1 | fremtpl2-raw-feature | tweedie-lightgbm | 0.341982 | 1.178104 | 74.721321 |
6.1 Log runs to local MLflow
MLflow uses a local SQLite backend. Its artifact root is also placed under the tutorial artifact directory. Each log_run call records the data fingerprint, experiment and model parameters, all finite metrics, and the generated artifacts. setup_mlflow_tracking is a tutorial helper in _shared.qmd; log_run is the Azoic API. Only reports written above are attached, so an older tariff workbook cannot be mistaken for an artifact of these new runs.
import mlflow
tracking_uri, experiment_name = setup_mlflow_tracking()
tracking_database = (ARTIFACT_DIR / "mlflow.db").resolve()
artifacts = [glm_card_path, gbm_card_path, dashboard_path]
mlflow_run_ids = [
log_run(
run,
tracking_uri=tracking_uri,
experiment_name=experiment_name,
artifacts=artifacts,
)
for run in runs
]Verify logged metrics and artifacts
client = mlflow.MlflowClient(tracking_uri=tracking_uri)
logged = [client.get_run(run_id) for run_id in mlflow_run_ids]
assert glm_run.data_fingerprint == gbm_run.data_fingerprint
assert all(
entry.data.params["experiment.data_fingerprint"] == glm_run.data_fingerprint
for entry in logged
)
assert all(
f"{model_name}.gini_test" in entry.data.metrics
for entry, run in zip(logged, runs, strict=True)
for model_name in run.models
)
assert all(
{Path(item.path).name for item in client.list_artifacts(run_id, path="artifacts")}
== {Path(path).name for path in artifacts}
for run_id in mlflow_run_ids
)
pd.DataFrame(
{
"run_id": mlflow_run_ids,
"tracking_database": [str(tracking_database.relative_to(ROOT))] * len(mlflow_run_ids),
"artifact_count": [len(artifacts)] * len(mlflow_run_ids),
}
)| run_id | tracking_database | artifact_count | |
|---|---|---|---|
| 0 | 992f11c9696e42f7b99bc56cb633f2da | examples/_artifacts/fremtpl2/mlflow.db | 3 |
| 1 | 7d4754a5f2674fb1a813582a9bc01ab7 | examples/_artifacts/fremtpl2/mlflow.db | 3 |
To inspect the run later, start the UI manually; rendering never starts a server:
uv run mlflow ui \
--backend-store-uri sqlite:///examples/_artifacts/fremtpl2/mlflow.db6.2 Drive the same workflow from the CLI
Save the two experiment configurations as YAML, then run these commands from the repository root. They write into the tutorial artifacts directory. The render executes each command; only its result is shown below the command.
glm_yaml = ARTIFACT_DIR / "glm_config.yaml"
gbm_yaml = ARTIFACT_DIR / "gbm_config.yaml"
glm_yaml.write_text(
yaml.safe_dump(glm_config.model_dump(exclude_none=True)), encoding="utf-8"
)
gbm_yaml.write_text(
yaml.safe_dump(gbm_config.model_dump(exclude_none=True)), encoding="utf-8"
)
assert glm_yaml.stat().st_size > 0 and gbm_yaml.stat().st_size > 0
print("Wrote glm_config.yaml and gbm_config.yaml")Wrote glm_config.yaml and gbm_config.yamluv run azoic profile --data examples/_artifacts/fremtpl2/portfolio_v3.parquet \
--target claim_amount --exposure exposure --claim-count claim_count \
--out examples/_artifacts/fremtpl2/cli_screening.csv column action reason
policy_id bin numeric high cardinality (667673)
exposure bin numeric high cardinality (106)
claim_count keep NaN
claim_amount bin numeric high cardinality (11943)
driver_age bin numeric high cardinality (83)
vehicle_age bin numeric high cardinality (78)
bonus_malus bin numeric high cardinality (115)
density bin numeric high cardinality (1607)
region group categorical high cardinality (22)
vehicle_brand group categorical high cardinality (11)
veh_power group categorical high cardinality (12)
veh_gas keep NaN
area keep NaNuv run azoic fit --config examples/_artifacts/fremtpl2/glm_config.yaml \
--out examples/_artifacts/fremtpl2/cli_model_card.md --quiet# Azoic model card -- fremtpl2-tariff-structured
## Experiment
- name: `fremtpl2-tariff-structured`
- data: `/home/runner/work/azoic/azoic/examples/_artifacts/fremtpl2/portfolio_v3.parquet`
- split: `random` (test_size=0.2, random_state=42)
- target: `claim_amount` exposure: `exposure`
- data fingerprint: `5c765c822ca04a254727eee1fb48074e7cf38537a0e7070cb541ff91eed1c8b1`
- rows: 667673 (train 534138 / test 133535)
- features (9): `driver_age`, `vehicle_age`, `bonus_malus`, `density`, `region`, `vehicle_brand`, `veh_power`, `veh_gas`, `area`
### Model: `direct-tweedie-glm` (glm)
- params: uv run azoic compare examples/_artifacts/fremtpl2/glm_config.yaml \
examples/_artifacts/fremtpl2/gbm_config.yaml \
--out examples/_artifacts/fremtpl2/cli_comparison.csv config model kind gini_train gini_test op_ratio_test deviance_test d2_test
fremtpl2-tariff-structured direct-tweedie-glm glm 0.335709 0.309850 1.022067 75.338331 0.048954
fremtpl2-raw-feature tweedie-lightgbm gbm 0.525721 0.341982 1.178104 74.721321 0.056743Export the direct GLM with --no-recalibrate to preserve its raw prediction scale. The training multiplier shown in scoring remains external to this workbook.
Without this flag, CLI export recalibrates on the stored training rows only, keeping holdout claims out of the base. The earlier raw metrics still describe the unadjusted estimator; evaluate the adjusted workbook separately on the holdout.
uv run azoic export-tariff --config examples/_artifacts/fremtpl2/glm_config.yaml \
--model direct-tweedie-glm --no-recalibrate \
--out examples/_artifacts/fremtpl2/cli_glm_tariff.xlsxDirect GLM workbook — raw rates| base_rate | family | link | intercept_ | recalibrated | |
|---|---|---|---|---|---|
| 0 | 48.05387 | tweedie | log | 5.52521 | False |
6.2.1 Optional: export a distilled GBM
As in optional distillation, create a separate structured teacher by copying the raw GBM’s settings into the GLM’s preprocessing configuration. Save it before running the export command. The CLI fits this teacher and its student on training rows, reserving the test rows for fidelity evaluation.
teacher_config = glm_config.model_copy(update={
"name": "fremtpl2-structured-teacher",
"models": {"tweedie-lightgbm-tariff": gbm_config.models["tweedie-lightgbm"].model_copy(deep=True)},
})
teacher_yaml = ARTIFACT_DIR / "teacher_config.yaml"
teacher_yaml.write_text(
yaml.safe_dump(teacher_config.model_dump(exclude_none=True)), encoding="utf-8"
)978uv run azoic export-tariff --config examples/_artifacts/fremtpl2/teacher_config.yaml \
--model tweedie-lightgbm-tariff --distill --no-recalibrate \
--out examples/_artifacts/fremtpl2/cli_distilled_tariff.xlsxDistilled workbook — raw student rates| base_rate | family | link | intercept_ | recalibrated | distilled_from | teacher_objective | teacher_student_deviance | student_teacher_total_ratio | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 51.015377 | tweedie | log | 5.225353 | False | RiskGBM | tweedie | 0.661562 | 1.003165 |
azoic tune is exercised end-to-end in the tuning chapter, using a bounded trial budget; its full search output is intentionally kept out of this page.
uv run azoic tune --helpUsage: azoic tune [OPTIONS] Tune model hyperparameters with optuna, then write a model card of the best fit. ╭─ Options ────────────────────────────────────────────────────────────────────╮ │ * --config <path> ExperimentConfig YAML. [required] │ │ --trials <int> Optuna trials per model; │ │ overrides YAML (default: YAML or │ │ 20). │ │ --calibration-penalty <float> O/P penalty weight; overrides │ │ YAML (default: YAML or 1.0). │ │ --out <path> Write the model card markdown │ │ here. │ │ --quiet -q Do not print the card to stdout. │ │ --help Show this message and exit. │ ╰──────────────────────────────────────────────────────────────────────────────╯
6.3 Operational boundaries
This book estimates the capped historical pure-premium layer. A deployable price still needs decisions about large losses above the cap, claims development, inflation, expenses, commission, profit, taxes, and business constraints. The local MLflow store and workbook are review artifacts, not a deployed scoring service or policy-system integration. Keep preprocessing, model, prediction scale, and calibration factor together when handing work over.
Once these holdout results inform a change to features, tuning, factors, or calibration, they become development evidence. Evaluate the revised decision on fresh data or a reserved later period before claiming independent performance.