The frequency and severity datasets are pinned by OpenML data IDs 41214 and 41215. They are fetched through sklearn.datasets.fetch_openml, following the scikit-learn insurance example.
Cleaning is deliberately deterministic and conservative:
cap individual claim amounts at 100,000 (large-loss stability; the tariff prices the retained layer);
aggregate severity rows to policy-level claim amount and left-join by policy ID;
retain unique policies with \(0 < exposure \le 1\), non-negative outcomes, positive density, plausible ages, bonus-malus in \([50, 230]\), vehicle power in \([2, 15]\), and consistent zero/positive claim count and amount;
rename only the columns used by Azoic and derive no outcome-informed features.
The cap applies to each claim before aggregation. A policy with two capped claims can therefore have a claim_amount above the cap. Costs above that layer need a separate pricing decision. The live tables below separate this cap effect from policy exclusions, then describe the sample actually used for modelling.
2.1 Retained columns
The CASdatasets field descriptions explain the source encodings. The retained columns are renamed as follows. Only the nine predictors enter model fitting; the identifier and outcomes are not rating inputs.
Column
Meaning and unit or encoding
Role
policy_id
Source IDpol; policy identifier, not a time field
Identifier
exposure
Observed fraction of a policy-year; retained range \((0, 1]\)
Exposure (rate denominator and weight)
claim_count
Number of claims during the observed period (ClaimNb)
Outcome
claim_amount
Policy aggregate cost: sum of individually capped claims, in euros; zero when no severity row matches
Outcome
driver_age
Main driver’s age in years (DrivAge), retained 18–100
Predictor
vehicle_age
Vehicle age in years (VehAge), retained 0–100
Predictor
bonus_malus
Bonus-malus index (BonusMalus); 100 is the reference, lower means a bonus, higher a penalty; retained 50–230
Predictor
density
Inhabitants per square kilometre in the driver’s city (Density), positive
Predictor
region
Source French region code (Region), treated as a category
Predictor
vehicle_brand
Source vehicle-brand category code (VehBrand), not a decoded brand name
Predictor
veh_power
Source ordered vehicle-power category (VehPower), retained 2–15 and converted to a string for grouping
Predictor
veh_gas
Fuel category (VehGas): diesel or regular
Predictor
area
Source density-area category (Area), A–F; retained as a category
Predictor
Missing region, vehicle_brand, veh_gas, or area values become the literal category "Missing". Missing numeric rating values fail their range or positivity rule. For pure-premium fitting, the target rate is claim_amount / exposure (euros per policy-year), weighted by exposure.
2.2 Cache and sample controls
load_portfolio is a tutorial helper defined in examples/_shared.qmd, where CLAIM_CAP, SAMPLE_SIZE, and RANDOM_STATE are set. SAMPLE_SIZE caps the cleaned policies used; the default exceeds the portfolio, so it uses all rows. Tuning and protected-group recipes use at most 100,000 of the available rows. Smaller samples change the evidence and may leave too few claims for a useful fit.
The cleaned cache does not detect changes to CLAIM_CAP or cleaning rules. After changing either, delete examples/_artifacts/fremtpl2/fremtpl2_joined_v3.parquetandexamples/_artifacts/fremtpl2/cleaning_audit_v1.csv before rerunning. Keep openml-cache/ to reuse the raw download. If either derived cache is missing, the loader rebuilds both from those raw inputs; it never reconstructs exclusions from already-cleaned policies. Each call regenerates portfolio_v3.parquet from the cleaned cache using the current sample settings; changing only sample size or seed does not require deleting the cleaned cache. Rerender affected chapters to regenerate matching reports and workbooks.
These totals cover every row in the pinned severity input, before the policy join, exclusions, or sampling. The exclusion audit starts from the left-joined frequency policies, so unmatched severity IDs do not contribute to that table. The configured cap is stored with the audit.
Read the rules from top to bottom. Each removed policy is attributed to its first failing rule, so overlapping problems count once. At each step, previous retained totals equal removed plus newly retained totals. The first row is the joined population; the last is the cleaned population before sampling. The default pinned inputs retain 667,673 policies.
Exposure is in policy-years; claim amounts are policy aggregates of capped claims in euros. Missing numeric values contribute no amount to sums, while those rows still count as policies. Negative invalid values remain signed in removed totals. The one-to-one join rejects duplicate source policy IDs before these rules; the ID rule also catches missing IDs or duplicates after numeric conversion. Zero/positive count–amount inconsistencies are excluded, not repaired.
Policies removed
Policies retained
Exposure removed
Exposure retained
rule
Joined policies before exclusions
0
678,013
0.00
358,499.45
Missing or duplicate policy ID
0
678,013
0.00
358,499.45
Exposure outside (0, 1]
1,224
676,789
1,363.34
357,136.11
Negative or missing claim count
0
676,789
0.00
357,136.11
Negative claim amount
0
676,789
0.00
357,136.11
Driver age outside [18, 100]
0
676,789
0.00
357,136.11
Vehicle age outside [0, 100]
0
676,789
0.00
357,136.11
Bonus-malus outside [50, 230]
0
676,789
0.00
357,136.11
Vehicle power outside [2, 15]
0
676,789
0.00
357,136.11
Non-positive or missing density
0
676,789
0.00
357,136.11
Inconsistent count and amount
9,116
667,673
4,605.46
352,530.65
Claims removed
Claims retained
EUR removed
EUR retained
rule
Joined policies before exclusions
0
36,102
0.00
49,302,196.05
Missing or duplicate policy ID
0
36,102
0.00
49,302,196.05
Exposure outside (0, 1]
54
36,048
87,927.38
49,214,268.67
Negative or missing claim count
0
36,048
0.00
49,214,268.67
Negative claim amount
0
36,048
0.00
49,214,268.67
Driver age outside [18, 100]
0
36,048
0.00
49,214,268.67
Vehicle age outside [0, 100]
0
36,048
0.00
49,214,268.67
Bonus-malus outside [50, 230]
0
36,048
0.00
49,214,268.67
Vehicle power outside [2, 15]
0
36,048
0.00
49,214,268.67
Non-positive or missing density
0
36,048
0.00
49,214,268.67
Inconsistent count and amount
9,657
26,391
0.00
49,214,268.67
2.5 Portfolio used in this render
This summary describes the actual SAMPLE_SIZE/RANDOM_STATE selection from the cleaned policies. Reducing the sample changes these totals, not the pre-sampling audit above.
2.6 Descriptive review before learned preprocessing
Profiling and screening are descriptive review aids. They do not learn tariff relativities and do not replace actuarial judgement. Binning and grouping are fitted in the next chapter, using training rows only.