Join Our 5-Week ML/AI Engineer Interview Bootcamp 🚀 led by ML Tech Leads at FAANGs
21. Calibration error
GeneralCompute the Expected Calibration Error (ECE) to measure how well a classifier’s predicted probabilities match actual outcomes. ECE bins predictions by confidence and compares average confidence to empirical accuracy in each bin.
The metric is defined as:
where (S_b) is the set of indices in bin (b), (n) is the total number of samples, (\mathrm{acc}(S_b)) is average correctness, and (\mathrm{conf}(S_b)) is average predicted confidence.
Requirements
Implement the function:
Rules:
- Use equal-width bins over the interval ([0, 1]) (e.g., for 10 bins: [0.0,0.1), …, [0.9,1.0]).
- For each bin, compute accuracy as the fraction of samples with
y_true == (y_prob >= 0.5)(i.e., predicted label uses threshold 0.5). - For each bin, compute confidence as the mean of
y_probin that bin. - Combine bins using the weighted sum shown in the formula (empty bins contribute 0).
- Use only NumPy and Python built-in libraries (no sklearn/scipy).
Example
Output:
| Argument | Type |
|---|---|
| n_bins | int |
| y_prob | np.ndarray |
| y_true | np.ndarray |
| Return Name | Type |
|---|---|
| value | float |
Constraints
Use NumPy only; no sklearn/scipy.
Equal-width bins on [0,1].
Threshold predictions at 0.5 for accuracy.
Hint 1
Start by converting y_true and y_prob to NumPy arrays, then create n_bins+1 equally spaced edges in [0,1] using np.linspace.
Hint 2
Assign each probability to a bin index (0..n_bins-1). Watch the edge case where y_prob == 1.0 should fall into the last bin (not overflow). np.digitize + clamping works well.
Hint 3
For each bin: compute count, acc = mean(y_true == (y_prob>=0.5)), and conf = mean(y_prob). Accumulate ece += (count/n) * abs(acc-conf); skip empty bins.
Roles
Companies
GeneralLevels
Tags
Python Editor