73. Calibration error
GeneralCompute the Expected Calibration Error (ECE) to measure how well a classifier’s predicted probabilities match actual outcomes. ECE bins predictions by confidence and compares average confidence to empirical accuracy in each bin.
The metric is defined as:
where is the set of indices in bin , is the total number of samples, is average correctness, and is average predicted confidence.
Requirements
Implement the function:
def expected_calibration_error(y_true, y_prob, n_bins=10):
"""
Computes Expected Calibration Error (ECE) for binary classification.
Args:
y_true (np.ndarray): True labels (0 or 1), length n.
y_prob (np.ndarray): Predicted probabilities for class 1, length n.
n_bins (int): Number of equal-width confidence bins in [0, 1].
Returns:
float: The Expected Calibration Error (ECE).
"""
# Your solution belowRules:
- Use equal-width bins over the interval (e.g., for 10 bins: [0.0,0.1), …, [0.9,1.0]).
- For each bin, compute accuracy as the fraction of samples with
y_true == (y_prob >= 0.5)(i.e., predicted label uses threshold 0.5). - For each bin, compute confidence as the mean of
y_probin that bin. - Combine bins using the weighted sum shown in the formula (empty bins contribute 0).
- Use only NumPy and Python built-in libraries (no sklearn/scipy).
Example
y_true = [1, 0, 1, 1, 0]
y_prob = [0.9, 0.6, 0.2, 0.8, 0.1]
expected_calibration_error(y_true, y_prob, n_bins=5)Output:
0.28- n_bins
- int
- y_prob
- np.ndarray
- y_true
- np.ndarray
- value
- float
Examples
{
"n_bins": 5,
"y_prob": [
0.9,
0.6,
0.2,
0.8,
0.1
],
"y_true": [
1,
0,
1,
1,
0
]
}0.4| Argument | Type |
|---|---|
| n_bins | int |
| y_prob | np.ndarray |
| y_true | np.ndarray |
| Return Name | Type |
|---|---|
| value | float |
Constraints
Use NumPy only; no sklearn/scipy.
Equal-width bins on [0,1].
Threshold predictions at 0.5 for accuracy.
Hint 1
Start by converting y_true and y_prob to NumPy arrays, then create n_bins+1 equally spaced edges in [0,1] using np.linspace.
Hint 2
Assign each probability to a bin index (0..n_bins-1). Watch the edge case where y_prob == 1.0 should fall into the last bin (not overflow). np.digitize + clamping works well.
Hint 3
For each bin: compute count, acc = mean(y_true == (y_prob>=0.5)), and conf = mean(y_prob). Accumulate ece += (count/n) * abs(acc-conf); skip empty bins.
Roles
Companies
General