ember

7Calibration

A reliability diagram plots each confidence bin's accuracy against its mean confidence; a perfectly calibrated advisor sits on the diagonal [11]. Bars above the diagonal mean ember was more often right than its confidence claimed (under-confidence); bars below it mean over-confidence, the dangerous direction for threshold-driven agents.

Reliability of choice answersAccuracy bars per confidence bin against the diagonal of perfect calibration.accuracymean confidenceperfect calibration0.00.00.20.20.40.40.60.60.80.81.01.0Top-label confidenceAccuracyn=4n=9n=12n=10n=19n=34n=104ECE 0.100 MCE 0.286
Figure 5 Reliability of choice answers (192 questions). Coral lines mark each bin's mean confidence; shading below a line marks over-confidence.
Reliability of noul answersAccuracy bars per confidence bin against the diagonal of perfect calibration.accuracymean confidenceperfect calibration0.00.00.20.20.40.40.60.60.80.81.01.0Top-label confidenceAccuracyn=13n=13n=31n=49n=62ECE 0.100 MCE 0.185
Figure 6 Reliability of noul answers (168 questions). Coral lines mark each bin's mean confidence; shading below a line marks over-confidence.
TypeNMean confidenceAccuracyAccuracy minus confidenceECEMCEBrier
choice1920.83893.8%+0.099 (under-confident)0.1000.2860.115
noul1680.82891.7%+0.088 (under-confident)0.1000.1850.066