ember

7Calibration

A reliability diagram plots each confidence bin's accuracy against its mean confidence; a perfectly calibrated advisor sits on the diagonal [11]. Bars above the diagonal mean ember was more often right than its confidence claimed (under-confidence); bars below it mean over-confidence, the dangerous direction for threshold-driven agents.

Reliability of choice answersAccuracy bars per confidence bin against the diagonal of perfect calibration.accuracymean confidenceperfect calibration0.00.00.20.20.40.40.60.60.80.81.01.0Top-label confidenceAccuracyn=5n=16n=21n=18n=39n=79n=181ECE 0.083 MCE 0.217
Figure 6 Reliability of choice answers (359 questions). Coral lines mark each bin's mean confidence; shading below a line marks over-confidence.
Reliability of noul answersAccuracy bars per confidence bin against the diagonal of perfect calibration.accuracymean confidenceperfect calibration0.00.00.20.20.40.40.60.60.80.81.01.0Top-label confidenceAccuracyn=25n=28n=60n=90n=105ECE 0.095 MCE 0.132
Figure 7 Reliability of noul answers (308 questions). Coral lines mark each bin's mean confidence; shading below a line marks over-confidence.
TypeNMean confidenceAccuracyAccuracy minus confidenceECEMCEBrier
choice3590.83992.2%+0.083 (under-confident)0.0830.2170.138
noul3080.82490.3%+0.078 (under-confident)0.0950.1320.073