ember

8Decisions under the agent kit

Confidence bands

The agent kit maps every answer to an action by its confidence. These tables replay that mapping on every answer in the run.

choice bandRuleAgent actionAnswersShareCorrectAccuracy
trustconfidence >= 0.85act on it413.3%250.0%
verify0.60 <= confidence < 0.85act, say so, and verify1343.3%1076.9%
deferconfidence < 0.60gather evidence or ask the user1343.3%538.5%
noul bandRuleAgent actionAnswersShareCorrectAccuracy
yesP >= 0.80treat as yes00.0%0n/a
noP <= 0.20treat as no00.0%0n/a
unsure0.20 < P < 0.80treat as unsure00.0%0n/a
What the kit does with each answerShare of answers in each action band.acts on itacts and verifiesdefers or askschoice13.3%43.3%43.3%noul
Figure 7 Share of answers the kit acts on, acts on with verification, or defers.

Coverage and accuracy

Coverage and accuracy, choiceCoverage and accuracy as the confidence threshold rises.coverage: share of answers acted onaccuracy of those answers0.00.00.20.20.40.40.60.60.80.81.01.0Confidence thresholdSharetrust 0.85coverage 13%accuracy 50%verify 0.60coverage 57%accuracy 71%
Figure 8 choice: coverage and accuracy as the threshold rises. Dashed lines mark the kit's thresholds.

Default project policy

The agent kit's AGENTS.md snippet ships a default project policy. The report replays it on every item and counts what an agent following it to the letter would have done: the right thing, a cautious extra step (an unnecessary question or review), or a wrong action (acting on a vague request, or shipping a risky change unreviewed).