ember

10Confusion matrices

Each matrix counts answers by gold label (rows) and ember's answer (columns). The outlined diagonal is agreement.

Confusion matrix for intentCounts of answers by gold label and ember's answer.rows: gold columns: ember's answerquestionimplementinvestigatefixquestion27000implement02601investigate00270fix01026
Figure 14 intent (choice).
Confusion matrix for specific_enoughCounts of answers by gold label and ember's answer.rows: gold columns: ember's answeryesnoyes5215no338
Figure 15 specific_enough (noul).
Confusion matrix for failure_kindCounts of answers by gold label and ember's answer.rows: gold columns: ember's answerlogic_bugenvironmentflakytest_buglogic_bug23000environment02021flaky00230test_bug10022
Figure 16 failure_kind (choice).
Confusion matrix for retryCounts of answers by gold label and ember's answer.rows: gold columns: ember's answeryesnoyes221no069
Figure 17 retry (noul).
Confusion matrix for riskCounts of answers by gold label and ember's answer.rows: gold columns: ember's answerNegligibleLowMediumHighNegligible22100Low02300Medium04181High05135
Figure 18 risk (score).
Confusion matrix for needs_reviewCounts of answers by gold label and ember's answer.rows: gold columns: ember's answeryesnoyes3511no046
Figure 19 needs_review (noul).
Confusion matrix for ownerCounts of answers by gold label and ember's answer.rows: gold columns: ember's answerapistorageuiinfraunclearapi150000storage015000ui001500infra000150unclear000213
Figure 20 owner (choice).
Confusion matrix for effortCounts of answers by gold label and ember's answer.rows: gold columns: ember's answerTrivial (minutes)Small (under an hour)Medium (a few hours)Large (a day or more)Trivial (minutes)21600Small (under an hour)018120Medium (a few hours)05112Large (a day or more)0361
Figure 21 effort (score).
Confusion matrix for approachCounts of answers by gold label and ember's answer.rows: gold columns: ember's answerminimal_patchrefactornew_componentask_userminimal_patch19000refactor11710new_component60130ask_user11017
Figure 22 approach (choice).
Confusion matrix for dominant_redCounts of answers by gold label and ember's answer.rows: gold columns: ember's answeryesnoyes40no04
Figure 23 dominant_red (noul).
Confusion matrix for dominant_colourCounts of answers by gold label and ember's answer.rows: gold columns: ember's answerredgreenbluemixedred2000green0200blue0020mixed0002
Figure 24 dominant_colour (choice).
Confusion matrix for alert_levelCounts of answers by gold label and ember's answer.rows: gold columns: ember's answerNo alert (calm green)Low (yellow)Medium (orange)High (red)No alert (calm green)2000Low (yellow)0200Medium (orange)0020High (red)0002
Figure 25 alert_level (score).
Confusion matrix for colour_changedCounts of answers by gold label and ember's answer.rows: gold columns: ember's answeryesnoyes40no04
Figure 26 colour_changed (noul).