You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Batch rechecks, and say which flip makes a Score decisive #1253
This was generated by AI during triage. Rewritten from the grill-with-docs session in docs/grill-with-docs/1252-1253-bar-and-rechecks.md (Q3.3). The original problem statement is in this issue's edit history.
Keep today's rule for a decisive Score: a Score is decisive when any one flip of its pass or its wrong finding would change the provisional Verdict. Borderline Scores are still always rechecked. Under #1252's two-level Bar, a pass with room has few decisive Scores, so rechecks gather where a Score really decides the Verdict. Make those rechecks explainable and cheap to record.
Which flip. In thirdshift experiments <id> --json, each entry in pending_rechecks says which flip makes it decisive (its pass, its wrong finding, or both) and the provisional Verdict that flip gives.
A batch recheck. One command lists every pending recheck, with its suite, Eval task, Arm, Trial and the reason it's pending, plus a fingerprint of that list. A second form takes the Experiment, the fingerprint, one decision and one reason, and records one recheck per pending Score in a single locked write. It refuses if the pending list has changed since the fingerprint was taken.
Rechecks stay in the CLI (ADR 0016). docs/guide.md's Experiments section describes the batch and the "which flip" field.
Acceptance criteria
pending_rechecks names the flip and the Verdict it would give, for a pass and for a wrong finding.
A batch recheck records every pending Score, each kept in its Score's history like a single recheck.
A batch with a stale fingerprint changes nothing and says why.
This was generated by AI from a grill-with-docs session.
Taken off #977's sub-issues so #977's Spec run can finish without it. It was grilled together with #1252, which is now the Spec for both: the cause is shared, since the standing Bar puts any pass right at the line, so every Score is decisive.
The decision (Q3.3) keeps today's decisive rule, and adds a batch recheck plus JSON naming which flip makes each Score decisive. This issue will be rewritten as that Spec's Ticket 6 and attached to #1252 when its Tickets are published.
changed the title [-]A clean pass waits on a recheck of every Candidate Trial[/-][+]Batch rechecks, and say which flip makes a Score decisive[/+]on Oct 24, 2026
Activity
JacobStephens2 commentedon Oct 24, 2026
Taken off #977's sub-issues so #977's Spec run can finish without it. It was grilled together with #1252, which is now the Spec for both: the cause is shared, since the standing Bar puts any pass right at the line, so every Score is decisive.
The decision (Q3.3) keeps today's decisive rule, and adds a batch recheck plus JSON naming which flip makes each Score decisive. This issue will be rewritten as that Spec's Ticket 6 and attached to #1252 when its Tickets are published.
[-]A clean pass waits on a recheck of every Candidate Trial[/-][+]Batch rechecks, and say which flip makes a Score decisive[/+]Merge pull request #1269 from JacobStephens2/grill-1252-1253
JacobStephens2 commentedon Oct 24, 2026
Closed by #1275, merged into issue-1252 by a thirdshift Ticket run with Self-merge.