Skip to content

Batch rechecks, and say which flip makes a Score decisive #1253

Closed

Description

@JacobStephens2

This was generated by AI during triage. Rewritten from the grill-with-docs session in docs/grill-with-docs/1252-1253-bar-and-rechecks.md (Q3.3). The original problem statement is in this issue's edit history.

Parent

#1252

What to build

Keep today's rule for a decisive Score: a Score is decisive when any one flip of its pass or its wrong finding would change the provisional Verdict. Borderline Scores are still always rechecked. Under #1252's two-level Bar, a pass with room has few decisive Scores, so rechecks gather where a Score really decides the Verdict. Make those rechecks explainable and cheap to record.

  • Which flip. In thirdshift experiments <id> --json, each entry in pending_rechecks says which flip makes it decisive (its pass, its wrong finding, or both) and the provisional Verdict that flip gives.
  • A batch recheck. One command lists every pending recheck, with its suite, Eval task, Arm, Trial and the reason it's pending, plus a fingerprint of that list. A second form takes the Experiment, the fingerprint, one decision and one reason, and records one recheck per pending Score in a single locked write. It refuses if the pending list has changed since the fingerprint was taken.
  • Rechecks stay in the CLI (ADR 0016). docs/guide.md's Experiments section describes the batch and the "which flip" field.

Acceptance criteria

  • pending_rechecks names the flip and the Verdict it would give, for a pass and for a wrong finding.
  • A batch recheck records every pending Score, each kept in its Score's history like a single recheck.
  • A batch with a stale fingerprint changes nothing and says why.
  • docs/guide.md describes both.

Activity

  1. JacobStephens2 commented on Oct 24, 2026

    @JacobStephens2
    OwnerAuthor

    This was generated by AI from a grill-with-docs session.

    Taken off #977's sub-issues so #977's Spec run can finish without it. It was grilled together with #1252, which is now the Spec for both: the cause is shared, since the standing Bar puts any pass right at the line, so every Score is decisive.

    The decision (Q3.3) keeps today's decisive rule, and adds a batch recheck plus JSON naming which flip makes each Score decisive. This issue will be rewritten as that Spec's Ticket 6 and attached to #1252 when its Tickets are published.

  2. changed the title [-]A clean pass waits on a recheck of every Candidate Trial[/-] [+]Batch rechecks, and say which flip makes a Score decisive[/+] on Oct 24, 2026
  3. added a commit that references this issue on Oct 24, 2026
  4. JacobStephens2 commented on Oct 24, 2026

    @JacobStephens2
    OwnerAuthor

    Closed by #1275, merged into issue-1252 by a thirdshift Ticket run with Self-merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions