Spec: a Bar a release that changes nothing can pass #1252

This was generated by AI from a grill-with-docs session.

Decided in docs/grill-with-docs/1252-1253-bar-and-rechecks.md (Q1.1–Q4.2; committed locally, on main once merged), which also holds the simulations behind every number here. It merges this issue with #1253, and builds on #977, which it is blocked by. This issue's earlier body is in its edit history.

Problem Statement

scripts/release.sh stops a release whose Experiment ends in a fail or inconclusive Verdict, and asks whether to release anyway. At our Eval suites' sizes, the standing Bar lets almost no real Experiment pass.

• Code review can never pass. With 10 Eval tasks, even Arms that score identically on every task give an interval whose lower bound is −12.8 points, below the −10-point margin.
• Spec review can barely pass. With 30 Eval tasks, it passes only if the Arms agree on every task. One task going each Arm's way gives −11.0.
• A pass that close to the line makes every Score decisive. One flipped Score tips it, so a clean Spec-review pass would ask you to recheck about 30 Candidate Trials by hand (Batch rechecks, and say which flip makes a Score decisive #1253). Today this is hidden, because code review's permanent inconclusive already decides any Experiment that selects both suites.

So every release that touches an Eval suite would stop and ask, and you'd always answer yes. A prompt that's always answered yes protects nothing.

Nobody knows how noisy the Arms really are. #429's replays and #549's Trial each ran one session per cell, so how often two runs of one setup disagree on a task was never measured. #698's research had already noted that detecting a 10-point difference needs about 175 Eval tasks.

Solution

Keep the release prompt, and make the Bar passable by a release that changes nothing.

• Two levels in the Bar. A pass needs the lower bound at a looser pass level above the −10-point margin. A fail still needs the upper bound at the 95% fail level below zero. Anything between is inconclusive. So "worse" still means 10 points, and a fail still means 95% sure the Candidate is worse at all. Only a pass asks for less certainty.
• Measure, then set. One A/A Experiment on the thirdshift host, with one commit in both Arms, measures how often the Arms disagree when nothing differs. The default pass level is then the highest one at which such an Experiment passes at least 80% of the time. It never goes below 70%. If 70% isn't enough, the suite grows instead.
• A bigger code-review suite. Seven bugs found since Security review: surface the session's reason when it deliberately ends incomplete #607, each visible in its PR's diff, become approved code-review Eval tasks, bringing the suite to 17.
• Rechecks where they decide. The rule for a decisive Score stays. With room under a pass, few Scores are decisive. A batch command records your decision on a reviewed pending list, and thirdshift experiments <id> --json says which flip makes each Score decisive.

Until the new default ships, releases keep today's Bar and its override.

User Stories

The Bar:

1. As the Day shift, I want a release that changes nothing to usually pass its Experiment, so that the release prompt stays rare enough to read.
2. As the Day shift, I want the release to keep stopping on fail and on inconclusive, so that "we couldn't tell" never ships without a question.
3. As the Day shift, I want a pass judged at the pass level against the −10-point margin, so that a pass is reachable at our suites' sizes.
4. As the Day shift, I want a fail still judged at the 95% fail level against zero, so that noise rarely fails a good change.
5. As the Day shift, I want the margin kept at −10 points, so that "worse" means what it meant under Spec: Experiments, which measure a change before it reaches production #977.
6. As the Day shift, I want one Bar for every suite, so that there's one rule to understand.
7. As the Day shift, I want the pass level and the fail level saved with the Experiment before its first Trial, and refused if they change afterwards, as the rest of the Bar is.
8. As the Day shift, I want --pass-level and --fail-level flags in place of --credible-level, so that I can set either level for one Experiment.
9. As the Day shift, I want the Verdict's method to name the two-level rule, so that an Experiment judged under the old rule is never mistaken for one under the new rule.
10. As the Day shift, I want the "about N more Eval tasks would settle it" prediction to use the same two levels, so that it predicts the Verdict I would actually get.
11. As the Day shift, I want each suite's intervals shown at the pass level and at the fail level, so that I can see which one decided its Verdict.
12. As the Day shift, I want an Experiment saved under the old Bar to keep its old Verdict, so that a change of method doesn't rewrite past results.

Measuring the noise:

13. As the Day shift, I want to run an A/A Experiment, with one commit and the same flags in both Arms, so that I can measure how often the Arms disagree when nothing differs.
14. As the Day shift, I want the A/A Experiment run on the User config's Harness, Model and Effort, so that the measurement matches the setup releases are judged on.
15. As the Day shift, I want the A/A Experiment run on the 17-task code-review suite and the 30-task Spec-review suite, so that it measures the suites the Bar will judge.
16. As the Day shift, I want each suite's disagreement rate posted on this issue, so that the default pass level is chosen from recorded numbers.
17. As the Day shift, I want the default pass level chosen as the highest one at which an A/A Experiment at that rate passes at least 80% of the time, so that the choice follows a written rule, not a feeling.
18. As the Day shift, I want the default pass level never set below 70%, so that a pass never becomes a coin toss. If 70% isn't enough, I want the shortfall named, so that I know which suite has to grow.
19. As the Day shift, I want the default pass level and the interval it gives at each suite's size written in docs/guide.md, where the standing Bar is described.

The code-review suite:

20. As the Day shift, I want Spec: a Bar a release that changes nothing can pass #1252's seven strong bugs added to code review as Eval tasks, so that the suite has 17 tasks.
21. As the Day shift, I want each one's success criteria drafted by an agent on the host and approved by me, so that no task runs that I haven't approved.
22. As the Day shift, I want the five partly visible bugs left out, so that code review keeps measuring what a review can see in the diff.

Rechecks:

23. As the Day shift, I want a Score to stay decisive when any one flip would change the provisional Verdict, so that a pass resting on one mis-scored Trial is still caught.
24. As the Day shift, I want borderline Scores still always rechecked, as Your rechecks of borderline and decisive Scores #1006 requires.
25. As the Day shift, I want thirdshift experiments <id> --json to say, for each pending recheck, which flip makes it decisive (its pass, or its wrong finding) and what the Verdict would become, so that the Experiment page (The Experiment pages in the Console #1020) can show why it's asked.
26. As the Day shift, I want one command that lists every pending recheck, with its Trial and the reason it's pending, so that I can review them all in one place.
27. As the Day shift, I want to record the same decision for every pending recheck in that list with one reason, so that a release doesn't cost me 30 commands.
28. As the Day shift, I want the batch refused if the pending list changed since it was shown, so that I never confirm a Score I didn't see.
29. As the Day shift, I want a batch recheck kept in each Score's history like any single recheck, so that the audit trail is the same either way.

Implementation Decisions

• The Bar replaces its one credible level with a pass level and a fail level. The fail level defaults to 95%. The pass level defaults to 95% until Ticket 5 sets it from the A/A measurement, so nothing changes for releases before then.
• The decision rule reads: fail when the upper bound at the fail level is below zero. Otherwise pass when the lower bound at the pass level is above the margin and the upper bound at the pass level is at least zero. Otherwise inconclusive. The posterior, the prior, the seed and the draws don't change. One set of draws gives both intervals.
• The method gets a new name, replacing paired-dirichlet-v1, and an ADR records why a pass and a fail are judged at different levels.
• The posterior-predictive prediction ("about N more Eval tasks…") uses the same two-level decision.
• The Verdict's JSON gives each suite's intervals at both levels, and the Bar's JSON gives both levels. Experiments saved before the change keep their stored Bar and Verdict, and are read as the old method.
• An Experiment may have the same commit and the same flags in both Arms. If anything refuses that today, it is allowed, and the A/A case is documented in docs/guide.md.
• The rule for a decisive Score doesn't change. pending_rechecks in experiments <id> --json gains, for each Score, the flip that makes it decisive and the provisional Verdict that flip would give.
• A batch recheck takes the Experiment, a decision and a reason, plus a fingerprint of the pending list it was shown. It refuses if the list has changed, and records one recheck per pending Score in a single locked write, as a single recheck does now. Rechecks stay in the CLI (ADR 0016).
• The seven code-review Eval tasks are Pickup reservation pushes run the repository's git hooks #847, Check re-run reads CI as passed before the re-run's checks are listed, and Self-merges red CI #686, Unit test startup_failure_preserves_its_cause_without_completion_reporting runs a real installed opencode #652, thirdshift console uses 50-74% of a core while idle #1153 and the other three in the "Strong" table of Spec: a Bar a release that changes nothing can pass #1252's triage comment, with the base and change commits given there. Eval data stays private on the host (Spec: Experiments, which measure a change before it reaches production #977), so they're written there, not in a Ticket run.
• scripts/release.sh doesn't change. It already prompts on fail and inconclusive, and waits on rechecks.

Testing Decisions

• Test external behaviour only. Tests judge Scores through the Verdict's public function, and drive the commands against the fake Harness. They don't reach into how intervals are computed.

• The Verdict, in its unit tests:

  • a pass at the pass level that the fail level alone wouldn't give;
  • a fail at the fail level;
  • an inconclusive between the two;
  • the new method name in the JSON;
  • the prediction under both levels;
  • a pinned case, the smallest code-review suite (3 Trials per task, identical Arms) that passes under the default Bar, so that a later change can't quietly bring Spec: a Bar a release that changes nothing can pass #1252 back.

  Prior art: the existing Verdict tests on the margin, wrong findings and the prediction budget.

• The commands, in the Experiment integration tests:

  • an A/A Experiment with one commit in both Arms;
  • the level flags saved in the Bar, and refused after the first Trial;
  • which flip pending_rechecks names;
  • a batch recheck that records every pending Score;
  • a batch refused after the pending list changes.

  Prior art: the existing Experiment and recheck integration tests.

• No release tests change, since release.sh doesn't.

• The host work (the seven tasks and the A/A run) has no automated tests. Its output is the approved tasks and the numbers posted here.

Out of Scope

• A Bar for each suite, and a smaller prior (this issue's former options 2 and 3).
• Prompting only on fail (the grill's Q1.3, option b).
• Checking only a sample of decisive Scores (Batch rechecks, and say which flip makes a Score decisive #1253's option 2), and counting only flips that make or undo a fail (its option 1).
• The five partly visible code-review candidates.
• Eval tasks proposed from Bug origins (Approve a proposed Eval task #1021).
• Moving rechecks into the Console.

Further Notes

• Tickets, in order:
  1. The two-level Verdict, with its ADR.
  2. Allowing an A/A Experiment.
  3. The seven code-review Eval tasks (ready-for-human).
  4. Running the A/A Experiment (ready-for-human, blocked by 2, 3 and Spec-review Trials have no Trial inputs or PR service #1251).
  5. Setting the default pass level (blocked by 1 and 4).
  6. Batch rechecks and which flip (Batch rechecks, and say which flip makes a Score decisive #1253, rewritten).
• A Spec run on this issue fails until Tickets 3 and 4 are closed, because it doesn't start a ready-for-human Ticket. Run the agent Tickets one at a time until then.
• The glossary's Bar entry already lists the pass level and the fail level.