The two-level Verdict: a pass level and a fail level #1264

This was generated by AI from a grill-with-docs session.

Parent

#1252

What to build

The Bar judges a pass and a fail at separate levels (grill Q2.3, Q3.4).

• The Bar has a pass level and a fail level in place of its one credible level. thirdshift experiment takes --pass-level and --fail-level in place of --credible-level. Both are saved before the first Trial and refused if they change afterwards, as the rest of the Bar is. Both default to 95%, so releases behave as they do today until the pass level is set from the A/A measurement.
• The decision: fail when the upper bound at the fail level is below zero. Otherwise pass when the lower bound at the pass level is above the margin and the upper bound at the pass level is at least zero. Otherwise inconclusive. One set of draws gives both intervals. The prior, seed, draws and margin are unchanged.
• The "about N more Eval tasks would settle it" prediction uses the same two-level decision.
• The Verdict's JSON gives each suite's intervals at both levels, and the Bar's JSON gives both levels. The method gets a new name in place of paired-dirichlet-v1. An Experiment saved under the old method keeps its stored Bar and Verdict, and is still read.
• An ADR records why a pass and a fail are judged at different levels. docs/guide.md describes the two levels where it describes the standing Bar.

Acceptance criteria

• Verdict tests show a pass at a pass level the fail level alone wouldn't give, a fail at the fail level, and an inconclusive between them.
• The prediction uses both levels.
• The JSON carries both levels, both intervals and the new method name. An Experiment saved under the old method still loads, with its Verdict unchanged.
• --pass-level and --fail-level are saved in the Bar and refused after the first Trial. --credible-level is gone.
• A test pins the smallest code-review suite (3 Trials per task, identical Arms) that passes under the default Bar.
• The ADR and the guide's Bar text are written.