The Verdict: a paired Bayesian interval and its rules #999

Parent

#977

What to build

A module that turns an Experiment's Scores into its Verdict, as #977's "The Verdict" decisions say, tested on its own.

• Inputs: for each Eval task, each Arm's Trials as pass or fail with their wrong-finding flags, and the Bar: the margin, credible level, prior, seed and draw count.
• The interval: each Eval task's difference is the Candidate's pass share minus the Baseline's. It takes a Dirichlet over the values a difference can take, seven with three Trials and three with one. The prior adds up to one pseudo Eval task, spread evenly. From 20,000 seeded draws, it gives the 2.5th to 97.5th percentile interval.
• The rules:
  • fail if the whole interval is below zero, even inside the margin;
  • pass if the whole interval is above the margin and some of it is at or above zero;
  • inconclusive otherwise. "How many more Eval tasks" is then the smallest number with at least a 0.8 chance of a clear Verdict, simulated from the posterior.
• Wrong findings are judged the same way, with the sign reversed. pass^3 is reported for suites with three Trials.
• An Eval task dropped as void leaves the interval for both Arms. If more than a tenth drop out, the Verdict is inconclusive.

Acceptance criteria

• The same inputs and seed always give the same Verdict.
• Tested cases: every difference equal (the interval never collapses to a point), a large loss fails, a large gain passes, a small gap is inconclusive with a "how many more" figure, and a sure loss inside the margin fails.
• Drop-outs leave the interval for both Arms, and more than a tenth dropped gives inconclusive.
• Wrong findings are judged with the sign reversed, and pass^3 is reported.
• Its JSON output holds the interval, the Verdict and everything it records with the Bar.