Repository navigation
Run the A/A Experiment on the host #1267
Description
This was generated by AI from a grill-with-docs session.
Parent
What to build
Run one A/A Experiment on the thirdshift host: one commit and the same flags in both Arms, on the User config's Harness, Model and Effort, with --suite code-review --suite spec-review (grill Q2.1, Q3.2). It runs under today's Bar, since only the disagreements are needed.
That's about 150 Trials: 17 × 3 × 2 for code review and 30 × 1 × 2 for Spec review. It needs a release that carries #977, #1251 and the A/A Experiment Ticket, and the 17-task code-review suite.
Acceptance criteria
- The A/A Experiment has finished, with any rechecks recorded.
- A comment here gives, for each suite: the Eval tasks that counted, how many went each Arm's way (on accuracy and on wrong findings), the Void Trials, and the intervals at 95%, 90% and 80%.
Activity
JacobStephens2 commentedon Oct 26, 2026
A/A Experiment results
One A/A Experiment ran on the
thirdshifthost. Both Arms used commit 71c7a49 (it includes #1341) with the User config's codex · gpt-6.1-sol · xhigh and--suite code-review --suite spec-review, under today's Bar: margin −0.10, credible level 0.95, prior 1. The Grader was claude · claude-opus-5-5 · medium. It ran 162 Trials from 2026-10-25 17:43 to 2026-10-26 07:00 EDT.Provisional Verdict: inconclusive. The Verdict itself reads
waiting_on_rechecks, with 11 borderline Trials. None of them is decisive, so no recheck can change the Verdict. I'll add the rechecks here once they're recorded.Code review (17 Eval tasks × 3 Trials per Arm)
All 17 tasks counted, and none was dropped.
The suite's Verdict is inconclusive on accuracy and passes on wrong findings. Getting a no-op to clear accuracy with 80% probability would take about 15 more tasks.
Only 3 of the 17 tasks passed in any Trial: bug-182-pr-99, bug-652-pr-462 and bug-686-pr-267. The other 14 failed all 6 of their Trials in both Arms. They count as ties, but they carry no information about a difference. In effect the suite measured 3 tasks, which matters for #1268's sizing.
Spec review (30 Eval tasks × 1 Trial per Arm)
All 30 tasks counted, and none was dropped. The tasks come from private repositories, so they're counted here, not named.
The suite's Verdict is inconclusive on accuracy and passes on wrong findings. A no-op would need about 73 more tasks to clear accuracy with 80% probability. Overall, 8 tasks passed in the baseline Arm and 7 in the candidate Arm.
The one task that went the candidate's way did so because its baseline Trial hit the 30-minute timeout, which counts as failed.
Void and failed Trials
Selected model is at capacity(A model-at-capacity error fails a Trial instead of voiding it #1362), three in the baseline Arm and one in the candidate Arm. They never reviewed anything. I archived their attempts, outside the Experiment, and reran them once the provider recovered. All four succeeded. Without the rerun, those four failures would have read as disagreements.Under other credible levels
The 95% figures are the Experiment's own. I computed the 90% and 80% intervals from the same paired-Dirichlet posterior (prior 1, linear quantiles) with 200,000 draws, and that calculation reproduces the 95% figures to within ±0.002.
At margin −0.10:
JacobStephens2 commentedon Oct 26, 2026
All 11 rechecks are recorded. Each matched the Grader's decision, so the counts above stand, and the final Verdict is inconclusive on both suites, with no pending rechecks.
Both acceptance criteria are met, so this issue is closing. These numbers shouldn't set #1268's default, though. A per-task look found that both suites measure much less than their size suggests: a broken task, one task targeting the wrong PR, and six code-review tasks whose bugs fall outside what the review skill is asked to find. Six Spec-review tasks have no reference findings, and Trial environments differed (#1369). #1370 repairs the suites and reruns the A/A Experiment, and #1268 is now blocked by it.