A Trial whose review session commits its fix is always void #1341

This was generated by AI during triage.

What happened

#1255's live check ran one imported Spec-review Eval task as an A/A Experiment on v0.16.0, on Codex gpt-6.1-sol at xhigh. Both Arms went void on every attempt, three attempts each and about 15 minutes per session, all with repository_changed: "Trial repository changed during its session".

The sessions did their job. Each wrote both reports and the review summary. Then, as the production Spec review prompt says, each session fixed what it found and committed in the Trial repository.

Why

execution.rs hashes the prepared repository before and after the session, with integrity::snapshot. The hash covers HEAD, every ref, the index, the diff against HEAD, and every file, ignored and untracked ones included. The session runs in that same repository, and any difference turns an otherwise successful Trial void (added in #1004).

So a session that commits is always void. That's every Spec-review session that finds something, and every code-review session too, since both production prompts fix and commit. The review report directory the session writes inside the repository may count as a change as well.

This contradicts #698's Q3.11 (docs/grill-with-docs/done/698-evals/round-3.md). Q3.11 chose a local bare origin so that "commits and pushes then work as in production, the gh update fails harmlessly, and only the review reports are graded."

None of the 47 approved Eval tasks has checks. Mechanical checks are the only grading that reads the repository after the session.

Impact

It blocks #1267, the A/A Experiment, and so #1268 and #1252. Every Trial there that finds anything would be void, so no Verdict is possible. Each void Trial is still retried twice, which costs roughly 45 minutes of sessions per Trial.

Desired behavior

• A review session's commits on the Issue branch, its pushes to the Trial's local origin, and the files it writes in the repository don't void a Trial. That's production behavior, per Q3.11.
• A Trial is void for a repository change only when the change touches what grading reads:
  • the Eval task's inputs: .thirdshift-issue.md and any [inputs] files;
  • the base and change commits under review;
  • for a task with checks, the state the checks run on. Running them on a fresh checkout of the prepared commit, not on the session's working tree, is one way to do that.
• Reports are graded as now.

Acceptance criteria

• A Spec-review Trial whose session commits a fix and writes its reports is not void, and its reports are graded.
• The same holds for a code-review Trial.
• A session that edits or deletes .thirdshift-issue.md or another task input is still void with repository_changed.
• A task with checks can't be passed by the session editing the code its checks run on.
• docs/eval-tasks.md and docs/guide.md say what counts as a repository change.

Found by #1255.