thirdshift experiment: the Command, the Bar and running Trials #1003

Parent

#977

What to build

thirdshift experiment is a new kind of Command.

• Inputs:
  • the Candidate, as a branch or commit;
  • the Baseline, by default the commit where the Candidate's branch left main;
  • the suites, by default those its change touches, from The Eval task format, suites and time-safe repositories #1000's declarations;
  • per-Arm Harness, Model and Effort flags.
• The Command: it writes a Command record naming both Arms and the Bar, a Command log and Activity log lines. They live in thirdshift's own repository's logs folder, in an experiment folder. It shows in the Inbox and can be Stopped.
• The Bar: the standing default is a 10-point margin, a 95% credible level, the prior, the seed and draw count, the Trials per Eval task, the per-Trial timeout and the Void Trial rules. It's recorded before the first Trial, and refused if it changes afterwards.
• Running Trials: it builds the Arms (Arm builds from commits, cached, with Harness versions fixed #1002), prepares the Eval tasks (The Eval task format, suites and time-safe repositories #1000), and runs each Trial through its Arm's Trial flag (The Trial flag: one session through a Run's code, one JSON line #1001). Two workers interleave the Arms, with a per-Trial timeout. A timeout counts as a fail. One result per Trial goes into the Experiment's folder under ~/.thirdshift/evals/experiments/.
• Resuming: running the same Experiment again resumes it, skipping finished Trials and re-running the ones cut short. A finished Experiment reports its results and doesn't run again.
• Listing: thirdshift experiments [--json] lists Experiments, and thirdshift experiments <id> [--json] shows one.

Trials run without a sandbox until #1014.

Acceptance criteria

• An integration test with fake Harnesses and a fixture suite checks the Command record (kind, both Arms, the Bar and the stages), the Command log, the Activity log lines and one result per Trial.
• At most two Trials run at once, and the Arms are interleaved.
• A changed Bar is refused after the first Trial.
• An interrupted Experiment resumes when run again, without re-running finished Trials.
• Stop ends it, from the CLI as for any Command.
• thirdshift experiments --json lists and shows Experiments, and the guide describes the command.