You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Arm builds from commits, cached, with Harness versions fixed #1002
Building: for each Arm, thirdshift builds itself from the Arm's commit, with cargo build --release --locked in a worktree.
Caching: builds are cached by commit under ~/.thirdshift/evals/builds/, so the same commit is built once. The Experiment records each Arm's commit, since thirdshift version can't tell a build from main apart from the last release.
Harness versions: when an Experiment starts, it resolves each Harness's binary to its real path and records its version. Every Trial then runs on that binary, so a Harness that updates itself partway through can't reach one Arm and not the other.
Tests seed the cache, and build nothing.
Acceptance criteria
A commit already in the cache isn't built again.
A failed build stops the Experiment before its first Trial, with the build's error.
Tests seed the cache with the test binary for both Arms' commits.
Each Harness's resolved path and version are recorded. Trials keep using that path even when the Harness's link changes mid-Experiment, tested with a fake Harness whose link is swapped.
Activity
thirdshift experiment: the Command, the Bar and running Trials #1003JacobStephens2 commentedon Oct 23, 2026
Closed by #1124, merged into issue-977 by a thirdshift Merge run.