Skip to content

Arm builds from commits, cached, with Harness versions fixed #1002

Closed

Description

@JacobStephens2

Parent

#977

What to build

  • Building: for each Arm, thirdshift builds itself from the Arm's commit, with cargo build --release --locked in a worktree.
  • Caching: builds are cached by commit under ~/.thirdshift/evals/builds/, so the same commit is built once. The Experiment records each Arm's commit, since thirdshift version can't tell a build from main apart from the last release.
  • Harness versions: when an Experiment starts, it resolves each Harness's binary to its real path and records its version. Every Trial then runs on that binary, so a Harness that updates itself partway through can't reach one Arm and not the other.
  • Tests seed the cache, and build nothing.

Acceptance criteria

  • A commit already in the cache isn't built again.
  • A failed build stops the Experiment before its first Trial, with the build's error.
  • Tests seed the cache with the test binary for both Arms' commits.
  • Each Harness's resolved path and version are recorded. Trials keep using that path even when the Harness's link changes mid-Experiment, tested with a fake Harness whose link is swapped.

Activity

  1. JacobStephens2 commented on Oct 23, 2026

    @JacobStephens2
    OwnerAuthor

    Closed by #1124, merged into issue-977 by a thirdshift Merge run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions