Spec: Experiments, which measure a change before it reaches production #977

From the #698 grilling. The decision record is .scratch/grill-with-docs/done/698-evals/: see its Settled list and Q2.3–Q2.6, Q3.2–Q3.11, Q3.13, Q3.20, Q4.1–Q4.4, Q4.9, Q5.3, Q5.5–Q5.7, Q6.1, Q6.3–Q6.11 and Q7.1–Q7.3. It is the second of #698's four Specs, built beside #700, the Session-trace Spec. The others are #978 (the Scoring run) and #981 (Run metrics).

Problem Statement

You change what the agents are told and run on: a Factory skill, a Session prompt, or a Harness, Model or Effort default. Nothing tells you whether a change makes the agents better or worse before it reaches production, which is the release that thirdshift update installs.

• Most such changes go out unmeasured. Since 1 October, 16 merges changed a Factory skill or a Session prompt, and they went out in 10 releases.
• The two times you measured, it took a throwaway script. That was Multi-family review: have another model family review a Run's work #429's replays and Trial: the Security review's two modes #549's Trial. Each built its repositories by hand and kept its sessions honest with a prompt that forbade network use. It graded by hand, and stopped at plan limits through its own regular expressions.
• Neither compared two setups with statistics that say when a difference is real.

You want to know, before a change ships, whether the Candidate is worse than what runs now, on accuracy first and with efficiency shown beside it. The answer has to come from sessions run exactly as production runs them, where no session can see the answer.

Solution

thirdshift experiment is a new Command. An Experiment compares a Baseline arm with a Candidate arm on the Eval suites a change touches, against a Bar fixed before it starts, and ends in a Verdict: pass, fail or inconclusive.

• An Arm is a thirdshift build from a commit, with its Factory skills and Session prompts, plus a Harness, Model and Effort. For a release, the Candidate is the commit being released and the Baseline is the last release.
• Each Trial is one session on one Eval task, run through the same code a Run uses. It works in a throwaway copy of the task's repository, inside a sandbox that shows it nothing else and lets it reach only its provider.
• Graders then score the Trials. Code checks come first. Then fresh, blind agent sessions score one packet per Eval task against its written success criteria.
• The Verdict pairs the Arms on each Eval task, with a Bayesian interval on the difference in pass share.

scripts/release.sh starts an Experiment whenever a release touches an Eval suite, and waits for its Verdict. The Console shows Experiments live, with every Trial's session one click away. Eval data stays private on the thirdshift host, under ~/.thirdshift/evals/.

User Stories

Starting an Experiment:

1. As the Day shift, I want one command that compares two setups on the Eval suites a change touches, so that measuring a change is never a throwaway script again.
2. As the Day shift, I want the suites chosen from what the change touches, and every suite that uses a Harness, Model or Effort default run when that default changes, so that an Experiment runs what the change could affect and no more.
3. As the Day shift, I want the Baseline arm run fresh beside the Candidate arm every time, so that the provider's drift and the day's luck reach both Arms alike.
4. As the Day shift, I want both Arms built the same way from their commits, so that the only difference between them is the change.
5. As the Day shift, I want a release's Experiment to build its Candidate from the commit being released and its Baseline from the last release, so that it measures exactly what would replace what production runs.
6. As the Day shift, I want a change tried on its own to build its Candidate from its branch and its Baseline from where that branch left main.
7. As the Day shift, I want a change of Harness, Model or Effort that needs no code to use one build with two sets of flags, so that I can test a User config change before I make it.
8. As the Day shift, I want every Trial of an Experiment to run on the Harness versions it started with, so that a Harness updating itself partway through can't reach one Arm and not the other.

The Bar:

9. As the Day shift, I want the Bar recorded with the Experiment before its first Trial, and refused if it changes afterwards, so that no result can move the goalposts.
10. As the Day shift, I want a standing default Bar, so that I don't set its parts by hand. Its parts are a 10-point margin, a 95% credible level, the prior, the seed and draw count, the Trials per Eval task, the per-Trial timeout and the rules for Void Trials.
11. As the Day shift, I want efficiency shown beside accuracy but kept out of the Bar, unless a change is meant to save time or cost, so that a faster, worse change never passes on speed.

Running Trials:

12. As the Day shift, I want each Trial run through the same session code a Run uses, so that an Experiment measures what production does.
13. As the Day shift, I want three Trials per Arm on each code-review Eval task and one on each Spec-review Eval task, as the Bar records.
14. As the Day shift, I want at most two Trials running at once beside the factory, so that production keeps its share of the machine.
15. As the Day shift, I want the two Arms' Trials interleaved, so that a slow hour or a provider incident reaches both.
16. As the Day shift, I want an Experiment to stop starting Trials at a Harness's plan-limit warning, so that it never spends the limit production needs.
17. As the Day shift, I want rerunning the same Experiment to resume it, skipping finished Trials and re-running the ones cut short, so that a stop costs only what was in flight.
18. As the Day shift, I want to Stop an Experiment from the CLI or the Console, as I stop any Command.
19. As the Day shift, I want a Trial that runs past its timeout to count as a fail, as a Run that never finishes fails.

Keeping Trials honest:

20. As the Day shift, I want each Trial to work in a throwaway repository with history only up to its Eval task's commit, so that no later fix is in reach.
21. As the Day shift, I want preparation to fail if any later commit survives in that repository, so that a leak stops the Experiment instead of passing it.
22. As the Day shift, I want a Spec-review Trial to get a local bare repository as its origin, so that its commits and pushes work as they do in production.
23. As the Day shift, I want each Trial to get a fresh home holding only its login and the instructions, rules, settings and hooks a Run's session reads, so that Harness memory is off and a Trial is told what a Run is told.
24. As the Day shift, I want the sandbox to show a Trial nothing else. That means no eval data, no real checkouts holding later fixes, no other Trials and no Grader's packet, so that the answer is out of reach.
25. As the Day shift, I want a Trial to have its own network, which reaches only its provider through an allowlisting proxy, so that it can't reach GitHub, the host's own services or anything else.
26. As the Day shift, I want a Trial to have its own process namespace, so that it can't read the host's command lines, and everything it leaves running ends with it.
27. As the Day shift, I want web search turned off in every Trial, since it runs at the provider and so gets past the proxy.
28. As the Day shift, I want Codex's login bound in place, so that a refresh inside a Trial reaches the file production uses.
29. As the Day shift, I want Claude's Trials to use a one-year subscription token from the credentials file, so that there's no login file to bind or refresh.
30. As the Day shift, I want each Trial to run under a memory cap of its own, and to give way first to the kernel's out-of-memory killer, so that a Trial can never take the factory down with it.

Void Trials:

31. As the Day shift, I want a Trial that failed for a reason outside its Arm marked as a Void Trial by rules fixed with the Bar, so that the host's failures don't count against an Arm.
32. As the Day shift, I want a timeout, a context-window overflow, a refusal or anything an agent did inside its sandbox to count as a fail, never a void, so that voids can't hide an Arm's failures.
33. As the Day shift, I want a Void Trial re-run fresh up to twice, and shown with its cause, counted per Arm.
34. As the Day shift, I want an Eval task whose Trial is still void after its re-runs to drop out for both Arms, and the Verdict to be inconclusive if more than a tenth drop out.
35. As the Day shift, I want an Arm that keeps hitting its memory cap reported as a finding, since Runs have no cap.

Grading:

36. As the Day shift, I want code checks to run first wherever a check is mechanical, so that a model never judges what code can.
37. As the Day shift, I want model Graders to score one packet per Eval task, holding both Arms' Trials with their labels shuffled and identifying lines removed, so that a Grader can't favour an Arm.
38. As the Day shift, I want each Score to say pass or fail and why, against the Eval task's written success criteria.
39. As the Day shift, I want each Trial flagged if it reported any wrong finding, so that a review that cries wolf costs the Arm.
40. As the Day shift, I want a Grader run on a Model from neither Arm when another is available, so that no model grades itself.
41. As the Day shift, I want Graders to run in the same sandbox as Trials, seeing only their packet.
42. As the Day shift, I want to recheck the borderline Scores and the ones that decide the Verdict, and my recheck kept beside the Grader's Score, so that the Verdict never rests on a call I haven't checked.

The Verdict:

43. As the Day shift, I want each Eval task's difference in pass share given a 95% Bayesian interval, so that the Verdict says how sure it is.
44. As the Day shift, I want the Candidate to fail when the whole interval is below zero, even inside the margin, so that a surely worse change never passes on its own.
45. As the Day shift, I want the Candidate to pass only when the whole interval is above the margin and some of it is at or above zero.
46. As the Day shift, I want any other result to be inconclusive, with how many more Eval tasks would settle it, so that I know what a clear answer would cost.
47. As the Day shift, I want wrong findings judged the same way, and the code-review suite's pass^3 reported, so that consistency shows beside accuracy.
48. As the Day shift, I want the seed and everything else that shaped the Verdict recorded, so that the same results always give the same Verdict.
49. As the Day shift, I want each Arm's minutes shown, and its tokens and cost once Session traces land, so that I see what a change costs beside what it does.

Releases:

50. As the Day shift, I want scripts/release.sh to work out which suites the changes since the last release touch, and to start their Experiment itself, so that no measured change ships without me knowing.
51. As the Day shift, I want the script to wait for the Verdict, printing progress, as it waits for CI.
52. As the Day shift, I want a pass to carry on, and a fail or an inconclusive Verdict to show me the interval and ask whether to release anyway.
53. As the Day shift, I want the script to list the Scores waiting on my recheck, and wait for them, so that a release never rests on unchecked Scores.
54. As the Day shift, I want an interrupted release to pick up the same Experiment when I run the script again.
55. As the Day shift, I want a release that touches no Eval suite to need no Experiment.

The Console and the CLI:

56. As the Day shift, I want thirdshift experiments to list Experiments and show one, with --json, so that the Console and scripts read the same thing.
57. As the Day shift, I want the Console to show each Experiment live. It should show the Verdict, the interval, each Eval task's pair of Arms, every Trial, its Scores and any void cause, each Arm's efficiency, and the Bar.
58. As the Day shift, I want each Trial's session one click from its Experiment, so that I can read why it passed or failed.
59. As the Day shift, I want a failed or inconclusive Experiment in Needs you, and a passed one in Done.
60. As the Day shift, I want the Experiment pages designed in a prototype before they're built.
61. As the Day shift, I want starting an Experiment to stay in the CLI, so that the Console still never starts work.
62. As the Day shift, I want to approve a proposed Eval task from the CLI or the Console, so that a bug found in production joins the code-review suite once I agree.

Eval data:

63. As the Day shift, I want Eval tasks, Trials, Scores and Verdicts kept private under ~/.thirdshift/evals/, so that nothing leaves the host but the sessions.
64. As the Day shift, I want the Eval task format documented in the repository and the Eval tasks kept off it, so that the format is reviewed while the answers stay private.
65. As the Day shift, I want Multi-family review: have another model family review a Run's work #429's nine code-review bugs and thirty Spec branches imported once as the first two Eval suites.
66. As the Day shift, I want Experiments to run only on the thirdshift host, where the Eval tasks are.
67. As the Day shift, I want a Security run's sessions never made into Eval tasks, so that ADR 0015 holds.

Maintenance:

68. As a maintainer, I want the Trial flag's input and output versioned, so that later builds stay comparable with earlier ones.
69. As a maintainer, I want to test whole Experiments against fake Harnesses without building thirdshift twice per test.
70. As a maintainer, I want the sandbox tested where it can run, and reported as skipped where it can't, never passed silently.

Implementation Decisions

The Command:

• thirdshift experiment is a new kind of Command. Like the Passes, it writes a Command record, a Command log and Activity log lines, appears in the Inbox and can be Stopped. Its record and log live in thirdshift's own repository's logs folder, under a new experiment folder, because the Inbox reads only Command records there.
• Where its Verdict lands: a fail or an inconclusive Verdict is an outcome that puts it in Needs you, and a pass puts it in Done.
• Its Command record names both Arms and the Bar, since a Run's record names only one Harness, Model and Effort.
• Inputs:
  • the Candidate, as a branch or commit;
  • the Baseline, by default the commit where the Candidate's branch left main;
  • the suites, by default those the change touches;
  • per-Arm Harness, Model and Effort flags.
• A release mode takes the commit being released. Its Baseline is the last release's tag, and its suites come from the changes since that tag.
• --json ends with one line holding the Verdict and its interval.
• Resuming: running the same Experiment again, with the same Arms and suites, resumes it. A finished Experiment reports its Verdict and doesn't run again.
• Listing: thirdshift experiments [--json] lists Experiments, and thirdshift experiments <id> [--json] shows one. These are the CLI commands the Console's pages pair with (ADR 0016).
• Your actions, each with --json:
  • a command records your recheck of a Trial's Score, kept beside the Grader's;
  • a command approves a proposed Eval task into its suite. Spec: the Scoring run, which scores a weekly sample of production work #978 makes the proposals, from confirmed Bug origins.

Arms and builds:

• Each Arm is built with cargo build --release --locked, in a worktree at its commit. Builds are cached by commit under ~/.thirdshift/evals/builds/. The Experiment records the commits, since thirdshift version can't tell a build from main apart from the last release.
• The Trial flag is new, so both Arms' builds must have it. The first Experiment's Baseline is the first release that ships it. The flag's input and its JSON line carry a version number, and a build refuses a version it doesn't know.
• Harness versions are fixed when the Experiment starts. It finds each Harness binary's real path then and runs every Trial on it, and records each Harness's version.

Trials:

• A hidden flag on the Arm's build runs one session, through the same session code a Run uses, in a prepared repository. It prints one JSON line: the outcome, the timings, any void cause, the session's ID, and where its reports and Session trace are. Trials write no Command record.
• A Trial's Session trace lives in its Experiment's folder, outside the factory's logs folder, so the Inbox, the Usual ranges and Silent ignore it. Its session is a span in the Experiment's Trace (Spec: Session traces, OTLP JSON Lines in place of Session logs #700).
• Session kinds:
  • The code-review suite uses a new Code review session kind and Session prompt. It runs the production code-review Factory skill on the diff as it was reviewed, under the implement session's sub-agent rules. Today only the implement session runs that review, inside itself.
  • The Spec-review suite uses the existing Spec review session and its prompt.
• Scheduling:
  • two workers, which interleave the Arms, so at most two Trials run at once;
  • a per-Trial timeout, recorded with the Bar, where a timeout is a fail.
• Plan limits:
  • thirdshift reads each Harness's limit warnings. For Claude that's its rate-limit events, both warnings and rejections. For Codex it's its rate-limit reports. The other Harnesses are added after the live checks.
  • At a warning, no new Trial starts, and a Trial cut short is void. The Experiment pauses until it's run again.

Eval tasks and suites:

• The Eval task format is documented in the repository. Each Eval task is a folder with a manifest that names:

  • its repository and commits: the base, and the change under review;
  • its session kind;
  • its inputs, such as a Spec's body as first published;
  • its success criteria, such as a bug's catch criterion or a Spec's known real findings.

• The Eval tasks themselves stay private, under ~/.thirdshift/evals/suites/<suite>/.

• Each suite declares what it depends on: the Factory skills, Session prompts and Harness, Model and Effort defaults its session uses. Those declarations decide which suites a change touches.

• Preparing a repository for an Eval task gives a throwaway clone with:

  • history only up to the base commit;
  • no origin, tags or other branches;
  • its reflog expired and pruned.

  Preparation fails if git log --all shows any later commit. A Spec-review Eval task's repository also gets a local bare origin.

• Multi-family review: have another model family review a Run's work #429's material is imported once: its nine code-review bugs with their catch criteria, and its thirty Spec branches with their findings scored real or wrong. This becomes the first two suites.

The sandbox:

• Each Trial runs in bwrap with --unshare-net and --unshare-pid. It binds only:
  • the prepared repository;
  • the Arm's build and the Harness's binary;
  • the Factory skills folder;
  • the fresh home;
  • the toolchains the session needs, read-only.
• Harness flags stay as ADR 0013 has them: approvals off, and the Harness's own sandbox off. The boundary is bwrap, outside the Harness.
• The fresh home holds the login, and the user-level instructions, rules, settings and hooks a Run's session reads, those read-only.
  • Codex's auth.json is bound read-write in place.
  • Claude gets CLAUDE_CODE_OAUTH_TOKEN, from a one-year claude setup-token token kept in a new section of thirdshift's credentials file. Setup can ask for it.
  • Each other Harness binds its one login file, once its live check passes.
• An allowlisting proxy runs outside the sandbox, listening on a Unix socket that is bound into it, with a relay inside that the Harness reaches through HTTPS_PROXY.
  • It allows only that Harness's provider hosts.
  • It refuses names that resolve to loopback or to the host's own addresses.
  • A Harness that ignores the proxy finds no route, and its Trial is void.
• Web search is off in every Trial, by each Harness's own flag.
• Memory: each Trial runs in a systemd-run --user --scope of its own, with MemoryHigh, MemoryMax and MemorySwapMax, and raises its own oom_score_adj. The caps' defaults are set once a Trial's footprint is measured.
• Graders run in the same sandbox, each seeing only its packet.

Void Trials:

• A Trial is void when:
  • the sandbox didn't start;
  • the Harness couldn't reach its provider;
  • the login failed or expired;
  • the provider returned a server error before any work;
  • a plan-limit stop ended it;
  • the host, or a memory cap that Runs don't have, killed it.
• Not void, but a fail: a timeout, a context-window overflow, a refusal, or anything the agent did inside its sandbox.
• A Void Trial is re-run fresh, up to twice, and shown with its cause, counted per Arm.
• An Eval task whose Trial is still void after its re-runs drops out of the interval for both Arms. If more than a tenth of the Eval tasks drop out, the Verdict is inconclusive.
• An Arm that keeps hitting its cap is reported as a finding.

Grading:

• Code Graders run first, on mechanical checks. For example, a Trial whose repository changed is void, and a review that left no reports fails.
• Model Graders score one blind packet per Eval task. The packet holds every Trial of both Arms, with the labels shuffled and identifying lines removed, and is scored against the Eval task's written success criteria.
  • A new Grader Session prompt drives them.
  • Each Grader runs on a Model from neither Arm when another is available.
• A Score says pass or fail, and why, for each Trial. Each Trial also gets a wrong-finding flag.
• Your rechecks: the Experiment lists the borderline Scores, which the Grader marks, and the decisive ones, whose flip would change the Verdict. A Verdict is final only once you've rechecked them all. Your rechecks are kept beside the Grader's Scores.

The Verdict:

• Each Eval task's difference is the Candidate's pass share minus the Baseline's.

• The interval comes from a Dirichlet over the values a difference can take: seven with three Trials, and three with one. The prior adds up to one pseudo Eval task, spread evenly. The interval is the 2.5th to 97.5th percentile of 20,000 seeded draws.

• The Verdict:

  • fail if the whole interval is below zero, even inside the margin;
  • pass if the whole interval is above the margin and some of it is at or above zero;
  • inconclusive otherwise. "How many more Eval tasks" is then the smallest number with at least a 0.8 chance of a clear Verdict, simulated from the posterior.

• Wrong findings are judged the same way, with the sign reversed. The code-review suite also reports pass^3.

• Recorded with the Bar:

  • the seed, draw count and prior;
  • the margin and credible level;
  • the Trials per Eval task, the per-Trial timeout and the rules for Void Trials;
  • both Arms' commits.

  The standing margin is 10 points.

• Efficiency is shown per Arm: minutes, plus tokens and cost once Spec: Session traces, OTLP JSON Lines in place of Session logs #700's Session traces land. It stays out of the Bar unless the Bar says the change is meant to save time or cost.

Releases:

• scripts/release.sh checks before it pushes anything. As part of its release checks, it works out which suites the changes since the last release touch. If any are touched, it runs the release mode of thirdshift experiment on the commit it will release and waits, printing progress.
• What it does with the Verdict:
  • a pass carries on;
  • a fail or an inconclusive Verdict shows the interval and asks whether to release anyway;
  • Scores waiting on your recheck are listed, and it waits for them.
• Resuming: an interrupted script picks up the same Experiment when it's run again. That includes resuming from a branch it already pushed, where it checks again before it opens the pull request.

The Console:

• Two new pages: the Experiments list and an Experiment's page, each paired with thirdshift experiments. They follow the pattern of the Cron jobs page.
• A /prototype session designs them first, borrowing from Phoenix's compare view.
• An Experiment's page is live while it runs, and each Trial links to its session's Session trace.
• You can approve a proposed Eval task in the Console, as ADR 0016 now allows. Starting an Experiment stays in the CLI.

Eval data and privacy:

• Everything is private under ~/.thirdshift/evals/. suites/ holds the Eval tasks, experiments/ holds each Experiment's Bar, Trials, Scores and Verdict, and builds/ is the Arm cache. Nothing leaves the host but the sessions, which talk to their providers as Runs do.
• Experiments run on the thirdshift host.
• A Security run's sessions never become Eval tasks (ADR 0015).

Steps only the Day shift can do:

• run loginctl enable-linger jacob once on the thirdshift host;

• make the one-year Claude token with claude setup-token, and save it through Setup;

• run the one-time import of Multi-family review: have another model family review a Run's work #429's material on the host;

• run the live checks before building on them:

  • whether each Harness logs in, refreshes and saves its login inside bwrap, with only its login file bound;
  • whether each Harness works through HTTPS_PROXY over the relay, and which hosts its provider needs;
  • how to turn each Harness's web search off;
  • whether a fresh home turns each Harness's memory off;
  • whether interruption still reaches the Harness through the PID namespace;
  • a Trial's real memory beside two building Runs.

  The results go in docs/research/eval-sandboxes.md.

Testing Decisions

• A good test checks what you or the Console see: the Command record, the Activity log, the results folder, what the CLI prints and what the Console serves. It doesn't check internals.

• Seam 1: the whole binary against fake Harnesses. This is the existing integration scenario, which runs the built binary with fake CLIs in a temporary home. A test sets up a fixture suite of a few Eval tasks, and seeds the build cache with the test binary for both Arms' commits, so nothing is built twice. These tests use a pass-through launcher in place of the sandbox. They assert:

  • the Experiment's Command record, stages, outcome and Activity log lines;
  • the results folder's Bar, Trials, Scores and Verdict;
  • thirdshift experiments --json;
  • a void Trial from a fake login failure is re-run, then drops its Eval task;
  • an interrupted Experiment resumes;
  • fake model Graders return Scores, and a recheck changes a decisive Score;
  • a failed Experiment lands in Needs you.

  Prior art: the integration tests of Security runs and Architect runs. The fake Claude can't yet print stream events before its result, so plan-limit cases are tested in the stream-reading tests, or the fake learns to.

• Seam 2: the Verdict on its own. It is tested with fixed seeds against known cases:

  • every difference equal;
  • small and large gaps;
  • the drop-out rule;
  • a sure loss inside the margin;
  • "how many more Eval tasks".

• Seam 3: the sandbox launcher on its own, covering bwrap, the proxy and the memory scope. It runs where bwrap works, which is the thirdshift host and CI if bwrap runs there. Anywhere else it reports itself skipped, never silently passed. Its tests check that:

  • a sandboxed process can't see an unbound path;
  • it can't reach loopback or GitHub;
  • it reaches an allowed host through the proxy;
  • nothing it starts outlives it.

• Seam 4: the Console's existing tests cover the new pages, the way the Cron jobs page was added: the Rust tests of its routes and the browser tests.

• Seam 5: scripts/release.sh's existing tests, which run it with a fake cargo, are extended to cover the wait for an Experiment: a pass, a fail that asks, a recheck waited for, and a resume.

Out of Scope

• The implement, plan-check and Architecture-review suites. Each needs its Eval tasks chosen first. Q6.7 has settled how the implement suite will run: on the host's toolchains in bwrap, with time-safe clones and code Graders that run the fix's tests offline.
• add evals -integrate with langfuse / oss eval tool? #698's other Specs: Session traces (Spec: Session traces, OTLP JSON Lines in place of Session logs #700), the Scoring run (Spec: the Scoring run, which scores a weekly sample of production work #978) and Run metrics (Spec: Run metrics, the signals each Run leaves, kept apart #981). Trials record Session traces once Spec: Session traces, OTLP JSON Lines in place of Session logs #700 lands, and until then show minutes but not tokens or cost.
• Making Bug-origin proposals. Spec: the Scoring run, which scores a weekly sample of production work #978 makes them, and this Spec only approves them into a suite.
• Self-merge settings for each kind of command (Self-merge settings for each kind of command in the User config #966).
• Pinning Harness versions in production, or blocking a User config change until an Experiment passes (Q7.2). Both were rejected. You run an Experiment before a deliberate change, though nothing enforces it.
• Containers or per-repository images for Trials (Q6.7).
• A second viewer for Experiments.

Further Notes

• Part of add evals -integrate with langfuse / oss eval tool? #698. It builds on Multi-family review: have another model family review a Run's work #429, whose material seeds the first suites, and on Trial: the Security review's two modes #549. add way for thirdshift to conduct autoresearch #348's autoresearch and update skills from mattpocock/skills #321's skill update are the kinds of change it measures.
• Order: it's built beside Spec: Session traces, OTLP JSON Lines in place of Session logs #700. Spec: the Scoring run, which scores a weekly sample of production work #978's Grader Ticket waits only for this Spec's sandbox Ticket.
• Research:
  • docs/research/eval-runners.md on the runner, the Verdict's statistics and their simulations;
  • docs/research/eval-sandboxes.md on the sandbox, logins, memory and Void Trials;
  • docs/research/second-opinion-replays.md and its replay.py, the prior art for repositories at a commit, Session prompts for review-only sessions, and limit handling.
• Tickets come next, with to-tickets. This Spec stays needs-triage until all of them and their blocking links are published.