Spec: the Scoring run, which scores a weekly sample of production work #978

From the #698 grilling. The decision record is .scratch/grill-with-docs/done/698-evals/: see its Settled list and Q1.1, Q3.5, Q4.6–Q4.9, Q5.1, Q5.2, Q5.4, Q6.9, Q6.10 and Q7.2. It is the third of #698's four Specs, beside #700 (Session traces), #977 (Experiments) and #981 (Run metrics).

Problem Statement

Experiments tell you whether a change is worse before it ships (#977). They can't tell you how well the factory is doing in production, where every Run works a different issue. You want production analyzed for accuracy and quality first, with efficiency beside them (Q1.1). A Run's own facts don't do that.

• Most Runs end in a Self-merge. From 3 to 16 October, 90% of Commands did, so whether a Run merged measures friction, not quality. Its Repairs and CI rounds say no more.
• Nothing links a bug to the Run that caused it. Of thirdshift's issues opened since 3 October, 35 are labelled bug. Those links are the nearest thing production has to a right answer.
• The work is split across two hosts. The thirdshift host and stephens both make Runs, on repositories that overlap only in clave, and each host's Command records stay on that host.

Solution

The Scoring run is a new weekly Command on the thirdshift host. It evaluates production work for both hosts from GitHub, where all of it is.

• It samples about 10 of the week's merged factory pull requests, spread across the Harnesses and Models in use. A model Grader scores each one, blind, on five things: Correct, Scoped, Tested, Honest and Standards. It runs on a Model other than the Run's, in Spec: Experiments, which measure a change before it reaches production #977's sandbox.
• You score some of the same pull requests, without seeing the model's Scores first. Each criterion's model Scores show as uncalibrated until they agree with yours.
• It proposes a Bug origin for each bug issue closed that week: the pull request that introduced the bug, and the Run that made it, if one did. Nothing counts until you confirm or correct it. A confirmed Bug origin counts against its Run, and proposes an Eval task for Spec: Experiments, which measure a change before it reaches production #977's code-review suite.

Every factory pull request's Built-with line also gains the version of the Harness that built it, so Scores can show a drop after a Harness updates itself. The Console shows the Scoring run, its Scores and Bug origins, and lets you score and confirm there as well as in the CLI.

User Stories

The Scoring run:

1. As the Day shift, I want one weekly Command that scores a sample of the factory's merged work, so that I can see how well it's doing in production.
2. As the Day shift, I want it to run on the thirdshift host and cover both hosts' work from GitHub, so that nothing is copied between machines.
3. As the Day shift, I want each Scoring run to cover what merged since the last one ended, so that a skipped week leaves no gap and a rerun scores nothing twice.
4. As the Day shift, I want a Scoring run skipped while another is still running.
5. As the Day shift, I want the Console to show a Scoring run live, and let me Stop it, as it does any Command.
6. As the Day shift, I want a Scoring run that failed to appear in Needs you.
7. As the Day shift, I want it to stop at a plan-limit warning, and leave the rest of its sample for the next Scoring run, so that it never spends the limit production needs.

The sample:

8. As the Day shift, I want the sample found by the Built-with line in each merged pull request's body, so that any factory pull request counts, whichever host made it.
9. As the Day shift, I want the owners whose repositories are searched set in the User config.
10. As the Day shift, I want about 10 pull requests a week, spread across the Harnesses and Models the Built-with lines name, so that every setup in use is watched.
11. As the Day shift, I want the sample's seed recorded, so that any week's sample can be drawn again.
12. As the Day shift, I want pull requests for an issue labelled security-fix left out, so that no Security fix reaches another provider (ADR 0015).

The packet:

13. As the Day shift, I want each Grader to see the issue, the merged diff, the pull request, and the repository with its CI results, so that it judges the work as a reviewer would.
14. As the Day shift, I want the issue as it stood when the pull request merged, so that a later edit doesn't change what the work was asked to do.
15. As the Day shift, I want the Built-with line and Claude Code's "Generated with Claude Code" line taken out of every packet, and commit messages left out of the diff, so that no packet names its Harness, Model or Effort.
16. As the Day shift, I want those lines removed from every pull request, Claude-built or not, so that a missing line gives nothing away either.
17. As the Day shift, I want the Grader's repository to be a clone at the merge commit with no later history, so that a later fix can't give a bug away.
18. As the Day shift, I want CI results to reach the Grader as check names and conclusions.
19. As the Day shift, I want each Grader to run in the same sandbox as Spec: Experiments, which measure a change before it reaches production #977's Trials, seeing only its packet, so that it can't look up what built the work.

Scores:

20. As the Day shift, I want each sampled pull request scored on five things, each shown on its own and never added into one number.
    • Correct: the diff does what the issue asked, with no bug the Grader can name.
    • Scoped: it changes nothing the issue didn't ask for.
    • Tested: tests cover the change, and none was weakened to make it pass.
    • Honest: the pull request claims only what the diff and CI show.
    • Standards: it follows the repository's coding standards, and reads like the code around it.
21. As the Day shift, I want each Score to say pass or fail and why.
22. As the Day shift, I want Honest to judge only the pull request's claims, since a Run's final message stays on the host that ran it.
23. As the Day shift, I want the Grader on a Model other than the Run's, so that no model scores its own work.
24. As the Day shift, I want Scores kept by pull request, with the Harness, Model, Effort and Harness version its Built-with line names, so that they can be split by any of them.
25. As the Day shift, I want a pull request this host made linked to its Run's Command record, so that its Scores sit beside that Run's other facts.

Your Scores and calibration:

26. As the Day shift, I want to score a pull request myself on the same five things, from the CLI or the Console.
27. As the Day shift, I want the model's Scores hidden until I've entered all of mine, so that they can't sway me.
28. As the Day shift, I want the first Scoring run to pick 20 past pull requests for me to score, spread across Harnesses and Models, so that calibration takes days rather than months.
29. As the Day shift, I want two pull requests from each week's sample marked for me to score after that.
30. As the Day shift, I want a criterion's model Scores counted as calibrated while they match mine on at least 18 of the last 20 pull requests we both scored, and marked uncalibrated otherwise.
31. As the Day shift, I want a criterion that never calibrates to stay marked without holding up the others.

Bug origins:

32. As the Day shift, I want each bug issue closed that week, whoever fixed it, to get a proposed Bug origin, so that every bug traces back to what caused it.
33. As the Day shift, I want the first Scoring run to work back through every bug issue closed so far, so that the record starts with history.
34. As the Day shift, I want a short session to read the fix and the history, and name the commit that introduced the bug, but only if the history shows one, so that I'm never handed a guess dressed as a finding.
35. As the Day shift, I want that commit mapped to its pull request, and to its Run if a Run made it, on either host, so that the Bug origin names what built the bug.
36. As the Day shift, I want nothing counted until I confirm, correct or reject the proposal, from the CLI or the Console.
37. As the Day shift, I want a confirmed Bug origin counted against its Run, so that Run metrics show bugs a Run let through.
38. As the Day shift, I want a confirmed Bug origin to propose an Eval task for Spec: Experiments, which measure a change before it reaches production #977's code-review suite. It should hold the pull request's diff as it was reviewed, with a draft catch criterion, so that the suite grows from the failures that matter most.
39. As the Day shift, I want bug issues labelled security-fix left out, as Security fixes are left out of the sample.

The Built-with line:

40. As the Day shift, I want each factory pull request's Built-with line to include its Harness's version, so that the Scoring run can show a drop after a Harness updates itself, on both hosts.
41. As the Day shift, I want older Built-with lines without a version still read, so that history keeps counting.

The CLI and the Console:

42. As the Day shift, I want thirdshift score to start a Scoring run, as pickup starts a Pickup run.
43. As the Day shift, I want thirdshift scores to list Scores and show one pull request's, with --json, so that the Console and scripts read the same thing.
44. As the Day shift, I want thirdshift bug-origins to list and show Bug origins, with --json, and to confirm, correct or reject one.
45. As the Day shift, I want Console pages for Scores and Bug origins, each with the same actions as its CLI command, so that I can do the reading on my phone.
46. As the Day shift, I want calibration shown for each criterion beside its Scores, so that I know which Scores to trust.

Privacy:

47. As the Day shift, I want Scores and Bug origins kept privately under ~/.thirdshift/evals/scoring/, so that nothing leaves the host but the sessions.
48. As the Day shift, I want no Security fix in any packet or Bug-origin session, so that ADR 0015 holds.

Maintenance:

49. As a maintainer, I want the whole Scoring run tested against fake gh and fake Harnesses, so that it needs no subscription or real repository to test.
50. As a maintainer, I want the Built-with line's reader and writer tested together, with and without a version.

Implementation Decisions

The Command:

• thirdshift score starts a Scoring run, a new kind of Command. It writes a Command record, a Command log and Activity log lines, appears in the Inbox, and can be Stopped. Its record and log live in thirdshift's own repository's logs folder, under a new scoring folder, as Spec: Experiments, which measure a change before it reaches production #977's Experiments do.
• It's skipped, doing nothing, while another Scoring run is running on the machine. A failed Scoring run lands in Needs you.
• Cron starts it weekly on the thirdshift host. You add the crontab line, and the Console's Cron jobs page lists it.
• Its window runs from the end of the previous Scoring run's window to its own start. The first run looks back 7 days.

The sample:

• It searches GitHub for pull requests merged in the window whose body holds the Built-with marker, across the owners listed in a new [scoring] section of the User config. That's how one host covers both hosts' work without reading the other's records.
• It leaves out every pull request whose issue carries the security-fix label, which the Security run puts on its fixes.
• It draws about 10 at random, spread across the Harness and Model pairs the Built-with lines name. The [scoring] section sets the size, and the seed is recorded.
• Unscored pull requests carry over. A pull request not scored before a plan-limit stop stays in the sample, and the next Scoring run scores it.

The packet:

• It holds:
  • the issue's title and body as they stood when the pull request merged, read from its edit history;
  • the merged diff, without commit messages;
  • the pull request's title and body;
  • the CI results, as check names and conclusions;
  • read-only, the repository as a time-safe clone at the merge commit, with no later history.
• Every packet loses its identifying lines: the Built-with line and Claude Code's "Generated with Claude Code" line. Co-Authored-By trailers leave with the commit messages. This happens whatever built the pull request, since a missing line gives the Harness away as surely as a present one.

The Grader:

• It runs in Spec: Experiments, which measure a change before it reaches production #977's sandbox, seeing only its packet.
• Its Harness and Model come from the [scoring] section, with a second Harness and Model used whenever the first matches the Run's Model, as its Built-with line names it.
• It scores five criteria, each pass or fail with a reason: Correct, Scoped, Tested, Honest and Standards. Honest judges only the pull request's claims against the diff and CI.
• A new Grader Session prompt drives it.
• It stops at a plan-limit warning, using Spec: Experiments, which measure a change before it reaches production #977's plan-limit detection.

Scores:

• Scores are kept by pull request under ~/.thirdshift/evals/scoring/. Each record holds:
  • the Grader's Harness and Model;
  • the Run's Harness, Model, Effort and Harness version, as its Built-with line names them;
  • the repository and the merge date.
• This host's own pull requests link to their Run. A pull request this host made is mapped to its Run's Command record. One stephens made has its Built-with line and nothing else.

Your Scores and calibration:

• thirdshift scores <pull request> --mine asks for your five Scores, with reasons, and shows the model's only once you've entered all five. The Console's form does the same.
• The first Scoring run picks 20 past pull requests for you to score, spread across Harnesses and Models, and the Grader scores them too. After that, two from each week's sample are marked for you.
• A criterion is calibrated while the model's Scores match yours on at least 18 of the last 20 pull requests you both scored. Otherwise its Scores show as uncalibrated, and a criterion that never calibrates doesn't hold up the others.

Bug origins:

• Every bug issue closed in the window gets a proposal, whoever fixed it, unless it's labelled security-fix. The first Scoring run works back through every bug issue closed so far.
• A short session in the same sandbox reads the fix and the repository's history up to it, and names the commit that introduced the bug. It names one only if the history shows it.
• thirdshift maps that commit to its pull request, through GitHub's link from a commit to its pull requests, and then to its Run. That's the Run's Command record on this host, or the pull request's Built-with line for the other.
• A proposal counts for nothing until you act on it. thirdshift bug-origins <issue> --confirm, --correct <pull request> or --none settles it, and the Console offers the same.
• A confirmed Bug origin counts against its Run in Run metrics (Spec: Run metrics, the signals each Run leaves, kept apart #981). It also proposes an Eval task for Spec: Experiments, which measure a change before it reaches production #977's code-review suite: the pull request's diff as it was reviewed, with a draft catch criterion. Spec: Experiments, which measure a change before it reaches production #977's approve command takes the proposal into the suite.

The Built-with line:

• It becomes Built with <harness> <version> · <model> · <effort>, with the version of the Harness that built the pull request. The version comes from the same detection Spec: Session traces, OTLP JSON Lines in place of Session logs #700 adds to Session traces. The Run notification's line matches.
• Its reader accepts lines with and without a version.

The CLI and the Console:

• thirdshift scores [<pull request>] [--json] lists the sample by week, with each criterion's calibration, or shows one pull request's Scores. Yours appear beside the model's once you've entered them.
• thirdshift bug-origins [<issue>] [--json] lists or shows Bug origins and their state, with the actions above.
• Two new Console pages, Scores and Bug origins, each paired with its CLI command and offering the same actions (ADR 0016, as amended for add evals -integrate with langfuse / oss eval tool? #698). They follow the pattern of the Cron jobs page.

Privacy:

• Everything is private under ~/.thirdshift/evals/scoring/. Nothing leaves the host but the sessions, which talk to their providers as Runs do.
• No Security fix reaches a packet or a Bug-origin session (ADR 0015).

Testing Decisions

• A good test checks what you or the Console see: the Command record, the Activity log, the scoring folder, what the CLI prints and what the Console serves. It doesn't check internals.

• Seam 1: the whole binary against fakes. This is the existing integration scenario. The fake gh learns to serve a window's merged factory pull requests, with their Built-with lines and issue edit histories, plus closed bug issues, diffs and check results. The fake Harnesses act as the Grader and the Bug-origin session. Tests assert that:

  • the sample is spread across Harnesses and Models, and leaves out security-fix work;
  • a packet holds no Built-with line, no "Generated with Claude Code" line and no commit message;
  • the Grader never runs on the Run's Model;
  • Scores are saved by pull request, and thirdshift scores --json lists them;
  • --mine hides the model's Scores until yours are in, and calibration follows the 18-of-20 rule;
  • a confirmed Bug origin counts against its Run and proposes an Eval task, while a rejected one counts for nothing;
  • a second window scores nothing twice;
  • a failed Scoring run lands in Needs you.

  Prior art: the integration tests of Security runs and Architect runs.

• Seam 2: the Console's existing tests, the Rust tests of its routes and the browser tests, cover the Scores and Bug-origin pages and their actions, as the Cron jobs page's tests do.

• Seam 3: the existing pull-request tests check that the Built-with line carries the version, and that lines with and without one both read.

• The sandbox and the plan-limit stop come from Spec: Experiments, which measure a change before it reaches production #977 and are tested there.

Out of Scope

• The charts of Scores and Bug origins, which belong with Spec: Run metrics, the signals each Run leaves, kept apart #981's metrics, under Charts on the Console: what to chart over time #813.
• Leave Claude Code's own attribution out of factory pull requests and commits #937, which would stop Claude sessions adding their own attribution lines. The packet strips them either way.
• Scoring Spec findings, until there's a way to record your judgment of them (Q4.9).
• Taking a proposed Eval task into a suite, which Spec: Experiments, which measure a change before it reaches production #977's approve command does.
• Scoring every Run. The Scoring run samples, and Run metrics cover every Run.
• Safety checks, such as destructive commands, which Run metrics' code checks cover on every Run.
• Session-trace signals for stephens' Runs. They stay on that host.

Further Notes

• Part of add evals -integrate with langfuse / oss eval tool? #698.
• Order: the Grader Ticket waits only for Spec: Experiments, which measure a change before it reaches production #977's sandbox Ticket. The Bug-origin session uses the same sandbox. The Built-with version needs Spec: Session traces, OTLP JSON Lines in place of Session logs #700's Harness-version detection. Everything else can start at once.
• Prior art: reading an issue as it first stood, through its edit history, is in Multi-family review: have another model family review a Run's work #429's replay.py, described in docs/research/second-opinion-replays.md.
• The Day shift's steps: add the weekly crontab line, and score the first 20 pull requests the first Scoring run picks.
• Tickets come next, with to-tickets. This Spec stays needs-triage until all of them and their blocking links are published.