Spec: Run metrics, the signals each Run leaves, kept apart #981

From the #698 grilling. The decision record is .scratch/grill-with-docs/done/698-evals/: see its Settled list and Q1.1, Q4.5, Q4.7, Q4.8, Q5.1, Q5.2, Q7.2 and Q7.4. It is the last of #698's four Specs: #700 (Session traces), #977 (Experiments), #978 (the Scoring run) and this one. Its Session-trace signals wait for #700.

Problem Statement

You want to know how efficiently the factory works, and where it struggles: minutes, tokens and cost, and the friction a Run meets on its way to a merge. You want those beside accuracy and quality, never folded into one number (Q1.1), and comparable across Harnesses, Models, Efforts and kinds of session. Today:

• A Command record holds a Run's outcome and stages, but nothing counts its Repairs or CI rounds.
• Unaddressed findings are only in the pull request's body, and the review's findings are only in free-form reports beside the Session log.
• No session records its tokens, cost, tool errors, permission denials or killed work.
• Nothing flags a destructive command a session ran.
• Nothing shows any of it over time, by Harness or Model. Charts on the Console: what to chart over time #813 wants charts, but has nothing to draw from.

A Run's outcome, Repairs and CI rounds don't count as evidence of quality on their own (Q1.1). They're friction, and need to be seen as friction, beside the Scores that #978 brings.

Solution

Run metrics are the signals each Run leaves, each kept apart, attached to the Run's Command record when it ends.

• From the Run:
  • its outcome;
  • its Repairs and CI rounds;
  • its Unaddressed findings and the review's findings, each by kind;
  • each stage's time against its Usual range.
• From its Session traces (Spec: Session traces, OTLP JSON Lines in place of Session logs #700):
  • tokens by Model, with the cache split;
  • cost, where the Harness reports one;
  • tool errors, permission denials, killed work and repeated tool calls;
  • destructive commands, flagged by a code check on every Run.

A confirmed Bug origin from #978 counts against the Run that made the bug. thirdshift metrics shows one Run's metrics, or weekly trends split by Harness, Model, Effort, Harness version, kind of session and repository, with --json for #813's charts. Past Runs get the metrics their records and Session traces hold. Each host shows its own Runs.

User Stories

From the Run:

1. As the Day shift, I want each Run's outcome kept with its metrics, so that trends can be split by how Runs ended.
2. As the Day shift, I want each Run's Repairs and CI rounds counted, so that I can see which setups meet the most friction.
3. As the Day shift, I want each Run's Unaddressed findings counted by kind (Standards, Spec and Security), so that I can see what agents choose not to fix.
4. As the Day shift, I want the review's findings counted by kind, so that I can see what the code review catches before a pull request is ready.
5. As the Day shift, I want each stage's time shown against its Usual range, so that a slow stage stands out in a Run's metrics as it does in the Inbox.
6. As the Day shift, I want each Run's minutes kept by stage and in total, so that efficiency sits beside accuracy.
7. As the Day shift, I want a Spec run's Ticket Runs and Base fixes to keep metrics of their own, so that each Run's friction is its own.

From its Session traces:

8. As the Day shift, I want each Run's tokens by Model, with the cache split, so that I can see what a Run used.
9. As the Day shift, I want cost shown only where a Harness reports one, with no price table of thirdshift's own, so that no figure is a guess.
10. As the Day shift, I want tool errors counted, so that I can see when a Harness or a Model fights its tools.
11. As the Day shift, I want permission denials counted, so that a setting that blocks agents shows up.
12. As the Day shift, I want killed work counted, so that I can see how often sessions leave background work behind.
13. As the Day shift, I want repeated tool calls counted, so that a session going round in circles shows up.
14. As the Day shift, I want destructive commands flagged by a code check on every Run, so that safety doesn't wait for a sample. Examples are a force push, or a recursive delete outside the worktree.
15. As the Day shift, I want a flag never to block or fail a Run, so that the check can't break the factory.

Bug origins:

16. As the Day shift, I want a confirmed Bug origin to count against the Run that made the bug, so that bugs a Run let through show in its metrics.
17. As the Day shift, I want a rejected or unconfirmed Bug origin to count for nothing.

Trends:

18. As the Day shift, I want thirdshift metrics to show weekly trends of every signal, so that I can see the factory change over time.
19. As the Day shift, I want those trends split by Harness, Model, Effort, Harness version, kind of session and repository, so that I can compare setups.
20. As the Day shift, I want every signal shown on its own, and never one combined score, so that no number hides what it's made of.
21. As the Day shift, I want --json, so that Charts on the Console: what to chart over time #813's charts and my scripts read exactly what the CLI shows.
22. As the Day shift, I want thirdshift metrics <command> to show one Run's metrics.

Past Runs:

23. As the Day shift, I want past Runs to get the metrics their Command records hold, so that trends start with history.
24. As the Day shift, I want past Runs to get their Session-trace signals once Spec: Session traces, OTLP JSON Lines in place of Session logs #700's conversion has turned their Session logs into Session traces.
25. As the Day shift, I want a signal a past Run never recorded shown as missing, not as zero, so that history doesn't look better than it was.

Where they're kept and shown:

26. As the Day shift, I want a Run's metrics written into its Command record when it ends, so that they stay with the Run they describe.
27. As the Day shift, I want each host to show its own Runs' metrics, since Session traces stay on the host that ran them.
28. As the Day shift, I want a Run's metrics in its details in the Console, and in show --json, so that I see them where I already look at a Run.
29. As a factory session, I want Run metrics never to read or send anything off the machine, so that the Console's rule that nothing is loaded from another site holds.

Maintenance:

30. As a maintainer, I want the Session-trace signals computed from one Session trace by itself, so that they can be tested against fixture traces for every Harness.
31. As a maintainer, I want the destructive-command patterns kept in one list in code, so that adding a pattern is one change with a test.

Implementation Decisions

Where metrics live:

• A new field of the Run's Command record. The Run's own process writes it when the Run ends, as it writes the rest of its record. The record's version goes up, and readers accept records without it.
• Every Run gets its own metrics, including a Ticket's Run and a Base fix.

From the Run:

• Outcome: as the record holds it.
• Repairs and CI rounds: counted from the record's Repair and CI watch stages.
• Unaddressed findings by kind: Standards, Spec and Security. They're read from the "Unaddressed findings" section of the pull request body when the Run ends, as Delivery last published it. The Security review's own count joins them.
• The review's findings by kind: the code review writes a short summary of its finding count per axis, Standards and Spec, beside its reports, so the count needs no parsing of prose. This is a small change to the code-review Factory skill and the Session prompt that asks for its reports. It changes what that skill does, so a release carrying it needs an Experiment once Spec: Experiments, which measure a change before it reaches production #977 exists.
• Stage times: each stage's time, and where it sits against its Usual range (below, within, above, or too few Runs to say). Minutes are kept by stage and in total.

From its Session traces (#700):

• Tokens by Model: input, output, cache read and cache write.
• Cost: only where the Harness reports one. There's no price table.
• Counts of tool errors, permission denials and killed work.
• Repeated tool calls: a call that uses the same tool with the same input as an earlier call in the same session.
• Destructive commands: a code check matches each session's shell commands against one list of patterns, such as a force push, a history rewrite of a shared branch, or a recursive delete outside the worktree. Matches are flagged with their command, and never block or fail a Run.
• Computed from one Session trace by itself. A Run's signals are the sum over its sessions, kept by session kind.

Bug origins:

• A confirmed Bug origin counts against its Run when metrics are read. It's taken from Spec: the Scoring run, which scores a weekly sample of production work #978's scoring data, so a confirmation that comes weeks later still counts. Unconfirmed and rejected Bug origins don't count.

The CLI:

• thirdshift metrics <command> [--json] shows one Run's metrics, or a Spec run's with its Ticket Runs.
• thirdshift metrics [--json] shows weekly trends over the machine's Command records. Every signal is kept apart, split by Harness, Model, Effort, Harness version, kind of session and repository, with options to narrow the weeks and repositories. Its JSON is what Charts on the Console: what to chart over time #813's charts draw from.
• show --json includes a Run's metrics, and the Console's details for a Run show them.
• No combined score. Neither the CLI nor the JSON adds signals together.

Past Runs:

• Past Runs' metrics are computed when read, from their Command records and from their Session traces once Spec: Session traces, OTLP JSON Lines in place of Session logs #700's conversion has run. Each process writes only its own files, so they're never written back into old records.
• A signal a past Run never recorded shows as missing. Its Unaddressed findings and review counts start with the switch.

Hosts and privacy:

• Each host reads only its own Command records and Session traces. Spec: the Scoring run, which scores a weekly sample of production work #978's Scores and Bug origins cover both hosts. The Session-trace signals cover only the host's own Runs.
• Nothing is read from or sent to another site.

Testing Decisions

• A good test checks what you or the Console see: the Command record's metrics, what thirdshift metrics and show --json print, and what the Console serves. It doesn't check internals.

• Seam 1: the whole binary against fake Harnesses. This is the existing integration scenario. Runs and Spec runs end, and tests assert:

  • the record's outcome, and its Repairs and CI rounds;
  • Unaddressed findings by kind, from a pull request body the fake gh holds;
  • the review's findings by kind, from the summary a fake review writes;
  • stage times against the Usual range;
  • thirdshift metrics --json for one Run and for trends, with past Runs' missing signals shown as missing;
  • a confirmed Bug origin from Spec: the Scoring run, which scores a weekly sample of production work #978's data counting against its Run, and a rejected one not counting.

  Once Spec: Session traces, OTLP JSON Lines in place of Session logs #700 lands, the same tests assert the Session-trace signals. Prior art: the integration tests of Command logs, show and logs, and Run health.

• Seam 2: Session-trace signals on their own. These are computed from one Session trace, and tested against Spec: Session traces, OTLP JSON Lines in place of Session logs #700's per-Harness fixture traces:

  • tokens by Model, with the cache split, and cost where it's reported;
  • tool errors, permission denials, killed work and repeated tool calls;
  • every destructive-command pattern, with a near miss for each that must not match.

• Seam 3: the Console's existing tests of a Run's details show its metrics.

Out of Scope

• The charts, which Charts on the Console: what to chart over time #813 decides after its own grilling, drawing from thirdshift metrics --json.
• Scores and their trends, which Spec: the Scoring run, which scores a weekly sample of production work #978's thirdshift scores keeps.
• A price table, or one combined score.
• Session-trace signals for the other host's Runs. Each host keeps its own.
• The SQLite index Charts on the Console: what to chart over time #813 asks about, until reading the Command records proves too slow.
• Counting the outcome, Repairs or CI rounds as quality on their own (Q1.1).

Further Notes

• Part of add evals -integrate with langfuse / oss eval tool? #698.
• Order: the Session-trace signals wait for Spec: Session traces, OTLP JSON Lines in place of Session logs #700's writer, and the Bug-origin counts for Spec: the Scoring run, which scores a weekly sample of production work #978's Bug origins. The rest can start at once.
• Safety: the destructive-command check is the safety check that Q4.7 left with code checks on every Run, rather than with Spec: the Scoring run, which scores a weekly sample of production work #978's sample.
• Tickets come next, with to-tickets. This Spec stays needs-triage until all of them and their blocking links are published.