[ifrit98]

Implements PRD-002 (plan): can a reader work out a result that an explanation never states? Also PRD-002a (plan): does that test notice when a building block is missing?

What's in it

• 30 rederivation targets in evals/artifact/questions.yaml. Each has a question the explanation was never asked to answer, 3–6 expected steps (building blocks plus derived steps), and a result.
  • A fresh agent checked every target (explainer eval targets, results in evals/artifact/rederive-checks.json).
  • All 30 were frozen in 6b61cdf, before any reader or grader code existed. A test pins them.
• The rederivation stage (explainer_kit/rederive.py).
  • A reader works the target out from the renderings, naming the part of the explanation each step rests on.
  • A blind grader marks each expected step as supported, unsupported, missing, or stated.
  • Every run makes three passes. The separation rule was fixed before the first run.
  • It runs with explainer eval rederive --builds <run>, and eval artifact now runs it too.
• A human check: explainer eval human plus evals/artifact/human-check.md.
• A dev baseline on the saved v0.8 builds, with no new builds: evals/artifact/results/2026-10-20-dev-rederive/. The v0.8 record is unchanged.

What it found (docs/evals.md)

• The stage is near its ceiling. 90–100% of steps are supported in every condition.
• One pair separates under the rule: none > published.
  • The gap is about 2 steps out of 38, all from one subject (two-tides).
  • There, the published explanation gives the Sun-to-Moon tide ratio but never says when the two tides add. The quiz already caught that gap.
  • The order roughly follows length.
• Reported alongside:
  • one target that overclaims (float-sum);
  • derived steps stated outright in 3 of 10 subjects;
  • compound building blocks graded inconsistently when only half present.
• Conclusion: this is not yet a measure of understanding beyond the quiz. PRD-002a, below, tests whether it notices a missing block.

Review

A final whole-branch review led to these fixes, each with a test that failed first:

• A failed build is now forced to score nothing, and the report shows it.
• A grader reply that doesn't fit is retried once, and its raw marks are kept.
• The test-split targets are pinned to the freeze.
• Two doc corrections: the gap behind the separation is half a building block, not a whole one, and the float-sum consistency claim was overstated.

PRD-002a: does the test catch a missing block? (docs/evals.md)

The baseline's near-ceiling score could mean good builds or a blind test. To tell them apart, one building block was cut from saved v0.8 principles explanations and the unchanged stage was run on the cut copies.

• Cuts (explainer eval cuts, explainer_kit/controls.py).
  • Six dev targets have an eligible block: one the question, background and other blocks don't supply, and that the baseline credited in every pass.
  • A fresh agent cut it, and a second checked that it couldn't be recovered and that the other blocks were intact. Each cut got up to 3 attempts.
  • Five passed. Bloom-filter's failed all three.
  • Cuts, checks and diffs were frozen in 6c8f028, before any run.
• Run (explainer eval controls). The cut copies were read fresh, and the baseline's reads were reused for the other conditions. All four were graded fresh, blind, over 3 passes. A leak check covers uncaught cuts.
• Bar, fixed in the PRD. At least 80% of counted subjects caught, at most one other block losing support, and at least five subjects counted.

What it found:

• Trusted, at the floor. 4 of 5 cuts were caught (4 needed), no other block lost support, and no leaks were found.
• The verdict rests on the grader treating rebuilt blocks inconsistently.
  • In heat-pump, the reader rebuilt the cut formula from a table the cut left in and got no credit: a catch.
  • In float-sum, a block rebuilt the same way was credited: the miss.
  • Had heat-pump been credited, the verdict would have been blind.
• On the four caught subjects, the baseline's ceiling reflects the builds, not a blind test; on float-sum it may not. The evidence is thin: five subjects, one model, a verdict at the floor.
• In greenhouse, key-exchange and two-tides, the reader's result went from right in 3 of 3 passes to 0 of 3.
• Four of the ten dev targets have no eligible block, so they can't show a missing block. The 20 test-split targets need the same sort before PRD-003 relies on them.

A final whole-branch review led to these fixes, each with a test that failed first:

• The cutter no longer sees other target steps. It had copied one into the two-tides copy; the frozen cuts predate the fix, and the docs disclose it.
• eval cuts refuses a folder that eval controls has run on.
• The write-up now says what the verdict rests on.

Test plan

• uv run pytest -q: 310 passed, offline.
• uv run explainer check passes.
• git diff main -- plugin/skills explainers evals/artifact/results/2026-10-04-dev* is empty.
• The PRD-002a freeze (6c8f028) comes before every run output.
• Results scanned for paths, usernames, emails, and keys: none.
• CI green.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DuVfjP13T9TDZwFJdP5uui