[shirgoldbird]

Rebased onto main after #479 merged. #479 fixed the gap-path bug that #450 targeted, so #450 is closed and its remaining fixes are folded in here (see "Carried over from #450" below).

Summary

Running the pipeline end to end (which I should have done before opening #450) turned up a third bug. The generation step didn't return a page — it returned a transcript of itself using tools:

I'll research the GitHub repo and peer pages before writing.
<tool_call>{"name": "Glob", "arguments": {"pattern": "docs/learning-how-tos/cookbook/**/*.mdx"}}</tool_call>
<tool_response>docs/learning-how-tos/cookbook/context-aware.mdx ...</tool_response>

build_authoring_system_prompt pastes .claude/agents/docs-writer.md verbatim into a plain Messages API call. That file is an agent definition:

• frontmatter declares tools: Read, Write, Edit, Grep, Glob, Bash, Agent
• the body says "Search with Grep and Glob" and to "delegate to the diataxis agent (via the Agent tool)"
• it closes with "Return the complete draft as file content (using Write or Edit)"

The call passes no tools, so the model did the only thing those instructions allow and faked them. diataxis.md and editorial-reviewer.md declare tools too, so rework.py and review.py shared the exposure.

This is pre-existing and intermittent — #447's pages came out as valid .mdx through the same assembly. #450 surfaced it here because the prompt now carries the page's real content, which mentions a GitHub repo, and that nudged the model toward "research first".

What changed

• load_agent() strips the YAML frontmatter from every agent file.
• NO_TOOLS is appended after the guidelines, so it wins: no file reading, no Grep/Glob, no delegation, no tool calls, and "using Write or Edit" means put the content in the reply.
• generate.py and rework.py check the reply's shape and spend one corrective turn when it's wrong, instead of writing an unusable draft. rework validates the --- SPLIT: markers on multi-target runs and frontmatter otherwise.

Two decisions worth reviewing:

• No salvaging. Cutting the page out of a transcript at the first --- is tempting and wrong: a faked tool result can quote another page's frontmatter, so salvage risks promoting spliced content. A corrective turn costs one call and can't mangle anything.
• Prefill would have been stronger, and isn't available. Seeding the assistant turn with --- makes a preamble structurally impossible, but the API rejects it: This model does not support assistant message prefill. I found that by running it.

Verification

Full run on the one live gap — the thin_page for cookbook/google-sheets, the same gap that produced the transcript:

Step	Result
generate	valid .mdx draft
evaluate	0 errors, 0 warnings
review	clean, 5 fixes applied
promote	wrote docs/learning-how-tos/cookbook/google-sheets.mdx

run.py exited 0. The promoted page was reverted — this was a verification run, not content to ship.

New pipeline/test_prompts.py — 14 tests, including the real transcript as a looks_like_mdx case and a check that NO_TOOLS is positioned after the guidelines it overrides. 55 tests total in pipeline/.

Two things this run exposed that I did not fix

1. Nothing gates on broken links. promote.py runs mint broken-links and prints the result, but doesn't fail on it. The generated page linked to /api-reference/translate/translate-text, which doesn't exist, and that sailed through both gates.
2. Without tools, the model fills gaps from priors. The page asserted the example repo's file layout (Code.gs, Sidebar.html, appsscript.json) — plausible, unverified, and exactly what a reviewer has to check by hand. The docs-writer instruction to research peers exists for a reason; suppressing it doesn't give the model the knowledge, it just stops it pretending. Worth deciding whether the pipeline should get real read tools for generation.

Carried over from #450

#479 fixed the gap paths. These parts of #450 it didn't cover are in the second commit:

• Hub pages aren't flagged as thin. A page with child pages is short because its content is on the children. Without this, the next run would ask the model to pad out breaking-changes-change-notices.
• load_existing_page() raises when the page a gap edits is missing or empty, so a failed read can't turn "expand this page" into "write it from scratch".
• Lazy Anthropic SDK import in generate.py, so --dry-run and the tests run without the SDK. CI now installs pyyaml, because the new tests import detect_gaps and generate.

Verified after the rebase: 52 tests pass. generate.py --dry-run runs in a venv with only pyyaml and leaves one generative gap (thin_page for cookbook/google-sheets). supported-languages is skipped as already changed in #403, and breaking-changes-change-notices is no longer flagged.

🤖 Generated with Claude Code

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
deepl-c950b784	🟢 Ready	View Preview	Oct 28, 2026, 6:55 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

[kabraham13]

Approving. This fixes the root cause (agent definitions that declare tools being pasted into a no-tools API call), the corrective turn and the decision not to salvage a page out of a transcript are well reasoned, and the live run on the real gap is good evidence. CI is green and it's rebased on current main.

Three non-blocking suggestions:

1. The corrective-turn path has no test. The tests cover looks_like_mdx and the retry prompt text, but nothing calls generate_content (or the rework flow) with a fake client to check that a bad first reply produces the user, assistant, user sequence and returns the second reply. The sequence looks valid on a read, so a small fake-client test would pin it.
2. A second bad reply isn't re-checked. After the retry, the reply is returned as is. If the model returns a transcript twice, it gets written as a draft. I'd expect evaluate to catch it and fail the run, but it would be worth either confirming that or failing fast in generate_content.
3. pip install pyyaml in the test workflow is unpinned. The old step was stdlib only. A pin, or a small requirements file, keeps CI reproducible.

The two gaps called out in the description (nothing gates on mint broken-links, and no read tools for generation) are worth tracking as follow-ups, but they don't need to block this.

[shirgoldbird]

Thanks, all three addressed in 343effe:

1. Corrective-turn test: pipeline/test_prompts.py now drives generate_content with a fake client. A good reply takes one call, a bad one produces the user/assistant/user retry and returns the second reply, and descriptions aren't shape-checked.
2. Second bad reply: evaluate would catch it (frontmatter_present is an error), but that fails the whole run, so one bad gap would block every valid draft. generate_content now raises instead, and main() records it as an error for that gap only. rework.py exits with an error in the same case. Covered by a test.
3. pyyaml: pinned to 6.0.3 in the test workflow.

Agreed on tracking the broken-links gate and read tools for generation as follow-ups.

[kabraham13]

Re-approving after 343effe. The fake-client tests pin the corrective-turn sequence (user, assistant, user), a second bad reply now raises so it only errors that gap instead of failing evaluate for the whole run, and rework exits 1 in the same case. The pyyaml==6.0.3 pin is in, and CI is green.