Repository navigation
feat(ci): Claude credential switch (subscription/oidc/api) + per-implementation daily regen - #12487
Merged
Merged
Conversation
…ode daily regen - CLAUDE_AUTH (repository variable): unset or `subscription` keeps every claude-code-action step on CLAUDE_CODE_OAUTH_TOKEN; `api` moves it to ANTHROPIC_API_KEY. Each of the 13 steps passes exactly one non-empty credential. A "Check Claude credentials" step opens every Claude job and stops on an unknown value or on `api` without the key, never falling back to the subscription. - "Claude cost to the job summary" after every Claude step: total_cost_usd, model and credential from the execution file, never fatal. collect_week.py --costs reads it, counts daily-regen too and splits out the API part. - daily-regen pair mode: REGEN_PAIRS_PER_DAY on the schedule, or the new `pairs` input, picks the N oldest (spec, library) pairs (automation/scripts/regen_pick_pairs.py), skips the spec polish and dispatches each pair straight to impl-generate with bulk-generate's arguments and pace. The watchdog's cron rescue forwards the variable. Unset, the schedule runs spec mode as before. - tests/unit/workflows/test_claude_auth.py pins the wiring and runs the check, cost, mode and dispatch scripts; docs and changelog fragment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
…-pick loop - CLAUDE_AUTH=oidc: workload identity federation through the pinned action's native inputs, with the CLAUDE_WIF_RULE_ID, _ORG_ID, _SERVICE_ACCOUNT_ID and _WORKSPACE_ID variables. "Resolve Claude credentials" (id claude_auth) replaces the check step and hands the mode and the federation IDs to every Claude step as outputs, so a new mode changes one step; it stops on a missing variable, a non-ID value or a job without id-token. No job gains id-token; review-retest stays without it. - "Claude session summary" (id <step>_cost) also reads, for api/oidc only, the API's spend-limit and credit messages and Claude Code's billing_error from the session's error fields. The retry is skipped, "Stop at the Claude spend limit" fails the job, impl-generate parks the pair (labels restored, own marker, no retry, no counted attempt, never :failed), impl-review and impl-repair hand the PR on without spending their auto-retry, and a daily-regen pre-flight at the limit skips that spec's dispatch. Subscription runs never take the new branches. - Pair mode: a pair's real attempts are its merged or closed implementation PRs and its failed generations (impl-fail comments without the infra tag, read lazily per spec issue); a pair touched within the attempt window (attempt_window_days, REGEN_ATTEMPT_WINDOW_DAYS, default 30) is skipped. Infrastructure failures and spend-limit stops are no attempt. - collect_week counts auth=oidc as API credits; docs, tests, changelog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
…ure stops in every mode - "Claude session summary" now detects the limit in all three modes, and only in the error fields of a session that failed (the last result is missing or is_error): the API's spend-limit (400 "specified [workspace] API usage limits"), tier spend-cap (429 enforced_spend_limit_reached) and credit messages, Claude Code's billing_error and "Credit balance (is) too low", and its subscription usage-limit messages (the SDK's USAGE_LIMIT_ERROR_PREFIXES, plus the older "usage limit reached"). A transient rate limit keeps today's retry and infra path. - Output limit_reached, ::error:: and the summary line "Claude spend/usage limit reached (auth=<mode>)"; the retry is skipped; impl-generate parks the pair before any attempt counting, for first implementations too (never impl:<lib>:failed); impl-review and impl-repair hand the PR on without spending their auto-retry; the pause comments carry their own *-paused-claude-limit markers, so the pair-mode pick never counts them. - daily-regen: each spec's similarity audit runs before its pairs are dispatched, so once a pair's chain reaches the limit, later specs of the night stop at their audit; pairs already dispatched fail fast and park. - Tests: fixtures with the API's and Claude Code's exact texts in every mode, and the negatives (transient rate limit, usage warning, quoted text, a session that recovered). Docs and changelog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
…ntials
Round-3 review fixes for the Claude credential switch:
- impl-review: a session Claude stopped gets no score, verdict or gate
decision even when it wrote quality_score.txt first ("Extract quality
score" is skipped on the stop).
- Stopped reviews and repairs get their own labels, ai-review-paused and
ai-repair-paused, instead of ai-review-failed; the watchdog skips paused
PRs, daily-regen's pair mode re-dispatches them first (oldest first,
counted against N), and the resumed run removes the label.
- With api or oidc, refused credentials (401/403, authentication_error,
permission_error, "Authentication failed", "invalid x-api-key") stop the
session like a limit: no retry, no counted attempt, never :failed. The
subscription keeps its handling. Outputs are now stopped/stop_reason.
- oidc routes the runs federation refuses (pull-request events, other
refs, jobs without id-token such as review-retest) to the subscription
with a notice, and fails fast when the OAuth token is missing.
- The three non-Claude steps that run model-written code in id-token jobs
unset ACTIONS_ID_TOKEN_REQUEST_URL/TOKEN first.
- regen_pick_pairs.py exits 1 when the implementation PR listing fails and
leaves out a spec whose failure comments can't be read.
- Docs: "When Claude stops a session", manual resume, the spend-limit test
on day one of oidc before REGEN_PAIRS_PER_DAY, and the federation
exposure (disable the Console rule when oidc is off, job_workflow_ref
pinning unverified).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
While a Claude stop lasts, every pair-mode night re-dispatches the paused reviews and repairs and each pauses again. impl-review and impl-repair now re-add the pause label but skip the comment when the PR's latest bot comment already starts with the same pause marker (spec, library, and for a repair the attempt). A failed read posts the comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Active moderate and critical findings remain: the implementation-branch listing can miss existing PRs, and invalid exception syntax prevents the pair-picker module and its tests from importing.
2 open issues
What changed in this PR
This pull request adds configurable Claude credential routing, session-cost reporting, clean handling of spend and authentication stops, and per-implementation daily regeneration across the CI workflows.
Changes:
- Adds
CLAUDE_AUTHsupport for subscription, OIDC, and API-key authentication. - Pauses or parks work when Claude reaches usage, spend, credit, or credential limits.
- Adds
REGEN_PAIRS_PER_DAYpair-mode regeneration with paused-work resumption and attempt tracking. - Updates documentation, automation scripts, workflows, and unit tests.
| File | Description |
|---|---|
tests/unit/workflows/test_regen_gate_workflows.py |
Updates regeneration gate expectations for Claude-stop paths. |
tests/unit/workflows/test_impl_generate_adopt.py |
Verifies stopped Claude sessions skip adopt-mode retries. |
tests/unit/skills/test_collect_week.py |
Tests credential-aware session-cost aggregation. |
tests/unit/automation/scripts/test_regen_pick_pairs.py |
Tests pair discovery, ordering, attempt windows, pauses, and fail-closed behavior. |
prompts/workflow-prompts/README.md |
Documents credential inputs for Claude workflow actions. |
prompts/README.md |
Updates the general Claude workflow wiring example. |
docs/workflows/overview.md |
Documents credential modes, stop handling, pair-mode regeneration, and operational procedures. |
changelog.d/ci-claude-auth-switch.md |
Adds the changelog fragment for the feature. |
automation/scripts/regen_pick_pairs.py |
Implements pair selection, attempt tracking, and paused-work resumption. |
automation/scripts/README.md |
Documents the pair-picker script and its interface. |
agentic/docs/project-guide.md |
Updates the workflow architecture guide with credentials and pair-mode behavior. |
.github/workflows/watchdog-stuck-jobs.yml |
Skips paused PRs and forwards pair-mode settings during cron rescue. |
.github/workflows/util-claude.yml |
Adds credential resolution, cost reporting, and stop handling. |
.github/workflows/spec-create.yml |
Adds configurable credentials, cost reporting, and stopped-session handling. |
.github/workflows/review-retest.yml |
Adds credential resolution and cost reporting for retest sessions. |
.github/workflows/report-validate.yml |
Adds configurable credentials and session-stop reporting. |
.github/workflows/impl-review.yml |
Adds credential routing, cost reporting, and paused-review handling. |
.github/workflows/impl-repair.yml |
Adds credential routing, cost reporting, and paused-repair handling. |
.github/workflows/impl-generate.yml |
Adds credential routing, cost reporting, and generation parking on Claude stops. |
.github/workflows/daily-regen.yml |
Adds pair-mode selection, paused-work resumption, and per-pair dispatching. |
.claude/skills/review-regen-week/SKILL.md |
Updates weekly regeneration-review guidance for pair mode and API costs. |
.claude/skills/review-regen-week/collect_week.py |
Aggregates session costs by credential mode, including daily regeneration jobs. |
Review effort: Lite
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The main() tests ran the picker as a subprocess, which coverage doesn't see. In-process tests with a fake runner now cover the gh listing wrappers, read_records and its fail-closed implementation-PR listing, issue_failures, the paused-label parsing edges, summary_table, argument checks and the GITHUB_OUTPUT and step-summary writing. No behaviour change; regen_pick_pairs.py is at 100 % line and branch coverage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
6 tasks done
MarkusNeusinger
added a commit
that referenced
this pull request
Oct 31, 2026
) ## Summary - **Watcher scripts end on their own.** `watch-deploys.sh` exits 0 with `no deploy for <sha>` after 6 empty polls (157 s measured on 523199b, which triggered no deploy), now passes `--project=anyplot` (the project guide's rule; a wrong default project would otherwise read as "no deploy"), and exits 2 at once when `gcloud builds list` fails. `watch-checks.sh` waits 90 polls (about 45 minutes) and its `TIMEOUT` line names the checks still pending. Both caps are stated in `open-pr` sections 3a and 4. - **Rules and skills from the retro's findings** (owner-approved list, items A1 to A14 and B15 to B20): `CLAUDE.md` gets the delegate stall check with a post-compaction re-check, a classifier denial that is final for every agent of the session, the first Copilot review at about 5 minutes, and merges only on a named OK (mirrored in `copilot-instructions.md`); `open-pr` gets a hadolint row (CI's 2.15.1, threshold `warning`), tests for new code before the PR opens (the codecov triage folded into section 1), the 16 real check names, the GraphQL `requestReviews` mechanic, the nvm prefix for site-smoke, a `pull --ff-only` of the shared checkout and a once-per-session worktree report; `zweitmeinung` gets the 60 KB diff rule, the `--attach` path gotcha, ledger-measured costs and the root-owned-files cleanup; `verify-core` the system-`python3` gotcha; `guardrails.md` the delegation, denial, Console-wizard and stalled-delegate sections. - **Recipes that lived only in chat:** the single-library `impl-generate.yml` dispatch with its optional inputs (`docs/workflows/overview.md`), the eval-matrix forms real runs used (`agents/README.md`), and the describe, edit, import route for a second-generation Cloud Build trigger (`agentic/docs/project-guide.md`). Every gate command that runs ruff now names the `dev` extra. Decisions taken within the brief: - The trigger route went into the project guide's deployment section, next to the existing "Checking whether something actually deployed"; `infra/` holds only Cloudflare. - The hadolint row covers all five Dockerfiles CI lints, not only the renderer and agents ones. "Must print nothing" became "no `warning` or `error` line": `agents/Dockerfile` and `toolchain.Dockerfile` print `info` lines (DL3059, DL3066) on main today, below CI's threshold. - `--extra dev` went into every skill, command and doc that runs ruff (also `docs/development.md`, `project-guide.md`, `copilot-instructions.md`, the audit and test commands); historical plans under `agentic/specs/` and `agentic/audits/` and the CI workflow (which syncs `--extra dev`) stay as they are. - The classifier-denial rule has no mirror in `copilot-instructions.md`: the classifier is a Claude Code harness mechanism, and that file carries no external-writes rule to extend. - The Gemini cost lines in `guardrails.md` were updated with the same ledger figures, so the skill and the guardrails agree. - `agents/README.md` said `--thinking-budget` takes 0 to 4,096; `matrix.py` caps it at 8,192 (`MAX_THINKING_BUDGET`). Fixed while there. ## Plan N/A. The source is the retro of session 402df83f (2026-10-28 to 2026-10-31), sections 3, 4 and 6: ranked findings, proposed changes, and the first-try accuracy table of documented commands. ## Test plan - [x] `watch-deploys.sh 523199b` (a `.github/workflows/` merge): `no deploy for 523199b`, exit 0 after 157 s; `f356a57b1` the same; `e9b11e373` (two deploys): `ALL DEPLOYS FINISHED for e9b11e3, 0 not SUCCESS`, exit 0 after 32 s - [x] `watch-checks.sh 12487`: exits at once, `ALL 16 CHECKS FINISHED on 3a6d00a, 0 not green`; the TIMEOUT name parsing checked on sample lines - [x] The worktree report and the `pull --ff-only` guard run read-only against the shared checkout; the `git diff` exclude pathspec and the hadolint command run locally (all five Dockerfiles exit 0) - [x] `RequestReviewsInput` lists `botIds` (GraphQL introspection); the Copilot bot id `BOT_kgDOCnlnWA` read from #12487's review - [x] `uv run --extra dev ruff check . && uv run --extra dev ruff format --check .`; `uv run python -m tools.changelog check --base origin/main` - [x] Gemini 3.8 Flash pre-PR check ($0.38): 6 findings, 1 real (detached worktrees missing from the report) fixed in 39f906e; `botIds` and the hadolint entrypoint were false positives (verified), the rest pre-existing or no false result https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CLAUDE_AUTHswitch for all 13 Claude steps in 8 workflows. Unset/subscription= today (CLAUDE_CODE_OAUTH_TOKEN);oidc= GitHub OIDC workload identity federation into the Console workspaceci-regen(the fourCLAUDE_WIF_*repo variables are already set; the Console rule accepts only runs onmain), so the nightly regen runs on the subscriber API credits instead of the owner's interactive quota;api=ANTHROPIC_API_KEYas a fallback. A "Resolve Claude credentials" step per job decides; each Claude step passes exactly one credential; bad values fail loudly, never a silent fallback. Underoidc, runs federation would refuse (PR review events, other refs,review-retest) use the subscription and say so.total_cost_usd, model and auth mode. Spend-limit / spend-cap / credit / subscription usage-limit errors (and, under api/oidc, refused credentials) are infrastructure stops: no retry, no counted attempt, neverimpl:<lib>:failed; reviews and repairs pause withai-review-paused/ai-repair-paused(no score from a cut session), the watchdog skips paused PRs, one comment per pause.REGEN_PAIRS_PER_DAY: daily-regen picks the N oldest implementations (spec × library) instead of whole specs; no re-pick loop (last real attempt within a 30-day window counts), paused PRs resume first, the pick fails closed. Unset = the old spec mode.daily-regen.ymlstays disabled.Plan
docs/workflows/overview.md("Claude credentials", "When Claude stops a session", "Daily regeneration modes"); the Console setup was done by the owner on 2026-10-31.Test plan
tests/unit(23,146 passed) incl.tests/unit/workflows/test_claude_auth.py(pins all 13 sites' credential expressions for the three modes, the resolve/summary scripts run under bash with fake gh, the stop/park/pause paths, id-token unchanged per job) andtest_regen_pick_pairs.py; ruff, mypy, changelog checkCLAUDE_AUTHunset: one ordinary regen pair (bulk-generate.yml -f specification_id=<spec> -f library=<lib>) behaves as before, summary showsauth=subscriptionwith a costCLAUDE_AUTH=oidc: one pair, Console usage lands inci-regenREGEN_PAIRS_PER_DAY=2claude-code-action refuses to run on a PR that changes Claude-running workflows, so the live paths are verified by the post-merge steps above.
https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8