[rmcdaniel]

Release mechanics for native 3.0.0, following the operator decisions of November 1 (recorded on #325): latest moves to native at the final release, and 3.0.0 ships without prepared local activities, workflow-bound Nexus handlers and external payload takeover. Compose and Helm defaults are a separate PR.

Rolling tags on a final native release

• New scripts/ci/native-release-rolling-tags.sh runs last in the publish job, only for a final release (the plan job's Docker Hub path). It points X.Y, X and latest at the attested index in ghcr.io/durable-workflow/server and durableworkflow/server, then checks each alias names exactly that index.
• Each alias moves only to the highest final native release it covers. X.Y follows the highest native-X.Y.*, X the highest native-X.*, and latest the highest native release when it is also above every PHP release. A backport or older patch leaves an alias where it is.
• Release candidates move nothing.

PHP releases never move latest back

• Native git tags are native-*, which PHP's plain-semver logic ignored. A PHP 2.11.9 after native 3.0.0 would therefore have moved latest back to PHP.
• resolve- and promote-release-image-rolling-tags.sh now leave latest alone whenever a final native-X.Y.Z tag is above the PHP release, and still move 2 and 2.Y. A native release candidate alone doesn't take latest.
• The publish evidence lists latest only when it moves (ROLLING_LATEST_ALIAS).

DW_RUST_EXPERIMENTAL

• A release build, with its version built in from the native- tag as DW_SERVER_VERSION, starts without the opt-in. A development build without a release version still requires it, so an untagged source build can't pass for a release. Setting it remains harmless.
• native-image-smoke.sh keys on the image's dev.durable-workflow.release.tag label. A development image must refuse to start without the opt-in; a release image runs the smoke without it.
• The takeover runbook, upgrade doc and rust/README.md are updated to match.

Port doc

• The header now describes native as the 3.x replacement.
• "Remaining for the release candidate" is replaced by "Remaining for the final release", with the scope and publication decisions and the current open items.

Checks run

• bash scripts/ci/check-native-release-tags.sh passes with new regressions, using a fake docker: first final, candidate, backport below a newer minor, older patch, newer minor, both registries from the index, an invalid digest, PHP after a native final (no latest) and PHP after a native candidate only (latest). With the PHP guard disabled, the check fails on the PHP-after-native-final case.
• ReleaseImageContinuationTest gains a PHPUnit case for the same rule. The rolling tests pass (3 tests, 23 assertions) under PHPUnit 11.5 on PHP 8.4. ReleaseImagePublishWorkflowContractTest locally fails the same 33 tests as a clean origin/main checkout; they need jq, node or docs-audit inputs that the bare PHP image lacks. CI runs both fully.
• Native binary, built locally:
  • development build: refuses to start without the opt-in with the new message, and with the opt-in proceeds to configuration;
  • build with DW_SERVER_VERSION=3.0.0-guardtest: proceeds to configuration without it.
• test_helm_chart_release.py passes.
• No real tag or registry write was made.

🤖 Generated with Claude Code

[rmcdaniel]

Security exact-head finding on a7046050c7f9f04649d20f99fa25332d5cf7330a: the new rolling-alias stage is not recoverable after a partial registry write.

The publish job first creates immutable exact tags and attestations. It then runs native-release-rolling-tags.sh, which mutates GHCR aliases before Docker Hub aliases and verifies only after both create calls. If Docker Hub creation or any later inspection fails, GHCR may already expose 3.0, 3 and latest while Docker Hub remains stale. A normal failed-job rerun starts the same publish job again, and native-release-publish.sh intentionally refuses as soon as either exact 3.0.0 tag already exists, so the rerun never reaches the rolling stage.

I reproduced this only with a disposable fake docker command: GHCR alias creation returned success, Docker Hub creation returned status 42, and a simulated normal rerun exited 1 with ghcr.io/example/server:3.0.0 already exists. No network or registry write occurred. The PHP protection compounds the stranded state: it sees the immutable native-3.0.0 Git tag and refuses to move latest, even on the registry whose native alias never advanced.

Required bounded correction: make alias reconciliation independently retryable from the already verified exact index. A separate post-publication job that re-verifies both exact tags equal the attested index before reconciling aliases is the clearest shape; alternatively, make exact publication idempotently admit only an existing index that exactly matches the two smoked digests and then continue. On every retry, recompute monotonic eligibility, converge every intended alias in both registries, and verify all aliases. Add fake-registry controls for failure between registries and failure during post-write verification, followed by a successful retry proving both registries converge without replacing the exact tag. Preserve a failed alias-stage receipt until convergence.

This is a #919 publication-path hold only, not a hold on unrelated Rust work or current PHP releases. Reviewed rolling script SHA-256 6922f99e730855399ed40108c09754e8b3868634d0eae204a3f33b24e84e5cc0; exact publisher SHA-256 85bf42637a8573cbe9d9ae6d9eff5070edd6a6b6df755c68ada580ea8a6a1c25.

[rmcdaniel]

Evidence-integrity follow-up on the same exact head: native-image-smoke.sh decides that an image is a release solely from a nonempty dev.durable-workflow.release.tag label, but never validates that label or binds it to the version compiled into the binary. The smoke reads /api/cluster/info for all three backends, then records only the number of keys; it never asserts version == ${release_tag#native-}. A build with any other nonempty DW_SERVER_VERSION can therefore start without the opt-in and pass this release smoke while advertising the wrong server version. The receipt also hardcodes opt_in_required:true for both development and release images, contradicting the behavior this PR is qualifying.

Please make the evidence self-consistent: for a release image, require a strict admitted native release-tag label, derive the expected version, assert every backend cluster-info response reports exactly that version, and record opt_in_required:false plus the release tag/version. For an unlabeled development image, preserve the no-opt-in refusal and record opt_in_required:true. Add a mismatch control proving a release label cannot bless a binary compiled with a different nonempty version. The workflow expressions currently intend the same version, but the smoke must independently prove the built artifact preserved that identity.

This is part of the bounded #919 correction, not a second unrelated hold. Reviewed smoke SHA-256 95d09ffefa929e024a32f8f447c0b2519803f285787a8f26009067fcad736bf2; main SHA-256 76d59649bc5c343b908961a31c484c5cb17b81e4819c2be187b27c202e35ac59.

[rmcdaniel]

Same exact head, same bounded alias-stage hold: the claimed monotonicity is also vulnerable to concurrent native releases. Workflow concurrency is native-release-${{ inputs.tag }}, so native-3.0.0 and native-3.1.0 publish jobs may run together. native-release-rolling-tags.sh snapshots Git tags once before writing. If the older job snapshots before the newer tag exists, the newer job can move 3 and latest first, then the older in-flight job can overwrite them with 3.0.0 and successfully verify its own older digest. Nothing rechecks eligibility against a serialized current state.

The retry-safe reconciliation stage should therefore own one shared repository-wide rolling-alias concurrency group across native releases, and ideally the PHP rolling stage as well, with a fresh tag/alias eligibility check after acquiring that serialization and immediately before writes. Add a deterministic two-release interleaving control proving an older job cannot finish last and move 3 or latest backward. This folds into the correction in comment 6560018912; it is not a second unrelated hold.

[rmcdaniel]

Security correction required — final release still admits unauthenticated administrator access

Reviewed exact head a7046050c7f9f04649d20f99fa25332d5cf7330a.

This head removes the experimental startup gate for release builds, but it also continues to accept DW_AUTH_DRIVER=none. That driver deliberately admits every request as an anonymous administrator. The final image sets DW_BIND_ADDRESS=0.0.0.0:8080, so an operator carrying the documented development-only setting into the final image gets a network-exposed administrator API with no request authentication.

This is the production-boundary condition already recorded in the Security review of #627: none was accepted there only because DW_RUST_EXPERIMENTAL=1 still fenced the runtime, with an explicit requirement that it “must not become an accidental deployment default when the experimental gate is eventually removed.”

Concrete path at this head:

• release_build() skips DW_RUST_EXPERIMENTAL when the binary has a nonempty compiled DW_SERVER_VERSION;
• validate_auth_configuration() accepts both token and none, including the legacy setting name;
• anonymous_driver() reaches the runtime, whose HTTP admission path creates an admin principal for every request;
• the image binds that runtime to all interfaces.

Required before final-release acceptance:

1. Preserve an explicit release boundary for the none driver. The smallest safe correction is for release builds to refuse both primary and legacy none settings before storage opens or the listener binds. If no-auth must remain available in a final binary, it needs a separate unmistakable development-only opt-in and a safely constrained bind, not inheritance from the removed global gate.
2. Add controls proving a versioned release build refuses the no-auth profile before storage/listen, while a development build with the experimental opt-in retains the intended parity behavior. Keep a release token control passing.
3. Make the Rust README and Server reference agree with the actual boundary; today the reference calls none development-only while the release path accepts it.

No live service, registry, credential, or network-auth test was performed. This is source-traced from the exact head and the previously recorded acceptance condition.

[rmcdaniel]

Security source review — workflow trust and credential boundary

Reviewed exact head a7046050c7f9f04649d20f99fa25332d5cf7330a. No additional finding in this bounded slice beyond the four corrections already recorded on this PR.

Covered the final native publisher's dispatch, permissions, source binding, registry credentials and attestation path:

• workflow-wide permission is contents: read; build jobs add only packages: write; only the publish/attestation job adds id-token: write and attestations: write;
• release tooling is checked out from the main-branch workflow SHA, while the release source is a separate tag checkout whose local HEAD and public peeled tag target must both equal the full lowercase planned commit;
• dispatch tag and commit inputs are passed through environment/action inputs rather than interpolated into shell source; the plan and source verifier enforce the native semver form and full SHA before the build jobs use them;
• every third-party action is immutable-SHA pinned;
• the release build context's allowlist excludes Git metadata and the checkout credential path; it includes only the Rust crate material needed by the pinned Dockerfile;
• Docker Hub/GHCR credentials are action inputs, not build arguments or image environment, and the smoke containers receive no registry credential or Docker socket;
• platform outputs must be lowercase SHA-256 digests, the created index must contain exactly the two smoked architecture digests, and the attestation subject is that verified index digest;
• release candidates still cannot select Docker Hub, while a final release cannot skip it.

Local bash -n passed for the publisher, rolling-tag, smoke, source-verification and evidence scripts. actionlint and shellcheck are not installed here, so no claim is made for those tools; hosted workflow parsing/checks remain authoritative.

This clearance does not waive the existing partial-publish recovery, concurrent alias ordering, tag-to-compiled-version smoke, or final-release no-auth boundary corrections. It adds no new hold and performs no registry, credential, service, or hardware-key operation.

[rmcdaniel]

Nonblocking documentation correction — keep the takeover API local or behind TLS

At exact head a7046050c7f9f04649d20f99fa25332d5cf7330a, the takeover runbook starts the final native API with:

docker run -d -p 8080:8080 $source -e DW_AUTH_TOKEN "$IMAGE"

The credential handling is good: -e DW_AUTH_TOKEN and -e DB_PASSWORD copy values from the invoking environment without putting their values in command arguments. The exact-image, stopped-writer, backup/fingerprint and no-in-place-rollback guidance is also fail-closed.

The host bind is not a safe default for an operator procedure. Docker documents that omitting the host IP publishes the port on all host interfaces and calls port publishing insecure by default. The API is plain HTTP and uses bearer credentials; the Server reference separately says to put TLS and public routing in a reverse proxy.

Requested docs-only correction:

• make the single-host example -p 127.0.0.1:8080:8080 (and add the IPv6 loopback form only when needed);
• say that remote access must publish only to the intended protected interface or private network and terminate HTTPS in the documented reverse proxy; do not expose the raw API port to an untrusted network;
• preserve the current environment-by-name credential pattern and digest-pinned image.

Official Docker behavior: https://docs.docker.com/engine/network/port-publishing/

This is a documentation hardening correction, not a new source/release hold and not a claim that the existing authenticated API is compromised. No container, listener, credential, or live service was touched. The previously recorded no-auth release boundary remains the separate blocking issue.

[rmcdaniel]

General's PHP publisher review at a7046050c7f9f04649d20f99fa25332d5cf7330a found one additional failure path in scripts/ci/promote-release-image-rolling-tags.sh and the corresponding resolver.

The new native_final_owns_latest() calls collect_release_tags inside a pipeline and treats an empty result as permission to promote latest. A failure from git ls-remote is lost in that pipeline. The promoter also fetches the tag inventory separately for the PHP ordering check and the native ownership check.

I reproduced this in an isolated container using fake git and docker, with no network or real registry writes:

• First tag lookup succeeds with 2.11.9 and native-3.0.0.
• The second lookup fails with exit 73.
• The script prints the lookup error, then invokes both registry promotions with --tag ...:latest from the PHP digest and exits 0.

This would move latest back to PHP despite an existing native final release. Please acquire the tag inventory with an explicitly checked status after the shared publication lock, derive PHP ordering and native ownership from that successful inventory, and refuse all alias writes when inventory acquisition fails. Add a regression that makes the lookup fail at this boundary and asserts nonzero exit and zero registry writes. The resolver should likewise fail rather than claim native ownership is absent when the inventory is unavailable.

This finding is scoped to #919. Security's separate serialization, partial-publication recovery, release smoke and authentication findings remain independently applicable. The intended continued 2 / 2.Y promotion is otherwise represented in the reviewed PHP changes.

[rmcdaniel]

Corrections for every finding at a7046050, now at head a3f78864:

1. Partial alias write (Security). Alias reconciliation is its own aliases job (needs: [plan, publish], final releases only), so "Re-run failed jobs" reruns it alone. It first checks that both exact tags name the attested index from publish's output, writes only aliases, and an alias that already names the index is rewritten unchanged. Controls: rerun after a partial write, and exact tag mismatch (fails, no writes).
2. Concurrent releases (Security). The job holds one repository-wide group (durable-workflow-server-rolling-aliases, no cancel-in-progress). Right before writing, the script also reads each alias's current release (its image's dev.durable-workflow.release.tag) in both registries, and leaves any alias that names a newer release. An alias that can't be read stops the run before any write. Control: older release finishing last, where 3.0.1 read Git tags before native-3.1.0 existed and 3.1.0 already holds 3/latest. Only 3.0 moves, and with the registry check disabled the control fails. The PHP stage keeps its own job; native final tags exist in Git before native publishes, and PHP now reads them fail-closed (5).
3. Smoke identity (Security). New scripts/ci/native-release-identity.sh admits only native-X.Y.Z or native-X.Y.Z-rc.N. The smoke requires every backend's /api/cluster/info version to equal the label's version, and records release: {tag, version, no_auth_refused} and opt_in_required as they are. Controls in check-native-release-tags.sh: malformed labels are refused, and a native-3.0.0 label doesn't admit 3.0.1, 3.0.0-rc.3, empty or 0.1.0.
4. No-auth driver (Security). A release build refuses DW_AUTH_DRIVER=none, and the legacy WORKFLOW_SERVER_AUTH_DRIVER=none, in validate_auth_configuration, before storage opens or the listener binds. A development build keeps it. Unit test a_release_build_refuses_the_none_driver; the release smoke checks both names and that no SQLite file is created. Rust README and Server reference updated.
5. Tag lookup (General). The promoter and resolver read the inventory once with a checked status, and derive both the PHP ordering and native ownership from it. Control: a failing git makes both scripts exit nonzero with zero registry calls.
6. Runbook (Security, docs). -p 127.0.0.1:8080:8080, with guidance to publish only to a protected interface and terminate HTTPS in the reverse proxy.

Run locally

• check-native-release-tags.sh passes all of the above.
• Rolling PHPUnit tests: 3 tests, 23 assertions.
• cargo test startup_tests: 2 passed.
• The full release-path smoke against a stand-in image: my locally built binary with DW_SERVER_VERSION=3.0.0-rc.99, in Dockerfile.native's runtime config on Ubuntu 24.04 for glibc, labelled native-3.0.0-rc.99. It passed on SQLite, PostgreSQL and MySQL, with each reporting 3.0.0-rc.99, no_auth_refused: true and opt_in_required: false.
• The same binary labelled native-3.0.0-rc.98 failed: the image is labelled native-3.0.0-rc.98 but its server reports version '3.0.0-rc.99'.

The development-image smoke path runs in this PR's Native image CI. No real tag or registry write was made. Re-review requested from Security and General.

Not changed, for awareness: native cluster info still reports "development": true for every build. Whether a release build should say otherwise is a separate parity question, not included here.

[rmcdaniel]

General's PHP publisher rereview at a3f7886483e8f5eb79667208bfc86c80b7456eda clears the tag-inventory failure reported in 6560160972.

Both scripts now read one explicitly checked inventory and derive PHP ordering/native ownership from it. Six isolated fake-registry controls pass: promoter and resolver each preserve PHP 2 / 2.Y without latest after a native final, retain latest after only a native candidate, and exit nonzero on inventory failure with zero registry writes. Every case asserts exactly one inventory read; the fake fails any second read. No real registry writes were made.

The original shared-serialization correction from Security's review is still incomplete on the PHP side:

• Native's new alias job uses durable-workflow-server-rolling-aliases.
• .github/workflows/release.yml still has only workflow concurrency release-${{ inputs.tag || github.ref_name }} and no job using the shared group.

Therefore a PHP promoter can snapshot an inventory without a native final, then write PHP latest after a concurrently created/published native final moves that alias. A one-time checked inventory alone does not serialize these two publishers. Please put PHP promotion behind the same repository-wide alias group and acquire its inventory there. This continues the existing alias-race correction rather than adding an unrelated release gate. The native/auth/smoke/recovery portions remain with Security.

[rmcdaniel]

Security focused rereview at exact head a3f7886483e8f5eb79667208bfc86c80b7456eda clears the native/auth/evidence portions of the prior hold:

• alias reconciliation is a separate rerunnable job that rebinds both immutable exact tags to the attested index before any alias write; a partial cross-registry write converges on rerun without reopening exact publication;
• one native-release-wide alias group plus a fresh per-registry alias-label read prevents an older native release from moving 3 or latest backward, and unreadable alias state fails before writes;
• the smoke admits only a strict native final/RC label, binds every backend's reported version to it, and records the actual release/opt-in/no-auth disposition;
• release builds refuse both primary and legacy none authentication settings before database open or listener bind, while development behavior remains explicit;
• the takeover example now binds loopback and states the protected-interface/TLS boundary.

Local source controls: bash -n passed for all six changed publication/smoke scripts; check-native-release-tags.sh passed its fake-registry partial-write, interleaving, unreadable-state, exact-tag, identity and inventory-failure controls; direct identity match/mismatch controls and git diff --check passed. No Rust build was run on the lab host. Exact-head native image smokes pass on amd64/arm64; Actions/JS/Python CodeQL, action policy, repository contracts, public-boundary checks and Sonar are green. Rust CodeQL and the broader backend matrix are still running and remain owner qualification.

The PR is not yet fully cleared: General correctly confirmed that the PHP release promoter is still outside durable-workflow-server-rolling-aliases, so the existing cross-workflow race remains (#919 (comment)). That is the remaining correction; this review adds no second finding and does not duplicate General's PHP publisher slice. Return the shared PHP alias-stage serialization/in-lock inventory delta for focused final review.

[rmcdaniel]

Security focused final review accepts exact head 53cbdb7287a7a6faace54f85913951a27d76f630; the remaining cross-workflow alias race is corrected and the Security hold is cleared.

I reviewed only a3f78864..53cbdb72. The PHP publish job now holds the same repository-wide durable-workflow-server-rolling-aliases group as native alias reconciliation, with cancellation disabled. Because the job owns that lock before its steps run, its resolver acquires one explicitly checked release-tag inventory inside the shared serialization boundary. The promoter receives that exact inventory through RELEASE_IMAGE_KNOWN_TAGS, no longer receives a tag-source URL in the workflow, and refuses missing, empty, or failed inventory before either registry call. This closes the previously demonstrated ordering in which PHP could snapshot before a native final and then move latest after native publication.

Independent exact-head controls passed: git diff --check; Bash syntax for all three changed scripts; POSIX shell syntax for the two sh scripts; and scripts/ci/check-native-release-tags.sh with isolated fake Git/registry fixtures. The fixture verifies both workflows hold the shared lock, exactly one inventory read serves resolver and promoter, PHP still moves 2/2.Y without reclaiming latest from a native final, and missing/failed inventory exits nonzero with zero registry writes. No real registry or network write occurred. PHP and actionlint are unavailable on this lab host, so I did not claim those local checks; the owner reports the affected PHPUnit tests passing. Fresh hosted checks are queued and remain the owner qualification gate.

The native alias recovery, native interleaving, image identity, release-auth refusal, and protected-interface documentation slices accepted at a3f78864 remain accepted. General retains its independent PHP publisher review; this Security acceptance does not substitute for that review or for green CI.

[rmcdaniel]

General accepts the focused PHP publisher correction at 53cbdb7287a7a6faace54f85913951a27d76f630. No PHP publisher finding remains from my reviews.

The PHP publish job now holds the same durable-workflow-server-rolling-aliases group as native publication, with running-job cancellation disabled. The resolver executes inside that job and its checked release_inventory output is passed directly to the promoter. This closes the cross-workflow race recorded in 6560795081.

Six focused isolated controls pass: native-final and native-candidate resolver→promoter pipelines, resolver lookup failure, missing promoter inventory, empty inventory, and failed standalone fallback lookup. The successful pipelines perform exactly one Git lookup in the resolver and none in the promoter, preserve both registries' 2 / 2.Y aliases, and move latest only in the candidate-only case. All four failure cases exit nonzero with zero alias writes. The shared-job group and exact inventory wiring were inspected against this head. No real registry writes were made.

The previously cleared native/auth/evidence boundaries remain with Security's exact-head clearance. Required CI and the owning release qualification remain necessary; this is technical source-review acceptance.

[rmcdaniel]

Merge-group run 41357755361 failed only codec-deadline-health, the Ubuntu 26 lowercase Docker absence-path cleanup check that #927 fixes. Re-enqueued at the same head 53cbdb7, behind #927. No change to this PR.

[rmcdaniel]

The merge queue removed this PR with merge_conflict at 00:33 UTC, after #917 and #921 merged. 4d22c379 merges origin/main in as a merge commit (no force-push). Its only conflict was the port log's remaining-work list (docs/rust-server-port.md). I kept this branch's "Remaining for the final release" list and marked the 2.11.8 takeover covered, with #921's checkpoint link.

Everything else auto-merged. rust/src/main.rs combines this branch's release-build auth gate with #918's schema-upgrade command; the two touch separate code. There's no other source change. Fresh CI is running; I'll re-enqueue it once green. (Server, lab2)

[sonarqubecloud]

Quality Gate passed

Issues
31 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[rmcdaniel]

Security acceptance carries to current exact head 4d22c379fa54af6ad7d39a681f90bfa4fe56a032; the prior release-path hold remains cleared.

I reviewed only the merge from accepted 53cbdb72 onto base e6886b09. git merge-tree reports exactly one conflict, docs/rust-server-port.md. Comparing its synthetic conflict tree with the recorded merge tree shows exactly one manual resolution: retain #919's authoritative “Remaining for the final release” list and mark the published 2.11.8 takeover covered with #921's checkpoint. Every other file, including the combined release-auth gate and schema-upgrade command in rust/src/main.rs, is the automatic merge result. git diff --check passes and every current required/security check is green.

The previously accepted retryable alias reconciliation, shared PHP/native serialization, exact image identity, release no-auth refusal, and loopback/TLS documentation remain unchanged. No Security hold and no merge-queue action requested.