[tangentstorm]

Fixes #6

What

• bin/deploy: builds (npx vite build), uploads the result as a new release (<docroot>.releases/<UTC timestamp>), then atomically switches the docroot symlink to it with a single rename(2). The live tree is never edited in place, and older releases are the backups (the newest DEPLOY_KEEP, default 5, are kept). Before every switch, files that the last DEPLOY_ASSET_GRACE (default 3) live releases shipped are hard-linked into the target, so a page loaded just before a switch can still fetch its hashed assets. Any failure exits nonzero. If the build fails, it stops before anything on the server is touched.
  • --dry-run builds, lists the itemized changes against the live tree, and touches nothing.
  • --rollback [NAME] is an atomic switch to the release before the live one (repeat to go further back), or to NAME, which can also roll forward. --rollback --dry-run previews.
  • --list-releases lists releases and marks the live one.
• deploy/remote.sh: the server-side half (prepare, activate, switch, prune, pick, diff, discard), piped to bash over ssh.
• .github/workflows/deploy.yml: PRs get build + deploy tests only. Push to main and workflow_dispatch on main run the tests, then bin/deploy in the production environment, then a curl check of https://implish.org/.
  • Deploy/test tooling always comes from the workflow's own commit. Only the application source comes from the optional ref input, which must already be merged into main. deploy/select-ref.sh checks that and exports it with git archive.
  • Deploys run one at a time with no cancellation.
• deploy/deploy.conf holds the config. deploy/known_hosts pins the server's host keys, so CI never trusts a key on first use. deploy/README.md is the full runbook. CLAUDE.md is updated to match.

⚠️ Server access: user and docroot unknown

This box can't reach the server over ssh: Permission denied (publickey,password). There's no private key here, and I didn't try any way around that. DEPLOY_HOST=45.79.174.182 is set. DEPLOY_USER and DEPLOY_PATH are empty and have to be filled in. Until they and the secret are set, bin/deploy refuses to run.

What Michal needs to add (details in deploy/README.md)

where	name	value
environment	production	Settings → Environments; deployment branches → Selected branches and tags → main
production env secret (not a repo secret)	DEPLOY_SSH_KEY	private half of a new ed25519 key (ssh-keygen -t ed25519 -N '' -C implish-deploy@github-actions -f implish-deploy)
production env variable	DEPLOY_USER	ssh user that owns the site's parent dir
production env variable	DEPLOY_PATH	the nginx root for implish.org, e.g. /var/www/implish/site (see the server prerequisites below)
server ~DEPLOY_USER/.ssh/authorized_keys	one line	restrict ssh-ed25519 AAAA…(implish-deploy.pub)… implish-deploy@github-actions

Commands: gh secret set DEPLOY_SSH_KEY --repo tangentcode/implish --env production < implish-deploy and gh variable set DEPLOY_USER|DEPLOY_PATH --env production --body …. Don't store the key as a repo-level secret: same-repo PR branches can read those, so a PR that edits the workflow could deploy. Also recommended: branch protection on main, since everything that lands there deploys.

Server prerequisites. These need root, so they're flagged to you:

• New with the atomic switch: the docroot becomes a symlink that the deploy swaps, so the deploy user needs write access to the docroot's parent dir. If the root currently sits directly in root-owned /var/www, the least-privilege option is to move the site one level down into a dir the deploy user owns, and point nginx (and certbot's webroot, if it uses one) at it. For example sudo mkdir /var/www/implish && sudo mv /var/www/implish.org /var/www/implish/site && sudo chown -R USER: /var/www/implish, set root /var/www/implish/site;, then reload nginx.
• nginx must follow symlinks. That's the default; check disable_symlinks isn't on.
• rsync and flock ship with Ubuntu.
• Please check the pinned host-key fingerprints in deploy/README.md against /etc/ssh/ssh_host_*_key.pub.

Until the secret is set, the deploy job on main fails with an error that points at deploy/README.md. I chose that over skipping so that a misconfiguration is visible.

Behavior notes

• First deploy onto today's plain-directory docroot: the old tree is moved into the releases dir as the oldest release (the rollback target), and the docroot becomes a symlink. That migration is two back-to-back renames, so the path is briefly missing. It's the only non-atomic step, and it happens once. A missing or empty docroot just becomes the symlink.
• Any failure before the atomic rename leaves the old site serving, including during the one-time migration, which restores the old directory if its last rename fails. Carry conflicts are warnings, not failures.
• A file you delete stays reachable for DEPLOY_ASSET_GRACE more deploys, which is the point of the grace period. Dry runs say so. Set it to 0 to disable.
• /.well-known/ (certbot) is copied from the live tree into each new release and protected from --delete.
• Files are 644 and dirs 755. Everything on the target is created under umask 022, so nginx can read it whatever umask the deploy shell has.
• Unchanged files are hard-linked from the live release (--link-dest), so they aren't re-sent and the releases share disk space. --checksum means CI checkouts with fresh mtimes only transfer changed content.
• The workflow uses actions/checkout@v5 / setup-node@v5 (Node 24 runtime).
• Vite 7 needs Node ≥ 20.19, so CI uses Node 22. Locally, /usr/local/bin/node 20.11 can't build the site, and /usr/bin/node 21.7 can.
• No lockfile, because package-lock.json is gitignored in this repo. CI uses npm install. Committing a lockfile would make builds fully reproducible; I left that as your call.

Findings fixed while building (root cause → neighboring cases covered)

1. --chmod is ignored without -p. Uploads use -rlp. Test: t_deploy_migrates_plain_docroot checks the mode is 644. Rollback no longer copies files, so the neighbor case from round 0 is gone.
2. Empty env vars clobbered the config file (GitHub maps an unset vars.X to ""). Every DEPLOY_* override applies only when non-empty. Test: t_conf_file_and_env_override.
3. --delete or rm -rf aimed at a wrong path. DEPLOY_PATH and DEPLOY_RELEASES are validated: relative paths, /, single-level paths, .., spaces/quotes, and a releases dir inside the docroot are all refused. A missing DEPLOY_SOURCE_DIR is refused too. Test: t_rejects_unsafe_paths.

Review fixes (MinBot, round 1)

4. The secret was reachable from unmerged code. The key and vars moved to the production environment, limited to main. Neighbors:

   • a PR that removes the if:;
   • dispatch from a non-main branch (also blocked by if: github.ref == 'refs/heads/main');
   • dispatch with an unmerged ref (select-ref.sh requires merge-base --is-ancestor);
   • option-like or unknown refs.

   Test: t_select_ref_only_merged, which also checks that nothing is exported for a rejected ref.

5. Every rm -rf goes through one name check (rm_release, ^20[0-9]{6}T[0-9]{6}Z$). It covers discard, prune and stale-.tmp cleanup. switch, activate and diff validate names the same way. Listing uses the exact regex. Test: t_remote_rejects_bad_names.

6. Partial uploads/backups left stray .tmp dirs. A failed upload is discarded, and stale .tmp dirs are cleaned before each prepare and on prune. Tests: t_failed_upload_leaves_live_intact, t_killed_deploy_recovers, t_stale_tmp_removed.

7. One lock covers the whole deploy or rollback. It's held through a coprocess ssh session and released on exit, crash or disconnect. Tests: t_lock_blocks_concurrent_runs, t_lock_held_during_upload. t_killed_deploy_recovers also shows that a kill -9'd deploy releases the lock.

8. Test-harness VAR=x out=$(cmd) bug (VAR was never exported): all uses are rewritten as out=$(VAR=x cmd).

Review fixes (Codex, round 2)

9. P2: deploy tooling was taken from the historical ref (deploy.yml line 29). Root cause: one checkout served as both the tooling and the source to build. Fix: both jobs check out github.sha (the workflow's own commit) for bin/deploy, deploy/*, package.json scripts and tests. select-ref.sh REF BASE DEST exports only the selected merged commit's tree via git archive (plus .deploy-sha), and bin/deploy builds it via the new DEPLOY_SOURCE_DIR. Neighboring cases of "tooling read from the target ref":

   • (a) the test job ran npm run test:deploy from the old ref's package.json. It now always runs from github.sha.
   • (b) deploy/known_hosts and select-ref.sh itself come from the tooling checkout.
   • (c) npm install runs in the exported source dir, so the old commit gets its own dependencies.
   • (d) the deploy log names the source commit (.deploy-sha), not the tooling checkout's HEAD.
   • (e) a commit too old to build fails at the build step, before the server is touched.

   Test: t_redeploy_pre_tooling_commit exports the real pre-tooling commit 6813323 from this repo (no bin/deploy, no test:deploy), deploys it with the current bin/deploy, and checks that the live index.html equals git show 6813323:web/index.html. CI uses fetch-depth: 0 so the commit is available, and the test ran in CI.

10. P2: assets were deleted before the new site was published (bin/deploy line 108). Root cause: rsync edited the live tree in place (--delay-updates doesn't defer deletions). Fix: the release dirs plus atomic symlink switch described above, so nothing under the live tree is ever written. Neighboring cases:

    • (a) rollback is now the same atomic switch instead of an rsync over live;
    • (b) an interrupted upload (rsync error) discards the .tmp and leaves live untouched;
    • (c) a killed deploy (kill -9) leaves live untouched and the lock released, and the next deploy cleans the stale .tmp;
    • (d) first deploy with a missing docroot, an empty docroot, or a plain directory (migrated into the oldest release, with its content kept byte-for-byte);
    • (e) prune never removes the live release, even after a rollback to an old one;
    • (f) switch refuses to replace a real directory;
    • (g) unchanged files are linked rather than copied, and the previous release is never modified.

    Tests:

    • t_live_untouched_during_upload: an rsync wrapper checks the live tree is byte-identical before and after the transfer. The fixtures use Vite-style hashed assets, and page_ok checks that the asset the live index.html names actually exists.
    • t_failed_upload_leaves_live_intact, t_killed_deploy_recovers
    • t_first_deploy_missing_docroot, t_first_deploy_empty_docroot, t_deploy_migrates_plain_docroot
    • t_rollback_is_one_atomic_switch, t_rollback_restores_previous (byte-for-byte, then roll forward by name), t_rollback_steps_back
    • t_prune_never_removes_live, t_switch_refuses_plain_dir, t_redeploy_reuses_unchanged_files

11. Found during a local end-to-end run: new dirs depended on the deploy shell's umask. With umask 077, the releases dir came out drwx------, which nginx can't read, so every request would get a 403. Root fix: umask 022 for everything that runs on the target (remote.sh and the lock session), plus an explicit chmod 755 on the releases dir, since the lock session can create it first. Neighbors: the releases dir, each release dir, .well-known copies, and the docroot's parent dirs on a first deploy. Test: t_nginx_can_read_everything deploys twice under umask 077 onto a missing parent path and checks every dir is o+rx and every file o+r.

Review fixes (Codex, round 3)

12. P2: a page loaded from the old release got a 404 for its hashed JS after the switch (remote.sh:64, thread). Root cause: URLs resolve through the docroot symlink, so a file is reachable only if the current release contains it. Keeping old releases on disk doesn't help. Fix: before every switch, the target gets hard links to every file that the last DEPLOY_ASSET_GRACE live releases shipped and the target lacks. The switch is still one atomic rename, done after the target is complete. Neighboring cases of "a page from release N requests a file missing from release N+1", each with a test:

    • (a) hashed JS/CSS after a deploy: the Codex repro over real HTTP (python3 -m http.server resolves the symlink per request, like nginx). GET /, deploy, then GET the asset that index named, across 3 consecutive deploys. → t_old_index_then_switch_no_404
    • (b) the first deploy's migration of the plain docroot: the old site's assets are carried into the first release before the symlink appears. Covered by step 1 of (a) and t_deploy_migrates_plain_docroot.
    • (c) rollback (page from N+1 → back to N) and roll-forward (page loaded during the rollback → forward again) → t_old_index_then_rollback_no_404
    • (d) non-hashed or fetched data (e.g. b4-image.json): any path is carried, not just assets/. A removed file stays reachable, and a file the new release ships itself is never overwritten by an older copy. → t_removed_data_file_stays_reachable
    • (e) pruning: releases in the window are never pruned, even with DEPLOY_KEEP=1, and carried files are hard links, so they survive their source release being pruned. → t_prune_keeps_grace_window
    • (f) window bound / no snowballing: only files a release shipped (RELEASES/<name>.files) are carried, not ones it carried itself, so a file ages out after N switches. → t_grace_window_is_bounded
    • (g) ordering after rollbacks: the window follows RELEASES/.history (which release was live when), not name order. This is exercised by (c) and (e), and the "ignore history" mutation fails 8 runs.
    • (h) manifest safety: entries with .. or a leading / are skipped, even when they'd resolve to a real file. → t_carry_ignores_bad_manifest_entries
    • (i) grace 0 disables carrying, and the setting is validated. → t_grace_zero_disables_carry

    Updated contracts: rollback/atomic-switch tests now assert that a release's own files (its manifest) are byte-identical and that only carried files are added. t_rollback_restores_previous checks that every file of the old tree is restored byte-for-byte and the only extra is the rolled-back release's asset.

Review fixes (Codex, round 4)

13. P2: a migration failure left the site down (remote.sh:139, thread). Root cause: fallible steps ran while the live site was gone or half-switched. Fix: every switch is stage (all fallible work, live site untouched) + commit (one rename) + warn-only bookkeeping. An exit trap removes the staged symlink and any never-live release on failure. Neighboring cases, each with a test:
    • (a) migration: stages from the plain docroot before moving it, and restores it if the final rename fails. Faults are injected at stage, before-commit, migrate-before-move and migrate-between-renames. → t_fault_migration_keeps_old_site; empty docroot → t_fault_empty_docroot_migration
    • (b) deploy (symlink docroot), faults at stage and before-commit → t_fault_deploy_keeps_old_site
    • (c) rollback and roll-forward, the same faults → t_fault_rollback_keeps_site
    • (d) after the switch: a history write failure warns and exits 0; a prune failure exits nonzero, saying the new release is live → t_after_switch_failures_report_live
    • (e) carry conflicts never fail a switch:
      • Codex's downloads/old.json vs. a downloads file during migration → t_migration_file_vs_dir_conflict
      • the same during a deploy → t_deploy_file_vs_dir_conflict
      • the reverse, dir vs. file → t_deploy_dir_vs_file_conflict
      • during rollback and roll-forward → t_rollback_file_vs_dir_conflict
    • (f) symlinked parent paths are never followed, on the target or the source side → t_carry_never_writes_through_symlinks
    • (g) found by the fault tests: a failed activation left a never-live release behind, which collided with a quick retry ("cannot migrate … exists") and could have become a rollback target. The trap now removes it, and the fault tests assert that nothing is left behind.

Mutation checks (each restored afterwards):

• round 4:
  • no restore after a failed second rename → 87/89
  • carry aborts on conflict (the pre-fix behavior) → 81/89
  • no cleanup trap → 83/89
  • fatal history write → 87/89
  • no source symlink check → 87/89
  • no target parent check → 87/89
• round 3:
  • no carry on switch → 55/69
  • no carry on the first-deploy migration → 65/69
  • carry overwrites existing files → 44/69
  • prune ignores the window → 67/69
  • ignore live history → 61/69
  • no manifest path guard → 67/69
  • unbounded window → 63/69
• round 2:
  • upload in place to live → 15/53
  • no discard on a failed upload → 51/53
  • ignore DEPLOY_SOURCE_DIR → 25/53
  • prune may remove live → 51/53
  • no stale-.tmp cleanup → 49/53
  • no umask fix → 53/55
• round 1: no rm_release name check, no lock

Tests

test/deploy.test.sh has 46 cases. 43 run in both modes and 3 are mode-specific, for 89 runs. The two modes: DEPLOY_HOST=local, and through a fake ssh that runs the remote half locally, so the real ssh + rsync -e path is exercised. The cases cover:

• dry runs and aborts: dry-run listing with nothing changed, dry run on a missing docroot, failed or empty build;
• deploys: migration of the plain docroot, first deploy onto a missing or empty docroot, link reuse, live untouched during upload, failed/killed uploads, stale .tmp, umask/nginx readability;
• rollback: restore and roll forward, atomic switch, stepping back, named rollback and bad names, no previous release;
• retention and prune-never-live;
• guards: remote name guards, switch-refuses-dir, unsafe paths and args, unreachable host, config/env precedence;
• locking: concurrent runs blocked, lock held during upload;
• failure safety: fault injection at every point of every switch kind; post-switch failures; file/dir conflicts in both directions during migration, deploy and rollback; symlinked parents;
• grace period: the HTTP old-index-then-switch reproduction for deploy, rollback and roll-forward; removed data files; no-overwrite; window bound; prune vs. the window; bad manifest entries; grace 0;
• CI: merged-only ref selection, redeploying the pre-tooling commit 6813323.

Manual check: a real vite build with a local temp docroot. The dry run listed the hashed assets. The deploy migrated the plain docroot into a release and switched the symlink, and --list-releases showed both releases. Rollback switched back to the original tree. There have been no real deploys and nothing was written to the server.

Results at head a01ddb4dedb1b4c14f27398c86b1dbe04d22676b (Node 21.7.3 locally):

• npm run build: passed (1/1)
• npm run test:deploy: 89/89 passed (3 mode-specific skips), locally and in CI run 40984171391. The deploy job was skipped on the PR, as designed.

🤖 Generated with Claude Code

[tangentstorm]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T01:19:47.563926Z	a01ddb4	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e6297d892a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[tangentstorm]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a374a0b894

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[tangentstorm]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a357e30dc8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[tangentstorm]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: a01ddb4ded

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".