Repository navigation
Several replicas on Postgres and a shared file:// volume #543
Description
The Helm chart says more than one replica needs object storage. Not every cluster has S3, but many have a ReadWriteMany volume (NFS, CephFS, Longhorn). I tested whether the proxy already works with several replicas on Postgres and a shared file:// directory. It mostly does. Two bugs and some duplicate work stand in the way of supporting it.
Test
Setup: three instances of current main, Postgres 17, one shared directory for storage.url, and nginx round robin in front.
| Test | Result |
|---|---|
3 parallel cold npm install (333 packages each) |
all succeed, no errors |
| DB records vs files on disk | 0 records without a file, 0 files without a record |
docker kill one instance during installs |
installs succeed, no partial files left |
| 3 instances started at once on an empty database | 2 of 3 exit in 8 of 8 runs: creating schema: executing schema: pq: duplicate key value violates unique constraint "pg_type_typname_nsp_index". A restart works. |
Eviction with max_size: 60MB, 93 MB cached |
all three evict at startup and the cache drops to 32 MB. Each instance frees the whole overshoot, and clearEntry skipping the entries another instance cleared makes it go further down the LRU list. |
| Duplicate upstream fetches | 406 replaced objects queued for deletion next to 961 live entries, so about 40 % extra upstream fetches. They are cleaned up correctly. |
The file side holds up because each fetch writes its own path and renames into place, the records are compare-and-swap, and Blob.Delete treats a missing file as deleted.
Proposal
- Lock schema creation and migrations. Take
pg_advisory_lockaroundCreate/MigrateSchemaon Postgres, so instances starting together run them one after the other. - Run background jobs on one instance at a time. Eviction, retention and mirror schedules each take
pg_try_advisory_lockper run. An instance that doesn't get the lock skips that run. This fixes the over-eviction and the duplicate schedule runs. - Coalesce cache misses across instances. Today
coalesceFetchand the metadata equivalent coalesce through an in-memory map. Add a lease tablefetch_leases(key, owner, expires_at):- The caller that runs the fetch in its process first takes the lease (
INSERT … ON CONFLICT (key) DO UPDATE … WHERE expires_at < now()). It renews the lease during long downloads and deletes it after the commit. - If another instance holds it, the caller polls the existing
recheckuntil the record appears, then serves the stored file, as in-process waiters do today. - An expired lease (holder crashed) is taken over.
- This costs one extra query per cache miss and nothing on a hit.
- The caller that runs the fetch in its process first takes the lease (
- Docs and chart. Document several replicas on Postgres with a ReadWriteMany
file://volume, and update the chart comment.
SQLite is unaffected: it is single-instance, and the locks and leases are no-ops there.
I'd do 1, 2 and 4 in one PR and 3 in a second one.
Activity
Coordinate instances sharing a Postgres database
Fetch each cache miss once across instances
simonchrz commentedon Oct 24, 2026
Done in #569 (points 1, 2 and 4) and #570 (point 3).