[simonchrz]

Point 3 of #543. This builds on #569: its migration comes after #569's, and the tests use #569's Postgres CI job. Until #569 is merged, the diff here includes its commits; the change is the last two commits, 84111c3 and 64162c7.

Instances sharing a Postgres database only coalesced cache misses within themselves, so each instance missing an artifact fetched it from upstream. Now the instance that fetches holds a lease on the miss, and the others wait for its result:

• Lease table. runSharedFetch takes a lease in a new fetch_leases(key, owner, expires_at, failed) table, after its recheck and before the upstream fetch. The key is a hash of the existing coalescing key. Taking the lease is a single INSERT … ON CONFLICT (key) DO UPDATE … WHERE expires_at < now(). Expiry uses the database clock.
• Holding it. The holder renews the lease every 10 seconds while it downloads. It deletes the lease once the artifact is stored or the fetch failed.
• Waiting. An instance that finds the lease held polls the existing recheck, starting at 50 ms and backing off to 2 s. Once the record appears, it serves the stored file, as in-process waiters do. It runs one poll per instance; callers within the instance still join it as before. A detached request keeps waiting after its client leaves, so callers joined to it are not cancelled.
• Failure and crashes. If the holder's fetch fails, it marks the lease failed. The instances waiting on it then serve their stale metadata copy if they have one, and otherwise fetch at the same time, so a failing upstream costs each instance one attempt in parallel. A new request that finds a failed lease takes it over. A lease from a crashed instance expires after 30 seconds. Reclaim deletes old expired rows.
• Database errors. If a lease query fails, the instance logs it and fetches without a lease.
• Metadata. Metadata misses use the same lease, but only with cache_metadata on and metadata_ttl above zero. Otherwise there is no stored copy to wait for.
• Cost. A cache hit costs nothing extra. A miss costs one query to take the lease and one to release it. SQLite never uses leases.
• Metric. proxy_fetch_lease_waits_total{kind} counts the misses that waited for another instance.

One behaviour to note: when a scanner blocks an artifact, the waiting instance fetches and scans it again. Within an instance, waiters get the holder's error instead.

Tested with three instances against Postgres 17 and one shared file:// directory. Three clients each fetched the same 30 npm tarballs, each client through a different instance, six requests at a time:

	#569	this PR
upstream fetches	84, 87, 90	30, 30, 30
replaced copies queued for deletion	54, 57, 60	0, 0, 0
waits on another instance	–	51, 41, 47

All 90 requests succeeded in every run. No lease rows were left afterwards.

Tests (Postgres, run by the test (postgres) job):

• TestFetchLeaseCoalescesArtifactAcrossInstances: two proxies share one database and storage, and the second one's miss waits for the first one's fetch.
• TestFetchLeaseFailedFetchLetsOtherInstanceFetch
• TestFetchLeaseFailedFetchWaitersFetchInParallel: three instances; after the holder fails, both waiting instances fetch at once.
• TestFetchLeaseFailedMetadataServesStale
• TestFetchLeaseWaitOutlivesDetachedCaller
• TestFetchLeaseExpiredIsTakenOver
• TestFetchLeaseCoalescesMetadataAcrossInstances
• TestFetchLeaseNotUsedWithoutMetadataCache
• TestFetchLeases and TestFetchLeaseFailure: acquire, renew, release, failure, takeover and cleanup on the table.
• TestFetchLeaseNotUsedOnSQLite

With leases switched off, the four cross-instance tests fail.

[andrew]

In runSharedFetch the wait on another instance's lease runs under the request ctx, while the commit uses the detached commitCtx. DetachUpstreamFetch is there so a client giving up on a slow miss (pacman after 10s) doesn't end the shared fetch. Here, if the leading caller on one instance disconnects while waiting, f.err = ctx.Err() and every other client joined to that fetch on the same instance gets context canceled, even though their connections are fine. Please wait under context.WithoutCancel(ctx) when upstreamFetchDetached(ctx), with a test for it. The wait makes no upstream request, so it needs no detach slot, and the holder's fetch timeout and the lease expiry still bound it.

One to decide: when the holder's fetch fails, each waiting instance takes the lease once that fetch ends and runs the same fetch again. Against an upstream that hangs until timeout, a client on the k-th instance in line waits about k timeouts, where on main the instances fail in parallel. For metadata that delays the stale copy each waiter would serve. Either note it in the "Several instances" docs, or have the holder mark the lease as failed before releasing it so waiters go straight to their own stale copy or error.

[simonchrz]

Both in 64162c7.

• A detached caller now waits under context.WithoutCancel. TestFetchLeaseWaitOutlivesDetachedCaller cancels the leading caller on one instance while another is joined to it; the joined caller still gets the artifact without a second upstream fetch.
• I went with marking the lease failed (a failed column, still migration 016). The holder sets it when its fetch fails, or for metadata when it served a stale copy. Instances waiting on it then serve their own stale metadata copy if they have one, and otherwise fetch at the same time, so each pays one timeout as on main. For artifacts they fetch rather than return an error, since a waiter can't tell a 404 from a timeout and would turn a 404 into a 502. A new request that finds a failed lease takes it over. Tests: TestFetchLeaseFailedFetchWaitersFetchInParallel, TestFetchLeaseFailedMetadataServesStale, TestFetchLeaseFailure; the docs paragraph describes it.