From fd097e91dff6838bc51bd38b1757ff11b903ca23 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 26 Oct 2026 17:27:49 -0600 Subject: [PATCH 01/24] fix(sqlite-persistence): make every persisted hydration wait for a reserved commit A sync transaction that called reserveCommitTurn only kept subset loads admitted during the reservation from crossing it. A subset load admitted earlier, a peer reload or reset, sequence-gap recovery, or a hydration held by the shared scheduler advanced the hydration cycle, and the commit rejected with "a persisted sync transaction cannot cross a hydration cycle" (#2226). Every persisted hydration now waits outside the apply mutex until the reserved transaction ends. Peer resets go through the notice queue, so a pre-reset commit can no longer publish a row the reset removed. Electric abandons its open transaction when its stream stops for good. The persisted-reads grammar, a real-adapter oracle and a Browser multi-tab oracle own the widened law. Co-authored-by: Isaac --- .../reserved-commit-defers-every-hydration.md | 11 + docs/contributing/glossary.md | 1 + docs/contributing/oracle-coverage.md | 68 +- .../tests/multi-tab-electric-oracle.test.ts | 495 ++++++ .../src/persisted.ts | 246 ++- .../tests/persisted-oracle.test.ts | 1431 ++++++++++++++--- ...erved-commit-sqlite-adapter-oracle.test.ts | 429 +++++ packages/db/src/types.ts | 9 +- .../electric-db-collection/src/electric.ts | 28 +- 9 files changed, 2407 insertions(+), 311 deletions(-) create mode 100644 .changeset/reserved-commit-defers-every-hydration.md create mode 100644 packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts create mode 100644 packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts diff --git a/.changeset/reserved-commit-defers-every-hydration.md b/.changeset/reserved-commit-defers-every-hydration.md new file mode 100644 index 0000000000..fbec83c149 --- /dev/null +++ b/.changeset/reserved-commit-defers-every-hydration.md @@ -0,0 +1,11 @@ +--- +'@tanstack/db-sqlite-persistence-core': patch +'@tanstack/db': patch +'@tanstack/electric-db-collection': patch +--- + +Keep a sync transaction that reserved its commit turn from failing when a persisted hydration starts while it is open. Before, only a subset load admitted during the reservation waited for the commit. A subset load admitted earlier, a reload or reset from a peer tab's notice, sequence-gap recovery, or a hydration that a shared scheduler held past its turn advanced the hydration cycle, and the commit then rejected with `a persisted sync transaction cannot cross a hydration cycle`. Electric reserves every stream transaction, so a deploy that made several tabs refetch at once could put their Collections into `error`. Each of these hydrations now waits outside the apply mutex until the reserved transaction commits, then starts. While it waits, readers keep the rows from before it. The `reserveCommitTurn` contract now covers every persisted hydration. + +A peer reset now applies in arrival order with peer commit notices, so a commit from before the reset can no longer publish a row that the reset removed. + +Electric now abandons its open transaction when its stream stops for good, so hydrations that wait for that transaction can run. diff --git a/docs/contributing/glossary.md b/docs/contributing/glossary.md index 898d5a1566..534354274f 100644 --- a/docs/contributing/glossary.md +++ b/docs/contributing/glossary.md @@ -34,6 +34,7 @@ production queues, caches, or semantic helpers merely to share their names. | optimistic state | The rows an optimistic transaction overlays on the applied synced rows while its mutation function runs. It drops when the mutation function settles, whether it fulfills or rejects. | Retained snapshot or accepted local snapshot. | | accepted sync transaction | A committed sync transaction that core, or a wrapping sync after its durable step, has taken. It always applies, in commit order. While an optimistic transaction is persisting it is held, and it applies in the publication that drops that transaction's optimistic state. Handler-facing writes and Collection readiness wait for acceptance; commit receipts and subset loads wait for visibility. | Visible or applied sync transaction. | | deferred sync transaction | A committed sync transaction that a wrapping sync has not handed to the Collection yet, because an earlier durable write is still running. Like a queued sync transaction, it holds settled optimistic rows on its keys until it applies. | Queued or accepted sync transaction. | +| reserved sync transaction | A sync transaction whose source called `reserveCommitTurn` and that has not committed or been abandoned. Every persisted hydration that would start while it is open waits for its commit. A source that stops for good abandons it. | Deferred sync transaction or held optimistic row. | | held optimistic row | A successful transaction's optimistic row kept only while a committed sync transaction that has not applied touches its key, so the drop and that transaction publish together. The transaction may be queued in the Collection or deferred behind a wrapping sync's durable step. Its source attribution follows the key-and-timing rules in [VirtualOrigin](../reference/type-aliases/VirtualOrigin.md), not source-client identity. | Retained or accepted snapshot. | | Collection status | One public Collection lifecycle value: `idle`, `loading`, `ready`, `error`, or `cleaned-up`. | Subscription status. | | subscription status | One subscription value: `ready` or `loadingSubset`. | Collection status. | diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 15a526e56d..1a41c797a3 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -1650,7 +1650,20 @@ consistency between them: while their streams interleave, a lagging tab's copy of an older transaction can be stored and announced after a newer one, so a tab can briefly show an older server state. Every tab delivers the same log in order, so durable state and every tab converge once the streams -catch up. No lane checks this yet. Ordering the copies would need a generic +catch up. No lane checks this yet. +A counterexample bounds that convergence claim +([#2209](https://github.com/TanStack/db/issues/2209)). Two Browser tabs +share one SQLite file through the real coordinator, with a controlled +Electric stream per tab. Tab A's stream applies v1 then v2 at `30_0`. Tab B's +lagging stream applies v1 at `20_0`, and tab B closes before it reaches v2. +Tab A then shows v1, and the durable `electric:resume` offset is back at +`20_0`. Nothing redelivers v2 to tab A while it stays open, because its +stream is already past it. A tab that opens later resumes from `20_0` and +repairs the cache. The convergence above therefore holds only while the +lagging tab keeps running or another tab restarts its stream. The +multi-tab Electric oracle in `browser-db-sqlite-persistence` pins this +history with `it.fails` beside a passing control in which the lagging tab +catches up. Ordering the copies would need a generic source position on sync commits; Electric already stores `{ handle, offset }` with each commit, PowerSync does not use this wrapper, and persisted Query collections have no log position. @@ -2467,6 +2480,59 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. not establish behavior after schema changes or invalidated resume points, or each intermediate publication during a multi-transaction truncate replay. The same owner needs those histories before claiming those stronger laws. + Reserved commits (#2226). The same driver and model also own the + reserved-commit law: a persisted hydration that would start while a + reserved sync transaction is open waits until it ends, and the commit is + accepted. Authority: `reserveCommitTurn` in `packages/db/src/types.ts`, + widened by maintainer decision on #2226 to every persisted hydration. + Further maintainer decisions fix its edges. Readers keep the rows from + before a hydration while it waits; a transaction opened after a + hydration began straddles it. A notice that waited may be retried as + gap recovery, with at most one more pullSince request and no added + upstream request. A source that stops for good abandons its + reservation. + A second registration crosses the routes with the reserving timings, + ending, and follow-up dimensions. The routes are subset, complete + subset, a follower's routed subset, the source's own baseline, + reset, full reload, sequence gap with or without pullSince, and a + targeted change that starts no hydration. The reserving timings are + across the mutex turn, across scope entry, during the read, in the + route's first publication, and across startup. The ending is commits, + abandoned, cleanup, an aborted demand, or the source awaiting its own + baseline. The follow-up is a reset then a pre-reset commit. + Observations: receipts, final rows, row metadata, cursor, status, + reads during the reservation, upstream requests, pullSince count, + rows while waiting, and every read against the fold of the durable + events before it. Every cell declares reach (ORC-015). + | Evidence | Result | + | --- | --- | + | Row-algebra cross | 360 cells pass. | + | Reservation cross | 188 cells pass. | + | `5649a07d4` | The open cells fail with `a persisted sync transaction cannot cross a hydration cycle`. Three `after read` reset follow-up cells also fail, because `main` applied a pre-reset commit before the reset set its term fence. Queueing resets with notices fixes this pre-existing defect. | + | Hostile mutants | Each mutant fails its cells, and every failure is an assertion. Deferring every notice fails the targeted change. Dropping the follower reply's wait fails the routed subset. Reporting hydration start before the check fails the aborted demand. A reset reload that starts its own cycle fails the first-publication reset. Applying a reset outside the notice queue fails the reset follow-ups. Retrying a waited notice while the reservation is open fails the follow-up and routed cells (repeated pullSince). Checking only at the mutex turn fails every scope-entry cell. Gating subset hydration only fails every reset, reload, and gap cell. Requeueing a waited notice behind later notices fails 12 reset follow-up cells: the pre-reset commit is published before its reset, and a reload is lost. | + Not covered: `forceReloadSubset`, which only a sync-absent Collection + has, and which therefore never meets a reservation; startup hydration, + which source entry follows (an `open across startup` observation); + and row visibility for on-demand Collections without demand beyond the + source's own rows. Cleaned-up cells judge only cleanup. + Real SQLite adapter: + `packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts` + crosses three routes (peer full reload, peer reset, and a subset + admitted earlier) with two timings (a held write of this Collection, + or the shared scheduler holding the scope after its turn) and two + endings (commits, abandoned). It runs on `SQLiteCorePersistenceAdapter`. + Its 10 cells pass. Every committing cell fails on `5649a07d4`. A peer + full reload behind this Collection's own held write cannot become + durable through the single SQLite writer, so it is excluded here and + left to the grammar's fixture leader. + Browser host: the multi-tab Electric oracle in + `browser-db-sqlite-persistence` crosses two or three tabs, eager or + on-demand Collections, which tab finishes first, a storm or staggered + arrival, and whether the last tab's stream fails while its refetch is + open. Its 32 cells pass. Every ordinary storm cell fails on + `5649a07d4`. Without Electric's abandon-on-failure, the eager storm + cells where a stream fails also fail. On-demand cells judge receipts + and status only. A claimed leftover-replay/reset failure still needs a concrete schedule; the bounded two-replay-then-reset probe drains replay before reset. Live Electric/browser OPFS delivery needs the matching schedule in diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts new file mode 100644 index 0000000000..f18307186e --- /dev/null +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -0,0 +1,495 @@ +// @vitest-environment node +/** + * # Do Electric tabs that share one persisted database agree with the server? + * + * Several Browser tabs open the same persisted Electric Collection. Each tab + * runs its own Electric stream, and all of them store through one SQLite + * database and one elected leader. Tabs coordinate only through durable + * commits and the notices those commits broadcast. This file is the Browser + * host owner for two laws about that arrangement. + * + * Law 1, reserved commits survive a refetch storm. Authority: + * SyncPersistenceCapabilityV1 `reserveCommitTurn` (packages/db/src/types.ts), + * widened by maintainer decision on issue #2226 to every persisted + * hydration. Electric reserves every stream transaction. After a deploy, each + * tab's stream sends must-refetch; the tab commits a reset resume marker at + * once, then holds a reserved refetch transaction open until up-to-date. A + * peer's marker or refetch commit reaches the other tabs while their + * refetches are open. The law: every tab's refetch commits, no Collection + * enters `error`, and once every stream has delivered the new snapshot every + * tab shows it. The controlled-history owner for the same law is the + * persisted-reads grammar in + * packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts; this + * file is its receiving witness on the real Browser coordinator. + * + * Law 2, an open tab converges to the server head. Authority: the coverage + * map's eventual-consistency note for persisted multi-tab Collections + * (docs/contributing/oracle-coverage.md, issue #2209): a lagging tab may + * briefly store an older transaction, and every tab converges once the + * streams catch up. The law judged here is narrower and observable: a tab + * whose own stream has applied the server head shows the head once the hosts + * are quiet. When the lagging tab closes before it catches up, nothing + * redelivers the head to the faster tab, so this law fails today. That + * history is pinned with `it.fails` until #2209 orders source writes by + * server position. + * + * Model: the server is a log of snapshots. A history names which snapshot + * each tab's stream has delivered; the expected public rows of a tab whose + * stream reached the head are the head snapshot. This file states no + * contract for which stream rows an on-demand Collection without subset + * demand exposes, so on-demand histories judge only commit acceptance and + * Collection status (a partial oracle). The model reads nothing from the + * coordinator, adapter, or Collection. + * + * Driver: each tab is a better-sqlite3 connection to one shared file, a + * `BrowserCollectionCoordinator` (BroadcastChannel and Web Locks in Node), and + * `createBrowserWASQLitePersistence`. A recording `ShapeStream` stands in for + * the Electric SDK; the test plays the server and delivers each tab's + * messages. Checkpoints wait for host quiescence: no SQL statement and no + * coordinator message for several tasks. Observations are each tab's + * Collection status and public rows. + * + * Limits: the stream is controlled, not the installed SDK over HTTP; one + * process hosts every tab; one Collection; no subset demand in on-demand + * Collections; no optimistic mutations; work amplification across tabs is + * not judged. + */ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { createCollection } from '@tanstack/db' +import { createOracleReach } from '../../db/tests/oracle-reach.js' +import { persistedCollectionOptions } from '../../db-sqlite-persistence-core/src' +import { electricCollectionOptions } from '../../electric-db-collection/src/electric' +import { BrowserCollectionCoordinator } from '../src/browser-coordinator' +import { createBrowserWASQLitePersistence } from '../src/browser-persistence' +import { createWASQLiteTestDatabase } from './helpers/wa-sqlite-test-db' +import type { + Message, + Row, +} from '../../electric-db-collection/node_modules/@electric-sql/client' +import type { ElectricCollectionUtils } from '../../electric-db-collection/src/electric' + +type Item = Row & { id: number; label: string } + +// --------------------------------------------------------------------------- +// Controlled Electric SDK boundary. Each stream records the position Electric +// started it at and exposes the subscriber, so the test can play the server. +// --------------------------------------------------------------------------- + +type ControlledStream = { + tab: string + shapeHandle: string | undefined + lastOffset: string + send: (messages: Array>) => void + // Electric's stream-error handler, which the test calls to fail the stream. + onError: (error: unknown) => unknown +} + +const streams: Array = [] + +vi.mock( + `../../electric-db-collection/node_modules/@electric-sql/client`, + async (importOriginal) => { + const actual = await importOriginal>() + return { + ...actual, + ShapeStream: vi.fn( + (options: { + headers?: Record + offset?: string + handle?: string + onError?: (error: unknown) => unknown + }) => { + const stream: ControlledStream = { + tab: options.headers?.[`x-tab`] ?? `unknown`, + shapeHandle: options.handle, + lastOffset: options.offset ?? `-1`, + send: () => { + throw new Error(`stream for tab ${stream.tab} is not subscribed`) + }, + onError: (error) => options.onError?.(error), + } + streams.push(stream) + return { + subscribe: (send: ControlledStream[`send`]) => { + stream.send = send + return () => undefined + }, + requestSnapshot: vi.fn().mockResolvedValue(undefined), + fetchSnapshot: vi + .fn() + .mockResolvedValue({ metadata: {}, data: [] }), + forceDisconnectAndRefresh: vi.fn().mockResolvedValue(undefined), + isUpToDate: false, + get shapeHandle() { + return stream.shapeHandle + }, + get lastOffset() { + return stream.lastOffset + }, + } + }, + ), + } + }, +) + +const upToDate: Message = { headers: { control: `up-to-date` } } +const mustRefetch: Message = { headers: { control: `must-refetch` } } +const insert = (row: Item): Message => ({ + key: String(row.id), + value: row, + headers: { operation: `insert` }, +}) +const update = (row: Item): Message => ({ + key: String(row.id), + value: row, + headers: { operation: `update` }, +}) + +const nextTask = () => new Promise((resolve) => setTimeout(resolve, 0)) + +// --------------------------------------------------------------------------- +// Browser host: one database file, one coordinator and persistence per tab. +// --------------------------------------------------------------------------- + +type SyncMode = `eager` | `on-demand` + +function openHost() { + const directory = mkdtempSync(join(tmpdir(), `tsdb-multi-tab-`)) + const filename = join(directory, `shared.sqlite`) + const dbName = `multi-tab-${Math.random()}` + const closers: Array<() => Promise> = [] + // Host activity for the quiescence checkpoint, and the number of commit + // notices, read from the coordinator's own channel. + let activity = 0 + let commitNotices = 0 + const channel = new BroadcastChannel(`tsdb:coord:${dbName}`) + channel.onmessage = (event: MessageEvent) => { + activity++ + const envelope = event.data as { + senderId?: string + payload?: { type?: string } + } + if (envelope.payload?.type === `tx:committed` && envelope.senderId) { + commitNotices++ + } + } + + const openTab = (tab: string, syncMode: SyncMode) => { + const database = createWASQLiteTestDatabase({ filename, timeout: 0 }) + const execute = database.execute.bind(database) + database.execute = async ( + sql: string, + params: ReadonlyArray = [], + ) => { + activity++ + for (;;) { + try { + return await execute(sql, params) + } catch (error) { + if ((error as { code?: string }).code !== `SQLITE_BUSY`) throw error + await nextTask() + } + } + } + const coordinator = new BrowserCollectionCoordinator({ dbName }) + const collection = createCollection( + persistedCollectionOptions< + Item, + string | number, + never, + ElectricCollectionUtils + >({ + ...electricCollectionOptions({ + id: `items`, + shapeOptions: { + url: `http://electric.test/v1/shape`, + params: { table: `items` }, + headers: { 'x-tab': tab }, + }, + syncMode, + getKey: (row) => row.id, + startSync: true, + }), + persistence: createBrowserWASQLitePersistence({ + database, + coordinator, + schemaMismatchPolicy: `reset`, + }), + }), + ) + let open = true + const close = async () => { + if (!open) return + open = false + await collection.cleanup().catch(() => undefined) + coordinator.dispose() + await database.close?.() + } + closers.push(close) + const stream = () => { + const own = streams.filter((candidate) => candidate.tab === tab) + const latest = own[own.length - 1] + if (!latest) throw new Error(`tab ${tab} has no stream`) + return latest + } + return { + tab, + collection, + close, + hasStream: () => streams.some((candidate) => candidate.tab === tab), + deliver: ( + messages: Array>, + position: { handle: string; offset: string }, + ) => { + const current = stream() + current.shapeHandle = position.handle + current.lastOffset = position.offset + current.send(messages) + }, + // A stream error with no retry stops the provider session for good. + failStream: () => stream().onError(new Error(`stream failed`)), + publicRows: () => + Array.from(collection.values(), ({ id, label }) => ({ + id, + label, + })).sort((left, right) => left.id - right.id), + } + } + + // The hosts are quiet when several tasks pass with no SQL statement and no + // coordinator message. The bound turns a livelock into a named failure. + const quiesce = async (label: string) => { + let quietTasks = 0 + for (let task = 0; task < 400 && quietTasks < 5; task++) { + const before = activity + await new Promise((resolve) => setTimeout(resolve, 2)) + quietTasks = activity === before ? quietTasks + 1 : 0 + } + if (quietTasks < 5) throw new Error(`hosts did not quiesce: ${label}`) + } + + return { + openTab, + quiesce, + commitNotices: () => commitNotices, + close: async () => { + for (const close of closers.reverse()) await close() + channel.close() + rmSync(directory, { recursive: true, force: true }) + }, + } +} + +type Host = ReturnType +type Tab = ReturnType + +async function openWarmTabs(host: Host, count: number, syncMode: SyncMode) { + const tabs: Array = [] + for (let index = 0; index < count; index++) { + const tab = host.openTab(String.fromCharCode(65 + index), syncMode) + for (let task = 0; task < 400 && !tab.hasStream(); task++) { + await new Promise((resolve) => setTimeout(resolve, 2)) + } + expect(tab.hasStream()).toBe(true) + tabs.push(tab) + } + return tabs +} + +// --------------------------------------------------------------------------- +// The server log. Snapshot 1 is the state before a deploy, snapshot 2 after: +// row 1 changes, row 2 is deleted, row 3 is new. +// --------------------------------------------------------------------------- + +const beforeDeploy: Array = [ + { id: 1, label: `one` }, + { id: 2, label: `two` }, +] +const afterDeploy: Array = [ + { id: 1, label: `one, refetched` }, + { id: 3, label: `three` }, +] + +describe(`multi-tab Electric over one persisted database`, () => { + let host: Host | undefined + afterEach(async () => { + await host?.close() + host = undefined + streams.length = 0 + }) + + // Law 1 grammar: two or three tabs, eager or on-demand Collections, which + // tab finishes its refetch first, whether every tab's must-refetch arrives + // before any refetch finishes (a storm) or each tab finishes before the next + // receives must-refetch (staggered, the neighbouring control), and whether + // the last tab's stream fails for good while its refetch is open. A source + // that stops for good abandons its reserved transaction (maintainer + // decision on #2226), so that tab still applies its peers' refetch from + // durable rows, as it did before reservations deferred hydrations. + it.each( + [2, 3].flatMap((tabCount) => + ([`eager`, `on-demand`] as const).flatMap((syncMode) => + ([`first`, `last`] as const).flatMap((finisher) => + ([`storm`, `staggered`] as const).flatMap((arrival) => + ([`ok`, `fails while refetching`] as const).map((stream) => ({ + tabCount, + syncMode, + finisher, + arrival, + stream, + })), + ), + ), + ), + ), + )( + `commits every reserved refetch across tabs ($tabCount tabs, $syncMode, $finisher finishes first, $arrival, stream $stream)`, + async ({ tabCount, syncMode, finisher, arrival, stream }) => { + host = openHost() + const tabs = await openWarmTabs(host, tabCount, syncMode) + for (const tab of tabs) { + tab.deliver([...beforeDeploy.map(insert), upToDate], { + handle: `h1`, + offset: `10_0`, + }) + await host.quiesce(`tab ${tab.tab} warm`) + } + type RefetchReach = + | `peer commit announced during an open refetch` + | `a stream failed while its refetch was open` + const reach = createOracleReach( + `browser.multi-tab-reserved-refetch`, + [ + ...(arrival === `storm` + ? ([`peer commit announced during an open refetch`] as const) + : []), + ...(stream === `ok` + ? [] + : ([`a stream failed while its refetch was open`] as const)), + ], + ) + reach.beginHistory() + + const finishOrder = finisher === `first` ? tabs : [...tabs].reverse() + // The last tab to finish is the one whose stream may fail. + const failingTab = + stream === `ok` ? undefined : finishOrder[finishOrder.length - 1] + const failStream = async (tab: Tab) => { + void Promise.resolve(tab.failStream()).catch(() => undefined) + await host!.quiesce(`tab ${tab.tab} stream failed`) + reach.reach(`a stream failed while its refetch was open`) + } + const refetch = (tab: Tab) => + tab.deliver([...afterDeploy.map(insert), upToDate], { + handle: `h2`, + offset: `20_0`, + }) + if (arrival === `storm`) { + for (const tab of tabs) + tab.deliver([mustRefetch], { handle: `h1`, offset: `10_0` }) + const noticesBeforeRefetch = host.commitNotices() + await host.quiesce(`every refetch open`) + // Each tab's reset marker is a peer notice for every other tab while + // its own reserved refetch is open. + if (host.commitNotices() > noticesBeforeRefetch) { + reach.reach(`peer commit announced during an open refetch`) + } + if (failingTab) await failStream(failingTab) + for (const tab of finishOrder) { + if (tab === failingTab) continue + refetch(tab) + await host.quiesce(`tab ${tab.tab} refetched`) + } + } else { + for (const tab of finishOrder) { + tab.deliver([mustRefetch], { handle: `h1`, offset: `10_0` }) + await host.quiesce(`tab ${tab.tab} refetch open`) + if (tab === failingTab) { + await failStream(tab) + continue + } + refetch(tab) + await host.quiesce(`tab ${tab.tab} refetched`) + } + } + reach.assertReached() + + // A rejected refetch commit puts its Collection in `error`. Eager rows + // must equal the head snapshot; on-demand rows are not judged (above). + expect( + tabs.map((tab) => ({ + tab: tab.tab, + status: tab.collection.status, + ...(syncMode === `eager` ? { rows: tab.publicRows() } : {}), + })), + ).toEqual( + tabs.map((tab) => ({ + tab: tab.tab, + status: `ready`, + ...(syncMode === `eager` ? { rows: afterDeploy } : {}), + })), + ) + }, + 30_000, + ) + + // Law 2 histories: tab A's stream applies v1 and v2; tab B's lagging stream + // applies v1 later. B then either catches up or closes. The checkpoint is + // host quiescence after the last delivery. + const lagging = async (bEnds: `catches up` | `closes`) => { + host = openHost() + const [a, b] = await openWarmTabs(host, 2, `eager`) + for (const tab of [a!, b!]) { + tab.deliver([insert({ id: 1, label: `v0` }), upToDate], { + handle: `h1`, + offset: `10_0`, + }) + await host.quiesce(`tab ${tab.tab} warm`) + } + a!.deliver([update({ id: 1, label: `v1` }), upToDate], { + handle: `h1`, + offset: `20_0`, + }) + await host.quiesce(`A applied v1`) + a!.deliver([update({ id: 1, label: `v2` }), upToDate], { + handle: `h1`, + offset: `30_0`, + }) + await host.quiesce(`A applied the head`) + b!.deliver([update({ id: 1, label: `v1` }), upToDate], { + handle: `h1`, + offset: `20_0`, + }) + await host.quiesce(`lagging B applied v1`) + if (bEnds === `catches up`) { + b!.deliver([update({ id: 1, label: `v2` }), upToDate], { + handle: `h1`, + offset: `30_0`, + }) + } else { + await b!.close() + } + await host.quiesce(`B ${bEnds}`) + // A's own stream applied the head, so A must show it. + expect(a!.publicRows()).toEqual([{ id: 1, label: `v2` }]) + } + + it.each([`catches up`] as const)( + `converges a tab that applied the head when the lagging tab %s`, + (bEnds) => lagging(bEnds), + 30_000, + ) + + // Known gap (#2209): nothing redelivers the head to tab A after the lagging + // tab stored v1 and closed. Ordering source writes by server position fixes + // it; move `closes` into the passing values with that change. + it.fails.each([`closes`] as const)( + `converges a tab that applied the head when the lagging tab %s`, + (bEnds) => lagging(bEnds), + 30_000, + ) +}) diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 69b1b3f2b4..f0cc3e60f1 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -1296,7 +1296,7 @@ type OpenSyncTransaction = Omit< applicationReceipt?: SyncAppliedReceipt operationKeys: Set queuedBecauseHydrating: boolean - // Set by `reserveCommitTurn`: subset hydration waits for this commit. + // Set by `reserveCommitTurn`: every hydration waits for this commit. reservesCommitTurn?: boolean hasDependentSuccessor: boolean publicationAdmissionWaiters?: Set<{ @@ -1313,6 +1313,22 @@ type SyncWriteNormalization = { operation: NormalizedSyncOperation } +/** + * Returns undefined when no reserved sync transaction (`reserveCommitTurn`) + * is open. Otherwise returns a promise that settles when it commits or ends. + * No hydration may start while one is open: a new hydration cycle would make + * its commit fail. + */ +type WaitForReservedCommit = () => Promise | undefined + +/** + * Thrown where a hydration would start while a reserved sync transaction is + * open, before the hydration changes any state. Its apply-mutex turn ends, + * waits for the commit outside the mutex, and retries: the commit's own + * durable write needs the mutex. + */ +class ReservedCommitPending extends Error {} + class ApplyMutex { private queue: Promise = Promise.resolve() private pending = 0 @@ -1546,6 +1562,7 @@ class PersistedCollectionRuntime< private readonly activeSubsets = new Map() private activeHydrationContext: { suppliedRowKeys: Set } | undefined private hydrationSequence = 0 + private waitForReservedCommit: WaitForReservedCommit | undefined // An unscheduled startup read may finish after a coordinator reset publishes. private resetSequence = 0 private readonly pendingRemoteSubsetEnsures = new Map< @@ -1566,10 +1583,13 @@ class PersistedCollectionRuntime< private readonly queuedHydrationTransactions: Array< BufferedSyncTransaction > = [] + // Peer commits and resets, in arrival order. private readonly queuedTxCommitted: Array<{ - txCommitted: TxCommitted + notice: TxCommitted | CollectionReset lifecycleGeneration: number storageCollectionId: string + // Set when the notice met an open reservation; it retries after that ends. + waitsForReservedCommit?: boolean }> = [] private readonly requestIds = new WeakMap() private readonly hydratedDemands = new Set() @@ -1898,6 +1918,45 @@ class PersistedCollectionRuntime< this.restartSourceAfterScopedRecovery = restart ?? null } + setWaitForReservedCommit(wait: WaitForReservedCommit): void { + this.waitForReservedCommit = wait + } + + // Every hydration start calls this before it changes state. The check runs + // inside the hydration's own scope, because a shared scheduler can hold the + // scope long after its apply-mutex turn began. + private throwIfReservedCommitOpen(): void { + if (this.waitForReservedCommit?.()) throw new ReservedCommitPending() + } + + /** Runs a hydration turn, retrying it after a reserved commit it met. */ + private async runHydrationTurn( + turn: () => Promise, + ): Promise { + for (;;) { + try { + return await this.applyMutex.run(turn) + } catch (error) { + if (!(error instanceof ReservedCommitPending)) throw error + await this.waitForReservedCommit?.() + } + } + } + + // A deferred notice stays at the head of the queue, so later notices still + // apply in arrival order once the reserved transaction commits. + private flushAfterReservedCommit(): void { + const lifecycleGeneration = this.lifecycleGeneration + void Promise.resolve(this.waitForReservedCommit?.()) + .then(() => { + if (lifecycleGeneration !== this.lifecycleGeneration) return + return this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) + }) + .catch((error) => { + this.markTerminalFailure(error, lifecycleGeneration) + }) + } + clearSourceRestartAfterScopedRecovery(): void { this.restartSourceAfterScopedRecovery = undefined } @@ -1969,6 +2028,17 @@ class PersistedCollectionRuntime< await this.ensureCacheClaimActive() } await this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) + // A broadcast whose reload waits for a reserved commit has not applied + // yet, so the load waits with it. + for (;;) { + const closed = + this.queuedTxCommitted.length > 0 + ? this.waitForReservedCommit?.() + : undefined + if (!closed) break + await closed + await this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) + } // A broadcast that failed to apply made the runtime terminal; its rows // are not visible, so the load must not settle successfully. this.throwIfTerminal() @@ -2314,7 +2384,7 @@ class PersistedCollectionRuntime< if (this.syncMode !== `on-demand`) return if (this.scopedRecovery) return - const appliedCursor = await this.applyMutex.run(async () => { + const appliedCursor = await this.runHydrationTurn(async () => { const result = await this.runInHydrationScope((adapter) => this.hydrateBaseline(lifecycleGeneration, adapter), ) @@ -2752,15 +2822,25 @@ class PersistedCollectionRuntime< this.activeSubsets.set(subsetKey, options) const appliedCursor = this.appliedReceiptSequence try { - const hydrated = await this.applyMutex.run( + const hydrated = await this.runHydrationTurn( async (): Promise => { - if (options.signal?.aborted) return `aborted` + // The turn may start long after admission, behind other work or a + // reserved commit; demand retired or cleaned up meanwhile reads + // nothing. + if ( + options.signal?.aborted || + (isDemandRetained !== undefined && !isDemandRetained()) + ) { + return `aborted` + } try { return await this.runInHydrationScope( (adapter) => { if (options.signal?.aborted) { return Promise.resolve(`aborted`) } + // Until the reserved commit lands, the load can still abort. + this.throwIfReservedCommitOpen() onHydrationStarted?.() return this.hydrateSubsetUnsafe( options, @@ -2985,6 +3065,7 @@ class PersistedCollectionRuntime< await this.ensureCacheClaimActive() const lifecycleGeneration = this.lifecycleGeneration // A one-shot refresh does not acquire an enduring subscription lease. + // Only a sync-absent Collection exposes it, so no reservation can be open. const hydrated = await this.applyMutex.run(async () => { const result = await this.runInHydrationScope((adapter) => this.hydrateSubsetUnsafe( @@ -3325,6 +3406,7 @@ class PersistedCollectionRuntime< }, adapter: HydrationPersistenceAdapter, ): Promise { + this.throwIfReservedCommitOpen() let rowsLoaded = false let boundRowVersion: number | undefined let replayFailure: { reason: unknown } | undefined @@ -4613,7 +4695,7 @@ class PersistedCollectionRuntime< // a renewed generation still owes its rows. A hydration flushes the // queue when it ends. this.queuedTxCommitted.push({ - txCommitted: payload, + notice: payload, lifecycleGeneration, storageCollectionId: message.collectionId, }) @@ -4635,32 +4717,41 @@ class PersistedCollectionRuntime< } if (isCollectionResetPayload(payload)) { - void this.applyMutex - .run(async () => { - if (message.collectionId !== this.storageCollectionId) return - await this.runInHydrationScope(async (adapter) => { - await this.truncateAndReloadUnsafe(adapter, lifecycleGeneration) - await this.rebaseDurableStreamFromSnapshotUnsafe( - adapter, - lifecycleGeneration, - ) - if (lifecycleGeneration === this.lifecycleGeneration) { - // Reloaded rows and the later position-only read are separate - // cuts. Zero is the conservative public version in the new epoch. - this.publicRowVersion = 0 - this.resetTermFence = this.latestTerm - } + // A reset queues with peer commits, so a commit that arrived after it + // applies after it, behind its term fence. + this.queuedTxCommitted.push({ + notice: payload, + lifecycleGeneration, + storageCollectionId: message.collectionId, + }) + if (!this.isHydratingNow()) { + void this.applyMutex + .run(() => this.flushQueuedTxCommittedUnsafe()) + .catch((error) => { + this.markTerminalFailure(error, lifecycleGeneration) }) - if (lifecycleGeneration === this.lifecycleGeneration) { - await this.flushQueuedTxCommittedUnsafe() - } - }) - .catch((error) => { - this.markTerminalFailure(error, lifecycleGeneration) - }) + } } } + private async applyCollectionResetUnsafe( + lifecycleGeneration: number, + ): Promise { + await this.runInHydrationScope(async (adapter) => { + await this.truncateAndReloadUnsafe(adapter, lifecycleGeneration) + await this.rebaseDurableStreamFromSnapshotUnsafe( + adapter, + lifecycleGeneration, + ) + if (lifecycleGeneration === this.lifecycleGeneration) { + // Reloaded rows and the later position-only read are separate + // cuts. Zero is the conservative public version in the new epoch. + this.publicRowVersion = 0 + this.resetTermFence = this.latestTerm + } + }) + } + private async flushQueuedTxCommittedUnsafe(): Promise { while (this.queuedTxCommitted.length > 0) { // Claim recovery flushes the queue once the generation is decided. @@ -4668,19 +4759,39 @@ class PersistedCollectionRuntime< this.scheduleCacheClaimRecovery(this.lifecycleGeneration) return } - const queued = this.queuedTxCommitted.shift() - if (!queued) { - continue + // A notice that waited retries only after the reservation ends, so + // notices that arrive meanwhile neither retry it nor overtake it. + if ( + this.queuedTxCommitted[0]!.waitsForReservedCommit && + this.waitForReservedCommit?.() + ) { + this.flushAfterReservedCommit() + return } + const queued = this.queuedTxCommitted.shift()! if (queued.storageCollectionId !== this.storageCollectionId) continue - // A queued commit has left the queue, so a failure must be terminal, - // as it is for a commit applied when it arrives. try { - await this.processCommittedTxUnsafe( - queued.txCommitted, - queued.lifecycleGeneration, - ) + if (isCollectionResetPayload(queued.notice)) { + await this.applyCollectionResetUnsafe(queued.lifecycleGeneration) + } else { + await this.processCommittedTxUnsafe( + queued.notice, + queued.lifecycleGeneration, + ) + } } catch (error) { + if (error instanceof ReservedCommitPending) { + // Only a notice that reaches a hydration waits; later notices stay + // queued behind it. The reserved commit lands first and takes the + // next stream position, so the retry may run as gap recovery, which + // rereads durable rows (maintainer decision on #2226). + queued.waitsForReservedCommit = true + this.queuedTxCommitted.unshift(queued) + this.flushAfterReservedCommit() + return + } + // A queued commit has left the queue, so a failure must be terminal, + // as it is for a commit applied when it arrives. throw this.markTerminalFailure(error, queued.lifecycleGeneration) } } @@ -4950,6 +5061,30 @@ class PersistedCollectionRuntime< this.refreshScopedSubsets() return } + // The reset is one hydration cycle from here, before its eager truncate + // publishes. A sync transaction that begins during it straddles the + // hydration instead of crossing it, so readers never wait on a truncated + // Collection. + this.throwIfReservedCommitOpen() + this.hydrationSequence++ + this.hydratingGeneration = lifecycleGeneration + try { + await this.truncateThenReloadUnsafe(adapter, lifecycleGeneration) + } finally { + // The reload ends the cycle; this covers an early return before it. + if ( + this.hydratingGeneration === lifecycleGeneration && + this.activeHydrationContext === undefined + ) { + this.hydratingGeneration = null + } + } + } + + private async truncateThenReloadUnsafe( + adapter: HydrationPersistenceAdapter, + lifecycleGeneration: number, + ): Promise { // A subscriber may reacquire reentrantly from the truncate commit. this.hydratedDemands.clear() this.resetSequence++ @@ -4977,6 +5112,7 @@ class PersistedCollectionRuntime< false, this.syncMode === `eager`, true, + true, ) } @@ -5126,11 +5262,14 @@ class PersistedCollectionRuntime< skipUnchangedReplace = false, restoreResetBaseline = false, replayDemand = false, + // A reset's reload continues the cycle its truncate began. + continuesHydrationCycle = false, ): Promise { if (this.scopedRecovery) { this.refreshScopedSubsets() return } + if (!continuesHydrationCycle) this.throwIfReservedCommitOpen() const lifecycleGeneration = this.lifecycleGeneration const truncateGeneration = this.sourceTruncateGeneration const activeSubsetOptions = Array.from(this.activeSubsets.values()) @@ -5159,7 +5298,7 @@ class PersistedCollectionRuntime< .map(([key]) => key) this.hydratedDemands.clear() - this.hydrationSequence++ + if (!continuesHydrationCycle) this.hydrationSequence++ const hydrationContext = { suppliedRowKeys: new Set() } this.activeHydrationContext = hydrationContext this.hydratingGeneration = lifecycleGeneration @@ -5863,6 +6002,24 @@ function createWrappedSyncConfig< }, ) + const findReservedTransaction = () => + transactionStack.find( + (transaction) => + transaction.reservesCommitTurn === true && !transaction.internal, + ) + runtime.setWaitForReservedCommit(() => { + const transaction = findReservedTransaction() + if (!transaction) return undefined + // Admission waiters settle when the transaction commits, aborts, or + // cleanup discards it. Every outcome closes the reservation. + return new Promise((resolve) => { + ;(transaction.publicationAdmissionWaiters ??= new Set()).add({ + resolve, + reject: () => resolve(), + }) + }) + }) + const persistenceCapability: SyncPersistenceCapabilityV1 = { protocol: SYNC_PERSISTENCE_PROTOCOL, version: SYNC_PERSISTENCE_VERSION, @@ -5879,6 +6036,13 @@ function createWrappedSyncConfig< }, hydrateBaseline: async () => { if (startupState.cleanedUp) return + // The source holds this reservation itself, so waiting for its + // commit here would wait for the caller. + if (findReservedTransaction()) { + throw new InvalidPersistedCollectionConfigError( + `hydrateBaseline cannot run while this source's reserved sync transaction is open`, + ) + } try { await runtime.ensureResumeBaselineHydrated() } catch (error) { @@ -6636,8 +6800,12 @@ function createWrappedSyncConfig< return } const currentTransaction = getOpenTransaction() - // A source transaction that reserved its FIFO turn commits before + // A sync transaction that reserved its FIFO turn commits before // subset hydration advances the generation it was built against. + // A load admitted during the reservation waits here, before it + // takes an apply-mutex turn, so it runs after the commit's durable + // write. The runtime's deferral covers hydrations whose turn + // began earlier; it cannot keep that FIFO order. if (currentTransaction?.reservesCommitTurn) { await waitForPublicationAdmission( currentTransaction, diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index 647a9520e4..c774772695 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -1,3 +1,4 @@ +import { isDeepStrictEqual } from 'node:util' import { describe, expect, it, vi } from 'vitest' import { fc, test as fcTest } from '@fast-check/vitest' import { @@ -20,6 +21,7 @@ import { oracleRuns, readOracleRunConfig, } from '../../db/tests/oracle-config.js' +import { createOracleReach } from '../../db/tests/oracle-reach.js' import { IndeterminateCommitError, InvalidPersistedCollectionConfigError, @@ -14910,7 +14912,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // insert/update, and absent/present dependent successors. The two-read // unreserved case // reconstructs the report; one and three distinguish a misplaced threshold. - // Open transactions crossing hydration are excluded here and rejected below. + // Open transactions crossing hydration are excluded here. Unreserved ones are + // rejected below; reserved ones are covered by the persisted-read precedence + // grammar's open commit timings. // This driver supplies adapter scheduling, not live Electric or OPFS delivery. it.each( ([`insert`, `update`] as const).flatMap((type) => @@ -17560,290 +17564,1205 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { }, ) - // This bounded grammar crosses insert/update/delete with no truncate replay, - // replay before the change, and replay after it. Each history runs with the - // persistence read already finished and with accepted commits waiting behind - // that read. Both schedules must satisfy the same independent result below. - // Eager and on-demand Collections receive reset/full-reload notifications; - // on-demand Collections acquire bounded or complete subsets. A full reload - // with no demand validates already public rows without publishing unrelated - // stored rows. The controlled coordinator does not prove host delivery. - it.each( - ( - [ - { syncMode: `on-demand`, route: `subset` }, - { syncMode: `on-demand`, route: `complete subset` }, - { syncMode: `on-demand`, route: `collection-reset` }, - { syncMode: `on-demand`, route: `full-reload` }, - { syncMode: `eager`, route: `collection-reset` }, - { syncMode: `eager`, route: `full-reload` }, - ] as const - ).flatMap(({ syncMode, route }) => - ([`insert`, `update`, `delete`] as const).flatMap((operation) => - ([`none`, `before`, `after`] as const).flatMap((truncateReplay) => - [false, true].map((commitBeforeRead) => ({ - syncMode, - route, - operation, - truncateReplay, - commitBeforeRead, - })), - ), - ), - ), - )( - `preserves sync history over persisted reads ($syncMode, $route, $operation, truncate=$truncateReplay, queued=$commitBeforeRead)`, - async ({ - syncMode, - route, - operation, - truncateReplay, - commitBeforeRead, - }) => { - const storedRow = { id: `shared`, title: `persisted`, detail: `old` } - const storedOnly = { id: `stored-only`, title: `removed by truncate` } - const gateRow = { id: `gate`, title: `earlier sync transaction` } - const networkRow = { id: `shared`, title: `network`, detail: `new` } - const replacementRow = { id: `replacement`, title: `truncate replay` } - const adapter = createRecordingAdapter([ - { ...storedRow }, - { ...storedOnly }, - ]) - adapter.rowMetadata.set(`shared`, { owner: `persisted` }) - adapter.collectionMetadata.set(`cursor`, `persisted`) - const coordinator = createFailStopCoordinatorHarness( - `persisted-read-precedence`, + // # Do persisted reads preserve sync history, and do reserved commits survive them? + // + // Law 1 (persisted-baseline precedence, maintainer decision on #2036): + // persisted rows are an older baseline for newer sync-adapter changes, and + // an internal read never revokes an accepted sync transaction. Every + // history must reach the independent final rows, row metadata, and cursor + // below, and a persistence read must observe the durable rows after exactly + // the source steps whose durable writes completed before it. + // + // Law 2 (reserved commits): a persisted hydration that would start while a + // reserved sync transaction is open waits until that transaction ends, and + // the commit is accepted. Authority: SyncPersistenceCapabilityV1 + // `reserveCommitTurn` (packages/db/src/types.ts), widened by maintainer + // decision on #2226 to every persisted hydration. Maintainer decisions on + // #2226 also fix its edges: + // - While a hydration waits, readers keep the rows from before it began. + // A hydration that already began when the reservation opened continues; + // the transaction straddles it. + // - A notice that waited may be retried as gap recovery, because the + // reserved commit lands first and takes the next stream position: at + // most one more pullSince request, and no added upstream request. + // - A source that stops for good abandons its reserved transaction; waiting + // hydrations then proceed and the abandoned writes never apply. + // - A peer change that starts no hydration is not delayed. A load aborted + // while it waits settles before the commit. Cleanup while a hydration + // waits settles and reads nothing more. A pre-reset commit never publishes + // its own rows. A source that awaits its own baseline hydration inside its + // reservation is rejected at once rather than waiting on itself. + // + // History grammar. A held earlier durable write ("gate") queues the route, + // the hydration that a cell exercises: + // - `subset`, `complete subset`: on-demand demand; + // - `routed subset`: a follower's demand, which the owner serves by + // committing a row, broadcasting it, then replying; + // - `source baseline`: the source's own `hydrateBaseline` request; + // - `collection-reset`, `full-reload`, `seq-gap`, `seq-gap pull`: peer + // notices (`seq-gap pull` recovers through pullSince); + // - `targeted`: a peer change for a key that is not public, which starts + // no hydration. + // The commit timing places the first sync transaction against the route: + // - `after read`: it begins after the read is published; + // - `queued`: it begins and commits while the read waits; + // - `open across mutex turn`: it begins and reserves while the read waits + // behind the gate, before the read's turn; + // - `open across scope entry`: it reserves after the read's turn began and + // its hydration scope was entered, as a shared scheduler can hold it; + // - `open during the read`: it reserves while the route's first persisted + // read is in progress, so the transaction straddles the hydration (for an + // eager reset, after its truncate published); + // - `open in the first publication`: it reserves from the route's first + // public change, through a microtask, as a source reacting to it would; + // for an eager reset that lands between its truncate and its reload; + // - `open across startup`: the source begins and reserves inside its sync + // function, before startup has settled. + // The ending says how the reserved transaction ends: it `commits`, is + // `abandoned` (committed with an aborted signal), meets `cleanup` of the + // Collection, or the route's demand is aborted while it waits (`demand + // aborted`); with `source awaits own baseline` the source also awaits + // `hydrateBaseline` inside its reservation. A follow-up delivers, while + // the route waits, a peer reset and then a pre-reset peer commit whose row + // the reset superseded. + // + // Two registrations share this one driver and model: the row-algebra cross + // (every operation and truncate replay; committed endings; the original + // routes and timings), and the reservation cross (every route, timing, + // ending, and follow-up, with one insert and no truncate replay). The + // controlled coordinator does not prove host delivery; the Browser-host + // receiving witness is the multi-tab Electric oracle. + type PersistedReadRoute = + | `subset` + | `complete subset` + | `routed subset` + | `source baseline` + | `collection-reset` + | `full-reload` + | `seq-gap` + | `seq-gap pull` + | `targeted` + type PersistedReadTiming = + | `after read` + | `queued` + | `open across mutex turn` + | `open across scope entry` + | `open during the read` + | `open in the first publication` + | `open across startup` + type PersistedReadEnding = + | `commits` + | `abandoned` + | `cleanup` + | `demand aborted` + | `source awaits own baseline` + type PersistedReadHistory = { + syncMode: `eager` | `on-demand` + route: PersistedReadRoute + operation: `insert` | `update` | `delete` + truncateReplay: `none` | `before` | `after` + commitTiming: PersistedReadTiming + ending: PersistedReadEnding + followUp: `none` | `reset then pre-reset commit` + } + type PersistedReadReach = + | `route performed a persisted read` + | `reservation opened while the read was pending` + | `reservation opened while a read was in progress` + | `reservation opened in the route's first publication` + | `reservation opened before startup settled` + | `peer change announced while reserved` + | `reset and pre-reset commit announced while waiting` + + const runPersistedReadHistory = async ({ + syncMode, + route, + operation, + truncateReplay, + commitTiming, + ending, + followUp, + }: PersistedReadHistory) => { + const storedRow = { id: `shared`, title: `persisted`, detail: `old` } + const storedOnly = { id: `stored-only`, title: `removed by truncate` } + const gateRow = { id: `gate`, title: `earlier sync transaction` } + const networkRow = { id: `shared`, title: `network`, detail: `new` } + const replacementRow = { id: `replacement`, title: `truncate replay` } + const peerRow = { id: `peer`, title: `peer change` } + const remoteRow = { id: `remote`, title: `owner` } + const supersededRow = { id: `superseded`, title: `before the reset` } + const adapter = createRecordingAdapter([ + { ...storedRow }, + { ...storedOnly }, + ]) + adapter.rowMetadata.set(`shared`, { owner: `persisted` }) + adapter.collectionMetadata.set(`cursor`, `persisted`) + const coordinator = createFailStopCoordinatorHarness( + `persisted-read-precedence`, + ) + const waits = + commitTiming === `open across mutex turn` || + commitTiming === `open across scope entry` + // The transaction straddles a hydration that already began. + const straddles = + commitTiming === `open during the read` || + commitTiming === `open in the first publication` + const reserves = + waits || straddles || commitTiming === `open across startup` + const persistenceEntered = createEventGate() + const releasePersistence = createEventGate() + const readApplied = createEventGate() + const trace: Array = [] + // Every adapter call advances this counter. The quiescence checkpoint + // reads it; it is driver evidence, not part of the model. + let adapterCalls = 0 + const applyCommittedTx = adapter.applyCommittedTx.bind(adapter) + adapter.applyCommittedTx = async (...args) => { + adapterCalls++ + if (args[1].mutations.some(({ key }) => key === `gate`)) { + persistenceEntered.resolve() + await releasePersistence.promise + } + await applyCommittedTx(...args) + // Record durable completion, including a delete or truncate. A rejected + // adapter write must never appear as a completed source obligation. + if (args[1].truncate) trace.push(`replacement`) + else if (args[1].mutations.some(({ key }) => key === `shared`)) { + trace.push(`network`) + } + } + // The fixture leader allocates each commit the next durable position, + // after every commit already stored, including the peer commits that + // notices announce. A Browser leader does the same + // (`nextPersistedTxIfLeader`), so two commits never share a position. + const leaderPosition = { seq: 0, rowVersion: 0 } + const advanceLeaderPosition = (seq: number, rowVersion: number) => { + leaderPosition.seq = Math.max(leaderPosition.seq, seq) + leaderPosition.rowVersion = Math.max( + leaderPosition.rowVersion, + rowVersion, ) - const persistenceEntered = createEventGate() - const releasePersistence = createEventGate() - const readApplied = createEventGate() - const trace: Array = [] - const applyCommittedTx = adapter.applyCommittedTx.bind(adapter) - adapter.applyCommittedTx = async (...args) => { - if (args[1].mutations.some(({ key }) => key === `gate`)) { - persistenceEntered.resolve() - await releasePersistence.promise - } - await applyCommittedTx(...args) - // Record durable completion, including a delete or truncate. A rejected - // adapter write must never appear as a completed source obligation. - if (args[1].truncate) trace.push(`replacement`) - else if (args[1].mutations.some(({ key }) => key === `shared`)) { - trace.push(`network`) - } + } + const nextPeerPosition = () => { + advanceLeaderPosition( + leaderPosition.seq + 1, + leaderPosition.rowVersion + 1, + ) + return { seq: leaderPosition.seq, rowVersion: leaderPosition.rowVersion } + } + // A peer commit stored durably at the next position, then announced. + const peerCommit = async (row: Todo, txId: string) => { + const position = nextPeerPosition() + await applyCommittedTx( + { kind: `eager`, collectionId: `persisted-read-precedence` }, + { + txId, + term: 1, + seq: position.seq, + rowVersion: position.rowVersion, + mutations: [{ type: `insert`, key: row.id, value: row }], + } as never, + ) + if (row.id === remoteRow.id) trace.push(`owner commit`) + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: position.seq, + txId, + latestRowVersion: position.rowVersion, + requiresFullReload: false, + changedRows: [{ key: row.id, value: row }], + deletedKeys: [], + }) + } + // Every route but `seq-gap` has a leader that answers pullSince with a + // full reload. + let pullSinceCalls = 0 + coordinator.pullSince = + route === `seq-gap` + ? undefined + : () => { + pullSinceCalls++ + return Promise.resolve({ + type: `rpc:pullSince:res`, + rpcId: `gap-pull`, + ok: true, + latestTerm: 1, + latestSeq: leaderPosition.seq, + latestRowVersion: leaderPosition.rowVersion, + requiresFullReload: true, + }) + } + // Upstream requests: direct source loads and remote-subset ensures, + // which owned demand, a follower's routed demand, and recovery's refetch + // go through. A follower's owner commits its row, broadcasts, then + // replies, as law 7 assumes. + let sourceLoads = 0 + if (route === `routed subset`) coordinator.isLeader = () => false + const requestEnsureRemoteSubset = + coordinator.requestEnsureRemoteSubset.bind(coordinator) + let ownerCommitted = false + coordinator.requestEnsureRemoteSubset = async (...args) => { + sourceLoads++ + if (route === `routed subset` && !ownerCommitted) { + ownerCommitted = true + await peerCommit(remoteRow, `owner-commit`) + return } - coordinator.requestApplyCommittedTx = async (id, tx) => { - await adapter.applyCommittedTx({ kind: `eager`, collectionId: id }, tx) - return { - type: `rpc:applyCommittedTx:res`, - rpcId: tx.txId, - ok: true, - term: tx.term, - seq: tx.seq, - latestRowVersion: tx.rowVersion, + return requestEnsureRemoteSubset(...args) + } + coordinator.requestApplyCommittedTx = async (id, tx) => { + const positioned = { + ...tx, + seq: Math.max(tx.seq, leaderPosition.seq + 1), + rowVersion: Math.max(tx.rowVersion, leaderPosition.rowVersion + 1), + } + advanceLeaderPosition(positioned.seq, positioned.rowVersion) + await adapter.applyCommittedTx( + { kind: `eager`, collectionId: id }, + positioned, + ) + return { + type: `rpc:applyCommittedTx:res`, + rpcId: tx.txId, + ok: true, + term: positioned.term, + seq: positioned.seq, + latestRowVersion: positioned.rowVersion, + } + } + // Reads. A reserved sync transaction is open between its reservation and + // its end. A read that starts in that window, outside a straddled + // hydration, belongs to a hydration that did not wait. Every adapter read + // method is recorded. When armed, the first read after the route was + // admitted is held, so a reservation can open while it is in progress. + let reservationOpen = false + let routeAdmitted = false + let cleanupStarted = false + const routeReads: Array = [] + const readsDuringReservation: Array = [] + const readsAfterCleanup: Array = [] + const startupOrder: Array = [] + let heldRead: + | { + entered: ReturnType + release: ReturnType } + | undefined + const recordRead = async (method: string) => { + adapterCalls++ + if (!routeAdmitted) startupOrder.push(`startup read`) + if (routeAdmitted) routeReads.push(method) + if (reservationOpen) readsDuringReservation.push(method) + if (cleanupStarted) readsAfterCleanup.push(method) + const hold = routeAdmitted ? heldRead : undefined + if (hold) { + heldRead = undefined + hold.entered.resolve() + await hold.release.promise } - const loadSubset = adapter.loadSubset.bind(adapter) - const readRows: Array>> = [] - adapter.loadSubset = async (...args) => { - const rows = await loadSubset(...args) - readRows.push(rows.map(({ value }) => ({ ...value }))) - trace.push(`persisted read`) - return rows + } + const loadSubset = adapter.loadSubset.bind(adapter) + const readRows: Array>> = [] + adapter.loadSubset = async (...args) => { + await recordRead(`loadSubset`) + const rows = await loadSubset(...args) + readRows.push(rows.map(({ value }) => ({ ...value }))) + trace.push(`persisted read`) + return rows + } + const loadResumeSnapshot = adapter.loadResumeSnapshot.bind(adapter) + adapter.loadResumeSnapshot = async (...args) => { + await recordRead(`loadResumeSnapshot`) + return loadResumeSnapshot(...args) + } + // The recording adapter defines every optional read method. + const loadCollectionMetadata = adapter.loadCollectionMetadata!.bind(adapter) + adapter.loadCollectionMetadata = async (...args) => { + await recordRead(`loadCollectionMetadata`) + return loadCollectionMetadata(...args) + } + const scanRows = adapter.scanRows!.bind(adapter) + adapter.scanRows = async (...args) => { + await recordRead(`scanRows`) + return scanRows(...args) + } + const sourceSteps = + truncateReplay === `none` + ? [`network`] + : truncateReplay === `before` + ? [`replacement`, `network`] + : [`network`, `replacement`] + let sourceParams!: TodoSyncParams + const writeSourceStep = (step: (typeof sourceSteps)[number]) => { + if (step === `replacement`) { + sourceParams.truncate() + sourceParams.write({ type: `insert`, value: { ...replacementRow } }) + sourceParams.metadata!.row.set(`replacement`, { owner: `replacement` }) + } else if (operation === `delete`) { + sourceParams.write({ type: `delete`, key: `shared` }) + } else { + sourceParams.write({ type: operation, value: { ...networkRow } }) + sourceParams.metadata!.row.set(`shared`, { owner: `network` }) } - let sourceParams!: TodoSyncParams - const collection = createCollection( - persistedCollectionOptions({ - id: `persisted-read-precedence`, - getKey: (row) => row.id, - syncMode, - sync: { - sync: (params) => { - sourceParams = params - params.markReady() - return { loadSubset: () => true } - }, + sourceParams.metadata!.collection.set(`cursor`, step) + } + const reach = createOracleReach( + `persistence.reserved-commit-read-precedence`, + [ + ...(ending === `cleanup` || + ((route === `targeted` || ending === `demand aborted`) && + followUp === `none`) + ? [] + : ([`route performed a persisted read`] as const)), + ...(waits + ? ([`reservation opened while the read was pending`] as const) + : []), + ...(commitTiming === `open during the read` + ? ([`reservation opened while a read was in progress`] as const) + : []), + ...(commitTiming === `open in the first publication` + ? ([`reservation opened in the route's first publication`] as const) + : []), + ...(commitTiming === `open across startup` + ? ([`reservation opened before startup settled`] as const) + : []), + ...(route === `targeted` && waits + ? ([`peer change announced while reserved`] as const) + : []), + ...(followUp !== `none` && waits + ? ([`reset and pre-reset commit announced while waiting`] as const) + : []), + ], + ) + reach.beginHistory() + const beginReservedSourceStep = () => { + sourceParams.begin() + sourceParams.metadata!.persistence!.reserveCommitTurn() + writeSourceStep(sourceSteps[0]!) + reservationOpen = true + } + const collection = createCollection( + persistedCollectionOptions({ + id: `persisted-read-precedence`, + getKey: (row) => row.id, + syncMode, + sync: { + sync: (params) => { + sourceParams = params + if (commitTiming === `open across startup`) { + startupOrder.push(`source entered`) + beginReservedSourceStep() + reach.reach(`reservation opened before startup settled`) + } + params.markReady() + return { + loadSubset: () => { + sourceLoads++ + return true + }, + } }, - persistence: { adapter, coordinator }, - }), - ) - const options = route === `complete subset` ? {} : { limit: 100 } - const sourceSteps = - truncateReplay === `none` - ? [`network`] - : truncateReplay === `before` - ? [`replacement`, `network`] - : [`network`, `replacement`] - const noDemandRead = - syncMode === `on-demand` && - route !== `subset` && - route !== `complete subset` - - // Independent snapshot algebra: a later replacement discards everything - // before it. Otherwise the baseline survives only when there is no - // replacement, and the newer change decides the shared key. Complete row - // values and explicit metadata avoid the separate partial-update contract. - // Source steps are input history, never observations from production. - const expectedRows = - truncateReplay === `after` + }, + persistence: { adapter, coordinator }, + }), + ) + const abortDemand = new AbortController() + const options = + route === `complete subset` + ? { signal: abortDemand.signal } + : { limit: 100, signal: abortDemand.signal } + const demandRoute = + route === `subset` || + route === `complete subset` || + route === `routed subset` + const noDemandRead = + syncMode === `on-demand` && !demandRoute && route !== `source baseline` + // Rows the route adds durably. The owner's row for a routed subset lands + // after the follower's own read, so reads never see it. + const routeRows = + route === `targeted` + ? [peerRow] + : route === `routed subset` + ? [remoteRow] + : [] + const lateRowIds = new Set(route === `routed subset` ? [remoteRow.id] : []) + const readBaselineRows = [ + storedRow, + storedOnly, + gateRow, + ...routeRows, + ].filter((row) => !lateRowIds.has(row.id)) + // Steps that apply: an abandoned reservation applies none of its writes. + const appliedSteps = + ending === `abandoned` ? sourceSteps.slice(1) : sourceSteps + const abandonedStep = ending === `abandoned` ? sourceSteps[0] : undefined + + // Independent snapshot algebra: a later replacement discards everything + // before it. Otherwise the baseline survives only when there is no + // replacement, and the newer change decides the shared key. Complete row + // values and explicit metadata avoid the separate partial-update contract. + // Source steps are input history, never observations from production. + const baselineRows = [storedRow, storedOnly, gateRow, ...routeRows] + const expectedRows = + abandonedStep !== undefined + ? baselineRows + : truncateReplay === `after` ? [replacementRow] : [ ...(truncateReplay === `before` ? [replacementRow] - : [storedOnly, gateRow]), + : [storedOnly, gateRow, ...routeRows]), ...(operation === `delete` ? [] : [networkRow]), ] - const expectedPublicRows = - noDemandRead && truncateReplay === `none` - ? [gateRow, ...(operation === `delete` ? [] : [networkRow])] - : expectedRows - const expectedMetadata = new Map([ - ...(truncateReplay === `none` - ? [] - : [[`replacement`, { owner: `replacement` }] as const]), - ...(truncateReplay !== `after` && operation !== `delete` - ? [[`shared`, { owner: `network` }] as const] - : []), - ]) - const expectedCursor = - truncateReplay === `after` ? `replacement` : `network` - // Await applied receipts and the optional subset-load promise. - const pendingOperations: Array> = [] - const retainOperation = (receipt: true | void | Promise) => { - const pending = Promise.resolve(receipt) - void pending.catch(() => undefined) - pendingOperations.push(pending) - } - let hasPrimaryFailure = false + // On-demand without demand exposes the source's own rows; an aborted + // demand loads nothing. Every other route exposes the durable rows. + // Uncertified on-demand cache rows stay quarantined after a baseline + // hydration (electric.ts: "loaded only through fresh subset snapshots"). + const exposesSourceRowsOnly = + noDemandRead || route === `source baseline` || ending === `demand aborted` + const expectedPublicRows = + exposesSourceRowsOnly && truncateReplay === `none` + ? [ + gateRow, + ...(abandonedStep !== undefined || operation === `delete` + ? [] + : [networkRow]), + ] + : expectedRows + const sharedMetadata = + abandonedStep !== undefined + ? exposesSourceRowsOnly + ? { public: undefined, durable: { owner: `persisted` } } + : { public: { owner: `persisted` }, durable: { owner: `persisted` } } + : truncateReplay !== `after` && operation !== `delete` + ? { public: { owner: `network` }, durable: { owner: `network` } } + : { public: undefined, durable: undefined } + const replacementMetadata = + truncateReplay === `none` ? undefined : { owner: `replacement` } + const expectedCursor = + abandonedStep !== undefined + ? `persisted` + : truncateReplay === `after` + ? `replacement` + : `network` + // Pending operations and the outcome each one owes. + const pendingOperations: Array> = [] + const expectedOutcomes: Array<`fulfilled` | `rejected`> = [] + const retainOperation = ( + receipt: unknown, + expected: `fulfilled` | `rejected` = `fulfilled`, + ) => { + const pending = Promise.resolve(receipt) + void pending.catch(() => undefined) + pendingOperations.push(pending) + expectedOutcomes.push(expected) + } + const commitSourceTransaction = (signal?: AbortSignal) => { try { - await atPersistedOracleCheckpoint( - collection.stateWhenReady(), - `precedence collection ready`, + retainOperation( + sourceParams.commit(signal), + signal ? `rejected` : `fulfilled`, ) - // Hold an earlier durable write so the real runtime queues its read and - // ordinary sync transactions. The scope callback witnesses publication, - // not just completion of the adapter's load promise. - adapter.runInHydrationScope = async (task) => { - const result = await task(adapter) + } catch (error) { + retainOperation( + Promise.reject(error), + signal ? `rejected` : `fulfilled`, + ) + } + } + const endReservation = () => { + reservationOpen = false + if (ending === `abandoned`) { + const cancelled = new AbortController() + cancelled.abort() + commitSourceTransaction(cancelled.signal) + } else commitSourceTransaction() + } + const published = new Set() + let hasPrimaryFailure = false + try { + await atPersistedOracleCheckpoint( + collection.stateWhenReady(), + `precedence collection ready`, + ) + const startupObservation = + commitTiming === `open across startup` + ? { + readsBeforeSource: startupOrder.indexOf(`source entered`) > 0, + sourceEnteredLast: + startupOrder[startupOrder.length - 1] === `source entered`, + } + : undefined + if (commitTiming === `open across startup`) endReservation() + const subscription = collection.subscribeChanges( + () => { + for (const key of collection.keys()) published.add(String(key)) + }, + { includeInitialState: false }, + ) + // The scope callback witnesses publication, not just completion of the + // adapter's load promise. When armed, it holds one scope after entry + // and before its task, as a shared logical scheduler can. + let heldScope: + | { + entered: ReturnType + release: ReturnType + } + | undefined + // Set inside the scope hook, so TypeScript cannot narrow it here. + let readPublished = false as boolean + adapter.runInHydrationScope = async (task) => { + const hold = heldScope + heldScope = undefined + if (hold) { + hold.entered.resolve() + await hold.release.promise + } + const result = await task(adapter) + if (routeAdmitted) { + readPublished = true readApplied.resolve() - return result } - sourceParams.begin() - sourceParams.write({ type: `insert`, value: { ...gateRow } }) - retainOperation(sourceParams.commit()) - await atPersistedOracleCheckpoint( - persistenceEntered.promise, - `precedence predecessor held`, + return result + } + // Progress checkpoint: wait until three consecutive tasks pass with no + // adapter call; the bound turns a livelock into a named failure. + const persistenceQuiesced = async (label: string) => { + let quietTasks = 0 + for (let task = 0; task < 200 && quietTasks < 3; task++) { + const callsBefore = adapterCalls + await flushAsyncWork() + quietTasks = adapterCalls === callsBefore ? quietTasks + 1 : 0 + } + if (quietTasks < 3) { + throw new Error( + `Persisted oracle checkpoint did not quiesce: ${label}`, + ) + } + } + sourceParams.begin() + sourceParams.write({ type: `insert`, value: { ...gateRow } }) + retainOperation(sourceParams.commit()) + await atPersistedOracleCheckpoint( + persistenceEntered.promise, + `predecessor held`, + ) + const rowsAtAdmission = new Map( + Array.from(collection.values(), (row) => [ + row.id, + stripVirtualProps(row), + ]), + ) + routeAdmitted = true + let ownBaselineRejection: string | undefined + if ( + route === `subset` || + route === `complete subset` || + route === `routed subset` + ) { + retainOperation( + collection._sync.loadSubset(options), + ending === `demand aborted` ? `rejected` : `fulfilled`, ) - if (route === `subset` || route === `complete subset`) { - retainOperation(collection._sync.loadSubset(options)) - } else if (route === `collection-reset`) { - coordinator.emit({ - type: `collection:reset`, - schemaVersion: 1, - resetEpoch: 1, - }) - } else { - coordinator.emit({ - type: `tx:committed`, - term: 1, - seq: 2, - txId: `remote-reload`, - latestRowVersion: 2, - requiresFullReload: true, - }) + } else if (route === `source baseline`) { + retainOperation(sourceParams.metadata!.persistence!.hydrateBaseline()) + } else if (route === `collection-reset`) { + coordinator.emit({ + type: `collection:reset`, + schemaVersion: 1, + resetEpoch: 1, + }) + } else if (route === `full-reload`) { + const position = nextPeerPosition() + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: position.seq, + txId: `remote-reload`, + latestRowVersion: position.rowVersion, + requiresFullReload: true, + }) + } else if (route === `targeted`) { + // Waiting timings deliver the change after the reservation opens. + if (!waits && commitTiming !== `open in the first publication`) { + await peerCommit(peerRow, `peer-change`) } - await flushAsyncWork() - expect(readRows).toEqual([]) - expect(collection.get(`shared`)?.title).toBe( - syncMode === `eager` ? `persisted` : undefined, + } else { + // Sequence gap, with or without pullSince. A peer commit this tab has + // not observed arrives past a sequence gap and carries no rows, as a + // peer's must-refetch reset commit does in a multi-tab refetch storm. + // Recovery rereads durable rows. + nextPeerPosition() + const position = nextPeerPosition() + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: position.seq, + txId: `remote-gap`, + latestRowVersion: position.rowVersion, + requiresFullReload: false, + changedRows: [], + deletedKeys: [], + }) + } + await flushAsyncWork() + expect(readRows).toEqual([]) + // A peer reset, then a pre-reset peer commit whose row the reset + // superseded. The reset removed that commit's row from durable storage. + const deliverFollowUp = () => { + if (followUp === `none`) return + coordinator.emit({ + type: `collection:reset`, + schemaVersion: 1, + resetEpoch: 1, + }) + const position = nextPeerPosition() + coordinator.emit({ + type: `tx:committed`, + term: 1, + seq: position.seq, + txId: `pre-reset-commit`, + latestRowVersion: position.rowVersion, + requiresFullReload: false, + changedRows: [{ key: supersededRow.id, value: supersededRow }], + deletedKeys: [], + }) + } + let remainingSteps: ReadonlyArray<(typeof sourceSteps)[number]> = + commitTiming === `open across startup` + ? sourceSteps.slice(1) + : sourceSteps + let rowsWhileReserved: Map | undefined + let peerWhileReserved: unknown + let abortedBeforeEnd: string | undefined + const observeWhileReserved = () => { + rowsWhileReserved = new Map( + Array.from(collection.values(), (row) => [ + row.id, + stripVirtualProps(row), + ]), ) - if (!commitBeforeRead) { - releasePersistence.resolve() + } + if ( + commitTiming === `after read` || + commitTiming === `open across startup` + ) { + releasePersistence.resolve() + if (route !== `targeted`) { await atPersistedOracleCheckpoint( readApplied.promise, - `persistence read published before commit`, + `read published`, ) - } - for (const step of sourceSteps) { - sourceParams.begin() - if (step === `replacement`) { - sourceParams.truncate() - sourceParams.write({ type: `insert`, value: { ...replacementRow } }) - sourceParams.metadata!.row.set(`replacement`, { - owner: `replacement`, + } else await persistenceQuiesced(`targeted change applied`) + deliverFollowUp() + await persistenceQuiesced(`follow-up applied`) + } else if (commitTiming === `queued`) { + // The sync transaction commits while the read waits (below). + } else if (commitTiming === `open in the first publication`) { + const reserved = createEventGate() + let armed = true + const firstPublication = collection.subscribeChanges( + () => { + if (!armed) return + armed = false + queueMicrotask(() => { + beginReservedSourceStep() + reach.reach(`reservation opened in the route's first publication`) + reserved.resolve() }) - } else if (operation === `delete`) { - sourceParams.write({ type: `delete`, key: `shared` }) + }, + { includeInitialState: false }, + ) + releasePersistence.resolve() + if (route === `targeted`) await peerCommit(peerRow, `peer-change`) + await atPersistedOracleCheckpoint(reserved.promise, `route published`) + firstPublication.unsubscribe() + await persistenceQuiesced(`straddled hydration finished`) + observeWhileReserved() + endReservation() + remainingSteps = sourceSteps.slice(1) + } else if (commitTiming === `open during the read`) { + heldRead = { entered: createEventGate(), release: createEventGate() } + const { entered, release } = heldRead + releasePersistence.resolve() + await atPersistedOracleCheckpoint( + entered.promise, + `route read in progress`, + ) + beginReservedSourceStep() + reach.reach(`reservation opened while a read was in progress`) + release.resolve() + await persistenceQuiesced(`straddled read finished`) + observeWhileReserved() + endReservation() + remainingSteps = sourceSteps.slice(1) + } else { + if (commitTiming === `open across mutex turn`) { + // The read was admitted before this reservation. + beginReservedSourceStep() + if (!readPublished) { + reach.reach(`reservation opened while the read was pending`) + } + releasePersistence.resolve() + await atPersistedOracleCheckpoint( + pendingOperations[0]!, + `predecessor applied while the reservation is open`, + ) + } else { + // The read already holds its mutex turn and hydration scope. + heldScope = { entered: createEventGate(), release: createEventGate() } + const { entered, release } = heldScope + releasePersistence.resolve() + if (route === `targeted`) { + await persistenceQuiesced(`targeted change pending`) + heldScope = undefined + release.resolve() } else { - sourceParams.write({ type: operation, value: { ...networkRow } }) - sourceParams.metadata!.row.set(`shared`, { owner: `network` }) + await atPersistedOracleCheckpoint( + entered.promise, + `read entered its scope`, + ) + } + beginReservedSourceStep() + if (!readPublished) { + reach.reach(`reservation opened while the read was pending`) } - sourceParams.metadata!.collection.set(`cursor`, step) - retainOperation(sourceParams.commit()) + release.resolve() } - releasePersistence.resolve() - const outcomes = await atPersistedOracleCheckpoint( - Promise.allSettled(pendingOperations), - `sync history applied and durable`, - ) - // Compare after applied receipts and the subset load settle, before cleanup. Exact - // keys detect resurrected deletes and rows wrongly retained by truncate. - // Read contents and completion order prove that the older baseline ran - // first in both schedules. No intermediate-publication claim is made. - const metadataKeys = [`shared`, `stored-only`, `gate`, `replacement`] - expect({ - outcomes, - trace, - readRows, - status: collection.status, - publicRows: new Map( - Array.from(collection.values(), (row) => [ - row.id, - stripVirtualProps(row), - ]), - ), - durableRows: adapter.rows, - publicMetadata: metadataKeys.map((key) => - sourceParams.metadata!.row.get(key), - ), - durableMetadata: metadataKeys.map((key) => - adapter.rowMetadata.get(key), - ), - publicCursor: sourceParams.metadata!.collection.get(`cursor`), - durableCursor: adapter.collectionMetadata.get(`cursor`), - }).toEqual({ - outcomes: pendingOperations.map(() => ({ - status: `fulfilled`, - value: undefined, - })), - trace: [...(noDemandRead ? [] : [`persisted read`]), ...sourceSteps], - readRows: noDemandRead ? [] : [[storedRow, storedOnly, gateRow]], - status: `ready`, - publicRows: new Map(expectedPublicRows.map((row) => [row.id, row])), - durableRows: new Map(expectedRows.map((row) => [row.id, row])), - publicMetadata: metadataKeys.map((key) => expectedMetadata.get(key)), - durableMetadata: metadataKeys.map((key) => expectedMetadata.get(key)), - publicCursor: expectedCursor, - durableCursor: expectedCursor, + if (route === `targeted`) { + await peerCommit(peerRow, `peer-change`) + reach.reach(`peer change announced while reserved`) + } + if (followUp !== `none`) { + deliverFollowUp() + reach.reach(`reset and pre-reset commit announced while waiting`) + } + await persistenceQuiesced(`reservation open, route waiting`) + observeWhileReserved() + peerWhileReserved = stripVirtualProps(collection.get(peerRow.id)) + if (ending === `cleanup`) { + cleanupStarted = true + reservationOpen = false + await atPersistedOracleCheckpoint( + collection.cleanup(), + `cleanup settles while the route waits`, + ) + } else { + if (ending === `demand aborted`) { + abortDemand.abort() + abortedBeforeEnd = await atPersistedOracleCheckpoint( + pendingOperations[1]!.then( + () => `fulfilled`, + () => `rejected`, + ), + `aborted demand settles before the reservation ends`, + ) + } + if (ending === `source awaits own baseline`) { + ownBaselineRejection = await atPersistedOracleCheckpoint( + sourceParams.metadata!.persistence!.hydrateBaseline().then( + () => `fulfilled`, + (error: unknown) => (error as Error).message, + ), + `own baseline settles inside the reservation`, + ) + } + endReservation() + } + remainingSteps = sourceSteps.slice(1) + } + if (ending !== `cleanup`) { + for (const step of remainingSteps) { + sourceParams.begin() + writeSourceStep(step) + commitSourceTransaction() + } + } + releasePersistence.resolve() + const outcomes = await atPersistedOracleCheckpoint( + Promise.allSettled(pendingOperations), + `sync history applied and durable`, + ) + if (ending === `cleanup`) { + await flushAsyncWork() + subscription.unsubscribe() + reach.assertReached() + expect({ status: collection.status, readsAfterCleanup }).toEqual({ + status: `cleaned-up`, + readsAfterCleanup: [], }) - } catch (error) { - hasPrimaryFailure = true - throw error - } finally { - releasePersistence.resolve() - await cleanupPersistedOracle( - [ - ...(route === `subset` || route === `complete subset` - ? [() => collection._sync.unloadSubset(options)] - : []), - () => Promise.allSettled(pendingOperations), - () => collection.cleanup(), - ], - hasPrimaryFailure, + return + } + // A read that waited runs after the reservation ends. + if (reserves && route !== `targeted` && ending !== `demand aborted`) { + await atPersistedOracleCheckpoint( + readApplied.promise, + `route read published`, ) } - }, + if (followUp !== `none`) await persistenceQuiesced(`follow-up settled`) + subscription.unsubscribe() + if (routeReads.length > 0) reach.reach(`route performed a persisted read`) + reach.assertReached() + + // Observations and the values the model owes. The trace records durable + // completions (source steps and the owner's commit) beside persisted + // reads. Each read must observe the fold of the durable events before + // it, which proves the baseline it saw. Without waiting, every read + // precedes every source step, so the older baseline runs first; after a + // startup reservation commits, its step precedes the reads; while a read + // waits for a reserved commit, reads and later steps may interleave. + // Rows are compared by key: the adapter's row order is not part of the law. + const readsSomething = + !noDemandRead && + route !== `targeted` && + route !== `source baseline` && + ending !== `demand aborted` + // A routed subset reads its local rows, then rereads to admit the + // owner's row; a reset follow-up reloads, and so does the fenced + // pre-reset notice. + const expectedReadCount = + (!readsSomething ? 0 : route === `routed subset` ? 2 : 1) + + (followUp === `none` ? 0 : 2) + const applyDurableEvent = (rows: Map, event: string) => { + if (event === `replacement`) { + rows.clear() + rows.set(replacementRow.id, replacementRow) + } else if (event === `network`) { + if (operation === `delete`) rows.delete(`shared`) + else rows.set(networkRow.id, networkRow) + } else if (event === `owner commit`) rows.set(remoteRow.id, remoteRow) + } + const durableBefore = (index: number) => { + const rows = new Map(readBaselineRows.map((row) => [row.id, row])) + for (const event of trace.slice(0, index)) + applyDurableEvent(rows, event) + return rows + } + const sameRows = ( + left: Map, + right: Map, + ) => + left.size === right.size && + [...left].every(([key, value]) => + isDeepStrictEqual(value, right.get(key)), + ) + const byKey = (rows: Array) => + new Map(rows.map((row) => [row.id, row])) + const readIndexes = trace.flatMap((event, index) => + event === `persisted read` ? [index] : [], + ) + const stepIndexes = trace.flatMap((event, index) => + event === `network` || event === `replacement` ? [index] : [], + ) + const startupStepCommitted = + commitTiming === `open across startup` && abandonedStep === undefined + const observedReadOrder = waits + ? `may interleave` + : straddles + ? readIndexes.length === 0 || + stepIndexes.every((step) => readIndexes[0]! < step) + ? `straddled read before every step` + : `other` + : startupStepCommitted + ? readIndexes.every( + (read) => + read > stepIndexes[0]! && + stepIndexes.slice(1).every((step) => read < step), + ) + ? `after the startup step` + : `other` + : readIndexes.every((read) => + stepIndexes.every((step) => read < step), + ) + ? `before every step` + : `other` + const noWaitPullSinceCalls = route === `seq-gap pull` ? 1 : 0 + // A route notice waits in the waiting timings; a follower's owner notice + // arrives during a straddled reservation and waits there. Each notice + // that waited may ask the leader once more; nothing else may. + const waitedNotices = + (waits && (route === `full-reload` || route === `seq-gap pull`)) || + (straddles && route === `routed subset`) + ? 1 + : 0 + const expectedSourceLoads = + route === `subset` || + route === `complete subset` || + route === `routed subset` + ? ending === `demand aborted` + ? 0 + : 1 + : route === `seq-gap` && syncMode === `eager` + ? 1 + : 0 + const metadataKeys = [`shared`, `stored-only`, `gate`, `replacement`] + const expectedMetadata = (side: `public` | `durable`) => + metadataKeys.map((key) => + key === `shared` + ? sharedMetadata[side] + : key === `replacement` + ? replacementMetadata + : undefined, + ) + const observed = { + outcomes: outcomes.map(({ status }) => status), + readsDuringReservation: straddles ? [] : readsDuringReservation, + stepOrder: trace.filter( + (event) => event === `network` || event === `replacement`, + ), + ownerCommits: trace.filter((event) => event === `owner commit`).length, + readCount: readIndexes.length, + readsObserveDurable: readIndexes.map((index, read) => + sameRows(byKey(readRows[read] as Array), durableBefore(index)), + ), + readOrder: observedReadOrder, + finalDurable: durableBefore(trace.length), + pullSinceWithinBound: + pullSinceCalls >= noWaitPullSinceCalls && + pullSinceCalls <= noWaitPullSinceCalls + waitedNotices, + sourceLoads, + status: collection.status, + publicRows: new Map( + Array.from(collection.values(), (row) => [ + row.id, + stripVirtualProps(row), + ]), + ), + durableRows: adapter.rows, + publicMetadata: metadataKeys.map((key) => + sourceParams.metadata!.row.get(key), + ), + durableMetadata: metadataKeys.map((key) => + adapter.rowMetadata.get(key), + ), + publicCursor: sourceParams.metadata!.collection.get(`cursor`), + durableCursor: adapter.collectionMetadata.get(`cursor`), + ...(waits && route !== `targeted` ? { rowsWhileReserved } : {}), + ...(straddles ? { rowsWhileReserved } : {}), + ...(route === `targeted` && waits ? { peerWhileReserved } : {}), + ...(followUp !== `none` + ? { publishedSuperseded: published.has(supersededRow.id) } + : {}), + ...(ending === `demand aborted` ? { abortedBeforeEnd } : {}), + ...(ending === `source awaits own baseline` + ? { ownBaselineRejection } + : {}), + ...(startupObservation ? { startupObservation } : {}), + } + const expected = { + outcomes: expectedOutcomes, + readsDuringReservation: [], + stepOrder: appliedSteps, + ownerCommits: route === `routed subset` ? 1 : 0, + readCount: expectedReadCount, + readsObserveDurable: Array.from( + { length: expectedReadCount }, + () => true, + ), + readOrder: waits + ? `may interleave` + : straddles + ? `straddled read before every step` + : startupStepCommitted + ? `after the startup step` + : `before every step`, + // The fold over durable events must agree with the closed form. + finalDurable: byKey(expectedRows), + pullSinceWithinBound: true, + sourceLoads: expectedSourceLoads, + status: `ready`, + publicRows: byKey(expectedPublicRows), + durableRows: byKey(expectedRows), + publicMetadata: expectedMetadata(`public`), + durableMetadata: expectedMetadata(`durable`), + publicCursor: expectedCursor, + durableCursor: expectedCursor, + // While a hydration waits, readers keep the rows from before it; a + // straddled read has published the pre-source durable rows. + ...(waits && route !== `targeted` + ? { rowsWhileReserved: rowsAtAdmission } + : {}), + ...(straddles + ? { + rowsWhileReserved: byKey( + exposesSourceRowsOnly ? [gateRow] : readBaselineRows, + ), + } + : {}), + ...(route === `targeted` && waits + ? { peerWhileReserved: peerRow } + : {}), + ...(followUp !== `none` ? { publishedSuperseded: false } : {}), + ...(ending === `demand aborted` + ? { abortedBeforeEnd: `rejected` } + : {}), + ...(ending === `source awaits own baseline` + ? { + ownBaselineRejection: `hydrateBaseline cannot run while this source's reserved sync transaction is open`, + } + : {}), + ...(startupObservation + ? { + startupObservation: { + readsBeforeSource: true, + sourceEnteredLast: true, + }, + } + : {}), + } + expect(observed).toEqual(expected) + } catch (error) { + hasPrimaryFailure = true + throw error + } finally { + releasePersistence.resolve() + await cleanupPersistedOracle( + [ + ...(demandRoute + ? [() => collection._sync.unloadSubset(options)] + : []), + () => Promise.allSettled(pendingOperations), + () => collection.cleanup(), + ], + hasPrimaryFailure, + ) + } + } + + // Row-algebra cross: the original routes and timings, every operation and + // truncate replay, with a reservation that commits. + it.each( + ( + [ + { syncMode: `on-demand`, route: `subset` }, + { syncMode: `on-demand`, route: `complete subset` }, + { syncMode: `on-demand`, route: `collection-reset` }, + { syncMode: `on-demand`, route: `full-reload` }, + { syncMode: `on-demand`, route: `seq-gap` }, + { syncMode: `on-demand`, route: `seq-gap pull` }, + { syncMode: `eager`, route: `collection-reset` }, + { syncMode: `eager`, route: `full-reload` }, + { syncMode: `eager`, route: `seq-gap` }, + { syncMode: `eager`, route: `seq-gap pull` }, + ] as const + ).flatMap(({ syncMode, route }) => + ([`insert`, `update`, `delete`] as const).flatMap((operation) => + ([`none`, `before`, `after`] as const).flatMap((truncateReplay) => + ( + [ + `after read`, + `queued`, + `open across mutex turn`, + `open across scope entry`, + ] as const + ).map((commitTiming): PersistedReadHistory => ({ + syncMode, + route, + operation, + truncateReplay, + commitTiming, + ending: `commits`, + followUp: `none`, + })), + ), + ), + ), + )( + `preserves sync history over persisted reads ($syncMode, $route, $operation, truncate=$truncateReplay, timing=$commitTiming)`, + runPersistedReadHistory, + ) + + // Reservation cross: every route, reserving timing, ending, and follow-up, + // with one insert and no truncate replay. `after read` is each route's + // control. An ending or follow-up appears only where it can happen: cleanup, + // aborted demand, and the source's own baseline need a waiting route; the + // follow-up's reset fence is judged on eager Collections, whose reset + // reloads every durable row. + const reservationRoutes = [ + { syncMode: `on-demand`, route: `subset` }, + { syncMode: `on-demand`, route: `complete subset` }, + { syncMode: `on-demand`, route: `routed subset` }, + { syncMode: `on-demand`, route: `source baseline` }, + { syncMode: `on-demand`, route: `collection-reset` }, + { syncMode: `on-demand`, route: `full-reload` }, + { syncMode: `on-demand`, route: `seq-gap` }, + { syncMode: `on-demand`, route: `seq-gap pull` }, + { syncMode: `eager`, route: `collection-reset` }, + { syncMode: `eager`, route: `full-reload` }, + { syncMode: `eager`, route: `seq-gap` }, + { syncMode: `eager`, route: `seq-gap pull` }, + { syncMode: `eager`, route: `targeted` }, + ] as const + const reservationHistories = reservationRoutes.flatMap( + ({ syncMode, route }) => + ( + [ + `after read`, + `open across mutex turn`, + `open across scope entry`, + `open during the read`, + `open in the first publication`, + `open across startup`, + ] as const + ).flatMap((commitTiming) => { + if (commitTiming === `open during the read` && route === `targeted`) + return [] + // Only a route whose hydration publishes rows reaches a publication. + if ( + commitTiming === `open in the first publication` && + syncMode === `on-demand` && + route !== `subset` && + route !== `complete subset` && + route !== `routed subset` + ) { + return [] + } + const waits = + commitTiming === `open across mutex turn` || + commitTiming === `open across scope entry` + const endings: Array = + commitTiming === `after read` + ? [`commits`] + : waits + ? [ + `commits`, + `abandoned`, + `cleanup`, + ...(route === `subset` || route === `complete subset` + ? ([`demand aborted`] as const) + : []), + ...(route === `source baseline` + ? ([`source awaits own baseline`] as const) + : []), + ] + : [`commits`, `abandoned`] + return endings.flatMap((ending) => + ( + [ + `none`, + ...(syncMode === `eager` && + (commitTiming === `after read` || waits) && + (ending === `commits` || ending === `abandoned`) + ? ([`reset then pre-reset commit`] as const) + : []), + ] as const + ).map((followUp): PersistedReadHistory => ({ + syncMode, + route, + operation: `insert`, + truncateReplay: `none`, + commitTiming, + ending, + followUp, + })), + ) + }), + ) + it.each(reservationHistories)( + `keeps reserved commits and their neighbours lawful ($syncMode, $route, timing=$commitTiming, ending=$ending, follow-up=$followUp)`, + runPersistedReadHistory, ) it(`rejects a source transaction that crosses a hydration cycle`, async () => { diff --git a/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts new file mode 100644 index 0000000000..81dfedd23d --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts @@ -0,0 +1,429 @@ +// @vitest-environment node +/** + * # Do reserved commits survive hydrations on the real SQLite adapter? + * + * Law: a persisted hydration that would start while a reserved sync + * transaction is open waits until it ends; a commit is then accepted, and an + * abandoned transaction applies nothing (`reserveCommitTurn`, + * packages/db/src/types.ts, widened by maintainer decision on #2226). The + * primary owner is the persisted-reads grammar in persisted-oracle.test.ts, + * which uses a recording adapter and simulates a held hydration scope. This + * oracle drives the same law through `SQLiteCorePersistenceAdapter` over + * `node:sqlite`, including the shared logical scheduler, which can hold a + * hydration after its apply-mutex turn began. Its histories were reduced from + * the report in https://github.com/TanStack/db/issues/2226. + * + * Grammar: the hydration route (a peer full-reload notice, a peer reset, or a + * subset load admitted before the reservation) crossed with the timing (the + * route waits behind this Collection's held write, or the shared scheduler + * holds its scope behind another Collection's write after its turn began) and + * the ending (the reserved transaction commits, or is abandoned). + * + * Model: row-1's durable value before the reservation follows from the + * history alone: the initial row, a peer edit, or the held write's value. A + * commit makes row-1 `refetched`; an abandoned transaction leaves the durable + * value. Each cell reaches its premise: the hydration entered its scope while + * the reservation was open. + * + * Limits: the simulated peer writes straight to the adapter and announces + * through a test coordinator, outside a leader that allocates positions, so + * only row-1, the commit outcome, and Collection status are judged. The + * Browser-host owner (the multi-tab Electric oracle) judges peer rows across + * real tabs. + */ + +import { DatabaseSync } from 'node:sqlite' +import { createCollection } from '@tanstack/db' +import { afterEach, describe, expect, it } from 'vitest' +import { createOracleReach } from '../../db/tests/oracle-reach.js' +import { + SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, + SQLiteCorePersistenceAdapter, + SingleProcessCoordinator, + eagerStorageTarget, + persistedCollectionOptions, +} from '../src' +import type { ProtocolEnvelope, SQLiteDriver, TxCommitted } from '../src' +import type { SyncConfig } from '@tanstack/db' +import type { SQLInputValue } from 'node:sqlite' + +interface ShapeRow extends Record { + readonly id: string + readonly value: string +} + +/** + * A real in-memory SQLite driver whose next write transaction can be held to keep the persistence queue busy. + * `sharedScheduling` marks it as one physical database, as the browser driver does, so every adapter on it + * shares one write/hydrate scheduler. + */ +const openHoldableDatabase = ({ + sharedScheduling = false, +}: { sharedScheduling?: boolean } = {}) => { + const database = new DatabaseSync(':memory:') + let hold: { reached: () => void; released: Promise } | undefined + let transactionQueue: Promise = Promise.resolve() + const driver: SQLiteDriver = { + ...(sharedScheduling + ? { [SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY]: {} } + : {}), + exec: async (sql) => { + database.exec(sql) + }, + query: async (sql: string, params: ReadonlyArray = []) => + // SQLiteDriver's generic contract erases SQL parameter and result types. + database + .prepare(sql) + .all(...(params as Array)) as Array, + run: async (sql, params = []) => { + database.prepare(sql).run(...(params as Array)) + }, + transaction: (work) => { + const transaction = transactionQueue.then(async () => { + const current = hold + hold = undefined + if (current) { + current.reached() + await current.released + } + database.exec('BEGIN IMMEDIATE') + try { + const result = await work(driver) + database.exec('COMMIT') + return result + } catch (error) { + database.exec('ROLLBACK') + throw error + } + }) + transactionQueue = transaction.catch(() => undefined) + return transaction + }, + } + const holdNextWrite = () => { + let release!: () => void + let reached!: () => void + const wasReached = new Promise((resolve) => { + reached = resolve + }) + hold = { + reached, + released: new Promise((resolve) => (release = resolve)), + } + return { release, wasReached } + } + return { database, driver, holdNextWrite } +} + +const waitFor = async ( + assertion: () => void, + timeoutMs: number = 2_000, +): Promise => { + const deadline = Date.now() + timeoutMs + for (;;) { + try { + assertion() + return + } catch (error) { + if (Date.now() >= deadline) throw error + await new Promise((resolve) => setTimeout(resolve, 5)) + } + } +} + +/** A coordinator for one tab that lets the test deliver another tab's broadcast, as BroadcastChannel does. */ +class PeerTabCoordinator extends SingleProcessCoordinator { + private readonly listeners = new Map< + string, + (message: ProtocolEnvelope) => void + >() + + override subscribe( + collectionId?: string, + onMessage?: (message: ProtocolEnvelope) => void, + ): () => void { + if (collectionId === undefined || onMessage === undefined) + return () => undefined + this.listeners.set(collectionId, onMessage) + return () => this.listeners.delete(collectionId) + } + + deliverFromPeer(collectionId: string, payload: unknown): void { + const listener = this.listeners.get(collectionId) + if (!listener) throw new Error(`No runtime subscribed to ${collectionId}.`) + listener({ + collectionId, + dbName: 'repro', + payload, + senderId: 'peer-tab', + ts: Date.now(), + v: 1, + }) + } +} + +const COLLECTION_ID = 'items' + +type SourceSyncParams = Parameters['sync']>[0] + +/** + * A persisted collection whose source the test drives by hand, with another tab committing to the same SQLite + * database. `reserve()` opens a source transaction that reserves its commit turn, as Electric's refetch does. + */ +const openPeerTabHarness = async ({ + sharedScheduling = false, + onDemand = false, +}: { sharedScheduling?: boolean; onDemand?: boolean } = {}) => { + const { database, driver, holdNextWrite } = openHoldableDatabase({ + sharedScheduling, + }) + // The concrete class exposes the stream-position and hydration-scope methods the factory's interface marks optional. + const adapter = new SQLiteCorePersistenceAdapter({ driver }) + // Reach checkpoint: resolves when a hydration scope starts its task, after + // any scheduler delay. A case commits only after the hydration it tests has + // tried to start, so the history overlaps the reservation however slow the + // runner is. + const scopeEntryWaiters: Array<() => void> = [] + const runScope = adapter.runInHydrationScope.bind(adapter) + adapter.runInHydrationScope = (task) => + runScope((scoped) => { + for (const resolve of scopeEntryWaiters.splice(0)) resolve() + return task(scoped) + }) + const nextScopeEntry = () => + new Promise((resolve) => scopeEntryWaiters.push(resolve)) + const coordinator = new PeerTabCoordinator() + let source: SourceSyncParams | undefined + const collection = createCollection( + persistedCollectionOptions({ + id: COLLECTION_ID, + persistence: { adapter, coordinator }, + schemaVersion: 1, + getKey: (row) => row.id, + ...(onDemand ? { syncMode: 'on-demand' as const } : {}), + sync: { + sync: (params) => { + source = params + params.begin() + params.write({ + type: 'insert', + value: { id: 'row-1', value: 'initial' }, + }) + params.commit() + params.markReady() + return onDemand ? { loadSubset: () => true } : {} + }, + }, + }), + ) + const requireSource = (): SourceSyncParams => { + if (!source) throw new Error('The collection has not started its source.') + return source + } + const start = async () => { + await collection.preload() + await waitFor(() => + expect(collection.get('row-1')).toMatchObject({ value: 'initial' }), + ) + } + /** Commits a row change from the peer tab into the shared database and returns its broadcast notice. */ + const commitFromPeer = async (value: string): Promise => { + const position = await adapter.getStreamPosition( + eagerStorageTarget(COLLECTION_ID), + ) + const tx = { + mutations: [ + { + key: 'row-1', + type: 'update' as const, + value: { id: 'row-1', value }, + }, + ], + rowVersion: position.latestRowVersion + 1, + seq: position.latestSeq + 1, + term: position.latestTerm, + txId: value, + } + await adapter.applyCommittedTx(eagerStorageTarget(COLLECTION_ID), tx) + return { + latestRowVersion: tx.rowVersion, + requiresFullReload: true, + seq: tx.seq, + term: tx.term, + txId: tx.txId, + type: 'tx:committed', + } + } + const reserve = (value?: string) => { + const params = requireSource() + params.begin() + if (value) params.write({ type: 'update', value: { id: 'row-1', value } }) + params.metadata?.persistence?.reserveCommitTurn() + } + const commit = () => requireSource().commit() + // Committing with an aborted signal abandons the transaction. + const abandon = () => { + const cancelled = new AbortController() + cancelled.abort() + return requireSource().commit(cancelled.signal) + } + await start() + return { + abandon, + adapter, + coordinator, + collection, + commit, + commitFromPeer, + database, + holdNextWrite, + nextScopeEntry, + reserve, + } +} + +type AdapterHistory = { + route: 'peer full reload' | 'peer reset' | 'subset admitted earlier' + timing: 'mutex turn' | 'scheduler hold' + ending: 'commits' | 'abandoned' +} + +const adapterHistories: Array = ( + ['peer full reload', 'peer reset', 'subset admitted earlier'] as const +).flatMap((route) => + (['mutex turn', 'scheduler hold'] as const).flatMap((timing) => + // A peer commit cannot become durable while this Collection's held write + // occupies the only SQLite writer, so that history needs a separate + // leader; the persisted-reads grammar's fixture leader covers it. + route === 'peer full reload' && timing === 'mutex turn' + ? [] + : (['commits', 'abandoned'] as const).map((ending) => ({ + route, + timing, + ending, + })), + ), +) + +/** A premise that must be reached; a miss fails by name instead of hanging. */ +const reached = (promise: Promise, label: string): Promise => + Promise.race([ + promise, + new Promise((_resolve, reject) => + setTimeout( + () => reject(new Error(`premise not reached: ${label}`)), + 2_000, + ), + ), + ]) + +describe('reserved commits across real-adapter hydrations', () => { + const cleanups: Array<() => Promise | void> = [] + afterEach(async () => { + for (const cleanup of cleanups.splice(0).reverse()) await cleanup() + }) + + it.each(adapterHistories)( + 'keeps a reserved commit lawful across a real-adapter hydration ($route, $timing, $ending)', + async ({ route, timing, ending }) => { + const harness = await openPeerTabHarness({ + sharedScheduling: timing === 'scheduler hold', + onDemand: route === 'subset admitted earlier', + }) + cleanups.push(() => harness.database.close()) + cleanups.push(() => harness.collection.cleanup()) + const reach = createOracleReach('sqlite.reserved-commit-hydration', [ + 'hydration entered its scope while reserved', + ]) + reach.beginHistory() + // row-1's durable value just before the reservation, from the history. + let durableBefore = 'initial' + const notice = + route === 'peer full reload' + ? await harness.commitFromPeer('peer edit') + : undefined + if (notice) durableBefore = 'peer edit' + let load: Promise | undefined + const admitRoute = () => { + if (notice) harness.coordinator.deliverFromPeer(COLLECTION_ID, notice) + else if (route === 'peer reset') { + harness.coordinator.deliverFromPeer(COLLECTION_ID, { + type: 'collection:reset', + schemaVersion: 1, + resetEpoch: 1, + }) + } else { + load = Promise.resolve( + harness.collection._sync.loadSubset({ limit: 1 }), + ).catch((error: unknown) => error) + } + } + if (timing === 'mutex turn') { + // This Collection's held write is ahead of the route on the mutex. + const write = harness.holdNextWrite() + harness.reserve('durable predecessor') + durableBefore = 'durable predecessor' + const blocker = Promise.resolve(harness.commit()).catch(() => undefined) + await write.wasReached + admitRoute() + // One task lets the route take its mutex turn before the reservation. + await new Promise((resolve) => setTimeout(resolve, 0)) + harness.reserve('refetched') + const entered = harness.nextScopeEntry() + write.release() + await blocker + await reached(entered, 'hydration entered its scope') + } else { + // Another Collection's held write holds the shared scheduler, so the + // route's scope waits after its mutex turn began. + await harness.adapter.loadSubset( + eagerStorageTarget('other_collection'), + {}, + ) + const otherWrite = harness.holdNextWrite() + const blocker = harness.adapter.applyCommittedTx( + eagerStorageTarget('other_collection'), + { mutations: [], rowVersion: 1, seq: 1, term: 1, txId: 'blocker' }, + ) + await otherWrite.wasReached + let turnBegan!: () => void + const began = new Promise((resolve) => { + turnBegan = resolve + }) + const runInHydrationScope = harness.adapter.runInHydrationScope.bind( + harness.adapter, + ) + harness.adapter.runInHydrationScope = (task) => { + turnBegan() + return runInHydrationScope(task) + } + admitRoute() + await reached(began, 'route turn began') + harness.reserve('refetched') + const entered = harness.nextScopeEntry() + otherWrite.release() + await blocker + await reached(entered, 'hydration entered its scope') + } + reach.reach('hydration entered its scope while reserved') + const outcome = await Promise.resolve( + ending === 'commits' ? harness.commit() : harness.abandon(), + ).then( + () => 'fulfilled', + () => 'rejected', + ) + await load + reach.assertReached() + const expectedValue = ending === 'commits' ? 'refetched' : durableBefore + await waitFor(() => + expect(harness.collection.get('row-1')).toMatchObject({ + value: expectedValue, + }), + ) + expect({ outcome, status: harness.collection.status }).toEqual({ + outcome: ending === 'commits' ? 'fulfilled' : 'rejected', + status: 'ready', + }) + }, + ) +}) diff --git a/packages/db/src/types.ts b/packages/db/src/types.ts index 298543dc42..38c45c2f63 100644 --- a/packages/db/src/types.ts +++ b/packages/db/src/types.ts @@ -583,8 +583,13 @@ export type SyncPersistenceCapabilityV1< value: unknown }) => Promise /** - * Reserve the open sync transaction's FIFO turn, so a subset hydration - * that starts before it commits waits for that commit. + * Reserve the open sync transaction's FIFO turn. Every persisted hydration + * that would start before it commits waits for that commit: a subset load, + * a reload or reset from a peer notice, and sequence-gap recovery. A subset + * load admitted while the turn is reserved also runs after the commit's + * durable write. The source must end the transaction: commit it, or abandon + * it by committing with an aborted signal. A source that stops for good + * abandons it, or every later hydration keeps waiting. */ readonly reserveCommitTurn: () => void readonly scanPersistedRows: ( diff --git a/packages/electric-db-collection/src/electric.ts b/packages/electric-db-collection/src/electric.ts index bb5f8b6048..093bb945c4 100644 --- a/packages/electric-db-collection/src/electric.ts +++ b/packages/electric-db-collection/src/electric.ts @@ -2001,6 +2001,9 @@ function createElectricSync>( const fail = (cause: unknown) => { rejectFullSnapshot(cause) if (collection.status === `loading`) failInitialSync(cause) + // No retry follows, so the open transaction can never commit. A + // source that stops must end its reserved transaction. + abandonOpenTransaction() } if (shapeOptions.onError) { @@ -2030,6 +2033,16 @@ function createElectricSync>( }, }) let transactionStarted = false + // Commit with an aborted signal: the wrapper discards the transaction and + // its reserved commit turn ends. + const abandonOpenTransaction = () => { + if (!transactionStarted) return + transactionStarted = false + const cancellation = new AbortController() + cancellation.abort() + const abandoned = commit(cancellation.signal) + if (abandoned !== true) void abandoned.catch(() => undefined) + } const newTxids = new Set() const newSnapshots: Array = [] // Track if we've completed initial sync in progressive mode. A persisted @@ -2306,12 +2319,7 @@ function createElectricSync>( const invalidateResume = () => { resumeInvalid = true - if (transactionStarted) { - const cancellation = new AbortController() - cancellation.abort() - commit(cancellation.signal) - transactionStarted = false - } + abandonOpenTransaction() syncedKeys.clear() newTxids.clear() newSnapshots.length = 0 @@ -2758,13 +2766,7 @@ function createElectricSync>( unsubscribeStream() // Abort the abort controller to stop the stream abortController.abort() - if (transactionStarted) { - transactionStarted = false - const cancellation = new AbortController() - cancellation.abort() - const abandoned = commit(cancellation.signal) - if (abandoned !== true) void abandoned.catch(() => undefined) - } + abandonOpenTransaction() pendingResumeBatches.length = 0 // Reset deduplication tracking so collection can load fresh data if restarted loadSubsetDedupe?.reset() From 3fd2e5e176cf70733afd2b16ef2cbbb4967063bd Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 26 Oct 2026 18:39:28 -0600 Subject: [PATCH 02/24] fix(electric): end a reserved transaction on every terminal stream stop Electric abandoned its open transaction only when onError gave no retry and on cleanup. When the SDK gave up its own retries, or the application's signal aborted the stream, the reservation stayed open, so hydrations that wait for it never ran. Both paths now abandon after the initial-failure decision. The multi-tab Electric oracle crosses every terminal stop path. Its eager storm cells for the two new paths fail without the repair. The Electric retry oracle's commit wrapper now passes the source's signal through, as a real persistence wrapper does. The real-adapter oracle now judges that a waited subset load succeeds; a mutant that rejects it passed before. The persisted-reads model is now a pure function of the history, separate from the driver. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 17 +- .../tests/multi-tab-electric-oracle.test.ts | 60 ++- .../tests/persisted-oracle.test.ts | 386 +++++++++++------- ...erved-commit-sqlite-adapter-oracle.test.ts | 19 +- .../electric-db-collection/src/electric.ts | 28 +- .../tests/electric-oracle.property.test.ts | 8 +- 6 files changed, 331 insertions(+), 187 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 735e826d0f..9a7d54ce6b 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2908,18 +2908,23 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. admitted earlier) with two timings (a held write of this Collection, or the shared scheduler holding the scope after its turn) and two endings (commits, abandoned). It runs on `SQLiteCorePersistenceAdapter`. - Its 10 cells pass. Every committing cell fails on `5649a07d4`. A peer + It also judges that a subset load that waited settles successfully. + Its 10 cells pass. Every committing cell fails on `5649a07d4`. A + mutant that rejects the waited load after the reservation fails the 4 + subset cells; before the load's outcome was judged, it passed. A peer full reload behind this Collection's own held write cannot become durable through the single SQLite writer, so it is excluded here and left to the grammar's fixture leader. Browser host: the multi-tab Electric oracle in `browser-db-sqlite-persistence` crosses two or three tabs, eager or on-demand Collections, which tab finishes first, a storm or staggered - arrival, and whether the last tab's stream fails while its refetch is - open. Its 32 cells pass. Every ordinary storm cell fails on - `5649a07d4`. Without Electric's abandon-on-failure, the eager storm - cells where a stream fails also fail. On-demand cells judge receipts - and status only. + arrival, and how the last tab's stream stops for good while its refetch + is open: no stop, an error that `onError` does not retry, the SDK + giving up its retries, or the application's signal aborting the + stream. Its 64 cells pass. Every ordinary storm cell fails on + `5649a07d4`. Without Electric's abandon on each stop path, the eager + storm cells for that path also fail (4 cells per path). On-demand + cells judge receipts and status only. A claimed leftover-replay/reset failure still needs a concrete schedule; the bounded two-replay-then-reset probe drains replay before reset. Live Electric/browser OPFS delivery needs the matching schedule in diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index f18307186e..6f62afb6ec 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -85,6 +85,9 @@ type ControlledStream = { send: (messages: Array>) => void // Electric's stream-error handler, which the test calls to fail the stream. onError: (error: unknown) => unknown + // The subscriber's error callback, which the SDK calls when it stops + // retrying. + giveUp: (error: unknown) => void } const streams: Array = [] @@ -110,11 +113,18 @@ vi.mock( throw new Error(`stream for tab ${stream.tab} is not subscribed`) }, onError: (error) => options.onError?.(error), + giveUp: () => { + throw new Error(`stream for tab ${stream.tab} is not subscribed`) + }, } streams.push(stream) return { - subscribe: (send: ControlledStream[`send`]) => { + subscribe: ( + send: ControlledStream[`send`], + giveUp: ControlledStream[`giveUp`], + ) => { stream.send = send + stream.giveUp = giveUp return () => undefined }, requestSnapshot: vi.fn().mockResolvedValue(undefined), @@ -156,6 +166,7 @@ const nextTask = () => new Promise((resolve) => setTimeout(resolve, 0)) // --------------------------------------------------------------------------- type SyncMode = `eager` | `on-demand` +type StreamStop = `error without retry` | `SDK gives up` | `signal aborted` function openHost() { const directory = mkdtempSync(join(tmpdir(), `tsdb-multi-tab-`)) @@ -196,6 +207,8 @@ function openHost() { } } const coordinator = new BrowserCollectionCoordinator({ dbName }) + // The application's signal, which can stop the stream from outside. + const external = new AbortController() const collection = createCollection( persistedCollectionOptions< Item, @@ -209,6 +222,7 @@ function openHost() { url: `http://electric.test/v1/shape`, params: { table: `items` }, headers: { 'x-tab': tab }, + signal: external.signal, }, syncMode, getKey: (row) => row.id, @@ -250,8 +264,15 @@ function openHost() { current.lastOffset = position.offset current.send(messages) }, - // A stream error with no retry stops the provider session for good. - failStream: () => stream().onError(new Error(`stream failed`)), + // Each way a stream stops for good: an error that `onError` does not + // retry, the SDK giving up its own retries, or the application's + // signal aborting the stream. + stopStream: (how: StreamStop) => { + const error = new Error(`stream failed`) + if (how === `error without retry`) return stream().onError(error) + if (how === `SDK gives up`) return stream().giveUp(error) + external.abort() + }, publicRows: () => Array.from(collection.values(), ({ id, label }) => ({ id, @@ -326,16 +347,24 @@ describe(`multi-tab Electric over one persisted database`, () => { // tab finishes its refetch first, whether every tab's must-refetch arrives // before any refetch finishes (a storm) or each tab finishes before the next // receives must-refetch (staggered, the neighbouring control), and whether - // the last tab's stream fails for good while its refetch is open. A source - // that stops for good abandons its reserved transaction (maintainer - // decision on #2226), so that tab still applies its peers' refetch from - // durable rows, as it did before reservations deferred hydrations. + // whether the last tab's stream stops for good while its refetch is open, + // through each terminal path Electric has. A source that stops for good + // abandons its reserved transaction (maintainer decision on #2226), so that + // tab still applies its peers' refetch from durable rows, as it did before + // reservations deferred hydrations. it.each( [2, 3].flatMap((tabCount) => ([`eager`, `on-demand`] as const).flatMap((syncMode) => ([`first`, `last`] as const).flatMap((finisher) => ([`storm`, `staggered`] as const).flatMap((arrival) => - ([`ok`, `fails while refetching`] as const).map((stream) => ({ + ( + [ + `ok`, + `error without retry`, + `SDK gives up`, + `signal aborted`, + ] as const + ).map((stream) => ({ tabCount, syncMode, finisher, @@ -347,7 +376,7 @@ describe(`multi-tab Electric over one persisted database`, () => { ), ), )( - `commits every reserved refetch across tabs ($tabCount tabs, $syncMode, $finisher finishes first, $arrival, stream $stream)`, + `commits every reserved refetch across tabs ($tabCount tabs, $syncMode, $finisher finishes first, $arrival, stream stop: $stream)`, async ({ tabCount, syncMode, finisher, arrival, stream }) => { host = openHost() const tabs = await openWarmTabs(host, tabCount, syncMode) @@ -360,7 +389,7 @@ describe(`multi-tab Electric over one persisted database`, () => { } type RefetchReach = | `peer commit announced during an open refetch` - | `a stream failed while its refetch was open` + | `a stream stopped while its refetch was open` const reach = createOracleReach( `browser.multi-tab-reserved-refetch`, [ @@ -369,19 +398,20 @@ describe(`multi-tab Electric over one persisted database`, () => { : []), ...(stream === `ok` ? [] - : ([`a stream failed while its refetch was open`] as const)), + : ([`a stream stopped while its refetch was open`] as const)), ], ) reach.beginHistory() const finishOrder = finisher === `first` ? tabs : [...tabs].reverse() - // The last tab to finish is the one whose stream may fail. + // The last tab to finish is the one whose stream may stop. const failingTab = stream === `ok` ? undefined : finishOrder[finishOrder.length - 1] const failStream = async (tab: Tab) => { - void Promise.resolve(tab.failStream()).catch(() => undefined) - await host!.quiesce(`tab ${tab.tab} stream failed`) - reach.reach(`a stream failed while its refetch was open`) + if (stream === `ok`) return + void Promise.resolve(tab.stopStream(stream)).catch(() => undefined) + await host!.quiesce(`tab ${tab.tab} stream stopped`) + reach.reach(`a stream stopped while its refetch was open`) } const refetch = (tab: Tab) => tab.deliver([...afterDeploy.map(insert), upToDate], { diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index c774772695..a89cbd7f1b 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -17674,15 +17674,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { | `peer change announced while reserved` | `reset and pre-reset commit announced while waiting` - const runPersistedReadHistory = async ({ - syncMode, - route, - operation, - truncateReplay, - commitTiming, - ending, - followUp, - }: PersistedReadHistory) => { + // The rows of the persisted-reads histories. Each history gets fresh + // objects, so no history can observe another's. + const createPersistedReadRows = () => { const storedRow = { id: `shared`, title: `persisted`, detail: `old` } const storedOnly = { id: `stored-only`, title: `removed by truncate` } const gateRow = { id: `gate`, title: `earlier sync transaction` } @@ -17691,15 +17685,44 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { const peerRow = { id: `peer`, title: `peer change` } const remoteRow = { id: `remote`, title: `owner` } const supersededRow = { id: `superseded`, title: `before the reset` } - const adapter = createRecordingAdapter([ - { ...storedRow }, - { ...storedOnly }, - ]) - adapter.rowMetadata.set(`shared`, { owner: `persisted` }) - adapter.collectionMetadata.set(`cursor`, `persisted`) - const coordinator = createFailStopCoordinatorHarness( - `persisted-read-precedence`, - ) + return { + storedRow, + storedOnly, + gateRow, + networkRow, + replacementRow, + peerRow, + remoteRow, + supersededRow, + } + } + type PersistedReadRows = ReturnType + + // Independent model. It reads only the history and its input rows, + // never an observation from production, so it can be reviewed and + // calibrated apart from the driver. + const expectedPersistedReadOutcome = ( + history: PersistedReadHistory, + fixtureRows: PersistedReadRows, + ) => { + const { + syncMode, + route, + operation, + truncateReplay, + commitTiming, + ending, + followUp, + } = history + const { + storedRow, + storedOnly, + gateRow, + networkRow, + replacementRow, + peerRow, + remoteRow, + } = fixtureRows const waits = commitTiming === `open across mutex turn` || commitTiming === `open across scope entry` @@ -17709,6 +17732,203 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { commitTiming === `open in the first publication` const reserves = waits || straddles || commitTiming === `open across startup` + const sourceSteps = + truncateReplay === `none` + ? [`network`] + : truncateReplay === `before` + ? [`replacement`, `network`] + : [`network`, `replacement`] + const demandRoute = + route === `subset` || + route === `complete subset` || + route === `routed subset` + const noDemandRead = + syncMode === `on-demand` && !demandRoute && route !== `source baseline` + // Rows the route adds durably. The owner's row for a routed subset lands + // after the follower's own read, so reads never see it. + const routeRows = + route === `targeted` + ? [peerRow] + : route === `routed subset` + ? [remoteRow] + : [] + const lateRowIds = new Set(route === `routed subset` ? [remoteRow.id] : []) + const readBaselineRows = [ + storedRow, + storedOnly, + gateRow, + ...routeRows, + ].filter((row) => !lateRowIds.has(row.id)) + // Steps that apply: an abandoned reservation applies none of its writes. + const appliedSteps = + ending === `abandoned` ? sourceSteps.slice(1) : sourceSteps + const abandonedStep = ending === `abandoned` ? sourceSteps[0] : undefined + + // Independent snapshot algebra: a later replacement discards everything + // before it. Otherwise the baseline survives only when there is no + // replacement, and the newer change decides the shared key. Complete row + // values and explicit metadata avoid the separate partial-update contract. + // Source steps are input history, never observations from production. + const baselineRows = [storedRow, storedOnly, gateRow, ...routeRows] + const expectedRows = + abandonedStep !== undefined + ? baselineRows + : truncateReplay === `after` + ? [replacementRow] + : [ + ...(truncateReplay === `before` + ? [replacementRow] + : [storedOnly, gateRow, ...routeRows]), + ...(operation === `delete` ? [] : [networkRow]), + ] + // On-demand without demand exposes the source's own rows; an aborted + // demand loads nothing. Every other route exposes the durable rows. + // Uncertified on-demand cache rows stay quarantined after a baseline + // hydration (electric.ts: "loaded only through fresh subset snapshots"). + const exposesSourceRowsOnly = + noDemandRead || route === `source baseline` || ending === `demand aborted` + const expectedPublicRows = + exposesSourceRowsOnly && truncateReplay === `none` + ? [ + gateRow, + ...(abandonedStep !== undefined || operation === `delete` + ? [] + : [networkRow]), + ] + : expectedRows + const sharedMetadata = + abandonedStep !== undefined + ? exposesSourceRowsOnly + ? { public: undefined, durable: { owner: `persisted` } } + : { public: { owner: `persisted` }, durable: { owner: `persisted` } } + : truncateReplay !== `after` && operation !== `delete` + ? { public: { owner: `network` }, durable: { owner: `network` } } + : { public: undefined, durable: undefined } + const replacementMetadata = + truncateReplay === `none` ? undefined : { owner: `replacement` } + const expectedCursor = + abandonedStep !== undefined + ? `persisted` + : truncateReplay === `after` + ? `replacement` + : `network` + const readsSomething = + !noDemandRead && + route !== `targeted` && + route !== `source baseline` && + ending !== `demand aborted` + // A routed subset reads its local rows, then rereads to admit the + // owner's row; a reset follow-up reloads, and so does the fenced + // pre-reset notice. + const expectedReadCount = + (!readsSomething ? 0 : route === `routed subset` ? 2 : 1) + + (followUp === `none` ? 0 : 2) + const applyDurableEvent = (rows: Map, event: string) => { + if (event === `replacement`) { + rows.clear() + rows.set(replacementRow.id, replacementRow) + } else if (event === `network`) { + if (operation === `delete`) rows.delete(`shared`) + else rows.set(networkRow.id, networkRow) + } else if (event === `owner commit`) rows.set(remoteRow.id, remoteRow) + } + const startupStepCommitted = + commitTiming === `open across startup` && abandonedStep === undefined + const noWaitPullSinceCalls = route === `seq-gap pull` ? 1 : 0 + // A route notice waits in the waiting timings; a follower's owner notice + // arrives during a straddled reservation and waits there. Each notice + // that waited may ask the leader once more; nothing else may. + const waitedNotices = + (waits && (route === `full-reload` || route === `seq-gap pull`)) || + (straddles && route === `routed subset`) + ? 1 + : 0 + const expectedSourceLoads = + route === `subset` || + route === `complete subset` || + route === `routed subset` + ? ending === `demand aborted` + ? 0 + : 1 + : route === `seq-gap` && syncMode === `eager` + ? 1 + : 0 + const metadataKeys = [`shared`, `stored-only`, `gate`, `replacement`] + const expectedMetadata = (side: `public` | `durable`) => + metadataKeys.map((key) => + key === `shared` + ? sharedMetadata[side] + : key === `replacement` + ? replacementMetadata + : undefined, + ) + return { + waits, + straddles, + reserves, + sourceSteps, + demandRoute, + readBaselineRows, + appliedSteps, + expectedRows, + exposesSourceRowsOnly, + expectedPublicRows, + expectedCursor, + expectedReadCount, + applyDurableEvent, + startupStepCommitted, + noWaitPullSinceCalls, + waitedNotices, + expectedSourceLoads, + metadataKeys, + expectedMetadata, + } + } + + const runPersistedReadHistory = async (history: PersistedReadHistory) => { + const { syncMode, route, operation, commitTiming, ending, followUp } = + history + const fixtureRows = createPersistedReadRows() + const { + storedRow, + storedOnly, + gateRow, + networkRow, + replacementRow, + peerRow, + remoteRow, + supersededRow, + } = fixtureRows + const { + waits, + straddles, + reserves, + sourceSteps, + demandRoute, + readBaselineRows, + appliedSteps, + expectedRows, + exposesSourceRowsOnly, + expectedPublicRows, + expectedCursor, + expectedReadCount, + applyDurableEvent, + startupStepCommitted, + noWaitPullSinceCalls, + waitedNotices, + expectedSourceLoads, + metadataKeys, + expectedMetadata, + } = expectedPersistedReadOutcome(history, fixtureRows) + const adapter = createRecordingAdapter([ + { ...storedRow }, + { ...storedOnly }, + ]) + adapter.rowMetadata.set(`shared`, { owner: `persisted` }) + adapter.collectionMetadata.set(`cursor`, `persisted`) + const coordinator = createFailStopCoordinatorHarness( + `persisted-read-precedence`, + ) const persistenceEntered = createEventGate() const releasePersistence = createEventGate() const readApplied = createEventGate() @@ -17887,12 +18107,6 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { await recordRead(`scanRows`) return scanRows(...args) } - const sourceSteps = - truncateReplay === `none` - ? [`network`] - : truncateReplay === `before` - ? [`replacement`, `network`] - : [`network`, `replacement`] let sourceParams!: TodoSyncParams const writeSourceStep = (step: (typeof sourceSteps)[number]) => { if (step === `replacement`) { @@ -17972,80 +18186,6 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { route === `complete subset` ? { signal: abortDemand.signal } : { limit: 100, signal: abortDemand.signal } - const demandRoute = - route === `subset` || - route === `complete subset` || - route === `routed subset` - const noDemandRead = - syncMode === `on-demand` && !demandRoute && route !== `source baseline` - // Rows the route adds durably. The owner's row for a routed subset lands - // after the follower's own read, so reads never see it. - const routeRows = - route === `targeted` - ? [peerRow] - : route === `routed subset` - ? [remoteRow] - : [] - const lateRowIds = new Set(route === `routed subset` ? [remoteRow.id] : []) - const readBaselineRows = [ - storedRow, - storedOnly, - gateRow, - ...routeRows, - ].filter((row) => !lateRowIds.has(row.id)) - // Steps that apply: an abandoned reservation applies none of its writes. - const appliedSteps = - ending === `abandoned` ? sourceSteps.slice(1) : sourceSteps - const abandonedStep = ending === `abandoned` ? sourceSteps[0] : undefined - - // Independent snapshot algebra: a later replacement discards everything - // before it. Otherwise the baseline survives only when there is no - // replacement, and the newer change decides the shared key. Complete row - // values and explicit metadata avoid the separate partial-update contract. - // Source steps are input history, never observations from production. - const baselineRows = [storedRow, storedOnly, gateRow, ...routeRows] - const expectedRows = - abandonedStep !== undefined - ? baselineRows - : truncateReplay === `after` - ? [replacementRow] - : [ - ...(truncateReplay === `before` - ? [replacementRow] - : [storedOnly, gateRow, ...routeRows]), - ...(operation === `delete` ? [] : [networkRow]), - ] - // On-demand without demand exposes the source's own rows; an aborted - // demand loads nothing. Every other route exposes the durable rows. - // Uncertified on-demand cache rows stay quarantined after a baseline - // hydration (electric.ts: "loaded only through fresh subset snapshots"). - const exposesSourceRowsOnly = - noDemandRead || route === `source baseline` || ending === `demand aborted` - const expectedPublicRows = - exposesSourceRowsOnly && truncateReplay === `none` - ? [ - gateRow, - ...(abandonedStep !== undefined || operation === `delete` - ? [] - : [networkRow]), - ] - : expectedRows - const sharedMetadata = - abandonedStep !== undefined - ? exposesSourceRowsOnly - ? { public: undefined, durable: { owner: `persisted` } } - : { public: { owner: `persisted` }, durable: { owner: `persisted` } } - : truncateReplay !== `after` && operation !== `delete` - ? { public: { owner: `network` }, durable: { owner: `network` } } - : { public: undefined, durable: undefined } - const replacementMetadata = - truncateReplay === `none` ? undefined : { owner: `replacement` } - const expectedCursor = - abandonedStep !== undefined - ? `persisted` - : truncateReplay === `after` - ? `replacement` - : `network` // Pending operations and the outcome each one owes. const pendingOperations: Array> = [] const expectedOutcomes: Array<`fulfilled` | `rejected`> = [] @@ -18414,26 +18554,6 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // startup reservation commits, its step precedes the reads; while a read // waits for a reserved commit, reads and later steps may interleave. // Rows are compared by key: the adapter's row order is not part of the law. - const readsSomething = - !noDemandRead && - route !== `targeted` && - route !== `source baseline` && - ending !== `demand aborted` - // A routed subset reads its local rows, then rereads to admit the - // owner's row; a reset follow-up reloads, and so does the fenced - // pre-reset notice. - const expectedReadCount = - (!readsSomething ? 0 : route === `routed subset` ? 2 : 1) + - (followUp === `none` ? 0 : 2) - const applyDurableEvent = (rows: Map, event: string) => { - if (event === `replacement`) { - rows.clear() - rows.set(replacementRow.id, replacementRow) - } else if (event === `network`) { - if (operation === `delete`) rows.delete(`shared`) - else rows.set(networkRow.id, networkRow) - } else if (event === `owner commit`) rows.set(remoteRow.id, remoteRow) - } const durableBefore = (index: number) => { const rows = new Map(readBaselineRows.map((row) => [row.id, row])) for (const event of trace.slice(0, index)) @@ -18456,8 +18576,6 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { const stepIndexes = trace.flatMap((event, index) => event === `network` || event === `replacement` ? [index] : [], ) - const startupStepCommitted = - commitTiming === `open across startup` && abandonedStep === undefined const observedReadOrder = waits ? `may interleave` : straddles @@ -18478,34 +18596,6 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ) ? `before every step` : `other` - const noWaitPullSinceCalls = route === `seq-gap pull` ? 1 : 0 - // A route notice waits in the waiting timings; a follower's owner notice - // arrives during a straddled reservation and waits there. Each notice - // that waited may ask the leader once more; nothing else may. - const waitedNotices = - (waits && (route === `full-reload` || route === `seq-gap pull`)) || - (straddles && route === `routed subset`) - ? 1 - : 0 - const expectedSourceLoads = - route === `subset` || - route === `complete subset` || - route === `routed subset` - ? ending === `demand aborted` - ? 0 - : 1 - : route === `seq-gap` && syncMode === `eager` - ? 1 - : 0 - const metadataKeys = [`shared`, `stored-only`, `gate`, `replacement`] - const expectedMetadata = (side: `public` | `durable`) => - metadataKeys.map((key) => - key === `shared` - ? sharedMetadata[side] - : key === `replacement` - ? replacementMetadata - : undefined, - ) const observed = { outcomes: outcomes.map(({ status }) => status), readsDuringReservation: straddles ? [] : readsDuringReservation, diff --git a/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts index 81dfedd23d..a85c2a3c13 100644 --- a/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts @@ -343,7 +343,8 @@ describe('reserved commits across real-adapter hydrations', () => { ? await harness.commitFromPeer('peer edit') : undefined if (notice) durableBefore = 'peer edit' - let load: Promise | undefined + // The subset load's settlement, judged after the reservation ends. + let load: Promise<'fulfilled' | 'rejected'> | undefined const admitRoute = () => { if (notice) harness.coordinator.deliverFromPeer(COLLECTION_ID, notice) else if (route === 'peer reset') { @@ -355,7 +356,10 @@ describe('reserved commits across real-adapter hydrations', () => { } else { load = Promise.resolve( harness.collection._sync.loadSubset({ limit: 1 }), - ).catch((error: unknown) => error) + ).then( + () => 'fulfilled' as const, + () => 'rejected' as const, + ) } } if (timing === 'mutex turn') { @@ -412,7 +416,7 @@ describe('reserved commits across real-adapter hydrations', () => { () => 'fulfilled', () => 'rejected', ) - await load + const loadOutcome = await load reach.assertReached() const expectedValue = ending === 'commits' ? 'refetched' : durableBefore await waitFor(() => @@ -420,9 +424,16 @@ describe('reserved commits across real-adapter hydrations', () => { value: expectedValue, }), ) - expect({ outcome, status: harness.collection.status }).toEqual({ + // A subset load that waited runs after the reservation and succeeds. + expect({ + outcome, + status: harness.collection.status, + loadOutcome, + }).toEqual({ outcome: ending === 'commits' ? 'fulfilled' : 'rejected', status: 'ready', + loadOutcome: + route === 'subset admitted earlier' ? 'fulfilled' : undefined, }) }, ) diff --git a/packages/electric-db-collection/src/electric.ts b/packages/electric-db-collection/src/electric.ts index 093bb945c4..1a33a22692 100644 --- a/packages/electric-db-collection/src/electric.ts +++ b/packages/electric-db-collection/src/electric.ts @@ -1925,6 +1925,17 @@ function createElectricSync>( // Abort controller for the stream - wraps the signal if provided const abortController = new AbortController() let cleanupAbortedStream = false + let transactionStarted = false + // Commit with an aborted signal: the wrapper discards the transaction and + // its reserved commit turn ends. + const abandonOpenTransaction = () => { + if (!transactionStarted) return + transactionStarted = false + const cancellation = new AbortController() + cancellation.abort() + const abandoned = commit(cancellation.signal) + if (abandoned !== true) void abandoned.catch(() => undefined) + } // A final initial failure decides its waiting loads. A cleanup that a // status listener runs inside that failure must not relabel it. let initialSyncFailedFinally = false @@ -1956,6 +1967,9 @@ function createElectricSync>( const fails = isActiveLifecycle() && collection.status === `loading` abortController.abort() if (fails) failInitialSync(new StreamAbortedError(collectionId)) + // A source that stops must end its reserved transaction. The initial + // failure decides first: abandoning commits. + abandonOpenTransaction() } if (shapeOptions.signal) { @@ -2032,17 +2046,6 @@ function createElectricSync>( return }, }) - let transactionStarted = false - // Commit with an aborted signal: the wrapper discards the transaction and - // its reserved commit turn ends. - const abandonOpenTransaction = () => { - if (!transactionStarted) return - transactionStarted = false - const cancellation = new AbortController() - cancellation.abort() - const abandoned = commit(cancellation.signal) - if (abandoned !== true) void abandoned.catch(() => undefined) - } const newTxids = new Set() const newSnapshots: Array = [] // Track if we've completed initial sync in progressive mode. A persisted @@ -2725,6 +2728,9 @@ function createElectricSync>( collection.status === `loading` ) failInitialSync(error) + // A source that stops must end its reserved transaction. The initial + // failure decides first: abandoning commits. + abandonOpenTransaction() }, ) diff --git a/packages/electric-db-collection/tests/electric-oracle.property.test.ts b/packages/electric-db-collection/tests/electric-oracle.property.test.ts index fa62022c93..40a50e3dda 100644 --- a/packages/electric-db-collection/tests/electric-oracle.property.test.ts +++ b/packages/electric-db-collection/tests/electric-oracle.property.test.ts @@ -5449,13 +5449,15 @@ describeUnlessQueuedPresenceReplay(`Electric adapter laws`, () => { const heldAcceptance = createDeferred() const holdAcceptance = (params: TParams): TParams => { const syncParams = params as unknown as { - commit: () => true | Promise + commit: (signal?: AbortSignal) => true | Promise } const commit = syncParams.commit return { ...params, - commit: () => { - const receipt = commit() + // A wrapper passes the source's signal through, so an abandoned + // transaction stays abandoned. + commit: (signal?: AbortSignal) => { + const receipt = commit(signal) if (!armAcceptance) return receipt return withAcceptedReceipt( receipt === true ? Promise.resolve() : receipt, From 7b14a9bc369ac1f95a1feb1700e382dc11ca7c9c Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 26 Oct 2026 18:53:56 -0600 Subject: [PATCH 03/24] test(browser-sqlite): return on every stream-stop path in the multi-tab oracle Co-authored-by: Isaac --- .../tests/multi-tab-electric-oracle.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 6f62afb6ec..6cbf3eddc8 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -271,7 +271,7 @@ function openHost() { const error = new Error(`stream failed`) if (how === `error without retry`) return stream().onError(error) if (how === `SDK gives up`) return stream().giveUp(error) - external.abort() + return external.abort() }, publicRows: () => Array.from(collection.values(), ({ id, label }) => ({ From 999457844c29a72569c818b980f2325462d7ce79 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 26 Oct 2026 21:46:40 -0600 Subject: [PATCH 04/24] test(browser-sqlite): cross every Electric stop path and recovery in the multi-tab oracle The storm grammar now crosses the stopped tab's open reserved transaction (its refetch, or an ordinary batch after a resume), every terminal stop (onError without retry, SDK give-up, signal abort, cleanup, resume invalidation), and the application's recovery (restart in the page, or a reload over the same database). It judges every tab at two checkpoints and the full-snapshot request after resume invalidation. The mocked stream gains unsubscribeAll, which main's Electric now calls, and rejects delivery to a stopped stream. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 20 +- .../tests/multi-tab-electric-oracle.test.ts | 326 ++++++++++++++---- 2 files changed, 273 insertions(+), 73 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 84393511bd..3bf0e9fa5f 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2933,12 +2933,20 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. Browser host: the multi-tab Electric oracle in `browser-db-sqlite-persistence` crosses two or three tabs, eager or on-demand Collections, which tab finishes first, a storm or staggered - arrival, and how the last tab's stream stops for good while its refetch - is open: no stop, an error that `onError` does not retry, the SDK - giving up its retries, or the application's signal aborting the - stream. Its 64 cells pass. Every ordinary storm cell fails on - `5649a07d4`. Without Electric's abandon on each stop path, the eager - storm cells for that path also fail (4 cells per path). On-demand + arrival, the last tab's open reserved transaction (its refetch, or an + ordinary batch after it resumed from the persisted position), how its + source stops for good (an error that `onError` does not retry, the SDK + giving up, a signal abort, cleanup, or resume invalidation), and how + the application recovers it (restart in the page, or reload). It + judges every tab's status and rows once every source has delivered or + stopped, again after recovery, and the full-snapshot request after + resume invalidation. Its 290 storm cells pass. Every ordinary storm + cell fails on `5649a07d4`. Removing Electric's abandon on the `onError`, + give-up, or signal-abort path fails 12 cells each. Two abandon mutants + survive here: cleanup's, because the persisted wrapper ends its + transactions on cleanup (the Electric descriptor-isolation oracle kills + it without persistence), and resume invalidation's, which changes only + the rows of a Collection in `error`, which no law promises. On-demand cells judge receipts and status only. A claimed leftover-replay/reset failure still needs a concrete schedule; the bounded two-replay-then-reset probe drains replay before reset. diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 6cbf3eddc8..2063021f32 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -47,7 +47,15 @@ * the Electric SDK; the test plays the server and delivers each tab's * messages. Checkpoints wait for host quiescence: no SQL statement and no * coordinator message for several tasks. Observations are each tab's - * Collection status and public rows. + * Collection status and public rows, and the position from which a + * recovered tab's new stream starts. + * + * A Collection in `error` after resume invalidation has no promised rows, so + * its rows are judged only after recovery. Removing Electric's abandon in + * resume invalidation changes only those unjudged rows, so that mutant + * survives here. Electric's own cleanup abandon is redundant behind the + * persisted wrapper, which ends its transactions on cleanup; the Electric + * descriptor-isolation oracle judges it without persistence. * * Limits: the stream is controlled, not the installed SDK over HTTP; one * process hosts every tab; one Collection; no subset demand in on-demand @@ -80,8 +88,12 @@ type Item = Row & { id: number; label: string } type ControlledStream = { tab: string + // The position Electric asked the server to start from. + startedAt: { handle: string | undefined; offset: string } shapeHandle: string | undefined lastOffset: string + // Set when Electric releases the stream; a stopped stream gets no messages. + stopped: boolean send: (messages: Array>) => void // Electric's stream-error handler, which the test calls to fail the stream. onError: (error: unknown) => unknown @@ -107,6 +119,11 @@ vi.mock( }) => { const stream: ControlledStream = { tab: options.headers?.[`x-tab`] ?? `unknown`, + startedAt: { + handle: options.handle, + offset: options.offset ?? `-1`, + }, + stopped: false, shapeHandle: options.handle, lastOffset: options.offset ?? `-1`, send: () => { @@ -127,6 +144,9 @@ vi.mock( stream.giveUp = giveUp return () => undefined }, + unsubscribeAll: () => { + stream.stopped = true + }, requestSnapshot: vi.fn().mockResolvedValue(undefined), fetchSnapshot: vi .fn() @@ -166,7 +186,12 @@ const nextTask = () => new Promise((resolve) => setTimeout(resolve, 0)) // --------------------------------------------------------------------------- type SyncMode = `eager` | `on-demand` -type StreamStop = `error without retry` | `SDK gives up` | `signal aborted` +type StreamStop = + | `error without retry` + | `SDK gives up` + | `signal aborted` + | `cleanup` + | `resume invalidated` function openHost() { const directory = mkdtempSync(join(tmpdir(), `tsdb-multi-tab-`)) @@ -208,7 +233,13 @@ function openHost() { } const coordinator = new BrowserCollectionCoordinator({ dbName }) // The application's signal, which can stop the stream from outside. - const external = new AbortController() + let external = new AbortController() + const shapeOptions = { + url: `http://electric.test/v1/shape`, + params: { table: `items` }, + headers: { 'x-tab': tab }, + signal: external.signal, + } const collection = createCollection( persistedCollectionOptions< Item, @@ -218,12 +249,7 @@ function openHost() { >({ ...electricCollectionOptions({ id: `items`, - shapeOptions: { - url: `http://electric.test/v1/shape`, - params: { table: `items` }, - headers: { 'x-tab': tab }, - signal: external.signal, - }, + shapeOptions, syncMode, getKey: (row) => row.id, startSync: true, @@ -255,23 +281,43 @@ function openHost() { collection, close, hasStream: () => streams.some((candidate) => candidate.tab === tab), + streamCount: () => + streams.filter((candidate) => candidate.tab === tab).length, + startedAt: () => stream().startedAt, deliver: ( messages: Array>, position: { handle: string; offset: string }, ) => { const current = stream() + if (current.stopped) { + throw new Error(`tab ${tab} delivered to a stopped stream`) + } current.shapeHandle = position.handle current.lastOffset = position.offset current.send(messages) }, - // Each way a stream stops for good: an error that `onError` does not - // retry, the SDK giving up its own retries, or the application's - // signal aborting the stream. + // Each way a source stops for good: an error that `onError` does not + // retry, the SDK giving up its own retries, the application's signal + // aborting the stream, the application cleaning up the Collection, or + // a resumed stream sending an update for a row it never sent, which + // invalidates the persisted resume position. stopStream: (how: StreamStop) => { const error = new Error(`stream failed`) if (how === `error without retry`) return stream().onError(error) if (how === `SDK gives up`) return stream().giveUp(error) - return external.abort() + if (how === `signal aborted`) return external.abort() + if (how === `cleanup`) return collection.cleanup() + const current = stream() + return current.send([update({ id: 99, label: `never sent` })]) + }, + // The application's recovery: clean up and start the Collection again. + // A signal that already aborted would stop the new stream too, so the + // restarted session gets a fresh signal. + restart: async () => { + await collection.cleanup() + external = new AbortController() + shapeOptions.signal = external.signal + void collection.preload().catch(() => undefined) }, publicRows: () => Array.from(collection.values(), ({ id, label }) => ({ @@ -321,6 +367,36 @@ async function openWarmTabs(host: Host, count: number, syncMode: SyncMode) { return tabs } +async function waitForStream(tab: Tab, count: number) { + for (let task = 0; task < 400 && tab.streamCount() < count; task++) { + await new Promise((resolve) => setTimeout(resolve, 2)) + } + expect(tab.streamCount()).toBe(count) +} + +// The server after the deploy, answering a new stream from where it starts: +// the head handle is up to date, a full-snapshot request gets the head +// snapshot, and the old handle must refetch. +async function serveHead( + host: Host, + tab: Tab, + startedAt: { handle: string | undefined; offset: string }, +) { + const head = { handle: `h2`, offset: `20_0` } + if (startedAt.handle === `h2`) tab.deliver([upToDate], head) + else if (startedAt.handle === undefined || startedAt.offset === `-1`) { + tab.deliver([...afterDeploy.map(insert), upToDate], head) + } else { + tab.deliver([mustRefetch], { + handle: startedAt.handle, + offset: startedAt.offset, + }) + await host.quiesce(`tab ${tab.tab} restart refetch open`) + tab.deliver([...afterDeploy.map(insert), upToDate], head) + } + await host.quiesce(`tab ${tab.tab} restarted`) +} + // --------------------------------------------------------------------------- // The server log. Snapshot 1 is the state before a deploy, snapshot 2 after: // row 1 changes, row 2 is deleted, row 3 is new. @@ -343,41 +419,72 @@ describe(`multi-tab Electric over one persisted database`, () => { streams.length = 0 }) - // Law 1 grammar: two or three tabs, eager or on-demand Collections, which - // tab finishes its refetch first, whether every tab's must-refetch arrives - // before any refetch finishes (a storm) or each tab finishes before the next - // receives must-refetch (staggered, the neighbouring control), and whether - // whether the last tab's stream stops for good while its refetch is open, - // through each terminal path Electric has. A source that stops for good - // abandons its reserved transaction (maintainer decision on #2226), so that - // tab still applies its peers' refetch from durable rows, as it did before - // reservations deferred hydrations. - it.each( - [2, 3].flatMap((tabCount) => - ([`eager`, `on-demand`] as const).flatMap((syncMode) => - ([`first`, `last`] as const).flatMap((finisher) => - ([`storm`, `staggered`] as const).flatMap((arrival) => + // Law 1 grammar. Dimensions: two or three tabs; eager or on-demand + // Collections; which tab finishes its refetch first; whether every tab's + // must-refetch arrives before any refetch finishes (a storm) or each tab + // finishes before the next receives must-refetch (staggered, the + // neighbouring control); and whether the last tab's source stops for good + // while it holds an open reserved transaction, through each terminal path + // Electric has. That transaction is either its refetch or an ordinary batch + // of a stream that resumed from the persisted position, with no up-to-date + // yet. Resume invalidation needs the resumed batch: a refetch already + // replaces the snapshot, and on-demand Collections never validate resume. + // + // A source that stops for good abandons its reserved transaction + // (maintainer decision on #2226). So a stopped tab that is still open + // applies its peers' refetch from durable rows, as it did before + // reservations deferred hydrations. Electric then fails no further, except + // that resume invalidation puts the Collection in `error`, as Electric + // documents. The application's recovery, cleanup and a restart, then brings + // the tab to the head. After resume invalidation the restart must request a + // full snapshot (electric.ts: "a full snapshot is required"). + type StormHistory = { + tabCount: number + syncMode: SyncMode + finisher: `first` | `last` + arrival: `storm` | `staggered` + open: `refetch` | `resumed batch` + stop: `none` | StreamStop + recovery: `none` | `restart` | `reload` + } + const stormHistories: Array = [2, 3].flatMap((tabCount) => + ([`eager`, `on-demand`] as const).flatMap((syncMode) => + ([`first`, `last`] as const).flatMap((finisher) => + ([`storm`, `staggered`] as const).flatMap((arrival) => + ([`refetch`, `resumed batch`] as const).flatMap((open) => ( [ - `ok`, + ...(open === `refetch` ? ([`none`] as const) : []), `error without retry`, `SDK gives up`, `signal aborted`, + `cleanup`, + ...(open === `resumed batch` && syncMode === `eager` + ? ([`resume invalidated`] as const) + : []), ] as const - ).map((stream) => ({ - tabCount, - syncMode, - finisher, - arrival, - stream, - })), + ).flatMap((stop) => + (stop === `none` + ? ([`none`] as const) + : ([`restart`, `reload`] as const) + ).map((recovery) => ({ + tabCount, + syncMode, + finisher, + arrival, + open, + stop, + recovery, + })), + ), ), ), ), ), - )( - `commits every reserved refetch across tabs ($tabCount tabs, $syncMode, $finisher finishes first, $arrival, stream stop: $stream)`, - async ({ tabCount, syncMode, finisher, arrival, stream }) => { + ) + it.each(stormHistories)( + `commits every reserved refetch across tabs ($tabCount tabs, $syncMode, $finisher finishes first, $arrival, open $open, stop: $stop, recovery: $recovery)`, + async ({ tabCount, syncMode, finisher, arrival, open, stop, recovery }) => { host = openHost() const tabs = await openWarmTabs(host, tabCount, syncMode) for (const tab of tabs) { @@ -389,29 +496,72 @@ describe(`multi-tab Electric over one persisted database`, () => { } type RefetchReach = | `peer commit announced during an open refetch` - | `a stream stopped while its refetch was open` + | `stopped tab resumed from the persisted position` + | `peer commit announced during the stopped tab's open transaction` + | `a source stopped while its transaction was open` + | `restarted tab opened a new stream` const reach = createOracleReach( `browser.multi-tab-reserved-refetch`, [ ...(arrival === `storm` ? ([`peer commit announced during an open refetch`] as const) : []), - ...(stream === `ok` + ...(open === `resumed batch` + ? ([`stopped tab resumed from the persisted position`] as const) + : []), + ...(stop === `none` ? [] - : ([`a stream stopped while its refetch was open`] as const)), + : ([ + `peer commit announced during the stopped tab's open transaction`, + `a source stopped while its transaction was open`, + `restarted tab opened a new stream`, + ] as const)), ], ) reach.beginHistory() + // The last tab to finish is the one whose source may stop. const finishOrder = finisher === `first` ? tabs : [...tabs].reverse() - // The last tab to finish is the one whose stream may stop. - const failingTab = - stream === `ok` ? undefined : finishOrder[finishOrder.length - 1] - const failStream = async (tab: Tab) => { - if (stream === `ok`) return - void Promise.resolve(tab.stopStream(stream)).catch(() => undefined) - await host!.quiesce(`tab ${tab.tab} stream stopped`) - reach.reach(`a stream stopped while its refetch was open`) + const stoppedIndex = + stop === `none` ? undefined : tabs.indexOf(finishOrder.at(-1)!) + if (open === `resumed batch`) { + // A page reload: the tab closes and reopens over the same database, + // resuming from its persisted position. Its first batch after the + // resume stays open, as a batch with no up-to-date does. + const closing = tabs[stoppedIndex!]! + const streamsBefore = closing.streamCount() + await closing.close() + await host.quiesce(`stopped tab closed`) + const reopened = host.openTab(closing.tab, syncMode) + await waitForStream(reopened, streamsBefore + 1) + tabs[stoppedIndex!] = reopened + if (reopened.startedAt().handle === `h1`) { + reach.reach(`stopped tab resumed from the persisted position`) + } + reopened.deliver([upToDate], { handle: `h1`, offset: `10_0` }) + await host.quiesce(`stopped tab resumed`) + reopened.deliver([update({ id: 1, label: `one, unfinished` })], { + handle: `h1`, + offset: `11_0`, + }) + await host.quiesce(`resumed batch open`) + } + const stoppedTab = + stoppedIndex === undefined ? undefined : tabs[stoppedIndex] + const refetchingTabs = ( + finisher === `first` ? tabs : [...tabs].reverse() + ).filter((tab) => open === `refetch` || tab !== stoppedTab) + const noticesBefore = host.commitNotices() + const stopSource = async () => { + if (!stoppedTab || stop === `none`) return + if (host!.commitNotices() > noticesBefore) { + reach.reach( + `peer commit announced during the stopped tab's open transaction`, + ) + } + void Promise.resolve(stoppedTab.stopStream(stop)).catch(() => undefined) + await host!.quiesce(`tab ${stoppedTab.tab} source stopped`) + reach.reach(`a source stopped while its transaction was open`) } const refetch = (tab: Tab) => tab.deliver([...afterDeploy.map(insert), upToDate], { @@ -419,7 +569,7 @@ describe(`multi-tab Electric over one persisted database`, () => { offset: `20_0`, }) if (arrival === `storm`) { - for (const tab of tabs) + for (const tab of refetchingTabs) tab.deliver([mustRefetch], { handle: `h1`, offset: `10_0` }) const noticesBeforeRefetch = host.commitNotices() await host.quiesce(`every refetch open`) @@ -428,41 +578,83 @@ describe(`multi-tab Electric over one persisted database`, () => { if (host.commitNotices() > noticesBeforeRefetch) { reach.reach(`peer commit announced during an open refetch`) } - if (failingTab) await failStream(failingTab) - for (const tab of finishOrder) { - if (tab === failingTab) continue + for (const tab of refetchingTabs) { + if (tab === stoppedTab) continue refetch(tab) await host.quiesce(`tab ${tab.tab} refetched`) } + await stopSource() } else { - for (const tab of finishOrder) { + for (const tab of refetchingTabs) { tab.deliver([mustRefetch], { handle: `h1`, offset: `10_0` }) await host.quiesce(`tab ${tab.tab} refetch open`) - if (tab === failingTab) { - await failStream(tab) - continue - } + if (tab === stoppedTab) continue refetch(tab) await host.quiesce(`tab ${tab.tab} refetched`) } + await stopSource() } - reach.assertReached() - // A rejected refetch commit puts its Collection in `error`. Eager rows - // must equal the head snapshot; on-demand rows are not judged (above). - expect( + // Checkpoint 1, every source has delivered or stopped. A rejected + // reserved commit puts its Collection in `error`. A stopped tab that is + // still open shows its peers' refetch, except after cleanup (no rows) + // or resume invalidation (`error`, rows not promised). Eager rows must + // equal the head snapshot; on-demand rows are not judged (above). + const stoppedStatus = + stop === `cleanup` + ? `cleaned-up` + : stop === `resume invalidated` + ? `error` + : `ready` + const judgesRowsBeforeRestart = (tab: Tab) => + syncMode === `eager` && + (tab !== stoppedTab || + (stop !== `cleanup` && stop !== `resume invalidated`)) + const observe = (judgesRows: (tab: Tab) => boolean) => tabs.map((tab) => ({ tab: tab.tab, status: tab.collection.status, - ...(syncMode === `eager` ? { rows: tab.publicRows() } : {}), - })), - ).toEqual( + ...(judgesRows(tab) ? { rows: tab.publicRows() } : {}), + })) + expect(observe(judgesRowsBeforeRestart)).toEqual( tabs.map((tab) => ({ tab: tab.tab, - status: `ready`, - ...(syncMode === `eager` ? { rows: afterDeploy } : {}), + status: tab === stoppedTab ? stoppedStatus : `ready`, + ...(judgesRowsBeforeRestart(tab) ? { rows: afterDeploy } : {}), })), ) + + // Checkpoint 2, after the application recovers the stopped tab and the + // server answers its new stream from where it starts. A restart cleans + // up and starts the same Collection again; a reload closes the tab and + // opens it again over the same database, so it reads only durable + // state. + if (stoppedTab) { + const streamsBefore = stoppedTab.streamCount() + let recovered = stoppedTab + if (recovery === `restart`) await stoppedTab.restart() + else { + await stoppedTab.close() + await host.quiesce(`tab ${stoppedTab.tab} closed`) + recovered = host.openTab(stoppedTab.tab, syncMode) + tabs[stoppedIndex!] = recovered + } + await waitForStream(recovered, streamsBefore + 1) + reach.reach(`restarted tab opened a new stream`) + const restartedAt = recovered.startedAt() + await serveHead(host, recovered, restartedAt) + if (stop === `resume invalidated`) { + expect(restartedAt).toEqual({ handle: undefined, offset: `-1` }) + } + expect(observe(() => syncMode === `eager`)).toEqual( + tabs.map((tab) => ({ + tab: tab.tab, + status: `ready`, + ...(syncMode === `eager` ? { rows: afterDeploy } : {}), + })), + ) + } + reach.assertReached() }, 30_000, ) From 767ea3a4c43b0388b6f6fd695ae66053e52bde09 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 26 Oct 2026 21:57:02 -0600 Subject: [PATCH 05/24] docs(oracles): state which multi-tab storm cells compare rows Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 3bf0e9fa5f..cfa807a0fb 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2938,9 +2938,11 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. source stops for good (an error that `onError` does not retry, the SDK giving up, a signal abort, cleanup, or resume invalidation), and how the application recovers it (restart in the page, or reload). It - judges every tab's status and rows once every source has delivered or - stopped, again after recovery, and the full-snapshot request after - resume invalidation. Its 290 storm cells pass. Every ordinary storm + judges every tab's status once every source has delivered or stopped, + and again after recovery. Eager cells also compare rows at both + checkpoints, except the stopped tab's rows before recovery after + cleanup or resume invalidation. It also judges the full-snapshot + request after resume invalidation. Its 290 storm cells pass. Every ordinary storm cell fails on `5649a07d4`. Removing Electric's abandon on the `onError`, give-up, or signal-abort path fails 12 cells each. Two abandon mutants survive here: cleanup's, because the persisted wrapper ends its From 5a7c00c13d95ff91c6b7480acb44eaa89ae1b3ef Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 08:12:11 -0600 Subject: [PATCH 06/24] test(browser-sqlite): judge the rows of a Collection in error after resume invalidation Maintainer decision on this PR: a persisted Collection in `error` keeps showing durable rows, because it still applies its peers' durable commits. The multi-tab oracle now judges the rows of the tab that resume invalidation put in `error`. Removing Electric's abandon in invalidateResume now fails 16 storm cells. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 19 ++++++------ .../tests/multi-tab-electric-oracle.test.ts | 31 ++++++++++--------- 2 files changed, 27 insertions(+), 23 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index cfa807a0fb..84402fa276 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2940,16 +2940,17 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. the application recovers it (restart in the page, or reload). It judges every tab's status once every source has delivered or stopped, and again after recovery. Eager cells also compare rows at both - checkpoints, except the stopped tab's rows before recovery after - cleanup or resume invalidation. It also judges the full-snapshot - request after resume invalidation. Its 290 storm cells pass. Every ordinary storm + checkpoints, except a cleaned-up tab before recovery. It also judges the + full-snapshot request after resume invalidation. A Collection in + `error` keeps showing durable rows (maintainer decision on PR #2265), + so the tab that resume invalidation put in `error` must show its peers' + refetch. Its 290 storm cells pass. Every ordinary storm cell fails on `5649a07d4`. Removing Electric's abandon on the `onError`, - give-up, or signal-abort path fails 12 cells each. Two abandon mutants - survive here: cleanup's, because the persisted wrapper ends its - transactions on cleanup (the Electric descriptor-isolation oracle kills - it without persistence), and resume invalidation's, which changes only - the rows of a Collection in `error`, which no law promises. On-demand - cells judge receipts and status only. + give-up, or signal-abort path fails 12 cells each, and on the resume + invalidation path 16 cells. Cleanup's abandon mutant survives here, + because the persisted wrapper ends its transactions on cleanup; the + Electric descriptor-isolation oracle kills it without persistence. + On-demand cells judge receipts and status only. A claimed leftover-replay/reset failure still needs a concrete schedule; the bounded two-replay-then-reset probe drains replay before reset. Live Electric/browser OPFS delivery needs the matching schedule in diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 2063021f32..4b51ad5fdf 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -6,7 +6,7 @@ * runs its own Electric stream, and all of them store through one SQLite * database and one elected leader. Tabs coordinate only through durable * commits and the notices those commits broadcast. This file is the Browser - * host owner for two laws about that arrangement. + * host owner for three laws about that arrangement. * * Law 1, reserved commits survive a refetch storm. Authority: * SyncPersistenceCapabilityV1 `reserveCommitTurn` (packages/db/src/types.ts), @@ -50,12 +50,16 @@ * Collection status and public rows, and the position from which a * recovered tab's new stream starts. * - * A Collection in `error` after resume invalidation has no promised rows, so - * its rows are judged only after recovery. Removing Electric's abandon in - * resume invalidation changes only those unjudged rows, so that mutant - * survives here. Electric's own cleanup abandon is redundant behind the - * persisted wrapper, which ends its transactions on cleanup; the Electric - * descriptor-isolation oracle judges it without persistence. + * Law 3, a Collection in `error` keeps showing durable rows. Authority: + * maintainer decision on PR #2265. A persisted Collection that a source + * failure put in `error` still applies its peers' durable commits, so its + * public rows match the durable rows. This file judges it after resume + * invalidation, the one stop that puts the Collection in `error`; the source + * must end its reserved transaction for the peers' refetch to apply. + * + * Electric's own cleanup abandon is redundant behind the persisted wrapper, + * which ends its transactions on cleanup, so that mutant survives here; the + * Electric descriptor-isolation oracle judges it without persistence. * * Limits: the stream is controlled, not the installed SDK over HTTP; one * process hosts every tab; one Collection; no subset demand in on-demand @@ -435,7 +439,8 @@ describe(`multi-tab Electric over one persisted database`, () => { // applies its peers' refetch from durable rows, as it did before // reservations deferred hydrations. Electric then fails no further, except // that resume invalidation puts the Collection in `error`, as Electric - // documents. The application's recovery, cleanup and a restart, then brings + // documents; that Collection still shows durable rows (Law 3). The + // application's recovery, cleanup and a restart, then brings // the tab to the head. After resume invalidation the restart must request a // full snapshot (electric.ts: "a full snapshot is required"). type StormHistory = { @@ -597,9 +602,9 @@ describe(`multi-tab Electric over one persisted database`, () => { // Checkpoint 1, every source has delivered or stopped. A rejected // reserved commit puts its Collection in `error`. A stopped tab that is - // still open shows its peers' refetch, except after cleanup (no rows) - // or resume invalidation (`error`, rows not promised). Eager rows must - // equal the head snapshot; on-demand rows are not judged (above). + // still open shows its peers' refetch, including one in `error` after + // resume invalidation (Law 3); a cleaned-up tab has no rows. Eager rows + // must equal the head snapshot; on-demand rows are not judged (above). const stoppedStatus = stop === `cleanup` ? `cleaned-up` @@ -607,9 +612,7 @@ describe(`multi-tab Electric over one persisted database`, () => { ? `error` : `ready` const judgesRowsBeforeRestart = (tab: Tab) => - syncMode === `eager` && - (tab !== stoppedTab || - (stop !== `cleanup` && stop !== `resume invalidated`)) + syncMode === `eager` && (tab !== stoppedTab || stop !== `cleanup`) const observe = (judgesRows: (tab: Tab) => boolean) => tabs.map((tab) => ({ tab: tab.tab, From 851e8d9da996c6584ca1bcbf21d76e0eb366a921 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 09:29:04 -0600 Subject: [PATCH 07/24] fix(sqlite-persistence): keep Collection status when a peer's commit reloads rows Maintainer decision on PR #2265: a Collection status changes only through its own source. The persisted runtime reloaded durable rows through a truncate that marks a Collection ready by default, so a peer's commit could mark a loading tab ready or move a failed tab from error to ready. The runtime's truncate is now status-neutral where it is bound. The multi-tab Electric oracle gains generated histories (fixed and random campaigns, seed and path replay). They interleave server changes, deploys, partial and complete deliveries, every non-invalidating stop, restarts, and reloads, and judge the durable rows against a fold of committed batches, every Collection status, and idle tabs' rows after every step. The fixed seed fails without the repair; removing the reservation gate or any of three Electric abandons fails both campaigns. They found #2267, which is pinned with it.fails and left out of the generated histories. Co-authored-by: Isaac --- .changeset/persisted-reload-keeps-status.md | 5 + docs/contributing/oracle-coverage.md | 19 + .../tests/multi-tab-electric-oracle.test.ts | 626 +++++++++++++++++- .../src/persisted.ts | 10 +- 4 files changed, 652 insertions(+), 8 deletions(-) create mode 100644 .changeset/persisted-reload-keeps-status.md diff --git a/.changeset/persisted-reload-keeps-status.md b/.changeset/persisted-reload-keeps-status.md new file mode 100644 index 0000000000..99c3bd75c9 --- /dev/null +++ b/.changeset/persisted-reload-keeps-status.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db-sqlite-persistence-core': patch +--- + +Keep a persisted Collection's status when another tab's commit reloads its rows. Before, a reload from durable rows could mark a tab `ready` while its own source was still loading, or move a tab whose initial sync failed from `error` to `ready`. Now only the Collection's own source changes its status. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 84402fa276..e192b6b129 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2951,6 +2951,25 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. because the persisted wrapper ends its transactions on cleanup; the Electric descriptor-isolation oracle kills it without persistence. On-demand cells judge receipts and status only. + Generated histories in the same file interleave server changes, + deploys, partial and complete deliveries, every stop except resume + invalidation, restarts, and reloads across two or three eager tabs + (fixed seed 952051961 and a random seed, 60 runs each; + `TANSTACK_DB_MULTI_TAB_ORACLE_SEED` and `_PATH` replay one). The + model folds committed batches into the expected durable rows. After + every step it judges the durable rows, every Collection status, and + the rows of each tab with no open transaction; after every tab catches + up it judges them again. A Collection status changes only through its + own source (maintainer decision on PR #2265): a peer's commit leaves a + `loading` or failed tab as it is. That law failed on `main` (a peer's + truncate reload marked such a tab ready); the fixed seed fails without + the repair. Removing the reservation gate, or the `onError`, give-up, + or signal-abort abandon, fails both campaigns. Each declared premise, + including a peer commit after each kind of stop mid-transaction, is + reached in every campaign. Known gap #2267 is left out of the + generated histories and pinned with `it.fails`: a resumed tab that lags + behind a peer can receive an update for a row the peer already + deleted, and Electric's resume check puts it in `error`. A claimed leftover-replay/reset failure still needs a concrete schedule; the bounded two-replay-then-reset probe drains replay before reset. Live Electric/browser OPFS delivery needs the matching schedule in diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 4b51ad5fdf..4928ebfd70 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -54,8 +54,20 @@ * maintainer decision on PR #2265. A persisted Collection that a source * failure put in `error` still applies its peers' durable commits, so its * public rows match the durable rows. This file judges it after resume - * invalidation, the one stop that puts the Collection in `error`; the source - * must end its reserved transaction for the peers' refetch to apply. + * invalidation and after a failed initial sync; the source must end its + * reserved transaction for the peers' commits to apply. + * + * Law 4, a Collection status changes only through its own source. Authority: + * maintainer decision on PR #2265. A peer's durable commit reloads a tab's + * rows but never moves its status: a tab whose own stream has not reached + * up-to-date stays `loading`, and a tab whose initial sync failed stays in + * `error` until it restarts. + * + * The generated histories below judge Laws 1, 3, and 4 over interleaved + * server changes, deploys, deliveries, stops, and recoveries, against a + * model of the durable rows. One known gap is left out of them and pinned: + * a resumed tab that lags behind a peer can receive an update for a row the + * peer already deleted, and Electric's resume check then puts it in `error`. * * Electric's own cleanup abandon is redundant behind the persisted wrapper, * which ends its transactions on cleanup, so that mutant survives here; the @@ -70,6 +82,7 @@ import { mkdtempSync, rmSync } from 'node:fs' import { tmpdir } from 'node:os' import { join } from 'node:path' import { afterEach, describe, expect, it, vi } from 'vitest' +import { fc } from '@fast-check/vitest' import { createCollection } from '@tanstack/db' import { createOracleReach } from '../../db/tests/oracle-reach.js' import { persistedCollectionOptions } from '../../db-sqlite-persistence-core/src' @@ -202,6 +215,7 @@ function openHost() { const filename = join(directory, `shared.sqlite`) const dbName = `multi-tab-${Math.random()}` const closers: Array<() => Promise> = [] + let observer: ReturnType | undefined // Host activity for the quiescence checkpoint, and the number of commit // notices, read from the coordinator's own channel. let activity = 0 @@ -331,8 +345,13 @@ function openHost() { } } - // The hosts are quiet when several tasks pass with no SQL statement and no - // coordinator message. The bound turns a livelock into a named failure. + // The hosts are quiet when five 2 ms tasks pass with no SQL statement and + // no coordinator message, within 400 tasks; the bound turns a livelock into + // a named failure. Quiet is evidence, not proof, that every notice and + // hydration finished, because the public API exposes no completion signal + // for peer notices. Its premises are calibrated instead: declared reach + // shows the peer commit arrived, and the abandon and readiness mutants fail + // only when a hydration ran after that commit. const quiesce = async (label: string) => { let quietTasks = 0 for (let task = 0; task < 400 && quietTasks < 5; task++) { @@ -347,7 +366,26 @@ function openHost() { openTab, quiesce, commitNotices: () => commitNotices, + // The durable rows of the shared Collection, read through a connection of + // its own so the read counts as no host activity. + durableRows: async (): Promise> => { + observer ??= createWASQLiteTestDatabase({ filename, timeout: 1000 }) + const registry = await observer.execute<{ table_name: string }>( + `SELECT table_name FROM collection_registry WHERE collection_id = ?`, + [`items`], + ) + const table = registry[0]?.table_name + if (!table) return [] + const rows = await observer.execute<{ value: string }>( + `SELECT value FROM "${table}"`, + ) + return rows + .map((row) => JSON.parse(row.value) as Item) + .map(({ id, label }) => ({ id, label })) + .sort((left, right) => left.id - right.id) + }, close: async () => { + await observer?.close?.() for (const close of closers.reverse()) await close() channel.close() rmSync(directory, { recursive: true, force: true }) @@ -371,6 +409,46 @@ async function openWarmTabs(host: Host, count: number, syncMode: SyncMode) { return tabs } +// Generated-history campaigns (ORC-007): a fixed seed and a random seed with +// the same property and budget. TANSTACK_DB_MULTI_TAB_ORACLE_SEED and +// TANSTACK_DB_MULTI_TAB_ORACLE_PATH replay one recorded counterexample; +// TANSTACK_DB_MULTI_TAB_ORACLE_RUNS sets the budget. +const GENERATED_FIXED_SEED = 952051961 +function generatedCampaigns( + environment: Record = process.env, +): Array<{ + mode: `fixed` | `random` | `replay` + parameters: { numRuns: number; seed?: number; path?: string } +}> { + const runText = environment.TANSTACK_DB_MULTI_TAB_ORACLE_RUNS ?? `60` + const numRuns = Number(runText) + if (!/^\d+$/.test(runText) || numRuns < 1) { + throw new Error(`Invalid TANSTACK_DB_MULTI_TAB_ORACLE_RUNS`) + } + const seedText = environment.TANSTACK_DB_MULTI_TAB_ORACLE_SEED + const path = environment.TANSTACK_DB_MULTI_TAB_ORACLE_PATH + if (seedText === undefined && path === undefined) { + return [ + { mode: `fixed`, parameters: { numRuns, seed: GENERATED_FIXED_SEED } }, + { mode: `random`, parameters: { numRuns } }, + ] + } + const seed = Number(seedText) + if (seedText === undefined || !/^-?\d+$/.test(seedText)) { + throw new Error(`Invalid TANSTACK_DB_MULTI_TAB_ORACLE_SEED`) + } + if (path !== undefined && !/^\d+(?::\d+)*$/.test(path)) { + throw new Error(`TANSTACK_DB_MULTI_TAB_ORACLE_PATH must be a shrink path`) + } + return [ + { + mode: `replay`, + parameters: + path === undefined ? { numRuns, seed } : { numRuns: 1, seed, path }, + }, + ] +} + async function waitForStream(tab: Tab, count: number) { for (let task = 0; task < 400 && tab.streamCount() < count; task++) { await new Promise((resolve) => setTimeout(resolve, 2)) @@ -662,6 +740,546 @@ describe(`multi-tab Electric over one persisted database`, () => { 30_000, ) + // Law 1 and Law 3, generated histories. The enumerated grammar above fixes + // one refetch storm and one stopped tab. These histories interleave, in any + // legal order: + // + // - server changes (insert, update, or delete of one of four rows); + // - deploys, after which every stream must refetch; + // - deliveries to one tab: part of what its stream still owes, with no + // up-to-date, which leaves a reserved transaction open, or everything it + // owes and up-to-date, which commits it; + // - a tab's source stopping for good (each terminal path except resume + // invalidation, which the enumerated grammar owns); + // - recovery of a stopped tab, and a reload of a running tab. + // + // Model. The server is a log per shape handle. A tab's stream owes the log + // after its position, or must-refetch then the snapshot when its handle is + // stale. A tab's open transaction holds the messages delivered since it + // began; a commit applies them to the durable rows in commit order, after a + // truncate when the transaction is a refetch or a fresh snapshot. An + // abandoned transaction applies nothing. So the expected durable rows are a + // fold of committed batches. A lagging tab can commit an older value after + // a peer committed a newer one; that is the eventual-consistency note in + // the coverage map (#2209), so the fold keeps it rather than the server + // head. + // + // Checkpoint after every step, at host quiescence: + // - no Collection is in `error`, and only a tab stopped by cleanup is + // `cleaned-up`; a rejected reserved commit would put its Collection in + // `error`; + // - the durable rows equal the fold; + // - every open tab with no open transaction shows the durable rows. That + // includes a tab whose source stopped: it abandoned its transaction, so + // its peers' commits apply to it (Law 1, Law 3). + // After the history, every tab recovers and catches up. Then every tab must + // show the durable rows, and the durable rows must be the server head when + // the fold says so. + // + // The model reads one production fact: the position a new stream asks the + // server to start from. The simulated server answers that request; the + // model does not judge it, except that a full-snapshot request must get a + // full snapshot. + type HistoryStep = + | { kind: `server change`; key: number; pick: number } + | { kind: `deploy` } + | { kind: `advance`; tab: number; pick: number } + | { kind: `catch up`; tab: number } + | { kind: `stop`; tab: number; pick: number } + | { kind: `recover`; tab: number; pick: number } + const historyStep: fc.Arbitrary = fc.oneof( + { + weight: 3, + arbitrary: fc.record({ + kind: fc.constant(`server change` as const), + key: fc.integer({ min: 1, max: 4 }), + pick: fc.nat(5), + }), + }, + { + weight: 1, + arbitrary: fc.record({ kind: fc.constant(`deploy` as const) }), + }, + { + weight: 3, + arbitrary: fc.record({ + kind: fc.constant(`advance` as const), + tab: fc.nat(2), + pick: fc.nat(5), + }), + }, + { + weight: 3, + arbitrary: fc.record({ + kind: fc.constant(`catch up` as const), + tab: fc.nat(2), + }), + }, + { + weight: 3, + arbitrary: fc.record({ + kind: fc.constant(`stop` as const), + tab: fc.nat(2), + pick: fc.nat(3), + }), + }, + { + weight: 1, + arbitrary: fc.record({ + kind: fc.constant(`recover` as const), + tab: fc.nat(2), + pick: fc.nat(1), + }), + }, + ) + const generatedHistory = fc.record({ + tabCount: fc.integer({ min: 2, max: 3 }), + steps: fc.array(historyStep, { minLength: 8, maxLength: 24 }), + }) + type GeneratedHistory = { tabCount: number; steps: Array } + + type ServerLog = { + handle: number + // Versions under the current handle; version 0 is the snapshot. + snapshot: Map + changes: Array<{ version: number; message: Message }> + rows: Map + } + type TabModel = { + stopped: StreamStop | undefined + // The position the stream has delivered: a handle and a version. An + // undefined handle means the stream owes a full snapshot. + handle: number | undefined + version: number + // Messages of the open transaction, and whether it truncates first. + open: Array> | undefined + truncates: boolean + // Collection readiness: a new sync run is `loading` until its own + // stream's first up-to-date. A stop before then is the final initial + // failure, which puts the Collection in `error` (Electric's initial-sync + // law). A Collection status changes only through its own source: a + // peer's durable commit reloads the rows but leaves the status alone + // (maintainer decision on PR #2265, Law 4). + ready: boolean + // A sync run that resumed from a persisted position, so Electric checks + // that each update names a row it holds. + resumed: boolean + failedInitialSync: boolean + // The stop that ended an open transaction, until the tab recovers. A + // peer commit in that interval is the premise of the abandon law. + stoppedMidTransaction: StreamStop | undefined + } + const stops: ReadonlyArray = [ + `error without retry`, + `SDK gives up`, + `signal aborted`, + `cleanup`, + ] + const rowsOf = (rows: Map): Array => + [...rows] + .map(([id, label]) => ({ id, label })) + .sort((left, right) => left.id - right.id) + const position = (log: ServerLog, version: number) => ({ + handle: `h${log.handle}`, + offset: `${version}_0`, + }) + // The request a new stream made, read as a model position. + const requestedPosition = ( + startedAt: { handle: string | undefined; offset: string }, + log: ServerLog, + ): Pick => { + if (startedAt.handle === undefined || startedAt.offset === `-1`) { + return { handle: undefined, version: -1 } + } + const handle = Number(startedAt.handle.slice(1)) + const version = Number(startedAt.offset.split(`_`)[0]) + return handle === log.handle + ? { handle, version } + : { handle, version: Number.NaN } + } + + const runGeneratedHistory = async ( + { tabCount, steps }: GeneratedHistory, + onKnownGap: () => void, + ) => { + host = openHost() + const activeHost = host + const tabs = await openWarmTabs(activeHost, tabCount, `eager`) + const log: ServerLog = { + handle: 1, + snapshot: new Map(beforeDeploy.map((row) => [row.id, row.label])), + changes: [], + rows: new Map(beforeDeploy.map((row) => [row.id, row.label])), + } + const durable = new Map() + const models: Array = tabs.map(() => ({ + stopped: undefined, + handle: undefined, + version: -1, + open: undefined, + truncates: false, + ready: false, + failedInitialSync: false, + resumed: false, + stoppedMidTransaction: undefined, + })) + let serverVersion = 0 + generatedReach.beginHistory() + + const commit = (model: TabModel) => { + if (model.truncates) durable.clear() + for (const message of model.open ?? []) { + if (!(`key` in message)) continue + const row = message.value + if (message.headers.operation === `delete`) durable.delete(row.id) + else durable.set(row.id, row.label) + } + model.open = undefined + model.truncates = false + } + const abandon = (model: TabModel) => { + model.open = undefined + model.truncates = false + } + // What the server sends one tab next: must-refetch for a stale handle, + // the snapshot for a full-snapshot request, or the changes it owes. + const deliver = async (index: number, upTo: number | `all`) => { + const tab = tabs[index]! + const model = models[index]! + if (model.handle !== undefined && model.handle !== log.handle) { + tab.deliver([mustRefetch], { + handle: `h${model.handle}`, + offset: `${Math.max(model.version, 0)}_0`, + }) + generatedReach.reach(`must-refetch delivered`) + abandon(model) + model.open = [] + model.truncates = true + model.handle = undefined + model.version = -1 + await activeHost.quiesce(`tab ${tab.tab} must-refetch`) + if (upTo !== `all`) return + } + const messages: Array> = [] + if (model.handle === undefined) { + model.open ??= [] + model.truncates = true + messages.push( + ...rowsOf(log.snapshot).map(insert), + ...log.changes.map((change) => change.message), + ) + model.handle = log.handle + model.version = serverVersion + } else { + const owed = log.changes.filter( + (change) => change.version > model.version, + ) + const count = upTo === `all` ? owed.length : Math.min(upTo, owed.length) + messages.push(...owed.slice(0, count).map((change) => change.message)) + model.version = count > 0 ? owed[count - 1]!.version : model.version + if (messages.length > 0) model.open ??= [] + } + // Known gap: a resumed tab that lags behind a peer can receive an + // update for a row the peer's commit already deleted from the shared + // durable rows. Electric's resume check then reports an unseen row and + // puts the Collection in `error` (issue #2267). + // Generated campaigns leave that history out; the pinned replay below + // keeps it. + if (model.resumed && !model.truncates) { + const held = new Set(durable.keys()) + for (const message of [...(model.open ?? []), ...messages]) { + if (!(`key` in message)) continue + const { id } = message.value + if (message.headers.operation === `update` && !held.has(id)) { + onKnownGap() + } + if (message.headers.operation === `delete`) held.delete(id) + else held.add(id) + } + } + model.open?.push(...messages) + const complete = upTo === `all` + if (messages.length === 0 && !complete) return + tab.deliver( + complete ? [...messages, upToDate] : messages, + position(log, model.version), + ) + if (complete) { + commit(model) + model.ready = true + } + await activeHost.quiesce(`tab ${tab.tab} delivered`) + } + const recover = async (index: number, how: `restart` | `reload`) => { + const stopped = tabs[index]! + const streamsBefore = stopped.streamCount() + let recovered = stopped + if (how === `restart`) await stopped.restart() + else { + await stopped.close() + await activeHost.quiesce(`tab ${stopped.tab} closed`) + recovered = activeHost.openTab(stopped.tab, `eager`) + tabs[index] = recovered + } + abandon(models[index]!) + await waitForStream(recovered, streamsBefore + 1) + generatedReach.reach(`tab recovered`) + const requested = requestedPosition(recovered.startedAt(), log) + Object.assign(models[index]!, { + stopped: undefined, + ready: false, + failedInitialSync: false, + resumed: requested.handle !== undefined, + stoppedMidTransaction: undefined, + ...requested, + }) + await activeHost.quiesce(`tab ${recovered.tab} recovered`) + } + + const check = async (label: string) => { + const expectedDurable = rowsOf(durable) + const judgesRows = (index: number) => + models[index]!.open === undefined && + models[index]!.stopped !== `cleanup` + const expectedStatus = (index: number) => + models[index]!.stopped === `cleanup` + ? `cleaned-up` + : models[index]!.failedInitialSync + ? `error` + : models[index]!.ready + ? `ready` + : `loading` + const expected = { + durableRows: expectedDurable, + tabs: tabs.map((tab, index) => ({ + tab: tab.tab, + status: expectedStatus(index), + ...(judgesRows(index) ? { rows: expectedDurable } : {}), + })), + } + const actual = { + durableRows: await activeHost.durableRows(), + tabs: tabs.map((tab, index) => ({ + tab: tab.tab, + status: tab.collection.status, + ...(judgesRows(index) ? { rows: tab.publicRows() } : {}), + })), + } + // fast-check reports only the message, so it carries both sides, plus + // each errored Collection's reason, which `preload()` rejects with. + const errors = await Promise.all( + tabs.map((tab) => + tab.collection.status === `error` + ? tab.collection.preload().then( + () => undefined, + (error: unknown) => `${tab.tab}: ${String(error)}`, + ) + : undefined, + ), + ) + expect( + actual, + `${label}\nactual ${JSON.stringify(actual)}\nexpected ${JSON.stringify(expected)}\nerrors ${JSON.stringify(errors.filter(Boolean))}`, + ).toEqual(expected) + } + + // Warm start: every tab delivers the snapshot and commits it. + for (const index of tabs.keys()) await deliver(index, `all`) + await check(`warm`) + + for (const [stepIndex, step] of steps.entries()) { + const openBefore = models.some((model) => model.open !== undefined) + const unreadyBefore = models.some( + (model) => + model.stopped !== `cleanup` && + (!model.ready || model.failedInitialSync), + ) + const noticesBefore = activeHost.commitNotices() + if (step.kind === `server change`) { + serverVersion++ + const present = log.rows.has(step.key) + const operation = !present + ? `insert` + : step.pick % 3 === 0 + ? `delete` + : `update` + const row = { id: step.key, label: `v${serverVersion}` } + if (operation === `delete`) log.rows.delete(step.key) + else log.rows.set(step.key, row.label) + log.changes.push({ + version: serverVersion, + message: { + key: String(row.id), + value: row, + headers: { operation }, + }, + }) + } else if (step.kind === `deploy`) { + log.handle++ + serverVersion = 0 + log.snapshot = new Map(log.rows) + log.changes = [] + } else { + // A stop goes to a tab with an open transaction when there is one, + // so the generator reaches the state the abandon law is about. + const openTabs = models.flatMap((model, index) => + model.open !== undefined && model.stopped === undefined + ? [index] + : [], + ) + // Only `recover` recovers a stopped tab. Another step aimed at a + // stopped tab goes to a running one, so a stopped tab stays stopped + // while its peers commit. + const runningTabs = models.flatMap((model, index) => + model.stopped === undefined ? [index] : [], + ) + const index = + step.kind === `stop` && openTabs.length > 0 + ? openTabs[step.tab % openTabs.length]! + : step.kind !== `recover` && + models[step.tab % tabs.length]!.stopped !== undefined && + runningTabs.length > 0 + ? runningTabs[step.tab % runningTabs.length]! + : step.tab % tabs.length + const model = models[index]! + if (model.stopped !== undefined || step.kind === `recover`) { + // A running tab can only reload; a stopped tab restarts or reloads. + const how = + model.stopped === undefined || + (step.kind === `recover` && step.pick === 1) + ? `reload` + : `restart` + await recover(index, how) + } else if (step.kind === `stop`) { + const how = stops[step.pick % stops.length]! + if (model.open !== undefined) { + generatedReach.reach(`source stopped with an open transaction`) + model.stoppedMidTransaction = how + } + void Promise.resolve(tabs[index]!.stopStream(how)).catch( + () => undefined, + ) + model.stopped = how + abandon(model) + model.failedInitialSync = how !== `cleanup` && !model.ready + await activeHost.quiesce(`tab ${tabs[index]!.tab} stopped`) + } else { + await deliver(index, step.kind === `catch up` ? `all` : step.pick + 1) + } + } + if (activeHost.commitNotices() > noticesBefore) { + for (const model of models) { + const stop = model.stoppedMidTransaction + if ( + stop === `error without retry` || + stop === `SDK gives up` || + stop === `signal aborted` + ) { + generatedReach.reach(`peer commit after ${stop} mid-transaction`) + } + } + } + if (unreadyBefore && activeHost.commitNotices() > noticesBefore) { + generatedReach.reach(`peer commit while a tab was not ready`) + } + if (openBefore && activeHost.commitNotices() > noticesBefore) { + const stillOpen = models.some((model) => model.open !== undefined) + if (stillOpen) { + generatedReach.reach( + `peer commit during an open reserved transaction`, + ) + } + } + if (rowsOf(durable).some(({ id, label }) => log.rows.get(id) !== label)) { + generatedReach.reach(`durable rows behind the server`) + } + await check(`after step ${stepIndex}: ${JSON.stringify(step)}`) + } + + // Every tab recovers and catches up; then every tab shows the durable + // rows, and those are the head unless a lagging commit left them behind. + for (const index of tabs.keys()) { + if (models[index]!.stopped !== undefined) await recover(index, `restart`) + await deliver(index, `all`) + } + await check(`after every tab caught up`) + } + + type GeneratedReach = + | `must-refetch delivered` + | `peer commit during an open reserved transaction` + | `source stopped with an open transaction` + | `tab recovered` + | `durable rows behind the server` + | `peer commit while a tab was not ready` + | `peer commit after error without retry mid-transaction` + | `peer commit after SDK gives up mid-transaction` + | `peer commit after signal aborted mid-transaction` + const createGeneratedReach = () => + createOracleReach(`browser.multi-tab-reserved-history`, [ + `must-refetch delivered`, + `peer commit during an open reserved transaction`, + `source stopped with an open transaction`, + `tab recovered`, + `durable rows behind the server`, + `peer commit while a tab was not ready`, + `peer commit after error without retry mid-transaction`, + `peer commit after SDK gives up mid-transaction`, + `peer commit after signal aborted mid-transaction`, + ]) + let generatedReach = createGeneratedReach() + + it.each(generatedCampaigns())( + `keeps reserved commits lawful across generated multi-tab histories ($mode campaign)`, + async ({ parameters }) => { + generatedReach = createGeneratedReach() + await fc.assert( + fc.asyncProperty(generatedHistory, async (history) => { + try { + // fc.pre discards a history that reaches the known gap. + await runGeneratedHistory(history, () => fc.pre(false)) + } finally { + await host?.close() + host = undefined + streams.length = 0 + } + }), + parameters, + ) + console.info( + `multi-tab generated reach`, + Object.fromEntries(generatedReach.counts()), + ) + generatedReach.assertReached() + }, + 600_000, + ) + + // Known gap #2267, pinned: the shrunk generated history in which a resumed tab + // that lags behind its peer receives an update for a row the peer already + // deleted. It replays through the same driver and checks with the known-gap + // detector off. Remove `fails` when Electric's resume check stops treating + // rows from peers' commits as this stream's evidence. + it.fails( + `keeps a resumed lagging tab out of error when a peer deleted the row it updates`, + () => + runGeneratedHistory( + { + tabCount: 2, + steps: [ + { kind: `recover`, tab: 1, pick: 0 }, + { kind: `server change`, key: 2, pick: 1 }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `server change`, key: 2, pick: 0 }, + { kind: `server change`, key: 1, pick: 0 }, + { kind: `advance`, tab: 0, pick: 0 }, + ], + }, + () => undefined, + ), + 60_000, + ) + // Law 2 histories: tab A's stream applies v1 and v2; tab B's lagging stream // applies v1 later. B then either catches up or closes. The checkpoint is // host quiescence after the last delivery. diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index f15c426f3c..0f401039b5 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -964,7 +964,9 @@ type SyncControlFns = { ) => void) | null commit: ((signal?: AbortSignal) => SyncAppliedReceipt) | null - truncate: ((options?: { markReady?: boolean }) => void) | null + // A runtime truncate replaces rows from durable state. It never changes + // Collection status: only the Collection's own source does. + truncate: (() => void) | null metadata: SyncMetadataApi | null markError: ((error: unknown) => void) | null } @@ -2563,7 +2565,7 @@ class PersistedCollectionRuntime< if (this.syncControls.begin && this.syncControls.commit) { const applied = this.withInternalApply(() => { this.syncControls.begin?.() - this.syncControls.truncate?.({ markReady: false }) + this.syncControls.truncate?.() if (resetMetadata) { this.syncControls.metadata?.collection.set( resetMetadata.key, @@ -5997,7 +5999,7 @@ function createWrappedSyncConfig< begin: params.begin, write: params.write as SyncControlFns[`write`], commit: params.commit, - truncate: params.truncate, + truncate: () => params.truncate({ markReady: false }), metadata: params.metadata ?? null, markError: params.markError, }, @@ -6861,7 +6863,7 @@ function createLoopbackSyncConfig< begin: params.begin, write: params.write as SyncControlFns[`write`], commit: params.commit, - truncate: params.truncate, + truncate: () => params.truncate({ markReady: false }), metadata: params.metadata ?? null, markError: params.markError, }, From b1fbac2698868c5b2b016ee2db45e7f6c09968d3 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 12:03:30 -0600 Subject: [PATCH 08/24] fix(sqlite-persistence): make gap recovery one hydration cycle Review finding: gap recovery advanced the stream position and applied targeted pullSince deltas before a later delta's reload deferred for a reserved commit, so the retried notice skipped the reload. Gap recovery now waits for a reserved commit before its first step and is one hydration cycle from there; a reservation that opens later straddles it. The persisted-reads grammar gains a gap route whose pullSince deltas reload partway through. It failed 28 cells on the parent; a gate only at recovery start still failed 15, and reloads that start their own cycle inside recovery fail 61. The multi-tab generated histories now judge Law 2: deliveries may commit a prefix of what a stream owes, and after every open tab catches up the durable rows and every tab show the server head. The #2209 history replays through the same driver and checks with it.fails, replacing the pinned lagging examples that carried their own expected answers. Electric clears its open-transaction flag before it commits, and the persisted wrapper uses one reservation lookup for subset admission. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 19 ++- .../tests/multi-tab-electric-oracle.test.ts | 150 +++++++++--------- .../src/persisted.ts | 43 +++-- .../tests/persisted-oracle.test.ts | 111 +++++++++++-- .../electric-db-collection/src/electric.ts | 3 +- 5 files changed, 220 insertions(+), 106 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index e192b6b129..1a8037b2a5 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2896,8 +2896,12 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. A second registration crosses the routes with the reserving timings, ending, and follow-up dimensions. The routes are subset, complete subset, a follower's routed subset, the source's own baseline, - reset, full reload, sequence gap with or without pullSince, and a - targeted change that starts no hydration. The reserving timings are + reset, full reload, sequence gap with or without pullSince, sequence + gap whose pullSince deltas reload partway through (a targeted delta, + then one past the targeted-invalidation limit), and a targeted change + that starts no hydration. Gap recovery is one hydration cycle from its + first step, so a reservation that opens at its scope entry straddles + it. The reserving timings are across the mutex turn, across scope entry, during the read, in the route's first publication, and across startup. The ending is commits, abandoned, cleanup, an aborted demand, or the source awaiting its own @@ -2908,8 +2912,8 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. events before it. Every cell declares reach (ORC-015). | Evidence | Result | | --- | --- | - | Row-algebra cross | 360 cells pass. | - | Reservation cross | 188 cells pass. | + | Row-algebra cross and reservation cross | 649 cells pass. | + | Gap recovery that reloads partway through, on `851e8d9da` | 28 cells fail: recovery advanced the stream position and applied a targeted delta before it deferred, so its retry skipped the reload. A gate only at recovery start still fails 15 scope-entry cells; reloads that start their own cycle inside recovery fail 61. | | `5649a07d4` | The open cells fail with `a persisted sync transaction cannot cross a hydration cycle`. Three `after read` reset follow-up cells also fail, because `main` applied a pre-reset commit before the reset set its term fence. Queueing resets with notices fixes this pre-existing defect. | | Hostile mutants | Each mutant fails its cells, and every failure is an assertion. Deferring every notice fails the targeted change. Dropping the follower reply's wait fails the routed subset. Reporting hydration start before the check fails the aborted demand. A reset reload that starts its own cycle fails the first-publication reset. Applying a reset outside the notice queue fails the reset follow-ups. Retrying a waited notice while the reservation is open fails the follow-up and routed cells (repeated pullSince). Checking only at the mutex turn fails every scope-entry cell. Gating subset hydration only fails every reset, reload, and gap cell. Requeueing a waited notice behind later notices fails 12 reset follow-up cells: the pre-reset commit is published before its reset, and a reload is lost. | Not covered: `forceReloadSubset`, which only a sync-absent Collection @@ -2970,6 +2974,13 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. generated histories and pinned with `it.fails`: a resumed tab that lags behind a peer can receive an update for a row the peer already deleted, and Electric's resume check puts it in `error`. + A delivery may also commit only a prefix of what a stream owes, as a + server answering at an earlier moment does, so lagging commits are + generated. The final checkpoint requires the server head (Law 2). + Known gap #2209 is the one history that breaks it: a lagging tab + commits an older row and closes for good. Generated histories never + close a tab for good, and an `it.fails` replay through the same driver + and checks pins that history. A claimed leftover-replay/reset failure still needs a concrete schedule; the bounded two-replay-then-reset probe drains replay before reset. Live Electric/browser OPFS delivery needs the matching schedule in diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 4928ebfd70..315afdaa9e 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -26,12 +26,13 @@ * map's eventual-consistency note for persisted multi-tab Collections * (docs/contributing/oracle-coverage.md, issue #2209): a lagging tab may * briefly store an older transaction, and every tab converges once the - * streams catch up. The law judged here is narrower and observable: a tab - * whose own stream has applied the server head shows the head once the hosts - * are quiet. When the lagging tab closes before it catches up, nothing - * redelivers the head to the faster tab, so this law fails today. That - * history is pinned with `it.fails` until #2209 orders source writes by - * server position. + * streams catch up. The generated histories below judge it: after every + * open tab catches up, the durable rows and every open tab show the server + * head, however earlier lagging commits left them. When a lagging tab + * commits an older row and then closes for good, nothing redelivers the head + * to the faster tab, so this law fails today. That history is pinned with + * `it.fails` until #2209 orders source writes by server position; generated + * histories never close a tab for good. * * Model: the server is a log of snapshots. A history names which snapshot * each tab's stream has delivered; the expected public rows of a tab whose @@ -784,7 +785,10 @@ describe(`multi-tab Electric over one persisted database`, () => { | { kind: `server change`; key: number; pick: number } | { kind: `deploy` } | { kind: `advance`; tab: number; pick: number } - | { kind: `catch up`; tab: number } + | { kind: `catch up`; tab: number; pick: number } + // A tab that closes for good. Generated histories never close a tab; + // only the pinned #2209 replay does (see below). + | { kind: `close`; tab: number } | { kind: `stop`; tab: number; pick: number } | { kind: `recover`; tab: number; pick: number } const historyStep: fc.Arbitrary = fc.oneof( @@ -813,6 +817,9 @@ describe(`multi-tab Electric over one persisted database`, () => { arbitrary: fc.record({ kind: fc.constant(`catch up` as const), tab: fc.nat(2), + // 0 delivers everything owed; n commits only the next n changes, as + // a server answering at an earlier moment does. + pick: fc.nat(3), }), }, { @@ -868,6 +875,7 @@ describe(`multi-tab Electric over one persisted database`, () => { // The stop that ended an open transaction, until the tab recovers. A // peer commit in that interval is the premise of the abandon law. stoppedMidTransaction: StreamStop | undefined + closed: boolean } const stops: ReadonlyArray = [ `error without retry`, @@ -922,6 +930,7 @@ describe(`multi-tab Electric over one persisted database`, () => { failedInitialSync: false, resumed: false, stoppedMidTransaction: undefined, + closed: false, })) let serverVersion = 0 generatedReach.beginHistory() @@ -943,7 +952,11 @@ describe(`multi-tab Electric over one persisted database`, () => { } // What the server sends one tab next: must-refetch for a stale handle, // the snapshot for a full-snapshot request, or the changes it owes. - const deliver = async (index: number, upTo: number | `all`) => { + const deliver = async ( + index: number, + upTo: number | `all`, + commits: boolean, + ) => { const tab = tabs[index]! const model = models[index]! if (model.handle !== undefined && model.handle !== log.handle) { @@ -958,7 +971,7 @@ describe(`multi-tab Electric over one persisted database`, () => { model.handle = undefined model.version = -1 await activeHost.quiesce(`tab ${tab.tab} must-refetch`) - if (upTo !== `all`) return + if (!commits) return } const messages: Array> = [] if (model.handle === undefined) { @@ -998,7 +1011,7 @@ describe(`multi-tab Electric over one persisted database`, () => { } } model.open?.push(...messages) - const complete = upTo === `all` + const complete = commits if (messages.length === 0 && !complete) return tab.deliver( complete ? [...messages, upToDate] : messages, @@ -1036,13 +1049,15 @@ describe(`multi-tab Electric over one persisted database`, () => { await activeHost.quiesce(`tab ${recovered.tab} recovered`) } - const check = async (label: string) => { - const expectedDurable = rowsOf(durable) + const check = async (label: string, head?: Array) => { + const expectedDurable = head ?? rowsOf(durable) + // A tab closed for good is a cleaned-up Collection with no rows. const judgesRows = (index: number) => models[index]!.open === undefined && - models[index]!.stopped !== `cleanup` + models[index]!.stopped !== `cleanup` && + !models[index]!.closed const expectedStatus = (index: number) => - models[index]!.stopped === `cleanup` + models[index]!.stopped === `cleanup` || models[index]!.closed ? `cleaned-up` : models[index]!.failedInitialSync ? `error` @@ -1084,7 +1099,7 @@ describe(`multi-tab Electric over one persisted database`, () => { } // Warm start: every tab delivers the snapshot and commits it. - for (const index of tabs.keys()) await deliver(index, `all`) + for (const index of tabs.keys()) await deliver(index, `all`, true) await check(`warm`) for (const [stepIndex, step] of steps.entries()) { @@ -1114,6 +1129,12 @@ describe(`multi-tab Electric over one persisted database`, () => { headers: { operation }, }, }) + } else if (step.kind === `close`) { + const index = step.tab % tabs.length + await tabs[index]!.close() + abandon(models[index]!) + models[index]!.closed = true + await activeHost.quiesce(`tab ${tabs[index]!.tab} closed for good`) } else if (step.kind === `deploy`) { log.handle++ serverVersion = 0 @@ -1164,7 +1185,15 @@ describe(`multi-tab Electric over one persisted database`, () => { model.failedInitialSync = how !== `cleanup` && !model.ready await activeHost.quiesce(`tab ${tabs[index]!.tab} stopped`) } else { - await deliver(index, step.kind === `catch up` ? `all` : step.pick + 1) + await deliver( + index, + step.kind === `catch up` + ? step.pick === 0 + ? `all` + : step.pick + : step.pick + 1, + step.kind === `catch up`, + ) } } if (activeHost.commitNotices() > noticesBefore) { @@ -1196,13 +1225,16 @@ describe(`multi-tab Electric over one persisted database`, () => { await check(`after step ${stepIndex}: ${JSON.stringify(step)}`) } - // Every tab recovers and catches up; then every tab shows the durable - // rows, and those are the head unless a lagging commit left them behind. + // Law 2: every open tab recovers and catches up, and then the durable + // rows and every open tab show the server head. Each tab's last batch + // rewrites every row that changed after its position, so no lagging + // commit survives the catch-up. for (const index of tabs.keys()) { + if (models[index]!.closed) continue if (models[index]!.stopped !== undefined) await recover(index, `restart`) - await deliver(index, `all`) + await deliver(index, `all`, true) } - await check(`after every tab caught up`) + await check(`after every tab caught up`, rowsOf(log.rows)) } type GeneratedReach = @@ -1280,59 +1312,29 @@ describe(`multi-tab Electric over one persisted database`, () => { 60_000, ) - // Law 2 histories: tab A's stream applies v1 and v2; tab B's lagging stream - // applies v1 later. B then either catches up or closes. The checkpoint is - // host quiescence after the last delivery. - const lagging = async (bEnds: `catches up` | `closes`) => { - host = openHost() - const [a, b] = await openWarmTabs(host, 2, `eager`) - for (const tab of [a!, b!]) { - tab.deliver([insert({ id: 1, label: `v0` }), upToDate], { - handle: `h1`, - offset: `10_0`, - }) - await host.quiesce(`tab ${tab.tab} warm`) - } - a!.deliver([update({ id: 1, label: `v1` }), upToDate], { - handle: `h1`, - offset: `20_0`, - }) - await host.quiesce(`A applied v1`) - a!.deliver([update({ id: 1, label: `v2` }), upToDate], { - handle: `h1`, - offset: `30_0`, - }) - await host.quiesce(`A applied the head`) - b!.deliver([update({ id: 1, label: `v1` }), upToDate], { - handle: `h1`, - offset: `20_0`, - }) - await host.quiesce(`lagging B applied v1`) - if (bEnds === `catches up`) { - b!.deliver([update({ id: 1, label: `v2` }), upToDate], { - handle: `h1`, - offset: `30_0`, - }) - } else { - await b!.close() - } - await host.quiesce(`B ${bEnds}`) - // A's own stream applied the head, so A must show it. - expect(a!.publicRows()).toEqual([{ id: 1, label: `v2` }]) - } - - it.each([`catches up`] as const)( - `converges a tab that applied the head when the lagging tab %s`, - (bEnds) => lagging(bEnds), - 30_000, - ) - - // Known gap (#2209): nothing redelivers the head to tab A after the lagging - // tab stored v1 and closed. Ordering source writes by server position fixes - // it; move `closes` into the passing values with that change. - it.fails.each([`closes`] as const)( - `converges a tab that applied the head when the lagging tab %s`, - (bEnds) => lagging(bEnds), - 30_000, + // Known gap #2209, pinned: tab A commits v1 and v2; tab B's lagging stream + // then commits only v1, so the durable row falls back to v1, and B closes + // for good. Nothing redelivers the head to A, so Law 2's final check fails. + // Generated histories never close a tab for good, which leaves this + // history out of them; it replays through the same driver and checks. + // Remove `fails` when source writes are ordered by server position. + it.fails( + `converges an open tab to the head after a lagging tab commits an older row and closes`, + () => + runGeneratedHistory( + { + tabCount: 2, + steps: [ + { kind: `server change`, key: 1, pick: 1 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `server change`, key: 1, pick: 1 }, + { kind: `catch up`, tab: 0, pick: 0 }, + { kind: `catch up`, tab: 1, pick: 1 }, + { kind: `close`, tab: 1 }, + ], + }, + () => undefined, + ), + 60_000, ) }) diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 0f401039b5..304b771c3d 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -4903,6 +4903,11 @@ class PersistedCollectionRuntime< lifecycleGeneration: number, ): Promise { if (lifecycleGeneration !== this.lifecycleGeneration) return false + // Gap recovery is one hydration cycle from its first step. It waits for + // a reserved commit before it asks the leader or moves the stream + // position; a sync transaction that begins later straddles it. + this.throwIfReservedCommitOpen() + this.hydrationSequence++ if (this.persistence.coordinator.pullSince && this.latestRowVersion >= 0) { let pullResponse: PullSinceResponse | undefined try { @@ -4939,7 +4944,13 @@ class PersistedCollectionRuntime< ) if (pullResponse.requiresFullReload) { await this.runInHydrationScope((adapter) => - this.reloadActiveSubsetsUnsafe(adapter), + this.reloadActiveSubsetsUnsafe( + adapter, + false, + false, + false, + true, + ), ) this.publicRowVersion = Math.max( this.publicRowVersion, @@ -4950,7 +4961,13 @@ class PersistedCollectionRuntime< const deltas = pullResponse.deltas if (!deltas) { await this.runInHydrationScope((adapter) => - this.reloadActiveSubsetsUnsafe(adapter), + this.reloadActiveSubsetsUnsafe( + adapter, + false, + false, + false, + true, + ), ) this.publicRowVersion = Math.max( this.publicRowVersion, @@ -4977,6 +4994,7 @@ class PersistedCollectionRuntime< delta.collectionMetadataMutations, }, adapter, + true, ) if (lifecycleGeneration !== this.lifecycleGeneration) return } @@ -4994,7 +5012,7 @@ class PersistedCollectionRuntime< // A gap is not a schema reset. Replace the durable snapshot in one // publication so readers never see an empty intermediate Collection. await this.runInHydrationScope((adapter) => - this.reloadActiveSubsetsUnsafe(adapter), + this.reloadActiveSubsetsUnsafe(adapter, false, false, false, true), ) this.publicRowVersion = Math.max( this.publicRowVersion, @@ -5122,6 +5140,7 @@ class PersistedCollectionRuntime< private async invalidateFromCommittedTxUnsafe( txCommitted: TxCommitted, adapter: HydrationPersistenceAdapter, + continuesHydrationCycle = false, ): Promise { if (this.scopedRecovery) { this.refreshScopedSubsets() @@ -5130,7 +5149,13 @@ class PersistedCollectionRuntime< const reloadActiveSubsets = (skipUnchangedReplace = false) => this.runInHydrationScope( (scopedAdapter) => - this.reloadActiveSubsetsUnsafe(scopedAdapter, skipUnchangedReplace), + this.reloadActiveSubsetsUnsafe( + scopedAdapter, + skipUnchangedReplace, + false, + false, + continuesHydrationCycle, + ), adapter, ) @@ -6007,8 +6032,7 @@ function createWrappedSyncConfig< const findReservedTransaction = () => transactionStack.find( - (transaction) => - transaction.reservesCommitTurn === true && !transaction.internal, + (transaction) => transaction.reservesCommitTurn === true, ) runtime.setWaitForReservedCommit(() => { const transaction = findReservedTransaction() @@ -6739,9 +6763,8 @@ function createWrappedSyncConfig< } } - const openTransaction = getOpenTransaction() const needsPublicationAdmission = - openTransaction?.reservesCommitTurn === true + findReservedTransaction() !== undefined let hydrationStarted = false const raceRetainedLoad = ( work: Promise, @@ -6802,14 +6825,14 @@ function createWrappedSyncConfig< ) { return } - const currentTransaction = getOpenTransaction() + const currentTransaction = findReservedTransaction() // A sync transaction that reserved its FIFO turn commits before // subset hydration advances the generation it was built against. // A load admitted during the reservation waits here, before it // takes an apply-mutex turn, so it runs after the commit's durable // write. The runtime's deferral covers hydrations whose turn // began earlier; it cannot keep that FIFO order. - if (currentTransaction?.reservesCommitTurn) { + if (currentTransaction) { await waitForPublicationAdmission( currentTransaction, options, diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index a89cbd7f1b..2a9a4785e9 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -17599,8 +17599,11 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // - `routed subset`: a follower's demand, which the owner serves by // committing a row, broadcasting it, then replying; // - `source baseline`: the source's own `hydrateBaseline` request; - // - `collection-reset`, `full-reload`, `seq-gap`, `seq-gap pull`: peer - // notices (`seq-gap pull` recovers through pullSince); + // - `collection-reset`, `full-reload`, `seq-gap`, `seq-gap pull`, + // `seq-gap deltas`: peer notices (`seq-gap pull` recovers through a + // pullSince full reload; `seq-gap deltas` through pullSince deltas, a + // targeted row change and then a delta too large to apply in place, + // which reloads partway through the recovery); // - `targeted`: a peer change for a key that is not public, which starts // no hydration. // The commit timing places the first sync transaction against the route: @@ -17641,6 +17644,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { | `full-reload` | `seq-gap` | `seq-gap pull` + | `seq-gap deltas` | `targeted` type PersistedReadTiming = | `after read` @@ -17732,6 +17736,18 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { commitTiming === `open in the first publication` const reserves = waits || straddles || commitTiming === `open across startup` + // Gap recovery is one hydration cycle from its first step, which comes + // before it enters its hydration scope. A reservation that opens at scope + // entry therefore straddles the recovery instead of waiting for it + // (law: a transaction opened after a hydration began straddles it). Every + // other route begins its hydration inside the scope. + const recoveryStraddles = + commitTiming === `open across scope entry` && + (route === `seq-gap` || + route === `seq-gap pull` || + route === `seq-gap deltas`) + const hydrationWaits = waits && !recoveryStraddles + const hydrationStraddles = straddles || recoveryStraddles const sourceSteps = truncateReplay === `none` ? [`network`] @@ -17747,7 +17763,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // Rows the route adds durably. The owner's row for a routed subset lands // after the follower's own read, so reads never see it. const routeRows = - route === `targeted` + route === `targeted` || route === `seq-gap deltas` ? [peerRow] : route === `routed subset` ? [remoteRow] @@ -17834,12 +17850,16 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { } const startupStepCommitted = commitTiming === `open across startup` && abandonedStep === undefined - const noWaitPullSinceCalls = route === `seq-gap pull` ? 1 : 0 + const noWaitPullSinceCalls = + route === `seq-gap pull` || route === `seq-gap deltas` ? 1 : 0 // A route notice waits in the waiting timings; a follower's owner notice // arrives during a straddled reservation and waits there. Each notice // that waited may ask the leader once more; nothing else may. const waitedNotices = - (waits && (route === `full-reload` || route === `seq-gap pull`)) || + (hydrationWaits && + (route === `full-reload` || + route === `seq-gap pull` || + route === `seq-gap deltas`)) || (straddles && route === `routed subset`) ? 1 : 0 @@ -17866,6 +17886,8 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { waits, straddles, reserves, + hydrationWaits, + hydrationStraddles, sourceSteps, demandRoute, readBaselineRows, @@ -17901,8 +17923,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { } = fixtureRows const { waits, - straddles, reserves, + hydrationWaits, + hydrationStraddles, sourceSteps, demandRoute, readBaselineRows, @@ -18003,6 +18026,39 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ? undefined : () => { pullSinceCalls++ + if (route === `seq-gap deltas`) { + // A targeted delta for the row the peer stored at the skipped + // position, then a delta past the targeted-invalidation limit. + const bulkKeys = Array.from( + { length: 129 }, + (_, index) => `absent-${index}`, + ) + return Promise.resolve({ + type: `rpc:pullSince:res`, + rpcId: `gap-deltas`, + ok: true, + latestTerm: 1, + latestSeq: leaderPosition.seq, + latestRowVersion: leaderPosition.rowVersion, + requiresFullReload: false, + changedKeys: [peerRow.id], + deletedKeys: bulkKeys, + deltas: [ + { + txId: `gap-delta-row`, + latestRowVersion: leaderPosition.rowVersion - 1, + changedRows: [{ key: peerRow.id, value: peerRow }], + deletedKeys: [], + }, + { + txId: `gap-delta-bulk`, + latestRowVersion: leaderPosition.rowVersion, + changedRows: [], + deletedKeys: bulkKeys, + }, + ], + } as never) + } return Promise.resolve({ type: `rpc:pullSince:res`, rpcId: `gap-pull`, @@ -18332,8 +18388,21 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // Sequence gap, with or without pullSince. A peer commit this tab has // not observed arrives past a sequence gap and carries no rows, as a // peer's must-refetch reset commit does in a multi-tab refetch storm. - // Recovery rereads durable rows. - nextPeerPosition() + // Recovery rereads durable rows. For `seq-gap deltas` the skipped + // position holds a peer row that only pullSince reports. + const skipped = nextPeerPosition() + if (route === `seq-gap deltas`) { + await applyCommittedTx( + { kind: `eager`, collectionId: `persisted-read-precedence` }, + { + txId: `gap-row`, + term: 1, + seq: skipped.seq, + rowVersion: skipped.rowVersion, + mutations: [{ type: `insert`, key: peerRow.id, value: peerRow }], + } as never, + ) + } const position = nextPeerPosition() coordinator.emit({ type: `tx:committed`, @@ -18576,9 +18645,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { const stepIndexes = trace.flatMap((event, index) => event === `network` || event === `replacement` ? [index] : [], ) - const observedReadOrder = waits + const observedReadOrder = hydrationWaits ? `may interleave` - : straddles + : hydrationStraddles ? readIndexes.length === 0 || stepIndexes.every((step) => readIndexes[0]! < step) ? `straddled read before every step` @@ -18598,7 +18667,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { : `other` const observed = { outcomes: outcomes.map(({ status }) => status), - readsDuringReservation: straddles ? [] : readsDuringReservation, + readsDuringReservation: hydrationStraddles + ? [] + : readsDuringReservation, stepOrder: trace.filter( (event) => event === `network` || event === `replacement`, ), @@ -18629,8 +18700,10 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ), publicCursor: sourceParams.metadata!.collection.get(`cursor`), durableCursor: adapter.collectionMetadata.get(`cursor`), - ...(waits && route !== `targeted` ? { rowsWhileReserved } : {}), - ...(straddles ? { rowsWhileReserved } : {}), + ...(hydrationWaits && route !== `targeted` + ? { rowsWhileReserved } + : {}), + ...(hydrationStraddles ? { rowsWhileReserved } : {}), ...(route === `targeted` && waits ? { peerWhileReserved } : {}), ...(followUp !== `none` ? { publishedSuperseded: published.has(supersededRow.id) } @@ -18651,9 +18724,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { { length: expectedReadCount }, () => true, ), - readOrder: waits + readOrder: hydrationWaits ? `may interleave` - : straddles + : hydrationStraddles ? `straddled read before every step` : startupStepCommitted ? `after the startup step` @@ -18671,10 +18744,10 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { durableCursor: expectedCursor, // While a hydration waits, readers keep the rows from before it; a // straddled read has published the pre-source durable rows. - ...(waits && route !== `targeted` + ...(hydrationWaits && route !== `targeted` ? { rowsWhileReserved: rowsAtAdmission } : {}), - ...(straddles + ...(hydrationStraddles ? { rowsWhileReserved: byKey( exposesSourceRowsOnly ? [gateRow] : readBaselineRows, @@ -18732,10 +18805,12 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { { syncMode: `on-demand`, route: `full-reload` }, { syncMode: `on-demand`, route: `seq-gap` }, { syncMode: `on-demand`, route: `seq-gap pull` }, + { syncMode: `on-demand`, route: `seq-gap deltas` }, { syncMode: `eager`, route: `collection-reset` }, { syncMode: `eager`, route: `full-reload` }, { syncMode: `eager`, route: `seq-gap` }, { syncMode: `eager`, route: `seq-gap pull` }, + { syncMode: `eager`, route: `seq-gap deltas` }, ] as const ).flatMap(({ syncMode, route }) => ([`insert`, `update`, `delete`] as const).flatMap((operation) => @@ -18779,10 +18854,12 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { { syncMode: `on-demand`, route: `full-reload` }, { syncMode: `on-demand`, route: `seq-gap` }, { syncMode: `on-demand`, route: `seq-gap pull` }, + { syncMode: `on-demand`, route: `seq-gap deltas` }, { syncMode: `eager`, route: `collection-reset` }, { syncMode: `eager`, route: `full-reload` }, { syncMode: `eager`, route: `seq-gap` }, { syncMode: `eager`, route: `seq-gap pull` }, + { syncMode: `eager`, route: `seq-gap deltas` }, { syncMode: `eager`, route: `targeted` }, ] as const const reservationHistories = reservationRoutes.flatMap( diff --git a/packages/electric-db-collection/src/electric.ts b/packages/electric-db-collection/src/electric.ts index ee8b225147..0a5a38c98b 100644 --- a/packages/electric-db-collection/src/electric.ts +++ b/packages/electric-db-collection/src/electric.ts @@ -2624,8 +2624,9 @@ function createElectricSync>( if (!isResettingSnapshot || finishesReset) { stageResumeMetadata() } - applied = commitSourceTransaction() + // The commit consumes the transaction even if it throws. transactionStarted = false + applied = commitSourceTransaction() } else if (commitPoint === `up-to-date` && metadata) { beginSourceTransaction() stageResumeMetadata() From e9ab9b3026b4cb8ef8066ef0d910bc3f2f83b611 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 12:45:11 -0600 Subject: [PATCH 09/24] fix(sqlite-persistence): release waiting loads when a reservation is abandoned Maintainer decision on PR #2265: an abandoned reservation releases the subset loads that waited for it, as a commit does. A load admitted while the turn was reserved waited in the wrapper and rejected on abandon, while a load admitted a moment earlier waited in the runtime and ran. The persisted-reads grammar gains a timing that reserves before a subset load is admitted. Its abandoned cells failed on the parent with a rejected load. The admission wait now only resolves; a demand's own abort still rejects its load through the retained-load race, which makes the wait's separate rejection path redundant, so it is removed. The reserveCommitTurn contract now states that an abandon releases waiting hydrations, and that a source must not await a baseline it started before reserving while the reservation is open (maintainer decision on PR #2265). Co-authored-by: Isaac --- .../reserved-commit-defers-every-hydration.md | 2 + docs/contributing/oracle-coverage.md | 10 +++-- .../src/persisted.ts | 42 +++++++------------ .../tests/persisted-oracle.test.ts | 35 +++++++++++++++- packages/db/src/types.ts | 8 +++- 5 files changed, 63 insertions(+), 34 deletions(-) diff --git a/.changeset/reserved-commit-defers-every-hydration.md b/.changeset/reserved-commit-defers-every-hydration.md index fbec83c149..24a1f7499a 100644 --- a/.changeset/reserved-commit-defers-every-hydration.md +++ b/.changeset/reserved-commit-defers-every-hydration.md @@ -9,3 +9,5 @@ Keep a sync transaction that reserved its commit turn from failing when a persis A peer reset now applies in arrival order with peer commit notices, so a commit from before the reset can no longer publish a row that the reset removed. Electric now abandons its open transaction when its stream stops for good, so hydrations that wait for that transaction can run. + +An abandoned reservation now releases the subset loads that waited for it, as a commit does, so they run. Before, a load admitted while the turn was reserved rejected with `SyncTransactionAbortedError` when the source abandoned the transaction. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 1a8037b2a5..22c73f20ed 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2902,8 +2902,12 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. that starts no hydration. Gap recovery is one hydration cycle from its first step, so a reservation that opens at its scope entry straddles it. The reserving timings are - across the mutex turn, across scope entry, during the read, in the - route's first publication, and across startup. The ending is commits, + before a subset load's admission, across the mutex turn, across scope + entry, during the read, in the route's first publication, and across + startup. A load that waits for a reservation runs after it ends, + whether it commits or is abandoned (maintainer decision on PR #2265); + on `851e8d9da` the abandoned admission cells failed with a rejected + load. The ending is commits, abandoned, cleanup, an aborted demand, or the source awaiting its own baseline. The follow-up is a reset then a pre-reset commit. Observations: receipts, final rows, row metadata, cursor, status, @@ -2912,7 +2916,7 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. events before it. Every cell declares reach (ORC-015). | Evidence | Result | | --- | --- | - | Row-algebra cross and reservation cross | 649 cells pass. | + | Row-algebra cross and reservation cross | 657 cells pass. | | Gap recovery that reloads partway through, on `851e8d9da` | 28 cells fail: recovery advanced the stream position and applied a targeted delta before it deferred, so its retry skipped the reload. A gate only at recovery start still fails 15 scope-entry cells; reloads that start their own cycle inside recovery fail 61. | | `5649a07d4` | The open cells fail with `a persisted sync transaction cannot cross a hydration cycle`. Three `after read` reset follow-up cells also fail, because `main` applied a pre-reset commit before the reset set its term fence. Queueing resets with notices fixes this pre-existing defect. | | Hostile mutants | Each mutant fails its cells, and every failure is an assertion. Deferring every notice fails the targeted change. Dropping the follower reply's wait fails the routed subset. Reporting hydration start before the check fails the aborted demand. A reset reload that starts its own cycle fails the first-publication reset. Applying a reset outside the notice queue fails the reset follow-ups. Retrying a waited notice while the reservation is open fails the follow-up and routed cells (repeated pullSince). Checking only at the mutex turn fails every scope-entry cell. Gating subset hydration only fails every reset, reload, and gap cell. Requeueing a waited notice behind later notices fails 12 reset follow-up cells: the pre-reset commit is published before its reset, and a reload is lost. | diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 304b771c3d..313808c468 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -5903,42 +5903,30 @@ function createWrappedSyncConfig< cancelAdmissionWait?: () => void } const acquisitions = new Map() + // Resolves when the reservation ends (committed, abandoned, or + // discarded by cleanup; maintainer decision on PR #2265) or when the + // load's own demand aborts. The load then runs, or rejects through its + // own signal. const waitForPublicationAdmission = ( transaction: OpenSyncTransaction, options: LoadSubsetOptions, acquisition: SubsetAcquisition, - ): Promise => { - if (options.signal?.aborted) { - return Promise.reject( - options.signal.reason ?? new SyncTransactionAbortedError(), - ) - } - return new Promise((resolve, reject) => { - const waiters = - transaction.publicationAdmissionWaiters ?? - (transaction.publicationAdmissionWaiters = new Set()) - let settled = false - const settle = (action: () => void) => { - if (settled) return - settled = true + ): Promise => + new Promise((resolve) => { + const waiters = (transaction.publicationAdmissionWaiters ??= + new Set()) + const settle = () => { waiters.delete(waiter) - options.signal?.removeEventListener(`abort`, abort) + options.signal?.removeEventListener(`abort`, settle) acquisition.cancelAdmissionWait = undefined - action() + resolve() } - const waiter = { - resolve: () => settle(resolve), - reject: (error: unknown) => settle(() => reject(error)), - } - const abort = () => - waiter.reject( - options.signal?.reason ?? new SyncTransactionAbortedError(), - ) + const waiter = { resolve: settle, reject: settle } + if (options.signal?.aborted) return settle() waiters.add(waiter) - options.signal?.addEventListener(`abort`, abort, { once: true }) - acquisition.cancelAdmissionWait = waiter.resolve + options.signal?.addEventListener(`abort`, settle, { once: true }) + acquisition.cancelAdmissionWait = settle }) - } const getTerminalFailure = () => runtime.getCurrentTerminalFailure() const reportUpstreamError = (error: unknown) => { if (persistedReadiness?.getSnapshot().status !== `loading`) { diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index 2a9a4785e9..dfcd2fe3ed 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -17609,6 +17609,10 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // The commit timing places the first sync transaction against the route: // - `after read`: it begins after the read is published; // - `queued`: it begins and commits while the read waits; + // - `open before admission`: it begins and reserves before the route's + // subset load is admitted, so the load waits in the wrapper's admission + // path. Whether the reservation commits or is abandoned, the load runs + // after it ends (maintainer decision on PR #2265); // - `open across mutex turn`: it begins and reserves while the read waits // behind the gate, before the read's turn; // - `open across scope entry`: it reserves after the read's turn began and @@ -17649,6 +17653,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { type PersistedReadTiming = | `after read` | `queued` + | `open before admission` | `open across mutex turn` | `open across scope entry` | `open during the read` @@ -17672,6 +17677,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { type PersistedReadReach = | `route performed a persisted read` | `reservation opened while the read was pending` + | `reservation opened before the route was admitted` | `reservation opened while a read was in progress` | `reservation opened in the route's first publication` | `reservation opened before startup settled` @@ -17728,6 +17734,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { remoteRow, } = fixtureRows const waits = + commitTiming === `open before admission` || commitTiming === `open across mutex turn` || commitTiming === `open across scope entry` // The transaction straddles a hydration that already began. @@ -18185,9 +18192,12 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { followUp === `none`) ? [] : ([`route performed a persisted read`] as const)), - ...(waits + ...(waits && commitTiming !== `open before admission` ? ([`reservation opened while the read was pending`] as const) : []), + ...(commitTiming === `open before admission` + ? ([`reservation opened before the route was admitted`] as const) + : []), ...(commitTiming === `open during the read` ? ([`reservation opened while a read was in progress`] as const) : []), @@ -18350,6 +18360,10 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { stripVirtualProps(row), ]), ) + if (commitTiming === `open before admission`) { + beginReservedSourceStep() + reach.reach(`reservation opened before the route was admitted`) + } routeAdmitted = true let ownBaselineRejection: string | undefined if ( @@ -18507,7 +18521,14 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { endReservation() remainingSteps = sourceSteps.slice(1) } else { - if (commitTiming === `open across mutex turn`) { + if (commitTiming === `open before admission`) { + // The reservation opened before the route was admitted. + releasePersistence.resolve() + await atPersistedOracleCheckpoint( + pendingOperations[0]!, + `predecessor applied while the reservation is open`, + ) + } else if (commitTiming === `open across mutex turn`) { // The read was admitted before this reservation. beginReservedSourceStep() if (!readPublished) { @@ -18867,6 +18888,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ( [ `after read`, + `open before admission`, `open across mutex turn`, `open across scope entry`, `open during the read`, @@ -18874,6 +18896,14 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { `open across startup`, ] as const ).flatMap((commitTiming) => { + // Only a subset load is admitted through the wrapper's admission. + if ( + commitTiming === `open before admission` && + route !== `subset` && + route !== `complete subset` + ) { + return [] + } if (commitTiming === `open during the read` && route === `targeted`) return [] // Only a route whose hydration publishes rows reaches a publication. @@ -18887,6 +18917,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { return [] } const waits = + commitTiming === `open before admission` || commitTiming === `open across mutex turn` || commitTiming === `open across scope entry` const endings: Array = diff --git a/packages/db/src/types.ts b/packages/db/src/types.ts index 459be2a92e..c02d349b3d 100644 --- a/packages/db/src/types.ts +++ b/packages/db/src/types.ts @@ -588,8 +588,12 @@ export type SyncPersistenceCapabilityV1< * a reload or reset from a peer notice, and sequence-gap recovery. A subset * load admitted while the turn is reserved also runs after the commit's * durable write. The source must end the transaction: commit it, or abandon - * it by committing with an aborted signal. A source that stops for good - * abandons it, or every later hydration keeps waiting. + * it by committing with an aborted signal. An abandon releases the waiting + * hydrations as a commit does, so they run. A source that stops for good + * abandons it, or every later hydration keeps waiting. The source must not + * await a `hydrateBaseline` that it started before reserving while the + * reservation is open: that baseline waits for the commit. Calling + * `hydrateBaseline` while the reservation is open rejects. */ readonly reserveCommitTurn: () => void readonly scanPersistedRows: ( From fb7420840cc2dd59e6573c85de5b222736978b22 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 13:20:48 -0600 Subject: [PATCH 10/24] test(browser-sqlite): run the generated multi-tab campaigns through the shared oracle config The oracle lint rule rejects a one-run replay with an environment seed. Register browser.multi-tab-reserved-history and run the fixed seed through oracleRuns and the random or replayed campaign through oraclePropertyOptions, as the other generated properties do. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 5 +- .../tests/multi-tab-electric-oracle.test.ts | 111 ++++++++---------- packages/db/tests/oracle-config.ts | 1 + 3 files changed, 54 insertions(+), 63 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 22c73f20ed..d21a28df5f 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2962,8 +2962,9 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. Generated histories in the same file interleave server changes, deploys, partial and complete deliveries, every stop except resume invalidation, restarts, and reloads across two or three eager tabs - (fixed seed 952051961 and a random seed, 60 runs each; - `TANSTACK_DB_MULTI_TAB_ORACLE_SEED` and `_PATH` replay one). The + (fixed seed 952051961 and a random seed, 60 runs each; the shared + `TANSTACK_DB_ORACLE_SEED`, `_PATH`, and `_PROPERTY= + browser.multi-tab-reserved-history` replay one). The model folds committed batches into the expected durable rows. After every step it judges the durable rows, every Collection status, and the rows of each tab with no open transaction; after every tab catches diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 315afdaa9e..0d32a05f2f 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -86,6 +86,11 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { fc } from '@fast-check/vitest' import { createCollection } from '@tanstack/db' import { createOracleReach } from '../../db/tests/oracle-reach.js' +import { + oraclePropertyOptions, + oracleReplayRequested, + oracleRuns, +} from '../../db/tests/oracle-config' import { persistedCollectionOptions } from '../../db-sqlite-persistence-core/src' import { electricCollectionOptions } from '../../electric-db-collection/src/electric' import { BrowserCollectionCoordinator } from '../src/browser-coordinator' @@ -410,45 +415,14 @@ async function openWarmTabs(host: Host, count: number, syncMode: SyncMode) { return tabs } -// Generated-history campaigns (ORC-007): a fixed seed and a random seed with -// the same property and budget. TANSTACK_DB_MULTI_TAB_ORACLE_SEED and -// TANSTACK_DB_MULTI_TAB_ORACLE_PATH replay one recorded counterexample; -// TANSTACK_DB_MULTI_TAB_ORACLE_RUNS sets the budget. +// Generated-history campaigns (ORC-007): a fixed seed and a random seed +// with the same property and budget. TANSTACK_DB_ORACLE_SEED with +// TANSTACK_DB_ORACLE_PATH and TANSTACK_DB_ORACLE_PROPERTY= +// browser.multi-tab-reserved-history replays one recorded counterexample; +// TANSTACK_DB_ORACLE_RUNS_MULTIPLIER scales the budget. +const GENERATED_PROPERTY = `browser.multi-tab-reserved-history` const GENERATED_FIXED_SEED = 952051961 -function generatedCampaigns( - environment: Record = process.env, -): Array<{ - mode: `fixed` | `random` | `replay` - parameters: { numRuns: number; seed?: number; path?: string } -}> { - const runText = environment.TANSTACK_DB_MULTI_TAB_ORACLE_RUNS ?? `60` - const numRuns = Number(runText) - if (!/^\d+$/.test(runText) || numRuns < 1) { - throw new Error(`Invalid TANSTACK_DB_MULTI_TAB_ORACLE_RUNS`) - } - const seedText = environment.TANSTACK_DB_MULTI_TAB_ORACLE_SEED - const path = environment.TANSTACK_DB_MULTI_TAB_ORACLE_PATH - if (seedText === undefined && path === undefined) { - return [ - { mode: `fixed`, parameters: { numRuns, seed: GENERATED_FIXED_SEED } }, - { mode: `random`, parameters: { numRuns } }, - ] - } - const seed = Number(seedText) - if (seedText === undefined || !/^-?\d+$/.test(seedText)) { - throw new Error(`Invalid TANSTACK_DB_MULTI_TAB_ORACLE_SEED`) - } - if (path !== undefined && !/^\d+(?::\d+)*$/.test(path)) { - throw new Error(`TANSTACK_DB_MULTI_TAB_ORACLE_PATH must be a shrink path`) - } - return [ - { - mode: `replay`, - parameters: - path === undefined ? { numRuns, seed } : { numRuns: 1, seed, path }, - }, - ] -} +const GENERATED_RUNS = 60 async function waitForStream(tab: Tab, count: number) { for (let task = 0; task < 400 && tab.streamCount() < count; task++) { @@ -1261,29 +1235,44 @@ describe(`multi-tab Electric over one persisted database`, () => { ]) let generatedReach = createGeneratedReach() - it.each(generatedCampaigns())( - `keeps reserved commits lawful across generated multi-tab histories ($mode campaign)`, - async ({ parameters }) => { - generatedReach = createGeneratedReach() - await fc.assert( - fc.asyncProperty(generatedHistory, async (history) => { - try { - // fc.pre discards a history that reaches the known gap. - await runGeneratedHistory(history, () => fc.pre(false)) - } finally { - await host?.close() - host = undefined - streams.length = 0 - } - }), - parameters, - ) - console.info( - `multi-tab generated reach`, - Object.fromEntries(generatedReach.counts()), - ) - generatedReach.assertReached() - }, + const runGeneratedCampaign = async ( + parameters: ReturnType, + ) => { + generatedReach = createGeneratedReach() + await fc.assert( + fc.asyncProperty(generatedHistory, async (history) => { + try { + // fc.pre discards a history that reaches the known gap. + await runGeneratedHistory(history, () => fc.pre(false)) + } finally { + await host?.close() + host = undefined + streams.length = 0 + } + }), + parameters, + ) + console.info( + `multi-tab generated reach`, + Object.fromEntries(generatedReach.counts()), + ) + generatedReach.assertReached() + } + it.skipIf(oracleReplayRequested())( + `keeps reserved commits lawful across generated multi-tab histories (fixed seed)`, + () => + runGeneratedCampaign({ + numRuns: oracleRuns(GENERATED_RUNS), + seed: GENERATED_FIXED_SEED, + }), + 600_000, + ) + it( + `keeps reserved commits lawful across generated multi-tab histories (random or replayed)`, + () => + runGeneratedCampaign( + oraclePropertyOptions(GENERATED_RUNS, GENERATED_PROPERTY), + ), 600_000, ) diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index bcf9de7f68..c381ca03a6 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -73,6 +73,7 @@ const staticOracleProperties = [ `sqlite-resume.managed-claim-read`, `sqlite-resume.retired-storage`, `sqlite-reload.class`, + `browser.multi-tab-reserved-history`, `sqlite-reload.class.electron`, `persistence.generation-notice`, `persistence.follower-remote-demand`, From 2047f42819ac88f574064bd8a2c33d9c7abfdbd9 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 13:37:43 -0600 Subject: [PATCH 11/24] test(oracles): name the owner of the generated multi-tab property Co-authored-by: Isaac --- packages/db/tests/oracle-replay-manifest.ts | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index 1304ba6a95..80e73fa4c9 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -379,6 +379,11 @@ const ownerGroups: ReadonlyArray< `sqlite-reload`, `class`, ], + [ + `browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts`, + `browser`, + `multi-tab-reserved-history`, + ], [ `electron-db-sqlite-persistence/tests/reload-class-oracle.test.ts`, `sqlite-reload`, From 2ff2e2fd49e4501190c7d2b92952ca466808737d Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 15:41:16 -0600 Subject: [PATCH 12/24] fix(sqlite-persistence): let a peer reset abandon an older reservation Maintainer decisions on PR #2265, from the third review: - A peer reset abandons a reserved transaction that began before it. Its writes are dropped and its commit rejects with SyncTransactionAbortedError, so no pre-reset rows reach the new epoch; the reset does not wait. Before, the reset waited and the stale commit landed, while a peer's pre-reset commit was fenced. The grammar failed 53 cells on the parent; abandoning only at the reset's turn fails 6 follow-up cells, only on arrival 40. - Gap recovery is a hydration cycle only from the point where it will change rows: after pullSince answers, or at its reload without pullSince. An unreserved transaction may commit during pullSince, and a reservation opened then makes recovery wait. Two new pullSince timings failed 12 cells on the parent; dropping the check after pullSince fails 8. - No fairness rule for hydrations released by a commit; recorded as a known limit. The model now folds the expected rows, metadata, and cursor over the source steps that apply, which a reset that abandons a first step needs. The follower's reply waits only for a head notice that waits, and gap recovery names its in-cycle reload once. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 26 +- .../src/persisted.ts | 99 +++---- .../tests/persisted-oracle.test.ts | 249 ++++++++++++------ ...erved-commit-sqlite-adapter-oracle.test.ts | 15 +- packages/db/src/types.ts | 6 +- 5 files changed, 266 insertions(+), 129 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 9dabcac25a..b6c55fc7dd 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2975,10 +2975,16 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. reset, full reload, sequence gap with or without pullSince, sequence gap whose pullSince deltas reload partway through (a targeted delta, then one past the targeted-invalidation limit), and a targeted change - that starts no hydration. Gap recovery is one hydration cycle from its - first step, so a reservation that opens at its scope entry straddles - it. The reserving timings are - before a subset load's admission, across the mutex turn, across scope + that starts no hydration. Gap recovery is a hydration cycle only from + the point where it will change rows (maintainer decision on PR #2265): + after pullSince answers, or at its reload without pullSince. A + reservation opened after that point straddles it; one opened during + pullSince makes it wait, and a transaction that did not reserve may + commit during pullSince. A peer reset abandons a reserved transaction + that began before it (maintainer decision on PR #2265): the commit + rejects and the reset does not wait. The reserving timings are + before a subset load's admission, during gap recovery's pullSince + (with or without a reservation), across the mutex turn, across scope entry, during the read, in the route's first publication, and across startup. A load that waits for a reservation runs after it ends, whether it commits or is abandoned (maintainer decision on PR #2265); @@ -2992,10 +2998,17 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. events before it. Every cell declares reach (ORC-015). | Evidence | Result | | --- | --- | - | Row-algebra cross and reservation cross | 657 cells pass. | + | Row-algebra cross and reservation cross | 677 cells pass. | + | Peer reset against an older reservation, on `2047f4281` | 53 cells fail: the reset waited and the stale commit landed in the new epoch. Abandoning only at the reset's turn fails 6 follow-up cells; abandoning only on arrival fails 40 route cells. | + | Gap recovery's cycle start, on `2047f4281` | 12 pullSince cells fail: an unreserved commit during pullSince rejected, and a reservation opened then straddled. Dropping the reservation check after pullSince fails 8. | | Gap recovery that reloads partway through, on `851e8d9da` | 28 cells fail: recovery advanced the stream position and applied a targeted delta before it deferred, so its retry skipped the reload. A gate only at recovery start still fails 15 scope-entry cells; reloads that start their own cycle inside recovery fail 61. | | `5649a07d4` | The open cells fail with `a persisted sync transaction cannot cross a hydration cycle`. Three `after read` reset follow-up cells also fail, because `main` applied a pre-reset commit before the reset set its term fence. Queueing resets with notices fixes this pre-existing defect. | | Hostile mutants | Each mutant fails its cells, and every failure is an assertion. Deferring every notice fails the targeted change. Dropping the follower reply's wait fails the routed subset. Reporting hydration start before the check fails the aborted demand. A reset reload that starts its own cycle fails the first-publication reset. Applying a reset outside the notice queue fails the reset follow-ups. Retrying a waited notice while the reservation is open fails the follow-up and routed cells (repeated pullSince). Checking only at the mutex turn fails every scope-entry cell. Gating subset hydration only fails every reset, reload, and gap cell. Requeueing a waited notice behind later notices fails 12 reset follow-up cells: the pre-reset commit is published before its reset, and a reload is lost. | + Known limit (maintainer decision on PR #2265): no fairness rule. A + hydration released by a commit waits again for a reservation that + opens before its next mutex turn, so a source that reserves back to + back can delay it; Electric reserves across batches only during an + initial sync or refetch. Not covered: `forceReloadSubset`, which only a sync-absent Collection has, and which therefore never meets a reservation; startup hydration, which source entry follows (an `open across startup` observation); @@ -3008,6 +3021,9 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. or the shared scheduler holding the scope after its turn) and two endings (commits, abandoned). It runs on `SQLiteCorePersistenceAdapter`. It also judges that a subset load that waited settles successfully. + Its peer-reset cells judge that the reset abandons the reservation: + the commit rejects and row-1 keeps its durable value (both committing + peer-reset cells fail on `2047f4281`). Its 10 cells pass. Every committing cell fails on `5649a07d4`. A mutant that rejects the waited load after the reservation fails the 4 subset cells; before the load's outcome was judged, it passed. A peer diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 313808c468..98eca29aab 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -1566,6 +1566,7 @@ class PersistedCollectionRuntime< private activeHydrationContext: { suppliedRowKeys: Set } | undefined private hydrationSequence = 0 private waitForReservedCommit: WaitForReservedCommit | undefined + private abandonReservedCommit: (() => void) | undefined // An unscheduled startup read may finish after a coordinator reset publishes. private resetSequence = 0 private readonly pendingRemoteSubsetEnsures = new Map< @@ -1921,8 +1922,12 @@ class PersistedCollectionRuntime< this.restartSourceAfterScopedRecovery = restart ?? null } - setWaitForReservedCommit(wait: WaitForReservedCommit): void { + setReservedCommitControls( + wait: WaitForReservedCommit, + abandon: () => void, + ): void { this.waitForReservedCommit = wait + this.abandonReservedCommit = abandon } // Every hydration start calls this before it changes state. The check runs @@ -2032,12 +2037,12 @@ class PersistedCollectionRuntime< } await this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) // A broadcast whose reload waits for a reserved commit has not applied - // yet, so the load waits with it. + // yet, so the load waits with it. Notices behind a head that does not + // wait arrived after this flush and are not the owner's. for (;;) { - const closed = - this.queuedTxCommitted.length > 0 - ? this.waitForReservedCommit?.() - : undefined + const closed = this.queuedTxCommitted[0]?.waitsForReservedCommit + ? this.waitForReservedCommit?.() + : undefined if (!closed) break await closed await this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) @@ -4720,6 +4725,9 @@ class PersistedCollectionRuntime< } if (isCollectionResetPayload(payload)) { + // A reservation built before the reset cannot commit into it + // (maintainer decision on PR #2265). + this.abandonReservedCommit?.() // A reset queues with peer commits, so a commit that arrived after it // applies after it, behind its term fence. this.queuedTxCommitted.push({ @@ -4903,11 +4911,15 @@ class PersistedCollectionRuntime< lifecycleGeneration: number, ): Promise { if (lifecycleGeneration !== this.lifecycleGeneration) return false - // Gap recovery is one hydration cycle from its first step. It waits for - // a reserved commit before it asks the leader or moves the stream - // position; a sync transaction that begins later straddles it. + // Gap recovery waits for a reserved commit before it asks the leader. It + // is a hydration cycle only from the point where it will change rows + // (maintainer decision on PR #2265): after pullSince answers, and before + // it moves the stream position. A sync transaction that begins later + // straddles it; one that ends before then never crosses it. this.throwIfReservedCommitOpen() - this.hydrationSequence++ + // Every reload in this recovery continues its hydration cycle. + const reloadInCycle = (adapter: HydrationPersistenceAdapter) => + this.reloadActiveSubsetsUnsafe(adapter, false, false, false, true) if (this.persistence.coordinator.pullSince && this.latestRowVersion >= 0) { let pullResponse: PullSinceResponse | undefined try { @@ -4937,21 +4949,15 @@ class PersistedCollectionRuntime< if (lifecycleGeneration !== this.lifecycleGeneration) return false if (pullResponse.ok) { + this.throwIfReservedCommitOpen() + this.hydrationSequence++ this.observeStreamPosition( pullResponse.latestTerm, pullResponse.latestSeq, pullResponse.latestRowVersion, ) if (pullResponse.requiresFullReload) { - await this.runInHydrationScope((adapter) => - this.reloadActiveSubsetsUnsafe( - adapter, - false, - false, - false, - true, - ), - ) + await this.runInHydrationScope((adapter) => reloadInCycle(adapter)) this.publicRowVersion = Math.max( this.publicRowVersion, pullResponse.latestRowVersion, @@ -4960,15 +4966,7 @@ class PersistedCollectionRuntime< } const deltas = pullResponse.deltas if (!deltas) { - await this.runInHydrationScope((adapter) => - this.reloadActiveSubsetsUnsafe( - adapter, - false, - false, - false, - true, - ), - ) + await this.runInHydrationScope((adapter) => reloadInCycle(adapter)) this.publicRowVersion = Math.max( this.publicRowVersion, pullResponse.latestRowVersion, @@ -5012,7 +5010,7 @@ class PersistedCollectionRuntime< // A gap is not a schema reset. Replace the durable snapshot in one // publication so readers never see an empty intermediate Collection. await this.runInHydrationScope((adapter) => - this.reloadActiveSubsetsUnsafe(adapter, false, false, false, true), + this.reloadActiveSubsetsUnsafe(adapter), ) this.publicRowVersion = Math.max( this.publicRowVersion, @@ -5083,10 +5081,10 @@ class PersistedCollectionRuntime< return } // The reset is one hydration cycle from here, before its eager truncate - // publishes. A sync transaction that begins during it straddles the - // hydration instead of crossing it, so readers never wait on a truncated - // Collection. - this.throwIfReservedCommitOpen() + // publishes. It abandons a reservation that opened before it, instead of + // waiting for it; a sync transaction that begins during the reset + // straddles it, so readers never wait on a truncated Collection. + this.abandonReservedCommit?.() this.hydrationSequence++ this.hydratingGeneration = lifecycleGeneration try { @@ -6022,18 +6020,31 @@ function createWrappedSyncConfig< transactionStack.find( (transaction) => transaction.reservesCommitTurn === true, ) - runtime.setWaitForReservedCommit(() => { - const transaction = findReservedTransaction() - if (!transaction) return undefined - // Admission waiters settle when the transaction commits, aborts, or - // cleanup discards it. Every outcome closes the reservation. - return new Promise((resolve) => { - ;(transaction.publicationAdmissionWaiters ??= new Set()).add({ - resolve, - reject: () => resolve(), + runtime.setReservedCommitControls( + () => { + const transaction = findReservedTransaction() + if (!transaction) return undefined + // Admission waiters settle when the transaction commits, aborts, or + // cleanup discards it. Every outcome closes the reservation. + return new Promise((resolve) => { + ;(transaction.publicationAdmissionWaiters ??= new Set()).add({ + resolve, + reject: () => resolve(), + }) }) - }) - }) + }, + // A peer reset abandons a reservation built before it: later writes + // are dropped, the commit rejects, and its waiters run. + () => { + const transaction = findReservedTransaction() + if (!transaction) return + transaction.reservesCommitTurn = false + transaction.terminalFailure ??= { + error: new SyncTransactionAbortedError(), + } + settlePublicationAdmissionWaiters(transaction) + }, + ) const persistenceCapability: SyncPersistenceCapabilityV1 = { protocol: SYNC_PERSISTENCE_PROTOCOL, diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index c3a2915804..4efb97a54d 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -17608,6 +17608,13 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // subset load is admitted, so the load waits in the wrapper's admission // path. Whether the reservation commits or is abandoned, the load runs // after it ends (maintainer decision on PR #2265); + // - `unreserved across pullSince`: it begins without reserving before a + // gap notice and commits while recovery waits for pullSince. Recovery + // is a hydration cycle only from the point where it will change rows, so + // the commit is accepted (maintainer decision on PR #2265); + // - `open during pullSince`: it begins and reserves while recovery waits + // for pullSince, before recovery's cycle begins, so recovery waits for + // its commit and asks the leader again; // - `open across mutex turn`: it begins and reserves while the read waits // behind the gate, before the read's turn; // - `open across scope entry`: it reserves after the read's turn began and @@ -17649,6 +17656,8 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { | `after read` | `queued` | `open before admission` + | `unreserved across pullSince` + | `open during pullSince` | `open across mutex turn` | `open across scope entry` | `open during the read` @@ -17673,6 +17682,8 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { | `route performed a persisted read` | `reservation opened while the read was pending` | `reservation opened before the route was admitted` + | `unreserved transaction committed while recovery waited for pullSince` + | `reservation opened while recovery waited for pullSince` | `reservation opened while a read was in progress` | `reservation opened in the route's first publication` | `reservation opened before startup settled` @@ -17730,6 +17741,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { } = fixtureRows const waits = commitTiming === `open before admission` || + commitTiming === `open during pullSince` || commitTiming === `open across mutex turn` || commitTiming === `open across scope entry` // The transaction straddles a hydration that already began. @@ -17738,18 +17750,26 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { commitTiming === `open in the first publication` const reserves = waits || straddles || commitTiming === `open across startup` - // Gap recovery is one hydration cycle from its first step, which comes - // before it enters its hydration scope. A reservation that opens at scope - // entry therefore straddles the recovery instead of waiting for it - // (law: a transaction opened after a hydration began straddles it). Every - // other route begins its hydration inside the scope. + // Gap recovery is a hydration cycle only from the point where it will + // change rows (maintainer decision on PR #2265). With pullSince that is + // after the leader answers, before recovery enters its hydration scope, + // so a reservation that opens at scope entry straddles the recovery. + // Without pullSince it is the reload inside the scope, so such a + // reservation comes first and the recovery waits. Every other route + // begins its hydration inside the scope. const recoveryStraddles = commitTiming === `open across scope entry` && - (route === `seq-gap` || - route === `seq-gap pull` || - route === `seq-gap deltas`) - const hydrationWaits = waits && !recoveryStraddles - const hydrationStraddles = straddles || recoveryStraddles + (route === `seq-gap pull` || route === `seq-gap deltas`) + // A peer reset abandons a reserved transaction that opened before the + // reset's hydration began: the route reset when the reservation opened + // before its turn, or a follow-up reset delivered while the reservation + // is open (maintainer decision on PR #2265). Its commit rejects, its + // first step applies nothing, and the reset runs without waiting. + const resetAbandons = + waits && (route === `collection-reset` || followUp !== `none`) + const hydrationWaits = waits && !recoveryStraddles && !resetAbandons + const hydrationStraddles = straddles || recoveryStraddles || resetAbandons + const noticeWaited = waits && !recoveryStraddles const sourceSteps = truncateReplay === `none` ? [`network`] @@ -17778,58 +17798,61 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ...routeRows, ].filter((row) => !lateRowIds.has(row.id)) // Steps that apply: an abandoned reservation applies none of its writes. - const appliedSteps = - ending === `abandoned` ? sourceSteps.slice(1) : sourceSteps - const abandonedStep = ending === `abandoned` ? sourceSteps[0] : undefined - - // Independent snapshot algebra: a later replacement discards everything - // before it. Otherwise the baseline survives only when there is no - // replacement, and the newer change decides the shared key. Complete row - // values and explicit metadata avoid the separate partial-update contract. - // Source steps are input history, never observations from production. + const firstStepAbandoned = ending === `abandoned` || resetAbandons + const appliedSteps = firstStepAbandoned ? sourceSteps.slice(1) : sourceSteps + const abandonedStep = firstStepAbandoned ? sourceSteps[0] : undefined + + // Independent snapshot algebra over the source steps that apply, in + // order. A replacement discards every earlier row and leaves only its + // own; a network step writes or deletes the shared key. An abandoned step + // applies nothing. Complete row values and explicit metadata avoid the + // separate partial-update contract. Source steps are input history, never + // observations from production. const baselineRows = [storedRow, storedOnly, gateRow, ...routeRows] - const expectedRows = - abandonedStep !== undefined - ? baselineRows - : truncateReplay === `after` - ? [replacementRow] - : [ - ...(truncateReplay === `before` - ? [replacementRow] - : [storedOnly, gateRow, ...routeRows]), - ...(operation === `delete` ? [] : [networkRow]), - ] - // On-demand without demand exposes the source's own rows; an aborted - // demand loads nothing. Every other route exposes the durable rows. - // Uncertified on-demand cache rows stay quarantined after a baseline - // hydration (electric.ts: "loaded only through fresh subset snapshots"). + const foldSteps = (initialRows: Array) => { + const rows = new Map( + initialRows.map((row) => [row.id, row]), + ) + for (const step of appliedSteps) { + if (step === `replacement`) { + rows.clear() + rows.set(replacementRow.id, replacementRow) + } else if (operation === `delete`) rows.delete(`shared`) + else rows.set(networkRow.id, networkRow) + } + return [...rows.values()] + } + let durableSharedMetadata: unknown = { owner: `persisted` } + let expectedCursor = `persisted` as string + for (const step of appliedSteps) { + durableSharedMetadata = + step === `replacement` || operation === `delete` + ? undefined + : { owner: `network` } + expectedCursor = step + } + const expectedRows = foldSteps(baselineRows) + // On-demand without demand exposes the source's own rows, folded from its + // earlier row; an aborted demand loads nothing. Every other route exposes + // the durable rows. Uncertified on-demand cache rows stay quarantined + // after a baseline hydration (electric.ts: "loaded only through fresh + // subset snapshots"). const exposesSourceRowsOnly = noDemandRead || route === `source baseline` || ending === `demand aborted` - const expectedPublicRows = - exposesSourceRowsOnly && truncateReplay === `none` - ? [ - gateRow, - ...(abandonedStep !== undefined || operation === `delete` - ? [] - : [networkRow]), - ] - : expectedRows - const sharedMetadata = - abandonedStep !== undefined - ? exposesSourceRowsOnly - ? { public: undefined, durable: { owner: `persisted` } } - : { public: { owner: `persisted` }, durable: { owner: `persisted` } } - : truncateReplay !== `after` && operation !== `delete` - ? { public: { owner: `network` }, durable: { owner: `network` } } - : { public: undefined, durable: undefined } - const replacementMetadata = - truncateReplay === `none` ? undefined : { owner: `replacement` } - const expectedCursor = - abandonedStep !== undefined - ? `persisted` - : truncateReplay === `after` - ? `replacement` - : `network` + const expectedPublicRows = exposesSourceRowsOnly + ? foldSteps([gateRow]) + : expectedRows + // The persisted shared metadata is public only where the durable rows are. + const sharedMetadata = { + public: + exposesSourceRowsOnly && appliedSteps.length === 0 + ? undefined + : durableSharedMetadata, + durable: durableSharedMetadata, + } + const replacementMetadata = appliedSteps.includes(`replacement`) + ? { owner: `replacement` } + : undefined const readsSomething = !noDemandRead && route !== `targeted` && @@ -17850,6 +17873,10 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { else rows.set(networkRow.id, networkRow) } else if (event === `owner commit`) rows.set(remoteRow.id, remoteRow) } + // The first step committed before the route's reads inside startup. A + // commit made while gap recovery waits for pullSince is accepted, but its + // durable write waits behind recovery's apply-mutex turn, so recovery's + // reads still come first. const startupStepCommitted = commitTiming === `open across startup` && abandonedStep === undefined const noWaitPullSinceCalls = @@ -17858,7 +17885,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // arrives during a straddled reservation and waits there. Each notice // that waited may ask the leader once more; nothing else may. const waitedNotices = - (hydrationWaits && + (noticeWaited && (route === `full-reload` || route === `seq-gap pull` || route === `seq-gap deltas`)) || @@ -17889,6 +17916,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { straddles, reserves, hydrationWaits, + resetAbandons, hydrationStraddles, sourceSteps, demandRoute, @@ -17927,6 +17955,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { waits, reserves, hydrationWaits, + resetAbandons, hydrationStraddles, sourceSteps, demandRoute, @@ -18023,11 +18052,29 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // Every route but `seq-gap` has a leader that answers pullSince with a // full reload. let pullSinceCalls = 0 + // Holds the next pullSince request open for the pullSince timings. + let pullSinceHold: + | { + entered: ReturnType + release: ReturnType + } + | undefined = + commitTiming === `unreserved across pullSince` || + commitTiming === `open during pullSince` + ? { entered: createEventGate(), release: createEventGate() } + : undefined + const pullHold = pullSinceHold coordinator.pullSince = route === `seq-gap` ? undefined - : () => { + : async () => { pullSinceCalls++ + const hold = pullSinceHold + pullSinceHold = undefined + if (hold) { + hold.entered.resolve() + await hold.release.promise + } if (route === `seq-gap deltas`) { // A targeted delta for the row the peer stored at the skipped // position, then a delta past the targeted-invalidation limit. @@ -18187,12 +18234,24 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { followUp === `none`) ? [] : ([`route performed a persisted read`] as const)), - ...(waits && commitTiming !== `open before admission` + ...(waits && + commitTiming !== `open before admission` && + commitTiming !== `open during pullSince` ? ([`reservation opened while the read was pending`] as const) : []), ...(commitTiming === `open before admission` ? ([`reservation opened before the route was admitted`] as const) : []), + ...(commitTiming === `unreserved across pullSince` + ? ([ + `unreserved transaction committed while recovery waited for pullSince`, + ] as const) + : []), + ...(commitTiming === `open during pullSince` + ? ([ + `reservation opened while recovery waited for pullSince`, + ] as const) + : []), ...(commitTiming === `open during the read` ? ([`reservation opened while a read was in progress`] as const) : []), @@ -18259,17 +18318,14 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { pendingOperations.push(pending) expectedOutcomes.push(expected) } - const commitSourceTransaction = (signal?: AbortSignal) => { + const commitSourceTransaction = ( + signal?: AbortSignal, + expected: `fulfilled` | `rejected` = signal ? `rejected` : `fulfilled`, + ) => { try { - retainOperation( - sourceParams.commit(signal), - signal ? `rejected` : `fulfilled`, - ) + retainOperation(sourceParams.commit(signal), expected) } catch (error) { - retainOperation( - Promise.reject(error), - signal ? `rejected` : `fulfilled`, - ) + retainOperation(Promise.reject(error), expected) } } const endReservation = () => { @@ -18278,7 +18334,13 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { const cancelled = new AbortController() cancelled.abort() commitSourceTransaction(cancelled.signal) - } else commitSourceTransaction() + } else { + // A reset that abandoned the reservation rejects its commit. + commitSourceTransaction( + undefined, + resetAbandons ? `rejected` : `fulfilled`, + ) + } } const published = new Set() let hasPrimaryFailure = false @@ -18355,6 +18417,10 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { stripVirtualProps(row), ]), ) + if (commitTiming === `unreserved across pullSince`) { + sourceParams.begin() + writeSourceStep(sourceSteps[0]!) + } if (commitTiming === `open before admission`) { beginReservedSourceStep() reach.reach(`reservation opened before the route was admitted`) @@ -18475,6 +18541,19 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { } else await persistenceQuiesced(`targeted change applied`) deliverFollowUp() await persistenceQuiesced(`follow-up applied`) + } else if (commitTiming === `unreserved across pullSince`) { + releasePersistence.resolve() + await atPersistedOracleCheckpoint( + pullHold!.entered.promise, + `recovery waits for pullSince`, + ) + commitSourceTransaction() + reach.reach( + `unreserved transaction committed while recovery waited for pullSince`, + ) + pullHold!.release.resolve() + await persistenceQuiesced(`recovery finished`) + remainingSteps = sourceSteps.slice(1) } else if (commitTiming === `queued`) { // The sync transaction commits while the read waits (below). } else if (commitTiming === `open in the first publication`) { @@ -18516,7 +18595,16 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { endReservation() remainingSteps = sourceSteps.slice(1) } else { - if (commitTiming === `open before admission`) { + if (commitTiming === `open during pullSince`) { + releasePersistence.resolve() + await atPersistedOracleCheckpoint( + pullHold!.entered.promise, + `recovery waits for pullSince`, + ) + beginReservedSourceStep() + reach.reach(`reservation opened while recovery waited for pullSince`) + pullHold!.release.resolve() + } else if (commitTiming === `open before admission`) { // The reservation opened before the route was admitted. releasePersistence.resolve() await atPersistedOracleCheckpoint( @@ -18884,6 +18972,8 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { [ `after read`, `open before admission`, + `unreserved across pullSince`, + `open during pullSince`, `open across mutex turn`, `open across scope entry`, `open during the read`, @@ -18891,6 +18981,15 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { `open across startup`, ] as const ).flatMap((commitTiming) => { + // Only a gap recovery that asks the leader waits for pullSince. + if ( + (commitTiming === `unreserved across pullSince` || + commitTiming === `open during pullSince`) && + route !== `seq-gap pull` && + route !== `seq-gap deltas` + ) { + return [] + } // Only a subset load is admitted through the wrapper's admission. if ( commitTiming === `open before admission` && @@ -18913,10 +19012,12 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { } const waits = commitTiming === `open before admission` || + commitTiming === `open during pullSince` || commitTiming === `open across mutex turn` || commitTiming === `open across scope entry` const endings: Array = - commitTiming === `after read` + commitTiming === `after read` || + commitTiming === `unreserved across pullSince` ? [`commits`] : waits ? [ diff --git a/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts index a85c2a3c13..9b89016a62 100644 --- a/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts @@ -5,7 +5,10 @@ * Law: a persisted hydration that would start while a reserved sync * transaction is open waits until it ends; a commit is then accepted, and an * abandoned transaction applies nothing (`reserveCommitTurn`, - * packages/db/src/types.ts, widened by maintainer decision on #2226). The + * packages/db/src/types.ts, widened by maintainer decision on #2226). A peer + * reset is the exception: it abandons a reserved transaction that began + * before it, so that commit rejects and the reset does not wait (maintainer + * decision on PR #2265). The * primary owner is the persisted-reads grammar in persisted-oracle.test.ts, * which uses a recording adapter and simulates a held hydration scope. This * oracle drives the same law through `SQLiteCorePersistenceAdapter` over @@ -21,8 +24,8 @@ * * Model: row-1's durable value before the reservation follows from the * history alone: the initial row, a peer edit, or the held write's value. A - * commit makes row-1 `refetched`; an abandoned transaction leaves the durable - * value. Each cell reaches its premise: the hydration entered its scope while + * commit makes row-1 `refetched`; an abandoned transaction, or one a peer + * reset abandoned, leaves the durable value. Each cell reaches its premise: the hydration entered its scope while * the reservation was open. * * Limits: the simulated peer writes straight to the adapter and announces @@ -418,7 +421,9 @@ describe('reserved commits across real-adapter hydrations', () => { ) const loadOutcome = await load reach.assertReached() - const expectedValue = ending === 'commits' ? 'refetched' : durableBefore + // A peer reset abandons the reservation, which began before it. + const commitAccepted = ending === 'commits' && route !== 'peer reset' + const expectedValue = commitAccepted ? 'refetched' : durableBefore await waitFor(() => expect(harness.collection.get('row-1')).toMatchObject({ value: expectedValue, @@ -430,7 +435,7 @@ describe('reserved commits across real-adapter hydrations', () => { status: harness.collection.status, loadOutcome, }).toEqual({ - outcome: ending === 'commits' ? 'fulfilled' : 'rejected', + outcome: commitAccepted ? 'fulfilled' : 'rejected', status: 'ready', loadOutcome: route === 'subset admitted earlier' ? 'fulfilled' : undefined, diff --git a/packages/db/src/types.ts b/packages/db/src/types.ts index c02d349b3d..ba5bc2691f 100644 --- a/packages/db/src/types.ts +++ b/packages/db/src/types.ts @@ -589,7 +589,11 @@ export type SyncPersistenceCapabilityV1< * load admitted while the turn is reserved also runs after the commit's * durable write. The source must end the transaction: commit it, or abandon * it by committing with an aborted signal. An abandon releases the waiting - * hydrations as a commit does, so they run. A source that stops for good + * hydrations as a commit does, so they run. A peer reset abandons a + * reserved transaction that began before the reset: its commit rejects + * with `SyncTransactionAbortedError`, and the reset does not wait. + * Sequence-gap recovery counts as a hydration only from the point where it + * changes rows. A source that stops for good * abandons it, or every later hydration keeps waiting. The source must not * await a `hydrateBaseline` that it started before reserving while the * reservation is open: that baseline waits for the commit. Calling From 5ac5a8711182739828162018f5f927da55f5ac87 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 19:33:12 -0600 Subject: [PATCH 13/24] test(browser-sqlite): cross peer resets with Electric in the generated multi-tab histories The generated histories gain a peer collection:reset step, posted on the tabs' coordinator channel because no shipped host sends one yet. The reset abandons every open reserved transaction. Electric still holds it, so its next commit rejects and the Collection goes to error until the app restarts it (maintainer decision on PR #2265, Law 5). A reset that waits and accepts the stale commit fails both campaigns. The budget rises to 80 runs so every declared premise stays reached. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 6 + .../tests/multi-tab-electric-oracle.test.ts | 104 ++++++++++++++++-- 2 files changed, 101 insertions(+), 9 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index b6c55fc7dd..7cd8a4eab5 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -3071,6 +3071,12 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. generated histories and pinned with `it.fails`: a resumed tab that lags behind a peer can receive an update for a row the peer already deleted, and Electric's resume check puts it in `error`. + The histories also post a peer `collection:reset` on the tabs' + coordinator channel; no shipped host sends one yet, so this premise is + controlled. The reset abandons every open reserved transaction, and + Electric's next commit then puts that Collection in `error` until it + restarts (maintainer decision on PR #2265). A reset that waits and + accepts the stale commit fails both campaigns. A delivery may also commit only a prefix of what a stream owes, as a server answering at an earlier moment does, so lagging commits are generated. The final checkpoint requires the server head (Law 2). diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 0d32a05f2f..6683e439a2 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -64,7 +64,15 @@ * up-to-date stays `loading`, and a tab whose initial sync failed stays in * `error` until it restarts. * - * The generated histories below judge Laws 1, 3, and 4 over interleaved + * Law 5, a peer reset abandons an older reservation, and Electric fails + * loudly. Authority: maintainer decisions on PR #2265. A peer + * `collection:reset` abandons every reserved transaction that began before + * it. Electric still holds that transaction, so its next commit rejects and + * puts the Collection in `error`; the app recovers by restarting the tab. + * No shipped host sends `collection:reset` yet, so the histories post it on + * the tabs' coordinator channel as a peer would (a controlled premise). + * + * The generated histories below judge Laws 1, 3, 4, and 5 over interleaved * server changes, deploys, deliveries, stops, and recoveries, against a * model of the durable rows. One known gap is left out of them and pinned: * a resumed tab that lags behind a peer can receive an update for a row the @@ -226,15 +234,19 @@ function openHost() { // notices, read from the coordinator's own channel. let activity = 0 let commitNotices = 0 + let storageCollectionId: string | undefined + let resetEpoch = 0 const channel = new BroadcastChannel(`tsdb:coord:${dbName}`) channel.onmessage = (event: MessageEvent) => { activity++ const envelope = event.data as { senderId?: string + collectionId?: string payload?: { type?: string } } if (envelope.payload?.type === `tx:committed` && envelope.senderId) { commitNotices++ + storageCollectionId ??= envelope.collectionId } } @@ -372,6 +384,27 @@ function openHost() { openTab, quiesce, commitNotices: () => commitNotices, + // A peer tab's collection reset, posted on the tabs' coordinator channel + // as a peer would post it. No shipped host sends this notice yet, so + // the premise is controlled; the tabs, their coordinators, and Electric + // receive it through production. + broadcastPeerReset: () => { + if (!storageCollectionId) { + throw new Error(`no commit notice named the storage collection yet`) + } + channel.postMessage({ + v: 1, + dbName, + collectionId: storageCollectionId, + senderId: `peer-reset-tab`, + ts: Date.now(), + payload: { + type: `collection:reset`, + schemaVersion: 1, + resetEpoch: ++resetEpoch, + }, + }) + }, // The durable rows of the shared Collection, read through a connection of // its own so the read counts as no host activity. durableRows: async (): Promise> => { @@ -422,7 +455,7 @@ async function openWarmTabs(host: Host, count: number, syncMode: SyncMode) { // TANSTACK_DB_ORACLE_RUNS_MULTIPLIER scales the budget. const GENERATED_PROPERTY = `browser.multi-tab-reserved-history` const GENERATED_FIXED_SEED = 952051961 -const GENERATED_RUNS = 60 +const GENERATED_RUNS = 80 async function waitForStream(tab: Tab, count: number) { for (let task = 0; task < 400 && tab.streamCount() < count; task++) { @@ -721,6 +754,7 @@ describe(`multi-tab Electric over one persisted database`, () => { // // - server changes (insert, update, or delete of one of four rows); // - deploys, after which every stream must refetch; + // - a peer reset, which abandons every open reserved transaction (Law 5); // - deliveries to one tab: part of what its stream still owes, with no // up-to-date, which leaves a reserved transaction open, or everything it // owes and up-to-date, which commits it; @@ -758,6 +792,7 @@ describe(`multi-tab Electric over one persisted database`, () => { type HistoryStep = | { kind: `server change`; key: number; pick: number } | { kind: `deploy` } + | { kind: `peer reset` } | { kind: `advance`; tab: number; pick: number } | { kind: `catch up`; tab: number; pick: number } // A tab that closes for good. Generated histories never close a tab; @@ -778,6 +813,10 @@ describe(`multi-tab Electric over one persisted database`, () => { weight: 1, arbitrary: fc.record({ kind: fc.constant(`deploy` as const) }), }, + { + weight: 1, + arbitrary: fc.record({ kind: fc.constant(`peer reset` as const) }), + }, { weight: 3, arbitrary: fc.record({ @@ -850,6 +889,11 @@ describe(`multi-tab Electric over one persisted database`, () => { // peer commit in that interval is the premise of the abandon law. stoppedMidTransaction: StreamStop | undefined closed: boolean + // A peer reset abandoned this tab's open transaction. Electric still + // holds it, so its next commit rejects and puts the Collection in + // `error` (maintainer decision on PR #2265); the app then restarts it. + abandonedByReset: boolean + failedByReset: boolean } const stops: ReadonlyArray = [ `error without retry`, @@ -905,6 +949,8 @@ describe(`multi-tab Electric over one persisted database`, () => { resumed: false, stoppedMidTransaction: undefined, closed: false, + abandonedByReset: false, + failedByReset: false, })) let serverVersion = 0 generatedReach.beginHistory() @@ -991,7 +1037,14 @@ describe(`multi-tab Electric over one persisted database`, () => { complete ? [...messages, upToDate] : messages, position(log, model.version), ) - if (complete) { + if (complete && model.abandonedByReset) { + generatedReach.reach( + `reset-abandoned commit put the Collection in error`, + ) + abandon(model) + model.abandonedByReset = false + model.failedByReset = true + } else if (complete) { commit(model) model.ready = true } @@ -1018,6 +1071,8 @@ describe(`multi-tab Electric over one persisted database`, () => { failedInitialSync: false, resumed: requested.handle !== undefined, stoppedMidTransaction: undefined, + abandonedByReset: false, + failedByReset: false, ...requested, }) await activeHost.quiesce(`tab ${recovered.tab} recovered`) @@ -1033,7 +1088,7 @@ describe(`multi-tab Electric over one persisted database`, () => { const expectedStatus = (index: number) => models[index]!.stopped === `cleanup` || models[index]!.closed ? `cleaned-up` - : models[index]!.failedInitialSync + : models[index]!.failedInitialSync || models[index]!.failedByReset ? `error` : models[index]!.ready ? `ready` @@ -1109,6 +1164,21 @@ describe(`multi-tab Electric over one persisted database`, () => { abandon(models[index]!) models[index]!.closed = true await activeHost.quiesce(`tab ${tabs[index]!.tab} closed for good`) + } else if (step.kind === `peer reset`) { + // A peer reset abandons every open reserved transaction that began + // before it (maintainer decision on PR #2265). It changes no durable + // row, so every tab reloads the same durable rows. + if (models.some((model) => model.open !== undefined)) { + generatedReach.reach(`peer reset abandoned an open transaction`) + } + for (const model of models) { + if (model.open !== undefined && model.stopped === undefined) { + model.abandonedByReset = true + } + abandon(model) + } + activeHost.broadcastPeerReset() + await activeHost.quiesce(`peer reset applied`) } else if (step.kind === `deploy`) { log.handle++ serverVersion = 0 @@ -1126,21 +1196,26 @@ describe(`multi-tab Electric over one persisted database`, () => { // stopped tab goes to a running one, so a stopped tab stays stopped // while its peers commit. const runningTabs = models.flatMap((model, index) => - model.stopped === undefined ? [index] : [], + model.stopped === undefined && !model.failedByReset ? [index] : [], ) const index = step.kind === `stop` && openTabs.length > 0 ? openTabs[step.tab % openTabs.length]! : step.kind !== `recover` && - models[step.tab % tabs.length]!.stopped !== undefined && + (models[step.tab % tabs.length]!.stopped !== undefined || + models[step.tab % tabs.length]!.failedByReset) && runningTabs.length > 0 ? runningTabs[step.tab % runningTabs.length]! : step.tab % tabs.length const model = models[index]! - if (model.stopped !== undefined || step.kind === `recover`) { + if ( + model.stopped !== undefined || + model.failedByReset || + step.kind === `recover` + ) { // A running tab can only reload; a stopped tab restarts or reloads. const how = - model.stopped === undefined || + (model.stopped === undefined && !model.failedByReset) || (step.kind === `recover` && step.pick === 1) ? `reload` : `restart` @@ -1155,6 +1230,8 @@ describe(`multi-tab Electric over one persisted database`, () => { () => undefined, ) model.stopped = how + // Electric abandons the dead transaction itself, so nothing errors. + model.abandonedByReset = false abandon(model) model.failedInitialSync = how !== `cleanup` && !model.ready await activeHost.quiesce(`tab ${tabs[index]!.tab} stopped`) @@ -1205,7 +1282,12 @@ describe(`multi-tab Electric over one persisted database`, () => { // commit survives the catch-up. for (const index of tabs.keys()) { if (models[index]!.closed) continue - if (models[index]!.stopped !== undefined) await recover(index, `restart`) + if ( + models[index]!.stopped !== undefined || + models[index]!.failedByReset + ) { + await recover(index, `restart`) + } await deliver(index, `all`, true) } await check(`after every tab caught up`, rowsOf(log.rows)) @@ -1221,6 +1303,8 @@ describe(`multi-tab Electric over one persisted database`, () => { | `peer commit after error without retry mid-transaction` | `peer commit after SDK gives up mid-transaction` | `peer commit after signal aborted mid-transaction` + | `peer reset abandoned an open transaction` + | `reset-abandoned commit put the Collection in error` const createGeneratedReach = () => createOracleReach(`browser.multi-tab-reserved-history`, [ `must-refetch delivered`, @@ -1232,6 +1316,8 @@ describe(`multi-tab Electric over one persisted database`, () => { `peer commit after error without retry mid-transaction`, `peer commit after SDK gives up mid-transaction`, `peer commit after signal aborted mid-transaction`, + `peer reset abandoned an open transaction`, + `reset-abandoned commit put the Collection in error`, ]) let generatedReach = createGeneratedReach() From 032fdc2629aad1897c567105df2bd97b6bc4b1a8 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 19:52:07 -0600 Subject: [PATCH 14/24] fix(sqlite-persistence): abandon every older reservation on a peer reset Review finding: the reset abandoned only the first reserved transaction on the stack, so a nested reserved transaction kept its reservation and later rejected with the cross-cycle error instead of the reset's abort. The real-adapter oracle gains a nesting dimension; its two nested peer-reset commit cells failed on the parent and pass now. Every wait for one reservation now shares one promise, so repeated checks no longer add waiters, and the notices that wait behind one reservation share one deferred flush. Positional flags are named at their call sites. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 6 +- .../src/persisted.ts | 58 ++++++++++------- ...erved-commit-sqlite-adapter-oracle.test.ts | 63 ++++++++++++++----- 3 files changed, 88 insertions(+), 39 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 7cd8a4eab5..2c1f69070a 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -3024,7 +3024,11 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. Its peer-reset cells judge that the reset abandons the reservation: the commit rejects and row-1 keeps its durable value (both committing peer-reset cells fail on `2047f4281`). - Its 10 cells pass. Every committing cell fails on `5649a07d4`. A + A nesting dimension adds a second reserved transaction inside the + first. A peer reset must abandon both, so both reject with the same + abort; on `5ac5a8711` the inner one rejected with the cross-cycle + error instead. Its 20 cells pass. + Every committing single-transaction cell fails on `5649a07d4`. A mutant that rejects the waited load after the reservation fails the 4 subset cells; before the load's outcome was judged, it passed. A peer full reload behind this Collection's own held write cannot become diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 98eca29aab..28f3071da2 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -1301,6 +1301,8 @@ type OpenSyncTransaction = Omit< queuedBecauseHydrating: boolean // Set by `reserveCommitTurn`: every hydration waits for this commit. reservesCommitTurn?: boolean + // Settles once the reservation ends; every wait shares it. + reservationEnded?: Promise hasDependentSuccessor: boolean publicationAdmissionWaiters?: Set<{ resolve: () => void @@ -1567,6 +1569,7 @@ class PersistedCollectionRuntime< private hydrationSequence = 0 private waitForReservedCommit: WaitForReservedCommit | undefined private abandonReservedCommit: (() => void) | undefined + private reservedCommitFlush: Promise | undefined // An unscheduled startup read may finish after a coordinator reset publishes. private resetSequence = 0 private readonly pendingRemoteSubsetEnsures = new Map< @@ -1955,7 +1958,11 @@ class PersistedCollectionRuntime< // apply in arrival order once the reserved transaction commits. private flushAfterReservedCommit(): void { const lifecycleGeneration = this.lifecycleGeneration - void Promise.resolve(this.waitForReservedCommit?.()) + const ended = this.waitForReservedCommit?.() + // One deferred flush per reservation drains every notice behind it. + if (ended && ended === this.reservedCommitFlush) return + this.reservedCommitFlush = ended + void Promise.resolve(ended) .then(() => { if (lifecycleGeneration !== this.lifecycleGeneration) return return this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) @@ -4919,7 +4926,13 @@ class PersistedCollectionRuntime< this.throwIfReservedCommitOpen() // Every reload in this recovery continues its hydration cycle. const reloadInCycle = (adapter: HydrationPersistenceAdapter) => - this.reloadActiveSubsetsUnsafe(adapter, false, false, false, true) + this.reloadActiveSubsetsUnsafe( + adapter, + /* skipUnchangedReplace */ false, + /* restoreResetBaseline */ false, + /* replayDemand */ false, + /* continuesHydrationCycle */ true, + ) if (this.persistence.coordinator.pullSince && this.latestRowVersion >= 0) { let pullResponse: PullSinceResponse | undefined try { @@ -5128,10 +5141,10 @@ class PersistedCollectionRuntime< // follower requests its demand again. await this.reloadActiveSubsetsUnsafe( adapter, - false, - this.syncMode === `eager`, - true, - true, + /* skipUnchangedReplace */ false, + /* restoreResetBaseline */ this.syncMode === `eager`, + /* replayDemand */ true, + /* continuesHydrationCycle */ true, ) } @@ -5150,8 +5163,8 @@ class PersistedCollectionRuntime< this.reloadActiveSubsetsUnsafe( scopedAdapter, skipUnchangedReplace, - false, - false, + /* restoreResetBaseline */ false, + /* replayDemand */ false, continuesHydrationCycle, ), adapter, @@ -6026,23 +6039,26 @@ function createWrappedSyncConfig< if (!transaction) return undefined // Admission waiters settle when the transaction commits, aborts, or // cleanup discards it. Every outcome closes the reservation. - return new Promise((resolve) => { - ;(transaction.publicationAdmissionWaiters ??= new Set()).add({ - resolve, - reject: () => resolve(), - }) - }) + return (transaction.reservationEnded ??= new Promise( + (resolve) => { + ;(transaction.publicationAdmissionWaiters ??= new Set()).add({ + resolve, + reject: () => resolve(), + }) + }, + )) }, - // A peer reset abandons a reservation built before it: later writes + // A peer reset abandons every reservation built before it: later writes // are dropped, the commit rejects, and its waiters run. () => { - const transaction = findReservedTransaction() - if (!transaction) return - transaction.reservesCommitTurn = false - transaction.terminalFailure ??= { - error: new SyncTransactionAbortedError(), + for (const transaction of transactionStack) { + if (!transaction.reservesCommitTurn) continue + transaction.reservesCommitTurn = false + transaction.terminalFailure ??= { + error: new SyncTransactionAbortedError(), + } + settlePublicationAdmissionWaiters(transaction) } - settlePublicationAdmissionWaiters(transaction) }, ) diff --git a/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts index 9b89016a62..7df54a1d2e 100644 --- a/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/reserved-commit-sqlite-adapter-oracle.test.ts @@ -19,8 +19,11 @@ * Grammar: the hydration route (a peer full-reload notice, a peer reset, or a * subset load admitted before the reservation) crossed with the timing (the * route waits behind this Collection's held write, or the shared scheduler - * holds its scope behind another Collection's write after its turn began) and - * the ending (the reserved transaction commits, or is abandoned). + * holds its scope behind another Collection's write after its turn began), + * the nesting (one reserved transaction, or a second one nested inside it), + * and the ending (the reserved transactions commit, or are abandoned). A + * peer reset abandons every reserved transaction that began before it, so a + * nested one rejects with the same abort as its outer one. * * Model: row-1's durable value before the reservation follows from the * history alone: the initial row, a peer edit, or the held write's value. A @@ -263,6 +266,13 @@ const openPeerTabHarness = async ({ if (value) params.write({ type: 'update', value: { id: 'row-1', value } }) params.metadata?.persistence?.reserveCommitTurn() } + // A second reserved transaction nested inside the open one. + const reserveNested = () => { + const params = requireSource() + params.begin() + params.write({ type: 'insert', value: { id: 'row-2', value: 'nested' } }) + params.metadata?.persistence?.reserveCommitTurn() + } const commit = () => requireSource().commit() // Committing with an aborted signal abandons the transaction. const abandon = () => { @@ -272,6 +282,7 @@ const openPeerTabHarness = async ({ } await start() return { + reserveNested, abandon, adapter, coordinator, @@ -288,6 +299,7 @@ const openPeerTabHarness = async ({ type AdapterHistory = { route: 'peer full reload' | 'peer reset' | 'subset admitted earlier' timing: 'mutex turn' | 'scheduler hold' + nesting: 'single' | 'nested' ending: 'commits' | 'abandoned' } @@ -300,11 +312,14 @@ const adapterHistories: Array = ( // leader; the persisted-reads grammar's fixture leader covers it. route === 'peer full reload' && timing === 'mutex turn' ? [] - : (['commits', 'abandoned'] as const).map((ending) => ({ - route, - timing, - ending, - })), + : (['single', 'nested'] as const).flatMap((nesting) => + (['commits', 'abandoned'] as const).map((ending) => ({ + route, + timing, + nesting, + ending, + })), + ), ), ) @@ -327,8 +342,8 @@ describe('reserved commits across real-adapter hydrations', () => { }) it.each(adapterHistories)( - 'keeps a reserved commit lawful across a real-adapter hydration ($route, $timing, $ending)', - async ({ route, timing, ending }) => { + 'keeps a reserved commit lawful across a real-adapter hydration ($route, $timing, $nesting, $ending)', + async ({ route, timing, nesting, ending }) => { const harness = await openPeerTabHarness({ sharedScheduling: timing === 'scheduler hold', onDemand: route === 'subset admitted earlier', @@ -376,6 +391,7 @@ describe('reserved commits across real-adapter hydrations', () => { // One task lets the route take its mutex turn before the reservation. await new Promise((resolve) => setTimeout(resolve, 0)) harness.reserve('refetched') + if (nesting === 'nested') harness.reserveNested() const entered = harness.nextScopeEntry() write.release() await blocker @@ -407,18 +423,26 @@ describe('reserved commits across real-adapter hydrations', () => { admitRoute() await reached(began, 'route turn began') harness.reserve('refetched') + if (nesting === 'nested') harness.reserveNested() const entered = harness.nextScopeEntry() otherWrite.release() await blocker await reached(entered, 'hydration entered its scope') } reach.reach('hydration entered its scope while reserved') - const outcome = await Promise.resolve( - ending === 'commits' ? harness.commit() : harness.abandon(), - ).then( - () => 'fulfilled', - () => 'rejected', - ) + // Commits the innermost transaction first; a rejection keeps its name, + // so an abort and a cross-cycle failure stay distinct. + const settle = async () => { + try { + await (ending === 'commits' ? harness.commit() : harness.abandon()) + return 'fulfilled' + } catch (error) { + return `rejected (${(error as Error).name})` + } + } + const outcomes: Array = [] + if (nesting === 'nested') outcomes.push(await settle()) + outcomes.push(await settle()) const loadOutcome = await load reach.assertReached() // A peer reset abandons the reservation, which began before it. @@ -431,11 +455,16 @@ describe('reserved commits across real-adapter hydrations', () => { ) // A subset load that waited runs after the reservation and succeeds. expect({ - outcome, + outcomes, + nestedRow: harness.collection.get('row-2')?.value, status: harness.collection.status, loadOutcome, }).toEqual({ - outcome: commitAccepted ? 'fulfilled' : 'rejected', + outcomes: Array.from({ length: nesting === 'nested' ? 2 : 1 }, () => + commitAccepted ? 'fulfilled' : 'rejected (AbortError)', + ), + nestedRow: + nesting === 'nested' && commitAccepted ? 'nested' : undefined, status: 'ready', loadOutcome: route === 'subset admitted earlier' ? 'fulfilled' : undefined, From 09e027412148e9607e66fa8fa231577824db38e5 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 20:53:52 -0600 Subject: [PATCH 15/24] docs(electric): state that awaitTxId resolves on stream evidence Maintainer decision on PR #2265: awaitTxId keeps its evidence meaning. It resolves when the txid appears in the stream, before persistence accepts the commit; a commit that later rejects puts the Collection in error. The coverage map records the limit. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 4 ++++ packages/electric-db-collection/src/electric.ts | 5 ++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 2c1f69070a..e89976f605 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -3004,6 +3004,10 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. | Gap recovery that reloads partway through, on `851e8d9da` | 28 cells fail: recovery advanced the stream position and applied a targeted delta before it deferred, so its retry skipped the reload. A gate only at recovery start still fails 15 scope-entry cells; reloads that start their own cycle inside recovery fail 61. | | `5649a07d4` | The open cells fail with `a persisted sync transaction cannot cross a hydration cycle`. Three `after read` reset follow-up cells also fail, because `main` applied a pre-reset commit before the reset set its term fence. Queueing resets with notices fixes this pre-existing defect. | | Hostile mutants | Each mutant fails its cells, and every failure is an assertion. Deferring every notice fails the targeted change. Dropping the follower reply's wait fails the routed subset. Reporting hydration start before the check fails the aborted demand. A reset reload that starts its own cycle fails the first-publication reset. Applying a reset outside the notice queue fails the reset follow-ups. Retrying a waited notice while the reservation is open fails the follow-up and routed cells (repeated pullSince). Checking only at the mutex turn fails every scope-entry cell. Gating subset hydration only fails every reset, reload, and gap cell. Requeueing a waited notice behind later notices fails 12 reset follow-up cells: the pre-reset commit is published before its reset, and a reload is lost. | + Known limit (maintainer decision on PR #2265): Electric publishes a + transaction's txids as stream evidence before persistence accepts its + commit, so `awaitTxId` can resolve for a commit that a peer reset or a + terminal failure later rejects; the Collection then goes to `error`. Known limit (maintainer decision on PR #2265): no fairness rule. A hydration released by a commit waits again for a reservation that opens before its next mutex turn, so a source that reserves back to diff --git a/packages/electric-db-collection/src/electric.ts b/packages/electric-db-collection/src/electric.ts index 0a5a38c98b..a4c2e12e85 100644 --- a/packages/electric-db-collection/src/electric.ts +++ b/packages/electric-db-collection/src/electric.ts @@ -888,7 +888,10 @@ function createLoadSubsetDedupe>({ } /** - * Type for the awaitTxId utility function + * Type for the awaitTxId utility function. It resolves when the txid + * appears in the Electric stream (the server's confirmation), before this + * tab's persistence accepts the transaction. If that commit later rejects, + * the Collection goes to `error` instead (maintainer decision on PR #2265). */ export type AwaitTxIdFn = (txId: Txid, timeout?: number) => Promise From a7de6254143f2d0c2258c6131787361ab4a2346f Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 21:14:36 -0600 Subject: [PATCH 16/24] test(browser-sqlite): judge source failures through health after #2168 #2168 splits Collection status from sync errors: after readiness a source failure keeps the Collection ready and records the error on its health node, and `error` means only a failed first load. Resume invalidation and a reset-abandoned commit happen after readiness, so the multi-tab oracle now expects `ready` with a health error there, and `error` only when the first load failed. Law 3 and Law 5 prose follow. Removing Electric's failure on a rejected commit fails both generated campaigns. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 13 ++-- .../tests/multi-tab-electric-oracle.test.ts | 65 ++++++++++++------- 2 files changed, 50 insertions(+), 28 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index df8e974a34..2ae6444eec 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -3011,7 +3011,8 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. Known limit (maintainer decision on PR #2265): Electric publishes a transaction's txids as stream evidence before persistence accepts its commit, so `awaitTxId` can resolve for a commit that a peer reset or a - terminal failure later rejects; the Collection then goes to `error`. + terminal failure later rejects; the source then fails, and the + Collection's health node records it. Known limit (maintainer decision on PR #2265): no fairness rule. A hydration released by a commit waits again for a reservation that opens before its next mutex turn, so a source that reserves back to @@ -3055,8 +3056,8 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. checkpoints, except a cleaned-up tab before recovery. It also judges the full-snapshot request after resume invalidation. A Collection in `error` keeps showing durable rows (maintainer decision on PR #2265), - so the tab that resume invalidation put in `error` must show its peers' - refetch. Its 290 storm cells pass. Every ordinary storm + so the tab whose source resume invalidation failed must show its + peers' refetch; it stays `ready` with an error on its health node. Its 290 storm cells pass. Every ordinary storm cell fails on `5649a07d4`. Removing Electric's abandon on the `onError`, give-up, or signal-abort path fails 12 cells each, and on the resume invalidation path 16 cells. Cleanup's abandon mutant survives here, @@ -3086,8 +3087,10 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. The histories also post a peer `collection:reset` on the tabs' coordinator channel; no shipped host sends one yet, so this premise is controlled. The reset abandons every open reserved transaction, and - Electric's next commit then puts that Collection in `error` until it - restarts (maintainer decision on PR #2265). A reset that waits and + Electric's next commit then fails the source until the tab restarts + (maintainer decision on PR #2265): a ready tab keeps `ready` with an + error on its health node, which the histories judge, and a loading tab + goes to `error` (#2168). A reset that waits and accepts the stale commit fails both campaigns. A delivery may also commit only a prefix of what a stream owes, as a server answering at an earlier moment does, so lagging commits are diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 6683e439a2..02dd86fc30 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -51,12 +51,15 @@ * Collection status and public rows, and the position from which a * recovered tab's new stream starts. * - * Law 3, a Collection in `error` keeps showing durable rows. Authority: - * maintainer decision on PR #2265. A persisted Collection that a source - * failure put in `error` still applies its peers' durable commits, so its - * public rows match the durable rows. This file judges it after resume - * invalidation and after a failed initial sync; the source must end its - * reserved transaction for the peers' commits to apply. + * Law 3, a Collection whose source failed keeps showing durable rows. + * Authority: maintainer decision on PR #2265. A persisted Collection whose + * source failed still applies its peers' durable commits, so its public rows + * match the durable rows. A failure before the first load completes puts it + * in `error`; a later one leaves it `ready` with an error on its health node + * (#2168: after readiness, sync failures never change Collection status). + * This file judges it after resume invalidation, after a failed initial + * sync, and after a reset abandoned the source's transaction; the source + * must end its reserved transaction for the peers' commits to apply. * * Law 4, a Collection status changes only through its own source. Authority: * maintainer decision on PR #2265. A peer's durable commit reloads a tab's @@ -68,7 +71,9 @@ * loudly. Authority: maintainer decisions on PR #2265. A peer * `collection:reset` abandons every reserved transaction that began before * it. Electric still holds that transaction, so its next commit rejects and - * puts the Collection in `error`; the app recovers by restarting the tab. + * the source fails: a ready Collection keeps `ready` with an error on its + * health node, one that is still loading goes to `error`. The app recovers by + * restarting the tab. * No shipped host sends `collection:reset` yet, so the histories post it on * the tabs' coordinator channel as a peer would (a controlled premise). * @@ -524,8 +529,9 @@ describe(`multi-tab Electric over one persisted database`, () => { // (maintainer decision on #2226). So a stopped tab that is still open // applies its peers' refetch from durable rows, as it did before // reservations deferred hydrations. Electric then fails no further, except - // that resume invalidation puts the Collection in `error`, as Electric - // documents; that Collection still shows durable rows (Law 3). The + // that resume invalidation fails the source, as Electric documents: the + // ready tab keeps `ready` with an error on its health node and still shows + // durable rows (Law 3). The // application's recovery, cleanup and a restart, then brings // the tab to the head. After resume invalidation the restart must request a // full snapshot (electric.ts: "a full snapshot is required"). @@ -687,29 +693,33 @@ describe(`multi-tab Electric over one persisted database`, () => { } // Checkpoint 1, every source has delivered or stopped. A rejected - // reserved commit puts its Collection in `error`. A stopped tab that is - // still open shows its peers' refetch, including one in `error` after + // reserved commit would fail its source. A stopped tab that is still + // open shows its peers' refetch, including one whose source failed on // resume invalidation (Law 3); a cleaned-up tab has no rows. Eager rows // must equal the head snapshot; on-demand rows are not judged (above). - const stoppedStatus = - stop === `cleanup` - ? `cleaned-up` - : stop === `resume invalidated` - ? `error` - : `ready` + // Resume invalidation comes after the tab was ready, so the tab stays + // `ready` and its health node records the source's error (#2168: after + // readiness, sync failures never change Collection status). + const stoppedStatus = stop === `cleanup` ? `cleaned-up` : `ready` const judgesRowsBeforeRestart = (tab: Tab) => syncMode === `eager` && (tab !== stoppedTab || stop !== `cleanup`) + const judgesSourceError = (tab: Tab) => + tab === stoppedTab && stop === `resume invalidated` const observe = (judgesRows: (tab: Tab) => boolean) => tabs.map((tab) => ({ tab: tab.tab, status: tab.collection.status, ...(judgesRows(tab) ? { rows: tab.publicRows() } : {}), + ...(judgesSourceError(tab) + ? { sourceError: tab.collection.health.error !== null } + : {}), })) expect(observe(judgesRowsBeforeRestart)).toEqual( tabs.map((tab) => ({ tab: tab.tab, status: tab === stoppedTab ? stoppedStatus : `ready`, ...(judgesRowsBeforeRestart(tab) ? { rows: afterDeploy } : {}), + ...(judgesSourceError(tab) ? { sourceError: true } : {}), })), ) @@ -735,11 +745,13 @@ describe(`multi-tab Electric over one persisted database`, () => { if (stop === `resume invalidated`) { expect(restartedAt).toEqual({ handle: undefined, offset: `-1` }) } + // The new sync run starts without the retired run's problems. expect(observe(() => syncMode === `eager`)).toEqual( tabs.map((tab) => ({ tab: tab.tab, status: `ready`, ...(syncMode === `eager` ? { rows: afterDeploy } : {}), + ...(judgesSourceError(tab) ? { sourceError: false } : {}), })), ) } @@ -774,9 +786,10 @@ describe(`multi-tab Electric over one persisted database`, () => { // head. // // Checkpoint after every step, at host quiescence: - // - no Collection is in `error`, and only a tab stopped by cleanup is - // `cleaned-up`; a rejected reserved commit would put its Collection in - // `error`; + // - every status is the one the model owes: `error` only for a failed + // first load, `cleaned-up` only for a tab stopped by cleanup; a + // reset-abandoned commit fails the source, which keeps a ready tab + // `ready` with an error on its health node (Law 5); // - the durable rows equal the fold; // - every open tab with no open transaction shows the durable rows. That // includes a tab whose source stopped: it abandoned its transaction, so @@ -890,8 +903,8 @@ describe(`multi-tab Electric over one persisted database`, () => { stoppedMidTransaction: StreamStop | undefined closed: boolean // A peer reset abandoned this tab's open transaction. Electric still - // holds it, so its next commit rejects and puts the Collection in - // `error` (maintainer decision on PR #2265); the app then restarts it. + // holds it, so its next commit rejects and the source fails (maintainer + // decision on PR #2265, Law 5); the app then restarts it. abandonedByReset: boolean failedByReset: boolean } @@ -1088,7 +1101,8 @@ describe(`multi-tab Electric over one persisted database`, () => { const expectedStatus = (index: number) => models[index]!.stopped === `cleanup` || models[index]!.closed ? `cleaned-up` - : models[index]!.failedInitialSync || models[index]!.failedByReset + : models[index]!.failedInitialSync || + (models[index]!.failedByReset && !models[index]!.ready) ? `error` : models[index]!.ready ? `ready` @@ -1099,6 +1113,8 @@ describe(`multi-tab Electric over one persisted database`, () => { tab: tab.tab, status: expectedStatus(index), ...(judgesRows(index) ? { rows: expectedDurable } : {}), + // The reset-abandoned source failed (Law 5). + ...(models[index]!.failedByReset ? { sourceError: true } : {}), })), } const actual = { @@ -1107,6 +1123,9 @@ describe(`multi-tab Electric over one persisted database`, () => { tab: tab.tab, status: tab.collection.status, ...(judgesRows(index) ? { rows: tab.publicRows() } : {}), + ...(models[index]!.failedByReset + ? { sourceError: tab.collection.health.error !== null } + : {}), })), } // fast-check reports only the message, so it carries both sides, plus From b3cc7df6007982b7c770e034cffdb6eaa31dde37 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 21:48:31 -0600 Subject: [PATCH 17/24] fix(sqlite-persistence): keep a follower's load waiting until the owner's notice applies Review finding: the follower's reply loop stopped when its reservation ended between the flush and the wait check, while the owner's notice was still deferred, so the load settled before the owner's row was public (law 7). The loop now waits while the head notice is deferred. The persisted-reads grammar judges law 7 at the routed load's settlement; a temporary hook that ended the reservation in that window failed 4 straddled cells on the old loop and passes on the new one. The grammar also gains a peer reset for a retired storage generation, which abandons nothing. One sentinel signals every deferral, and a beginHydrationCycle helper pairs the reservation check with the sequence step. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 7 ++ .../src/persisted.ts | 41 +++++++----- .../tests/persisted-oracle.test.ts | 67 +++++++++++++++---- 3 files changed, 87 insertions(+), 28 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 2ae6444eec..d1b956d0a6 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -3008,6 +3008,13 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. | Gap recovery that reloads partway through, on `851e8d9da` | 28 cells fail: recovery advanced the stream position and applied a targeted delta before it deferred, so its retry skipped the reload. A gate only at recovery start still fails 15 scope-entry cells; reloads that start their own cycle inside recovery fail 61. | | `5649a07d4` | The open cells fail with `a persisted sync transaction cannot cross a hydration cycle`. Three `after read` reset follow-up cells also fail, because `main` applied a pre-reset commit before the reset set its term fence. Queueing resets with notices fixes this pre-existing defect. | | Hostile mutants | Each mutant fails its cells, and every failure is an assertion. Deferring every notice fails the targeted change. Dropping the follower reply's wait fails the routed subset. Reporting hydration start before the check fails the aborted demand. A reset reload that starts its own cycle fails the first-publication reset. Applying a reset outside the notice queue fails the reset follow-ups. Retrying a waited notice while the reservation is open fails the follow-up and routed cells (repeated pullSince). Checking only at the mutex turn fails every scope-entry cell. Gating subset hydration only fails every reset, reload, and gap cell. Requeueing a waited notice behind later notices fails 12 reset follow-up cells: the pre-reset commit is published before its reset, and a reload is lost. | + A routed load also judges law 7 at its settlement: the owner's row is + public when the load settles. A reservation that ends between the + follower's flush and its wait check is a one-microtask window no + timing reaches; a temporary hook that ended the reservation there + showed the old wait loop settling early in 4 straddled cells and the + current loop keeping the owner's row public. A peer reset for a + storage generation this Collection no longer uses abandons nothing. Known limit (maintainer decision on PR #2265): Electric publishes a transaction's txids as stream evidence before persistence accepts its commit, so `awaitTxId` can resolve for a commit that a peer reset or a diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index f05175657d..3925201dda 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -1335,6 +1335,8 @@ type WaitForReservedCommit = () => Promise | undefined * durable write needs the mutex. */ class ReservedCommitPending extends Error {} +// One instance signals every deferral; callers compare it by identity. +const reservedCommitPending = new ReservedCommitPending() class ApplyMutex { private queue: Promise = Promise.resolve() @@ -1940,8 +1942,15 @@ class PersistedCollectionRuntime< // Every hydration start calls this before it changes state. The check runs // inside the hydration's own scope, because a shared scheduler can hold the // scope long after its apply-mutex turn began. + // A hydration cycle begins: it waits for an open reservation, then moves + // the sequence that open sync transactions are bound to. + private beginHydrationCycle(): void { + this.throwIfReservedCommitOpen() + this.hydrationSequence++ + } + private throwIfReservedCommitOpen(): void { - if (this.waitForReservedCommit?.()) throw new ReservedCommitPending() + if (this.waitForReservedCommit?.()) throw reservedCommitPending } /** Runs a hydration turn, retrying it after a reserved commit it met. */ @@ -1952,7 +1961,7 @@ class PersistedCollectionRuntime< try { return await this.applyMutex.run(turn) } catch (error) { - if (!(error instanceof ReservedCommitPending)) throw error + if (error !== reservedCommitPending) throw error await this.waitForReservedCommit?.() } } @@ -2101,15 +2110,18 @@ class PersistedCollectionRuntime< } await this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) // A broadcast whose reload waits for a reserved commit has not applied - // yet, so the load waits with it. Notices behind a head that does not - // wait arrived after this flush and are not the owner's. + // yet, so the load waits with it, including when the reservation ends + // before this check. Notices behind a head that does not wait arrived + // after this flush and are not the owner's. for (;;) { - const closed = this.queuedTxCommitted[0]?.waitsForReservedCommit - ? this.waitForReservedCommit?.() - : undefined - if (!closed) break - await closed + const head = this.queuedTxCommitted[0] + if (!head?.waitsForReservedCommit) break + const reservation = this.waitForReservedCommit?.() + await reservation await this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) + // With no reservation left, a head still queued is held by other work, + // such as a claim recovery, which flushes it. + if (!reservation && this.queuedTxCommitted[0] === head) break } // A broadcast that failed to apply made the runtime terminal; its rows // are not visible, so the load must not settle successfully. @@ -3469,12 +3481,11 @@ class PersistedCollectionRuntime< }, adapter: HydrationPersistenceAdapter, ): Promise { - this.throwIfReservedCommitOpen() + this.beginHydrationCycle() let rowsLoaded = false let boundRowVersion: number | undefined let replayFailure: { reason: unknown } | undefined const resetSequence = this.resetSequence - this.hydrationSequence++ const hydrationContext = { suppliedRowKeys: new Set() } this.activeHydrationContext = hydrationContext try { @@ -4845,7 +4856,7 @@ class PersistedCollectionRuntime< ) } } catch (error) { - if (error instanceof ReservedCommitPending) { + if (error === reservedCommitPending) { // Only a notice that reaches a hydration waits; later notices stay // queued behind it. The reserved commit lands first and takes the // next stream position, so the retry may run as gap recovery, which @@ -5009,8 +5020,7 @@ class PersistedCollectionRuntime< if (lifecycleGeneration !== this.lifecycleGeneration) return false if (pullResponse.ok) { - this.throwIfReservedCommitOpen() - this.hydrationSequence++ + this.beginHydrationCycle() this.observeStreamPosition( pullResponse.latestTerm, pullResponse.latestSeq, @@ -5346,7 +5356,7 @@ class PersistedCollectionRuntime< this.refreshScopedSubsets() return } - if (!continuesHydrationCycle) this.throwIfReservedCommitOpen() + if (!continuesHydrationCycle) this.beginHydrationCycle() const lifecycleGeneration = this.lifecycleGeneration const truncateGeneration = this.sourceTruncateGeneration const activeSubsetOptions = Array.from(this.activeSubsets.values()) @@ -5375,7 +5385,6 @@ class PersistedCollectionRuntime< .map(([key]) => key) this.hydratedDemands.clear() - if (!continuesHydrationCycle) this.hydrationSequence++ const hydrationContext = { suppliedRowKeys: new Set() } this.activeHydrationContext = hydrationContext this.hydratingGeneration = lifecycleGeneration diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index 1570bd4335..15a7648f2f 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -599,7 +599,7 @@ function createLocalCoordinatorHarness() { } type FailStopCoordinatorHarness = PersistedCollectionCoordinator & { - emit: (payload: unknown, senderId?: string) => void + emit: (payload: unknown, senderId?: string, storageId?: string) => void publishCalls: Array> remoteEnsureCalls: Array remoteReleaseCalls: Array @@ -645,11 +645,11 @@ function createFailStopCoordinatorHarness( seq: tx.seq, latestRowVersion: tx.rowVersion, }), - emit: (payload, senderId = `remote-fail-stop-peer`) => { + emit: (payload, senderId = `remote-fail-stop-peer`, storageId) => { subscriber?.({ v: 1, dbName: `test-db`, - collectionId, + collectionId: storageId ?? collectionId, senderId, ts: Date.now(), payload, @@ -17713,7 +17713,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { truncateReplay: `none` | `before` | `after` commitTiming: PersistedReadTiming ending: PersistedReadEnding - followUp: `none` | `reset then pre-reset commit` + followUp: `none` | `reset then pre-reset commit` | `retired-storage reset` } type PersistedReadReach = | `route performed a persisted read` @@ -17726,6 +17726,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { | `reservation opened before startup settled` | `peer change announced while reserved` | `reset and pre-reset commit announced while waiting` + | `retired-storage reset announced while reserved` // The rows of the persisted-reads histories. Each history gets fresh // objects, so no history can observe another's. @@ -17803,7 +17804,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // is open (maintainer decision on PR #2265). Its commit rejects, its // first step applies nothing, and the reset runs without waiting. const resetAbandons = - waits && (route === `collection-reset` || followUp !== `none`) + waits && + (route === `collection-reset` || + followUp === `reset then pre-reset commit`) const hydrationWaits = waits && !recoveryStraddles && !resetAbandons const hydrationStraddles = straddles || recoveryStraddles || resetAbandons const noticeWaited = waits && !recoveryStraddles @@ -17900,7 +17903,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // pre-reset notice. const expectedReadCount = (!readsSomething ? 0 : route === `routed subset` ? 2 : 1) + - (followUp === `none` ? 0 : 2) + (followUp === `reset then pre-reset commit` ? 2 : 0) const applyDurableEvent = (rows: Map, event: string) => { if (event === `replacement`) { rows.clear() @@ -18268,7 +18271,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { [ ...(ending === `cleanup` || ((route === `targeted` || ending === `demand aborted`) && - followUp === `none`) + followUp !== `reset then pre-reset commit`) ? [] : ([`route performed a persisted read`] as const)), ...(waits && @@ -18301,9 +18304,12 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ...(route === `targeted` && waits ? ([`peer change announced while reserved`] as const) : []), - ...(followUp !== `none` && waits + ...(followUp === `reset then pre-reset commit` && waits ? ([`reset and pre-reset commit announced while waiting`] as const) : []), + ...(followUp === `retired-storage reset` + ? ([`retired-storage reset announced while reserved`] as const) + : []), ], ) reach.beginHistory() @@ -18464,15 +18470,27 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { } routeAdmitted = true let ownBaselineRejection: string | undefined + let ownerVisibleAtSettlement: boolean | undefined if ( route === `subset` || route === `complete subset` || route === `routed subset` ) { + const load = collection._sync.loadSubset(options) retainOperation( - collection._sync.loadSubset(options), + load, ending === `demand aborted` ? `rejected` : `fulfilled`, ) + // Law 7 at the load's settlement: a routed load settles only after + // the owner's broadcast rows are public. + if (route === `routed subset`) { + void Promise.resolve(load).then( + () => { + ownerVisibleAtSettlement = collection.has(remoteRow.id) + }, + () => undefined, + ) + } } else if (route === `source baseline`) { retainOperation(sourceParams.metadata!.persistence!.hydrateBaseline()) } else if (route === `collection-reset`) { @@ -18533,6 +18551,16 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // superseded. The reset removed that commit's row from durable storage. const deliverFollowUp = () => { if (followUp === `none`) return + // A reset for a storage generation this Collection no longer uses + // does not apply here, so it abandons nothing. + if (followUp === `retired-storage reset`) { + coordinator.emit( + { type: `collection:reset`, schemaVersion: 1, resetEpoch: 1 }, + undefined, + `retired-storage`, + ) + return + } coordinator.emit({ type: `collection:reset`, schemaVersion: 1, @@ -18686,7 +18714,11 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { } if (followUp !== `none`) { deliverFollowUp() - reach.reach(`reset and pre-reset commit announced while waiting`) + reach.reach( + followUp === `retired-storage reset` + ? `retired-storage reset announced while reserved` + : `reset and pre-reset commit announced while waiting`, + ) } await persistenceQuiesced(`reservation open, route waiting`) observeWhileReserved() @@ -18846,9 +18878,10 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { : {}), ...(hydrationStraddles ? { rowsWhileReserved } : {}), ...(route === `targeted` && waits ? { peerWhileReserved } : {}), - ...(followUp !== `none` + ...(followUp === `reset then pre-reset commit` ? { publishedSuperseded: published.has(supersededRow.id) } : {}), + ...(route === `routed subset` ? { ownerVisibleAtSettlement } : {}), ...(ending === `demand aborted` ? { abortedBeforeEnd } : {}), ...(ending === `source awaits own baseline` ? { ownBaselineRejection } @@ -18898,7 +18931,12 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ...(route === `targeted` && waits ? { peerWhileReserved: peerRow } : {}), - ...(followUp !== `none` ? { publishedSuperseded: false } : {}), + ...(followUp === `reset then pre-reset commit` + ? { publishedSuperseded: false } + : {}), + ...(route === `routed subset` + ? { ownerVisibleAtSettlement: true } + : {}), ...(ending === `demand aborted` ? { abortedBeforeEnd: `rejected` } : {}), @@ -19078,6 +19116,11 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { (ending === `commits` || ending === `abandoned`) ? ([`reset then pre-reset commit`] as const) : []), + ...(syncMode === `eager` && + waits && + (ending === `commits` || ending === `abandoned`) + ? ([`retired-storage reset`] as const) + : []), ] as const ).map((followUp): PersistedReadHistory => ({ syncMode, From 7a6f32fd762e8d57a3e91d9bb01954a47c3bc2c6 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 27 Oct 2026 22:11:51 -0600 Subject: [PATCH 18/24] test(browser-sqlite): fold only updates for rows a tab holds A random CI campaign found a history the model got wrong: a peer committed an update and then a delete of a row, and a lagging tab then delivered only the update. Electric's updates carry only changed columns by default, so an update cannot create a row the tab no longer holds, and Electric drops it. The fold now applies the same rule; a resumed tab in that position stays the #2267 known gap. Both mutants still fail both campaigns. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 4 ++ .../tests/multi-tab-electric-oracle.test.ts | 44 +++++++++++-------- 2 files changed, 30 insertions(+), 18 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index d1b956d0a6..8063cab4c5 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -3099,6 +3099,10 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. error on its health node, which the histories judge, and a loading tab goes to `error` (#2168). A reset that waits and accepts the stale commit fails both campaigns. + Electric's updates carry only changed columns by default, so an update + for a row the tab does not hold applies nothing; the fold follows that + rule (a CI random campaign found the history where a lagging tab's + update follows a peer's delete). A delivery may also commit only a prefix of what a stream owes, as a server answering at an earlier moment does, so lagging commits are generated. The final checkpoint requires the server head (Law 2). diff --git a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts index 02dd86fc30..25596c177d 100644 --- a/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts +++ b/packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts @@ -1025,25 +1025,33 @@ describe(`multi-tab Electric over one persisted database`, () => { model.version = count > 0 ? owed[count - 1]!.version : model.version if (messages.length > 0) model.open ??= [] } - // Known gap: a resumed tab that lags behind a peer can receive an - // update for a row the peer's commit already deleted from the shared - // durable rows. Electric's resume check then reports an unseen row and - // puts the Collection in `error` (issue #2267). - // Generated campaigns leave that history out; the pinned replay below - // keeps it. - if (model.resumed && !model.truncates) { - const held = new Set(durable.keys()) - for (const message of [...(model.open ?? []), ...messages]) { - if (!(`key` in message)) continue - const { id } = message.value - if (message.headers.operation === `update` && !held.has(id)) { - onKnownGap() - } - if (message.headers.operation === `delete`) held.delete(id) - else held.add(id) - } + // Electric's updates carry only changed columns (the shape's default + // replica), so an update cannot create a row the tab does not hold, + // and Electric drops it. The tab holds the durable rows, or, inside a + // refetch, the snapshot it has received. Known gap: a resumed tab + // instead reports an unseen row and fails (issue #2267). Generated + // campaigns leave that history out; the pinned replay below keeps it. + const held = new Set(model.truncates ? [] : durable.keys()) + const track = (message: Message) => { + if (!(`key` in message)) return + if (message.headers.operation === `delete`) + held.delete(message.value.id) + else held.add(message.value.id) } - model.open?.push(...messages) + for (const message of model.open ?? []) track(message) + const applied = messages.filter((message) => { + if ( + `key` in message && + message.headers.operation === `update` && + !held.has(message.value.id) + ) { + if (model.resumed && !model.truncates) onKnownGap() + return false + } + track(message) + return true + }) + model.open?.push(...applied) const complete = commits if (messages.length === 0 && !complete) return tab.deliver( From 383ce484128013f3b13231ba8f42a92398cba04a Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 07:35:23 -0600 Subject: [PATCH 19/24] fix(persistence): close the sixth review's reservation gaps - A follower's routed load checks for a new reservation after each flush, so a reservation that opens before the flush cannot settle the load before the owner's broadcast applies. - A transaction that a peer reset abandoned cannot reserve again. - A reset's reload skips an aborted demand still waiting for its turn. - Electric's begin and commit helpers own transactionStarted, so the progressive swap's reserved transaction is abandoned when the source stops. - Admission waiters are plain release callbacks; their rejection path was unreachable. The persisted-reads grammar gains a re-reserve follow-up and a reset follow-up for aborted demands. The Electric descriptor-isolation oracle gains a failed-swap give-up witness. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 21 +++++- .../src/persisted.ts | 54 ++++++++-------- .../tests/persisted-oracle.test.ts | 64 +++++++++++++++---- .../electric-db-collection/src/electric.ts | 12 ++-- ...ectric-descriptor-isolation-oracle.test.ts | 58 ++++++++++++++++- 5 files changed, 156 insertions(+), 53 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 8063cab4c5..e0fec281fc 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2995,7 +2995,13 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. on `851e8d9da` the abandoned admission cells failed with a rejected load. The ending is commits, abandoned, cleanup, an aborted demand, or the source awaiting its own - baseline. The follow-up is a reset then a pre-reset commit. + baseline. The follow-up is a reset then a pre-reset commit, or a + reset, a new reservation, and then the pre-reset commit. A reservation + that a reset abandoned cannot reserve again; on `7a6f32fd7` 8 + re-reserve cells failed because a later hydration waited for the + abandoned transaction. An aborted demand still waiting for its turn + also receives the reset follow-up; on `7a6f32fd7` the reset's reload + read for it, and 4 cells failed on the read count. Observations: receipts, final rows, row metadata, cursor, status, reads during the reservation, upstream requests, pullSince count, rows while waiting, and every read against the fold of the durable @@ -3013,7 +3019,11 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. follower's flush and its wait check is a one-microtask window no timing reaches; a temporary hook that ended the reservation there showed the old wait loop settling early in 4 straddled cells and the - current loop keeping the owner's row public. A peer reset for a + current loop keeping the owner's row public. A new reservation that + opens between the loop's wait and its flush defers the owner's + broadcast again. The loop checks for a reservation after each flush; a + temporary hook that opened one there showed the loop on `7a6f32fd7` + settling the routed load in 2 cells before the owner's row was public. A peer reset for a storage generation this Collection no longer uses abandons nothing. Known limit (maintainer decision on PR #2265): Electric publishes a transaction's txids as stream evidence before persistence accepts its @@ -4004,7 +4014,12 @@ public and durable state. A second delivery lands during rotation, before the replacement stream starts; it rejects old callback admission into the new cache. Initial scoped recovery also forgets imported old txid evidence, and retiring an incomplete old stream transaction releases its commit turn so a -new demand can start. These histories killed the one-phase restart, retained +new demand can start. A progressive atomic swap that throws in `getKey` +leaves its reserved transaction open; when the SDK then gives up, Electric +must abandon it, and a subset load that waited runs. On `7a6f32fd7` the swap +did not mark its transaction open, so no stop could abandon it and the load +waited until its checkpoint timed out. The metadata-only commit shares the +same begin and commit helpers but has no fallible step between them. These histories killed the one-phase restart, retained acknowledgement, and open-transaction designs at their named checkpoints. The Electric descriptor owner also holds rotation while aborting one old demand: that demand rejects promptly and a sibling later applies fresh source rows. diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 3925201dda..c00c362182 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -1302,10 +1302,8 @@ type OpenSyncTransaction = Omit< // Settles once the reservation ends; every wait shares it. reservationEnded?: Promise hasDependentSuccessor: boolean - publicationAdmissionWaiters?: Set<{ - resolve: () => void - reject: (error: unknown) => void - }> + // Every outcome of the transaction releases its admission waiters. + publicationAdmissionWaiters?: Set<() => void> terminalFailure?: { error: unknown } // Set just before the transaction is handed to the Collection; from then // on core holds its keys itself. @@ -2121,7 +2119,8 @@ class PersistedCollectionRuntime< await this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) // With no reservation left, a head still queued is held by other work, // such as a claim recovery, which flushes it. - if (!reservation && this.queuedTxCommitted[0] === head) break + if (this.queuedTxCommitted[0] === head && !this.waitForReservedCommit?.()) + break } // A broadcast that failed to apply made the runtime terminal; its rows // are not visible, so the load must not settle successfully. @@ -5359,7 +5358,10 @@ class PersistedCollectionRuntime< if (!continuesHydrationCycle) this.beginHydrationCycle() const lifecycleGeneration = this.lifecycleGeneration const truncateGeneration = this.sourceTruncateGeneration - const activeSubsetOptions = Array.from(this.activeSubsets.values()) + // An aborted demand still waiting for its turn loads nothing. + const activeSubsetOptions = Array.from(this.activeSubsets.values()).filter( + (options) => !options.signal?.aborted, + ) if (restoreResetBaseline && activeSubsetOptions.length === 0) { activeSubsetOptions.push({}) } @@ -5802,15 +5804,10 @@ function createWrappedSyncConfig< transactionStack[transactionStack.length - 1] const settlePublicationAdmissionWaiters = ( transaction: OpenSyncTransaction, - error?: unknown, ) => { const waiters = transaction.publicationAdmissionWaiters transaction.publicationAdmissionWaiters = undefined - if (!waiters) return - for (const waiter of waiters) { - if (error === undefined) waiter.resolve() - else waiter.reject(error) - } + waiters?.forEach((release) => release()) } const bindToCurrentHydration = ( transaction: OpenSyncTransaction, @@ -5926,7 +5923,7 @@ function createWrappedSyncConfig< error: unknown, ) => { removePendingPublicationTransaction(transaction) - settlePublicationAdmissionWaiters(transaction, error) + settlePublicationAdmissionWaiters(transaction) return createHandledRejection(error) } const updateMetadataWrite = ( @@ -5975,14 +5972,13 @@ function createWrappedSyncConfig< const waiters = (transaction.publicationAdmissionWaiters ??= new Set()) const settle = () => { - waiters.delete(waiter) + waiters.delete(settle) options.signal?.removeEventListener(`abort`, settle) acquisition.cancelAdmissionWait = undefined resolve() } - const waiter = { resolve: settle, reject: settle } if (options.signal?.aborted) return settle() - waiters.add(waiter) + waiters.add(settle) options.signal?.addEventListener(`abort`, settle, { once: true }) acquisition.cancelAdmissionWait = settle }) @@ -6109,10 +6105,9 @@ function createWrappedSyncConfig< // cleanup discards it. Every outcome closes the reservation. return (transaction.reservationEnded ??= new Promise( (resolve) => { - ;(transaction.publicationAdmissionWaiters ??= new Set()).add({ + ;(transaction.publicationAdmissionWaiters ??= new Set()).add( resolve, - reject: () => resolve(), - }) + ) }, )) }, @@ -6141,7 +6136,12 @@ function createWrappedSyncConfig< }, reserveCommitTurn: () => { const openTransaction = getOpenTransaction() - if (openTransaction && !openTransaction.internal) + // A transaction a reset abandoned cannot reserve again. + if ( + openTransaction && + !openTransaction.internal && + !openTransaction.terminalFailure + ) openTransaction.reservesCommitTurn = true }, hydrateBaseline: async () => { @@ -6499,7 +6499,7 @@ function createWrappedSyncConfig< bindToCurrentHydration(openTransaction) assertHydrationSequenceCurrent(openTransaction) } catch (error) { - settlePublicationAdmissionWaiters(openTransaction, error) + settlePublicationAdmissionWaiters(openTransaction) throw error } const transaction = { @@ -6577,11 +6577,9 @@ function createWrappedSyncConfig< let applied: Promise try { applied = runtime.applyHydrationBufferedTransaction(transaction) - } catch (error) { - settlePublicationAdmissionWaiters(openTransaction, error) - throw error + } finally { + settlePublicationAdmissionWaiters(openTransaction) } - settlePublicationAdmissionWaiters(openTransaction) settleRuntimeTransaction(openTransaction, applied) return visibleAfterDurable(openTransaction, applied) }, @@ -6756,9 +6754,8 @@ function createWrappedSyncConfig< // open old-source transaction is abandoned before the new cache binds. if (!sourceCleanupPromise) { sourceSession++ - const abortError = new SyncTransactionAbortedError() for (const transaction of transactionStack) { - settlePublicationAdmissionWaiters(transaction, abortError) + settlePublicationAdmissionWaiters(transaction) } transactionStack.length = 0 const retiredResult = sourceResult @@ -6834,7 +6831,6 @@ function createWrappedSyncConfig< // reenters cleanup before returning, wait for that exact synchronous // entry to publish its cleanup callback before sampling the result. const sourceSyncEntry = activeSourceSyncEntry - const cleanupError = new SyncTransactionAbortedError() for (const acquisition of acquisitions.values()) { acquisition.cancelAdmissionWait?.() } @@ -6842,7 +6838,7 @@ function createWrappedSyncConfig< acquisitions.clear() pendingPublicationTransactions.length = 0 for (const transaction of transactionStack) { - settlePublicationAdmissionWaiters(transaction, cleanupError) + settlePublicationAdmissionWaiters(transaction) } transactionStack.length = 0 let sourceCleanup: Promise diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index 15a7648f2f..1ec43add7e 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -17713,7 +17713,11 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { truncateReplay: `none` | `before` | `after` commitTiming: PersistedReadTiming ending: PersistedReadEnding - followUp: `none` | `reset then pre-reset commit` | `retired-storage reset` + followUp: + | `none` + | `reset then pre-reset commit` + | `reset, re-reserve, then pre-reset commit` + | `retired-storage reset` } type PersistedReadReach = | `route performed a persisted read` @@ -17806,7 +17810,8 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { const resetAbandons = waits && (route === `collection-reset` || - followUp === `reset then pre-reset commit`) + followUp === `reset then pre-reset commit` || + followUp === `reset, re-reserve, then pre-reset commit`) const hydrationWaits = waits && !recoveryStraddles && !resetAbandons const hydrationStraddles = straddles || recoveryStraddles || resetAbandons const noticeWaited = waits && !recoveryStraddles @@ -17900,10 +17905,16 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ending !== `demand aborted` // A routed subset reads its local rows, then rereads to admit the // owner's row; a reset follow-up reloads, and so does the fenced - // pre-reset notice. + // pre-reset notice. Those reloads read only active demand: an eager + // Collection's baseline, or a subset whose demand was not aborted. An + // aborted demand loads no rows. const expectedReadCount = (!readsSomething ? 0 : route === `routed subset` ? 2 : 1) + - (followUp === `reset then pre-reset commit` ? 2 : 0) + ((followUp === `reset then pre-reset commit` || + followUp === `reset, re-reserve, then pre-reset commit`) && + (syncMode === `eager` || readsSomething) + ? 2 + : 0) const applyDurableEvent = (rows: Map, event: string) => { if (event === `replacement`) { rows.clear() @@ -18271,7 +18282,8 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { [ ...(ending === `cleanup` || ((route === `targeted` || ending === `demand aborted`) && - followUp !== `reset then pre-reset commit`) + followUp !== `reset then pre-reset commit` && + followUp !== `reset, re-reserve, then pre-reset commit`) ? [] : ([`route performed a persisted read`] as const)), ...(waits && @@ -18304,7 +18316,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ...(route === `targeted` && waits ? ([`peer change announced while reserved`] as const) : []), - ...(followUp === `reset then pre-reset commit` && waits + ...((followUp === `reset then pre-reset commit` || + followUp === `reset, re-reserve, then pre-reset commit`) && + waits ? ([`reset and pre-reset commit announced while waiting`] as const) : []), ...(followUp === `retired-storage reset` @@ -18566,6 +18580,14 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { schemaVersion: 1, resetEpoch: 1, }) + // The source reserves its turn again on the transaction the reset + // abandoned; a dead transaction cannot reserve. + if ( + followUp === `reset, re-reserve, then pre-reset commit` && + reservationOpen + ) { + sourceParams.metadata!.persistence!.reserveCommitTurn() + } const position = nextPeerPosition() coordinator.emit({ type: `tx:committed`, @@ -18712,7 +18734,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { await peerCommit(peerRow, `peer-change`) reach.reach(`peer change announced while reserved`) } - if (followUp !== `none`) { + const announceFollowUp = () => { deliverFollowUp() reach.reach( followUp === `retired-storage reset` @@ -18720,6 +18742,10 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { : `reset and pre-reset commit announced while waiting`, ) } + // An aborted demand meets its follow-up after the abort (below). + if (followUp !== `none` && ending !== `demand aborted`) { + announceFollowUp() + } await persistenceQuiesced(`reservation open, route waiting`) observeWhileReserved() peerWhileReserved = stripVirtualProps(collection.get(peerRow.id)) @@ -18740,6 +18766,12 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ), `aborted demand settles before the reservation ends`, ) + // A reset after the abort reloads only demand that is still + // active, so the aborted demand loads nothing. + if (followUp !== `none`) { + announceFollowUp() + await persistenceQuiesced(`follow-up after the abort applied`) + } } if (ending === `source awaits own baseline`) { ownBaselineRejection = await atPersistedOracleCheckpoint( @@ -18878,7 +18910,8 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { : {}), ...(hydrationStraddles ? { rowsWhileReserved } : {}), ...(route === `targeted` && waits ? { peerWhileReserved } : {}), - ...(followUp === `reset then pre-reset commit` + ...(followUp === `reset then pre-reset commit` || + followUp === `reset, re-reserve, then pre-reset commit` ? { publishedSuperseded: published.has(supersededRow.id) } : {}), ...(route === `routed subset` ? { ownerVisibleAtSettlement } : {}), @@ -18931,7 +18964,8 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ...(route === `targeted` && waits ? { peerWhileReserved: peerRow } : {}), - ...(followUp === `reset then pre-reset commit` + ...(followUp === `reset then pre-reset commit` || + followUp === `reset, re-reserve, then pre-reset commit` ? { publishedSuperseded: false } : {}), ...(route === `routed subset` @@ -19111,10 +19145,14 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ( [ `none`, - ...(syncMode === `eager` && - (commitTiming === `after read` || waits) && - (ending === `commits` || ending === `abandoned`) - ? ([`reset then pre-reset commit`] as const) + ...((syncMode === `eager` && + (commitTiming === `after read` || waits) && + (ending === `commits` || ending === `abandoned`)) || + ending === `demand aborted` + ? ([ + `reset then pre-reset commit`, + `reset, re-reserve, then pre-reset commit`, + ] as const) : []), ...(syncMode === `eager` && waits && diff --git a/packages/electric-db-collection/src/electric.ts b/packages/electric-db-collection/src/electric.ts index 76f64a23db..9605b30a19 100644 --- a/packages/electric-db-collection/src/electric.ts +++ b/packages/electric-db-collection/src/electric.ts @@ -2112,8 +2112,10 @@ function createElectricSync>( usesBaseline = false } + // The commit consumes the transaction even if it throws. const commitSourceTransaction = (): SyncAppliedReceipt => { const committedPresenceRevision = pendingPresenceRevision + transactionStarted = false const applied = commit() if (metadata?.persistence) { const retireAppliedPresence = () => { @@ -2129,8 +2131,10 @@ function createElectricSync>( } // Subset hydration must not split a stream transaction it overlaps. + // Every reserved transaction is marked open, so a stop can abandon it. const beginSourceTransaction = () => { begin() + transactionStarted = true metadata?.persistence?.reserveCommitTurn() } @@ -2371,7 +2375,6 @@ function createElectricSync>( if (freshSnapshotPending) { freshSnapshotPending = false beginSourceTransaction() - transactionStarted = true truncate() recordPendingTruncate() syncedKeys.clear() @@ -2476,7 +2479,6 @@ function createElectricSync>( // Normal processing: write changes immediately if (!transactionStarted) { beginSourceTransaction() - transactionStarted = true } processChangeMessage(message) @@ -2535,7 +2537,6 @@ function createElectricSync>( // Start a transaction and truncate the collection if (!transactionStarted) { beginSourceTransaction() - transactionStarted = true } truncate() @@ -2618,8 +2619,7 @@ function createElectricSync>( bufferedMsg.headers.patterns, begin, write, - // The swap already opened a transaction, even though the - // normal-stream transactionStarted flag is still false. + // The swap already opened a transaction. true, (rowId) => { recordPendingPresence(rowId, false) @@ -2651,8 +2651,6 @@ function createElectricSync>( if (!isResettingSnapshot || finishesReset) { stageResumeMetadata() } - // The commit consumes the transaction even if it throws. - transactionStarted = false applied = commitSourceTransaction() } else if (commitPoint === `up-to-date` && metadata) { beginSourceTransaction() diff --git a/packages/electric-db-collection/tests/electric-descriptor-isolation-oracle.test.ts b/packages/electric-db-collection/tests/electric-descriptor-isolation-oracle.test.ts index 131b65a619..f3473e9b5e 100644 --- a/packages/electric-db-collection/tests/electric-descriptor-isolation-oracle.test.ts +++ b/packages/electric-db-collection/tests/electric-descriptor-isolation-oracle.test.ts @@ -129,6 +129,7 @@ function expectWholeTagRecovery( } type StreamHarness = { send: (messages: Array>) => void + giveUp: (error: Error) => void unsubscribe: ReturnType holdSnapshots: boolean ignoreAbort: boolean @@ -167,11 +168,15 @@ vi.mock(`@electric-sql/client`, async () => { ) let harness: StreamHarness | undefined return { - subscribe: (send: StreamHarness[`send`]) => { + subscribe: ( + send: StreamHarness[`send`], + giveUp: StreamHarness[`giveUp`], + ) => { harness = { // Deliberately retain the raw callback for late-delivery tests. // Release calls are observed separately from these injections. send, + giveUp, unsubscribe, holdSnapshots: holdNewSnapshots, ignoreAbort: false, @@ -1645,6 +1650,57 @@ fixedCase( }, ) +// A progressive Collection opens its atomic swap as a reserved transaction. +// A throw inside the swap leaves that transaction open. When the SDK then +// gives up, the source has stopped for good and must abandon the transaction, +// as for any other reserved transaction. Otherwise a subset load that waits +// for the reservation never runs. +fixedCase( + `releases a waiting subset when the SDK gives up after a failed atomic swap`, + async () => { + const { adapter } = tagPersistence() + let poisoned = false + const collection = createCollection( + persistedCollectionOptions< + TestRow, + string | number, + never, + ElectricCollectionUtils + >({ + ...descriptor(`once-spread`, false, `progressive`), + id: `failed-atomic-swap`, + getKey: (row) => { + if (poisoned) throw new Error(`getKey failed`) + return row.id + }, + persistence: { adapter }, + }), + ) + let demand: Promise | undefined + try { + collection.startSyncImmediate() + await vi.waitFor(() => expect(streams).toHaveLength(1)) + const stream = streams[0]! + stream.send([insert(1, `buffered`)]) + poisoned = true + expect(() => stream.send([upToDate])).toThrow(`getKey failed`) + demand = Promise.resolve(collection._sync.loadSubset({ limit: 1 })).then( + () => `ran`, + () => `rejected`, + ) + stream.giveUp(new Error(`SDK gave up`)) + // An abandon releases a waiting load as a commit does, so it runs. + expect( + await atCheckpoint(demand, `subset after the failed atomic swap`), + ).toBe(`ran`) + } finally { + poisoned = false + await atCheckpoint(collection.cleanup(), `failed atomic swap cleanup`) + await demand + } + }, +) + fixedCase.each([`eager`, `on-demand`, `progressive`] as const)( `delivers abandoned tag-recovery callbacks after replacement starts in %s mode`, async (syncMode) => { From 9c799e284d872d11f77af24c1169f941a9acad62 Mon Sep 17 00:00:00 2001 From: "autofix-ci[bot]" <114827586+autofix-ci[bot]@users.noreply.github.com> Date: Wed, 28 Oct 2026 13:37:00 +0000 Subject: [PATCH 20/24] ci: apply automated fixes --- packages/db-sqlite-persistence-core/src/persisted.ts | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index c00c362182..9813328721 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -2119,7 +2119,10 @@ class PersistedCollectionRuntime< await this.applyMutex.run(() => this.flushQueuedTxCommittedUnsafe()) // With no reservation left, a head still queued is held by other work, // such as a claim recovery, which flushes it. - if (this.queuedTxCommitted[0] === head && !this.waitForReservedCommit?.()) + if ( + this.queuedTxCommitted[0] === head && + !this.waitForReservedCommit?.() + ) break } // A broadcast that failed to apply made the runtime terminal; its rows From f67c9f5025284c8b3d298605e163cccc72a1986e Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 07:55:37 -0600 Subject: [PATCH 21/24] fix(persistence): retire an aborted demand while it waits for its turn An aborted subset load that waited for its hydration turn stayed in the active subsets until its turn came. Reloads and targeted invalidation treated it as live demand, so a peer change could reach the leader for a demand nobody held. The load now leaves the active subsets when its signal aborts, which replaces the reload-only filter. The persisted-reads grammar gains a peer change after the abort, and judges reach after its comparison so an unexpected read fails as an assertion. The coverage map records the follower window that no public timing reaches. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 20 ++++++++++--- .../src/persisted.ts | 16 +++++++--- .../tests/persisted-oracle.test.ts | 29 +++++++++++++++---- 3 files changed, 52 insertions(+), 13 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index e0fec281fc..2bea6c1cba 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2999,9 +2999,14 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. reset, a new reservation, and then the pre-reset commit. A reservation that a reset abandoned cannot reserve again; on `7a6f32fd7` 8 re-reserve cells failed because a later hydration waited for the - abandoned transaction. An aborted demand still waiting for its turn - also receives the reset follow-up; on `7a6f32fd7` the reset's reload - read for it, and 4 cells failed on the read count. + abandoned transaction. An aborted demand still waiting for its turn is + no longer active demand. It receives the reset follow-up, or a peer's + targeted change for a row no live demand covers; nothing reads or asks + the leader for it. On `7a6f32fd7`, 12 of the 24 aborted-demand cells + fail: the reset's reload read for it, and the peer change sent a + pullSince past the bound. Filtering aborted demand only from reloads + still fails the 4 peer-change cells that wait at the mutex turn or + scope entry. Observations: receipts, final rows, row metadata, cursor, status, reads during the reservation, upstream requests, pullSince count, rows while waiting, and every read against the fold of the durable @@ -3023,7 +3028,14 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. opens between the loop's wait and its flush defers the owner's broadcast again. The loop checks for a reservation after each flush; a temporary hook that opened one there showed the loop on `7a6f32fd7` - settling the routed load in 2 cells before the owner's row was public. A peer reset for a + settling the routed load in 2 cells before the owner's row was public. + Not reached by a public timing: that history needs the reservation to + end in the same microtask window, between the loop's first flush and + its check. A reservation lasts through its own durable write, so a + held write after the commit does not open the window; a follow-up that + held an unreserved write and then reserved again passed on both loops. + The missing witness is a source that ends its reservation in a + microtask after a publication from the follower's first flush. A peer reset for a storage generation this Collection no longer uses abandons nothing. Known limit (maintainer decision on PR #2265): Electric publishes a transaction's txids as stream evidence before persistence accepts its diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index c00c362182..4ab4ccf530 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -2902,6 +2902,15 @@ class PersistedCollectionRuntime< // Wire admission failures are permanent input errors, not transport retries. if (routeRemoteDemandDuringHydration) validateRemoteSubsetOptions(options) this.activeSubsets.set(subsetKey, options) + // An aborted demand is not active, even while it waits for its turn. + const retireAbortedDemand = () => { + if (this.activeSubsets.get(subsetKey) === options) { + this.activeSubsets.delete(subsetKey) + } + } + options.signal?.addEventListener(`abort`, retireAbortedDemand, { + once: true, + }) const appliedCursor = this.appliedReceiptSequence try { const hydrated = await this.runHydrationTurn( @@ -2983,6 +2992,8 @@ class PersistedCollectionRuntime< this.activeSubsets.delete(subsetKey) } throw error + } finally { + options.signal?.removeEventListener(`abort`, retireAbortedDemand) } if ( @@ -5358,10 +5369,7 @@ class PersistedCollectionRuntime< if (!continuesHydrationCycle) this.beginHydrationCycle() const lifecycleGeneration = this.lifecycleGeneration const truncateGeneration = this.sourceTruncateGeneration - // An aborted demand still waiting for its turn loads nothing. - const activeSubsetOptions = Array.from(this.activeSubsets.values()).filter( - (options) => !options.signal?.aborted, - ) + const activeSubsetOptions = Array.from(this.activeSubsets.values()) if (restoreResetBaseline && activeSubsetOptions.length === 0) { activeSubsetOptions.push({}) } diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index 1ec43add7e..a34cc1f18f 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -17670,7 +17670,10 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // aborted`); with `source awaits own baseline` the source also awaits // `hydrateBaseline` inside its reservation. A follow-up delivers, while // the route waits, a peer reset and then a pre-reset peer commit whose row - // the reset superseded. + // the reset superseded. After an aborted demand, the follow-up + // `peer change after the abort` delivers a peer's targeted change while the aborted + // load still waits for its turn. No live demand covers that row, so it + // stays out of the Collection and nothing reads for it. // // Two registrations share this one driver and model: the row-algebra cross // (every operation and truncate replay; committed endings; the original @@ -17718,6 +17721,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { | `reset then pre-reset commit` | `reset, re-reserve, then pre-reset commit` | `retired-storage reset` + | `peer change after the abort` } type PersistedReadReach = | `route performed a persisted read` @@ -17731,6 +17735,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { | `peer change announced while reserved` | `reset and pre-reset commit announced while waiting` | `retired-storage reset announced while reserved` + | `peer change announced after the demand aborted` // The rows of the persisted-reads histories. Each history gets fresh // objects, so no history can observe another's. @@ -17830,7 +17835,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // Rows the route adds durably. The owner's row for a routed subset lands // after the follower's own read, so reads never see it. const routeRows = - route === `targeted` || route === `seq-gap deltas` + route === `targeted` || + route === `seq-gap deltas` || + followUp === `peer change after the abort` ? [peerRow] : route === `routed subset` ? [remoteRow] @@ -18324,6 +18331,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ...(followUp === `retired-storage reset` ? ([`retired-storage reset announced while reserved`] as const) : []), + ...(followUp === `peer change after the abort` + ? ([`peer change announced after the demand aborted`] as const) + : []), ], ) reach.beginHistory() @@ -18768,7 +18778,11 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ) // A reset after the abort reloads only demand that is still // active, so the aborted demand loads nothing. - if (followUp !== `none`) { + if (followUp === `peer change after the abort`) { + await peerCommit(peerRow, `peer-change`) + reach.reach(`peer change announced after the demand aborted`) + await persistenceQuiesced(`peer change after the abort applied`) + } else if (followUp !== `none`) { announceFollowUp() await persistenceQuiesced(`follow-up after the abort applied`) } @@ -18817,8 +18831,6 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { } if (followUp !== `none`) await persistenceQuiesced(`follow-up settled`) subscription.unsubscribe() - if (routeReads.length > 0) reach.reach(`route performed a persisted read`) - reach.assertReached() // Observations and the values the model owes. The trace records durable // completions (source steps and the owner's commit) beside persisted @@ -18989,6 +19001,10 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { : {}), } expect(observed).toEqual(expected) + // Reach is judged after the comparison, so an unexpected read fails on + // the read count rather than as an undeclared reach state. + if (routeReads.length > 0) reach.reach(`route performed a persisted read`) + reach.assertReached() } catch (error) { hasPrimaryFailure = true throw error @@ -19154,6 +19170,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { `reset, re-reserve, then pre-reset commit`, ] as const) : []), + ...(ending === `demand aborted` + ? ([`peer change after the abort`] as const) + : []), ...(syncMode === `eager` && waits && (ending === `commits` || ending === `abandoned`) From 4e2d0db3272b7ec9fea668bb36a8b18e48c72de4 Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 08:27:13 -0600 Subject: [PATCH 22/24] test(persistence): reach the follower window without a hook A follower's routed load must settle only after the owner's broadcast applies, even when the source ends its reservation and reserves again while the broadcast waits. The persisted-reads grammar gains a third registration: the owner's reply is held, then the source commits and reserves an empty second transaction after 0 to 8 and 0 to 6 microtasks. The loop from before the re-check settles the load early in the 6 cells that commit 3 microtasks after the reply; all 63 cells pass on the current loop. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 19 +++-- .../tests/persisted-oracle.test.ts | 76 +++++++++++++++++-- 2 files changed, 81 insertions(+), 14 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 2bea6c1cba..e540c51f5f 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -3029,13 +3029,18 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. broadcast again. The loop checks for a reservation after each flush; a temporary hook that opened one there showed the loop on `7a6f32fd7` settling the routed load in 2 cells before the owner's row was public. - Not reached by a public timing: that history needs the reservation to - end in the same microtask window, between the loop's first flush and - its check. A reservation lasts through its own durable write, so a - held write after the commit does not open the window; a follow-up that - held an unreserved write and then reserved again passed on both loops. - The missing witness is a source that ends its reservation in a - microtask after a publication from the follower's first flush. A peer reset for a + A third registration reaches it without a hook. The owner's reply is + held while its broadcast waits; the source commits 0 to 8 microtasks + after the reply and reserves an empty second transaction 0 to 6 + microtasks later. On `7a6f32fd7` the 6 cells that commit 3 microtasks + after the reply settle the routed load before the owner's row is + public; all 63 cells pass on the current loop. A held write after the + commit does not open the window, because a reservation lasts through + its own durable write. + Open (law question on PR #2265): a second reservation that opens 7 or + more microtasks after the commit makes the broadcast wait twice, and + the notice asks the leader once per wait. The pullSince bound counts + one request per waited notice, so those delays are not registered. A peer reset for a storage generation this Collection no longer uses abandons nothing. Known limit (maintainer decision on PR #2265): Electric publishes a transaction's txids as stream evidence before persistence accepts its diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index a34cc1f18f..2bc10ebfdc 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -17673,12 +17673,23 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // the reset superseded. After an aborted demand, the follow-up // `peer change after the abort` delivers a peer's targeted change while the aborted // load still waits for its turn. No live demand covers that row, so it - // stays out of the Collection and nothing reads for it. + // stays out of the Collection and nothing reads for it. For a follower's + // routed subset, `owner replies, then the source reserves again` releases + // the owner's held reply while the owner's broadcast waits for the + // reservation. The source commits a few microtasks later and reserves an + // empty second transaction a few microtasks after that, as a source that + // reacts through promise continuations does. The follower can then find no + // reservation open while the broadcast still waits, and the second + // reservation defers the broadcast again. Law 7 still requires the load to + // settle after the owner's row is public. The delays range over 0 to 8 and + // 0 to 6 microtasks; a later second reservation makes the broadcast wait + // twice, which the pullSince bound does not yet decide. // - // Two registrations share this one driver and model: the row-algebra cross - // (every operation and truncate replay; committed endings; the original - // routes and timings), and the reservation cross (every route, timing, - // ending, and follow-up, with one insert and no truncate replay). The + // Three registrations share this one driver and model: the row-algebra + // cross (every operation and truncate replay; committed endings; the + // original routes and timings), the reservation cross (every route, timing, + // ending, and follow-up, with one insert and no truncate replay), and the + // follower window (the second-reservation delays above). The // controlled coordinator does not prove host delivery; the Browser-host // receiving witness is the multi-tab Electric oracle. type PersistedReadRoute = @@ -17716,12 +17727,17 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { truncateReplay: `none` | `before` | `after` commitTiming: PersistedReadTiming ending: PersistedReadEnding + // For `owner replies, then the source reserves again`: the microtasks + // the source takes after the owner replies to commit its reservation, + // then to reserve a second transaction. + reReserveDelays?: readonly [number, number] followUp: | `none` | `reset then pre-reset commit` | `reset, re-reserve, then pre-reset commit` | `retired-storage reset` | `peer change after the abort` + | `owner replies, then the source reserves again` } type PersistedReadReach = | `route performed a persisted read` @@ -17736,6 +17752,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { | `reset and pre-reset commit announced while waiting` | `retired-storage reset announced while reserved` | `peer change announced after the demand aborted` + | `source reserved again after the owner replied` // The rows of the persisted-reads histories. Each history gets fresh // objects, so no history can observe another's. @@ -18185,11 +18202,16 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { const requestEnsureRemoteSubset = coordinator.requestEnsureRemoteSubset.bind(coordinator) let ownerCommitted = false + const ownerReply = createEventGate() + if (followUp !== `owner replies, then the source reserves again`) { + ownerReply.resolve() + } coordinator.requestEnsureRemoteSubset = async (...args) => { sourceLoads++ if (route === `routed subset` && !ownerCommitted) { ownerCommitted = true await peerCommit(remoteRow, `owner-commit`) + await ownerReply.promise return } return requestEnsureRemoteSubset(...args) @@ -18334,6 +18356,9 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ...(followUp === `peer change after the abort` ? ([`peer change announced after the demand aborted`] as const) : []), + ...(followUp === `owner replies, then the source reserves again` + ? ([`source reserved again after the owner replied`] as const) + : []), ], ) reach.beginHistory() @@ -18610,6 +18635,26 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { deletedKeys: [], }) } + // With `owner replies, then the source reserves again`, the owner's + // reply is released first; the source then ends its reservation and + // reserves an empty second transaction after the history's delays. + const endStraddledReservation = async () => { + if (followUp !== `owner replies, then the source reserves again`) { + return endReservation() + } + const [commitDelay, reserveDelay] = history.reReserveDelays! + ownerReply.resolve() + for (let i = 0; i < commitDelay; i++) await Promise.resolve() + endReservation() + for (let i = 0; i < reserveDelay; i++) await Promise.resolve() + sourceParams.begin() + sourceParams.metadata!.persistence!.reserveCommitTurn() + reservationOpen = true + reach.reach(`source reserved again after the owner replied`) + await persistenceQuiesced(`owner's broadcast waits again`) + reservationOpen = false + commitSourceTransaction() + } let remainingSteps: ReadonlyArray<(typeof sourceSteps)[number]> = commitTiming === `open across startup` ? sourceSteps.slice(1) @@ -18674,7 +18719,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { firstPublication.unsubscribe() await persistenceQuiesced(`straddled hydration finished`) observeWhileReserved() - endReservation() + await endStraddledReservation() remainingSteps = sourceSteps.slice(1) } else if (commitTiming === `open during the read`) { heldRead = { entered: createEventGate(), release: createEventGate() } @@ -18689,7 +18734,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { release.resolve() await persistenceQuiesced(`straddled read finished`) observeWhileReserved() - endReservation() + await endStraddledReservation() remainingSteps = sourceSteps.slice(1) } else { if (commitTiming === `open during pullSince`) { @@ -19191,6 +19236,23 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ) }), ) + // The follower window: a second reservation after the owner replied. + const reReserveHistories = Array.from({ length: 9 }, (_, commitDelay) => + Array.from({ length: 7 }, (_, reserveDelay): PersistedReadHistory => ({ + syncMode: `on-demand`, + route: `routed subset`, + operation: `insert`, + truncateReplay: `none`, + commitTiming: `open during the read`, + ending: `commits`, + followUp: `owner replies, then the source reserves again`, + reReserveDelays: [commitDelay, reserveDelay], + })), + ).flat() + it.each(reReserveHistories)( + `keeps a follower's load behind a second reservation (delays=$reReserveDelays)`, + runPersistedReadHistory, + ) it.each(reservationHistories)( `keeps reserved commits and their neighbours lawful ($syncMode, $route, timing=$commitTiming, ending=$ending, follow-up=$followUp)`, runPersistedReadHistory, From 60adb19763f533cfacc4ab6e9939880d66b59bb1 Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 08:31:05 -0600 Subject: [PATCH 23/24] test(persistence): bound pullSince per wait, not per notice Maintainer decision on PR #2265: a notice that waited may send one more pullSince for each reservation it waited for. The model's bound now counts the second reservation that can hold a follower's owner notice, and the follower-window registration widens its second delay to 0 to 12 microtasks. All 117 cells pass; on 7a6f32fd7 the 12 cells that commit 3 microtasks after the owner's reply settle the routed load early. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 18 +++++++++--------- .../tests/persisted-oracle.test.ts | 18 +++++++++++------- 2 files changed, 20 insertions(+), 16 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index e540c51f5f..36bd88a9ed 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2970,8 +2970,9 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. Further maintainer decisions fix its edges. Readers keep the rows from before a hydration while it waits; a transaction opened after a hydration began straddles it. A notice that waited may be retried as - gap recovery, with at most one more pullSince request and no added - upstream request. A source that stops for good abandons its + gap recovery, with at most one more pullSince request for each + reservation it waited for (maintainer decision on PR #2265) and no + added upstream request. A source that stops for good abandons its reservation. A second registration crosses the routes with the reserving timings, ending, and follow-up dimensions. The routes are subset, complete @@ -3031,16 +3032,15 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. settling the routed load in 2 cells before the owner's row was public. A third registration reaches it without a hook. The owner's reply is held while its broadcast waits; the source commits 0 to 8 microtasks - after the reply and reserves an empty second transaction 0 to 6 - microtasks later. On `7a6f32fd7` the 6 cells that commit 3 microtasks + after the reply and reserves an empty second transaction 0 to 12 + microtasks later. On `7a6f32fd7` the 12 cells that commit 3 microtasks after the reply settle the routed load before the owner's row is - public; all 63 cells pass on the current loop. A held write after the + public; all 117 cells pass on the current loop. A held write after the commit does not open the window, because a reservation lasts through its own durable write. - Open (law question on PR #2265): a second reservation that opens 7 or - more microtasks after the commit makes the broadcast wait twice, and - the notice asks the leader once per wait. The pullSince bound counts - one request per waited notice, so those delays are not registered. A peer reset for a + A second reservation that opens 7 or more microtasks after the commit + holds the broadcast twice, and the notice asks the leader once per + wait, as the bound allows. A peer reset for a storage generation this Collection no longer uses abandons nothing. Known limit (maintainer decision on PR #2265): Electric publishes a transaction's txids as stream evidence before persistence accepts its diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index 2bc10ebfdc..7032ddde4c 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -17616,7 +17616,8 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // the transaction straddles it. // - A notice that waited may be retried as gap recovery, because the // reserved commit lands first and takes the next stream position: at - // most one more pullSince request, and no added upstream request. + // most one more pullSince request for each reservation it waited for, + // and no added upstream request (maintainer decision on PR #2265). // - A source that stops for good abandons its reserved transaction; waiting // hydrations then proceed and the abandoned writes never apply. // - A peer change that starts no hydration is not delayed. A load aborted @@ -17682,8 +17683,8 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // reservation open while the broadcast still waits, and the second // reservation defers the broadcast again. Law 7 still requires the load to // settle after the owner's row is public. The delays range over 0 to 8 and - // 0 to 6 microtasks; a later second reservation makes the broadcast wait - // twice, which the pullSince bound does not yet decide. + // 0 to 12 microtasks. A late second reservation holds the broadcast twice, + // so it may ask the leader twice. // // Three registrations share this one driver and model: the row-algebra // cross (every operation and truncate replay; committed endings; the @@ -17957,15 +17958,18 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { const noWaitPullSinceCalls = route === `seq-gap pull` || route === `seq-gap deltas` ? 1 : 0 // A route notice waits in the waiting timings; a follower's owner notice - // arrives during a straddled reservation and waits there. Each notice - // that waited may ask the leader once more; nothing else may. + // arrives during a straddled reservation and waits there. A notice may + // ask the leader once more for each reservation it waited for; nothing + // else may. A second reservation can hold the owner's notice again. const waitedNotices = (noticeWaited && (route === `full-reload` || route === `seq-gap pull` || route === `seq-gap deltas`)) || (straddles && route === `routed subset`) - ? 1 + ? followUp === `owner replies, then the source reserves again` + ? 2 + : 1 : 0 const expectedSourceLoads = route === `subset` || @@ -19238,7 +19242,7 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { ) // The follower window: a second reservation after the owner replied. const reReserveHistories = Array.from({ length: 9 }, (_, commitDelay) => - Array.from({ length: 7 }, (_, reserveDelay): PersistedReadHistory => ({ + Array.from({ length: 13 }, (_, reserveDelay): PersistedReadHistory => ({ syncMode: `on-demand`, route: `routed subset`, operation: `insert`, From 82f496c52544a5e573802b2ca5d67b38cd417e68 Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 28 Oct 2026 08:36:33 -0600 Subject: [PATCH 24/24] docs(persistence): state the pullSince law as gap filling A tab asks the leader only to fill a gap it has not already filled. The per-wait count is how the oracle checks that law: each reservation a notice waited for can open one gap. Comments and the coverage map only. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 7 ++++--- .../tests/persisted-oracle.test.ts | 15 +++++++++------ 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 36bd88a9ed..d845d4e4c7 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -2970,9 +2970,10 @@ a hostile wrong-answer control, and an explicit statement of remaining limits. Further maintainer decisions fix its edges. Readers keep the rows from before a hydration while it waits; a transaction opened after a hydration began straddles it. A notice that waited may be retried as - gap recovery, with at most one more pullSince request for each - reservation it waited for (maintainer decision on PR #2265) and no - added upstream request. A source that stops for good abandons its + gap recovery. A tab asks the leader only to fill a gap it has not + already filled, and sends no added upstream request. Each reservation + a notice waited for can open one gap, so the oracle checks at most one + more pullSince request per wait (maintainer decision on PR #2265). A source that stops for good abandons its reservation. A second registration crosses the routes with the reserving timings, ending, and follow-up dimensions. The routes are subset, complete diff --git a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts index 7032ddde4c..f95713068e 100644 --- a/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts +++ b/packages/db-sqlite-persistence-core/tests/persisted-oracle.test.ts @@ -17615,9 +17615,11 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { // A hydration that already began when the reservation opened continues; // the transaction straddles it. // - A notice that waited may be retried as gap recovery, because the - // reserved commit lands first and takes the next stream position: at - // most one more pullSince request for each reservation it waited for, - // and no added upstream request (maintainer decision on PR #2265). + // reserved commit lands first and takes the next stream position. A tab + // asks the leader only to fill a gap it has not already filled, and + // sends no added upstream request. Each reservation a notice waited for + // can open one gap, so the oracle allows one more pullSince request per + // wait (maintainer decision on PR #2265). // - A source that stops for good abandons its reserved transaction; waiting // hydrations then proceed and the abandoned writes never apply. // - A peer change that starts no hydration is not delayed. A load aborted @@ -17958,9 +17960,10 @@ describeUnlessOracleReplay(`persistedCollectionOptions`, () => { const noWaitPullSinceCalls = route === `seq-gap pull` || route === `seq-gap deltas` ? 1 : 0 // A route notice waits in the waiting timings; a follower's owner notice - // arrives during a straddled reservation and waits there. A notice may - // ask the leader once more for each reservation it waited for; nothing - // else may. A second reservation can hold the owner's notice again. + // arrives during a straddled reservation and waits there. A tab asks the + // leader only to fill a new gap, and each reservation a notice waited + // for can open one, so the bound is one request per wait; nothing else + // opens a gap. A second reservation can hold the owner's notice again. const waitedNotices = (noticeWaited && (route === `full-reload` ||