Skip to content

Load-subset replay oracle: failed marker load after marker truncate leaves the live query in a terminal error #2239

Closed as not planned

Description

@KyleAMathews

Summary

The random campaign of the load-subset replay refinement oracle fails on main (427a7c31e). A live query ends in the error status after a failed marker load. The oracle's model says no fatal load failed in this history, so it expects the window to recover.

  • Test: packages/db/tests/query/load-subset-replay-refinement-oracle.test.ts › "Ordered windows across truncate replays › keeps the ordered window under load and release faults (random campaign)"
  • Assertion: convergence: terminal error: expected false to be true (line 1875: expect(fatalLoadFailed(), ...).toBe(true))

Reproduce

cd packages/db
TANSTACK_DB_ORACLE_SEED=1930131799 npx vitest run \
  tests/query/load-subset-replay-refinement-oracle.test.ts \
  -t "load and release faults \(random"

fast-check output:

{ seed: 1930131799, path: "33:3:1:4:0:2:3:2:2", endOnFailure: true }
Counterexample: [[{"type":"hold"},{"type":"truncateMarkers"},{"type":"settle","ok":false}],{"direction":"asc","offset":0,"limit":1,"ties":false,"peer":false,"joined":"none","markerMode":"on-demand","includes":true}]
Shrunk 8 time(s)

The shrunk history:

  1. hold the loads;
  2. truncate the marker source;
  3. settle the held loads with a failure.

The scenario is an ascending window (offset 0, limit 1) with on-demand markers and includes, no ties, no peer and no join.

What the oracle checks

After the faults, the recovery step truncates every source, settles, and moves the window again explicitly. A failed replay keeps the last complete publication until a later authoritative replay, and a failed window operation stays private until a later explicit move revalidates it (see ARCHITECTURE.md). After that, the live query must either show the expected window, or be in error only if fatalLoadFailed() (demandFailed || failedWhileLoading) holds. In this history the query is in error, but fatalLoadFailed() is false.

Undecided

Two explanations fit this result, and I haven't decided between them:

  • Production defect: a marker load that fails while held across a marker truncate becomes a terminal query error, though the documented recovery should clear it.
  • Model gap: this failure really is fatal, and fatalLoadFailed() misses the case where a marker load fails after a marker truncate.

The fixed campaign passes. I found this while working on #2220 (branch issue-2220-sync-metadata-held-reads), but it reproduces on a clean main checkout, and that branch doesn't touch this oracle or its production path.

Activity

  1. KyleAMathews commented on Oct 26, 2026

    @KyleAMathews
    CollaboratorAuthor

    Duplicate of #2224, which is fixed by #2229. The shrunk trace here (hold, truncateMarkers, then a failed settle on an ascending limit: 1 window with on-demand markers and includes) passes on main at 1640de834. Details are in #2224.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions