Repository navigation
Conversation
Independent all-source predicate check — latest validationThe acceptance checker now independently executes cumulative merge/split predicates over every original source spike, including omitted evidence and actions from earlier curation rounds. A high Sol 6.1 audit exposed a coherent forged-threshold false acceptance in the previous checker; the original reproduction is now rejected at source spike 4. No incorrect output from the normal curation producer was demonstrated. All 469,725,567 spikes passed the strengthened standalone check: two merges and one threshold split, 46 metadata fields, 20 immutable files, saved generation and complete inventories. This is Carl's earlier converted engineering dataset with unverified exposure, not the newer 500-hour/native-export acceptance. Contract and validation · Pinned result summary. Local full suite: 1,041 passed, one existing skip, including lint/formatting; documentation checks passed. All 52 file-contract cases, 26 literal oracle cases, three actual-controller plans and 18 comparison cases pass. The coherent current/inherited forgeries, omitted-row-only contradiction, comparator equality, nonfinite policy and incorporated source history have regressions. Independent Sol 6.1 high review found no further issue. New commit CI is pending. The same user-action script still runs through both actual controllers with real graphical threshold gestures, unseen source spikes and four curation rounds across three sampling configurations. Existing before/after screenshots and their scope remain below; this checker fix does not change the UI. The separate phylib draft PR #70 remains unchanged: 308 tests and CI passed, and an isolated patched reopen of the copied full sorting had zero tracebacks with unchanged assignments/groups/notes. PR #1428's dependency pin remains unchanged. Follow-up: actual main GUI versus PR replayThe proposed comparison is now automated against unmodified main Main's actual threshold preview, merged unit 705 from 40/44/48, exact source threshold -424.0037077454673. This is the shared ledger's split; the older bounded workflow images below show a separate demonstration on unit 708.
Main after fresh-process reopen. Rendering caveat: main logged seven Feature View errors during curation and five on reopen. Unmodified phylib 2.7.1 attempts PCA when a feature request intersects no stored waveforms; a minimal reproduction confirms the division by zero. The assignment/label workflow still completed and the full comparator passes. Bounded mode filters such IDs before projection. This is curation-output equivalence with a documented baseline renderer failure, not an error-free all-view pass; no #1428 code change was needed. Full main-GUI record · Exact comparison summary · Updated user-test protocol. Independent high-effort audit confirmed the result and limitations. This earlier validation-only commit passed local documentation checks; the permanent executable test follow-up is recorded above. Carl's manual usability/scientific review, his newer 500-hour dataset and Brandon's native-export acquisition evidence remain separate. Update: graphical amplitude splitting and curation comparisonImplemented on this same PR branch at These actual Qt captures show before the action → preview → after the split in the current implementation, on a disposable copy of Carl's earlier 112.859-clock-hour converted test export, unit 708 after replay. They are a separate demonstration from the comparison ledger, not a main-versus-PR visual comparison. Exposure is an unverified test conversion; this is not the new 500-hour recording. Before the threshold action:
Preview: orange points and matching waveform highlights:
After K: the two retained children:
The screenshot split retained 3721 and 3643 spikes, with 34 highlighted waveform traces. All 88 input package files retained their hashes. The capture tool uses a temporary copy. Independent complete-data check: a separate shared ledger exercised merge → graphical split → merge, undo/redo and history branching, GOOD/MUA/notes, and save/reopen. Full bounded replay matched the ordinary Supervisor reference on all 469,725,567 spikes, with zero partition or manual-label mismatches. That earlier ordinary reference ran complete curation commands without GUI views, on the same PR code; the subsequent actual main GUI comparison is recorded above. A further no-edit prepare/save/apply cycle preserved all 18 NPY files and both manual-label tables byte-for-byte. All 67 original download files were unchanged. Local validation: 960 tests passed, one existing skip; lint, formatting and strict documentation checks passed; independent Sol 6.1-high audit found no remaining concrete blocker. Final cross-platform CI, documentation and wheel checks are green; spelling checks also passed. User-test protocol and evidence · Exact shared ledger · Comparison summary Earlier unit-metadata and Unit Stability galleriesUpdate: all seven items in the unit-metadata PDFImplemented on the same PR branch at
Existing saved bounded sessions need an explicit one-time migration to adopt the metadata policy; legacy journal meaning is preserved until then. Close the GUI, call Current real camera capture, units 24, 26, 27 and 32 on Carl's earlier downloaded 112.9-hour converted test export (declared exposure unverified):
Separate generated 513-hour stress fixture: twelve selected units, 6156 native five-minute bins versus 3078 ten-minute display bins. This is synthetic, not Carl's new 500-hour recording.
Validation: 936 distinct tests covered, final cross-platform CI/docs/build green, independent Sol-high audit clear. An independent process checked all 469,725,567 assignments in each of three merge → split → merge rounds, all 45 source table names, explicit labels, two accepted/91 pending suggestions, and stable 892-byte params. All 67 original files and 12 immutable files per output retained their hashes. Legacy migration preserved assignments, reviews, the original manifest and explicit GOOD labels. A combined GUI/data harness reached PASS then exited 143 with unknown SIGTERM source; the independent data-only verifier, captures and test suites exited normally. Seven-item evidence and limitations · CI. Historical first comparison galleryThe earlier text/images below document the first implementation. Its legend and eight-unit display cap have since been replaced by the refinements above. Unit Stability: cluster comparisons and optional plotsCarl's two requests are implemented on this PR branch. Unit Stability overlays selected clusters with the same IDs/colors as Cluster, Similarity and Merge views. Its dock menu now has saved Show firing rate, Show amplitude, and Show depth toggles. Up to eight clusters are displayed, with an explicit notice for larger selections; this never changes merge membership. Counts/rates still come from complete-source exposure-aware summaries. These are actual Qt captures of the same earlier downloaded Carl test export, with clusters 40, 44 and 48 selected and the same 2100 × 1200 window. Its elapsed clock spans 112.9 hours including declared gaps. Rate and amplitude are available; depth is explicitly unavailable. The conversion's acquisition mask is not independently verified. This is not the newly mentioned 500-hour recording.
Rate-only comparison — turn off Show amplitude and Show depth in the Unit Stability dock menu:
Merge candidates — staged IDs and colors match the curves:
Separate synthetic 500-hour check (generated test data)This generated fixture has 120,000 spikes and 1,000 summary bins, with rate, amplitude and depth. It checks long-duration display and does not represent Carl's 500-hour recording.
Validation: 864 tests passed, one existing benchmark skip; lint, formatting, strict documentation checks and cross-platform CI passed. Independent Sol 6.1-high audit completed; its hidden-view color finding was fixed and regression-tested. All 72 files in the original bounded screenshot package retained their hashes; captures used a disposable copy. Both real and synthetic capture processes exited normally. Reproduction instructions · Detailed validation and limitations · CI |
Problem and behavior
Opening a complete chronic sorting in the existing in-memory curation model does not scale to 1000-hour recordings. This PR prepares bounded contiguous observation windows, supports semantic curation of that evidence, and replays committed actions against the complete immutable source into a separate validated output.
phy prepare-large SOURCE_PARAMS OUTPUTvalidates the source, preserves exact int64 sample/global-row coordinates, generates full-duration counts and exposure-aware summaries, and writes a budgeted evidence package.phy template-gui OUTPUT/params.pyactivates a separate bounded controller only when its manifest is present.phy apply-curation SOURCE_PARAMS BOUNDED OUTPUTstreams supported actions into a new complete sorting.Implementation
The commit sequence covers the contract, services, GUI, hardening, and documentation within this single PR. The guide is
docs/long-recordings.md; measured evidence and acceptance limits are indesign/large-recording-validation.md.AIND elapsed-time and waveform-window compatibility
The follow-up supports explicitly declared acquisition-sample or elapsed-tick coordinates on one monotonic integer axis. Elapsed sources require authoritative acquisition exposure; all event times, window bounds and waveform sample-index sidecars must use consistent coordinates. Tick and physical waveform frequencies must currently agree, and elapsed coordinates cannot reference raw voltage paths.
timestamps.jsonrequires an explicit contract and is preserved as hashed provenance; Phy does not infer clock conversions or valid acquisition intervals from its endpoint map.Preparation now prefers validated producer waveform windows, offers
--window-policy auto|producer|generated, spreads budgeted producer-window selection across recording time, and reports source/retained/omitted waveform rows. Complete windows and the existing byte/spike limits are retained. Source clock endpoints and contradictory declarations fail before expensive full scans.The coordinate contract and exporter handoff describe the exporter changes still required. The reported mixed-clock AIND export is intentionally rejected until its producer supplies consistent coordinates and resolves exposure/filler handling. A different computation is now downloaded and tested through an explicitly modified diagnostic derivative; authoritative acquisition exposure and acceptance of the untouched export remain outstanding.
Full-size AIND diagnostic
Downloaded computation
b1b0782a-b3ea-4f34-81d7-b3c6e36452b3contains 469,725,567 spikes and 140,240 stored waveforms. It remains intentionally rejected unchanged: coordinates are mixed and authoritative exposure is missing.A separately labeled engineering derivative normalized coordinates and explicitly quarantined 33 spikes in mapped filler regions. Its inferred exposure is not acquisition truth. This derivative exposed a further Phy save bug after merge undo/redo and amplitude splitting: NumPy child IDs remained as split journal metadata keys. Both split-event paths now normalize metadata before retention; API and GUI regressions cover saving and reopening.
On the 469,725,534-row derivative, default preparation, all default GUI views, merge/split undo/redo, save/reopen, full replay and repreparation passed. An independent streamed oracle matched every replayed assignment and checked complete-source count/amplitude summaries. Every bounded spike and all 27,865 retained waveform/channel/center rows matched their source projection. Original-download SHA-256 checks remained unchanged. The 532,625,401-byte bounded package has 1,486,361 spikes; all 705 units have spike evidence and 702 have stored waveforms.
Observed single warm-cache runs: preparation 50.1 s / 14.2 GiB peak RSS; GUI actions/reopen 9.1 s / 0.68 GiB; replay 37.0 s / 25.0 GiB; repreparation 51.0 s / 14.1 GiB. Offline RSS includes resident memory-mapped pages and is not constrained by the package byte cap. These are integration measurements, not a controlled benchmark or scientific acceptance of the original export. See the real-data diagnostic record for assumptions, methods and limits.
SLAY suggestions across curation rounds
Bounded preparation, committed saves and full replay now preserve exact
curation.jsonbytes and the complete source-boundcuration_review.jsonledger. Committed checkpoint reviews take precedence over loose copies; direct saves and disabled proposition loading retain inherited history. Pending suggestions affected by merges/splits become stale with cumulative retirement reasons, while historical accept/reject decisions remain intact. Mutated, null, missing or unbound documents fail explicitly; older bounded packages with proposition artifacts require migration rather than silently losing reviews.On Carl's converted test overlay, three rounds retained 2 accepted and 91 pending suggestions, all 140,240 waveforms at his explicit 8M-spike/3GB caps, and both GOOD labels. An independent oracle matched every one of 469,725,567 full assignments in each round, verified 12 immutable files per replay, and confirmed all 67 raw download files unchanged. A separate final-code GUI-disabled save retained both accepted reviews. The default-budget validator also passed (1,486,361 evidence spikes, 27,865 waveform rows, 532,625,749 output bytes).
The combined roundtrip completed every assertion and wrote its success report, then exited 139 after its final PASS. Two isolated six-controller/three-GUI lifecycle runs exited cleanly; the shutdown fault remains recorded, with no reproduced product cause. See roundtrip evidence and limitations.
This is engineering validation of Carl's conversion, not acceptance of Brandon's native exporter or its acquisition mask. The new
tools/validate_aind_export.pyproduces source-hashed JSON reports and optional budgeted preparation; the native-export acceptance checklist records the remaining external evidence.Unit Stability comparisons and optional plots
Unit Stability now overlays a configurable number of selected clusters (32 by default; 0 for all) using the authoritative Cluster/Similarity/Merge selection and colors. Saved dock-menu toggles independently show firing rate, amplitude and depth, with rate-only filling the panel. The display cap never changes merge membership. Existing prepared folders work unchanged; full-source exposure-weighted counts/rates and finite-value weighted means are preserved, with split children individually pending until replay/re-preparation.
The before/after screenshot gallery compares the same selected clusters on Carl's earlier downloaded 112.9-clock-hour converted test export, including a rate-only view and matching Merge candidates. It separately labels a synthetic 500-hour example; Carl's new 500-hour recording was not available locally. The capture tool uses a disposable copy and verified all 72 input file hashes unchanged. Validation and reproduction.
Unit metadata across curation rounds: seven-item PDF follow-up
All seven items are addressed on this branch. Metadata tables preserve original basenames/CSV-TSV columns; new merge/split memberships have blank scalar source IDs and imported producer values rather than first-parent measurements. New memberships start unreviewed; explicit labels on the new unit survive unchanged rounds, with parent metadata and decisions retained in cumulative provenance. Legacy event meaning is preserved; already saved sessions require an explicit
BoundedSession.migrate_metadata(original_source_params); save()before using the new policy. Migration instructions.Replay parameters now use one maintained final override block, preserving original Python, comments, line endings, encoding cookies and BOMs without round-by-round growth. The display cap is configurable (32 default, 0 all), legends are removed, the actual camera action works, and saved rate width defaults to 0.6 points. Dataset-local display bins default to prepared resolution; coarsening exactly sums counts/exposure and measurement sums/counts before dividing, preserving gaps and final edges.
An independent data-only verifier checked all 469,725,567 assignments in each of three merge → split → merge rounds, unchanged-unit metadata, all 45 source table names, blank producer values, explicit labels, two accepted/91 pending suggestions and stable 892-byte params. All 67 original files and 12 immutable files per output retained their hashes. Migration of a copied legacy checkpoint retained its original manifest, assignments and reviews while clearing ambiguous producer values and preserving explicit GOOD labels. The combined GUI/data harness reached PASS then exited 143 with an unknown SIGTERM source; independent data verification, capture processes and the suites exited normally.
The updated existing screenshot gallery includes the real camera capture and separate twelve-unit synthetic 513-hour native/coarsened views. Detailed seven-item acceptance record. Final CI, documentation and wheel checks are green. Independent Sol-high audits found no remaining blocker, including legacy blank/typed values, migration identity, save snapshots and exact rebinning.
Graphical amplitude splits and curation acceptance
Evidence Amplitude now defaults to an overview of all retained windows on the original recording clock, with a saved overview/window toggle. With one unit selected outside Merge mode, Alt-right-drag previews a source-amplitude threshold and highlights matching waveforms; K splits the retained rows and journals its semantic predicate. Full
apply-curationlater evaluates every source spike. Finite values strictly below the threshold are selected; equal/nonfinite values stay in the remainder. Context changes clear previews, invalid splits report the reason without mutation, and arbitrary selected-row splits remain guarded.The new independent
tools/compare_curation_modes.pycompares complete outputs using an exact two-way ID mapping over every spike, verifies row-aligned times, and checks explicitly requested manual-label fields. Its ordinary reference uses full Supervisor commands without GUI views; the separate streamed engine is not ordinary GUI evidence. The user-test protocol supplies a shared ledger and practical variants for merge/split, labels, history, persistence and sampling. Ordinary graphical comparisons must select template amplitudes and use finite source values; its display sanitizes nonfinite values differently from raw bounded replay.On Carl's earlier converted 112.859-hour test overlay, a native Qt drag/K workflow, merge → split → merge, undo/redo and history branching, explicit GOOD/MUA/notes, and save/reopen produced full replay matching the ordinary reference on all 469,725,567 spikes, with zero partition or manual-label mismatches. That earlier engine reference used the same PR code; the subsequent actual main GUI comparison is recorded below. A further no-edit prepare/save/reopen/apply cycle preserved all 18 NPY files and both manual-label tables byte-for-byte. All 67 original download files and 19 immutable replay files in each output retained their hashes. This validates engineering behavior of the conversion, not its inferred exposure or Brandon's native exporter.
The subsequent actual main GUI at
ad64e913completed the same ledger with the full ordinary TemplateModel/controller, native Qt Alt-right-drag/K, GUI saves and a fresh-process reopen. Its saved full output also matched bounded replay on all 469,725,567 spikes, with exact times, zero partition mismatches and zero group/notes differences. An independent scan confirmed all 1,156,852 parent amplitudes are finite and the actual threshold exactly matches the ledger. All 67 original download files and 19 immutable ordinary working-copy files retained their hashes. Main's absent-raw-data string sentinel was normalized only in its writable copy; its code and event arrays were unchanged.This is curation completed with baseline Feature View errors, not error-free all-view acceptance: unmodified phylib 2.7.1 emitted seven empty-waveform PCA errors during main curation and five on reopen. A minimal independent reproduction confirms that renderer failure; the bounded feature-ID filter avoids that empty projection path. The complete assignment/label comparison still passes, and no #1428 code fix was required. Main-GUI evidence and limitations · Comparison summary. Carl's manual usability/scientific review and native-export acceptance remain separate.
The existing screenshot discussion now includes actual before-action, threshold-preview and after-split captures on a disposable copy of that real output. The bounded workflow images are a separate demonstration from the shared comparison ledger; follow-up main GUI captures document that ledger without claiming a controlled layout comparison. The capture tool checks input hashes and supports
--amplitude-split. Recorded ledger · All-row comparison summary.Permanent full-versus-subset output equivalence
The permanent CI regression now drives the same action script through both actual controllers, rather than comparing an ordinary run with manually constructed bounded journal events. Three sampling configurations use real subsets, omitted source spikes and a five-point display cap. Four rounds exercise merge → graphical threshold split → merge, GOOD/MUA/noise/clear and notes, undo/redo and history branching, GUI saves/reopens, further curation, a byte-preserving no-edit round and an unreviewed merge. Threshold-equal values remain in the remainder. Explicit final label expectations are independent of exported values.
tools/compare_curation_modes.py compare --source ORIGINAL --bounded-dir FINAL_PACKAGE --expectations EXPECTED.jsonenables the whole-output contract: every immutable array/sidecar, all metadata fields, replay table schema, parameters, exact saved generation, retained-row/full-row assignment agreement, journal/lineage/ancestry, review bindings and file inventory. Only explicitly validated native artifacts and membership-policy differences are admitted; unknown/missing files or arbitrary producer values fail. New source-only corruption regressions repair hashes before testing semantics, and a real subprocess test covers the standalone CLI. Contract and scope · Controller test · File-contract regressions.The fresh standalone check passed on all 469,725,567 spikes, all 46 metadata fields and 20 immutable files, bound to the actual saved generation. It recognized the unchanged plural SI-ID alias, three byte-pinned blank columns and a correctly derived native whitening inverse. These are explicit format artifacts, not a blanket metadata exclusion. Real-data result summary. Acceptance remains scoped to the tested ledgers and documented exceptions; inherited producer values may differ, raw nonfinite predicates are tested separately from finite graphical comparisons, and older parent artifacts are recorded ancestry rather than fetched snapshots.
The separate baseline Feature View failure is fixed in phylib draft PR #70: five on-disk regressions, 308 full-suite tests and cross-platform/minimum-SpikeInterface CI passed. With only that isolated phylib change, a copied real full sorting reopened in main with zero tracebacks, preserving assignments/groups/notes byte-for-byte. The original failed-renderer baseline and installed environments were preserved. No dependency bump is required for bounded mode, which already filters empty feature requests.
Independent all-source predicate validation
A further Sol 6.1 high audit found a checker false acceptance: an altered split threshold with coherently repaired journal/history/checkpoint hashes could pass without changing assignments. The checker now independently executes cumulative merges and split predicates over every original source spike, including omitted evidence and earlier curation rounds. It checks exact replay IDs, comparator equality, remainder/selected child order and parent-local nonfinite policy without using the production projector/evaluator. This was a validation gap; no incorrect output from the normal curation producer was demonstrated.
The original audit reproduction now fails at the contradictory spike. 31 new regressions cover coherent current/inherited forgeries, an unseen-row-only contradiction, incorporated source history and literal predicate semantics. Independent high review found no further issue.
The committed checker passed on all 469,725,567 source rows, including the recorded two merges and one split, 46 metadata fields, 20 immutable files and saved-generation/file-inventory checks. This remains the earlier converted engineering dataset with unverified exposure, not acceptance of the newer 500-hour/native export. Contract and results · Pinned report summary.
Validation
Latest runtime verification, 2026-10-26, Linux / Python 3.12.3 / phylib 2.7.1 / Xvfb and software OpenGL:
make test-full: 1,041 passed, 1 existing benchmark skip (499 core and 542 application tests), including lint and formatting checks.make doc-checkpassed locally. The previous runtime CI passed. Local strict documentation checks passed; the previous cross-platform CI/docs/build is green. The new predicate-oracle commits are awaiting their CI results.Earlier streamed replay benchmarks matched every assignment for synthetic 250k and 2m-spike sources over a 1000-hour clock, at approximately 204–206 MiB peak RSS. These synthetic measurements do not establish capacity or curator acceptance on the original AIND export; the separate full-size diagnostic above retains its explicit exposure and filtering limitations. See
design/large-recording-validation.mdfor methods and limits.Earlier synthetic screenshots
Actual Qt application captures from a reproducible synthetic eight-hour fixture. These illustrate the UI; they are not measurements of chronic recording quality. Regenerate with
tools/capture_long_recording_views.pyunder Xvfb/software Mesa; the command and captions are indocs/long-recordings.md.Unit Stability: complete-source rates and finite-value amplitude/depth means, with acquisition gaps in gray and retained windows in cyan.
Windowed evidence: source amplitudes at top left, explicitly labeled as seconds from window start with the recording-time range above the plot; sampled ISI at top right, sampled correlograms below it, and window-relative features along the bottom.
Acceptance limits
Draft pending real chronic-recording and curator acceptance. Synthetic sparse-clock fixtures do not establish billion-spike capacity, Qt responsiveness on every hardware/source layout, or biological identity over 1000 hours. Survey/rescue evidence is not a probability sample, and sampled diagnostics must not be interpreted as full-recording quality metrics. The workflow requires one monotonic integer event-time axis, explicit authoritative exposure, supported canonical arrays, and nonnegative int32-compatible unit IDs; time/global-row coordinates are int64. Explicit elapsed ticks and physical waveform sample rates must currently use the same frequency. Producer metrics are not recomputed. New membership-policy merge/split units clear imported metrics, source IDs and quality until explicitly reviewed; legacy journals keep their original behavior until migrated.