Spike: how to avoid re-implementing capture per language #298

Activity

1. vishnuv688

   self-assigned this

   on Aug 12, 2026
2. vishnuv688

   added

   capture-architectureCross-language capture architectureCross-language capture architecture

   on Aug 12, 2026
3. vishnuv688

   added a parent issue on Aug 12, 2026

   • Epic: Cross-language capture architecture #278
4. vishnuv688 commented on Aug 25, 2026

   

   vishnuv688

   on Aug 25, 2026

   MemberAuthor

   More actions

   Decision

   Keep the common parts common. The trace transforms move to a package the
   backend may legitimately import, so every language reuses one implementation
   instead of growing its own copy. The JS adapters keep calling them in-process;
   the backend gains an export path for adapters that only stream.

   Lever taken: #301, move stream post-processing behind the wire — scoped, see
   "What stays per-language" below. Levers not taken now: #299 and #300 stay open
   and are unaffected by this.

   Why this rather than porting to Python

   backend, script and shared are already common to every adapter — the
   backend runs as a Node subprocess, the page collector is injected as JS at
   runtime, and shared's values are generated into _contract.py with a drift
   guard. core is the exception: it is TypeScript, Python cannot import it,
   and everything it provides was therefore written a second time
   (frames.py, capturer.py, types.py, snapshot.py, screencast.py,
   performance.py — ~1,206 lines that are not language-specific, per this
   issue's own measurement).

   That gap has already cost us three times. Force-anchor drain (#285), recovery
   drain (#286) and document-start preload (#284) were each fixed in the JS
   adapters and never propagated to Python. Trace assembly is another ~2,600 lines;
   ported, it becomes the second copy, and Java would make it the third.

   What moves

   Measured by walking the import graph: every trace module depends only on
   @wdio/devtools-shared, yazl, and node builtins. None touches a driver, an
   adapter or a framework — they are pure transforms over shared types, which is
   what makes this possible at all.

   ~2,573 lines move — trace-exporter, trace-action-events,
   trace-frame-snapshots, trace-snapshots, trace-retention, trace-har,
   screencast-trace, trace-mutations, trace-transcript, plus their leaves
   action-mapping, trace-console, trace-sources, trace-zip-writer,
   trace-hierarchy, uid, artifact-naming, console, error.

   Destination: a new workspace-internal package (private, bundled into consumers
   like core and shared are). Not into shared: the app bundles shared
   for the browser, and yazl + node:fs have no business there.

   What stays per-language

   ~1,137 lines stay in core as adapter orchestration: trace-finalizer
   (drives the adapter's finalize lifecycle), spec-trace-helpers (slices by test
   boundaries only the adapter knows), artifacts-manifest and attempt-tracker
   (fed by adapter hooks).

   Irreducibly per-language, and unchanged by this: driver instrumentation, the
   BiDi/CDP bindings, session lifecycle, the transport, and the test-framework
   plugin. That is the ~1,595 lines this issue already measured as genuinely
   language-specific.

   The constraint this respects

   The backend is forbidden from importing core — enforced in eslint.config.cjs,
   stated at CLAUDE.md:87 — because core is adapter logic. That rule is kept,
   not weakened: trace assembly is pure data transformation and was arguably never
   adapter logic, so it moves out rather than the boundary moving.

   The constraint this does not break

   Trace mode deliberately runs no backend (launcher.ts: "Trace mode —
   skipping backend and Chrome window"). The JS adapters keep calling the
   transforms in-process, so backend-free trace mode is untouched and its output
   stays byte-stable. Python's export goes through the backend — and Python has no
   backend-free trace mode to lose, because it has no trace mode at all today.

   Still to solve in #330

   actionSnapshots is a defined TraceLog scope no adapter sends (core
   synthesizes from command screenshots when it is absent), and screencastFrames
   is an exporter argument rather than a wire message. Both must travel for a
   stream-fed export to match in-process fidelity. Neither blocks the move.
5. vishnuv688 commented on Aug 25, 2026

   

   vishnuv688

   on Aug 25, 2026

   MemberAuthor

   More actions

   Correction to the line counts above

   Walking the reverse dependencies changed the split. Two things I got wrong:

   1. Five modules I counted as movable are not trace-only. error.ts,
   uid.ts and artifact-naming.ts are used across core (assert-patcher, bidi,
   script-loader, test-reporter, screenshot-artifact, video-slice) and the trace
   transforms do not need them at all — they stay put. That removes ~230 lines from
   the moving set.

   2. Two genuinely are needed by both. action-mapping.ts (how a command
   renders as an action) and console.ts (console-entry normalisation) are
   imported by the transforms and by session-capturer / bidi /
   assert-patcher. Both depend on nothing but @wdio/devtools-shared, so they
   move to shared rather than into the trace package — they are contract
   shaped, and putting them in a package named for trace assembly would be the
   wrong home for bidi.ts to import from.

   So: the trace package holds the transforms only, shared gains two
   dependency-free helpers, and core keeps its general utilities.

   Dependency direction, confirmed

   session-capturer.ts already imports a transform (reattributeDomAnchors from
   trace-mutations), so core will import the new package. That is one-way and
   cycle-free: trace ← core ← adapters, and trace ← backend. The trace
   package imports only shared.

   Worth noting this leaves the enforced rule genuinely intact rather than
   technically satisfied — eslint.config.cjs blocks backend → core because core
   is adapter logic, and the backend still will not import it.
6. vishnuv688 commented on Aug 31, 2026

   

   vishnuv688

   on Aug 31, 2026

   MemberAuthor

   More actions

   The fourth lever, evaluated

   The record above named three of the four levers this issue listed and skipped
   one: push more work into the shared page script. Filling that in, because
   #341 is about to run into it.

   Verdict: the page script is already the right home, and already common — the
   gap is not what runs there, it is who can reach it.

   Measured:

   	lines	runs where	reachable from Python
   packages/script	503	the page	yes — served at /api/collector, and the adapter injects it
   core/element-scripts.ts	425	the page	no
   core/element-snapshot.ts	1057	Node	no
   core/action-snapshot.ts	176	Node (adapter)	no

   element-scripts.ts says so itself: "Browser-injectable script strings … each
   function returns a self-contained JavaScript string designed to run inside a
   browser page … must be ES5-compatible." That is 425 lines of page code sitting
   in a package Python cannot import — the same barrier the trace transforms hit,
   for the same reason, and it is why a Python trace carries 0
   *-elements.json where a JS Selenium trace carries 12.

   So this lever does not mean "write more browser code". The browser code exists.
   It means the page-side half is in the wrong package, and the route out is one
   that already works: the collector reaches Python over HTTP from the backend, so
   the element scripts can travel the same way rather than being ported.

   element-snapshot.ts is a different case. It imports only shared and the
   locator helpers, takes a tree and returns indented text — a pure transform,
   which is trace-shaped, not page-shaped. It is not part of this lever.

   What stays per-language is unchanged. Reading the page needs the live
   driver, so running the script and shipping the result is adapter work; only
   the script body is shared. That is the same split #301 drew — transforms
   shared, capture in-language.

   Concretely for #341: serve the element scripts the way the collector is served,
   rather than porting element-scripts.ts to Python. Anything else re-creates the
   duplication this issue exists to prevent.
7. vishnuv688 commented on Oct 28, 2026

   

   vishnuv688

   on Oct 28, 2026

   MemberAuthor

   More actions

   Re-analysis, two months on: all four levers shipped

   Lever	Issue	State
   Move trace post-processing behind the wire	#301	Done. packages/trace exists, and Python's trace_export.py asks the backend to build its zip.
   Push work into the shared page script	#341	Done for the collector and the element scripts, which Python fetches from the backend.
   Generate contract types	#299	Done. _wire_types.py and _contract.py are generated from shared.
   Cross-adapter conformance suite	#300	Done, and it now runs each adapter through its real test runner.

   Where the Python adapter stands

   7,108 logic lines, up from about 2,800, mostly from features (trace mode, mobile, waits, reruns, native network):

   Kind	Lines	Modules
   Generated, or fetched from the backend	~700	_wire_types, _contract, element_scripts, collector_source, trace_export
   Hand-written copies of core or JS logic	~1,600, plus part of bidi.py	screencast, snapshot, capturer, wait_tracking, interceptor_network, element_locators, performance, device, run_id, output_dir, read_value_locators, bidi_preload
   Genuinely Python-specific	~4,800	instrumentation, pytest plugin, lifecycle, reruns, backend install, transport, BiDi binding

   What the remaining copies cost

   Four drifts surfaced in one day, each a case where two copies disagreed and only one was right: core's performance script lacked the return Python's had, so no JS adapter attached navigation timings (#467); the cross-process row id fix landed in core and had to be rediscovered for Python (#473, #474); Python's assertion-target rule exposed a bug in core's (#476); and wait tracking was built twice, with the same nested-wait bug found separately in each (#453, #474). The conformance suite caught many cross-adapter bugs, but three of these four were found in review.

   Decision

   The levers stand. What is left is narrower than this spike: the copies that remain, and the checks that would have caught the four drifts. Tracked under #278:

   • Serve the remaining page scripts from the backend instead of copying them into Python #477 Serve the remaining page scripts (performance capture, collector drain and ready probe, injection) from the backend, so Python keeps no browser code of its own.
   • Shared test vectors for logic implemented in both TypeScript and Python #478 Shared test vectors for the logic that has to exist in both languages, run by vitest and pytest from one file.
   • Conformance rules for navigation timings and page-level assertion targets #479 Conformance rules for navigation timings and page-level assertion targets.

   Irreducibly per-language, unchanged: driver patching, the BiDi/CDP bindings, session lifecycle, transport and the pytest plugin. Moving Python's screencast and capture-session logic into the backend would cut more, but changes when capture happens; that is a design question, not one of these follow-ups.
8. vishnuv688

   closed this as completedon Oct 28, 2026