feat(workflow-operator): compare Filter's exported predicates the way the engine does by kz930 · Pull Request #8980 · apache/texera

kz930 · 2026-10-14T08:53:01Z

What changes were proposed in this PR?

Filter's exported Python compared the column to the value with pandas operators. The engine's FilterPredicate.evaluate compares by the column's type instead, so the two disagreed on numbers stored as text, booleans, NaN and -0.0.

Each comparison in the export now goes through _texera_filter_keep, a transcription of FilterPredicate.evaluate:

Column type	Comparison
STRING	as Java doubles when both sides parse as one, otherwise by UTF-16 text
BOOLEAN	the lower-case text "true"/"false"
INTEGER, DOUBLE	by `Double.compareTo`: NaN equals NaN and sorts above every number, -0.0 sorts below 0.0; a hex value past the largest double is infinity, as in Java
LONG	as integers, rejecting what `Long.valueOf` rejects
TIMESTAMP	by whole milliseconds; a value with an offset is read as the wall clock it shows in the machine's zone, with the offset that zone had at that instant

The column type comes from the input schema. Without one, it is inferred from the pandas dtype. is null and is not null are unchanged.

One approximation remains. The engine reads a TIMESTAMP value with DateParserUtils, which accepts more formats than pandas. A value pandas cannot read raises an error, so the export never gives a silently wrong answer.

A value with an offset is placed in the machine's zone by _texera_local_wall_clock from #8597. It uses the zone's IANA history, as the JVM does, and not Windows' localtime. Before 1900 it takes the zone's standard offset, as the engine does.

Any related issues, documentation, discussions?

Closes #8977

Depends on #8597, which adds _texera_local_wall_clock, and on #9035, which reads a column named in a different case. This PR stays a draft until both merge.

How was this PR tested?

A new test in SpecializedFilterOpDescSpec runs 33 predicates across the six column types through SpecializedFilterOpExec and through the export, and compares the kept rows by input position. A second test does the same for timestamp values with an offset, with both the JVM and Python set to Asia/Kolkata. Both fail on main and pass with this change. A third test sets both to America/Mexico_City and compares 2021-07-01T00:00:00Z (summer time) and 1931-01-01T00:00:00Z (an older offset). With #8597 merged in, the Filter specs pass (42 tests). A fourth test compares a LONG column Key through a predicate on KEY, past 2^53, and passes with #8597 and #9035 merged in.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

🤖 Generated with Claude Code

…does The export compared the column to the value with pandas operators, while FilterPredicate compares by type: a STRING field and the value as numbers when both parse as one, a BOOLEAN as its lower-case text, an INTEGER as a double, a DOUBLE with Double.compareTo, and a TIMESTAMP by getTime. Each comparison now runs through a transcription of FilterPredicate.evaluate. Closes apache#8977 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-14T11:20:59Z

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

No candidates found from git blame history.

codecov-commenter · 2026-10-14T12:44:54Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.18%. Comparing base (7b8422e) to head (1f8e1ef).

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #8980   +/-   ##
=========================================
  Coverage     92.18%   92.18%           
+ Complexity     5051     5049    -2     
=========================================
  Files          1286     1286           
  Lines         55937    55934    -3     
  Branches       7140     7140           
=========================================
- Hits          51563    51561    -2     
+ Misses         2612     2610    -2     
- Partials       1762     1763    +1

Flag	Coverage Δ		*Carryforward flag
access-control-service	`77.44% <ø> (ø)`
agent-service	`99.16% <ø> (ø)`		Carriedforward from f225ca5
amber	`86.93% <ø> (-0.01%)`	⬇️	Carriedforward from f225ca5
computing-unit-managing-service	`64.02% <ø> (ø)`
config-service	`87.61% <ø> (ø)`
file-service	`82.29% <ø> (ø)`
frontend	`96.48% <ø> (-0.01%)`	⬇️	Carriedforward from f225ca5
notebook-migration-service	`83.77% <ø> (ø)`
pyamber	`98.54% <ø> (+0.06%)`	⬆️	Carriedforward from f225ca5
workflow-compiling-service	`74.25% <ø> (ø)`

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-10-14T14:05:22Z

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 11 worse · ⚪ 4 noise (<±5%) · 0 without baseline

Compared against main 7b8422e benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	374	0.228	26,909/35,799/35,799 us	🔴 +18.2% / 🔴 +154.1%
🔴	bs=100 sw=10 sl=64	831	0.507	115,482/165,528/165,528 us	🔴 +26.4% / 🔴 +71.7%
🔴	bs=1000 sw=10 sl=64	958	0.585	1,043,040/1,172,656/1,172,656 us	🔴 +6.5% / 🔴 +26.3%

Baseline details

Latest main 7b8422e from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	374 tuples/sec	433 tuples/sec	888.52 tuples/sec	-13.6%	-57.9%
bs=10 sw=10 sl=64	MB/s	0.228 MB/s	0.264 MB/s	0.542 MB/s	-13.6%	-58.0%
bs=10 sw=10 sl=64	p50	26,909 us	23,020 us	11,467 us	+16.9%	+134.7%
bs=10 sw=10 sl=64	p95	35,799 us	30,296 us	14,087 us	+18.2%	+154.1%
bs=10 sw=10 sl=64	p99	35,799 us	30,296 us	17,519 us	+18.2%	+104.3%
bs=100 sw=10 sl=64	throughput	831 tuples/sec	889 tuples/sec	1,174 tuples/sec	-6.5%	-29.2%
bs=100 sw=10 sl=64	MB/s	0.507 MB/s	0.543 MB/s	0.716 MB/s	-6.6%	-29.2%
bs=100 sw=10 sl=64	p50	115,482 us	112,426 us	89,952 us	+2.7%	+28.4%
bs=100 sw=10 sl=64	p95	165,528 us	130,987 us	96,426 us	+26.4%	+71.7%
bs=100 sw=10 sl=64	p99	165,528 us	130,987 us	110,188 us	+26.4%	+50.2%
bs=1000 sw=10 sl=64	throughput	958 tuples/sec	978 tuples/sec	1,204 tuples/sec	-2.0%	-20.4%
bs=1000 sw=10 sl=64	MB/s	0.585 MB/s	0.597 MB/s	0.735 MB/s	-2.0%	-20.4%
bs=1000 sw=10 sl=64	p50	1,043,040 us	1,014,804 us	886,711 us	+2.8%	+17.6%
bs=1000 sw=10 sl=64	p95	1,172,656 us	1,100,795 us	928,644 us	+6.5%	+26.3%
bs=1000 sw=10 sl=64	p99	1,172,656 us	1,100,795 us	972,356 us	+6.5%	+20.6%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,535.22,200,128000,374,0.228,26908.88,35798.93,35798.93
1,100,10,64,20,2406.34,2000,1280000,831,0.507,115482.30,165528.44,165528.44
2,1000,10,64,20,20883.44,20000,12800000,958,0.585,1043040.04,1172656.20,1172656.20

kz930 · 2026-10-15T01:18:39Z

@carloea2 Could you review this? Filter's export now compares each predicate by the column's type, as FilterPredicate.evaluate does, and a new spec test checks 26 predicates against the engine.

carloea2

The 40 existing Filter tests and scalafmtCheckAll pass against current upstream main on JDK 17. Two additional native versus export parity cases fail below. Finite hexadecimal, unzoned timestamp and null controls pass.

Commands: sbt -batch "WorkflowOperator/testOnly *SpecializedFilterOpDescSpec *FilterPredicateSpec *SpecializedFilterOpExecSpec" scalafmtCheckAll; sbt -batch "WorkflowOperator/testOnly *FilterBoundaryReviewSpec"

…flow as the engine does An offset-bearing timestamp value is an instant, which the engine holds as the wall clock it shows in the machine's zone, so the export now converts it there before comparing instead of raising on a naive column. A hex double past the largest double is infinity in Java, so the export no longer lets float.fromhex raise OverflowError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kz930 · 2026-10-16T06:22:05Z

@carloea2 Thanks, both findings are fixed in 55ce604 with regressions against the native executor. Could you take another look?

carloea2

Both original regressions now pass. A remaining Windows timezone problem is reproduced below. Of 47 focused tests, 44 pass and 3 fail: the two historical cases and the new Asia/Kolkata test, whose TZ setting is not honored as an IANA zone by Windows localtime. scalafmtCheckAll passes.

Command: sbt -batch "WorkflowOperator/testOnly *SpecializedFilterOpDescSpec *FilterPredicateSpec *SpecializedFilterOpExecSpec *FilterBoundaryReviewSpec" scalafmtCheckAll

…offset A timestamp value with an offset now becomes the machine-zone wall clock through apache#8597's _texera_local_wall_clock, which uses the zone's IANA history instead of Windows localtime. Adds a Mexico City test for 2021 summer time and 1931. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

carloea2

The historical timezone finding is fixed. With #8597 at f0eedd9 merged locally into current upstream main, all 61 focused tests and scalafmtCheckAll pass on Windows. That includes my original hexadecimal, offset and historical regressions. The PR head alone lacks _texera_local_zone, so the CI dependency is real. Leaving the draft unchanged until #8597 lands and CI passes.

Command: sbt -batch "WorkflowOperator/testOnly *SpecializedFilterOpDescSpec *FilterPredicateSpec *SpecializedFilterOpExecSpec *FilterBoundaryReviewSpec *TypeCastingOpDescSpec" scalafmtCheckAll

carloea2

Tested together with #8597 and #9035 after resolving only the import conflicts. Nine existing tests pass. The #9035 generated-code assertion needs updating for the new helper, and an added native-versus-export regression finds a LONG comparison mismatch.

sbt -batch 'WorkflowOperator/testOnly *SpecializedFilterOpDescSpec -- -oDF'

…insensitive lookup The engine finds a predicate's field through Schema, which ignores case. An exact-name lookup found no type for a column named in another case, fell back to the dtype, and compared a LONG as a double. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ported script _texera_filter_keep explained each comparison in Python comments, which every exported script with a Filter printed. The rules now sit in the Scaladoc, one sentence per type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… type needs _texera_filter_keep held every type's comparison in one function, so a Filter on a DOUBLE column carried the LONG, BOOLEAN, TIMESTAMP and STRING code too. Each type now has its own function, the generated code calls the one its column's declared type needs, and the translator drops the rest. Without a schema, _texera_filter_keep still picks one by dtype. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…umber parsers Filter kept its own String.trim, Double.parseDouble and a LONG check that matched Long.parseLong. It now calls _texera_java_trim, _texera_java_double and _texera_parse_int from AttributeCasts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

carloea2

The LONG Key and KEY regression is fixed. All 12 SpecializedFilterOpDescSpec tests pass with the #9035 lookup API and timestamp helpers supplied. The native filter and export now keep only the correct row beyond 2^53. scalafmtCheckAll passes. The combined #9035 generated-code assertion was updated mechanically for the new helper. CI is still red, so the dependencies and checks need clearing before merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kz930 · 2026-10-26T22:20:05Z

@carloea2 Since your review, Filter orders doubles with the shared _texera_double_key and its helpers are spaced like the rest of the export. Could you take another look?

@carloea2

…way the engine does (apache#9035) ### What changes were proposed in this PR? The engine looks a column up through `Schema`, which ignores case. So a setting that names `Name` reads the column `name`, and the workflow runs in Texera. The exported script indexed the pandas frame with the name as written and stopped with a `KeyError`. `StandaloneCodeGenerator` gains `columnAsStored`, which resolves a setting's column name against the input schema and returns the table's spelling. A name the schema does not hold comes back unchanged, so the script still fails where the run would. Seven operators read their column through it: Projection, Filter, Keyword Search, Regular Expression, Substring Search, Dictionary Matcher and Unnest String. Projection still names each output column by its alias, which falls back to the name as the setting writes it. That matches `ProjectionOpExec`. Type Casting is the exception. Its schema propagation and its executor both match the column by its exact name, so in the engine a unit whose column differs in case casts nothing and the column passes through unchanged. The export now skips that unit too, instead of raising a `KeyError`. Operators that run as Python in the engine already compare names exactly on both sides, so they are left alone. ### Any related issues, documentation, discussions? Closes apache#9004. @carloea2 found the mismatch while reviewing apache#8918. apache#8980 rewrites Filter's export and looks up the declared type by the column's name. Whichever of the two merges second should look that type up under the resolved name as well. ### How was this PR tested? Each of the eight operators' `OpDescSpec` gains a test with a setting in a different case from the table. It asserts that the generated code reads the table's spelling. For Type Casting, the test also checks that the engine's output schema leaves that column's type unchanged, and that the export skips the unit. All of `WorkflowOperator/test` passes. Merged with the other open export PRs, the verification suite still passes all 150 cases (3 ignored, as before). ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ine-comparisons # Conflicts: # common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDesc.scala # common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDescSpec.scala

github-actions Bot assigned kz930 Oct 14, 2026

github-actions Bot added fix common labels Oct 14, 2026

kz930 mentioned this pull request Oct 15, 2026

A workflow cannot be taken out of Texera as a script that runs on its own #8325

Open

17 of 31 tasks

carloea2 suggested changes Oct 16, 2026

View reviewed changes

Comment thread ...perator/src/main/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDesc.scala Resolved

Comment thread ...perator/src/main/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDesc.scala Resolved

carloea2 suggested changes Oct 16, 2026

View reviewed changes

Comment thread ...perator/src/main/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDesc.scala Resolved

kz930 marked this pull request as draft October 16, 2026 09:24

carloea2 reviewed Oct 19, 2026

View reviewed changes

kz930 mentioned this pull request Oct 20, 2026

feat(workflow-operator): read a column named in a different case the way the engine does #9035

Merged

carloea2 suggested changes Oct 23, 2026

View reviewed changes

Comment thread ...perator/src/main/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDesc.scala Resolved

kz930 and others added 4 commits October 23, 2026 14:43

carloea2 approved these changes Oct 24, 2026

View reviewed changes

kz930 and others added 2 commits October 24, 2026 14:54

feat(workflow-operator): order Filter's doubles with the shared key

c21c9a2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(workflow-operator): space Filter's helpers by one blank line

4de04d3

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Conversation

kz930 commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Oct 14, 2026

Automated Reviewer Suggestions

Uh oh!

codecov-commenter commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Benchmark changes need a look

Uh oh!

kz930 commented Oct 15, 2026

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kz930 commented Oct 16, 2026

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carloea2 left a comment

Choose a reason for hiding this comment

Uh oh!

kz930 commented Oct 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kz930 commented Oct 14, 2026 •

edited

Loading

codecov-commenter commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading