Repository navigation
Conversation
…ython Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8709 +/- ##
============================================
- Coverage 92.18% 92.18% -0.01%
+ Complexity 5051 5047 -4
============================================
Files 1286 1286
Lines 55937 55933 -4
Branches 7140 7140
============================================
- Hits 51566 51562 -4
Misses 2610 2610
Partials 1761 1761
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 387 | 0.236 | 23,979/38,871/38,871 us | 🔴 +18.6% / 🔴 +175.9% |
| 🔴 | bs=100 sw=10 sl=64 | 784 | 0.478 | 120,780/170,015/170,015 us | 🔴 -8.4% / 🔴 +76.3% |
| ⚪ | bs=1000 sw=10 sl=64 | 951 | 0.58 | 1,057,406/1,096,700/1,096,700 us | ⚪ within ±5% / 🔴 -21.1% |
Baseline details
Latest main 7b8422e from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 387 tuples/sec | 438 tuples/sec | 888.52 tuples/sec | -11.6% | -56.4% |
| bs=10 sw=10 sl=64 | MB/s | 0.236 MB/s | 0.267 MB/s | 0.542 MB/s | -11.6% | -56.5% |
| bs=10 sw=10 sl=64 | p50 | 23,979 us | 21,225 us | 11,467 us | +13.0% | +109.1% |
| bs=10 sw=10 sl=64 | p95 | 38,871 us | 32,776 us | 14,087 us | +18.6% | +175.9% |
| bs=10 sw=10 sl=64 | p99 | 38,871 us | 32,776 us | 17,519 us | +18.6% | +121.9% |
| bs=100 sw=10 sl=64 | throughput | 784 tuples/sec | 855 tuples/sec | 1,174 tuples/sec | -8.3% | -33.2% |
| bs=100 sw=10 sl=64 | MB/s | 0.478 MB/s | 0.522 MB/s | 0.716 MB/s | -8.4% | -33.3% |
| bs=100 sw=10 sl=64 | p50 | 120,780 us | 112,755 us | 89,952 us | +7.1% | +34.3% |
| bs=100 sw=10 sl=64 | p95 | 170,015 us | 176,271 us | 96,426 us | -3.5% | +76.3% |
| bs=100 sw=10 sl=64 | p99 | 170,015 us | 176,271 us | 110,188 us | -3.5% | +54.3% |
| bs=1000 sw=10 sl=64 | throughput | 951 tuples/sec | 965 tuples/sec | 1,204 tuples/sec | -1.5% | -21.0% |
| bs=1000 sw=10 sl=64 | MB/s | 0.58 MB/s | 0.589 MB/s | 0.735 MB/s | -1.5% | -21.1% |
| bs=1000 sw=10 sl=64 | p50 | 1,057,406 us | 1,034,650 us | 886,711 us | +2.2% | +19.3% |
| bs=1000 sw=10 sl=64 | p95 | 1,096,700 us | 1,127,940 us | 928,644 us | -2.8% | +18.1% |
| bs=1000 sw=10 sl=64 | p99 | 1,096,700 us | 1,127,940 us | 972,356 us | -2.8% | +12.8% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,516.29,200,128000,387,0.236,23978.94,38870.62,38870.62
1,100,10,64,20,2551.38,2000,1280000,784,0.478,120780.12,170015.48,170015.48
2,1000,10,64,20,21034.97,20000,12800000,951,0.580,1057405.55,1096699.88,1096699.88
Contributor
Author
carloea2
suggested changes
Oct 7, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
One empty-input mismatch remains. Reproduced with the emitted aggregate fragment and pandas 2.2.3, then checked against AggregateOpExec.
…h no group keys The engine opens a group only when a row arrives, so an empty input with no group keys finishes with nothing to emit. The exported script built one row anyway, for example a COUNT of 0. It now emits an empty frame with the result columns, and a parity test runs both on zero rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
carloea2
suggested changes
Oct 8, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
The empty Aggregate case now passes. The 26 existing focused tests pass too, but a native versus exported string-sort probe found one remaining mismatch.
The engine orders strings with String.compareTo, which compares UTF-16 code units, while pandas compares code points. The two disagree once a character above U+FFFF meets one between U+E000 and U+FFFF, so the generated script now sorts a string key by its UTF-16 big-endian bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
carloea2
suggested changes
Oct 12, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
The Unicode sort finding is fixed. I ran all four focused specs: 27 tests pass, but a new native versus generated aggregate regression fails with a stored NaN.
…e engine does A DOUBLE column read from Arrow keeps a stored NaN apart from null. The engine skips only the nulls and folds the rest as Java doubles, so SUM, AVERAGE and MAX answer NaN and MIN answers the smallest number. pandas' reductions answered null instead, and grouped MIN and MAX skipped the NaN. The exported script now writes the engine's folds out for a declared DOUBLE column and keeps grouped results in a nullable column, so NaN and null stay apart. A parity test runs both on the same rows, with and without group keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
carloea2
suggested changes
Oct 13, 2026
Read nulls from the column's mask instead of pd.isna on each value, which took a stored NaN for a null too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
carloea2
reviewed
Oct 13, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
The CONCAT NaN fix passes. With the declared #8597 cast helper supplied locally, all 28 focused tests pass, plus my original grouped and scalar reproductions with separate NaN and null cells. Formatting passes. The branch alone still fails with _texera_java_text undefined, so it needs that prerequisite before merging.
sbt -batch 'WorkflowOperator/testOnly *AggregateOpDescSpec *StableMergeSortOpDescSpec *SortPartitionsOpDescSpec *SortOpDescSpec -- -oDF' 'scalafmtCheckAll'
…fuses them Without a zone file, as on Windows, gettz() answers tzlocal(), which asks localtime, and Windows refuses an instant before 1970. Reading the column raised, and the sum's scalar tz_convert swallowed the refusal and answered in UTC. Read a refused cell at the zone's standard offset, and convert the total through datetime so the refusal reaches the same fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the engine does The engine keys a group on a List of the key fields in a HashMap, under Scala's ==, so every NaN row opens a group of its own. groupby, and the merge that joined the results back by key, put every NaN in one group. Group by an id that follows the engine's equality instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tamps as the engine does Double.compare puts -0.0 below 0.0, where pandas reads the two as equal, so both sorts gain a sign tier for a double key. Sort Partitions compares a timestamp by getTime, so two inside one millisecond keep their input order, and compares strings by UTF-16 code unit as Stable Merge Sort does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…instant gettz() falls back to tzlocal() where there is no zone file, as on Windows, and tzlocal() knows only today's two offsets; on macOS and Linux the zone file reads a wall clock before 1900 at local mean time, where java.util.Date takes today's standard offset. Read the machine's zone by its IANA name through the shared helpers, and keep the standard offset before 1900 in both directions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… engine does AttributeTypeUtils.compare reads a timestamp by getTime, so two in the same millisecond tie and the engine's fold keeps the one it met first, with its own digits past the millisecond. pandas' min and max compared the full resolution and picked the other one. Fold a declared TIMESTAMP column by its floored milliseconds instead, grouped or not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kz930
marked this pull request as draft
October 15, 2026 22:58
Contributor
Author
|
@carloea2 Five commits since your review, all on timestamps, NaN keys and signed zeros in Aggregate and Sort. Could you take another look? |
carloea2
approved these changes
Oct 23, 2026
carloea2
left a comment
carloea2
left a comment
Contributor
There was a problem hiding this comment.
Reviewed after #8597. The unmodified Aggregate, Stable Merge Sort and Sort Partitions suites give 28 passes and one Windows test-fixture failure. Setting the mocked standard offset explicitly makes all 29 pass, with no implementation change. scalafmtCheckAll passes. Please make the fixture portable as noted below.
sbt -batch 'WorkflowOperator/testOnly *AggregateOpDescSpec *StableMergeSortOpDescSpec *SortPartitionsOpDescSpec -- -oDF' 'scalafmtCheckAll'
…ck probe A C runtime that cannot read the zone name leaves time.timezone at 0, so the probe gave 12:00 where it asserts 18:00. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orted script Aggregate's helpers explained themselves in about 50 lines of Python comments, which every exported script with an Aggregate printed. The notes now sit above the helpers in Scala, one line per function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d key Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wall clock helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…odes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re left out Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed sum Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mputed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
@carloea2 Since your approval, Aggregate groups by the shared key codes, orders MIN and MAX with the shared Double.compareTo key, and averages through its own sum. Its helpers are declared so a script only carries the ones it calls, and each result is joined as it is computed. Could you take another look? |
…piling getPhysicalPlan swapped the operator's aggregations for their global-stage form and left them there. The export compiles a plan and then reads the same operators, so the generated code grouped on the result column names. The global stage's desc is unchanged; the operator now keeps what the user set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…regate-and-sort # Conflicts: # common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/aggregate/AggregateOpDesc.scala
renovate-bot
pushed a commit
to renovate-bot/apache-_-texera
that referenced
this pull request
Nov 1, 2026
…es (apache#8597) ### What changes were proposed in this PR? Type Casting to a timestamp read text with `pd.to_datetime(col, errors="coerce")` in the exported script. That is wrong in two ways, and both empty a cell the run itself filled. pandas parses into nanoseconds, which reach only 1677-09-21 to 2262-04-11. The engine holds a `java.sql.Timestamp`, where `2500-01-01 00:00:00` is an ordinary moment: it reads it, and the exported script answered with an empty cell. The same for `1500-06-15 08:30:00` and `9999-12-31 23:59:59`. pandas also infers one format for the whole column and coerces every row that does not match it. The engine hands `DateParserUtils` one field at a time, so a row states its own format. A column holding `2024-03-05 14:09:07` and `March 5, 2024` kept the first and emptied the second. The text branch now reads the column cell by cell and holds the result at microsecond resolution, which covers the years the engine covers. Text no format reads now ends the run, as the engine's `parseField` ends its own. The script raises the engine's message, `Failed to parse type str to Timestamp`, where it used to leave the cell empty. dateutil reads every format the verification hands the engine, so the two refuse the same text. Review and the verification's edge values found more places where the cast parted from the engine: - A reading that states an offset, such as `Z` or `+05:30`, stopped the cast. The engine moves it to the machine's zone and keeps that wall clock, and the script now does the same. - A column that already held timestamps went through the text parse and lost its nanoseconds. `parseField` returns a `java.sql.Timestamp` untouched, so the script now hands such a column back at the resolution it arrived in. - Text with digits past the millisecond kept them. `DateParserUtils` reads into a `java.util.Date`, which counts milliseconds, so the script now cuts there too. - Casting to text wrote a value the way Python's `str()` does. Java writes a double past 1e7 or under 1e-3 in E notation, a boolean in lower case, and a timestamp with its nanoseconds and at least `.0`. A shared `_texera_java_text` now writes each one as Java's `toString` does. - A double column holding a NaN as a value, as an Arrow or Parquet source hands one over, lost it. The casts asked each cell whether it was empty, and pandas hands an empty cell over as NaN too, so a stored NaN came out null where the engine writes `NaN` as text, 0 as an integer and `true` as a boolean. The casts now read empty cells from the column's own mask, and the double cast builds a nullable `Float64`, so a NaN read from text stays one through the next cast. A double past an integer's bounds now saturates as `toInt` and `toLong` do, where `int()` raised on an infinity. - A moment was read with dateutil's `tzlocal()`, which knows only today's two offsets, so an instant from before a zone changed its rules came out an hour or more off on every platform. The script now reads the machine's zone by its IANA name and, before 1900, keeps today's standard offset as `java.util.Date` does. - On Windows, `localtime` refuses an instant before 1970, which stopped the cast. The same helper now falls back to the zone's standard offset there. That `Float64` column changed what a Python operator after this cast is handed. pyamber hands a Python operator a DOUBLE column as `float64`, where a NaN and a null are one value, and writes a NaN out as null. Given `Float64` instead, Sort ordered a NaN apart from the nulls, a chart drew points the engine dropped, and a passed-through NaN stayed NaN. `StandaloneCodeGenerator` gains `standaloneInputAdapter()`, and for a Python operator the translator feeds the fragment a `float64` copy of each input, leaving the upstream frame as it was. The number branch is left alone. A LONG is read as epoch milliseconds and has the same window, but it also disagrees about the calendar before 1582-10-15, where `new Timestamp(long)` is Julian and pandas is proleptic Gregorian. Widening the window there would answer with a moment eight days off instead of an empty cell, so what that branch should do is a separate decision. The issue records it. ### Any related issues, documentation, discussions? Closes apache#8595. apache#8480 closed the same nanosecond gap for Extract Date/Time Fields, which parses strictly and could take the one-line form this operator cannot. apache#8341, apache#8704, apache#8705 and apache#8709 call the timestamp and text helpers this adds, so this PR merges before them. ### How was this PR tested? Eight tests in `TypeCastingOpDescSpec` run the generated Python and compare it cell by cell against `AttributeTypeUtils.parseField`. They are tagged to run in the integration job, which has pandas. - The text-to-timestamp test reads moments in three formats, the rows either side of the nanosecond edge, `2500-01-01`, `1500-06-15`, `9999-12-31`, readings with `Z` and `+05:30`, and readings past the millisecond. It fails on the old generator. - One test gives a column that already holds timestamps and checks its nanoseconds are kept. - The cast-to-string test covers a double at 1e7 and 1e-4. - One test checks that unreadable text ends the script with the engine's message, as the engine refuses it. - Two tests cast a column holding a stored NaN, both infinities and an empty cell to each type and compare against `AttributeTypeUtils`, and check that a NaN read from text survives a second cast. - Two tests read moments from `America/Mexico_City` in 1899, 1921, 1931 and 2021, at the offset the zone had then, and the same moments with `localtime` refusing those before 1970. - Three tests in `WorkflowToPythonTranslatorSpec` check that a Python operator reads a converted copy of its input and an operator the engine runs on the JVM reads its upstream as it is. Two of them fail without the change. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Aggregate and the sort family (Sort, Stable Merge Sort and Sort Partitions) implement
StandaloneCodeGenerator. The sort family's row order is its contract, soLogicalOpgainsorderSensitive, which those three override. Anything comparing two runs of an operator reads it to decide whether the rows have to arrive in the same order or only be the same rows.Review then found five places where the block answered a question the engine answers differently. Aggregate summed an INTEGER column without the wrap Java ints have, refused a TIMESTAMP column outright, answered a timestamp average with a timestamp where AVERAGE is declared DOUBLE, and spelled a boolean in Python's capitals rather than Java's lower case. The sort put a NaN last on the way down, where the engine puts it first.
The edge values the shared table now carries found one more. Aggregate's concatenation wrote a double past 1e7 or under 1e-3, and a timestamp, as Python prints them, where the engine writes what Java's toString gives. It now writes every value, the lower-case boolean the review asked for included, through the helper #8597 adds for casting to text.
One change reaches code the engine runs.
getPhysicalPlanswapped Aggregate's aggregations for their global-stage form and left them on the operator. The export compiles a plan and then translates the same operators, so the script grouped on the result column names and failed. The operator now keeps what the user set. The desc the global stage receives is unchanged.Any related issues, documentation, discussions?
Part of #8325, step 11b; that issue lists the set in order. Aggregate's concatenation calls a helper #8597 adds, so #8597 merges first.
Sorting a NaN apart from a null, and grouping on one, need the nullable dtypes #8704 reads an Arrow file into. Without it a null and a NaN share a slot and both behave as they did before, so this PR is no worse on its own and only complete alongside that one.
Closes #8707, the task this change is the whole of.
How was this PR tested?
Each operator asserts the block it emits in its own spec. The Aggregate cases the review named run in the verify tables of #8363, which put the same input through the engine and through the script and compare the two. What those tables cannot reach stays here as executed tests: a timestamp at two resolutions and in a zone other than UTC, and a null kept apart from a NaN in the sort and the group key.
AggregateOpDescSpecbuilds the physical plan and then checks that the operator still holds the user's aggregations and that its generated code still reads the input column.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
🤖 Generated with Claude Code