Issues

• map_from_entries dispatches a calendar-interval result into the microsecond overflow

  requires-triage
  apache/datafusion-comet#7083 · Filter by author dwsmith1983 opened on Oct 17, 2026
  1

  

• bench: add the shuffle benchmark cases that the #5905 review found missing

  area:shuffleenhancementgood first issueperformancerequires-triage
  apache/datafusion-comet#7080 · Filter by author andygrove opened on Oct 16, 2026

• perf: Celeborn push completion is polled every 10 ms and the transport hook proxies every channel of the shared factory

  area:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7079 · Filter by author andygrove opened on Oct 16, 2026

• perf: the native Celeborn writer pays a large fixed cost per frame (admission, schema, estimates, re-encodes)

  area:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7078 · Filter by author andygrove opened on Oct 16, 2026

• perf: the native Celeborn writer sends one frame per partition chunk with no coalescing

  area:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7077 · Filter by author andygrove opened on Oct 16, 2026

• perf: the native Celeborn writer reserves about 7x a batch and holds it through the network round trip

  area:memoryarea:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7076 · Filter by author andygrove opened on Oct 16, 2026

• perf: the JVM shuffle's native radix sort always runs six passes, and the JVM reserves scratch it never uses

  area:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7075 · Filter by author andygrove opened on Oct 16, 2026

• perf: the JVM columnar shuffle insert loop boxes, locks and copies every row more than once

  area:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7074 · Filter by author andygrove opened on Oct 16, 2026

• perf: a shuffle over a native plan takes the JVM row shuffle for round-robin and nested hash partitioning

  area:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7073 · Filter by author andygrove opened on Oct 16, 2026

• perf: use extend when buffering partition row indices in the native shuffle writer

  area:shuffleenhancementgood first issueperformancerequires-triage
  apache/datafusion-comet#7072 · Filter by author andygrove opened on Oct 16, 2026
  1

  

• perf: the native multi-partition shuffle writer copies each partition's tail batch and flushes the output file once per partition

  area:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7071 · Filter by author andygrove opened on Oct 16, 2026
  

• perf: NativeBatchDecoderIterator discards its direct buffer after every block over 128 KiB

  area:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7070 · Filter by author andygrove opened on Oct 16, 2026

• perf: investigate decoding shuffle blocks ahead of the consumer in native shuffle read

  area:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7069 · Filter by author andygrove opened on Oct 16, 2026

• perf: a stage that reads shuffle and ends in columnar-to-row crosses the FFI boundary twice per batch

  area:ffiarea:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7068 · Filter by author andygrove opened on Oct 16, 2026

• perf: the shuffle reader builds a new decompressor and zeroes a fresh body buffer for every block

  area:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7067 · Filter by author andygrove opened on Oct 16, 2026

• perf: Arrow IPC write paths still copy through Channels.newChannel's 8 KiB buffer

  enhancementperformancerequires-triage
  apache/datafusion-comet#7064 · Filter by author andygrove opened on Oct 16, 2026

• Native hashing of NullType and ANSI interval values fails with "Unsupported data type in hasher"

  area:expressionsarea:shufflebugpriority:highrequires-triage
  apache/datafusion-comet#7063 · Filter by author andygrove opened on Oct 16, 2026

• JVM columnar shuffle panics on a struct column with a NullType field

  area:shufflebugpriority:highrequires-triage
  apache/datafusion-comet#7062 · Filter by author andygrove opened on Oct 16, 2026

• perf: the sort-based JVM columnar shuffle is 2x slower than Spark at 2,000 partitions because every partition write pays builder, file and codec setup

  area:shuffleenhancementperformancerequires-triage
  apache/datafusion-comet#7061 · Filter by author andygrove opened on Oct 16, 2026

• perf: every native plan creates and deletes a temp directory on each Spark local disk, even when it never spills

  enhancementperformancerequires-triage
  apache/datafusion-comet#7060 · Filter by author andygrove opened on Oct 16, 2026
  

• perf: the bypass JVM columnar shuffle writer writes one-row blocks when its partitions' pages don't fit in a task's memory

  area:shufflebugperformancepriority:medium
  apache/datafusion-comet#7058 · Filter by author andygrove opened on Oct 16, 2026 · 1.2.0
  

• [EPIC] Nested types in the native Parquet and Iceberg writers: correctness and performance follow-ups

  area:Icebergarea:writerbugcorrectnessEPICperformancepriority:mediumrequires-triage
  apache/datafusion-comet#7051 · Filter by author andygrove opened on Oct 16, 2026

• Native Iceberg writes into NOT NULL columns fall back on Spark 3.5+ because Comet has no native AssertNotNull

  area:Icebergarea:writerenhancementperformancerequires-triage
  apache/datafusion-comet#7049 · Filter by author andygrove opened on Oct 16, 2026

• Native Parquet writer ignores outputTimestampType and writeLegacyFormat, and omits Spark's schema metadata

  area:writerbugpriority:lowrequires-triage
  apache/datafusion-comet#7048 · Filter by author andygrove opened on Oct 16, 2026
  

• Native Parquet writer ignores the datetime rebase modes, so Spark rejects or shifts its pre-1582 dates

  area:writerbugcorrectnesspriority:mediumrequires-triage
  apache/datafusion-comet#7047 · Filter by author andygrove opened on Oct 16, 2026