Repository navigation
fix(spark): create QueryExecution via SessionState.executePlan - #1432
Conversation
Constructing `QueryExecution` directly binds to Apache Spark's exact constructor signature. Vendor builds such as the Databricks Runtime change that signature, so conversion fails there with `NoSuchMethodError`. `SessionState.executePlan` is the entry point Spark itself uses (`Dataset.ofRows`), and it leaves the constructor to the session. The change is applied to all three variants so vendor builds based on Spark 3.x are covered too.
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Delegate query execution to session state spark/src/main/spark-*/io/substrait/spark/compat/SparkCompatImpl.scala |
Each Spark compatibility implementation calls sessionState.executePlan(plan) instead of directly constructing QueryExecution. |
Priority:
Estimated code review effort: 2 (Simple) | ~10 minutes
Change: Bug fix
Merge Risk: 
21228
-
Query executions for Spark 3.4, 3.5 and 4.0 are now created through Spark's standard planning entry point, which should avoid constructor-signature mismatches on vendor Spark builds. No remaining issue blocks merging.
Pre-merge checks: 
-
Check name Status Explanation Title check The title clearly describes the primary change and follows the required Conventional Commit format: fix(spark): create QueryExecution via SessionState.executePlan.Description check The description provides a clear rationale, identifies the vendor-build compatibility issue, explains the replacement API, and states that all three Spark variants are updated. It satisfies the reposi… Docstring Coverage No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within survi… Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Full details: Description check
-
Explanation
The description provides a clear rationale, identifies the vendor-build compatibility issue, explains the replacement API, and states that all three Spark variants are updated. It satisfies the repository template.
Full details: Docstring Coverage
-
Explanation
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (3 skipped: 3 unsupported.)
-
Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
Constructing
QueryExecutiondirectly binds to Apache Spark's exact constructor signature. Vendor builds such as the Databricks Runtime change that signature, so conversion fails there withNoSuchMethodError.SessionState.executePlanis the entry point Spark itself uses (Dataset.ofRows), and it leaves the constructor to the session. The change is applied to all three variants so vendor builds based on Spark 3.x are covered too.