Repository navigation
Let greedy join reorder own join decisions with column statistics - #392
Merged
Merged
Conversation
…gurable - GreedyJoinReorder defers Cartesian products by connectivity when a region containing a cross join lacks row estimates; malformed join keys are now reported as InvalidPlan instead of silently skipping the region. - Add DataBaseBuilder::hll_precision (default 2^14 registers) and statistics_relative_error (KLL / Count-Min, default 0.001) and validate them when the database is built. - Reject duplicate visible table names in one FROM namespace. - Bind non-LATERAL derived tables without the sibling FROM sources and reject LATERAL explicitly.
- Treat `AliasType::Expr` targets as output identity only: evaluate, remap and substitute through `expr`; the default visitors skip the target, while deep clone and referenced-column collection still walk it. - Run Collapse Project before Combine Operators so filters reach the scan through rename projections, keeping the final collapse pass. - Let Guard Mark Apply skip Sort/Limit/TopK between the projection and filter, checking those operators do not read the markers.
- Invalidate schema caches along rewrite paths without clearing untouched subtrees, and preallocate concatenated join schemas. - Restore join output using tracked column positions instead of schema lookups, sharing projection construction with nested-loop orientation. - Remove the unused child replacement helper and streamline reorder tests.
- Limit InnerJoinApply conversion to complete primary/unique-key point probes. - Move matched rows into the output instead of cloning and rotating the left values, and remove the cross-call join input state. - Keep non-unique index joins on the ordinary join path and cover NULL, residual rejection, repeated probes, and complete composite keys.
Keep ordered name bindings with a single-source lookup fast path.\nAccount for intermediate/build width when reordering joins, move the final hash probe match, and classify predicates in one traversal.
Rewrite Project elimination as local bottom-up transformations and bound join-key source searches. Add guarded child traversal to operator visitors and adapt expression consumers.
- Add rfc/0001: GreedyJoinReorder decides join order, implementation, direction and probe index; unified row cost model; OrientNestedLoopJoin merged into Greedy. - Lower the default ANALYZE histogram bucket count from 100 to 10.
- PropagateFilterConstants: derive `b = c` from `a = b AND a = c` inside a filter, closing chains. - PropagateJoinConstants: derive constants across join equi-keys onto the inputs. Inner joins propagate both ways; left joins only to the right side, right joins only to the left side; full joins are skipped. Outer joins also read the preserved side's ON constants kept on the join. - Only plain table columns of identical types participate; NULL constants are ignored. - Run both rules in the "Predicate Pushdown" batch so derived predicates become index ranges. - Add unit tests and slt cases in where.slt. - Fix a missing AggregateOperator import in elimination tests.
HepBatch::new, before_batch/after_batch and implementations only iterate the given rules to build their own grouped storage, so accept &[NormalizationRuleImpl] / &[ImplementationRuleImpl] instead of owned Vecs. The pipeline builder now builds ImplementationRuleIndex directly instead of keeping an intermediate Vec. Also drop constant propagation from RFC 0001 future work, since it is implemented in 179c4b9.
- PushPredicateThroughJoin pushes a filter recursively through nested joins (push_through_join / push_into_child) instead of one level per fixpoint iteration. - ExprVisitor::visit_column_ref and all_referenced_columns now carry the column position, so join filter classification checks the slot at that position before falling back to a linear contains. - PropagateFilterConstants skips the propagation loop when the predicate has no column = column conjunct.
…izing Introduce TupleImpl (a plain Tuple or a binary Concat of two rows) as the executor result type. Joins (MarkApply, nested loop, hash) emit Concat rows instead of copying both inputs into one buffer, and every consumer reads positions through TupleLike, so no operator flattens a joined row. - Sort/top-k/window/spill/set membership/hash join state store TupleImpl; spill encodes Concat with the same wire format as a flat Tuple. - Storage accepts TupleImpl (append_tuple, serialize_to) and generic TupleLike index values (Index, with_index, add/del_index). - Public API: ResultIter::next_tuple, FromQueryRow and FromQueryTuple now take &mut TupleImpl. - Remove MoveableTuple, the ProjectMove rule and ProjectOperator::moveable; rewrite evaluates against TupleImpl and reuses the result buffer. - Add a Miri CI job for unsafe-covering unit tests.
- Add ConcatPool on ExecResult: Concat boxes are recycled as MaybeUninit when the arena splits a joined row (rewrite, output_mut) or MarkApply drops an unmatched left row, and reused by arena.concat. - Hash join and nested loop join copy the side that is matched many times into one flat buffer instead of cloning it into a new Concat. - Add TupleImpl::prepend/Extend for NULL padding and outer-row emission. - Add the `make miri` target used by the CI miri job.
- Drop the equality predicates already enforced by a parameterized index probe; MarkApply now treats NULL probe values as no match up front. - IndexScan reset rewinds static ranges and overwrites the Eq key of runtime probes in place instead of rebuilding the ranges. - Inner hash join builds the last match in the probe row's buffer, and TupleImpl::prepend extends the leftmost leaf of a joined row.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #392 +/- ##
==========================================
- Coverage 93.47% 93.39% -0.09%
==========================================
Files 260 272 +12
Lines 34472 39076 +4604
==========================================
+ Hits 32224 36495 +4271
- Misses 2248 2581 +333
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Inner-join probes shared MarkApplyOperator through a MarkApplyKind::InnerJoin variant, which forced kind checks across the executor, schema, costing and rewrite rules. Give them a dedicated IndexProbeJoin operator and executor, make the MarkApply output column required, and fold the quantified ANY scan over its Eq(x) and Eq(NULL) ranges into one pass.
…im redundant tests
- Replace the cached PhysicalOption cost with LogicalPlan::price, so a cost always reflects the current rows; drop WorkCost and clear_work_cost. - Join reorder inspect only collects schemas and keys; inner join rows were discarded by the rebuild. - Guard MarkApply walks referenced columns through the operator visitor instead of collecting them first. - Return an error instead of panicking on a loaded column without an id. - Drop tests already covered elsewhere and table-drive near-duplicates.
- StorageTransaction::transaction_with_isolation and Database::new_transaction take a read_only flag; LMDB read-only transactions skip the single-writer lock and reject writes. - run(), prepared plans and ORM reads pick read-only transactions by statement or plan classification. - CommandType moves to the planner; Operator::command_type classifies plans the same way as SQL statements (COPY .. TO is DQL). - Rename the storage Transaction trait to StorageTransaction and DBTransaction to Transaction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Join order was fixed by the query text and costed with coarse row estimates, quantified subqueries were re-evaluated per outer row, and the executor copied rows and projections between operators.
Issue link:
What is changed and how it works?
rfc/0001-greedy-join-reorder-owns-join-decisions.md); propagate constants through equality predicates and push predicates through nested joins in one pass.DataBaseBuilder) and account for filtered, parameterized, and cross-join input work in join costs.IndexProbeJoinoperator instead of a MarkApply variant.TupleImplthrough joins, reuse Concat boxes, eliminate intermediate projections, and move single-use values; fix index scans under aliased tables.make miriCI job and atest-slt-sqliterunner over the SQLite sqllogictest corpus from thetests/kitesql-testingsubmodule.Code changes
Check List
Tests
Side effects
Note for reviewer
make test,make test-slt, andmake native-examplespass locally; Miri and fuzz were not run locally and rely on CI.make test-slt-sqliteneedsgit submodule update --init; it is not in CI yet and currently fails 8x IN ()cases (empty IN lists are not parsed).HistogramMetalayout changed (column statistics,index_idmoved out), so statistics written by olderANALYZEmust be regenerated.