Commit 5dd720d
committed
feat: shared subplans (ReferenceRel) / DataFrame.cache() (CTEs)
Add CTE / shared-subplan support at both the builders.plan and native
DataFrame layers, carrying the shared subtrees in-band inside the resolved
Plan (the leading `rel` entries of Plan.relations, the way a Plan already
carries its extension declarations) rather than out-of-band via a contextvar.
type_inference: thread an optional `subtrees` list through infer_rel_schema /
infer_expression_type and add a `reference` case that resolves a ReferenceRel's
schema against subtrees[subtree_ordinal]. infer_plan_schema extracts the leading
subtrees and wraps them in a _SubtreeScope that memoizes each subtree's schema
(so a subtree referenced from many places is inferred once, not exponentially)
and detects reference cycles (a clear error instead of RecursionError).
builders: a `reference(plan)` builder promotes a plan to a shared subtree and
returns a ReferenceRel-rooted plan; every relational builder propagates its
inputs' shared subtrees upward and rebases subtree_ordinals when merging inputs,
structurally deduping identical subtrees so a cached frame reused across
branches is emitted once and referenced many times. A shared _plan_from helper
owns subtree propagation + Plan assembly, so each builder is a single call.
dataframe: DataFrame.cache() marks a frame as a reusable common subplan; the
hint() guard for RelCommon-less relations (e.g. a ReferenceRel) is restored.
A cached frame consumed inside a subquery is inlined into that subquery (a
plan-global ReferenceRel cannot cross the subquery boundary), keeping the
emitted plan valid.
Rel-walking primitives (plan_subtrees, rebase_reference_ordinals,
inline_reference_rels) live in substrait.utils and discover child-Rel fields
from the protobuf descriptor rather than a hand-maintained table.
Closes #2111 parent 154ebbe commit 5dd720d
9 files changed
Lines changed: 968 additions & 358 deletions
File tree
- src/substrait
- builders
- dataframe
- utils
- tests
- builders/plan
- dataframe
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
16 | 17 | | |
17 | 18 | | |
| 19 | + | |
18 | 20 | | |
19 | 21 | | |
20 | 22 | | |
| |||
1041 | 1043 | | |
1042 | 1044 | | |
1043 | 1045 | | |
1044 | | - | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
1045 | 1057 | | |
1046 | 1058 | | |
1047 | 1059 | | |
| |||
0 commit comments