Antalya 26.6: Hybrid tables - #2144
Conversation
…next commit) --- Original cherry-pick message follows: Merge pull request #1694 from Altinity/feature/antalya-26.3/pr-1442 Antalya 26.3 forward-port: Hybrid tables # Conflicts: # src/Core/Settings.cpp # src/Interpreters/ClusterProxy/executeQuery.cpp # src/Planner/Utils.cpp # src/Storages/StorageDistributed.cpp
…next commit) --- Original cherry-pick message follows: Merge pull request #1788 from Altinity/mkmkme/antalya-26.3/hybrid-segment-pruning Antalya 26.3 Hybrid: added support for segment pruning # Conflicts: # src/Storages/StorageDistributed.cpp
Kept the PR's `StorageDistributed::getStorageSnapshot()` override, which freezes the Hybrid watermark snapshot in `HybridSnapshotData`. Adapted: re-added the `getStorageSnapshot()` declaration in StorageDistributed.h, removed by antalya-26.6 commit 5d159b3 ("rewrite getStorageSnapshot to getInMemoryMetadataPtr"), so the source PR's override body compiles. Source-PR: #1788 (#1788)
zvonand
added
releasy
zvonand
added
the
forwardport
CI triage for #2144 (Antalya 26.6: Hybrid tables)Verdict — the only PR-caused failures are the two new Hybrid stateless tests ( Summary:
1. PR-caused:
|
|
@mkmkme could you pls take a look at the fails? ^^ |
Yup, will do |
The `Hybrid` table engine forward-ported here registers unconditionally in `registerStorages`, so it appears in `system.table_engines` regardless of `allow_experimental_hybrid_table`. It was not present in the `engine_to_type` map in `commandSelector.cpp`, so `commandGetTableTypes` and `commandGetTables` translated it through the `'UNKNOWN TABLE TYPE'` fallback of `transform`. That made `test_arrowflight_interface/test_sql_server.py::test_get_table_types` fail with `'UNKNOWN TABLE TYPE' in types`, and it would also report Hybrid tables as `UNKNOWN TABLE TYPE` to Flight SQL and ADBC clients doing table discovery. `Hybrid` is registered from `StorageDistributed.cpp` and is a member of the `Distributed` family, so it is mapped to `REMOTE TABLE` like `Distributed`. This did not show up before the forward port because the two halves lived on different branches: `antalya-26.3` has the `Hybrid` engine but no ArrowFlight server, while `antalya-26.6` has the ArrowFlight server but no `Hybrid` engine. Follows the same fix as b3a647c, which added `Filesystem` to the same map for the same test failure. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2144&sha=78baf9814831565b534e6a0338c83e500ca12aff&name_0=PR Related: #2144 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`03643_hybrid` and `03646_hybrid_segment_pruning` are the only Hybrid tests that assert `EXPLAIN` output, and both fail intermittently because `clickhouse-test` randomizes session settings that change the rendered plan. Three separate culprits were confirmed from the CI artifacts, each randomized to a non-default value with probability 0.05: * `query_plan_optimize_prewhere = 0` — `03646` on amd_debug. With the prewhere optimization disabled the WHERE step stays a `FilterStep` instead of being replaced by an `ExpressionStep`, so the plan renders `Filter ((WHERE + Change column names to column identifiers))` where the reference has `Expression (...)`. See `optimizePrewhere`, which only substitutes `ExpressionStep` when the filter is fully moved to `PREWHERE`. * `optimize_move_to_prewhere = 0` — `03643` on arm_binary. Same rendering flip, gated earlier in the same function. * `optimize_use_projections = 0` — `03646` on arm_binary. Test 6 then reads `ReadFromMergeTree` instead of `ReadFromPreparedSource (_exact_count_projection)`, which is a different failure mode from the two above. Chasing individual settings does not converge: these tests assert whole plan shapes, so any current or future randomized plan setting reopens the same failure. `no-random-settings` pins the session to the defaults the references were generated with and closes the whole class. The existing `SET` statements in `03646` are kept. The tag is not a superset of them: the `ParallelReplicas` job variants enable parallel replicas through the server config (`--parallel-rep` installs `users.d/enable_parallel_replicas.xml`, which puts `enable_parallel_replicas = 1` in the default profile) rather than through settings randomization, and that file does not set `parallel_replicas_local_plan`, so it stays at its default there and still has to be pinned by the test. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2144&sha=78baf9814831565b534e6a0338c83e500ca12aff&name_0=PR Related: #2144 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`03645_hybrid_watermarks`, `03646_hybrid_segment_pruning` and
`03682_system_hybrid_watermarks` all carried `no-fasttest` with the reason
`requires remote() table function`. That reason does not hold: 366 stateless
tests call `remote` or `remoteSecure`, carry no `no-fasttest` tag, and pass the
Fast test job, some with dozens of calls.
The strongest evidence is in this same group of tests. `03643_hybrid` is
untagged, passes Fast test, and exercises a strictly larger set of remote
infrastructure than any of the three: `cluster('test_shard_localhost')` plus
`remote` to `127.0.0.1:9000`, `127.0.0.2:9000` and `localhost:9000`. The three
tagged tests only use `remote('localhost:9000')` or `remote('127.0.0.1:9000')`.
Failpoints are not a reason either, which is the only other feature that
distinguishes `03682` from the untagged Hybrid tests: 61 stateless tests use
`SYSTEM ENABLE FAILPOINT` and run to completion in the Fast test job.
The `no-parallel` tag on `03682` is kept, since a process-wide failpoint really
does conflict with parallel execution.
Fast test is the earliest signal in the pipeline, so skipping three of the seven
Hybrid tests there — including the segment pruning test, which is the one that
has actually been failing — costs real coverage for no reason.
Related: #2144
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Pushed three commits:
|
mkmkme
left a comment
mkmkme
left a comment
There was a problem hiding this comment.
From my perspective looks correct. I haven't read the whole PR body thoroughly. I deliberately offloaded this task to Claude, and it reported that everything seems to be correct and fully front-ported.
The integration test run failure is not related to the PR. Good to go.
4 participants
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Add support for Hybrid tables, which combine local MergeTree storage with data in object storage behind a configurable watermark. Watermarks can be moved with ALTER TABLE MODIFY SETTING and inspected via the new system.hybrid_watermarks table, and queries now skip segments that cannot match the combination of the WHERE clause and the segment predicate.
DONE
Add support for Hybrid tables, which combine local MergeTree storage with data in object storage behind a configurable watermark. Watermarks can be moved with ALTER TABLE MODIFY SETTING and inspected via the new system.hybrid_watermarks table, and queries now skip segments that cannot match the combination of the WHERE clause and the segment predicate.
DONE (#1694 by @zvonand, #1723 by @mkmkme, #1788 by @mkmkme, #1789 by @mkmkme).
CI/CD Options
Exclude tests:
Regression jobs to run:
Combined port of 4 PR(s) (group
auto-grp-pr-1694). Cherry-picked from #1694, #1723, #1788, #1789.