Export partition - check name and pos of columns in partition key. by k-morozov · Pull Request #2134 · Altinity/ClickHouse

k-morozov · 2026-07-30T14:58:29Z

Closes: #2123

Changelog category (leave one):

Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixed ALTER TABLE ... EXPORT PART and EXPORT PARTITION schema validation to prevent silently writing values into incorrect destination columns.

The export now verifies, by name and position only, that the top-level column owning each PARTITION BY column or subcolumn occupies the same position in the source and destination schemas. If that column contains a named
Tuple, its element names must also be declared in the same order - recursively through nested tuples and through container types such as Array and Map (both the key and value type of a Map are checked). This protects both by-name subcolumn access (PARTITION BY t.a) and positional expressions (PARTITION BY tupleElement(t, 1), PARTITION BY tupleElement(arr[1], 'a')) from silently changing meaning after the positional cast.

This check only compares names, positions, and Tuple element layout of the owning columns - it does not verify that the corresponding types are compatible (e.g. it does not detect a DateTime/DateTime64 partition key column
using different time zones in the source and destination tables, nor a Tuple element whose name matches but whose type differs). Such type mismatches, like all non-partition-key type mismatches, are still caught separately by
the existing positional cast check (rejected unless export_merge_tree_part_allow_lossy_cast = 1 is set).

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

Regression jobs to run:

github-actions · 2026-07-30T14:59:47Z

Workflow [PR], commit [9286316]

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

k-morozov · 2026-08-03T13:46:41Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: faef463617

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-03T13:55:04Z

+        auto partition_key_columns = source_metadata->getColumnsRequiredForPartitionKey();
+        const std::unordered_set<String> partition_key_column_set(
+            std::make_move_iterator(partition_key_columns.begin()),
+            std::make_move_iterator(partition_key_columns.end()));


Normalize subcolumns before checking partition positions

When PARTITION BY references a subcolumn such as t.a, getColumnsRequiredForPartitionKey returns the subcolumn name, while source_columns contains only the top-level readable column t; consequently, this set never matches and the new positional check is skipped. For example, source columns (t Tuple(a UInt32), u Tuple(a UInt32)) and destination columns (u Tuple(a UInt32), t Tuple(a UInt32)), both partitioned by t.a, have positionally compatible types and pass validation, but export source u into destination t while constructing the partition from source t.a, silently mispartitioning the data. Map required subcolumns back to their top-level storage columns before constructing this set.

Useful? React with 👍 / 👎.

It's not a problem. Source-id here https://github.com/Altinity/ClickHouse/blob/antalya-26.3/src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp#L985 mapped only for columns (not for subcolumns).

up: not quite.

If it is not a problem, can you write a test for it?

ianton-ru · 2026-08-03T14:14:44Z

            const auto & source_column = source_columns[i];
            const auto & destination_column = destination_columns[i];
+
+            if (partition_key_column_set.contains(source_column.name) && source_column.name != destination_column.name)


What if partitioning is for several columns, and destination has same columns but in different order?
Both have columns 'a' and 'b', in same order, but source with partition by a,b, and destination partition by b,a.

Or when destination has more columns in partition list than source.

I added the tests for simular cases:

What if partitioning is for several columns, and destination has same columns but in different order?
https://github.com/Altinity/ClickHouse/pull/2134/changes#diff-28d3e30160a3442e2b9ef01e3bf0b10b4c27de6a39efea3175ccec0a3f2d8949R1786

Or when destination has more columns in partition list than source.
https://github.com/Altinity/ClickHouse/pull/2134/changes#diff-28d3e30160a3442e2b9ef01e3bf0b10b4c27de6a39efea3175ccec0a3f2d8949R1799

ianton-ru · 2026-08-03T14:22:41Z


+    namespace
+    {
+        std::optional<String> getDateTimeTimeZoneName(const DataTypePtr & type)


I found method getExplicitTimeZoneOfDateTimeArgument like this.
And fragment from method extractTimeZoneNameFromFunctionArguments.
May be possible to reuse something?

Seems they are a slightly defferent. But I could use checkAndGetDataType from them.

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

mkmkme

The code indeed does what is expected from the description.

However I'm a bit hesitant still to just approve it because I'm not sure whether the description matches to what we actually want to do.

Let's say we have source columns (a, b, c, d, e) with partition by (a, b).
As expected, destination having (e, d, c, b, a) or even (b, a, c, d, e) will be rejected.
However, situations like (a, b, e, d, c) will be accepted (assuming matching types). And sometimes, especially considering how easy it can be to make a typo/mistake, it can lead to quite surprising results.

Overall, for me this fix looks good, but I would like to continue the discussion about all possible outcomes tomorrow. If we decide that this solution is fine for now I'll approve it :)

mkmkme · 2026-08-04T11:38:20Z

Here's the output of /audit-report:

High: Tuple subcolumns bypass positional validation
    Impact: Data can be silently written into the wrong nested field and Hive partition.
    Anchor: src/Storages/MergeTree/ExportPartitionUtils.cpp / verifyExportSchemaCastable, lines 675–714
    Trigger: Both tables partition by `t.a`, but destination changes `Tuple(a, b)` to `Tuple(b, a)` with compatible element types.
    Why defect: Required key name `t.a` never matches the top-level iterated name `t`; positional tuple conversion then swaps values while the path retains source `t.a`.
    Fix direction (short): Resolve subcolumns to physical columns and recursively validate relevant tuple member names/order.
    Regression test direction (short): Export the reordered tuple case to Hive and verify synchronous rejection.

High: Wrapped temporal types bypass timezone protection
    Impact: Files can be committed under one incorrect Hive path or Iceberg partition tuple, breaking pruning and metadata correctness.
    Anchor: src/Storages/MergeTree/ExportPartitionUtils.cpp / getDateTimeTimeZoneName and verifyExportSchemaCastable, lines 643–649 and 714–735
    Trigger: `Nullable` or `LowCardinality(DateTime[64])` uses different source/destination timezones with `export_merge_tree_part_allow_lossy_cast = 1`.
    Why defect: Timezone extraction handles only bare temporal types; lossy-cast opt-in then skips the remaining rejection.
    Fix direction (short): Unwrap `Nullable` and `LowCardinality` before timezone comparison.
    Regression test direction (short): Cover wrapped timestamps spanning a destination timezone boundary for direct and replicated exports.

Medium: Timezone-invariant partition expressions are rejected
    Impact: Valid Hive exports fail with `BAD_ARGUMENTS`.
    Anchor: src/Storages/MergeTree/ExportPartitionUtils.cpp / verifyExportSchemaCastable, lines 714–728
    Trigger: Differing column timezones with `PARTITION BY toUnixTimestamp(ts)` or an explicitly normalized timezone expression.
    Why defect: The guard checks required input columns without determining whether the partition result depends on their declared timezone.
    Fix direction (short): Validate computed partition semantics or recognize timezone-invariant expressions.
    Regression test direction (short): Confirm normalized expressions succeed while timezone-sensitive expressions remain protected.

Note the first one. It's actually the very same as was reported by codex and then dismissed. Could you double-check that please whether this is something we can simply dismiss or need to fix?

mkmkme · 2026-08-05T14:40:12Z

As discussed on the call, this change is good to go. Still, @k-morozov could you double-check the audit report on whether its first "High" issue is actually bogus and inapplicable?

arthurpassos · 2026-08-06T11:55:28Z

+                    i,
+                    destination_column.name);
+
+            if (partition_key_column_set.contains(source_column.name))


This is not very related to this PR, is it? It seems like a fix for #2138. I pushed a similar fix to it in my pr, but a more permissive one since the goal on my pr is to allow the monotonic stuff.

I think it is ok to keep this change here, and then I'll modify it in my pr. Your pr is likely going to be merged first.

arthurpassos · 2026-08-06T11:58:26Z

+        }
+    }
+
+    void verifyMergeTreePartitionCompatibility(


I would rename this to something like assertPartitionKeyASTAreEqual

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

k-morozov · 2026-08-06T14:51:36Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9ce7987f46

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

k-morozov · 2026-08-06T16:40:47Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2d7f3b2e68

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-06T16:47:58Z

+                const auto source_time_zone = getDateTimeTimeZoneName(source_resolved->type);
+                const auto destination_time_zone = getDateTimeTimeZoneName(destination_resolved->type);


Inspect nested DateTime types in partition columns

When the partition key is a composite column, such as t Tuple(ts DateTime('UTC')), and the destination declares the same t column with DateTime('Asia/Tokyo'), these calls pass the whole Tuple type to getDateTimeTimeZoneName, which returns no timezone. canBeSafelyCast also accepts the tuple because DateTime::equals ignores timezone, so the export bypasses the new guard; with the wildcard strategy, WildcardPartitionStrategy computes the path using toString(t), causing the nested timestamp to be rendered in the destination timezone and silently changing the partition identifier. Recursively compare nested DateTime/DateTime64 members of partition-key columns.

Useful? React with 👍 / 👎.

The comment isn't related to the current changes — yes, we have a problem with DateTime (and more), but I'm adding name and position checks for the column in the partition by expression.

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

k-morozov · 2026-08-07T10:58:25Z

@codex review

chatgpt-codex-connector · 2026-08-07T11:07:33Z

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: bce08c3b15

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

k-morozov · 2026-08-07T11:10:49Z

Here's the output of /audit-report:

High: Tuple subcolumns bypass positional validation
    Impact: Data can be silently written into the wrong nested field and Hive partition.
    Anchor: src/Storages/MergeTree/ExportPartitionUtils.cpp / verifyExportSchemaCastable, lines 675–714
    Trigger: Both tables partition by `t.a`, but destination changes `Tuple(a, b)` to `Tuple(b, a)` with compatible element types.
    Why defect: Required key name `t.a` never matches the top-level iterated name `t`; positional tuple conversion then swaps values while the path retains source `t.a`.
    Fix direction (short): Resolve subcolumns to physical columns and recursively validate relevant tuple member names/order.
    Regression test direction (short): Export the reordered tuple case to Hive and verify synchronous rejection.

High: Wrapped temporal types bypass timezone protection
    Impact: Files can be committed under one incorrect Hive path or Iceberg partition tuple, breaking pruning and metadata correctness.
    Anchor: src/Storages/MergeTree/ExportPartitionUtils.cpp / getDateTimeTimeZoneName and verifyExportSchemaCastable, lines 643–649 and 714–735
    Trigger: `Nullable` or `LowCardinality(DateTime[64])` uses different source/destination timezones with `export_merge_tree_part_allow_lossy_cast = 1`.
    Why defect: Timezone extraction handles only bare temporal types; lossy-cast opt-in then skips the remaining rejection.
    Fix direction (short): Unwrap `Nullable` and `LowCardinality` before timezone comparison.
    Regression test direction (short): Cover wrapped timestamps spanning a destination timezone boundary for direct and replicated exports.

Medium: Timezone-invariant partition expressions are rejected
    Impact: Valid Hive exports fail with `BAD_ARGUMENTS`.
    Anchor: src/Storages/MergeTree/ExportPartitionUtils.cpp / verifyExportSchemaCastable, lines 714–728
    Trigger: Differing column timezones with `PARTITION BY toUnixTimestamp(ts)` or an explicitly normalized timezone expression.
    Why defect: The guard checks required input columns without determining whether the partition result depends on their declared timezone.
    Fix direction (short): Validate computed partition semantics or recognize timezone-invariant expressions.
    Regression test direction (short): Confirm normalized expressions succeed while timezone-sensitive expressions remain protected.

Note the first one. It's actually the very same as was reported by codex and then dismissed. Could you double-check that please whether this is something we can simply dismiss or need to fix?

@mkmkme Regarding first comment - in this description we don;t have a problem. It is safely use Tuple(a, b) instead Tuple(b,a). But we have problem if we use tupleElement with position. I added tests. Other points are not related with current PR.

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

k-morozov added the antalya label Jul 30, 2026

k-morozov added 4 commits July 31, 2026 17:11

add tests

9209361

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

refactoring

1a6d63c

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

add check names and position

420f2e8

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

add tests

5322e48

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

k-morozov force-pushed the bugfix/antalya/26.3/export-incorect-order branch from 9e63fd6 to 5322e48 Compare July 31, 2026 15:11

k-morozov added 2 commits August 3, 2026 14:37

update tests, add hive part check

cb859eb

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

up tests

faef463

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

k-morozov changed the title ~~[WIP] Export partition - Positional column matching might lead to incorrect partitioning~~ Export partition - Positional column matching might lead to incorrect partitioning Aug 3, 2026

k-morozov marked this pull request as ready for review August 3, 2026 12:54

chatgpt-codex-connector Bot reviewed Aug 3, 2026

View reviewed changes

ianton-ru reviewed Aug 3, 2026

View reviewed changes

refactoring

31116d6

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

mkmkme reviewed Aug 4, 2026

View reviewed changes

k-morozov changed the title ~~Export partition - Positional column matching might lead to incorrect partitioning~~ Export partition - check name and pos of columns in partition key. Aug 5, 2026

arthurpassos reviewed Aug 6, 2026

View reviewed changes

k-morozov added 3 commits August 6, 2026 15:34

verify subcolumns

9c4ee5a

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

lowcardinality and nullable

a536161

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

sensetive column

9ce7987

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

chatgpt-codex-connector Bot reviewed Aug 6, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/ExportPartitionUtils.cpp Outdated

Comment thread src/Storages/MergeTree/ExportPartitionUtils.cpp Outdated

k-morozov marked this pull request as draft August 6, 2026 15:17

k-morozov added 2 commits August 6, 2026 17:38

resolve some cases

f6be56b

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

fix problems with func

7bf6fa5

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

issue with func with tz

2d7f3b2

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

k-morozov marked this pull request as ready for review August 6, 2026 16:39

Altinity deleted a comment from chatgpt-codex-connector Bot Aug 6, 2026

chatgpt-codex-connector Bot reviewed Aug 6, 2026

View reviewed changes

k-morozov marked this pull request as draft August 6, 2026 17:48

k-morozov added 3 commits August 7, 2026 11:53

refuse type checking

1680452

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

fix tupleElement

acf634a

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

tuple in array/map

bce08c3

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

k-morozov marked this pull request as ready for review August 7, 2026 11:08

k-morozov added 2 commits August 7, 2026 14:59

support unnamed tuple

c5bdb02

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

up doc

9286316

Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

		const auto source_time_zone = getDateTimeTimeZoneName(source_resolved->type);
		const auto destination_time_zone = getDateTimeTimeZoneName(destination_resolved->type);

Uh oh!

Conversation

k-morozov commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k-morozov commented Aug 3, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

k-morozov Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkmkme left a comment

Choose a reason for hiding this comment

Uh oh!

mkmkme commented Aug 4, 2026

Uh oh!

mkmkme commented Aug 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-morozov commented Aug 6, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

k-morozov commented Aug 6, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-morozov commented Aug 7, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 7, 2026

Uh oh!

k-morozov commented Aug 7, 2026

Uh oh!

Reviewers

k-morozov commented Jul 30, 2026 •

edited

Loading

github-actions Bot commented Jul 30, 2026 •

edited

Loading

k-morozov Aug 4, 2026 •

edited

Loading