alter_replace: fix tests exposed by the 26.6 empty-source guard

[CarlosFelipeOR]

Fix #160

ClickHouse 26.6 refuses REPLACE PARTITION when the source has no parts in the requested partition. That broke 32 tests in alter_replace plus the concurrent actions scenario — and 26 of the 32 turned out to be tests that had been green while never replacing any data.

Why they were green before

Every scenario validates with check_partition_was_replaced, which asserts:

assert partition_values_destination == partition_values_source

Until 26.5, replacing from an empty source silently emptied the destination partition. The assertion then compared empty to empty and passed. It could not distinguish "replaced successfully" from "both are empty". The guard did not break these tests — it revealed they were not testing anything.

Upstream context: ClickHouse#23727, open since 2021, classified the old behaviour as silent data loss. The previous behaviour is still available via allow_replace_partition_from_empty_source = 1.

Changes

1. concurrent replace partitions — off-by-one (13 failures)

for parts in range(1, number_of_parts) with number_of_parts = random.randrange(1, 50) runs zero inserts when the draw is 1, so ~2% of partitions were created with no parts at all. Measured in the failing run: 96 of 99 partitions created, and every partition that failed had zero inserts logged.

2. engines with a SummingMergeTree source (8 failures)

SummingMergeTree drops rows whose summing columns are all zero. The shared insert only fills p and i, which are the partition and sorting keys and therefore not summed, so every row was discarded and the source ended up completely empty. Verified on 26.6.2: INSERT (p, i) → 0 rows, INSERT (p, i, Value) → 3 rows.

This became visible when #111 moved these tables from ORDER BY tuple() to ORDER BY i, which took i out of the summing set. Same pattern as #129.

3. engines with a CollapsingMergeTree source (5 failures)

The helper reused the partition column as the sign column (sign="p"), which forced partitions to {-1, 1} and one partition per table. To keep signs valid, the population drew random.choice([-1, 1]) as the partition value — so half the time the only partition was -1 while the test replaces partition 1. All 5 tables that drew -1 failed; all 11 that drew 1 passed.

Collapsing engines now take a dedicated Sign column. This frees p to hold the same 5 partitions as every other engine, and the sign varies per row, which preserves the intent of the original change without randomising the partition. As a side effect, collapsing becomes structurally possible: with sign="p", rows with opposite signs landed in different partitions and could never collapse.

The other callers of these wrappers keep the partition column as the sign, and unifying the population initially broke them: the shared helper inserts p = 1..N, so p = 2 was rejected with Code: 117. Incorrect data: Sign = 2 (must be 1 or -1). When the sign is the partition column, the population now picks the partition value from [-1, 1], which makes the generated SQL byte-for-byte identical to the helper that was removed. Caught by a 26.3 run of s3/export part.

4. partition types and data integrity — intentional empty-source tests (6 failures)

These deliberately replace from a source with no parts (ALTER TABLE ... DROP PARTITION 1, or a partition beyond the source's range). They now expect the refusal on 26.6+ and assert that the destination data is kept; older versions keep the previous expectation.

The expectation is matched by the DB::Exception: Source table prefix. Matching only has no parts in partition was not enough: query() skips its generic "output has exception" check only when the expected message itself contains Exception:, so the test still failed on the very exception it was asserting.

For data integrity this also resolves a contradiction. The requirement it declares says:

RQ.SRS-032.ClickHouse.Alter.Table.ReplacePartition.NonExistentPartition — "[ClickHouse] SHALL keep the data of the destination table when replacing partition form the non-existent partition of a source table."

The test asserted the opposite — that the destination was emptied. The new ClickHouse behaviour is what the requirement always asked for.

5. Swapped engine wrappers

partitioned_summing_merge_tree_table called create_aggregating_merge_tree_table and vice versa. Both engines were exercised, but under each other's names, so the reported engine in every combination involving them was wrong.

6. concurrent actions — the partition to replace was drawn blindly

The scenario deliberately destroys partitions (DROP, DETACH, MOVE) over 100 iterations, but drew partition_to_replace from random.randrange(1, 500) — the range the tables were created with, not the partitions that survived. Once a draw landed on a destroyed partition the guard fired, and the surrounding retries(timeout=60) could not recover: the draw sat outside the retry, so every attempt asked for the same partition, and that partition never comes back.

This is accumulated state, not a race. 128 of the 147 guard hits in the run were on partition 12, which iteration 2 had dropped 195s earlier.

The partition now comes from system.parts and is drawn inside the retry, so a partition removed by a concurrent action mid-attempt is simply reconsidered on the next attempt.

This corrects an earlier assessment in this PR that concurrent actions needed no change. That was based on a single job log which happened to be the one that passed — replace partition along other actions failed in 51 of the 53 results recorded in the CI database.

Decisions

• create_partitions_with_random_uint64 takes an optional extra_columns. With the default the generated SQL is byte-for-byte unchanged, so the ~41 other callers are unaffected.
• The collapsing wrappers keep sign="p" / version="i" as defaults, so the s3 and parquet callers keep the schema they had; only the engines test passes a dedicated sign column. Keeping the default was not enough on its own — see the note in change 3.
• We did not use allow_replace_partition_from_empty_source = 1 anywhere. It would have turned everything green while leaving 26 tests asserting a behaviour upstream fixed as a data-loss bug.
• Two other blind draws in concurrent_actions.py were left alone on purpose. replace_partition_with_single_concurrent_action has the same flaw, but the feature only runs one_replace_partition and replace_partition_along_other_actions, so it is dead code. replace_partition_from_another_table draws blindly too, but from inside the action, so each retry re-draws and it self-heals — and none of the 147 guard hits name the destination table as the source, which is what those actions would use.

Not covered

RQ.SRS-032.ClickHouse.Alter.Table.ReplacePartition.Concurrent.Manipulating.Partitions.Drop requires that DROP PARTITION waits for a REPLACE PARTITION on the same partition to finish. This suite cannot exercise it reliably: it needs two independent draws to pick the same partition (~1/500 per iteration) and to overlap in time. Covering it needs a dedicated test that forces the overlap. Tracked in #160.

References

• Investigation issue: Failure: alter_replace — REPLACE PARTITION from an empty source refused on 26.6+ #160
• Failing CI run: https://github.com/Altinity/ClickHouse/actions/runs/31008527066/job/92685321160
• Full-suite run with these changes: https://github.com/Altinity/clickhouse-regression/actions/runs/31151708532
• 26.3 run that caught the collapsing regression: https://github.com/Altinity/clickhouse-regression/actions/runs/31202263970/job/92944853146

[CarlosFelipeOR]

Running all suites to confirm these changes and check for fallout elsewhere: https://github.com/Altinity/clickhouse-regression/actions/runs/31151708532

Worth watching because helpers/create.py and create_partitions_with_random_uint64 are shared with s3/export_part, s3/export_partition and parquet. Most of it is a no-op for them (extra_columns defaults to None, collapsing keeps sign="p"), except that the swapped summing/aggregating wrappers now create the engine their name says.