Skip to content

guard delete against breaking incremental backups chains - #1494

Merged
Slach merged 7 commits into
masterfrom
1493-delete-required-backup-guard
Aug 7, 2026
Merged

guard delete against breaking incremental backups chains#1494
Slach merged 7 commits into
masterfrom
1493-delete-required-backup-guard

Conversation

@Slach

@Slach Slach commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Fix #1493.

delete local|remote <backup_name> deleted a backup which other backups reference via required_backup without any check. The breakage surfaced only later, when a descendant was downloaded or restored (ReadBackupMetadataRemote(RequiredBackup)), list remote kept printing +<parent> for a backup which no longer existed, and for object disks the descendant required parts blobs were deleted together with the <object_disk_path>/<backup_name> prefix of the parent. backups_to_keep_remote retention already protected such chains (storage.GetBackupsToDeleteRemote), only the explicit delete didn't.

Behavior

  • Both delete paths (CLI, POST /backup/delete/{where}/{name}, system.backup_actions) now collect the backups whose required_backup points at the deleted one and refuse to delete, naming the dependents:

    'full_backup' is required by remote backup(s) increment_backup, deleting it breaks the
    incremental backups chain, delete the dependent backup(s) first, run `rebase` for them,
    set `general.rebase_during_delete: true` or use `--force`
    
  • --force (force=1/force=true for the API) keeps the old chain-breaking behavior. Note it must precede the positional arguments: delete --force remote <name>.

  • New general.rebase_during_delete (env REBASE_DURING_DELETE, default false): rebase every dependent increment first (same as the rebase command), so the chain stays restorable and the backup becomes deletable. Opt-in, because rebase copies the deleted backup parts into its dependents and turns an O(1) delete into an unbounded data copy; a rebase failure aborts the delete.

  • backups_to_keep_local/backups_to_keep_remote retention, watch, clean_local_broken/clean_remote_broken, create rollback and --delete-source pass force=true, so their behavior is unchanged.

  • Not covered: remote_storage: customRemoveBackupRemote delegates to custom.DeleteRemote before any metadata is read, so there is nothing to check there.

Tests

  • unit: TestFindDependentBackups (local and remote chain link detection, self-reference, multiple children).
  • integration: new TestDeleteRequiredBackup — guard on delete local, backup survives the refused delete, --force, guard on delete remote, then REBASE_DURING_DELETE=true rebases the increment and the rebased increment restores after its ancestor is gone.
  • existing integration/testflows tests which deleted a parent while a descendant was still present now delete children first (childrenFirst in test/integration/utils.go, fullCleanup, rebase/restoreResolveIncrement/downloadRequiredBackupConnection/keepBackupRemote/hardlinks cleanups, cloud_storage.py), or pass --force where they break the chain on purpose (testBackupSpecifiedPartitions, hardlinksExistsFiles).

Local runs (macOS/arm64, ClickHouse 26.3): make test green; TestS3, TestDeleteRequiredBackup, TestRebaseS3, TestHardlinksExistsFiles, TestKeepBackupRemoteAndDiffFromRemote, TestKeepBackupRemoteWithRebase, TestRestoreResolveIncrementS3, TestDownloadRequiredBackupConnectionS3, TestProjections, TestDiffFromChecksums, TestResumeOperationsAfterRestartS3, TestWatchSchedule, TestServerAPIRebase all pass.

Slach added 7 commits August 3, 2026 20:45
`delete local|remote <backup_name>` deleted a backup which other backups
reference via `required_backup` without any check: the breakage surfaced
only later, when a descendant was downloaded or restored, `list remote`
kept printing `+<parent>` for a backup which no longer existed, and for
object disks the descendant `required` parts blobs were deleted together
with the `<object_disk_path>/<backup_name>` prefix of the parent.
`backups_to_keep_remote` retention already protected such chains, only
the explicit delete didn't.

Both delete paths now collect the backups whose `required_backup` points
to the deleted one and refuse to delete, naming the dependents. `--force`
(`force=1` for `POST /backup/delete/{where}/{name}`) restores the old
behavior, and `general.rebase_during_delete` rebases every dependent
increment first (same as the `rebase` command) so the chain stays
restorable - opt-in, because rebase copies the parent parts into the
dependents and turns an O(1) delete into an unbounded data copy.

Retention, watch and cleanup paths pass force=true, so their behavior is
unchanged. Integration tests which deleted a parent while a descendant
was still present now delete children first, or pass --force where they
break the chain on purpose.
…itions

restore_remote of the increment downloads it locally, so the local
fullBackupName it requires can't be deleted without --force anymore.
@Slach Slach added this to the 2.8.1 milestone Aug 4, 2026
@Slach
Slach merged commit 42db6a5 into master Aug 7, 2026
82 of 84 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

delete remote/local silently breaks incremental chains: no required_backup dependency check

1 participant