Inserting a delete command into the system.backup_actions is sync and leading to timeouts

[manasmulay]

Hello!
We have around 1PB data backed up using the clickhouse backup tool in azure blob storage, using a flat file system. Our setup is as follows - Some tables are hourly partitioned, some are partitioned on a day level. Every time data is inserted into a partition, we trigger a backup for that partition. Data is sharded too, and spread across a bunch of nodes. The backup mechanism is based on retries - try backup for all shards for a given partition, if some fail, delete all backups for that partition and try again (delete is only done because data merges might change the partition files, and a new backup without delete would not overwrite all the data in blob storage).
This delete mechanism is where we are facing issues and seeing timeouts. It looks like the delete action handler is sync and any queries to insert a delete remote statement into the system.backup_actions table are timing out mostly due to azure storage being slow. While we are working on making azure storage performant, our issue was as follows --

Is there any way to have the delete operation be sync? Also is there any use case of the delete operation being sync, and rest of the operations create, restore, upload etc being async?

[Slach]

deletion operation is sync
no way to async deletion

increase http_receive_timeout settings in your backup user profile
/etc/clickhouse-server/users.d/backup_http_receive_timeout.xml

<clickhouse>
<profiles><your-backup-user-profile><http_receive_timeout>3600</http_receive_timeout></profiles>
</clickhouse>

[manasmulay]

Thanks for this!