Host crash mid-activation permanently leaks the 'activating' VDI sm_config lock, blocking activations pool-wide (MAP_DUPLICATE_KEY)

[olivierlambert]

Summary

The activating VDI sm_config key that blktap2 uses as a pool-wide activation lock has no crash-safety: it is released in a finally clause, which protects against Python exceptions but not against the host dying mid-activation (HA fence, power loss, watchdog, sysrq reset). The leaked key then persists in the pool database forever, failing every subsequent RW activation of that VDI pool-wide with MAP_DUPLICATE_KEY, and it cannot be cleared with xe (sm-config is a read-only map in the CLI) nor by xe-toolstack-restart.

Observed incident

After chaos testing that hard-reset hosts during VM activity (lab pool, LVMoISCSI SR, XCP-ng 8.x), six VDIs were left with sm_config = activating: True and no activation in flight anywhere. Every subsequent start of the affected VMs failed on every host:

Error code: SR_BACKEND_FAILURE_46
The VDI is not available [opterr=['MAP_DUPLICATE_KEY', 'VDI', 'sm_config', 'OpaqueRef:...', 'activating']]

• xe vdi-param-remove cannot clear it: VDI sm-config is exposed by the CLI with only get/get_map handlers (xapi-cli-server/records.ml), so it fails with Cannot remove parameters from read-only map (cli_operations.ml).
• xe-toolstack-restart does not clear it: the key lives in the pool database.
• The only remedy is raw API surgery from dom0: session.xenapi.VDI.remove_from_sm_config(vdi_ref, 'activating').

Mechanism

VDI._add_tag() (libs/sm/blktap2.py) implements the activation lock through the xapi pool DB:

try:
    self._session.xenapi.VDI.add_to_sm_config(
        vdi_ref, 'activating', 'True')
except XenAPI.Failure as e:
    if e.details[0] == 'MAP_DUPLICATE_KEY' and not writable:
        # Someone else is activating - a retry might succeed
        return False
    raise                            # RW attach (every VM start): hard fail
host_key = "host_%s" % host_ref
...
self._session.xenapi.VDI.add_to_sm_config(vdi_ref, host_key, attach_mode)

and the release is a finally in _activate_locked():

finally:
    vdi_ref = self._session.xenapi.VDI.get_by_uuid(vdi_uuid)
    self._session.xenapi.VDI.remove_from_sm_config(vdi_ref, 'activating')

If the host dies between the add and the remove, the key stays behind. The key carries no owner identity and no timestamp, so nothing can arbitrate staleness afterwards — in explicit contrast to the sibling host_<host_ref> attach keys, which encode the owning host and get crash-recovery treatment in two places, both of which ignore activating:

• SM side: resetvdis.reset_sr(), run on every sr_attach (PBD plug), clears the re-attaching host's host_* key and paused, but not activating.
• xapi side: refresh_local_vdi_activations (storage_access.ml, called from dbsync_slave.ml at xapi startup) clears host_* keys owned by the restarting host or by hosts no longer in the pool, but not activating.

This combination is why the observed leak signature is exactly activating: True with no host_ key: after the crashed host reboots, its host_* key is cleaned up, and activating is orphaned forever.

Note the RO path is not meaningfully better off: MAP_DUPLICATE_KEY on a RO attach returns False, but the retry loop in activate() gives up after ATTACH_DETACH_RETRY_SECS and raises VDI locked.

A leaked key also breaks GC/coalesce

cleanup.py:_tagChildrenForRelink() waits 15×2s for activating to clear before tagging a leaf for relink, then raises SMException("Failed to tag vdi %s for relink"). So a leaked activating also permanently prevents leaf-coalesce on that VDI chain — silent snapshot-chain growth in addition to the visible VM start failures.

Reproduction

1. Pool with a shared SR (LVMoISCSI or any blktap2-activated type), one VM.
2. Start the VM; during the activation window on the executing host, hard-reset that host:
   xe vm-start uuid=$VM & then on the host, within ~1s: echo b > /proc/sysrq-trigger
   (Looping start/reset catches the window within tens of iterations; chaos testing catches it reliably. The blktap_activate_inject_failure/blktap_activate_error_handling FIST points can be used to widen the window.)
3. After the host returns: xe vdi-param-get uuid=$VDI param-name=sm-config → activating: True, no host_ key.
4. xe vm-start uuid=$VM → SR_BACKEND_FAILURE_46 / MAP_DUPLICATE_KEY, on every host, forever.

Impact

Any host crash during any VM start/attach on a shared SR can permanently block that VM's disks pool-wide until manual raw-API surgery, and permanently stall coalesce for the affected chains. HA deployments hit this naturally: the fence HA performs is exactly the reset that leaks the lock of whatever was activating at that moment.

Proposed fix

Give activating the same owner encoding and crash-recovery treatment as the host_* keys, without renaming the key (the GC's DB_VDI_ACTIVATING check and the snapshot sm_config filter in LVHDSR.py depend on the literal name):

1. _add_tag() stores the owning host_ref as the value (instead of 'True').
2. resetvdis.reset_sr() (already run on every sr_attach) removes an activating entry owned by the re-attaching host — a crashed host cleans up after itself as soon as it replugs its PBDs, before anything can attempt an activation.
3. resetvdis.reset_vdi() removes the entry under --force, or when the recorded owner is no longer part of the pool — making resetvdis single <uuid> a supported remedy for hosts that never come back.

Legacy bare 'True' values written by older code cannot be attributed to an owner and are only cleared by --force (no behaviour change for mixed pools during rolling upgrade).

An alternative considered — arbitrating at acquisition time on MAP_DUPLICATE_KEY by stealing the key when no host_* key is present — has a live race: a concurrent starter can observe the window between the two adjacent add_to_sm_config calls of a healthy activation and steal a live lock, allowing two RW tapdisks on shared storage. The owner-encoded cleanup above avoids that entirely.

[olivierlambert]

PR with the fix + unit tests: #804