CLDSRV-973: keep serving when the delete path throws synchronously

[DarkIsDude]

Issue: CLDSRV-973

Draft — the containment fix is complete, but the broader question in Open questions should be settled before this merges.

Problem

A synchronous throw in the data layer while deleting an object does not become a 500 for that request. It escapes into the promise machinery, crosses an async callback that has already fired, and produces:

{"name":"S3","action":"DeleteObjects","bucket":"im-bucket-001",
 "object":"QRh8XuQk/19.pQUofZcHqIeBa3E1.rnd",
 "error":"Cannot read properties of undefined (reading 'protectAzureBlocks')"}

unhandledRejection:
Error: Callback was already called.
    at node_modules/async/dist/async.js:969:32
    at .../mongoclient/MongoClientInterface.js:1125:28
{"name":"S3","error":"Callback was already called.","level":"fatal"}

Node's default for an unhandled rejection is to exit, so one request kills the pod serving it. Clients retry automatically, so the failure walks the deployment: in the field this took all 25 cloudserver pods into CrashLoopBackOff and the ARTESCA S3 endpoint fully offline, from a single delete-objects call. Found during the IronMountain XDM scale campaign (OS-1095) on ARTESCA 4.3.0-rc2 / cloudserver:9.3.9.

Note the first log line: the request-scoped error was caught and logged with full context. The process still died on the second one.

Change

Contain it at the delete path in services.deleteObject: wrap the data.protectAzureBlocks(...) call so a synchronous throw is logged and returned as InternalError for that request, instead of escaping. The pod keeps serving traffic.

The specific trigger seen in the field (an unguarded backend-client lookup for a location removed from the overlay) is fixed separately in scality/Arsenal#2686 / ARSN-623. This PR is deliberately about the blast radius — so that the next throw on this path is a failed request rather than a fleet-wide outage.

Open questions for review

1. Is a local try/catch the right depth of fix? It closes the path with a confirmed outage, but any other synchronous throw reachable from a request handler has the same consequence. A process-level unhandledRejection handler that logs and attributes the rejection to its request — rather than exiting — would address the class instead of this one instance. That is a deliberate availability-vs-fail-fast policy decision, so I have not made it here. lib/server.js:89 already installs an uncaughtException handler; there is no unhandledRejection equivalent.
2. Double-callback specifically. Callback was already called. means the first callback already ran, i.e. the request had been answered. Arguably that case should always be logged and swallowed rather than escalated to fatal.
3. Target branch. Opened against development/9.3 since 9.3.9 is the deployed version, so Bert-E forward-ports to 9.4.
4. No test added — reproducing this needs a data-layer client that throws synchronously, and the meaningful assertion (the process stays alive) is awkward in-process. Suggestions welcome; the Arsenal PR carries the unit-level regression test for the trigger.

[bert-e]

Hello darkisdude,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request.
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Incorrect fix version

The Fix Version/s in issue CLDSRV-973 contains:

• None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 9.3.17

• 9.4.2

Please check the Fix Version/s of CLDSRV-973, or the target
branch of this pull request.

[codecov]

Codecov Report

❌ Patch coverage is 57.14286% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.16%. Comparing base (24fb6b3) to head (b497126).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
lib/services.js	57.14%	3 Missing ⚠️

❌ Your patch check has failed because the patch coverage (57.14%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/services.js	87.10% <57.14%> (-0.43%)	⬇️

@@                 Coverage Diff                 @@
##           development/9.3    #6255      +/-   ##
===================================================
- Coverage            85.17%   85.16%   -0.02%     
===================================================
  Files                  206      206              
  Lines                13424    13427       +3     
===================================================
+ Hits                 11434    11435       +1     
- Misses                1990     1992       +2     

Flag	Coverage Δ
unit	71.08% <57.14%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bert-e]

Branches have diverged

This pull request's source branch bugfix/CLDSRV-973/keep-serving-on-delete-path-typeerror has diverged from
development/9.4 by more than 50 commits.

To avoid any integration risks, please re-synchronize them using one of the
following solutions:

• Merge origin/development/9.4 into bugfix/CLDSRV-973/keep-serving-on-delete-path-typeerror
• Rebase bugfix/CLDSRV-973/keep-serving-on-delete-path-typeerror onto origin/development/9.4

Note: If you choose to rebase, you may have to ask me to rebuild
integration branches using the reset command.