Handle Spider terminal failures after compression-job commit

[coderabbitai]

Summary

A Spider compression job can be reported as Failed or Cancelled after its commit task has already completed. The commit task may already have published archive metadata and marked the corresponding CLP compression job as Succeeded. In that case, overwriting the job status based solely on the final Spider outcome produces an inconsistent result.

Immediate safeguard

When processing a Spider Failed or Cancelled outcome in S3CompressionJobHandle::to_completion, first determine whether the CLP compression job is already Succeeded. If it is, preserve Succeeded rather than changing it to Failed or Killed.

Proposed durable design

Introduce a Committed compression-job state between Running and Succeeded:

Running -> Committed -> Succeeded

Suggested responsibilities:

• The commit task transitions the compression job from Running to Committed while publishing archive metadata, within the same database transaction.
• Archive metadata created by a Spider job is associated with that Spider job ID. The archive table already has a creator_id column that can support this association but is not currently used by the package.
• A later Spider terminal failure/cancellation can use the Committed state and Spider-job association to identify and roll back the database side effects before marking the compression job terminally failed/killed.
• The final transition to Succeeded happens only after the system has confirmed the job should be considered successful.

Affected areas

• components/compression-coordinator/src/job_handle.rs (S3CompressionJobHandle::to_completion)
• Compression-job status/state definitions and database schema/migrations
• Spider commit-task transaction that publishes archive metadata
• Archive metadata persistence, including use of creator_id

Acceptance criteria

• Failed/cancelled Spider outcomes do not overwrite an already-Succeeded compression job.
• A Committed state and valid state-transition rules are defined and persisted.
• The commit task atomically records the committed state and archive metadata publication.
• Archive metadata can be traced to the originating Spider job ID.
• A failure/cancellation after commit can roll back the associated database side effects and leave a consistent terminal compression-job state.
• Tests cover normal success, failure before commit, cancellation before commit, and failure/cancellation reported after commit.

Backlinks

• PR: feat(compression-coordinator): Complete S3 compression job handle implementation. #2420
• Review discussion: feat(compression-coordinator): Complete S3 compression job handle implementation. #2420 (comment)
• Related context: feat(clp-tdl-package): Implement the clp-s compression commit task. #2415

Requested by @LinZhihao-723.