Make worker jobs recoverable after process interruption

[Cjay-Cyber-2]

Description

Harden the background worker so jobs interrupted by a process crash or deployment can be resumed or safely marked for retry instead of remaining stuck indefinitely. This improves settlement and anchor processing reliability without creating duplicate ledger actions.

Context & Requirements

The repository exposes a worker entry point at src/worker/index.ts and uses Prisma-backed application state. Stellar submissions and SEP-24 polling can outlive a single process. A worker must distinguish an in-progress lease from a completed operation and must not blindly replay a transaction after losing its local state. Recovery should use a lease timeout, bounded attempts, and the existing settlement or anchor state machine rather than adding a second job store.

State updates need to be conditional so two worker instances cannot process the same job simultaneously. Recovery should preserve auditability and record when a job is retried or moved to a terminal failure state.

Acceptance Criteria

• Processing jobs have a lease or equivalent ownership marker with an expiration time.
• A later worker can reclaim an expired job while an active worker remains protected from concurrent processing.
• Lease acquisition and state transitions use conditional database operations or transactions to prevent double ownership.
• Retry attempts are bounded and terminal failures are recorded with a safe reason.
• Recovery does not blindly resubmit a Stellar transaction when the ledger may already contain it; it checks the existing operation state first.
• Worker shutdown stops taking new work and allows the current operation to release or safely expire its lease.

Implementation Guidance

Start with src/worker/index.ts and inspect the Prisma schema and service modules used by settlement and SEP-24 processing. Reuse existing status fields and audit patterns where possible. Keep external Stellar and anchor calls in src/services/*, and design the lease duration around the longest supported polling or submission operation rather than a fixed arbitrary sleep.

Testing & Validation

Add tests for lease acquisition, concurrent workers, expired lease recovery, attempt exhaustion, graceful shutdown, and the case where a transaction was submitted before the process stopped. Use a fake clock or controlled timestamps. Run npm run build, npm test, and npm run lint; manually stop and restart the worker against a local test database if the project setup permits it.

Submission Guidelines

Assignment is required before starting. Open one PR that includes Closes #<issue-number>, includes any Prisma migration, and explains how the recovery path avoids duplicate Stellar or anchor actions.

---

Wave complexity: Medium