[Bug]: Emulator retries invalid invocation outputs that the service fails immediately

[yaythomas]

Expected Behavior

The emulator should mirror the service's handling of invalid invocation outputs. Verified service behavior:

Most invalid outputs fail the execution immediately, with no retry. This applies to:

• Status=FAILED with a Result present
• Status=SUCCEEDED with an Error present
• a missing or unrecognized Status
• an unparseable/malformed invocation output payload

In each case the service fails the execution with an error object of the form:

{
  "ErrorType": "InvalidParameterValueException",
  "ErrorMessage": "<the specific validation message>"
}

There is exactly one retried case: Status=PENDING when the execution has no pending operations. The service treats this as a transient runtime-level error (it can arise from SDK race conditions that resolve on replay) and retries the invocation, up to 3 consecutive attempts, before failing the execution with the same InvalidParameterValueException-shaped error.

Actual Behavior

In aws_durable_execution_sdk_python_testing.executor.Executor._finish_invocation(), all validation failures from _validate_invocation_response_and_store() are routed through the same retry path:

except (InvalidParameterValueException, IllegalStateException) as e:
    ...
    self._set_invocation_gate(execution_arn, InvocationState.PRE_INVOKE)
    self._retry_invocation(execution, error_obj)
    return

_retry_invocation re-invokes the handler up to MAX_CONSECUTIVE_FAILED_ATTEMPTS = 5 times with a flat RETRY_BACKOFF_SECONDS = 5 backoff before failing.

Consequences:

• A deterministically invalid output (e.g. SUCCEEDED with an Error) is re-invoked 5 times before failing, where the service fails on the first response. For a deterministic handler this adds ~25 seconds and 4 pointless invocations, and every re-invocation replays the handler.
• The one case the service does retry (PENDING with no pending operations) gets 5 attempts instead of 3.
• The terminal error surfaced to the customer is ErrorObject.from_exception(e) rather than the service's {ErrorType: "InvalidParameterValueException", ErrorMessage: ...} shape.

Proposed Fix

In _finish_invocation, distinguish the two classes:

• Fail-fast validation errors (FAILED+Result, SUCCEEDED+Error, missing/unknown status, malformed output): fail the execution immediately with an InvalidParameterValueException-typed error object carrying the validation message.
• PENDING with no pending operations: keep the retry path, with a consecutive-attempt cap of 3.

Regression tests:

• each fail-fast case produces a FAILED execution after a single invocation, with the InvalidParameterValueException error shape;
• PENDING with no pending operations retries and fails after 3 consecutive attempts;
• a transient case (invalid output once, then a valid PENDING/SUCCEEDED) recovers without failing the execution.

Related

• Found while investigating [Bug]: FAILED invocation without ErrorObject is completed as SUCCEEDED by local emulator #656 (FAILED without ErrorObject completed as SUCCEEDED by local emulator). This issue covers the adjacent divergence in the same code path: how invalid outputs are retried vs. failed.

SDK Version

aws-durable-execution-sdk-python-testing current main as of 2026-08-21.

Python Version

3.13

Is this a regression?

No known working version.