[OPTIMIZATION] Cancel pending shards when fan-out fails in ALLSHARD search

[mohanrajendran]

What operations aren't optimal?

When an ALLSHARDS query receives its first shard error, the coordinator cancels only its own token. Active remote searches continue running because the coordinator never calls grpc::ClientContext::TryCancel().

Remote leaves already receive a CallbackServerContext and poll it through their existing cancellation checks. However:

• Client::SearchIndexPartition() does not expose a cancellation handle.
• Each outbound ClientContext is private to request state that is released from the completion callback.
• Remote searches continue until their normal completion or the RPC deadline, currently 25 seconds by default.
• gRPC retries may continue while the fan-out is already unrecoverable.
• Late successful callbacks can still perform conversion, interning, and aggregation after the result is known to be unusable.

The original first_node_error must remain the final client error. A cancellation caused by another shard failure must not replace it with CANCELLED or DEADLINE_EXCEEDED.

How could they be improved?

When the fan-out becomes unrecoverable:

• Transition the tracker to a thread-safe terminal state exactly once.
• Register each outbound search’s cancellation handle before dispatch.
• Safely call TryCancel() on active remote RPCs, including requests queued for asynchronous dispatch.
• Ensure terminal-state publication and handle registration are synchronized, so a request cannot start after cancellation without immediately receiving it.
• Keep request state and ClientContext alive until the RPC callback completes.
• Perform RPC cancellation outside the tracker mutex.
• Skip late callback aggregation while retaining callback cleanup.
• Preserve the first shard error and distinguish parent cancellation from deadline cancellation.
• Reuse the existing remote cancellation polling; no new leaf polling checks are required.
• Add deterministic remote tests, including cancellation versus dispatch/completion races, retry behavior, late callbacks, and verification that leaves observe gRPC cancellation.

Consistency-failure and SOMESHARDS semantics should be handled separately.