Skip to content

Attribute wrapped blob-hydration failures to their inner cause - #2077

Merged
tyrielv merged 1 commit into
microsoft:masterfrom
tyrielv:tyrielv/hydration-retryable-unwrap
Aug 6, 2026
Merged

Attribute wrapped blob-hydration failures to their inner cause#2077
tyrielv merged 1 commit into
microsoft:masterfrom
tyrielv:tyrielv/hydration-retryable-unwrap

Conversation

@tyrielv

@tyrielv tyrielv commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What

Attribute a wrapped blob-hydration failure to its real inner cause when categorizing telemetry.

RetryableException carries the real cause in InnerException. The BlobHydrationFailureCategory
logic added in #2071 checked the RetryableException type itself:

category = errorArgs.Error is IOException ? LocalIO : NetworkUnavailable;

RetryableException is not an IOException, so every RetryableException was tagged
NetworkUnavailable - even when it wrapped a local disk/IO cause. This over-counts the
"external / network" slice and under-counts LocalIO.

This matters because RetryableException is the largest blob-hydration failure bucket in the
field (~255 os.2020 mount machines, ~2.85M events in 30 days).

Change

  • Unwrap RetryableException.InnerException before categorizing.
  • Map IOException / UnauthorizedAccessException / Win32Exception (the local disk/IO family) to
    LocalIO.
  • A RetryableException whose inner cause is not local (e.g. HttpRequestException), or that has no
    inner cause, stays NetworkUnavailable.
  • Telemetry metadata only. No behavior change. The Warning-while-retrying / Error-on-terminal
    split is unchanged.

Which path actually reaches this branch

On this branch the RetryableException arrives from Context.Repository.TryCopyBlobContentStream -
typically StreamUtil wrapping an IOException while reading a corrupt or truncated local loose
object
(or the ProjFS write callback failing). Note: a RetryableException thrown while writing
a loose object (e.g. a full disk during download) is caught by the inner download retrier and
surfaces as DownloadFailed (no exception on the outer handler), so it does not reach this
branch. UnauthorizedAccessException / Win32Exception are mapped to LocalIO because they belong
to the same local disk/IO family.

Testing

  • Full unit suite: 898 passed, 0 failed (11 pre-existing skips).
  • Per-cause tests: IOException, UnauthorizedAccessException, Win32Exception -> LocalIO;
    HttpRequestException (non-local inner) and a bare no-inner RetryableException ->
    NetworkUnavailable.
  • The fixture resets the process-global RetryCircuitBreaker in both SetUp and TearDown so
    these failure-driving tests cannot open the circuit for one another or leave it dirty for a later
    fixture.

Rollout note (metadata distribution)

This moves a large volume of failures from NetworkUnavailable to LocalIO. The dashboard folds
both into "external", so the external-vs-actionable split is unaffected, but any per-category
NetworkUnavailable / LocalIO trend or alert will show a one-time step change at the deploy
boundary. Annotate the boundary and re-baseline per-category thresholds before using this data for a
release-readiness comparison. Fully reversible (pure metadata).

Relationship to #2071

Follow-up to #2071 (now merged), which introduced BlobHydrationFailureCategory. This branch has
been rebased onto master, so the diff is just the unwrap change (two files).

Companion

Consumes the BlobHydrationFailureCategory tag from #2071. The dashboard-side split that reads these
tags is EngSys PR 16223444 in devprod.git.telemetry.

@tyrielv
tyrielv force-pushed the tyrielv/hydration-retryable-unwrap branch from c5d0f3e to c788f02 Compare August 5, 2026 21:29
RetryableException wraps its real cause in InnerException. The blob-
hydration failure categorization checked the RetryableException type
itself, so every RetryableException - the largest hydration failure
bucket in the field - was tagged NetworkUnavailable, even when the real
cause was local disk/IO.

On this branch the RetryableException reaches OnFailure from
Context.Repository.TryCopyBlobContentStream - typically StreamUtil
wrapping an IOException while reading a corrupt or truncated local loose
object. Unwrap RetryableException.InnerException before categorizing, and
map IOException / UnauthorizedAccessException / Win32Exception (the local
disk/IO family) to LocalIO. A RetryableException whose inner cause is not
local (e.g. HttpRequestException), or that has no inner cause, stays
NetworkUnavailable. Telemetry metadata only; no behavior change.

Add unit tests for each inner-cause mapping (IOException, Unauthorized-
AccessException, Win32Exception -> LocalIO; HttpRequestException and no
inner -> NetworkUnavailable), and reset the process-global
RetryCircuitBreaker in the fixture SetUp and TearDown so these failure-
driving tests cannot open the circuit for one another or for a later
fixture.

Stacked follow-up to PR microsoft#2071; do not publish until microsoft#2071 merges.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tyrielv
tyrielv force-pushed the tyrielv/hydration-retryable-unwrap branch from c788f02 to 13794ab Compare August 5, 2026 23:00
@tyrielv tyrielv changed the title Attribute wrapped blob-hydration failures to their inner cause (stacked on #2071) Attribute wrapped blob-hydration failures to their inner cause Aug 5, 2026
@tyrielv
tyrielv marked this pull request as ready for review August 5, 2026 23:01
@tyrielv
tyrielv enabled auto-merge August 5, 2026 23:26
@tyrielv
tyrielv merged commit 2e85256 into microsoft:master Aug 6, 2026
68 of 69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants