Skip to content

[Microsoft.Extensions.AI.Evaluation.Reporting] Evaluation report redesign - #7609

Open
grafanaKibana wants to merge 24 commits into
dotnet:mainfrom
grafanaKibana:ai-eval-report-redesign
Open

[Microsoft.Extensions.AI.Evaluation.Reporting] Evaluation report redesign#7609
grafanaKibana wants to merge 24 commits into
dotnet:mainfrom
grafanaKibana:ai-eval-report-redesign

Conversation

@grafanaKibana

@grafanaKibana grafanaKibana commented Jul 4, 2026

Copy link
Copy Markdown

Closes #7593.

Summary

Redesigns the generated AI Evaluation HTML report as a shared Fluent UI app used by both the standalone report and the Azure DevOps extension. The serialized Dataset, ScenarioRunResult, and EvaluationMetric contracts are unchanged.

The previous single-page report is replaced by four focused views:

  • Overview — pass/fail KPIs, largest raw deltas, needs-attention metrics, and pass rate by scenario group.
  • Cases — searchable and filterable case list with inline transcripts, metrics, diagnostics, pagination, and new-case indicators.
  • History — per-metric run trends with mean, median, spread, and chronological deltas.
  • Comparison — side-by-side execution comparison with explicit added, removed, shared, and unavailable metric states.

What changed

  • Reorganized report components by feature and removed superseded Fluent UI v8 code.
  • Added responsive light/dark styling, deterministic development data, and Markdown/GFM rendering.
  • Corrected execution ordering, scenario aggregation, missing-metric handling, and delta presentation without fabricating values.
  • Improved keyboard navigation, ARIA relationships, announcements, focus behavior, and Azure DevOps resize/root lifecycle handling.
  • Added a Vitest and Testing Library suite covering model invariants, views, accessibility, routing, and host behavior.

Verification

  • npm test -- --no-file-parallelism216/216 tests across 31 committed test files.
  • Forced TypeScript build — passed.
  • ESLint — 0 errors; one Fast Refresh warning in the Azure DevOps entry point.
  • Standalone report build — passed.
  • Azure DevOps typecheck and build — passed.
  • Light/dark, responsive, keyboard, and shared-host behavior were exercised.

Risk

The primary risk is visual regression because both hosts share the same components. No public API or serialized data-shape changes are introduced.

Microsoft Reviewers: Open in CodeFlow

grafanaKibana and others added 13 commits July 1, 2026 14:31
Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
…ified comparisons

Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
…ed unused styles, refactored context and metric formatting

Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
@grafanaKibana

Copy link
Copy Markdown
Author

@dotnet-policy-service agree

…pendency

Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
…related logic

Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
…lity logic. Added more tests to test accessibility.

Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
…lity fixes, refactoring

Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
…on abstractions consistency

Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
- Fixed Summary.ts — collision & duplicate handling
- Removed orphaned @fluentui/react v8
- Fixed yellow shade in light theme
- Deleted more dead code

Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
@grafanaKibana
grafanaKibana marked this pull request as draft July 22, 2026 11:55
grafanaKibana and others added 3 commits July 22, 2026 14:56
…ting ones

Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
Signed-off-by: nikitareshetnik <reshetnik.nikita@gmail.com>
@grafanaKibana
grafanaKibana marked this pull request as ready for review July 26, 2026 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[AI Evaluation] Redesign the generated HTML report

1 participant