Add typed reporting for Vally experiments

[Larry Osterman (LarryOsterman)]

What does this PR do?

Moves Vally experiment post-processing from PowerShell into a typed TypeScript reporter plugin so reporter API changes fail at compile time and experiment results have a reusable, dashboard-ready contract.

The reporter records per-experiment JSON and Markdown artifacts, aggregates all selected tools and iterations into consolidated summaries, compares each candidate against baseline, and ranks every variant including baseline. Invoke-VallyEval.ps1 now focuses on orchestration, keeps reporter dependencies current from the lockfile, rebuilds the reporter for every invocation, and reports aggregation failures on the error stream.

The Vally README documents the version 2 output schema, field semantics, effectiveness categories, ordering, and cross-repository consumption requirements. Validation included reporter unit tests, Azure MCP Server build, spelling checks, report-only reconstruction, and an Event Hubs evaluation run.

GitHub issue number?

N/A

Pre-merge Checklist

• Required for All PRs
  • Read contribution guidelines
  • PR title clearly describes the change
  • Commit history is clean with descriptive messages (cleanup guide)
  • Added comprehensive tests for new/modified functionality
  • Created a changelog entry if the change falls among the following: new feature, bug fix, UI/UX update, breaking change, or updated dependencies. Follow the changelog entry guide
• For MCP tool changes: N/A - this changes evaluation infrastructure only
  • One tool per PR: This PR adds or modifies only one MCP tool for faster review cycles
  • Updated servers/Azure.Mcp.Server/README.md and/or servers/Fabric.Mcp.Server/README.md documentation
  • Validate README.md changes running the script ./eng/scripts/Process-PackageReadMe.ps1. See Package README
  • For new or modified tool descriptions, ran ToolDescriptionEvaluator and obtained a score of 0.4 or more and a top 3 ranking for all related test prompts
  • For tools with new names, including new tools or renamed tools, update consolidated-tools.json
  • For renamed tools, follow the Tool Rename Checklist and tag the PR with the breaking-change label
  • For new tools associated with Azure services or publicly available tools/APIs/products, add URL to documentation in the PR description
• Extra steps for Azure MCP Server tool changes: N/A - no tool registration or descriptions changed
  • Updated command list in servers/Azure.Mcp.Server/docs/azmcp-commands.md
  • Ran ./eng/scripts/Update-AzCommandsMetadata.ps1 to update tool metadata in azmcp-commands.md (required for CI)
  • Updated test prompts in servers/Azure.Mcp.Server/docs/e2eTestPrompts.md
  • 👉 For Community (non-Microsoft team member) PRs:
    • Security review: Reviewed code for security vulnerabilities, malicious code, or suspicious activities before running tests (crypto mining, spam, data exfiltration, etc.)
    • Manual tests run: added comment /azp run mcp - pullrequest - live to run Live Test Pipeline

Invoking Livetests

Copilot submitted PRs are not trustworthy by default. Users with write access to the repo need to validate the contents of this PR before leaving a comment with the text /azp run mcp - pullrequest - live. This will trigger the necessary livetest workflows to complete required validation.

[azure-pipelines]

Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.