Add support for vally

[Connie Yau (conniey)]

What does this PR do?

• Add automated Vally evaluations for Azure MCP tool changes, including result artifacts.
• Generate evaluation specs from existing end-to-end prompts and build metadata.
• Add shared evaluator utilities, App Configuration evaluations, tests, and contributor documentation.

GitHub issue number?

Fixes #2848

Pre-merge Checklist

• Required for All PRs
  • Read contribution guidelines
  • PR title clearly describes the change
  • Commit history is clean with descriptive messages (cleanup guide)
  • Added comprehensive tests for new/modified functionality
  • Created a changelog entry if the change falls among the following: new feature, bug fix, UI/UX update, breaking change, or updated dependencies. Follow the changelog entry guide
• For MCP tool changes:
  • One tool per PR: This PR adds or modifies only one MCP tool for faster review cycles
  • Updated servers/Azure.Mcp.Server/README.md and/or servers/Fabric.Mcp.Server/README.md documentation
  • Validate README.md changes running the script ./eng/scripts/Process-PackageReadMe.ps1. See Package README
  • For new or modified tool descriptions, ran ToolDescriptionEvaluator and obtained a score of 0.4 or more and a top 3 ranking for all related test prompts
  • For tools with new names, including new tools or renamed tools, update consolidated-tools.json
  • For renamed tools, follow the Tool Rename Checklist and tag the PR with the breaking-change label
  • For new tools associated with Azure services or publicly available tools/APIs/products, add URL to documentation in the PR description
• Extra steps for Azure MCP Server tool changes:
  • Updated command list in servers/Azure.Mcp.Server/docs/azmcp-commands.md
  • Ran ./eng/scripts/Update-AzCommandsMetadata.ps1 to update tool metadata in azmcp-commands.md (required for CI)
  • Updated test prompts in servers/Azure.Mcp.Server/docs/e2eTestPrompts.md
  • 👉 For Community (non-Microsoft team member) PRs:
    • Security review: Reviewed code for security vulnerabilities, malicious code, or suspicious activities before running tests (crypto mining, spam, data exfiltration, etc.)
    • Manual tests run: added comment /azp run mcp - pullrequest - live to run Live Test Pipeline

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds first-class support for running vally evaluations against Azure MCP tools by introducing a Vally evaluation generator (based on e2eTestPrompts.md + build_info.json), wiring it into a new GitHub Actions workflow, and documenting how contributors author/run eval specs.

Changes:

• Introduces eng/tools/VallyEvaluator to generate eval.yaml specs per tool namespace and supporting YAML/JSON serialization models.
• Adds a PR-triggered GitHub Actions workflow to build the server, generate evals, run vally eval, and publish vally-results artifacts.
• Adds contributor documentation and baseline configuration (.vally.yaml) plus an initial AppConfig eval spec.

Reviewed changes

Copilot reviewed 38 out of 39 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
tools/Azure.Mcp.Tools.AppConfig/tests/eval.yaml	Adds initial vally eval spec for App Configuration prompts.
eng/tools/VallyEvaluator/tests/VallyUtilitiesTests.cs	Adds unit test coverage for placeholder replacement utility.
eng/tools/VallyEvaluator/tests/VallyEvaluator.Tests.csproj	New test project for VallyEvaluator utilities.
eng/tools/VallyEvaluator/src/VallyYamlStaticContext.cs	YAML static context registration for source-generated YAML serialization.
eng/tools/VallyEvaluator/src/VallyUtilities.cs	Implements YAML serialization and eval spec generation helpers.
eng/tools/VallyEvaluator/src/VallyJsonContext.cs	Source-gen JSON context for reading build info metadata.
eng/tools/VallyEvaluator/src/VallyEvaluator.csproj	New VallyEvaluator tool project + dependencies.
eng/tools/VallyEvaluator/src/RunConfiguration.cs	Adds configuration model for the evaluator tool CLI binding.
eng/tools/VallyEvaluator/src/Resources/eval.template.yaml	Provides a starter YAML template for eval specs.
eng/tools/VallyEvaluator/src/Resources/eval.instructions.md	Adds vally-run instructions injected into AGENTS.md during workflow.
eng/tools/VallyEvaluator/src/Properties/launchSettings.json	Adds launch profiles for running the evaluator locally.
eng/tools/VallyEvaluator/src/Program.cs	Implements CLI entrypoint to generate per-namespace eval specs.
eng/tools/VallyEvaluator/src/Models/StimulusGraderConfig.cs	Adds models for vally grader config serialization.
eng/tools/VallyEvaluator/src/Models/Stimulus.cs	Adds model for vally stimulus entries.
eng/tools/VallyEvaluator/src/Models/Evaluation.cs	Adds model for vally evaluation spec root object.
eng/tools/VallyEvaluator/src/Models/BuildInfoModels.cs	Adds models matching build_info.json structure.
eng/tools/VallyEvaluator/src/BuildInfo.cs	Adds build-info loader using source-generated JSON context.
eng/tools/Tools.sln	Adds new tool projects to the tools solution and updates VS metadata.
eng/tools/McpToolEvaluator.Core/tests/PromptParserTests.cs	Updates test namespace to match the new core evaluator library.
eng/tools/McpToolEvaluator.Core/tests/McpToolEvaluator.Core.Tests.csproj	New test project for prompt parsing/datastore.
eng/tools/McpToolEvaluator.Core/src/Utilities.cs	Adds repo-root discovery utility for tools.
eng/tools/McpToolEvaluator.Core/src/PromptParser.cs	Moves/renames prompt parsing into shared core library and makes it public.
eng/tools/McpToolEvaluator.Core/src/PromptDatastore.cs	Adds prompt datastore grouping prompts by namespace.
eng/tools/McpToolEvaluator.Core/src/Models/TestPrompt.cs	Introduces shared TestPrompt model for evaluators/testers.
eng/tools/McpToolEvaluator.Core/src/McpToolEvaluator.Core.csproj	New shared evaluator core library project.
eng/tools/CopilotCliTester/src/Program.cs	Updates CopilotCliTester to use the new shared evaluator core types.
eng/tools/CopilotCliTester/src/Models/TestPrompt.cs	Removes now-duplicated local TestPrompt model.
eng/tools/CopilotCliTester/src/Models/JsonContext.cs	Updates JSON source-gen context to reference shared TestPrompt.
eng/tools/CopilotCliTester/src/CopilotCliTester.csproj	Adds project reference to shared evaluator core library.
eng/scripts/Invoke-VallyEvalTests.ps1	Adds script to run vally evals using build_info + generated evals.
docs/testing-with-vally.md	Adds contributor doc for authoring and running vally evaluations.
Directory.Packages.props	Adds central package versions needed by VallyEvaluator (binder/commandline + YAML generator/analyzers).
CONTRIBUTING.md	Links new vally testing documentation from contributing guide.
AGENTS.md	Adds guidance about vally runs (assume az login, avoid subscription_list).
.vally.yaml	Adds repo-level vally configuration for environments and suites.
.gitignore	Ignores .vscode/ and vally-results/ output.
.github/workflows/vally-eval.yml	Adds workflow to generate and run vally evals on tool PR changes and upload artifacts.
.github/aw/actions-lock.json	Updates gh-aw firewall container pins and adds GitHub MCP server image pin.
.github/agents/agentic-workflows.agent.md	Updates referenced gh-aw documentation links to newer version.

Comments suppressed due to low confidence (1)

.vally.yaml:27

• The suite entries for generated evals reference .work/evals/..., but the evaluator + workflow place them under .work/vally/evals/.... As written, vally suite basic won’t include the generated evals.

    evals:
      - ".work/evals/appconfig/"
      - ".work/evals/storage/"
      - ".work/evals/keyvault/"

[vcolin7]

I assume these are simply templates and that users can use any namespaces they want to test, right?

[Connie Yau (conniey)]

This is correct

[vcolin7]

I assume these are tool arguments. Do you think we'll ever need to pass a complex object that requires JSON serialization?

[Connie Yau (conniey)]

It does not look like it from the vally spec