Add support for vally - #3163

Open

Connie Yau (conniey) wants to merge 77 commits into

mainfrom

conniey/vally-experiment

Commits on Aug 6, 2026

Ignore any vally-results
Connie Yau (conniey)
committed
ea9cfa1
Initial Commit.
Connie Yau (conniey)
committed
983b070
Add VallyEvaluator and CopilotCliTester into Tools.sln
Connie Yau (conniey)
committed
1877e47
Move PromptParser and TestPrompt into .Core project
Connie Yau (conniey)
committed
44c7366
Move PromptParserTests.
Connie Yau (conniey)
committed
710537c
Add configuration to bind command line options and AOT compat for YAML serde
Connie Yau (conniey)
committed
56e08ca
Move prompt parser into common library
Connie Yau (conniey)
committed
90a04fc
Add VallyEvaluator.csproj
Connie Yau (conniey)
committed
17bc377
Add support to parse prompts and output evals
Connie Yau (conniey)
committed
a981e3e
Add development yaml
Connie Yau (conniey)
committed
8e28303
Update eval paths
Connie Yau (conniey)
committed
fbb0c79
Add evals for appconfig.
Connie Yau (conniey)
committed
81562f6
Get All namespaces
Connie Yau (conniey)
committed
c413803
Change to class.
Connie Yau (conniey)
committed
2382e11
Update to write to .work root.
Connie Yau (conniey)
committed
bb933b5
Add replacement variables.
Connie Yau (conniey)
committed
58ff551
Add dictionary for args
Connie Yau (conniey)
committed
921063b
Add test.
Connie Yau (conniey)
committed
f6bd7aa
- Add .vscode gitignore
Connie Yau (conniey)
committed
c6726d4
Allow manual invocation of workflow. Use github variables.
Connie Yau (conniey)
committed
a0a1534
Fix interpolation of environment variable to see if it is a PR or not.
Connie Yau (conniey)
committed
fe0eaf2
* Add different environments
Connie Yau (conniey)
committed
2848533
* Add Get-EvalPaths
Connie Yau (conniey)
committed
884edc9
Fix project path (#10)
Connie Yau (conniey)
committed
ddec7ff
Upload artifacts (#11)
Connie Yau (conniey)
committed
e49bd70
Add parameter to deserialize build_info and parse server information
Connie Yau (conniey)
committed
d513dd8
Update AGENTS.md to understand vally testing.
Connie Yau (conniey)
committed
fb05a16
Pass build info into VallyEvaluator project. Also limit concurrency.
Connie Yau (conniey)
committed
8f23116
* Add assumed env variables from Azure Pipelines used in our scripts.
Connie Yau (conniey)
committed
d3fa1ab
Move "Run vally logic" all into ps script
Connie Yau (conniey)
committed
c63b7ff
Add documentation
Connie Yau (conniey)
committed
be10477
Add "basic" suite
Connie Yau (conniey)
committed
77dc898
Set --output-dir
Connie Yau (conniey)
committed
0de9b03
Change script name.
Connie Yau (conniey)
committed
53e5ffe
Rename vally workflow
Connie Yau (conniey)
committed
bf5d1a7
Fix path to vally eval.
Connie Yau (conniey)
committed
657ee7a
Set working directory for vally
Connie Yau (conniey)
committed
e19525b
Add documentation to run config
Connie Yau (conniey)
committed
6779294
Add documentation to RunConfiguration
Connie Yau (conniey)
committed
1d65ecd
Add eval.instructions.md
Connie Yau (conniey)
committed
d74d30a
Replace AGENTS.md with eval instructions.
Connie Yau (conniey)
committed
fe9c90e
Add number of runs
Connie Yau (conniey)
committed
94ec41a
Adding VallyBaseDirectory. Evals written to baseDirectory/eval
Connie Yau (conniey)
committed
bf8f37f
Add baseVallyDirectory
Connie Yau (conniey)
committed
0a0db69
Add Deserializer
Connie Yau (conniey)
committed
d8bba14
Add optional command
Connie Yau (conniey)
committed
8cf6f23
Fix spelling
Connie Yau (conniey)
committed
b464ebc
Replacing hard coded environment with VALLY_ENVIRONMENT parameter
Connie Yau (conniey)
committed
3c181f6
Add instructions to CONTRIBUTING.md
Connie Yau (conniey)
committed
da6e48d
(FIXUP) Eval APpConfig
Connie Yau (conniey)
committed
c67ad26
Update .vally.yaml environment name to linux
Connie Yau (conniey)
committed
1db4a23
update environment name to linux
Connie Yau (conniey)
committed
352df58
Add documentation for testing.
Connie Yau (conniey)
committed
16f41dc
Update Microsoft solution.
Connie Yau (conniey)
committed
026b958
Fix formatting
Connie Yau (conniey)
committed
b2f02c6
Add header to files.
Connie Yau (conniey)
committed
f421f19
Potential fix for pull request finding

Connie Yau (conniey)
and
Copilot
committed
c91b252
Potential fix for pull request finding

Connie Yau (conniey)
and
Copilot
committed
d88e022
Potential fix for pull request finding

Connie Yau (conniey)
and
Copilot
committed
2577697
Potential fix for pull request finding

Connie Yau (conniey)
and
Copilot
committed
9f92c11
Potential fix for pull request finding

Connie Yau (conniey)
and
Copilot
committed
427202a
Potential fix for pull request finding

Connie Yau (conniey)
and
Copilot
committed
e5c490a
Potential fix for pull request finding

Connie Yau (conniey)
and
Copilot
committed
276f867
Potential fix for pull request finding

Connie Yau (conniey)
and
Copilot
committed
13239e5
Fix spacing error.
Connie Yau (conniey)
committed
e878b5b
Fix build error
Connie Yau (conniey)
committed
8b126fb
Fix invocation issue.
Connie Yau (conniey)
committed
faba5f4
Scope the run
Connie Yau (conniey)
committed
2094642
Update default environment to be placeholder environment
Connie Yau (conniey)
committed
30ca7ac
Move environment
Connie Yau (conniey)
committed
a0b049c
Add "interaction" column to clarify if prompt is enough.
Connie Yau (conniey)
committed
6651d6a
Update to parse interaction requirement.
Connie Yau (conniey)
committed
724fd9e
Add Interaction parsing from string value
Connie Yau (conniey)
committed
68d5f9f
Update instructions
Connie Yau (conniey)
committed
e73318a
Remove replacement logic.
Connie Yau (conniey)
committed
72f25d2
Add "none" column to line 169.
Connie Yau (conniey)
committed
202166e
Update with more clarification-required.
Connie Yau (conniey)
committed
535e3fe