You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| `api_key` | Yes | Your [Braintrust API key](https://www.braintrust.dev/app/settings/api-keys). |
48
+
| `runtime` | Yes | The runtime to use: `node`, `python`, or `go`. |
49
+
| `root` | No | Root directory containing your evals. Defaults to `.`. |
50
+
| `paths` | No | Paths or glob patterns, relative to `root`, containing evals to run. Defaults to `.`. |
51
+
| `package_manager` | No | `npm` or `pnpm` for Node; `pip` or `uv` for Python; `go` for Go. Can be omitted for the default package manager. |
52
+
| `use_proxy` | No | Set to `true` to use the Braintrust proxy at `https://braintrustproxy.com/v1`, which can cache repetitive LLM calls and speed up evals. Defaults to `true`. |
53
+
| `terminate_on_failure` | No | Set to `true` to stop the eval process when an error occurs. Defaults to `false`. Ignored for Go evals. |
54
+
| `github_token` | No | GitHub token used to create or update PR comments. Defaults to `${{ github.token }}`. |
32
55
33
56
## Full example
34
57
35
58
```yaml
36
59
name: Run pnpm evals
37
60
38
61
on:
62
+
pull_request:
39
63
push:
40
-
# Uncomment to run only when files in the 'evals' directory change
41
-
# - paths:
42
-
# - "evals/**"
64
+
# Uncomment to run only when files in the 'evals' directory change.
65
+
# paths:
66
+
# - "evals/**"
43
67
44
68
permissions:
45
-
pull-requests: write
46
69
contents: read
70
+
pull-requests: write
47
71
48
72
jobs:
49
73
eval:
@@ -52,74 +76,68 @@ jobs:
52
76
53
77
steps:
54
78
- name: Checkout
55
-
id: checkout
56
79
uses: actions/checkout@v4
57
80
with:
58
81
fetch-depth: 0
59
82
60
83
- name: Setup Node.js
61
-
id: setup-node
62
84
uses: actions/setup-node@v4
63
85
with:
64
-
node-version: 20
86
+
node-version: 22
65
87
66
-
- uses: pnpm/action-setup@v3
88
+
- name: Setup pnpm
89
+
uses: pnpm/action-setup@v4
67
90
with:
68
-
version: 8
91
+
version: 10
69
92
70
-
- name: Install Dependencies
71
-
id: install
72
-
run: pnpm install
93
+
- name: Install dependencies
94
+
run: pnpm install --frozen-lockfile
73
95
74
-
- name: Run Evals
75
-
uses: braintrustdata/eval-action@v1
96
+
- name: Run evals
97
+
uses: braintrustdata/eval-action@v2
76
98
with:
77
99
api_key: ${{ secrets.BRAINTRUST_API_KEY }}
78
100
runtime: node
101
+
package_manager: pnpm
79
102
root: my_eval_dir
80
103
```
81
104
82
-
> [!IMPORTANT] You must specify `permissions` for the action to leave comments
83
-
> on your PR. Without these permissions, you'll see Github API errors.
84
-
85
-
To see examples of fully configured templates, see the `examples` directory:
105
+
For more fully configured workflows, see the `examples` directory:
86
106
87
107
- [`node with npm`](examples/node/npm.yml)
88
108
- [`node with pnpm`](examples/node/pnpm.yml)
89
109
- [`python with pip`](examples/python/pip.yml)
90
110
- [`python with uv`](examples/python/uv.yml)
91
111
- [`go`](examples/go/go.yml)
92
112
93
-
## Go evals
94
-
95
-
The Go runtime executes `go run ${paths}`. To include Go eval results in the PR
96
-
comment, print each `ExperimentSummary` as one JSON line after calling
97
-
`result.Summarize(ctx)`:
98
-
99
-
```go
100
-
summary, err := result.Summarize(ctx)
101
-
if err != nil {
102
-
log.Fatal(err)
103
-
}
104
-
b, err := json.Marshal(summary)
105
-
if err != nil {
106
-
log.Fatal(err)
107
-
}
108
-
fmt.Println(string(b))
109
-
```
113
+
## Runtime behavior
114
+
115
+
- **Node and Python:** the action runs `braintrust eval --jsonl` from `root` and
116
+
collects the emitted experiment summaries.
117
+
- **Go:** the action runs `go run ${paths}` from `root`. To include Go eval
118
+
results in the PR comment, print each `ExperimentSummary` as one JSON line
119
+
after calling `result.Summarize(ctx)`:
110
120
111
-
## How it works
121
+
```go
122
+
summary, err := result.Summarize(ctx)
123
+
if err != nil {
124
+
log.Fatal(err)
125
+
}
126
+
b, err := json.Marshal(summary)
127
+
if err != nil {
128
+
log.Fatal(err)
129
+
}
130
+
fmt.Println(string(b))
131
+
```
112
132
113
-
For Node and Python, the action runs `braintrust eval`. For Go, the action runs
114
-
`go run`on `paths` from `root`. It collects experiment results emitted as JSONL
115
-
and posts them as a comment in the PR alongside a link to Braintrust. For
116
-
example:
133
+
The action creates or updates a single PR comment with a Braintrust link and
134
+
result table. For example:
117
135
118
-
### Example braintrust eval report
136
+
### Example Braintrust eval report
119
137
120
138
**[Say Hi Bot (HEAD-1714341466)](https://www.braintrustdata.com/app/braintrustdata.com/p/Say%20Hi%20Bot/experiments/HEAD-1714341466)**
0 commit comments