docs(evaluation): ship revamped Overview and Concepts to production#737
Merged
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs(evaluation): Eval types (three types)
…s/eval-composite-evals
docs(evaluation): Composite evals & groups (concept)
Repoints the judge-model card to concepts/judge-models, where that content now lives after a84dce3 folded it in.
docs(evaluation): Overview revamp
…ation docs(evaluation): Understanding Evaluation concept
…e-agi/docs into docs/eval-custom
What a template holds (criteria, required inputs, output type, threshold, optional
reason), input mapping with {{variables}}, and the versions model (numbered
snapshots, default, pin so a re-run scores the same and a CI gate stays stable).
Rewritten for one-read clarity: no '## About', minimal frontmatter, forward cards,
no em-dashes, all links resolve.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- drop About heading and --- dividers, open at a real first H2 - bold error localization on first mention, link judge model/eval template/Observe - drop the fake configured-threshold framing; state the real pass/fail vs low-score trigger - soften code-type/non-scorable rows to what's grounded, not the private ee engine's internals - compress How to enable it to the checkbox + the real OR-of-two-flags inheritance - call it the Error Localization checkbox, not the API field name - reword Review/Where results appear to the confirmed surfaces (trace detail eval view, dataset row drawer, Playground run tests) - fix stale link to guides/running-evaluations, cut the backward Understanding-evaluation link - Next steps -> Keep exploring, as cards TH-6675
Add the tool-use mental model (response -> Agent Evaluator -> connector -> verdict) and remove the annotated screenshot, which is how-to material and belongs in the Configure connectors guide, not the concept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e-agi/docs into docs/eval-feedback
…e-agi/docs into docs/eval-error-localization
Cut the threshold-calibration section and the SDK feedback pointer, rewrite few-shot injection platform-first with a run-time steering diagram, add the feedback actions, complete the sources to all seven values, and move to the concept shape (opens at the first H2, no separators, Keep exploring cards). Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
docs(evaluation): Feedback (concept)
"Judge models" read as the LLM-as-Judge eval type (Rishav's feedback), so the concept is now Evaluator model: page moved to concepts/evaluator-models, term renamed across the evaluation tree, FAQ, nav, and redirect targets; the LLM-as-Judge type name stays. Also addresses review on the page: stale guides/ links repointed, modality line made consistent with the table, first mention bolded, backward card swapped for Guardrails, redundant How to choose section dropped, and "Bring your own LLM" phrasing everywhere. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
…ew (error localization)
docs(evaluation): Judge models concept
…e-agi/docs into docs/eval-ground-truth
… to Guides Add the two-jobs mental model (answer key + few-shot calibration), switch to evaluator-model terminology, point the reference-based eval mentions at the Built-in evals reference, state the upload formats, and convert the footer to Keep exploring cards. Rename the Evaluation nav section from How to guides to Guides, matching Observe. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
docs(evaluation): Ground truth (concept)
Add the templates-and-configs mental model (template as the image, config as a running instance, an "Eval" colloquially meaning a config), repoint judge-models to evaluator-models, fold the grounding clause in place of the dropped Context page, correct the built-in count to 156, and fix a nonexistent card icon. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
… statement Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
docs(evaluation): Eval templates & versions concept
docs(evaluation): Connectors concept
…s/eval-error-localization
…to docs/eval-error-localization
…model rename, drop false skip condition) Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
docs(evaluation): Error localization (concept)
Ships the rewritten Evaluation Overview and the ten concept pages (Understanding Evaluation, Eval types, Eval templates & versions, Evaluator models, Output types, Error localization, Guardrails, Composite evals, Feedback, Ground truth). The old features pages and their nav section stay live; the how-to guides ship separately later. Adds redirects for the two renamed concept URLs and completes the judge model -> evaluator model rename in the FAQ and cookbook. Signed-off-by: khushalsonawat <khushalsonawat@gmail.com>
atharva-bhange
approved these changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Ships the revamped Evaluation Overview and the ten concept pages to production while keeping the old Features pages and their nav section live. The how-to guides are not part of this release; they ship separately as they land, and until then every reference that pointed at a guides URL is repointed to the live features equivalent.
Type of Change
Changes
Testing