Reframe Kitaru docs around executions and replay; fix unshipped API samples - #5099
Merged
Conversation
…amples Invert the framing across introduction, user-guides, agents-guide, and SDK reference pages: replay against your real code leads, durable checkpoints are the recording mechanism, and production runs become the test bed. Restructure the agents-guide so recording and the three-runs replay discipline carry the narrative, with crash economics compressed to a secondary benefit. Also corrects the replay-and-improve code samples, which documented an API that does not exist in the shipped Kitaru SDK (run_cohort, metric callables, from_= and bare model= kwargs on replay). All samples now use the real surface: flow.replay(at=, flow_overrides=), ReplaySubmission results, kitaru.diff/diff_cohort/diff_matrix, and the executions CLI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The introduction bullet ran three sentences and leaned hard on "test bed"; replace with the canonical hook (traces you can run, not just read) plus one sentence of mechanics. Same trim in the SDK reference overview and the durable-agent intro. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
htahir1
added
documentation
Contributor
Documentation Link Check Results✅ Absolute links check passed |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
|
Kitaru v2 replaced flows, checkpoints, and executions with sessions recorded by adapters (or imported from Langfuse) and replayed with tool calls answered from the recording — and it runs its own server rather than sharing ZenML stacks. The agents-guide spine (record page, replay page, guide README), the sdk-docs Kitaru pages, and every cross-mention (introduction, user-guide, sdk-docs, component-guide, first-ai-pipeline) now describe that surface; the dead v1 API samples (flow.replay, kitaru.diff, checkpoint_strategy, executions CLI) are gone. Part 3 of the agents guide still targets the v1 harness example and carries a visible refresh note pending a decision on that example's future. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The section pitched durable execution, crash recovery, HITL, and @flow/@checkpoint — none of which is Kitaru's story anymore. It now leads with record → replay → improve and the clean split: ZenML is for ML pipelines, Kitaru is for agents. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Parts 1 and 2 are now a guided read of Kitaru's canonical returns-resolver example: import ten Langfuse-traced runs as tagged sessions, surface the outliers with built-in evaluators, record the support lead's judgment as an investigation, encode it as the returns-policy evaluator, freeze target and control cohorts, and replay a strict-policy candidate through an experiment with evaluate-baselines — every command lifted verbatim from the example so the guide stays true as the example evolves in CI. The invented investigator agent is gone; the tool-policy section explains why the example runs passthrough (mock tools) and what the history/on_miss=fail posture is for. Part 3 keeps its refresh note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A command-by-command diff against the example README caught three abbreviated blocks (evaluator register lost its description and display version, agent version register its description, investigation create its curated session view) and one missing reading caveat (replay cost is unavailable from the adapter — don't read it as zero). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The canonical example now relies on startup-registered official plugins (kitaru/langfuse, kitaru/cost, ...) instead of a seed script, and experiment create names its agent. Guide commands re-verified against the refreshed README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kitaru packaged its PydanticAI adapter separately on Aug 8: src/kitaru/ adapters/ no longer exists and there is no pydantic-ai extra, so `from kitaru.adapters.pydantic_ai import KitaruAgent` raises ImportError against current Kitaru. The import is now `from kitaru_pydantic_ai import KitaruAgent`, matching Kitaru's canonical example. Both pages showed the wrapper with no install step, which would have left a reader with an ImportError and nothing to search for, so each now names the distribution it needs. Checked the other Kitaru imports on these pages against Kitaru's source: kitaru.client.KitaruAPIClient, api_models.v1.session.SessionListParams and api_models.v1.replay.ReplayCreateRequest are all still correct, and ReplayOverride is re-exported from replay.py even though it is defined in replay_config.py — so those samples are left alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guide was built on Kitaru's v1 durable-workflow surface: 06-hitl.md is an entire stage premised on kitaru.wait(), and 02-sandbox.md teaches kitaru.log() and kitaru.save()/load() for state across runs. None of those exist in Kitaru v2 — src/kitaru/__init__.py exports nothing, and there is no definition of wait, log, save, load or llm anywhere in the package. Readers were being taught an API they cannot install. It also linked to docs.zenml.io/kitaru/guides/wait-and-resume and /kitaru/guides/artifacts, neither of which exists, which is the likeliest cause of the long-standing check-absolute-links failure on this branch. Rather than leave nine pages teaching a dead API, the guide is removed and its URLs redirect to the user guide index. The three cards that advertised it — two in introduction.md, one in the user guide README — now point at the Kitaru docs, where the current agent story lives. Stages 2-5 covered genuine ZenML-side patterns (sandboxing, skills, credential proxy, typed services) that do not depend on the removed API. If those are worth keeping they should be rewritten free of the v1 surface rather than preserved as-is; they are recoverable from history. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
check_gitbook reported "9 missing files" for redirects whose targets looked correct. The cause is in check_toc_entries: a redirect target must appear in the parsed TOC, and parse_toc_file only collects entries that sit under a "##" section header. The space README is listed above the first header in toc.md, so it is not in the 76 parsed paths and every redirect pointing at it counted as missing. Retargeted to llmops-guide/README.md, which is in the TOC and is the closest in-space home for someone who came looking for agent content. The real destination is the Kitaru docs, but this map cannot redirect across GitBook spaces. Verified by running the script's own parse_toc_file and check_toc_entries against the edited config: missing_files is empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
strickvl
approved these changes
Aug 15, 2026
htahir1
added a commit
that referenced
this pull request
Aug 17, 2026
Reframe Kitaru docs around executions and replay; fix unshipped API samples (cherry picked from commit 991c3fb)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two layers of change across the Kitaru-related docs in this repo (introduction, Learn/user-guides, agents-guide, SDK reference):
1. Framing inversion. Kitaru descriptions previously led with durable checkpoints ("records every model call and tool call as a durable checkpoint, so you can replay..."). Inverted everywhere: replay-against-your-real-code leads (canonical hook: "traces you can run, not just read"), durable checkpoints are the mechanism that makes the replay faithful. The agents-guide promotes "Replay is the part other tooling can't do" to the headline, reorders its promise list to Replay / Improve / Run, renames Part 1 to "Record", and labels Part 3 (sandbox/skills/proxy/services/HITL) as the operating-at-scale annex.
2. Correctness fix (the important one). The published replay-and-improve chapter documents an API that does not exist in the shipped Kitaru SDK:
run_cohort, metric callables (cost,latency,quality_judge),from_=and baremodel=kwargs on replay, and.exec_idon replay results. Anyone following the chapter gets code that throws. All samples now use the verified shipped surface:flow.replay(execution, at=, flow_overrides=)→ReplaySubmission.results[].replay_exec_id,kitaru.diff/diff_cohort/diff_matrix,handle.exec_id, and the realkitaru executionsCLI. The chapter's roadmap hint is also corrected — it under-claimed: per-call input/output/model/code overrides all ship; only forcing a call to raise does not.Also: 01-durable-agent restructured (recording leads, new "What the recording looks like" section, crash economics compressed to one paragraph), "cohort" replaced with filter phrasing in prose (API names untouched), and stale meta descriptions updated.
9 prose files + toc display titles; no code, no structure moves, all URLs stable.
Review
Preview of this exact branch is live on the internal "Hamza" GitBook site (all spaces synced to
kitaru-vision-docs). Key pages: user-guides/agents-guide, .../01-durable-agent, .../replay-and-improve, getting-started/introduction.🤖 Generated with Claude Code