Problem statement
Workspai lets developers and AI agents search a Workspace Knowledge Graph without loading the complete graph artifact.
The search is deterministic, local, and bounded, but we do not yet have a realistic regression corpus proving that important engineering terms consistently rank above conversational filler.
For example, a query such as “Where is user authentication handled?” should prioritize authentication services, endpoints, and supporting proofs—not unrelated entities that happen to contain common words such as “is”, “where”, or “handled”.
Without a stable relevance fixture, ranking regressions can look valid while returning misleading results to users and AI agents.
Proposed solution
Add a compact polyglot workspace fixture and a deterministic relevance test suite for natural-language graph queries.
The fixture should include:
- A frontend and at least two backend services
- An authentication API or interface contract
- A shared package or dependency
- One CI workflow
- One infrastructure artifact
- Proof-carrying relationships between these entities
Add at least five realistic queries, such as:
- Where is user authentication handled?
- Which service owns the checkout API?
- What depends on the shared database package?
- Where is the release workflow defined?
- Which project exposes the health endpoint?
For every query:
- Define the expected entity or entities
- Require the primary relevant entity to appear within the top three results
- Ensure duplicate logical entities do not consume the bounded result budget
- Verify that repeated runs return identical ordering and scores
Suggested starting points:
packages/cli/src/workspace-knowledge-graph-query.tspackages/cli/src/__tests__/workspace-intelligence-benchmark.test.ts- Existing Workspace Knowledge Graph query tests
Acceptance criteria:
Alternatives considered
Embedding-based or remote semantic search could improve recall, but it would introduce network access, model variability, additional cost, and non-deterministic results.
This issue intentionally focuses on a deterministic regression corpus for the existing local retrieval engine. Embeddings and remote search are outside its scope.
Expected impact
This gives contributors and maintainers an objective way to improve graph retrieval without guessing.
It also helps ensure that developers, IDE integrations, and AI agents receive small, relevant, and repeatable evidence sets instead of confident but unrelated search results.
Problem statement
Workspai lets developers and AI agents search a Workspace Knowledge Graph without loading the complete graph artifact.
The search is deterministic, local, and bounded, but we do not yet have a realistic regression corpus proving that important engineering terms consistently rank above conversational filler.
For example, a query such as “Where is user authentication handled?” should prioritize authentication services, endpoints, and supporting proofs—not unrelated entities that happen to contain common words such as “is”, “where”, or “handled”.
Without a stable relevance fixture, ranking regressions can look valid while returning misleading results to users and AI agents.
Proposed solution
Add a compact polyglot workspace fixture and a deterministic relevance test suite for natural-language graph queries.
The fixture should include:
Add at least five realistic queries, such as:
For every query:
Suggested starting points:
packages/cli/src/workspace-knowledge-graph-query.tspackages/cli/src/__tests__/workspace-intelligence-benchmark.test.tsAcceptance criteria:
Alternatives considered
Embedding-based or remote semantic search could improve recall, but it would introduce network access, model variability, additional cost, and non-deterministic results.
This issue intentionally focuses on a deterministic regression corpus for the existing local retrieval engine. Embeddings and remote search are outside its scope.
Expected impact
This gives contributors and maintainers an objective way to improve graph retrieval without guessing.
It also helps ensure that developers, IDE integrations, and AI agents receive small, relevant, and repeatable evidence sets instead of confident but unrelated search results.