Skip to content

chore(evaluation): type strict - #1121

Open
AkhileshNegi wants to merge 2 commits into
mainfrom
fix/evaluation-v2-typecheck
Open

chore(evaluation): type strict#1121
AkhileshNegi wants to merge 2 commits into
mainfrom
fix/evaluation-v2-typecheck

Conversation

@AkhileshNegi

@AkhileshNegi AkhileshNegi commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Issue

Part of #1083

Summary

  • Before: Evaluation modules failed static type checking — raw model attrs in query builders, TypedDict payloads passed as dict[str, Any], Optional values used without narrowing.
  • Now: Evaluation path (CRUD, services, storage utils) type-checks clean, and the None paths the checker exposed are handled.

Types: col() in order_by/is_not; typed score payloads (EvaluationScore, SummaryScore, TraceData) threaded through merge/summary/processing/fast; read-only args widened to Sequence/Mapping; TraceData.categoryNotRequired; failure_responseAPIResponse[T].

Behaviour (only runtime-affecting bits): skip traces with None question_id / missing item_id; skip + warn on missing cosine_similarity instead of sending None to Langfuse; read string_value for Langfuse text scores; fail the run explicitly when config/provider is unresolvable; raise trace_download_failed when score_trace_url is missing.

Checklist

Before submitting a pull request, please ensure that you mark these task.

  • Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
  • If you've fixed a bug or added code that is tested and has test cases.

Notes

Evaluation modules only — TTS/STT from #1083 not covered here.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)
  • ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 91b51d89-9265-46d6-8c9c-16054d6e8ed2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot changed the title Evaluations: Typechecks feat(evaluation): Introduce type checks Aug 10, 2026
@AkhileshNegi AkhileshNegi changed the title feat(evaluation): Introduce type checks feat(evaluation): Added type checks Aug 10, 2026
@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown

OpenAPI changes   ⚪ No API surface changes

Note

This PR does not modify the API contract.

mainbea5f505 · generated by oasdiff

@AkhileshNegi AkhileshNegi changed the title feat(evaluation): Added type checks chore(evaluation): type strict Aug 10, 2026
@AkhileshNegi AkhileshNegi self-assigned this Aug 10, 2026
@AkhileshNegi
AkhileshNegi marked this pull request as ready for review August 10, 2026 04:40
@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.79245% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
backend/app/crud/evaluations/core.py 72.72% 3 Missing ⚠️
backend/app/crud/evaluations/langfuse.py 60.00% 2 Missing ⚠️
backend/app/crud/evaluations/processing.py 87.50% 1 Missing ⚠️
...end/app/services/evaluations/prompt_improvement.py 50.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@AkhileshNegi
AkhileshNegi requested a review from Ayush8923 August 10, 2026 05:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant