Skip to content

README.md

Latest commit

 

History

History
47 lines (43 loc) · 2.67 KB

README.md

File metadata and controls

47 lines (43 loc) · 2.67 KB
Raw

Evaluation Result

Notes

  • QnAs should not be used for model training or tuning. This is created only for evaluation purpose.
  • QnAs were generated from OCP docs by LLMs. It is possible that some of the questions/answers are not entirely correct. We are constantly trying to verify both Questions & Answers manually. If you find any QnA pair to be modified or removed, please create a PR.
  • Instruction to run eval script

Result

  • Last Execution date: pending re-run with updated models
  • Evaluated Provider/Models:
    • (watsonx) ibm/granite-3-2-8b-instruct (API Version: TBD)
    • (azure) gpt-5.4-mini (Model Version: TBD, API Version: TBD)
  • Judge provider/model (LLM based eval): (openai) gpt-5.4-mini
  • QnA evaluation dataset: QnAs from OCP doc
  • API run mode: without tool calling
  • RAG SHA: 338e5ec469754d1ac190852ab0d997a99892acbe262bf75971ed91234e3d184a
    • OCP doc: 4.19
    • Embedding model: all-mpnet-base-v2
    • Splitter: Normal (chunk: 380)
    • Vector DB: faiss
    • Retrieval: Top 5 (cosine similarilty)

Scores

Please look at below box-plots to have an idea about overall score for QnAs. Ideal result: Box should be as condense as possible and near to right side (close to 1).

Correctness/Similarity score generated by Model

Here we ask a judge model to score/grade response generated by OLS against the pre-defined/ground-truth answer. Similarity Score by LLM

Cosine Similarity

Cosine similarity between vector representation (generated by embedding model) of ground-truth answer & OLS response. Cosine Similarity

Answer Relevancy

Here we generate multiple questions from the response using a Model and calculate average cosine similarity score between the original query & the generated questions. Answer Relevancy score

Rouge-L

These scores are based on stem words. Here the longest common subsequence (LCS) is being considered (not necessarily in order) between the predefined answer and the OLS response.

Rouge-L - Precision

Ratio of the length of the LCS over the unigrams in the response. Rouge-L Precison score

Rouge-L - Recall

Ratio of the length of the LCS over the unigrams in the pre-defined answer. Rouge-L Recall score

Rouge-L - F1

Harmonic mean of precision & recall. Rouge-L F1 score

Score Summary Json file contains different quantile values for each of the score.