Inqui-Lab: Improving duplicate-detection

[kartpop]

Is your feature request related to a problem?
The pilot testing of the input app revealed inadequate duplicate detection, the need for a global knowledge base, and the correlation between duplicate detection and topic relevance. This lack affects processing effectiveness and user experience.

Describe the solution you'd like

• Keep duplicate detection separate from L2 scoring, using weighted averages in post-processing.
• Improve communication about duplicate detection accuracy (40–50%) and maintain a human-in-loop.
• Utilize the model's trained knowledge for cost-effective duplicate assessment.
• Integrate duplicate detection results as input for the next LLM stage.
• Adjust prompts to guide LLM on handling duplicate results and novelty scoring.
• Collaborate with the Input app team on prompt modifications while limiting Google search use.

Original issue

Context

Input app pilot testing (~2,000 dataset) surfaced missing duplicate detection, a desire for a global knowledge base, and duplicate detection linked to topic relevance. These are mostly prompt + pipeline changes.

Approach

Option 1:

• Keep duplicate detection independent of L2 scoring; handle it in post-processing via a weighted average (a flagged idea may still be strong relative to the cohort — a decision input, not a score penalty).
• Duplicate detection is ~40–50% accurate — communicate plainly, keep human-in-loop.
• Cheaper alternative to global web search: lean on the model's trained knowledge via rubric wording ("if the idea is generally common, score lower on novelty").
  Option 2:
• Feed the duplicate-detection result as an input into the next LLM stage (not done today).
• Prompt then instructs the LLM to consider the duplicate result and assign appropriate novelty scores.
• Need the Input app team's help on how they want the prompt changes shaped.
• Keep some scope for Google search but treat cautiously (nondeterministic wide search).