Psychological safety for conversational AI.
Your AI talks to people who may be in crisis, in distress, or emotionally dependent on it. You have evals for accuracy, latency, and toxicity — but nothing that tells you whether your model handled a suicidal disclosure the way a trained human would.
We build the instruments that measure it. The core ones are open source.
An open benchmark for the psychological safety of conversational AI. Run it locally against your own model and see where it actually stands. Free, open source, no account.
⚖️ aipsy-judge
A specialized, psychologist-corrected local judge. Apache-2.0. It runs inside your contour — your users' conversations never leave your environment. There is no API to send data to.
Keyword-free clinical vignettes for research on emotion processing in language models. 8 Plutchik emotions × 6 domains, with matched neutral controls, intensity gradients, and discriminant-validity controls — validated by a three-method NLP leakage battery (VADER, NRC, GoEmotions).
from datasets import load_dataset
ds = load_dataset("keidolabs/aipsy-affect")Most evaluation of AI emotional behavior uses keyword-contaminated stimuli — prompts containing the very emotion words being tested. That confounds detecting an emotion with naming it, and produces inflated, uninterpretable numbers.
We borrow from clinical psychology: stimuli must evoke the construct without naming it. Controls must match surface structure while removing emotional content. Alternative explanations get ruled out before anything is called a finding.
The methodology is as much the contribution as the findings.
| Paper | Finding |
|---|---|
| Empathy Is Not What Changed (Keep4o) | When OpenAI updated GPT-4o, the shift wasn't in empathy — it was in psychological safety posture. Invisible to standard NLP metrics. Visible with clinical evaluation. |
| Whether, Not Which | Affect reception is real and dissociable from emotion categorization. AUROC 1.000 across 6 models. Clinical methodology removed the keyword confounds that contaminated prior work. |
| AIPsy-Affect | The dataset and the keyword-free stimulus methodology. |
aipsy-bench and aipsy-judge preprints: submitted, IDs pending.
A preregistered human-validation study of the judge against expert raters is in progress (OSF, embargoed). Until it lands, our posture is explicit: recommendation, not rubber-stamp.
Keido Labs — an AI Psychology R&D lab. We apply clinical-psychology methodology to AI alignment and red-teaming.
🔬 Monitoring platform: EmpathyC — we monitor the machine, not the person.
