Qresp 2.0 modernization, auth, ownership, drafts, and admin management - #68
Open
hongsimi7 wants to merge 235 commits into
Open
Qresp 2.0 modernization, auth, ownership, drafts, and admin management#68hongsimi7 wants to merge 235 commits into
hongsimi7 wants to merge 235 commits into
Conversation
…ent/develop New Release, v2.0.4
…ent/develop New Release, v2.0.5
DEPENDENCY_AUDIT.md records latest-stable versions (PyPI/npm, 2026-07-02), risk, and breaking changes for every dependency before any edits. requirements.txt / setup.py: remove every declared-but-never-imported package (Flask-API, Flask-HTTPAuth, flask-profiler, Flask-WTF, paramiko, schedule, py3dns, pyasn1, validate-email, pyOpenSSL, swagger-spec-validator, coveralls, python-dateutil, expiringdict, cffi, cryptography) and redundant explicit transitives (itsdangerous, Jinja2, urllib3). Add `requests` (used by project/util.py, was transitive-only). setup.py: python_requires >=3.10 (jsonschema/coverage/gunicorn already require it), test tools moved to a `test` extra. Flask<2.3 / connexion<3 caps unchanged in this phase. Verified (fresh CPython 3.11.5 venv): pip check OK, GET / -> 200, nose2 17 OK. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e B) flask-mongoengine 1.0 is unmaintained and hard-blocks Flask>=2.3 (it uses the removed flask.json.JSONEncoder APIs). The models are plain mongoengine Documents; the extension was only a connection shim, so: - project/__init__.py: mongoengine.connect() straight from Config settings; username/password only passed when configured. Drops the write-only app.config['MONGODB_*'] mirror (never read anywhere). - project/db.py: MongoDBConnection re-points via mongoengine.disconnect() + connect() (default alias cannot be silently reused with new settings); catches both pymongo and mongoengine ConnectionFailure. - requirements.txt / setup.py: drop flask-mongoengine (also drops its Flask-WTF / WTForms[email] / email-validator transitive chain; views.py uses EmailField for rendering only, no Email() validator). Verified (fresh CPython 3.11.5 venv): pip check OK, flask-mongoengine absent from the tree, GET / -> 200, nose2 17 OK. Real-MongoDB round-trip re-verified in the Docker phase. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Connexion 3 keeps FlaskApp but runs routing/validation/swagger-ui as ASGI
middleware around Flask, so the servable object changes:
- __init__.py: FlaskApp(jsonifier=Jsonifier(cls=MongoJSONEncoder)); swagger.yml
resolved relative to the package (cwd-independent); app.json provider set.
- NEW project/jsonutil.py: restores mongoengine-document JSON serialization
that flask-mongoengine's patched encoder used to provide, in the identical
bson json_util shape, for BOTH layers (Connexion jsonifier for /api/*,
Flask JSON provider for routes.py jsonify()).
- api.py: `from connexion import request, jsonifier` (gone in v3) -> flask
request proxy; jsonifier import was unused.
- run.py / __main__.py: serve via connexionapp.run() (uvicorn) so dev serving
includes the middleware; __main__ gains the main() the qresp console script
always pointed at (previously broken entry point).
- compose: gunicorn now uses -k uvicorn_worker.UvicornWorker on
project:connexionapp; dev compose serves uvicorn --reload (flask run would
bypass middleware). deps: connexion[flask,swagger-ui,uvicorn]>=3.3 +
uvicorn-worker; caps dropped.
- NEW tests/test_api_endpoints.py: 9 tests through the real ASGI middleware
(validation 400, Swagger-2 body-name mapping into `req`, EmbeddedDocument
serialization on /api/paper/{id}, Flask passthrough, /api/ui/).
Flask stays 2.2.5 in this phase (cap lifted next, Phase C).
Verified (fresh CPython 3.11.5 venv): pip check OK, GET / -> 200,
nose2 26 tests OK (was 17).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With Connexion 3 in and flask-mongoengine gone, the last <2.3 caps are
removed: Flask 2.2.5 -> 3.1.3, Werkzeug 2.2.3 -> 3.1.8. App code needed no
changes for removed Flask APIs (no before_first_request / JSONEncoder /
flask.escape usage); flask-sitemap 0.4.0, Flask-Session 0.8.0, flask-cors
6.0.5 all verified working on Flask 3 (boot + /sitemap.xml render).
Two latent bugs exposed by the new page-render tests (both pre-existing on
the current WTForms 3.2.2 baseline, not caused by Flask 3):
- views.py RequiredIf: WTForms-2 tuple field_flags crashed every form that
binds it ("'tuple' object has no attribute 'items'" on /qrespcurator);
now a dict, and the broken super(RequiredIf).__init__() call fixed.
- util.py Servers: the federated-servers registry was fetched with no
timeout and no error handling, so an outage or non-JSON reply 500'd
/qrespcurator + /qrespexplorer (reproduced live against
paperstack.uchicago.edu today); now degrades to an empty list.
tests: +3 page-render tests (curator page mocks the registry fetch to stay
hermetic). Verified (fresh CPython 3.11.5 venv): pip check OK, GET / -> 200,
nose2 29 tests OK.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…se F) - requirements.lock.txt: regenerated from the verified clean venv. 81 -> 65 pins: Flask 3.1.3, Werkzeug 3.1.8, connexion 3.3.0 (+starlette/httpx/ a2wsgi/uvicorn/uvicorn-worker), flask-mongoengine chain and all unused packages gone. Header documents Windows provenance (uvloop absent -> Docker uvicorn falls back to asyncio; harmless). - backend/Dockerfile: python:3.11-slim -> python:3.14-slim. Verified in-container: lock installs, gunicorn -k UvicornWorker serves /, /api/search, /api/ui/ (200); DAO insert->read against mongo:4.4; nose2 29 OK. - backend/Dockerfile.dev: python:3.6-alpine (EOL + apk toolchain) -> python:3.14-slim, mirroring the prod image. Verified: build + uvicorn --reload serves / and /api/search (200) against the dev db. - docker-compose.dev.yml: dev db mongo:3.6.18-xenial -> mongo:4.4. 3.6 is EOL and PyMongo 4.17 requires MongoDB >= 4.0, so the old dev db could no longer connect at all. Production mongodb default stays mongo:4.4 (unchanged). - CI backend-smoke: python matrix 3.11 (dev baseline) + 3.14 (Docker runtime); all YAML validated locally. - tests: assertEquals -> assertEqual (unittest aliases removed in Python 3.12; was 16 in-container errors on 3.14). - .coverage untracked + gitignored (binary artifact rewritten by every run). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FULL_STACK_MODERNIZATION_REPORT.md rewritten for wave 2: dependency table (before/after), changed-files list, code-migration summary, full verification matrix (local venvs, lock reproducibility, Docker prod/dev on py3.14, real- MongoDB round trip, validation-middleware checks), latent bugs fixed, deployment risks, and exact next steps. QUICKSTART/TROUBLESHOOTING: serving commands updated for ASGI (uvicorn / gunicorn -k UvicornWorker), test counts 17 -> 29, stale Docker/Mongo status rows corrected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…st 30 Toolchain: Node 24.18.0 / Yarn 1.22.22 (Yarn 1 kept -- minimal churn). Verified locally: yarn install OK, yarn build OK (Next 16.2.10, Turbopack, 7 routes), yarn test OK (RTL, 2 suites / 5 tests). Dependency majors (see DEPENDENCY_AUDIT.md for the audit): - next 9.4.4 -> 16.2.10; react/react-dom 16.13.1 -> 19.2.7 - @material-ui core v5-alpha + icons/lab v4 mix -> @mui/material 9.1.2 + @mui/icons-material 9.1.1 (+ @mui/material-nextjs, emotion incl. @emotion/server); @mui/lab dropped (Alert/Autocomplete/Pagination in core) - react-hook-form 6 -> 7.80; @hookform/resolvers 0.1 -> 5.4; yup 0.29 -> 1.7 - jest 26+enzyme -> jest 30 + next/jest + React Testing Library - axios 0.19 -> 1.18; ajv 6 -> 8 (strict:false, draft-07 schema unchanged) - simple-react-lightbox (dead, React 16-only) -> yet-another-react-lightbox - vis-network 7 -> 10 (standalone import; hammerjs/keycharm/component-emitter peer shims dropped); fontawesome 5 -> 7; react-checkbox-tree 1.6 -> 2.0 Code migrations: - JSS -> emotion: 12 makeStyles/withStyles files converted to styled()/sx; _app/_document rebuilt on @mui/material-nextjs v16-pagesRouter (replaces ServerStyleSheets + jss-server-side removal); createTheme. - MUI v4 API sweep: justify->justifyContent (20), Hidden -> responsive sx (4 files), Accordion TransitionProps -> slotProps.transition, Popover PaperProps/classes -> slotProps/sx, visuallyHidden util for sort labels. - RHF v7: register-as-ref removed -- TextInput/NameInput/RadioInput wrappers register internally (callers pass register=); Controller as= -> render; errors moved to formState.errors (13 forms); unregister(object) -> (name); bracket field names (PIs[0].x) -> dot syntax; touched -> touchedFields. - yup 1: when() then/otherwise now function form (ToolsInfoForm). - next/link: no child <a>; MUI Buttons render component={Link}; href object replaces as=; styled-jsx anchors rescoped via :global(). - React 19: CSSTransition needs nodeRef (findDOMNode removed) -- FadeTableRow wrapper keeps the row fade; Turbopack: imported-binding reassign fixed (explorer.js); @mui/icons-material 9 dropped *Outline aliases -> *Outlined equivalents. - Enzyme specs rewritten with RTL/user-event; babel-jest/.babelrc removed (SWC via next/jest), custom cssTransform deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Next 16 requires Node >= 20.9; Node 24 matches the verified local toolchain (local yarn build + test green before this change, per the required order). pm2 no longer needs the Node-14 pidusage pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Frontend section rewritten from "blocked on Node 14" to DONE: Next 16.2.10 / React 19.2.7 / @mui 9 / RHF 7 / jest 30+RTL on Node 24, build+tests verified; deployment risks now include the gui image rebuild and a staging click-through list (forms, lightbox, workflow graph, visual parity); next steps updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five goals (modernization done; UI regression repair; Google identity-only auth; owner/admin edit+deactivate; agentic literature explorer) with MVP/defer splits, ordering, file map, risk list, and branch plan. No code changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MUI v9 removed two APIs the whole layout was built on, and both fail SILENTLY (unstyled div / ignored props), which broke the staging UI vs https://paperstack.uchicago.edu: - Box system props (display/flexDirection/flexGrow/alignItems/m/p/ fontWeight/...) emit zero CSS in v9 (class `css-0`) -> header/nav layout, home hero spacing, action-button rows, footer logo rows and all bold-text Box wrappers rendered as plain block divs. - The legacy Grid API is gone: `item` leaks to the DOM and xs/sm/md sizes are ignored -> every grid cell (forms, tables, footer rows, curator layout) lost its sizing. Fix (mechanical, no redesign, values preserved verbatim): - 47 Box tags: system props -> sx={{ ... }} (dynamic expressions kept). - 156 Grid tags across 27 files: item dropped, xs/sm/md/lg -> size={n} / size={{ xs: .., sm: .. }} (bare xs -> size="grow"). Verified: yarn build OK (7 routes), yarn test OK (5 tests), zero legacy props left (grep), and a jsdom computed-style probe confirms Layout root is flex-column again, footer rows are flex/space-evenly/32px padding, home action row is centered with m=1 wrappers, and Grid emits size classes (MuiGrid-grid-xs-12 etc). Browser-level comparison against the working site remains for staging QA. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backend-only development auth endpoints (Qresp 2.0 checklist goal 3,
identity skeleton — no ownership, no permissions, no Google yet):
- GET /api/auth/me -> {authenticated, user} from the Flask session
- POST /api/auth/logout -> clears only session["auth_user"]
- POST /api/auth/dev-login -> dev/staging-only login; body {email required,
name defaults to email, is_admin defaults false}; email trimmed+lowercased
Session shape (no secrets): session["auth_user"] =
{email, name, is_admin, provider: "dev"}.
Safety gate: dev-login is OFF by default and checked per request —
enabled only when QRESP_ENABLE_DEV_LOGIN (env, via the existing
Config.get_setting override; optional [AUTH] config.ini fallback) is
1/true/yes/on; otherwise the endpoint returns 404 JSON. Production runs
without the variable. (app.config['env'] is hardcoded 'DEV' even in prod,
so the gate deliberately does NOT trust it.)
Routes are spec-first in swagger.yml (Connexion validates the body;
missing email -> 400). New project/auth.py module keeps /me and /logout
provider-agnostic for the upcoming Google OAuth swap-in.
tests: +6 in tests/test_auth.py through the real ASGI middleware with
cookie round-trip (anonymous /me; login->me->logout->me; name default;
admin flag; email validation 400s; disabled-by-default 404). No MongoDB
needed. Frontend untouched.
Verified: boot GET / -> 200; nose2 35 tests OK (was 29).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ownership MVP on top of the session skeleton (no update API yet, no
frontend, no Google):
- models.py: Paper.owner_email (StringField) — the VERIFIED session email
of the publisher, distinct from the curator-declared
info.insertedBy.emailId. Absent on all legacy records => "ownerless".
- auth.py helpers: get_current_user(), is_admin() (QRESP_ADMIN_EMAILS
env/config allowlist, plus the session is_admin claim from dev-login),
can_edit_paper(paper, user) -> (allowed, reason), stamp_owner(payload).
Rules: anonymous no; admin yes; owner yes; ownerless -> admin only;
everyone else no.
- api.py publish handler stamps owner_email from the session BEFORE the
payload is validated/stored, so ownership survives the email-verification
round trip into insertIntoPapers (schema has no additionalProperties cap;
Paper(**data) accepts the new field). Anonymous publishing stays allowed
and simply yields an ownerless record — no existing publish flow or test
breaks.
- NEW GET /api/paper/{id}/permissions (spec-first in swagger.yml; singular
/paper/{id}/... to match the existing routes) returning {can_edit,
reason, owner_email, authenticated, is_admin} so the frontend can
show/hide edit controls later; the same can_edit_paper rule will guard
the future update/deactivate endpoints. Unknown id -> 404.
tests: +9 (tests/test_permissions.py, mongomock + real ASGI middleware):
anonymous/owner/non-owner/admin on owned records, ownerless admin-only,
session admin flag, 404, can_edit_paper unit matrix, stamp_owner with and
without a session.
Verified: boot GET / -> 200; nose2 44 tests OK (was 35). Frontend untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Connects the frontend to the existing backend auth/permission MVP (commits 1678e07 + edc9279). No Google OAuth, no update API, no redesign. - NEW Context/Auth (AuthState/authReducer/authContext + types), following the existing Alert/Loading context pattern. State: {loading, authenticated, user{email,name,is_admin,provider}, error}; helpers refresh()/devLogin()/logout(). GET /api/auth/me on app load. All calls use relative same-origin /api paths so the browser carries the Flask session cookie itself; nothing is stored in localStorage. - NEW components/AuthControls.js in the header links row (desktop row and mobile drawer share the fragment): authenticated -> name/email (+"(admin)") and Sign out; anonymous -> "Dev sign in" opening a small dialog (email, optional name, dev-only admin checkbox) that POSTs /api/auth/dev-login and shows "Development login is unavailable on this server." when the backend gate returns 404. Explicitly labelled staging/development-only; no Google branding or scopes. - NEW components/Paper/PermissionNotice.js on paperdetails (skipped for previews): fetches GET /api/paper/{id}/permissions (backend decision, never frontend logic; refetches when auth state changes) and renders a small notice - "You can edit this record (owner|admin)" / "Sign in to edit this record" / "Only the record owner or an admin can edit this record". Fetch failure (previews, older backend) renders nothing. No edit button yet - there is no update API to point it at. - Publish/session behavior: NO change needed - publish/preview already go through getServer() = window.location origin, i.e. same-origin absolute URLs, so session cookies are attached automatically. tests: +8 (AuthControls anonymous/authenticated/logout/disabled-gate; PermissionNotice owner/anonymous/non-owner/silent-failure) with axios mocked. Verified: yarn build OK (7 routes), yarn test 13 OK (was 5), backend nose2 44 OK (untouched). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Smallest vertical slice proving owner/admin editing end to end (no Google,
no curator redesign, no delete):
Backend — PUT /api/paper/{id} (same singular path as the existing GET,
spec-first in swagger.yml):
- 404 unknown id; can_edit_paper gate -> 401 anonymous / 403 authenticated
non-owner (ownerless records stay admin-only).
- Top-level payload fields are merged into the stored document
(existing.to_mongo + payload), filtered to defined Paper fields, then
re-validated through the Paper model constructor and saved under the
same pk -> embedded docs coerce, required fields stay enforced (400 on
violation).
- Server-owned fields can never come from the payload: id/_id,
owner_email (forced back from the stored record), version/versions.
Frontend — PermissionNotice grows the minimal edit flow:
- "Edit metadata" button only when the BACKEND permission decision says
can_edit (no frontend-only logic).
- MVP dialog edits one harmless field (tags, comma separated) and PUTs
/api/paper/{id}; success reloads the page so getServerSideProps
refetches; 401/403 show the backend reason; other failures show a
generic error. Full curator-integrated editing is a later phase.
tests: backend +8 (test_update_paper.py on the shared permission fixture:
anonymous 401, non-owner 403, owner/admin persist, ownerless admin-only,
owner_email immutable, 404, invalid payload 400); frontend PermissionNotice
suite extended to 6 (edit action shown/hidden by backend decision, PUT
payload + reload, forbidden save shows backend reason, silent failure).
Verified: nose2 52 OK (was 44); yarn build OK; yarn test 15 OK (was 13).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Google becomes the real identity provider on top of the existing session
and permission system. Identity ONLY: scopes are hardcoded to
openid/email/profile in project/auth.py (never read from config), so this
flow can never request Drive/Gmail/other Google APIs.
Backend (spec-first, requests_oauthlib which was already a dependency):
- GET /api/auth/google: builds the consent URL, stores the OAuth state in
the session, 302 to Google. 503 JSON when unconfigured -- the app still
boots and dev-login keeps working.
- GET /api/auth/google/callback: rejects provider errors, missing flows
and state mismatches (400); exchanges the code and fetches userinfo
server-side; tokens are used for that single call and discarded (never
stored in the session or exposed to the frontend). Stores the existing
session shape: {email (trimmed+lowercased), name (defaults to email),
is_admin, provider: "google", google_sub}. is_admin comes EXCLUSIVELY
from the QRESP_ADMIN_EMAILS / [AUTH] ADMIN_EMAILS allowlist -- Google is
never trusted for roles. 302 back to "/" -> AuthState refetches /me.
- Config via existing conventions: QRESP_GOOGLE_CLIENT_ID /
QRESP_GOOGLE_CLIENT_SECRET / QRESP_GOOGLE_REDIRECT_URI env overrides,
[GOOGLE_API] config.ini fallback; the section's existing
auth_uri/token_uri/user_info endpoint entries are honored with Google
defaults otherwise. No secrets committed.
- /auth/me, /paper/{id}/permissions and PUT /paper/{id} work unchanged on
a Google session (same auth_user shape).
Frontend: anonymous header now shows "Sign in with Google" (plain text,
no Google branding) navigating to /api/auth/google; "Dev sign in" stays
as the staging tool. No redesign.
tests: backend +7 (mocked OAuth2Session, no network: unconfigured 503 +
dev-login unaffected; redirect carries identity-only scopes + state;
state mismatch and cold-callback 400; user stored with provider google
and normalized email; allowlist -> is_admin; provider error 400).
frontend +1 (Google link href for anonymous users).
Verified: nose2 59 OK (was 52); yarn build OK; yarn test 16 OK (was 15).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hardening/readiness only — no new product features.
1) OAUTHLIB_INSECURE_TRANSPORT is no longer hardcoded on (it silently
weakened production OAuth). It is now set ONLY when explicitly enabled
via QRESP_OAUTHLIB_INSECURE_TRANSPORT (env, or [AUTH] ini) for local/
plain-HTTP development. HTTPS staging/production need nothing; local
tests never exchange real tokens, so they pass without it.
2) Minimal CSRF protection (double-submit header, session-bound):
- /api/auth/me now also issues csrf_token (secrets.token_urlsafe,
stored in the session, constant-time compared).
- X-CSRF-Token is REQUIRED on mutating routes when the request carries
an authenticated session: POST /api/auth/logout, POST /api/publish,
PUT /api/paper/{id}. Anonymous API/CLI usage (incl. anonymous
publish) is unchanged; dev-login stays unwrapped by design (it only
establishes a session; Google login is protected by OAuth state).
- Frontend: AuthState captures the token from /me and an axios request
interceptor attaches it to MUTATING SAME-ORIGIN requests only (covers
the publish/preview calls built on getServer(); never leaks the
header to external hosts like the DOI scraper).
3) Google return path: /api/auth/google?next=<path> remembers a validated
same-origin, path-only target (no scheme/host, no //, no backslash) in
the session; the callback re-validates and redirects there instead of
"/". The header button now passes the current page (router.asPath).
Open redirects are covered by tests.
4) Staging readiness docs: NEW STAGING_QA_CHECKLIST.md (env vars, curl
smoke, browser matrix incl. mobile drawer; qresp_staging only, no
secrets committed); QRESP_2_IMPLEMENTATION_CHECKLIST.md gains a
pre-production blockers section (done: transport gate, CSRF, open
redirect; open: id_token verification, cookie flags via nginx, rate
limiting, unset dev-login in prod, util.py verify=False).
tests: backend +4 / updated for CSRF (logout/PUT require the token; next
round-trip; unsafe-next fallback) -> nose2 63 OK (was 59). frontend:
AuthControls spec covers the next-carrying Google href with next/router
mocked -> yarn test 16 OK; yarn build OK.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/curator crashed client-side with "TypeError: findDOMNode is not a function" (performExit -> updateStatus -> componentDidUpdate). Root cause: components/switchFade.js used react-transition-group's <Transition> WITHOUT nodeRef, so it fell back to ReactDOM.findDOMNode -- removed in React 19 -- the moment SwitchTransition (mode="out-in") exited a side. All six CuratorElements (PaperInfo/Reference/License/FileServer/ Documentation/Curator) render through SwitchFade, taking the whole curator page down. Fix: FadeTransition now owns a nodeRef attached to the div it renders and passes it to <Transition> -- the same pattern as the FadeTableRow fix in Table/Table.js from the modernization wave. Visuals/timings unchanged. (The wave-3 audit grepped for CSSTransition|TransitionGroup and missed this lone `Transition` import; a fresh repo-wide react-transition-group audit confirms switchFade.js and Table.js are the only two usages, both now nodeRef-correct.) tests: NEW SwitchFade.spec.js -- toggling form<->display, which throws under React 19 without the nodeRef, now passes (+2, yarn test 18 OK). yarn build OK. Backend untouched. Not addressed here (observed on staging, reported separately): the MUI Dialog aria-hidden focus warning (cosmetic), and the ERR_TLS_CERT_ALTNAME_INVALID from the qresp.hybrid3.duke.edu federated node (its cert is for materials.hybrid3.duke.edu; search.js already try/catches per server+endpoint, so /search degrades instead of dying). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
paperdetails showed "Error Getting Paper Data" on staging: its
getServerSideProps fetched `${query.server}/api/paper/{id}` with
query.server = https://localhost:8443 — but SSR runs INSIDE the gui
container, where localhost is the container itself, not the host tunnel
/nginx, so the fetch could never reach the backend.
Fix: NEW Utils/serverSideApi.js resolveServerSideApiBase(ctx, server) —
SSR-only decision of the fetch base:
- external federation nodes (host differs from the request host and is
not local): used as-is, unchanged behavior;
- missing server, localhost/127.0.0.1/::1/*.localhost, or same host as
the incoming request (honoring x-forwarded-host): rewritten to
QRESP_INTERNAL_API_URL (e.g. http://backend:5000); without the env var
it falls back to the original behavior;
- unparseable or non-http(s) input (ftp:, javascript:, //host, ...) is
never used as a fetch target (internal or null -> the page's existing
error path).
The env var is deliberately not NEXT_PUBLIC_* and the public query.server
passed to components (file-server links, chart paths) is untouched;
preview fetches go through the same resolver.
docker-compose.yml: gui gains QRESP_INTERNAL_API_URL=http://backend:5000
and joins the backend network so SSR can reach the backend service.
tests: +9 (resolver matrix: localhost/same-origin/x-forwarded-host ->
internal; external unchanged; missing -> internal or null; malicious
schemes neutralized; env-unset fallbacks) -> yarn test 27 OK (was 18);
yarn build OK; compose YAML validated. Backend untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
paperdetails SSR returned Next 500 ("TypeError: D is not iterable") even
after the API base fix: React 19 no longer applies .defaultProps on
FUNCTION components, so ChartInfo/DatasetInfo/ToolsInfo/ScriptsInfo
received editColumn = undefined from the page (which never passes it) and
crashed on `...editColumn` when building their table columns during SSR.
Fix: every function-component .defaultProps in the frontend (16 sites,
none remain) moved into destructuring default parameters — behavior and
UI unchanged:
- crash class (iterable spread on paperdetails): Charts, Datasets, Tools,
Scripts (editColumn = [], inDrawer = true, showSlider/showWorkflows);
- same failure class nearby: Workflow/Graph (manipulate = {}),
Workflow/Legend, pages/paperdetails ({preview = false});
- shared/benign but same regression: drawer (defaultOpen), labelvalue
(SimpleLabelValue/LabelValue), Form InputFields/NameInput/RadioInput/
SelectInput/TextInput (type moved out of ...rest and passed explicitly)
and Form/Util SubmitAndReset.
tests: NEW PaperInfoDefaults.spec.js renders ChartInfo/DatasetInfo/
ToolsInfo/ScriptsInfo with ONLY their required props (exactly how
paperdetails renders them) — these throw "not iterable" before this fix
and pass after (+4, yarn test 31 OK; yet-another-react-lightbox is
ESM-only so the suite mocks it, and the router mock provides events for
LoadingState). yarn build OK. Backend/Docker/auth untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owners/admins can now edit a whole record through the EXISTING curator
forms: paperdetails -> "Edit in Curator" -> /curator?edit=<id>&server=...
-> forms pre-populated from the stored document -> Save Changes -> PUT
/api/paper/{id} -> back to paperdetails. Create/publish mode is untouched
(TopActions/Publish render exactly as before when ?edit is absent).
Backend:
- NEW GET /api/paper/{id}/raw (spec-first): the edit flow's data source.
Gated by the same can_edit_paper rule as updates (401 anonymous / 403
non-owner / 404 unknown; ownerless -> admin only), returns the stored
document with _id/owner_email stripped. The public display-shaped read
stays GET /api/paper/{id}.
- PUT /api/paper/{id} unchanged (already merge + Paper._fields allowlist +
model re-validation + id/_id/owner_email/version/versions blocked);
now covered by full curator-payload tests.
Frontend:
- Utils/model.js: convertReqSchematoState (previously dead code) fixed and
hardened -- referenceUtil.set takes an object (was positional -> would
have produced "undefined ..." publication), journal.fullName unwrapped,
Person objects trimmed to the name triple before namesUtil.set, legacy
records with missing sections tolerated. NEW convertStateToUpdatePayload:
publish payload + fields the curator does not manage preserved from the
original document (info.downloadPath/gitPath/isPublic/..., original
schema URL); identity/server-owned fields stripped client-side and
enforced server-side regardless.
- NEW CuratorElements/EditMode.js: EditModeController (backend permission
gate -> loads /raw -> setAll into existing curator state; unauthorized
users get a clear message and NO forms/save controls) + SaveChangesBar
(reuses Publish's validate(), PUTs with the session CSRF token via the
existing axios interceptor, Cancel/Save, returns to paperdetails).
- pages/curator.js: EditModeController wraps the existing element tree;
edit mode hides TopActions and swaps Publish for Save Changes.
- PermissionNotice: the MVP tag-edit dialog is REPLACED by the single
"Edit in Curator" entry point (no two competing edit paths).
- Publish.js: validate() exported for reuse, and its ajv step no longer
crashes -- ajv 8 THROWS compiling schema_v1.2.json (duplicate
draft-04-style `id` anchors make "#/properties/collections/items"
ambiguous), which had silently broken the publish button since the ajv
upgrade; compile failures now log and fall through (backend re-validates
every payload anyway).
tests: backend +10 (test_edit_flow.py: /raw permission matrix + stored
shape; full curator-shaped PUT persists reference/tags/charts/datasets,
admin allowed, non-owner 403, owner_email immutable, invalid payload 400)
-> nose2 73 OK. frontend +13 (model round-trip on the real fixture,
EditModeController create/unauthorized/anonymous/load/save/forbidden,
PermissionNotice link) -> yarn test 44 OK; yarn build OK.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two staging blockers in /curator?edit=<id>:
1) "collections must be a `string` type, but the final value was:
[\"MICCOM\"]" — curator state keeps collections/tags as ARRAYS and
PaperInfoForm edits them as comma-separated strings, but its
defaultValues joined only tags; collections passed the raw array into
the yup string field. One missing join — also a latent CREATE-mode bug
(re-editing an already-saved Paper Information section failed the same
way); edit mode just surfaced it immediately. Both fields now join
defensively; onSubmit still splits back to arrays, so the backend shape
is unchanged and create mode behaves exactly as before.
2) "CSRF token missing or invalid" on Save Changes — the interceptor
attached the token only when the module cache happened to be populated
by AuthState's initial /me; any path where that fetch failed or the
backend session store was replaced (e.g. staging rebuild between login
and save) left the cache empty/stale with no recovery. The CSRF wiring
is now self-healing, still same-origin-only, CSRF never disabled:
- mutating same-origin requests fetch the token JUST IN TIME from
/api/auth/me when nothing is cached (async request interceptor; /me
is a GET, so no recursion);
- a 403 "CSRF" response drops the cache so the user's retry refetches
a fresh token for the new session.
tests: NEW CsrfIntegration.spec.js runs the REAL axios interceptor
pipeline against a stub adapter — Save Changes carries X-CSRF-Token from
the cached /me, from the just-in-time fetch when nothing cached it, and
re-fetches after a stale-token 403 (+3). NEW PaperInfoForm.spec.js loads
array-backed state (collections: ["MICCOM"]) — renders "MICCOM, PARADIM"
in the field and saves back clean arrays (+2). Backend proof that a
missing token is rejected and a valid one succeeds already exists
(test_update_paper.test_update_without_csrf_token_denied + owner tests).
Verified: yarn test 49 OK (was 44); yarn build OK; nose2 73 OK.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1) Phantom empty "Extra Fields" row when editing tools/charts/datasets/
scripts: stored records routinely carry a legacy placeholder row
[{extrakey: "", extravalue: ""}] (not even this form's label/value
keys), so the edit dialog seeded one blank row and the required-field
schema then blocked saving. Shared handling in ExtraFieldInput:
- cleanExtraFields() drops rows without a usable label AND value; used
when seeding the field array from an existing item AND on submit in
all four forms (both edit and add branches), so empty rows are never
rendered unasked and never sent in payloads;
- extraFieldsSchema (shared yup): an untouched empty row passes (it is
filtered on submit), a half-filled row errors — replacing the four
copies of label/value-required schemas, so a user-created blank row
no longer blocks saving;
- rows now take their default values from the field-array state (no
defaults/index mismatch after filtering); the plus button still adds
fresh rows; also fixes the value column showing the LABEL's error.
Real saved extra fields load, edit and save exactly as before.
2) Overlapping First/Middle/Last labels with multiple PIs (and authors —
same pattern in ReferenceInfoForm): the rows are plain nested Grids,
and MUI v9's gap-based grid no longer gives nested non-container items
the vertical padding the v4 negative-margin system did, so shrunk
labels collided with the row above. Each map is now wrapped in a
`container direction="column" spacing={2}` so every PI/author renders
as a cleanly separated row with its remove button; data shape and
submit behavior unchanged.
tests: NEW ExtraFieldInput.spec.js (+5: legacy placeholder row renders
nothing, empty/missing defaults render nothing, real fields still render
with values and survive mixed legacy junk, plus-button adds a row,
cleanExtraFields unit matrix); PaperInfoForm.spec.js +1 (two PIs render
as two distinct rows). yarn test 55 OK (was 49); yarn build OK;
nose2 73 OK (backend untouched).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…resp server Qresp federates in the browser only: the Explorer picks servers, and the detail page fetches the record from whichever one `?server=` names. The backend was never part of that, so a record opened from PaperStack made the browser ask the LOCAL server about an id only PaperStack has. That id is not in the local database, and the endpoint answered 404 -- correctly, for the question it was asked. Every federated record therefore had no Related Research at all. Pass the server through and read the record and the corpus from it. The peer's corpus is what the record is scored against, because "specific to this field" measured against a different server's vocabulary is a different question; a corpus that cannot be read is a failure of the section, never a reason to substitute the local one. Everything downstream -- profiles, IDF, evidence families, the quality gate, the five-result cap, the external provider -- is the same code on the same shapes. project/federation.py is the whole boundary. `?server=` selects from a list, it never supplies a target: shape (no credentials, query, fragment, path, odd scheme, or non-ASCII host), then "is this us", then HTTPS, then a literal-address rule that refuses loopback/private/link-local BEFORE the allowlist so a compromised registry cannot name one, then an exact origin match. On the wire: no redirects, an 8s timeout, and an 8MB cap enforced while reading rather than from a Content-Length the peer controls. A refused server is a 400 with no fallback to the local database -- answering with whichever local record shares the id would be a wrong answer presented as a right one. Only the published scientific metadata is copied out of a peer's answer. That answer also carries the curator's name, e-mail and affiliation, the RCC server path and the file paths; the allowlist is positive, so none of it crosses, and a field a peer invents is dropped by construction. Nothing is written: a federated record is read, scored and discarded, so a node can never accumulate shadow copies of another node's records. The cache key becomes server + id. A local record keeps its bare id, so every entry written before federation existed is still a hit and there is no migration; a remote record is namespaced by its origin, so two servers that happen to issue the same ObjectId cannot serve each other's answers. The allowlist needs the shipped list, not just the registry: the registry URL in config.ini answers 404 today, which is why the Explorer has been running off its own checked-in copy. backend/project/data/qresp_servers.json mirrors frontend/data/qresp_servers.js, a test asserts the two do not drift, and QRESP_FEDERATION_SERVERS overrides both for an operator who wants neither. Verified read-only against a live peer (65 records): five named published records answered 200 with five peer-corpus results each, no local record in any answer, nothing written locally, the cap respected across all 65, and one record correctly returning zero. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two defects, one visible symptom. The request never carried the server the detail page was showing, so a federated record was always asked about on the wrong server. It now forwards `?server=`, and a local record produces exactly the request it always did. And the section caught any error and rendered `null`. A failed request, a deployment without the feature, and "nothing is related to this paper" were then indistinguishable -- all three were an absent section -- yet only the last is a statement about the record. On a published page with the feature on, the section now always renders, in one of four states: loading, results, a legitimate empty result, or unavailable with a retry. It disappears only when the backend says `enabled: false`, when there is no record to ask about, or on an unpublished preview, which the page excludes as before. A Semantic Scholar failure still marks the external subsection alone and leaves the Qresp list intact. Results carry the Qresp server they live on, and links use it: a federated id resolves only on its own server, so dropping the origin would send the reader to a 404 -- or, if the id happened to exist locally too, to the wrong paper. The effect depends on both paperId and server and resets loading, data and the error flag when either changes. The same id on a different Qresp server is a different paper, so showing the previous server's answer while the new one loads would attribute one server's results to another. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tion contract What `?server=` means and how a federated request is answered; the five ordered checks a server URL has to survive and what each refuses; what is copied out of a peer's answer and what is not; the server+id cache key and why it needs no migration; the four UI states and the three ways the section renders nothing. Records the live read-only QA against a 65-record peer, including the one caveat it turned up: 64 of 65 records saturate the five-result cap, so the tail of a list can rest on ordinary English words that happen to be rare in a small corpus. That is a property of relatedness.py's `is_specific`, not of federation, and it is written down rather than changed -- adjusting it moves local results too. The staging checklist gains a federated-records section: the request must carry `server`, the results must come from that server, the refused-server and peer-failure cases, and the proof that nothing is copied into the local database. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d author Two rules decided what a reader was told, and both were wrong on a real corpus. Rarity was the ONLY test for "specific research term". On a 65-record server that promoted any word appearing in fewer than ten abstracts, so readers were shown `python`, `http`, `user`, `another`, `related`, `discussed`, `play`, `will`, `proper`, `class`, `comparing`, `particular`, `region` and `yield` as the reason two papers were related. `particular` in two records out of 32 is arithmetically as rare as `chalcogenide`, and no amount of document counting can tell them apart. A term must now ALSO look like subject vocabulary: a curated tag, a multi-word phrase, a digit or internal hyphen, an acronym or formula read from the author's own typography before lowercasing, or failing all of those a plain word of at least nine characters. Ordinary English, academic boilerplate and web/file words are blocked outright, and both word lists are singular-folded at import because tokenize folds plurals first -- `technologies` would otherwise never match the `technologie` that arrives. A shared author was a MEDIUM, so one PI co-authoring half a corpus supplied half of every gate decision and, paired with any second weak signal, pushed unrelated subjects through. Authors are no longer evidence at all: the gate is topic-only, removing every author from both records cannot change a verdict, and the count survives solely to order candidates that already passed on their own subject. No rule guesses which author is the PI, and no reason mentions a person -- what a reader is shown has to be the overlap that actually decided it. Both titles carrying two of the same technical concepts is now strong on its own: a title is the most deliberate sentence a record has. The cap drops from five to three, and the order is gate, then sort, then cut, so a short list is what a reader gets when nothing else clears the bar. Measured read-only against a live 65-record peer: records filling the cap fall from 98% (5 slots) to 60% (3 slots), two records now correctly return nothing, and across 15 results on five named papers no ordinary word appears in any reason. The accepted cost is short prose-only terms (`exciton`, `phonon`): they are missed unless tagged or capitalised, which loses recommendations rather than inventing them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…r it twice "No external papers" had five causes and one sentence. A reader could not tell a rate limit from an empty index from a gate that rejected everything, and neither could an operator reading the response. Each stage of the pipeline is now counted and reported: whether the provider resolved this paper, what it answered, how many candidates it proposed, how many survived de-duplication, how many cleared Qresp's gate, and how many were shown. `reason` names the cause -- provider_returned_no_candidates, all_candidates_below_quality_gate, source_paper_not_in_provider_index, provider_rate_limited, provider_timeout, provider_error. `status` is unchanged, so the UI contract is untouched: the first two are a perfectly healthy `ok`, because the gate is never relaxed to fill a list. Counts only -- no title, no abstract, no provider body, no credential. Instrumenting it immediately corrected a claim this file made. A note here concluded from two DOIs that the default recommendation pool always answers empty and the external half was useless for Qresp. Of three real PaperStack records, one came back with 20 candidates of which 3 passed the gate. An empty answer is the provider's coverage, not a Qresp bug, and now says so. Caching, all of it in-process except the provider answer that was already persisted. Before, every view of a federated record read the peer twice -- once for the record, once for its entire corpus -- so five reloads cost the peer ten requests and five simultaneous readers cost ten more. Now: a peer's record and corpus are cached per origin, the computed response is cached behind a single flight, a stale answer is served immediately while it is refreshed behind the reader, and a failure is remembered for 45 seconds so one outage cannot become a request storm. Five reloads and five concurrent readers each cost two peer requests, pinned as exact numbers by tests. Local records are deliberately not cached: the answer comes from this server's own database, costs no peer and no provider request, and recomputing is what keeps the promises the product already makes -- a deactivated record disappears on the next reload. ALGORITHM_VERSION is part of every cache key, in memory and in Mongo, so tightening the quality gate stops the weak and empty answers the old gate produced from being served instead of waiting for them to age out. Entries predating the field are a miss; that is the whole migration. Tests also pin what this feature does NOT do: no Gemini call, no Gemini quota, no import of the assist client on any module in this path, and no outbound host other than the peer and Semantic Scholar. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…er list Three gaps from the previous round. The federation list existed twice -- shipped with the frontend, enforced by the backend -- and nothing kept them in step, so a server could be offered in the Explorer and then refused with a 400 by the endpoint that reads it. The backend now publishes the list it actually enforces at GET /api/federation/servers, and the Explorer asks for it; the checked-in file is only the offline fallback for a backend too old to answer. The registry was read through util.Servers, which fetches it with verify=False. That is not acceptable for a list whose job is to decide what this server may contact: anyone able to intercept an unverified fetch could add themselves to the allowlist. Federation now reads the same URL itself, with certificate verification and redirects refused. util.Servers is left alone -- changing it would alter the curator and publish flows, which are out of scope. Refusing literal private addresses was never enough on its own. An allowlisted NAME whose DNS answer is 127.0.0.1 or 169.254.169.254 would still have been fetched, which is the standard way an allowlist becomes a request against the machine itself. Every address a hostname currently resolves to must now be public, a name that does not resolve is refused, and the check happens before any request leaves the process. Verdicts are cached briefly so this costs one lookup per host, not one per page view. DNS rebinding is still not defeated, and that is recorded rather than implied: the check and the connection are separate steps, and pinning the connection to the checked address is not something requests exposes. On the UI side, the reason vocabulary is written down where the two sentences are chosen, so "the provider had nothing" and "nothing cleared the gate" keep reading as an answer while a rate limit and a timeout keep reading as a failure -- and a Semantic Scholar failure still leaves the Qresp list intact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… caching What now counts as a "specific research term" and why rarity alone was not enough; why a shared author is not evidence and what it is still used for; the seven reason codes behind an empty external list and which of them are a healthy `ok`; what is cached where, the reasoning behind each TTL, and the measured request counts for five reloads and five concurrent readers; the zero-Gemini contract; the DNS check and the verified registry, with DNS rebinding recorded as the residual risk. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The documented way to switch federation off was to set the variable to a single space. It did the opposite. The value was `.strip()`ed and an empty result was read as "the variable is not set", so the shipped list came back and every peer on it stayed reachable. An operator disabling a feature got it enabled. Presence is now decided by `os.environ` membership, and only the CONTENT decides what is allowed. Absent means registry plus shipped list; present means exactly the origins named, and naming none -- "", " ", ",", or junk that parses to nothing -- means an empty allowlist and no federation at all. An empty allowlist can never widen into an open one: every ?server= is refused. The Explorer had the mirror-image bug. It only adopted a published list if it was non-empty, so a backend that had switched federation off still had its shipped peers offered in the UI -- servers the backend would then refuse with a 400. An empty published list is an answer and is now respected; the shipped list is the fallback for exactly two cases, a failed request and an answer that is not the documented shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The registry decides this server's outbound allowlist. It was fetched with certificate verification, but the URL itself was only required to parse -- http:// was accepted. Verifying a certificate that is never presented buys nothing: anyone on the path of a plaintext fetch could add themselves to the list of servers this deployment will contact. An http:// (or any non-HTTPS) registry URL is now not requested at all. The result is "no registry", so the shipped list and an explicit QRESP_FEDERATION_SERVERS both still apply, and a misconfiguration narrows federation instead of weakening it. Redirects, the timeout and certificate verification are unchanged, and the URL is still never logged -- it comes from config.ini and may name an internal host. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ast good answer Stale-while-revalidate had no guard. Every reader of a stale entry called spawn_background, so five readers arriving together on a record whose peer caches had also expired started five refreshes and five rounds of peer reads -- the exact multiplication the caching was added to prevent. SingleFlight is the wrong tool here: it serialises callers who are all waiting for the same answer, and nobody waits for a background refresh. RefreshGuard instead lets exactly one reader start the work and tells the rest there is nothing for them to do. Different keys never block each other, and the guard is released in a finally, so an exception cannot strand one. A failed refresh also used to overwrite the good stale answer with the unavailable response it had just computed -- taking a true answer away from readers because a peer was briefly unreachable. It now keeps serving the last good result for the rest of its stale window, and the failure is recorded as a 45-second cooldown on that key instead, so one outage is not re-tried by every page view and one new attempt is let through once it expires. That distinction needed care: a refresh that preserves a good answer RETURNS success and IS a failure, so `_refresh_and_report` reports the attempt's own outcome rather than letting the caller infer it from the value served. Reading it off the served value silently cleared the cooldown, and a test passed for that wrong reason before this was split. The guard holds an entry only while a refresh is in flight or a cooldown is unexpired, and prunes on every call, so it tracks concurrent work rather than every record ever viewed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`after_gate` and `shown` were identical in every response ever produced, so the pair could not report the one thing it existed for: how much the three-result cap is discarding. `external_recommendations` had already truncated the list, and the count was taken from the truncated result. It now returns every candidate that cleared the gate when asked to, and the caller applies the cap, so `after_gate >= shown` and the gap is visible. The order is unchanged -- gate, sort, then cut -- and so is the default for other callers. The counts also vanished on a cache hit: they were computed live and never stored, so the second view of a record answered without them and looked like a different kind of answer. `reason` and `pipeline` are now stored alongside the results, and a cached response explains itself exactly as the live one did. The field is optional by design. An entry written before it existed has no pipeline, so the key is omitted rather than invented, nothing crashes, and the next real refresh fills it in -- the same migration-free pattern the fingerprint and algorithm-version fields already use. Only booleans, a status string and counts are stored: no title, no abstract, no provider body, no credential. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One class per defect, each reproducing it before the fix: the environment allowlist across unset / valid / empty / whitespace / comma-only / junk, five concurrent readers of a stale record refreshing once, a failed refresh preserving the last good answer and cooling down for 45 seconds before exactly one retry, `after_gate` exceeding `shown`, live and cached responses agreeing on reason and pipeline, a legacy entry without a pipeline still serving, and an http registry that is never requested. The refresh guard shares the result cache's injectable clock, so a cooldown is stepped over deliberately rather than waited out, and the peer caches are cleared between views so the cooldown assertions measure the guard and not the peer negative cache. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tract Descriptions left behind by earlier changes: "capped at five" and "five-result cap" (the cap is three), authors listed as one of the independent evidence families the gate reads (it is not evidence at all -- it only orders candidates that already passed), and "set it to a single space" as the way to switch federation off, which is now true rather than aspirational. Adds what the four fixes changed: the exact allowlist table for absent / named / empty / junk values, the pipeline field meanings with the `after_gate >= shown` guarantee and the live-versus-cache parity, the HTTPS-only registry row in the security order, and how a stale refresh is guarded, preserved on failure and cooled down. Documentation and comments only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The AI action on an RCC candidate received the candidate's name, its relative paths and the analyzer's structural sentences. Everything the analysis had already read off the file server was discarded on the way: _script_header() was defined and called from nowhere, README text reached only Tool manifest parsing, notebooks were excluded from evidence reads outright, and no function or class name was ever extracted. A Script whose module docstring said exactly what it does was described from its filename. project/evidence.py now extracts a structured, boundary-confined bundle per candidate -- README, module docstring (ast), top-level def/class NAMES, notebook MARKDOWN cells, manifest lines -- and curation.py attaches it as `ai_sources`. The paper's title and abstract travel as `paper_context`, and the prompt states an evidence hierarchy that forbids claiming what a script computes or a chart shows on the strength of the abstract alone. What is structurally impossible, not merely discouraged: raw dataset values, image bytes, notebook code cells/outputs/attachments, function bodies and string literals. No extractor for any of them exists. A Python file that does not parse yields nothing rather than being regex-guessed. Evidence never crosses a boundary, so a sibling dataset's README cannot describe this one. Credential-shaped values are redacted before the bundle is built and again on the way out, because it round-trips through the browser. Budgets are explicit: 1200 chars per source, 3000 per candidate, 8 sources, and a read plan spent ROUND ROBIN across candidates so one large folder cannot leave every later candidate's README unfetched. _script_header is now on the real Script path and delegates to the shared extractors, so the Details panel and the AI bundle cannot disagree about what a file's header is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
aiItem() built `context` from draft.readme + draft.description -- the curator's own answer to the very field the model was being asked to fill. A filled field produced a paraphrase of itself; an empty one produced nothing but the analyzer's structural sentences. It also made any benchmark against curator text self-fulfilling. The request now carries the candidate's structured `ai_sources` and `inventory`, plus the paper's title and abstract as background. The server dropped `context` from its allowlist in the previous commit, so an older client cannot reinstate the leak. The consent dialog was also no longer true: it promised no notebook contents while the payload now carries notebook markdown, and said nothing about the paper background. It now itemises the ACTUAL source list for that candidate -- type and path, one line each -- and warns, before the request is spent, when a candidate has no readable text and the answer will be "not enough evidence". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
65 new backend tests and 5 new frontend ones, over the properties that matter rather than the shape of the payload: - a sibling's README/docstring never describes this candidate, and `scripts/analysis2` is not inside `scripts/analysis`; - _script_header is actually CALLED (a spy, because "defined and unused" is exactly the bug being fixed); - top-level names only -- no bodies, no literals, no nested methods, no private helpers, and nothing at all from a file with a syntax error; - notebook markdown only -- no code, no outputs, no base64 attachments -- and a corrupt notebook skips its own evidence, not the analysis; - a Chart with only an image gets no sources, so abstention is the correct answer; - title/abstract travel but are not artifact evidence, asserted against the prompt's own wording; - Tool keywords are dropped server-side and a Tool is never even asked; - secret redaction, per-source/per-candidate/per-request caps, forged source types and paths, prompt injection, confidence clamping; - exactly one candidate per provider call, one quota unit, and none spent on a rejected request. Existing tests that pinned the OLD contract are updated, not deleted: `payload["item"]` became the `paper_context`/`artifact`/`sources` bundle, and the root requirements.txt is no longer fetched because a root file belongs to no boundary and nothing ever consumed it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every sampled candidate is now asked TWICE -- once with the pre-change input (name, paths, structural sentences) and once with the shipped bundle -- so the difference is paired on the same candidate instead of on two different samples. The structural sentences travel as `artifact.structure_notes`, not as a source, so a description copied out of them does not score as grounded. Reported per record type per mode: description groundedness (against the EVIDENCE, with the paper's abstract deliberately excluded from the denominator), a usefulness floor, keyword concept precision/recall, generic-term ratio measured before the server's stopword filter, and abstention correctness -- where `missed_abstention` (described an artifact with nothing to describe from) is the failure this change targets. Leakage work: - the reference corpus is the published corpus MINUS Qresp's own QA/test records, each exclusion printed with the rule that fired it. The first rule was over-eager and matched "Testing the limits of DFT for water"; a QA word now has to be used as a label, not as a sentence; - the target record's curated description AND keywords are scrubbed from every source excerpt, which makes keyword recall an inference test rather than a copying test; - the leave-one-record-out vocabulary is unchanged and still holds out the target; - adding `paper_context` opened a real channel: a paper titled "Band structure of monolayer transition metal dichalcogenides" contains two of its own artifacts' reference keywords. Deleting the title would benchmark a product that does not exist, so those keywords are counted and recall is reported a second time without them; - a unit whose FINAL payload still contains the curated description is dropped and never called, with the reason printed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…mark RCC_FOLDER_ANALYSIS.md now shows the actual request shape, a per-kind table of which source types each record type may carry, the budgets, and the two facts a reader most needs: nothing the curator typed is sent, and abstention is a correct answer for a Chart that is only an image. AI_ASSIST_EVALUATION.md documents the two evidence modes, what each metric means, and the paper-title caveat -- including why it is reported rather than scrubbed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The prompt asked the model to return an empty description when `sources` is
empty. That is a request, not a guarantee. The server called Gemini anyway,
spent a quota unit anyway, and returned whatever came back -- which for a
Chart holding nothing but an image meant a caption assembled from the file
name and the paper's abstract, the one thing the prompt most explicitly
forbids.
describe_candidates() now decides abstention itself, from the evidence,
after authentication/CSRF/consent/one-candidate (all unchanged) and BEFORE
the provider configuration is read:
{"suggestions": {}, "no_suggestion": ["chart-0"]} HTTP 200
No new API field: `no_suggestion` already carries this, and the browser
already handles it. The answer is the same on a server with no API key,
because whether a candidate can be described is a property of the folder,
not of the provider -- a candidate that DOES have evidence still gets 503
there.
_sanitize_sources() also had no idea what kind of candidate it was
validating, so the global type allowlist let a tampered client hang a
`docstring` on a Chart. It now takes the kind and filters against
evidence.accepted_source_types(), which is the single table both directions
read: the extractors decide what to produce from it, the endpoint decides
what to accept from it, and AI_SOURCE_TYPES is derived from it rather than
repeated beside it. A bundle filtered to nothing takes the abstention path.
swagger.yml's enum cannot express this -- it knows the seven type names, not
which kind may hold which -- so it stays a first gate only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`no_suggestion` now carries two different situations and the curator needs to tell them apart: the server declined to ask because the candidate has no evidence of its own, or it asked and the provider had nothing usable. One is fixed by adding a README to the folder; the other by trying again. The message is chosen from the candidate's own `ai_sources`, which the browser already holds -- no new API field, so the two cannot drift apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Written before the fix; 16 of them failed against 6893f7f. Backend (28 new). Every abstention case watches the provider AND the quota counter, because asserting only "no provider call" would still pass if the curator had been charged for a request that was then not made: - empty sources, for all four kinds, and sources that sanitize away to nothing -- 200, empty suggestions, id in no_suggestion, zero calls; - a Chart carrying only a forged docstring, a Dataset only python_symbols, a Script only notebook_markdown, a Tool only python_symbols -- all filtered to nothing, all abstaining; - a mixed bundle keeps exactly the types its kind can carry; - a Chart with a real README and a Script with docstring + symbols still make exactly one call and spend exactly one quota unit; - an unconfigured provider abstains on no evidence and still 503s on real evidence; - auth, CSRF, consent and the one-candidate rule are still enforced on the no-evidence path, which is not a shortcut around them; - the abstain path writes nothing and logs no evidence text; - the analyzer never emits a source its own kind's filter would reject. Also pins that swagger.yml parses and that its enum matches the code. An unquoted JSON brace in a description opens a YAML flow mapping and breaks the spec, which surfaces as every test module failing to IMPORT -- a confusing signal for a typo in one string. Frontend (3 new): the evidence-based notice, the provider-had-nothing notice, and that an abstention leaves the curator's typed value alone and adds nothing. Two existing Tool tests supplied a Script fixture's docstring as their evidence; they now supply a manifest and a declaration, because a Tool cannot carry a docstring and would otherwise (correctly) abstain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records the exact response, that it costs no quota and makes no provider call, that it answers the same way on a server with no API key, and that the per-kind source table is enforced on the way in as well as on the way out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Chart whose caption the analyser could not determine kept its analysis-time
"Needs input" chip AFTER the AI filled the field -- while the card header,
which reads the draft, correctly counted the field as no longer missing. The
same staleness would have let a "High evidence" chip, earned by a file path
the analyser detected, vouch for a path a curator had typed over it.
The cause was one line reading the wrong thing: the chip rendered
`candidate.field_evidence[field]` whenever the field was non-empty.
`field_evidence` is a statement about `candidate.proposal`, frozen at analysis
time; nothing re-read it, so it described a value that was no longer there.
Three facts were being conflated -- what the analysis proposed, what the field
holds now, and how strong the analysis' evidence was. `valueState`,
`evidenceChipFor` and `suggestionApplied` in Utils/artifactFields.js are now
the single place they are combined:
blank -> no chip. The asterisk, the helper text and the header's
missing count are the three required indicators; a fourth is
noise, and flagging OPTIONAL fields this way was a bug once
already.
unchanged -> the analysis' own high/medium standing.
changed -> no chip. Nothing verified this value.
`needs_input` is therefore unreachable in either direction rather than
special-cased.
The AI panel's "not applied" was a hardcoded literal that never changed. It
and both Use buttons are now DERIVED from the draft, so applying a suggestion
says "applied", editing or clearing the value says "not applied" again, and
the button stops telling the curator their text is being protected from the
AI when the text in the field is the AI's own.
Nothing about what is sent, saved, added or published changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Explorer is about to open on results instead of on a node picker, so
"which server" stops being something a visitor types and becomes something
the deployment has to answer. It is answered here, beside the allowlist,
rather than in React -- a default hardcoded in a page would be a second copy
of the federation config, and the first copy already drifted once.
`/api/federation/servers` gains `default_server`. Additive: a client that
only reads `servers` is unaffected.
QRESP_DEFAULT_EXPLORER_SERVER names it. The value goes through the same
`parse_origin` as every other origin -- so a trailing slash, a mixed-case
host and an explicit :443 all resolve to the spelling the allowlist holds --
and is then checked for MEMBERSHIP. Naming a server here can pick among the
federated ones and can never add one; an origin outside the allowlist is
ignored with a log line rather than obeyed, because a default the allowlist
refuses would send every first-time visitor into a 400 naming a server they
never chose.
Without the variable: the first origin in the published (sorted) order --
deterministic, and visibly the first row.
An empty allowlist yields "", which is an answer ("this deployment federates
with nobody") and not a failure. No SSRF, HTTPS, literal-address or DNS check
is touched: this only chooses among origins those checks already permit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Clicking EXPLORER asked a question before showing anything: pick a Qresp
node, then search. Answering it wrong -- Duke, currently unreachable --
produced a blocking "Search Error!" modal on top of a page reading "0 Records
Available", which is exactly what a healthy but empty node also says.
`/explorer` now redirects server-side to the default server's results, so the
navbar link, a typed URL, a refresh and the back button all behave the same.
The target comes from `/api/federation/servers`; no host is named in this
file, and a `default_server` the published list does not contain is refused
here too rather than taken on trust.
Federation is not reduced to one node: `/explorer?choose=1` still offers the
picker, `/search?servers=a,b` is unchanged, and each record keeps its own
source server in its detail link.
The search page had one failure mode for two situations. Now:
- some nodes failed -> results from the ones that answered, plus a
non-blocking warning naming the ones that did not;
- every node failed -> an in-page unavailable panel with Retry that says
this is a connection problem, NOT an empty node;
- navigating -> an explicit "Searching…" state, because Next keeps
the previous page mounted while it fetches and the
stale count would read as the new one.
No blocking modal is raised for a search failure at all any more.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All of these fail against 7379e69 and pass after the two fixes above. Folder Analysis (7 integration + 15 unit). The integration tests use the real analyzer's own field_evidence for a chart folder -- imageFile high, everything a human supplies needs_input -- which is the shape that produced the report: - the stale "Needs input" chip is gone once AI fills caption and keywords, and the header goes 3 -> 1 on the same click; - the panel says "applied" only when the suggestion's values are actually in the fields, and goes back to "not applied" when one is edited; - clearing an applied value restores the missing count and re-enables Use; - High evidence disappears when the value stops being the analysed one; - applying adds, saves and publishes nothing (2 posts: analyse, describe); - closing and re-analysing leaves no applied state on the same candidate id. Federation (10). The default is published, is always one of `servers`, is normalized before it is compared, is REFUSED when it names an origin outside the allowlist (falling back to the first listed one), never widens the allowlist, and is "" when this deployment federates with nobody. Explorer (9) and search (8). The redirect target comes from the backend and only the backend; no peer is contacted while deciding it; an unlisted default is refused; an empty or unreachable federation shows an unavailable page instead of redirecting; `?choose=1` still gets the picker without spending a request. The search page keeps partial results with a non-blocking warning, shows an unavailable panel with Retry only when everything failed, never raises a modal, never shows "0 Records Available" while loading, and keeps each record's source server in its link. One test reads pages/explorer.js and asserts no server hostname and no record count is hardcoded in it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
What the Explorer's default server is, the table of what each value of the new variable does, why an origin outside the allowlist is ignored rather than obeyed, and that the picker and multi-server URLs are still reachable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The panel had two states for a suggestion that can offer two things. Using only the keywords left the button reading "Applied to Keywords" directly below a header still reading "not applied" -- the same kind of self- contradiction the stale evidence chip produced. `suggestionState(kind, draft, offers)` in Utils/artifactFields.js is a small pure helper over what the suggestion ACTUALLY offered: an entry with no target field (a Tool has no keyword field) or an empty value was never an offer and cannot hold the state back. So a description-only suggestion is `applied` after one click rather than stranded waiting for a second field that does not exist. none of what it offered is in place -> not applied some -> partially applied all -> applied Still derived from the draft on every render, never stored, so editing or clearing an applied value walks the state back on its own. The state is spelled out in text; the colour is only a second cue. The per-field "Applied to ..." buttons, the stale-chip fix and the evidence-only-while-unchanged rule are untouched, and nothing here saves, publishes or adds anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Qresp node is asked for four endpoints and they are not equal: /api/search fills the results table, while /api/collections, /api/authors and /api/publications only fill dropdowns in AdvancedSearch. The loop treated them as one list, `break`ing on the first failure of either kind and marking the whole SERVER failed. So a node whose records had already loaded, but whose authors list 404'd, was announced as one whose "records are missing from these results" -- directly above the rows it had served. Worse, `total` was `failed.length >= servers.length`, so with a single configured node that one broken auxiliary endpoint replaced the whole page with an unavailable panel while its records sat in `data`. The core endpoint is now fetched first and staged in a local, committed only once it has actually arrived; its failure drops that node's records and skips its filters entirely. The auxiliary endpoints are then fetched independently -- no `break`, because one being down says nothing about the other two, and the old flow discarded filters that had nothing wrong with them. `error.failed` (records missing) and `error.filters` (records fine, filters short) are separate, and total failure is measured on how many nodes actually produced records rather than on a count of nodes with something wrong. `error.is`/`error.msg` are kept for older readers. The filter notice names the endpoints: "Records were loaded, but some search filters are unavailable from: https://x (authors, collections)." Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Searching PaperStack and Duke together put an un-dismissable dialog over
PaperStack's perfectly good matches. `/search`'s SSR load had already learned
to tell a failed node from a failed filter and to say so beside the results;
`AdvancedSearch.onSubmit` had not, and still called the global `setAlert()`
on any server error.
The cause was ownership. The component ran the search AND decided what the
page said about it, so its only vocabulary for "one of two nodes is down" was
a modal. The results live in `pages/search.js`, so the status that describes
them now lives there too: the component reports `{papers, failedServers,
totalFailure, retry}` and the page decides.
every node answered -> results replaced, no notice
some answered -> only those committed, inline warning naming the
others; the good matches are never discarded
none answered, results
already on screen -> nothing is committed, inline error saying the
previous results are still shown
none answered, nothing
on screen -> inline error, and the record count is withheld
rather than reading "0 Records Available"
answered with 0 rows -> an ordinary 0 Records Available, not a failure
Results are staged per server and committed only after every node has been
asked, so a late failure cannot land after a partial commit, and a total
failure never calls setData({}) over results that are still valid.
Retry re-runs the criteria and server list captured when the search started,
not whatever is in the form by the time it is pressed. `showLoader`/
`hideLoader` are paired in a `finally`, a submit already in flight cannot be
started again, and a result arriving after unmount is dropped. The thrown
error goes to the console; the page is told only WHICH server failed, so no
host or stack reaches the screen.
The SSR notices (`error.failed`, `error.filters`) and this runtime one are
separate state and can be shown together -- they describe different events.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
QA completed on staging
Production notes
Deployment safety
This PR has not been deployed to production. Production deployment should only happen after configuring production OAuth/SMTP/admin env vars and taking a code + MongoDB backup.