Duplicate entries in the artists' books list (same catalog record minted as multiple nodes)

[rybesh]

Problem

The artists' books list shows the same book more than once — e.g. (Ho + go)² = it and 28 days : a deck of cards each appear twice. Across the index, 51 titles appear on more than one row (a few on three).

Root cause

The construct query mints one ab:ArtistsBook node per Zotero itemKey and does no bibliographic deduplication. The source collection (Zotero lib 1, the curated Artists' Books Collection the pipeline builds from — see Zotero/README.md) contains multiple item records that map to the same catalog record, so each surfaces as its own list entry. The pipeline is faithfully reflecting the source; the duplication is upstream of the graph.

The repeated titles fall into three distinct categories, which is why a naive title-based dedup would be wrong:

1. Accidental double-entries — same bib record, identical call number (4)

The same item entered twice. (28 days was added to Zotero 88 seconds apart.) These are data-entry mistakes and safe to merge/remove.

title	itemKeys	call number	catalog
Nada para el destino = Nothing for destiny	U3BJIXIE, N9YFMRAC	N7433.4.V47 N23 2009	UNCb6975170
Performing an archive	RSGVH7R8, F5XMI2TG	TR655 .L352 2015	UNCb8400904
28 days : a deck of cards	TJEXM6ME, Q7GM2XJS	N7433.4.N425 A13 2000	UNCb5647598
A girl's life	FWSWMASH, T3KKZBKH	N7433.4.D83 G58 2002	UNCb4192956

2. Multiple physical copies — same bib record, differing (copy-specific) call numbers (40)

Legitimately separate holdings of the same work (… c. 2). One bibliographic record, multiple copies. Whether these should collapse to one browse entry is a policy call. Examples (12 of 40):

title	itemKeys	call numbers	catalog
Fragments of the peculiar institution	X3KX92BL, PXCQU5L6, IA54GVQX	…F73 2016 / …F73 2016 / N6537.S3636 A4 2016	UNCb8562276
Printed web.	VZC5BRP9, ZICHM787	…P75 no.2(2014) / …P75 no.1(2014)	UNCb7690550
El muro = The wall	YJ9V49Y6, CBUYF3ST	…M87 2009 c. 2 / …M87 2009	UNCb5944456
Sheherezade : a flip book	GAHFZ256, 566PNJJ7	…S54 1988 / …S54 1988 c. 2	UNCb2238039
(Ho + go)² = it	SQDPPBXK, YGVJRI4Y	…H6 1986 / …H6 c. 2	UNCb4293240

((Ho + go)² lands here — copy 1 vs c. 2 of the same UNC record UNCb4293240.)

3. Same title, DIFFERENT bib record — distinct works (9) — must NOT be merged

Different books that merely share a title (different catalog records, often different author/year). Title-based dedup would wrongly collapse these.

title	itemKeys (→ distinct catalog records)
Canto a Matanzas	4EC84LP7→UNCb7981223, XY9DI4TS→UNCb7672486
Memory loss	3L36U9YX→UNCb5645973, U64L9DKA→UNCb2262050
No man's land	885A8TBX→UNCb7614589, CJB8FZKV→UNCb7614590
Ohio	RYZJVH45→UNCb7406945, MKPJRPYM→UNCb7201879
Sierra suite : (overheard)	566FGEFI→UNCb9407230, 2B2582MX→UNCb11208956, EED7JCA6→UNCb11208955
Suite vénitienne	DBI69A84→UNCb4538590, 3M6P242S→UNCb8390381
Versus	3HXRSQYY+AS9A8WPX→UNCb8257169 (a copy pair), RZEMGHH9→UNCb9730172
War cut	TBSWMXFY→UNCb7356631, BBS6NWT3→UNCb4483215

(Versus spans categories 2 and 3 — two copies of one record plus a distinct work.)

Scope (counts, verified against Zotero/artists-books.csv, 1,341 rows)

• 44 catalog records are shared by >1 Zotero item → 45 redundant rows.
  • 4 are accidental double-entries (identical call number).
  • 40 are multi-copy holdings (differing call numbers).
• 9 repeated titles map to >1 distinct catalog record (distinct works — not duplicates).
• Also surfaced: 1 record with an empty title (9MQS4NXM, no catalog URL) — separate data-quality nit.

The right dedup key

Not the title. The shared catalog record is the reliable signal — it's already in the source as the url column (e.g. catalog.lib.unc.edu/catalog/UNCb4293240); OCLC (from the MARC 001/WorldCat) would work equally. Grouping by catalog record cleanly separates category 3 from 1 & 2.

Options

1. Fix at source — remove the 4 accidental double-entries in zotero.sqlite (correct fix for category 1; independent of the graph build).
2. Collapse in the construct query by catalog-record URL (or OCLC) — merges categories 1 and 2 onto one node; pick a surviving itemKey and optionally record "N copies" / both call numbers.
3. Leave as-is — the list then faithfully mirrors the library's holdings (every physical copy shown).

A decision is needed on category 2 (collapse copies vs. show every holding) before implementing 2.

Notes

• Source-data / pipeline-policy issue; independent of [Infra] Build the reference-page and results-page views (nothing consumes the reference graph yet) #63 / PR Build the reference-page and results-page views (consume the reference graph) #69 (the reference & results views), which only revealed it.
• Helper to regenerate these tables: a short csv-module script over Zotero/artists-books.csv grouping by url / title (used to produce the counts above).

[beeyop]

I had to stop at "Performing an archive" for the day. All the title dupes up to that point alphabetically can be merged up to that point except:

1. Christopher Williams (2739, 2740): different volumes but our artist book collection copy keeps both volumes under the same number, circulating copies are seperated. I think it is safe to merge.
2. Canto... (1783, 1841): Different books with different years.
3. Mariposa 5 (1820, 1821): same as Christopher Williams
4. Memory Loss (forgot to note numbers): two totally different books
5. No man's land (885A8TBX, CJB8FZKV): different volume numbers and different years. we catalog them seperately but the artist has them combined on her site. We might want to check citation notes to see if references are making a distinction.
6. Ohio - same book but one is special edition, no content differences
7. Parallax- one has handmade cover, but counted as same book
8. A humument (532)- not a title dupe in the artists-books.csv but the library hold 4 copies all of different editions with different content. Only one is held in the AB collection. We may want to check citation notes to make sure they reference the 1st edition (1980).

[beeyop]

Review items frtom dedup review csv:
Metamorfosis (correct spelling)- if by Pilar Álvarez Pablos in 2020(?) it is the same book, only one copy in catalog and no artist books in catalog by the other spelling
Avalanche- a periodical, the library service center hold individual issues, the art library copy is a facsimile from 2010 that covers issues 1-13 (1970-1976)
The bowl and the book- only one in catalog
Kidnapping mountains (correct title) - only one in catalog
Ingres and other parables- same book that has two dates 1971 (title page date); 1972 (copyright date); date in catalog as 1971 [i.e. 1972]
South America (proper title) - Richard Long: South America, 1972 and South America, 1972 are one book.
The practical speller : a manual of instruction in the rudiments of the English language- only one in catalog
Arbitra- This is a interesting one. It is an edition of 40 (ours is number 9) produced between 2005-2009 but each book is different. Should include the material note "unique book" but doesn't have any material genre notes. I assume the citation refers to them in collective.
En tierra, hilando vientos - only one in catalog
Googled English frontier deedbook- same book, limited edition and each book might be unique but are all from the same project and citation would probably be for the collective
Bobby: book nr. 100 - same book, two copies in catalog
Analemmic : an equation of time- same book, two copies in catalog
Groar! : (14 poemas-fósforo) - same book, questionable date range [between 2000 and 2009?]
Shadowed memory - same book as long as authors are Susan Kae Grant or Randall Hill
Words of Chan K'in : Lacandon jungle - english version; wilson holds non-english version Palabras de Chan K'in : Selva Lacandona.
The 24 hour self portrait- same book questionable dates [between 2000 and 2009?]
[I love you]. -same book, questionable dates [between 2000 and 2009?]
Identity- same book, questionable dates [between 2000 and 2009?]
Making stuff and doing things. - 2 books and a cassette in one case, questionable dates [between 2000 and 2009?]
Shirana Shahbazi : monstera -same book
The fortune teller - same book, Limited variable edition of 25 copies
Mirrors- same book
High tide - first published 2006, republished 2012 with additional material
Resolution - same book
Me garden - same book questionable dates [between 1990 and 2003?]
Ayotochtli- same book, may be handmade, questionable dates [between 2000 and 2099?]
De cuando no habia agua : cuento yaqui - same as ayotochtli
Versus - two volumes (Black book and White book). Cataloged together.
Drogas, sexo, rock 'n' roll. - same book
Trumped up empathy. - same book, Originally inserted into the Journal of Artists' Books 41 (Spring 2017).
Nancy Reagan is killing me - same book, Artist's book included in issue no. 41 (Spring 2017) of the Journal of Artists' Books.
Breathe- same book if both by Levi Sherman. Included with issue no.41 of the Journal of Artists' Books.
Tomorrow's man 4 and Tomorrow's man 3- different books
Exhausted geographies and Exhausted geographies II - different books
Blueprints (correct title) - if by Jesse Chun, is the same book
Mourning/Warning. Numbers and repeaters and Mourning/Warning - different books (numbers and repeaters is also called M/W 2)
How to make flipbooks - same book, questionable dates [between 2000 and 2009?]

[rybesh]

Manual dedup review — progress

The human review of the fuzzy lib-1 dedup attaches now has a stable, hand-owned home. Previously the decisions were being written into sources/artists-books-dedup-review.csv, but that file is generated by dedup.py --review and gets overwritten on every make, so the review work was volatile.

PR #101 (commit 31f7246) fixes that:

• artists-books-dedup-review.csv is restored to its generated 8-column form (machine-owned).
• New hand-owned sources/artists-books-dedup-decisions.csv holds the human calls, keyed by lib1Key and joined to the generated review file for context — the same generated/hand-owned split already used for construction/decisions.csv and subjects/decisions.csv. A dedup re-run no longer clobbers it.

Review status so far (480 fuzzy attach candidates):

decision	count	meaning
same	71	genuine duplicate — safe to collapse
no	8	distinct works sharing a title — must not merge (category 3 in this issue)
unsure	6	needs another signal (author/year/material) to call
(blank)	395	not yet reviewed

So 85 / 480 reviewed, 395 remaining. The no calls are exactly the false-merge risk this issue flagged — title-similarity alone would have wrongly collapsed them.

Not yet wired up: nothing consumes decisions.csv yet — dedup.py still auto-attaches every fuzzy match regardless of the human call. Feeding the decisions back into the attach logic (a no should block the merge, unsure should hold it for review) is the follow-up needed to actually act on this review.

[rybesh]

Update — manual dedup review wired into the pipeline (PR #101 merged)

PR #101 is merged to main (merge commit 057f52b). The manual review of the uncertain fuzzy dedup attaches is now a first-class, enforced part of the build, and its decisions are reflected all the way through to the graph.

What changed

• The human review lives in a hand-owned overlay, sources/artists-books-dedup-decisions.csv (keyed by lib1Key; decision ∈ same / no / unsure), separate from the generated artists-books-dedup-review.csv so a rebuild can't clobber it — same generated/hand-owned split as construction/decisions.csv and subjects/decisions.csv.
• dedup.py reads it back and gates the fuzzy merge: a lib-1 key marked no (distinct work) or unsure (unconfirmed) is left as its own canonical node instead of collapsing into its fuzzy twin. same and unreviewed keys attach as before. Conservative default for a frozen set — a wrong merge is harder to undo than a missed one.

Effect on the canonical set: sources/artists-books.csv went 7906 → 7920 canonical works — the 14 recorded no/unsure calls (8 + 6) split back out.

Directly addresses category 3 of this issue (same title, different bib record — must NOT merge). The review catches exactly these false merges; e.g. both distinct Canto a Matanzas works (4EC84LP7→UNCb7981223 and XY9DI4TS→UNCb7672486) are now kept as separate nodes rather than collapsed on title similarity.

Verified in the built graph (graph/artists-books.ttl, queried with arq):

• All 14 no/unsure works appear as distinct ab:ArtistsBook nodes (both Canto a Matanzas, both Tomorrow's man, etc.).
• All 71 same works: 0 appear as their own node — they correctly merged into their twins, lib-1 key retained as provenance in the twin's sourceKeys.

Still open / not covered by this change

• The review is partial: 85 of 480 fuzzy candidates decided (71 same, 8 no, 6 unsure); 395 remain unreviewed and currently auto-attach by confidence.
• The empty-title data-quality nit is unchanged: 8 canonical records still have no title (9MQS4NXM, Y2MQKHFR, EN42444Z, NERXSW2E, 657C6HGB, 7KVV2C6R, UM2HRVK3, UWWTV2WA) and so mint no book node — worth a separate cleanup.
• Categories 1 & 2 (same catalog record → accidental double-entries / multi-copy holdings) are collapsed by the exact-signal dedup (sameAs/ISBN/OCLC/title+author) from Rebuild the graph as three stages over the frozen critical index (supersedes #80) #82, not by this review layer.