Lessons Learned: Zotero collections ingest

Extracted from the session on 2026-06-30 that built zotero/ingest_to_wiki.py and ingested the Buckyverse Zotero group's curated collections into .wiki/raw/** (IMP-5). 7 lessons covering the Zotero local/web API, shell quoting, verification side-effects, idempotent-import design, curated-corpus data quality, and pluck branching on a large content repo.

Lesson 1: The local Zotero API is read-only; writes need a write-enabled web key

Category: gotcha
Context: Trying to remove off-topic items from the Buckyverse group collection programmatically.
Symptom: PATCH http://localhost:23119/api/groups/5189518/items/<key> → HTTP 501 Method not implemented. The committed web API key reported "groups": {"all": {"library": true, "write": false}}.
Root cause: Zotero's local connector API (port 23119) serves reads only; the shared group can only be mutated through the cloud Web API with a key that has group write permission.
Fix: Abandoned the automated write; the user removed the items in the Zotero desktop app, then the ingest re-verified.
Rule: You can READ a running Zotero instance over localhost:23119/api with no key, but any WRITE to a group library requires a write-enabled api.zotero.org key — check GET https://api.zotero.org/keys/<key> before attempting a mutation.

Lesson 2: Curated Buckyverse data lives in the Zotero GROUP library, not the personal one

Category: discovery
Context: First attempt to enumerate collections to ingest.
Symptom: GET /api/users/0/collections returned [].
Root cause: The personal library ("My Library", users/0) is empty; all curated Buckyverse material is in group 5189518 ("Buckyverse"), with collections Bucky Shelf L84AV9YL, Books BK23LBDC, By Bucky YAAHI5VV, Wish List HLYVN3EP, People R5ID4MAH.
Fix: Discover groups via GET /api/users/0/groups, then query GET /api/groups/5189518/collections and /collections/<key>/items/top.
Rule: For shared Zotero libraries, address /groups/<id>/…, not /users/0/…; find the group id from /users/0/groups first.

Lesson 3: zsh globs ? and & in unquoted curl URLs

Category: gotcha
Context: Probing the local API with curl …/collections?limit=200.
Symptom: zsh: no matches found: http://localhost:23119/api/users/0/collections?limit=200 — and a spurious 403 impression — even though the endpoint was fine.
Root cause: The interactive shell treated ?/& in the bare URL as glob/job-control metacharacters.
Fix: Single-quote the whole URL: curl -s 'http://localhost:23119/api/groups/5189518/collections?limit=100'.
Rule: Always single-quote URLs containing a query string in shell commands; an early "permission/no-match" error on a known-good endpoint is often shell quoting, not the server.

Lesson 4: Verify idempotency with --dry-run --no-fetch; never run write-mode just to "check"

Category: pattern
Context: Confirming the ingest was idempotent after the real run.
Symptom: Re-running python3 zotero/ingest_to_wiki.py (write mode) to check idempotency overwrote zotero/last-ingest-report.md with created=0 numbers and prepended spurious "0 items" bullets to every category _index.md; also --dry-run alone still hit the network because fetch defaults on.
Root cause: The check itself had side effects, and --dry-run only suppresses writes, not fetching.
Fix: Use --dry-run --no-fetch for verification; restored the clobbered files from the staged-good versions with git checkout --.
Rule: A verification step must be side-effect-free — pair --dry-run with --no-fetch, and never invoke a write-mode command merely to inspect state.

Lesson 5: Idempotent bulk import needs a provenance marker so cleanup is safe

Category: pattern
Context: Designing dedup + a --prune mode for the ingest.
Symptom: Risk that pruning "records whose item left the collection" would also delete pre-existing book sources that the ingest had merely enriched with a zotero_key.
Root cause: A zotero_key alone can't distinguish ingest-created records from hand-authored ones that were enriched.
Fix: Ingest-created records carry a zotero_collections: frontmatter field (written by build_meta, never by enrich_record); dedup matches by zotero_key then slug (incl. repo-root book dirs) and enriches-not-overwrites; --prune only deletes records that have zotero_collections and whose key is no longer in the live fetch.
Rule: A re-runnable importer should (a) key off a stable upstream id, (b) enrich rather than overwrite on match, and (c) stamp every record it creates with a provenance field so destructive cleanup can target only its own output.

Lesson 6: A "curated" Zotero collection is still noisy — filter, flag, don't trust

Category: discovery
Context: Reviewing the 348 ingested records.
Symptom: ~36% of records carried an emoji plugin-artifact tag (⛔ No INSPIRE recid found); several off-topic hard-engineering papers (reverse-Brayton exergy, phase-change roof insulation, hybrid-solar exergy) were mis-filed in Bucky Shelf via an INSPIRE-HEP plugin; some title fields held an editor credit / blurb / participant roster instead of a title; standalone attachment items would have become empty stubs. Separately, live-URL bodies captured page nav/boilerplate because the fetcher is a plain HTML tag-strip, not a readability extractor.
Root cause: The upstream library accumulated plugin cruft, mis-files, and bad metadata; the wiki ingest faithfully reproduced it.
Fix: Drop tags not starting with an alphanumeric (_is_clean_tag); skip attachment/note item types (SKIP_TYPES); manually correct malformed titles while preserving original_zotero_title; off-topic items removed upstream in Zotero (the proper fix), not patched in the wiki.
Rule: Treat any externally-curated corpus as noisy input — sanitize known artifacts, surface off-topic/malformed items for human review, push true fixes upstream, and remember tag-stripping ≠ readable full text.

Lesson 7: On a large content repo, pluck isolates via an in-place branch, not a worktree

Category: correction
Context: The pluck workflow's default is a dedicated git worktree.
Symptom: The working tree is ~38 GB (books/ebooks), so a second worktree would duplicate tens of GB and be slow.
Root cause: Worktree isolation assumes a modest checkout; it doesn't fit a large media/content repo.
Fix: Used the documented fallback — an in-place feature branch (git checkout -b pluck/imp-5-…) — skipping the worktree-only setup/verify steps, and squash-merged in place.
Rule: For large content repos, prefer an in-place feature branch over a full second worktree; reserve worktrees for code repos where parallel checkouts are cheap.