BUCKYVERSE

Design: Ingest Buckyverse Zotero collections into the wiki

Date: 2026-06-30
Source item: IMP-5 (work/IMPROVEMENTS.md) — "Ingest all items in Zotero collections for Buckyverse"
Status: Approved (brainstorming)

Goal

Pull every item in the Buckyverse Zotero group's curated collections into the
active llm-wiki ingest pipeline (.wiki/raw/**) as raw source records, so the
collection enters the knowledge base and can be compiled and queried. The job
must be idempotent — safe to re-run as the Zotero library grows.

Decisions (from brainstorming)

Architecture

A single reusable Python script, zotero/ingest_to_wiki.py, run from the repo
root. Pure standard library where possible (urllib for the local API). It is the
only new moving part; it reads Zotero and writes .wiki/raw/**.

Pipeline stages (each independently testable):

  1. Fetch — for each target collection, GET …/collections/<key>/items/top
    (paginated, format=json, Zotero-API-Version: 3). Merge and dedupe by
    item key    . Record which collection(s) each item came from (for tagging
    Wish List items). Write the merged set to zotero/buckyverse_collections.json
    as a committed audit snapshot.
  2. Classify — map each item's itemType to a wiki raw type and target path
    (table below).
  3. Resolve dedup — decide create / enrich / skip against existing .wiki/raw
    records and repo-root book dirs.
  4. Enrich (optional fetch) — pull attachment text and/or live URL content.
  5. Write — emit/patch the raw markdown record with frontmatter.
  6. Index + report — update _index.md files and print a summary.

A --dry-run flag runs stages 1–4 and reports planned actions without writing
any .wiki/raw files or the snapshot.

Type → raw mapping

Zotero itemType raw type path
book, bookSection repos raw/repos/<slug>.md (by ref)
journalArticle, report papers raw/papers/<slug>.md
webpage, blogPost, newspaperArticle, magazineArticle, document, radioBroadcast, tvBroadcast articles raw/articles/<ingested>-<slug>.md
encyclopediaArticle (incl. People collection) articles raw/articles/<ingested>-<slug>.md
videoRecording, film notes raw/notes/<slug>.md

Unknown/other types fall back to articles and are flagged in the report.

Frontmatter

Standard .wiki fields plus Zotero provenance:

---
title: "<item title>"
source: "<best URL: item url, else DOI link, else zotero://select/...>"
type: <repos|papers|articles|notes>
ingested: <YYYY-MM-DD>
tags: [<authorship-tag>, <zotero tags…>, <wish-list if applicable>]
summary: "<abstractNote, trimmed; else generated one-liner>"
zotero_key: <item key>           # provenance + idempotency anchor
zotero_collections: [<names>]    # which collections it came from
held: true|false                 # false for Wish List items
isbn: "<ISBN>"                   # books, when present
doi: "<DOI>"                     # papers, when present
fetch_status: ok|metadata-only|failed
---

authorship-tag follows the repo convention: buckminster-fuller (By Bucky
collection), else about-buckminster-fuller / fuller-adjacent by best guess
from creators/title (default about-buckminster-fuller).

Dedup & idempotency

Match an incoming item to an existing record in this order:

  1. By zotero_key found in any existing .wiki/raw/** frontmatter.
  2. By slug against existing raw/repos/*.md and repo-root book dirs
    (normalized: lowercase, punctuation stripped, spaces→-).

Re-running the whole script therefore creates zero new files on an unchanged
library (idempotency is the primary verification signal). Books that already
exist as repo-root sources are enriched in their raw/repos/<slug>.md pointer,
not re-ingested.

Full-text fetch

Books-without-local-copy (most By-Bucky/Books items not already a repo dir) get a
raw/repos/<slug>.md reference record with held reflecting ownership and a note
that no local file exists; they are not forced to fetch anything.

Indexes & report

Error handling

Verification

This is a content repo with no unit-test suite; verification is behavioral:

  1. --dry-run — review planned create/enrich/skip counts for sanity.
  2. Real run — generates records + snapshot + indexes.
  3. Idempotency: immediate second run reports 0 created (only re-confirms).
  4. Spot-check 3–5 generated records across types (a book, a People encyclopedia
    article, a webpage with fetched content, a Wish List item).
  5. /wiki:lint passes (no broken indexes/links introduced).

Out of scope