BUCKYVERSE

Improvements

Improvement ideas and feedback: changes that would make the product or workflow better, from reviews, users, or conversation. Use improve: or feedback:. Move sections to ITERATION.md when scheduling work.

Last id: IMP-21

Re-migrate catalog epubs to current codexis template (purple numbers + MathML)

Id: IMP-21

Date: 2026-07-04

Why: 58 of 60 readable books lack the compile-time section.paragraph purple
numbers (and true MathML). Their epubs predate the current codexis pipeline, so
the reader falls back to bare sequential per-chapter numbers (1,2,3, id=pn1)
instead of canonical numbers (Grunch: data-pn="3.1" id="3.1"). The leading
section number is a document-wide counter from the LaTeX structure (weaves
front/back matter, skips stub pages) that the reader CANNOT reconstruct from the
epub alone — so this is not a reader bug and must not be faked reader-side
(synthesized numbers = wrong references + permalinks that break on later
migration). Same root cause as the MathML gap (synergetics-epub-missing-mathml).

Description: Program (not a one-off): each book is its own codexis project
(elib-*) needing a legacy→current-template migration via the
codexis-template-migrate skill, which enables BOTH the purplenumbers package
(opt-in per book in config.yaml) and MathML (\Preamble{xhtml,mathml,…}). Per
book: migrate → codexis build --format epub → re-import to
<slug>/<slug>.epub → pnpm --filter reader build → wrangler pages deploy.
The reader is already correct and needs no change (trusts compile-time purple
numbers when present, falls back otherwise; Grunch proves the 3.1 right-margin
layout renders). First instance in flight: Synergetics (elib-synergetics IMP-1).
Sub-task 1 = inventory which of the 58 slugs have a ready codexis project vs need
one created. Detection: in apps/reader/dist, a migrated book has
data-pn="N.M" (dotted); a non-migrated one only bare-integer data-pn.

Inventory (2026-07-04) — codexis project availability for the 60 readable books:

Reshaped cost model: only 8/60 have a project (2 done, 6 migrate-able); 52/60
need a codexis project created before any purple-number/MathML work. Naming
convention: reader slug foo_bar ↔ repo elib-foo-bar under ~/Projects.

Pilot result (2026-07-04, cosmography) — "migrate-able" ≠ "mechanical". Set
purple_numbers: true + codexis build --format epub: the epub came out with 0
purple numbers AND 0 MathML (still 27 math SVGs). Root cause: the tex4ht/htlatex
run errored (Missing \begin{document} on a generated glossary) and codexis
masks it ("exited non-zero but produced an EPUB; treating as success"), so the
purple anchors (stamped during that run) never emit and the epub is degraded.
Codexis IS mathml+purple capable; the blocker is a clean tex4ht compile per book.
So each of the 6 needs real per-book migration (fix compile → green tex4ht build →
verify dotted data-pn + <math> → re-import), not a config flip — same as
Synergetics, best done per-repo with the codexis-template-migrate skill.
(cosmography config.yaml has purple_numbers: true set but UNCOMMITTED pending
a working build.)

Add an accession record for each collection item

Id: IMP-2

Date: 2026-05-09

Description: Add an accession record for each item in the collection so we can track details about each item.

Find digital receipts for each collection item

Id: IMP-3

Date: 2026-05-09

Description: Find digital receipts for each item in the collection so purchase and acquisition details are preserved.

Chrome extension to ingest items into the llm-wikis

Id: IMP-4

Date: 2026-06-28

Why: Capture sources directly from the browser into the wikis' ingest pipelines, instead of manually downloading files and dropping them into each repo's inbox — lowering the friction of adding sources across multiple knowledge bases.

Description: Build a Chrome extension that ingests items (pages, PDFs, selections, links) into the LLM wikis maintained across projects — e.g. buckyverse, jouleheist, etc. Should target the multiple wikis (likely via a selectable destination), feeding the same ingest-by-reference / raw-source flow these wikis already use.

Use readability extraction for Zotero ingest live-URL bodies

Id: IMP-6

Date: 2026-06-30

Why: During the IMP-5 Zotero ingest, live-URL fetching captured page nav/menus/footer chrome rather than clean article text, because zotero/ingest_to_wiki.py uses a plain HTMLParser tag-strip. Cleaner bodies make the raw sources far more useful for compile/query.

Issue: default_url_fetcher + extract_text_from_html strip tags but keep all visible text, so fetched ## Page content sections include boilerplate (e.g. the Macmillan "On Film-making" record is mostly site navigation).

Suggestion: Replace the tag-strip with a readability-style main-content extraction (e.g. a readability port, or heuristic largest-text-block selection) for fetch_status: ok records; optionally re-fetch existing noisy records.

Reconcile Zotero ingest slug-exact dedup with existing book dirs

Id: IMP-7

Date: 2026-06-30

Why: The IMP-5 Zotero ingest dedups by exact slug, so books that already exist as repo-root source dirs but under a subtitle-variant slug (e.g. Zotero "Utopia or oblivion: the prospects for humanity" → utopia-or-oblivion-the-prospects-for-humanity vs the repo dir utopia_or_oblivion/) were created as new raw/repos/*.md pointer records instead of matched — producing some redundancy with existing book sources.

Issue: scan_existing/resolve_action in zotero/ingest_to_wiki.py match slugs exactly; subtitle/edition differences defeat the match, so ~150 new repo pointers were created, some duplicating books already represented as root dirs.

Suggestion: Add fuzzy/normalized title matching (e.g. compare on title prefix before the first colon, or token-overlap against existing repo slugs and root book dirs) so near-duplicate books enrich the existing record instead of creating a new pointer; review the existing duplicates and reconcile.

Create updates/ and publish weekly summaries of books, articles, features, and changes

Id: IMP-10

Date: 2026-07-01

Why: buckyverse grows continuously (new book/article sources, wiki articles, features, page changes) but there is no running record of "what's new." A weekly summary gives readers a reason to return, surfaces recent additions that would otherwise be buried in a 23k-page wiki, and doubles as a changelog/newsletter for the project.

Issue: There is no updates/ section and no periodic digest; recent activity is scattered across the Wiki.js database (new/updated pages) and the git repo (commits, new source dirs, .wiki/ compiles) with no consolidated, reader-facing summary.

Suggestion: Create an updates/ area (repo-root pages + Wiki.js publish, mirroring the book-page pattern) and publish weekly summary posts covering new books, articles, features, and notable changes. Determine each week's content options from BOTH sources: the Wiki.js database (pages created/updated in the window — via the existing dev/wikijs tooling/GraphQL) and the git repo (commits, new root source dirs, .wiki/log.md ingest/compile entries) in the window. Draft the digest from that combined signal, then curate/publish. Consider automating the data-gathering step and a template for the post.

Add a "read in Bucky's voice" option to book pages

Id: IMP-11

Date: 2026-07-01

Why: An audio option that reads a page aloud in Buckminster Fuller's own synthesized voice would make the texts far more engaging and accessible — a distinctive, on-brand way to experience the corpus (audiobook-style, but in Bucky's voice), and a natural payoff for the voice-synthesis research already scoped.

Issue: Book/source pages are text-only; there is no way to listen to a page, let alone hear it in Fuller's voice.

Suggestion: Add a control on book pages (Wiki.js) that plays the page text as speech in a synthesized Bucky voice. Build on the TODO-6 Fuller voice-synthesis recipe (ACC-2, .wiki/output/projects/fuller-voice-synthesis/recipe.md): generate audio from page text (pre-rendered per page or on demand), host the audio, and add a play button/widget to the page template. Respects the recipe's ethics/rights stance (BFI permission + label audio as synthetic). Decide pre-generation vs. real-time TTS and where audio is stored/served.

Organization entity pages — Batch 2: publishers, agencies, and the long tail

Id: IMP-15

Date: 2026-07-01

Why: IMP-13 covered ~11 corpus-significant institutions, but many organizations that recur across the corpus still lack a node — completing coverage makes the organizational axis fully navigable alongside the people layer.

Issue: Organizations have no seed collection; the remaining ones (publishers, agencies, companies, minor institutions) live only in scattered corpus mentions and were capped at ~11 for Batch 1.

Suggestion: Continue the entity-organization model from IMP-13: create pages for the long tail — publishers (Garland, MIT Press, Princeton Architectural Press, and SIU Press if split from the university), agencies (NASA, the DEW-line/military dome programs), companies, and one-mention institutions not covered by Batch 1 — each with a description, "Role in Fuller's world", and bidirectional links, added to the fuller-organizations.md registry. Reuse the Batch-1 spec/plan (docs/superpowers/specs/2026-07-01-organization-entity-pages-design.md). With people (IMP-8/9, IMP-12, IMP-14) and organizations complete, the entity model spans the whole corpus.

Fork Wiki.js (or patch-overlay) to land pending page-render / tree-rebuild performance PRs

Id: IMP-20

Date: 2026-07-02

Why: Wiki.js rebuilds the entire page tree (O(all pages)) on every page create/update. On our ~23k-page site that makes bulk writes ~60s/page — measured during the llm-wiki knowledge/ projection, where a full ~1,068-page mirror extrapolates to ~17–18 hours and intermittently slows the live site during the run. Faster mutations would help not just bulk imports but everyday editing/publishing latency for readers and authors. Community PRs addressing render/tree performance reportedly exist but sit unmerged upstream (no maintainer to accept/implement them).

Issue: We run the stock ghcr.io/requarks/wiki:2 image; the per-mutation tree rebuild + render cost is a fixed upstream bottleneck we can't tune via config or GraphQL. Wiki.js 2.x is effectively in maintenance and Wiki.js 3 remains unreleased, so the pending perf improvements aren't reaching us.

Suggestion: Evaluate landing the pending performance work ourselves. Options in increasing order of commitment: (1) minimal patch overlay — cherry-pick just the tree-rebuild/render PRs into a thin Docker image layer over requarks/wiki:2 (least maintenance, no full fork); (2) maintained fork — fork Wiki.js, merge the perf PRs, build/host our own image (more control, but inherits the upstream security-patch/dependency burden); (3) wait for / migrate to Wiki.js 3. Note the safe one-time workaround already exists for bulk imports (scoped GraphQL projection run in off-hours phases), so this is about ongoing performance, not just the current migration.

Research findings (2026-07-02): Root cause is Page.rebuildTree() in server/models/pages.js — a full O(all-pages) tree rebuild run synchronously after every create/update/delete (≈60s/page at our 23k-page scale). Three upstream PRs address it, all touching the single file server/models/pages.js and all against main (= the 2.x line our requarks/wiki:2 image is built from), which makes a patch overlay very tractable:

Recommended next step: build a Docker image FROM ghcr.io/requarks/wiki:2 that patches server/models/pages.js with #7780, deploy to a staging/local Wiki.js, and benchmark create/update latency vs. the ~60s/page baseline; keep #6082's background approach as the fallback patch.

Benchmark result (2026-07-02): Confirmed against the real 45,937-page prod dump by replaying the exact rebuild-tree.js assembly loop and timing #7780's incremental lookups. Stock rebuildTree = ~56,000 ms per page write (O(n²) _.find scan — matches the ~60s/page seen live). #7780 = ~40 ms/write (~1,400×), or ~1 ms with a (localeCode,path) index the PR omits (~56,000×). Drops the full ~1,068-page projection from ~17h to a couple of minutes and fixes site-wide edit latency.

Image built + correctness-tested (2026-07-02): Patched image buckyverse/wiki:2-perf7780 builds cleanly on the droplet (patch applies to the exact live 2.x image). A correctness harness (dev/wikijs/perf-7780/harness.js) drove #7780's methods against the real 46k-page copy and diffed the result against a full rebuildTree() after every op — found two bugs that make #7780 undeployable as-written: (1) fatal — pageTree.id has no sequence in stock Wiki.js, so addPageToTree's id-less insert throws on every create/move and #7780's catch silently falls back to the slow full rebuild (→ zero speedup for our all-create workload); (2) cleanupEmptyFolders compares pg's string count to a number ('0' === 0), so emptied folders are never removed (orphan folders accumulate). Both fixed (pagetree-id-sequence.sql migration + a one-line numeric fix in 7780-fixed.patch) → harness ALL PASS. Artifacts + runbook under dev/wikijs/perf-7780/.

DEPLOYED + VERIFIED LIVE (2026-07-02): After a detour (the first deploy attempts were masked because the true write bottleneck was Wiki.js git storage, not the tree — see DEBT-2; once git storage was disabled, the patch's benefit became visible), the patched image is live behind Caddy and creates dropped from ~56s to ~3.5s, deletes to ~0.3s (incremental path, no fallback, id floored at 100M per the corrected pagetree-id-sequence.sql). That's ~16× — the full ~1,068-page knowledge/ projection went from 17h/3h estimates to ~1 hour. Two prerequisites, both now baked into the runbook: (1) the pageTree.id sequence must be floored high (100M) or rebuildTree desyncs it and every create collides→falls back (the bug that made early attempts look like failures); (2) the site must not be git-storage-bound. Caddy TLS termination (DEBT-1) makes the wiki-container swap cert-neutral, so this deploy is low-risk and one-line reversible.