Synergetics Dictionary Loader — Design (Sub-project 3)

Date: 2026-07-06
Status: Approved for planning
Sub-project: SP3 of the Wiki.js → Astro migration (docs/superpowers/specs/2026-07-06-wikijs-to-astro-migration-program-design.md). SP1 (reader) + SP2 (library/home/downloads) are live at future.buckyverse.org.
App: apps/reader/ (Astro 5 + pnpm, deploy via wrangler pages deploy).

1. Goal & success criteria

Bring the Synergetics Dictionary — ~21,188 cross-referenced cards — onto the Astro site as a browsable, searchable, deep-linkable section, at full parity with what Wiki.js serves today, while staying fully static and under Cloudflare Pages' 20,000-file limit.

Done when:

• Every card is reachable and readable on future.buckyverse.org.
• A card deep link (/dictionary/s#C18234) resolves to the right card.
• Card↔card cross-references resolve to real deep links.
• Pagefind returns individual cards as search results (not just the letter page).
• Card figure images render (served from R2, not the Pages deploy).
• The build stays fully static (no server/edge runtime) and well under 20k files.
• A redirect map from the old Wiki.js dictionary URLs is defined (activated at SP6).

2. Source of truth & the import artifact

• Canonical source: the elib-synergetics-dictionary repo's SQLite DB (data/synergetics_dictionary.db; git-synced SQL dump in data/git-sync/). Generated per-card markdown lives at output/generated/markdown/cards/<letter>/C#####.md; the Wiki.js sync (scripts/sync_cards_to_buckyverse.exs) pushes those to synergetics-dictionary/<letter>/<id>.
• Import artifact (chosen): the dict repo exports per-letter JSON shards (24 files, one per letter-group) from the DB — each card as structured data: { id, term, letter, html, crossRefs: [{id, label, date?}], prevId?, nextId?, imageId?, tags: [] }. These ~24 files are imported into buckyverse (true to "buckyverse is an import sink"; avoids 21k card files in the repo; structured and reproducible). The export script is owned by the dict repo (small addition).
• Fallback source: if we choose not to add the export step, dictionaryLoader consumes the existing generated per-card markdown from the dict repo directly. The loader contract (§4) is written so either source produces the same collection.

3. Card model (from the source)

A card carries: id (C#####, immutable, global), term (title, e.g. "Zero = Normal"), letter (one of 24 groups: a–w plus a combined xyz), definition html (verbatim RBF text with citations), crossRefs (links to other card ids, with optional dates), prevId/nextId (dictionary reading order), an optional figure imageId (image at images/cards/<letter>/<id>.jpg), and tags.

Distribution: 24 letter-groups, 21,188 cards. Largest: s 2,293, c 1,747, t 1,638, p 1,593; smallest j 46.

4. Architecture — one loader, grouped static pages

dictionaryLoader (apps/reader/src/loaders/dictionaryLoader.ts), a new Astro content-layer collection sibling to bookLoader/catalogLoader:

• Reads the 24 JSON shards.
• Emits one collection entry per letter-group (24 entries), each holding its ordered cards with rewritten links (§5).
• Pure card→HTML shaping lives in a testable module (src/lib/dictionary/), no Astro coupling — mirrors the src/lib/epub/ pattern.

Routes:

• src/pages/dictionary/index.astro — A–Z landing: letter nav, search entry, links to front-matter pages.
• src/pages/dictionary/[letter].astro — one static route per letter-group (24). Renders every card in the group as a <section id="C#####"> anchor.

Page-weight mitigation (large letters): each card <section> uses content-visibility: auto with a contain-intrinsic-size estimate, so offscreen cards cost ~nothing to render. The heaviest page (s, 2,293 cards) stays a single static file but renders responsively on mobile. Images are loading="lazy".

File budget: 24 letter pages + 1 index + ~10 front-matter pages ≈ ~40 files added to the deploy (currently 8,545). Comfortably under 20k with headroom as books grow.

5. URL model, cross-links, redirects

• /dictionary/ — landing (A–Z + search + front-matter links).
• /dictionary/<letter> — letter-group page.
• Card deep link: /dictionary/<letter>#<id> (e.g. /dictionary/s#C18234). Stable: a term's letter is stable; the anchor is the immutable card id.
• Front-matter pages (How-to-Use, Sources-Cited, Preamble, Contents-of-the-Set, Some-Perspectives, Annotated-Bibliography, …) — currently buckyverse root pages — re-homed to /dictionary/<slug>.
• Cross-link rewriting (loader): /synergetics-dictionary/<letter>/<id>.md → /dictionary/<letter>#<id> (same-page anchor when within the letter; cross-letter jump otherwise). prev/next become anchor/cross-page links in dictionary order.
• Redirect map (defined here; activated at SP6 cutover):
  • synergetics-dictionary/<letter>/<id> → /dictionary/<letter>#<id>
  • synergetics-dictionary/<letter>/index → /dictionary/<letter>
  • front-matter Wiki.js paths → their /dictionary/* equivalents
  • The existing A.md/B.md term-index root pages fold into /dictionary/<letter>.

6. Images → R2

The ~21,193 card figures do not enter the Pages deploy. They are offloaded to R2 (the existing downloads/assets bucket pattern) and referenced by URL, loading="lazy". A one-time upload of images/cards/** to R2 is part of SP3; the loader emits imageId → R2 URL. This removes ~21k files from the deploy on its own.

7. Search — per-card Pagefind results

Pagefind indexes the built letter pages, but must return individual cards, not "the S page." Each card <section> is marked as its own Pagefind result body (data-pagefind-body on the section, with a per-card result title/meta = the term + id) so a query surfaces the specific card and links to its #<id> anchor. This is the primary search-quality tuning of SP3.

8. Scope

In: the 24 letter-group card pages, /dictionary/ index, the dictionary front-matter pages, cross-link rewriting, per-card Pagefind, card images→R2, the redirect map (definition), and (if chosen) the dict-repo JSON export script.

Out: cross-references from cards into book passages or .wiki/ concepts (that inter-corpus linking is SP5); the Wiki.js sync script retirement (SP6); dictionary content editing/review (that stays in the dict repo).

9. Testing

• Unit (src/lib/dictionary/*.test.ts): shard/markdown → cards (assert per-letter counts match §3, cross-refs parsed, prev/next linked, image ids mapped, link rewriting /synergetics-dictionary/... → /dictionary/<letter>#<id>).
• Build smoke: 24 [letter] routes + /dictionary/ index generated; a known card #C21002 anchor present in output; a known cross-link rewritten; a deep link resolves.
• Search: Pagefind returns a per-card sub-result for a known term.
• Visual/perf: the heaviest letter (s, 2,293 cards) renders and scrolls smoothly on mobile + desktop (content-visibility verified); a card image loads from R2.

10. Risks

• Page weight on big letters — mitigated by content-visibility:auto + lazy images; measure s/c/t HTML size and mobile scroll perf early; if a single letter is still too heavy, shard that letter by a stable sub-key (e.g. first two term characters) rather than position-based pagination.
• Pagefind sub-result config — per-card result granularity needs the right data-pagefind-* wiring; validate against a query before declaring done.
• Source drift — the JSON export must stay in lockstep with the DB; the export script is owned by the dict repo and re-run on dictionary updates (a parallel content track, like .wiki/ ingest — does not gate).
• R2 image upload volume (~21k files) — a one-time bulk upload; script it and verify a sample resolves.

11. Definition of done

/dictionary/ is live on future.buckyverse.org: all 21,188 cards browsable by letter, deep-linkable by #<id>, cross-refs resolving, images loading from R2, Pagefind returning individual cards, the build fully static and ~40 files heavier, and the Wiki.js→Astro dictionary redirect map written for SP6.