Knowledge Layer Loader — Design (Sub-project 4)

Date: 2026-07-06
Status: Approved for planning
Sub-project: SP4 of the Wiki.js → Astro migration (docs/superpowers/specs/2026-07-06-wikijs-to-astro-migration-program-design.md). SP1 (reader), SP2 (library/home/downloads), SP3 (dictionary) are live at future.buckyverse.org.
App: apps/reader/ (Astro 5 + pnpm, deploy via wrangler pages deploy).

1. Goal & success criteria

Render the compiled llm-wiki knowledge layer (.wiki/wiki/ — topics, concepts, references) as a browsable, searchable, cross-linked /knowledge/* section of the static Astro site, at parity with what the Wiki.js projection serves — rendering whatever content exists today (not gated on ingest completeness).

Done when:

• All 378 articles (topics/concepts/references) are reachable and readable at future.buckyverse.org/knowledge/....
• Intra-knowledge links resolve: [[slug|label]]/[[slug]] wikilinks and relative .md links become real /knowledge/ links; unresolved wikilinks degrade to plain text (no dead links).
• A $…$ math expression renders via KaTeX.
• Pagefind indexes knowledge articles alongside books + dictionary.
• The landing links out to the relationship graph at graph.buckyverse.org.
• The build stays fully static and the added file count is ~382 (no 20k concern).

2. Source & the canonical URL scheme (reuse the projector)

• Source of truth: .wiki/wiki/ under the buckyverse repo — subtrees topics/ (335), concepts/ (17), references/ (26); theses/ is empty. Each file is markdown with rich frontmatter (title, category, sources, created, updated, tags, aliases, confidence, volatility, verified, compiled-from, summary).
• Out of scope: .wiki/raw/ (provenance/ingest records), .wiki/inventory/ (tracking records — only 2 reference pages link into it), .wiki/datasets/, and _index.md files. Cross-corpus deep links (knowledge→book-passage, knowledge→dictionary-card) are SP5.
• Canonical URLs come from the existing projector dev/wikijs/scripts/project-llm-wiki.exs, which already maps .wiki/ → /knowledge/<category>/<slug> and rewrites links. SP4 ports that logic to TypeScript and mirrors the scheme, so Astro URLs equal the projector's Wiki.js URLs (near-identity redirects at SP6). Namespace = knowledge; map_path yields knowledge/<category>/<slug> (e.g. knowledge/topics/synergetics).

3. Architecture — port the projector to a static loader

Pure, tested TS modules in apps/reader/src/lib/knowledge/ (mirrors src/lib/dictionary/):

• mapPath.ts — wiki/<cat>/<slug>.md → knowledge/<cat>/<slug>. Ports map_path/flatten_wiki/index_rename (strip wiki/, drop .md, _index→index).
• slugIndex.ts — build a global Map<slug, "/knowledge/<cat>/<slug>"> from all discovered articles; replicate the projector's collision dedupe (on two sources → same path, keep the shallower; warn).
• rewriteLinks.ts — ports rewrite_links:
  • [[slug|label]] and [[slug]] → [label](/knowledge/…) when slug is in the index, else the plain label/slug text (no dead link).
  • Relative ](target.md) → resolve target against the current file's dir, then branch on where it lands: inside wiki/ → map_path → ](/knowledge/…); into raw/ or inventory/ (provenance/tracking, not rendered) → strip the link, keep the link text only; into a repo book/source dir (e.g. ../../../<book_slug>/) → citeSource → /read/<slug> when that book is hosted, else text. Absolute/http(s)//-prefixed links are left untouched (cross-corpus deep links into book passages/dictionary cards are SP5).
• renderArticle.ts — markdown→HTML via markdown-it (already a dep) with a KaTeX plugin for $…$/$$…$$, then rewriteLinks. Returns { html }. Extracts nothing else (metadata comes from frontmatter in the loader).

knowledgeLoader (src/loaders/knowledgeLoader.ts): walk .wiki/wiki/{topics,concepts,references}/**/*.md (exclude _index.md), parse frontmatter, build the slug index (first pass), then emit one knowledge collection entry per article:
{ slug, category, path, title, summary, tags, updated, confidence, volatility, aliases, html, sources }.

Collection (src/content.config.ts): add knowledge with a zod schema matching the entry.

4. Routes & pages

• src/pages/knowledge/index.astro — landing: intro, three category sections (topics/concepts/references with counts + browse links), a prominent link to https://graph.buckyverse.org, Pagefind search entry.
• src/pages/knowledge/[category]/index.astro — category index: A–Z list of that category's articles (title + summary snippet).
• src/pages/knowledge/[category]/[slug].astro — article: title, summary lede, a light metadata line (updated <date> · <confidence> · <volatility> as subtle badges), then the body html (which already contains the article's ## Sources section — its links are handled during rewriteLinks per §3), and tags. Marked data-pagefind-body. Uses SiteLayout.

Citations helper (src/lib/knowledge/citeSource.ts): called by rewriteLinks (§3) for relative links that resolve into a repo book/source dir — maps to /read/<book-slug> when the source is a hosted book (present in books.config/catalog), otherwise returns plain text. Best-effort; unmatched → text. (Deep cross-corpus linking is SP5.)

5. Math (KaTeX)

Only 2 articles use inline $…$. Add a KaTeX markdown-it plugin (e.g. @vscode/markdown-it-katex or markdown-it-katex) in renderArticle, and include KaTeX CSS once in SiteLayout (self-hosted CSS asset, no external CDN — CSP-safe). Rendered at build; no client JS required.

6. Design system

Reuse SiteLayout + the reader's theme/typography so /knowledge/* is visually one site with /read and /dictionary. Article body reuses the readable measure (~72ch).

7. Redirects (for SP6)

Because Astro /knowledge/<cat>/<slug> mirrors the projector's Wiki.js paths, the map is a passthrough. Add to public/_redirects at SP6:

• If the live projection served under /en/knowledge/..., an /en/knowledge/:cat/:slug → /knowledge/:cat/:slug 301 rule (Cloudflare) — otherwise identity, no rule needed.
• Any legacy pre-projection knowledge URLs → their /knowledge/ equivalents (enumerated at cutover from the live page list).

8. Scope

In: 378 articles (topics/concepts/references) as static pages; landing + 3 category indexes; ported link/path/slug logic + tests; KaTeX; light metadata + tags; ## Sources citations with best-effort /read linking; graph link; Pagefind.

Out: raw/, inventory/, datasets/, _index.md, empty theses/; cross-corpus deep links (SP5); rebuilding the relationship graph (link only); editing/ingesting .wiki/ content (ongoing parallel track).

9. Testing

• Unit (src/lib/knowledge/*.test.ts), porting the projector's cases:
  • mapPath: wiki/topics/synergetics.md → knowledge/topics/synergetics; _index handling.
  • slugIndex: builds {slug→path}; collision dedupe keeps the shallower + warns.
  • rewriteLinks: [[synergetics|Synergetics]] → [Synergetics](/knowledge/topics/synergetics) when indexed; unknown [[ghost]] → plain ghost text; relative ](../topics/x.md) resolves; http(s)/absolute untouched.
  • renderArticle: a $x^2$ renders to KaTeX HTML (<span class="katex">).
  • citeSource: a hosted-book source → /read/<slug>; an unknown source → text.
• Build smoke: 378 article routes + 3 category indexes + landing generated; a known article contains a resolved /knowledge/... link; a known math article contains katex markup; find dist -type f | wc -l grows by ~382, well under 20k; Pagefind indexes /knowledge/.
• Live: deploy; /knowledge/, /knowledge/topics/synergetics, a category index, and a graph link all resolve (200); the graph link points at graph.buckyverse.org.

10. Risks

• Link-rewrite fidelity vs the Elixir projector — highest risk; mitigate by porting the exact regexes (@wikilink_labeled/@wikilink_bare/@relmd) and its test cases, and diffing a sample of rendered links against a projector dry-run.
• Slug collisions across categories — replicate the projector's dedupe; a build-time warning lists any collisions.
• ## Sources→/read mapping accuracy — best-effort; unmatched sources render as plain text (never a broken link).
• KaTeX asset/CSP — self-host KaTeX CSS (no CDN) to satisfy the site's CSP; verify the 2 math articles render.

11. Definition of done

/knowledge/ is live on future.buckyverse.org: all 378 articles browsable by category and cross-linked (wikilinks + relative links resolved, dead links degraded to text), math rendered, sources cited (with /read links where hosted), the graph linked, Pagefind covering knowledge, the build fully static and ~382 files heavier, and the near-identity redirect approach recorded for SP6.