Lessons Learned: Buckyverse wiki migration & ingest tooling

Extracted from the 2026-06-27 session that migrated the legacy llm-wiki/ into .wiki/, compiled ~80 book articles, tagged by authorship, and added the first dataset manifest. 7 lessons covering frontmatter tooling, sandbox quirks, layer boundaries, and provenance.

Lesson 1: Parse YAML frontmatter line-based, never with DOTALL or substring split

Category: gotcha
Context: Bulk-editing the sources: block in 78 topic articles with a regex, and parsing frontmatter with split("---", 2).
Symptom: All 78 articles had their frontmatter silently stripped to 3 fields (title/category/sources); separately, climate---architecture.md was mangled (its body folded into frontmatter).
Root cause: Two bugs. (a) A regex (?ms)^sources:\s*(\n(?:\s*-\s+.*\n?)*) used the s/DOTALL flag, so .* matched across newlines and ate every field after sources:. (b) split("---", 2) treated the --- inside the slug climate---architecture as a frontmatter delimiter.
Fix: Recovered the lost fields from the pre-damage _index.md (summary/confidence) plus the uniform fields, then regenerated tags. Switched all frontmatter handling to a line-based parser: find the closing --- as a whole line (lines.index("---", 1)), edit only the target key's lines.
Rule: Parse/split YAML frontmatter by matching --- only at line boundaries (^---$); never use a DOTALL .* over a frontmatter block, and never str.split("---") — slugs and bodies contain ---.

Lesson 2: cat and heredocs are blocked in this sandbox — use printf or Python

Category: gotcha
Context: Generating many _index.md files with cat <<EOF heredocs.
Symptom: mkindex:2: command not found: cat — files created empty.
Root cause: This environment blocks cat (and the prompt warns the same for head/tail/sed/awk). The redirect still created a 0-byte file, masking the failure.
Fix: Rewrote file generation with printf and, for anything structured, Python (open(p,"w").write(...)).
Rule: In this sandbox, never build files with cat/heredoc; use printf, the Write tool, or Python — and verify byte counts, since a failed cat >file silently leaves an empty file.

Lesson 3: Hundreds of row-like records belong in a dataset manifest, not individual inventory files

Category: pattern
Context: Migrating 365 collection records, then the ~20,900-card Synergetics Dictionary.
Symptom: 365 inventory files are slow to list and bloat context; the dictionary (63K files, ~3 GB) is impossible to ingest as articles.
Root cause: inventory/ is a small tracking layer; inventory.md's fit-check says hundreds of row-like items are "too big for inventory."
Fix: The user opted for 365 inventory files anyway (their call), but the dictionary correctly became the wiki's first datasets/ manifest — wiki as index, data stays external.
Rule: For collection-scale, homogeneous, row-like data (hundreds+), create one datasets/ manifest (+ optional corpus inventory record), not per-item files; surface the boundary explicitly and let the user override.

Lesson 4: In a Wiki.js content repo, ingest complete books by reference — don't copy into raw/

Category: correction
Context: Migrating ~80 Fuller books that live as directories in the repo root.
Symptom: Default ingest would copy book text into raw/, duplicating content the Wiki.js site already serves from the root.
Root cause: This repo is a Wiki.js site; book dirs and ebook formats are served from the root and must stay there.
Fix: Ingest-by-reference — a thin raw/repos/<slug>.md pointer (type: repos, source: → ../../../<book_dir>/, tagged ingest-by-reference) plus a compiled wiki/topics/<slug>.md; nothing copied. Documented in config.md.
Rule: A complete book ingests differently from an article: book → raw/repos pointer (type repos, kept in root, not copied) + wiki/topics overview; article → full copy into raw/articles. Skim large books (index + representative chapters), don't flatten.

Lesson 5: For bulk frontmatter edits, have subagents return JSON and apply changes deterministically

Category: pattern
Context: Regenerating tags/aliases and classifying authorship across 78 articles after the frontmatter-stripping bug.
Symptom: Letting code/agents rewrite frontmatter directly had already caused mass data loss (Lesson 1).
Root cause: Free-form frontmatter mutation is fragile; structured-data round-trips are not.
Fix: Subagents returned strict JSON ({slug, tags, aliases} / {slug, klass}); a deterministic line-based applier wrote the changes. Reserved agent direct-writes for whole-file prose (compiling articles), where there's no fragile in-place edit.
Rule: Split bulk metadata work — agents produce structured JSON, you apply it with a deterministic parser; only let agents write whole files when they're authoring prose, not patching frontmatter.

Lesson 6: When migrating bullet-field records, only treat a value as a path if it looks like one

Category: gotcha
Context: A migration script turned each accession record's Receipt: field into a sources: path.
Symptom: Two records ended up with dangling sources like "../../../llm-wiki/entities/not owned" and ".../in-store cash/card purchase; no digital receipt".
Root cause: The heuristic treated any field value as a path; some records had free-text receipt notes instead of a path.
Fix: Lint caught the 2 danglers; removed them. Path-extraction should require a path shape (starts with .//../, or contains / and ends in a file ext).
Rule: Field-to-path migration must validate that the value is path-shaped before emitting a link; free-text field values silently become dangling provenance otherwise.

Lesson 7: Verify a book's authorship from its content, not its slug or title

Category: discovery
Context: Compiling and authorship-tagging the book collection.
Symptom: Several slugs implied Fuller authorship but the works are by others.
Root cause: Slug/title under a Fuller collection doesn't establish authorship or subject.
Fix: Agents reading the bodies surfaced the truth: sunspots is by Steve Baer (Zomeworks), est is L. Clark Stevens' Electronic Social Transformation (not Werner Erhard's est), we-are-all-astronauts is a Marta Herford exhibition catalog, regular-polytopes/introduction-to-geometry are Coxeter. Tagged buckminster-fuller (by) / about-buckminster-fuller / fuller-adjacent accordingly.
Rule: Establish authorship/subject from the work's own text before tagging; in a single-subject collection, never infer "by X" from the folder it lives in.