Codexis

Overview

Codexis is a command-line book-digitization system for building a personal research library. It transforms printed books into readable, searchable, annotatable digital editions that capture a work's ideas and content — not the physical layout of the page it came from.

It began with a single trapped book: Buckminster Fuller's Cosmography, available only as a used, price-gouged, out-of-print volume. The desire was concrete — to actually engage with the work: to read it on a personal device, at a font size of one's choosing, to annotate it, and to search it. The institutional path (asking rights-and-archive holders to digitize a catalog) was tried first and stalled, so the work became self-directed. Over a hundred books have since been digitized this way into a private, copyright-agnostic library for personal study.

Core idea: ideas outlive their containers

A book's enduring value is its intellectual content — the argument, the vocabulary, the structure of thought — not the particular physical artifact that carried it. The printed vessel is scarce, fragile, fixed, and often gatekept; the ideas inside deserve to be freed into forms that endure and that the reader controls.

Preservation here means liberation for use, not archival custody. The goal is not a perfect copy of an object but ideas made readable, resizable, annotatable, and searchable on the reader's terms. To do that faithfully, the system captures meaning and structure and deliberately discards the page. Figures and diagrams are not page decoration — where an illustration is part of the argument, it is captured as content.

This implies a discipline: each captured work becomes a single canonical source of truth, and every readable form (PDF, ePub, mobi/kfx, Markdown, WikiJS) is mechanically derived from it — an edition as projection. No output form is hand-tuned and then treated as canonical; corrections flow back to the source. Fidelity is owed to the ideas, and consistency to the single source — never to the appearance of the original page.

What Codexis is — and isn't

• It captures meaning and structure — not the page. Typography, pagination, and visual facsimile of the original are explicitly not its concern.
• It transforms existing works — it does not author, summarize, paraphrase, or rewrite content. Machine assistance only restores what was captured; it never originates or alters the author's ideas.
• It builds a private library — it is not a publisher, host, storefront, or public archive, and it does not distribute or catalog editions for others.
• It is copyright-agnostic for personal study — it does not arbitrate rights; legal stewardship of works is not a responsibility it claims.

Codexis vs. this site

buckyverse.org consumes Codexis's WikiJS output as one derived projection among many. Publishing a derived form here is buckyverse's activity, not part of Codexis's identity — Codexis itself remains a private-library tool. A completed Codexis book is imported into this repo via codexis deploy --target buckyverse and curated as a repo-root source.

Process (single source → derived forms)

The canonical manuscript (LaTeX) is the source of truth; every other format is a projection built from it.

1. Physical texts are scanned into .tiff files (and a searchable PDF / DOCX).
2. Images are processed with optical character recognition (OCR) into formatted documents (.docx) — or scanned page images are OCR'd directly with a local vision model.
3. Document files are translated into the canonical LaTeX manuscript.
4. The manuscript is run through the toolchain to produce derived PDF, ePub, mobi, kfx, Markdown, and WikiJS forms.
5. Markdown/WikiJS versions are converted into Wiki.js pages for sites that consume them.
6. WikiJS indexes all pages for searching.
7. A Test Book is published at https://www.buckyverse.org/en/test_book/index to confirm the derived forms render content and structure faithfully from the LaTeX source.

Workflow

flowchart TD
    Start([Start]) --> ScanPrep[Prepare Book for CZUR ET24 Pro]
    
    subgraph Scanning["Physical to Digital"]
        ScanPrep --> Scanner[CZUR ET24 Scanning]
        Scanner --> ExportTIFF[Export TIFFs]
        Scanner --> ExportPDF[Export Searchable PDF]
        Scanner --> ExportDOCX[Export DOCX]
    end
    
    subgraph FileOrg["File Organization"]
        ExportTIFF --> TIFFDir[assets/scans/tiffs]
        ExportPDF & ExportDOCX --> OCRDir[assets/ocr]
    end
    
    subgraph Conversion["Format Conversion"]
        OCRDir --> ConvertTex[docx_to_tex_converter.rb]
        ConvertTex --> WipTex[content/chapters/wip.tex]
    end
    
    subgraph ContentOrg["Content Organization"]
        WipTex --> ContentTypes[Sort Content]
        ContentTypes --> Chapters[content/chapters/ ]
        ContentTypes --> Figures[content/figures/ ]
        ContentTypes --> Tables[content/tables/ ]
        ContentTypes --> Equations[content/equations/ ]
        ContentTypes --> Code[content/listings/ ]
        ContentTypes --> References[content/references/ ]
    end
    
    subgraph Proofing["Quality Control"]
        Chapters & Figures & Tables & Equations & Code & References --> Proof[Proofread Content]
        Proof --> CheckScannos[Check Scannos]
        Proof --> CheckLaTeX[Check LaTeX]
        Proof --> CheckLayout[Verify Layout]
        Proof --> CheckTypography[Check Typography]
    end
    
    subgraph Building["Generate Formats"]
        CheckScannos & CheckLayout & CheckTypography --> BuildScript[codexis build --all]
        BuildScript --> Outputs[/Output Formats/]
        Outputs --> PDF[PDF]
        Outputs --> HTML[HTML]
        Outputs --> EPUB[EPUB]
        Outputs --> MOBI[MOBI]
        HTML --> MARKDOWN[Markdown]
        EPUB --> KPF[KPF] --> KPX[KFX]
        MARKDOWN --> WikiJS[WikiJS]
    end

Roadmap

Status reconciled against the Codexis source on 2026-06-30. Items marked DONE were verified shipped; remaining TODO/FIX items are either confirmed not-yet-implemented or unverified. The WikiJS Markdown Conversion fixes below have not been re-verified and are kept as-is.

Wiki.js

1. DONE Mirror Zotero collection to wiki pages — zotero/ tooling (fetch_from_zotero, zotero-to-wikijs, ingest_to_wiki.py); the Buckyverse collections were ingested into the knowledge base on 2026-06-30
2. TODO List recent changes to front page

Technical

1. DONE Convert separate book repos into a buckyverse monorepo — this repo holds the book-project source trees at its root
2. DONE Any time a file type is built, copy updated output to project — codexis deploy --target buckyverse
3. TODO Add ssh keys
4. TODO Try Ketty
5. TODO Try Ed
6. TODO Run book through FlightDeck
7. TODO Run ePub through ebook-polish
8. TODO Build a Docker for LaTex to re-build books when anything changes, content or toolchain

LaTeX Manuscripts

1. DONE Numbered, titled chapter files wired into the main manuscript — codexis scaffolds NN-title.tex chapter inputs (e.g. 01-introduction.tex) referenced from manuscript.tex
2. DONE Build an RTF version — codexis build RTF target
3. DONE Build a DOCX version — included in codexis build --all
4. TODO Redo raster images using ImageMagick + Potrace (VectorMagic, commercial) then to tikz.
5. TODO Change indexing to xindy from makeindex (templates still use makeindex)
6. TODO Use awesomebox pkg
7. TODO Create a dictionary file of Bucky/synergetic terms for WebStorm and MS Word
8. TODO Build a DAISY version
9. TODO Build an audio version
10. TODO Build a braille version

WikiJS Markdown Conversion

Verified 2026-06-30 by a fresh codexis build --all of the bundled feature-test book, then root-caused. Root cause: the Markdown/WikiJS format builds its own prerequisite HTML via tex4ebook/t4ht, which for book-class (book/scrbook) \backmatter docs is poorer than the standalone --format html output — it drops the Index (still open), and tex4ht emits no bibliography-entry anchors so citations dangled (now fixed, see below). The Feb-2026 deployed page predates this rework: its custom HtmlToMarkdownBuilder carried the index through and synthesized #ref- bib anchors. (The deployed "List of Equations" was a phantom — injected by that old converter even though the source has no \listofequations.) Tracked in codexis work/BUGS.md.

1. DONE List of Tables — renders as a ## List of Tables section
2. DONE List of Figures — renders as a ## List of Figures section
3. DONE Skip title and half-title pages — no title-page text leaks into the output
4. DONE Footnote number-before-content — definitions render as [^n]: content (correct order). Minor remaining artifact: the inline marker renders as [^1^](#fn1x0) (stray caret + redundant anchor link)
5. DONE citation links to bibliography entries — fixed in codexis (WikiJSBuilder.resolve_tex4ht_citations/1, branch fix/wikijs-citation-anchors): rewrites dangling #X0- cite links to #ref-<key>, injects matching <a id="ref-<key>"> anchors on the **[Label]** bibliography entries, and strips stray empty []() links. Unit-tested; verified 11/11 citations resolve on the test book. (Take effect requires mix escript.build.)
6. WONTFIX List of Equations — phantom: the source has no \listofequations, so none should render; the deployed page's LoE was an erroneous converter injection (the old "duplicate list-of-equations injection" bug). Only emit if the source requests one
7. FIX Index missing — real regression: source has \makeindex and the standalone --format html build renders an index page, but the markdown/tex4ebook HTML drops it; fix by making the markdown path consume the standalone HTML
8. FIX Todo List missing — no consolidated Todo List section; only inline \change/todonote items render
9. FIX Code listings lose backslashes — LaTeX shown in code blocks renders as title{…}/maketitle instead of \title{…}/\maketitle
10. TODO Add ids/labels for hyperlinks for every paragraph — not present
11. TODO Add ids/labels for mapping to physical page numbers — not present
12. TODO On scans page, add a link to .tiff version of scan image — scans page links .jpg page images, no .tiff
13. TODO If a larger image exists, click to view a full-size version lightbox-style — not present

docs