Buckyverse: Elixir Static Site Generator with LaTeX → HTML + MathML
Project Overview
Build a static site generator in Elixir that converts LaTeX book sources into HTML with native MathML for math rendering. Designed for hosting digitized out-of-print books, specifically R. Buckminster Fuller's works.
Source Material: Books are managed using Codexis (a book digitization tool) which produces LaTeX manuscripts. The static site generator will consume these LaTeX sources and generate a web-friendly reading experience.
Technical Stack
- Elixir - Core language
- Pandoc - LaTeX to HTML conversion (with MathML support)
- MathML - Native browser math rendering (no JavaScript)
- Purple Numbers - Paragraph-level anchors for precise citation
- Alternative: tex4ht/make4ht - Direct LaTeX to HTML+MathML conversion
- NimbleParsec - Optional: Custom LaTeX parser for specific elements
- Static HTML output - Deploy anywhere (Netlify, Cloudflare Pages, S3)
Project Structure
buckyverse/
├── lib/
│ ├── buckyverse/
│ │ ├── application.ex
│ │ ├── library.ex # Content loading
│ │ ├── library/
│ │ │ ├── book.ex # Book struct
│ │ │ ├── chapter.ex # Chapter struct
│ │ │ └── page.ex # Page struct
│ │ ├── latex/
│ │ │ ├── converter.ex # LaTeX → HTML conversion (Pandoc/tex4ht)
│ │ │ ├── parser.ex # Parse LaTeX structure (\chapter, \section)
│ │ │ └── post_processor.ex # Clean up HTML output
│ │ ├── templates/
│ │ │ ├── layout.ex # HTML layout
│ │ │ ├── page.ex # Page template
│ │ │ ├── home.ex # Home page template
│ │ │ └── components.ex # Reusable components
│ │ ├── build.ex # Static site generator
│ │ └── watch.ex # Dev server with live reload
│ └── buckyverse.ex
├── priv/
│ ├── books/ # Content directory (Codexis projects)
│ │ └── grunch-of-giants/ # Example Codexis book project
│ │ ├── meta.yaml # Codexis book metadata
│ │ ├── config.yaml # Codexis build config
│ │ ├── manuscript.tex # Main LaTeX file
│ │ ├── content/
│ │ │ ├── front-matter/
│ │ │ │ ├── foreword.tex
│ │ │ │ └── ...
│ │ │ └── chapters/
│ │ │ ├── 01-fee-x-fie-x-fo-x-fum.tex
│ │ │ ├── 02-astro-age-davids-sling.tex
│ │ │ └── ...
│ │ ├── assets/ # Images, fonts, etc.
│ │ └── styles/ # LaTeX style files
│ └── static/
│ ├── css/
│ │ └── style.css
│ └── js/
│ └── app.js
├── _site/ # Build output
├── mix.exs
└── README.md
Phase 0: Configuration Decisions
0.1 Library Choices
LaTeX to HTML conversion:
-
Option 1: Pandoc (Recommended)
- Excellent LaTeX support with MathML output
- Call via
System.cmd/3orrambo - Command:
pandoc --from=latex --to=html5 --mathml input.tex - Pros: Mature, handles complex LaTeX, good MathML
- Cons: External dependency, must be installed on system
-
Option 2: tex4ht/make4ht
- Part of TeX Live distribution
- Command:
make4ht -f html5+mathml input.tex - Pros: Designed for LaTeX, very comprehensive
- Cons: Complex output, requires cleanup, slower
-
Option 3: Custom parser (Not recommended for v1)
- Parse LaTeX structure with NimbleParsec
- Convert subset of LaTeX commands
- Pros: Full control, no external deps
- Cons: Huge effort, won't handle all LaTeX features
Recommendation: Use Pandoc for LaTeX conversion
HTTP server:
- Use
plug_cowboy~> 2.7 - Better developer experience - Add
plug~> 1.15 as dependency - Alternative:
:inets(built-in, no deps, but less features)
Template system:
- Use EEx (built-in) with pure Elixir functions
- No need for
phoenix_html- keep it simple
0.2 Configuration File (config/config.exs)
import Config
config :buckyverse,
books_path: "priv/books", # Directory containing Codexis projects
static_path: "priv/static",
output_path: "_site",
dev_server_port: 4000,
pandoc_path: "pandoc", # Pandoc executable
enable_tex4ht_fallback: false,
tex4ht_path: "/usr/local/bin/make4ht"
import_config "#{config_env()}.exs"
0.3 Environment Configs
config/dev.exs:
import Config
config :buckyverse,
enable_live_reload: true,
minify_html: false
config/prod.exs:
import Config
config :buckyverse,
enable_live_reload: false,
minify_html: true
config/test.exs:
import Config
config :buckyverse,
books_path: "test/fixtures/books",
output_path: "test/output"
0.4 Git Configuration (.gitignore)
# Build artifacts
/_build/
/deps/
/_site/
# Elixir LS
/.elixir_ls/
# Temp files
*.tmp
/tmp/
# tex4ht artifacts
*.4ct
*.4tc
*.dvi
*.idv
*.lg
*.tmp
*.xref
# OS files
.DS_Store
Phase 1: Project Setup
1.1 Create new Elixir project
mix new buckyverse --sup
cd buckyverse
1.2 Configure mix.exs
Dependencies:
yaml_elixir~> 2.9 - YAML parsing for Codexis metadatafloki~> 0.36 - HTML parsing and post-processingslugify~> 1.3 - URL slug generationplug_cowboy~> 2.7 - HTTP server for dev modeplug~> 1.15 - Web server interfacerambo~> 0.3 - Shell commands (Pandoc/tex4ht)briefly~> 0.5 - Temp files for LaTeX processingfile_system~> 1.0 - File watching for dev serverjason~> 1.4 - JSON encoding for search index
System Requirements:
- Pandoc 2.19+ (install via:
brew install pandocorapt-get install pandoc) - OR TeX Live with tex4ht (for make4ht option)
Note: Books should be Codexis projects with meta.yaml, manuscript.tex, and content/ directory.
Mix aliases:
defp aliases do
[
"bucky.build": ["run -e 'Buckyverse.Build.run()'"],
"bucky.watch": ["run --no-halt -e 'Buckyverse.Watch.start()'"],
test: ["test"]
]
end
1.3 Create directory structure
Create all directories listed in project structure above.
mkdir -p lib/buckyverse/{library,latex,templates}
mkdir -p priv/{books,static/{css,js}}
mkdir -p test/{fixtures/books,support}
mkdir -p config
1.4 Create configuration files
Create config/config.exs, config/dev.exs, config/prod.exs, config/test.exs as specified in Phase 0.
Phase 2: Data Structures
2.1 Book struct (lib/buckyverse/library/book.ex)
Fields:
slug- URL identifiertitle- Book titlesubtitle- Optional subtitleauthor- Author nameyear- Publication yeardescription- Book descriptionchapters- List of Chapter structs
2.2 Chapter struct (lib/buckyverse/library/chapter.ex)
Fields:
slug- URL identifier (from LaTeX label or filename)title- Chapter title (from\chapter{})position- Sort order (from manuscript.tex include order)source_file- Path to .tex filehtml- Rendered HTML contenttoc- Table of contents (list of heading maps with level, id, text)book_slug- Parent book identifier
Note: In LaTeX-based system, each chapter is a single page. No separate Page struct needed unless chapters are very long and need splitting.
Phase 3: LaTeX Conversion
3.1 LaTeX Structure Parser (lib/buckyverse/latex/parser.ex)
Parse LaTeX document structure to extract chapters and metadata.
Parse from meta.yaml (Codexis metadata):
title[0].text- Book titlecreator[0].text- Author name (where role=author)description- Book descriptioncopyright_year- Publication yearpublisher- Publisher nameidentifier- ISBN and other identifiers
Parse from manuscript.tex:
\include{...}or\input{...}- Chapter files- Detect
\frontmatter,\mainmatter,\backmattersections - Extract chapter order from include statements
Parse from chapter files:
\chapter{...}- Chapter title\label{ch:...}- Chapter identifier\section{...},\subsection{...}- Section headings
Public API:
@spec parse_codexis_metadata(path :: String.t()) :: {:ok, map()} | {:error, term()}
def parse_codexis_metadata(path) # Parse meta.yaml
@spec parse_manuscript(path :: String.t()) :: {:ok, %{chapters: list()}} | {:error, term()}
def parse_manuscript(path) # Extract chapter includes
@spec extract_chapter_info(path :: String.t()) :: {:ok, %{title: String.t(), label: String.t()}} | {:error, term()}
def extract_chapter_info(path)
Implementation:
- Use regex or simple string parsing (not full LaTeX parser)
- Extract minimal metadata needed for navigation
- Return list of chapter file paths in order
3.2 LaTeX to HTML Converter (lib/buckyverse/latex/converter.ex)
Convert individual LaTeX chapter files to HTML with MathML.
Using Pandoc (recommended):
@spec convert_file(input_path :: String.t(), opts :: keyword()) :: {:ok, String.t()} | {:error, term()}
def convert_file(input_path, opts \\ []) do
args = [
"--from=latex",
"--to=html5",
"--mathml",
"--standalone",
"--no-highlight", # We'll handle code highlighting
input_path
]
case Rambo.run("pandoc", args) do
{:ok, %{status: 0, out: html}} -> {:ok, html}
{:ok, %{status: status, err: err}} -> {:error, "Pandoc failed: #{err}"}
{:error, reason} -> {:error, reason}
end
end
Options:
:mathml- Use MathML for math (default: true):extract_body- Extract only body content, not full document (default: true):base_header_level- Adjust header levels (default: 1)
Alternative: tex4ht/make4ht:
def convert_with_tex4ht(input_path, opts \\ []) do
# Create temp directory
{:ok, tmp_dir} = Briefly.create(directory: true)
# Copy input file to temp
basename = Path.basename(input_path)
tmp_input = Path.join(tmp_dir, basename)
File.cp!(input_path, tmp_input)
# Run make4ht
args = ["-f", "html5+mathml", "-d", tmp_dir, basename]
case Rambo.run("make4ht", args, cd: tmp_dir) do
{:ok, %{status: 0}} ->
html_file = Path.join(tmp_dir, Path.basename(basename, ".tex") <> ".html")
{:ok, File.read!(html_file)}
error -> error
end
end
3.3 HTML Post-Processor (lib/buckyverse/latex/post_processor.ex)
Clean up and enhance Pandoc/tex4ht HTML output.
Processing steps:
- Extract body content (remove Pandoc's
<head>, keep only<body>contents) - Remove or adjust header levels (h1 → h2, etc.)
- Add IDs to headings for TOC
- Fix image paths (relative to book directory)
- Extract table of contents
- Clean up unnecessary classes/attributes
- Wrap figures with captions
- Add section numbers if present in LaTeX
Public API:
@spec process(html :: String.t(), opts :: keyword()) :: {String.t(), list()}
def process(html, opts \\ [])
Returns: {processed_html, toc_items}
Use Floki for HTML manipulation:
def process(html, opts) do
{:ok, document} = Floki.parse_document(html)
document
|> extract_body()
|> adjust_headers()
|> add_heading_ids()
|> fix_image_paths(opts[:book_slug])
|> extract_toc()
end
Phase 4: Content Processing (Removed)
Note: This phase is replaced by Phase 3 (LaTeX Conversion). Since source content is in LaTeX, we don't need a separate Markdown renderer. All content processing happens through:
- LaTeX structure parsing (Phase 3.1)
- LaTeX to HTML conversion via Pandoc (Phase 3.2)
- HTML post-processing (Phase 3.3)
Phase 5: Content Loading
5.1 Library module (lib/buckyverse/library.ex)
Public API:
list_books()- Returns list of Book structsget_book(slug)- Returns{:ok, book}or{:error, :not_found}get_chapter(book_slug, chapter_slug)- Returns chapter or errorget_nav_tree(book)- Returns nested navigation structureget_prev_next(book, current_chapter)- Returns{prev_chapter | nil, next_chapter | nil}books_path()- Returns path to priv/books (from config)
Loading logic:
- Scan
priv/books/for directories (each is a Codexis project) - For each book directory:
- Look for
manuscript.tex(main LaTeX file) - Load
meta.yaml(Codexis metadata) for book info - Parse manuscript.tex to extract chapter list and structure
- Extract book slug from directory name
- Look for
- For each chapter referenced in manuscript.tex:
- Parse chapter .tex file to extract title and label
- Convert chapter LaTeX to HTML using Pandoc
- Post-process HTML (extract TOC, fix paths, etc.)
- Create Chapter struct with HTML content
- Build nested struct hierarchy
- Copy
figures/directory reference (don't load into memory)
Codexis metadata parsing (meta.yaml):
# From meta.yaml, extract:
- title[0].text → book title
- creator (where role=author) → author name
- description → book description
- copyright_year → publication year
- publisher → publisher name
LaTeX manuscript parsing:
# From manuscript.tex, extract:
- \include{content/chapters/01-fee-x-fie-x-fo-x-fum} → chapter file path
- Order of \include statements → chapter order
Chapter file parsing:
# From chapter .tex file, extract:
- \chapter{...} → chapter title
- \label{ch:...} → chapter identifier/slug
Slug extraction:
- From directory name:
grunch-of-giants→grunch-of-giants - From LaTeX label:
\label{ch:fee-x-fie}→fee-x-fie - From filename:
01-fee-x-fie-x-fo-x-fum.tex→fee-x-fie-x-fo-x-fum
Metadata precedence:
- Book:
meta.yaml(Codexis metadata) > manuscript.tex metadata - Chapter: LaTeX
\chapter{}and\label{}commands
Navigation tree structure:
%{
chapters: [
%{
slug: "fee-x-fie-x-fo-x-fum",
title: "(Fee X fie X fo X fum)⁴",
position: 1,
toc: [
%{level: 2, id: "grunch-definition", text: "GRUNCH Definition"},
%{level: 2, id: "invisible-giants", text: "Invisible Giants"}
]
},
%{
slug: "astro-age-davids-sling",
title: "Astro-Age David's Sling",
position: 2,
toc: [...]
}
]
}
Prev/Next calculation:
- Chapters are already ordered from manuscript.tex
- Find current chapter index
- Return previous and next chapters
- Handle frontmatter/backmatter sections appropriately
Error handling:
- Missing
manuscript.tex: Return error, skip book - Invalid LaTeX syntax: Log error, try to continue
- Pandoc conversion failure: Include error message in HTML, continue
- Missing chapter files: Log error, skip chapter
- Missing figures: Log warning, continue (broken image in output)
Caching:
- Cache parsed LaTeX structure to avoid re-parsing
- Cache converted HTML during development
- Clear cache on file changes in watch mode
Phase 6: HTML Templates
All templates use EEx (Embedded Elixir) with pure functions returning strings.
6.1 Layout template (lib/buckyverse/templates/layout.ex)
Full HTML document with:
- DOCTYPE, html, head, body
- Meta charset UTF-8 and viewport
- Title from assigns (default: "Buckyverse")
- Meta description from assigns (optional)
- Link to
/css/style.css - Content slot (passed as string)
- Script tag for
/js/app.js(optional, for search) - MathML namespace support
Function signature:
@spec render(content :: String.t(), assigns :: map()) :: String.t()
def render(content, assigns \\ %{})
Assigns:
:title- Page title:description- Meta description:body_class- CSS class for body element
6.2 Chapter template (lib/buckyverse/templates/chapter.ex)
Three-column layout:
- Sidebar - Book title, chapter navigation list
- Content - Chapter title, rendered HTML from LaTeX, prev/next links
- TOC - "On this page" with section/subsection links
Function signature:
@spec render(assigns :: map()) :: String.t()
def render(assigns)
Required assigns:
:book- Book struct:chapter- Current chapter struct:nav_tree- Navigation structure:prev_chapter- Previous chapter or nil:next_chapter- Next chapter or nil
6.3 Book index template (lib/buckyverse/templates/book_index.ex)
- Book title, subtitle, author, year
- Book description
- List of chapters with their pages
- Links to first page of each chapter
Function signature:
@spec render(assigns :: map()) :: String.t()
def render(assigns)
Required assigns:
:book- Book struct with chapters and pages
6.4 Home template (lib/buckyverse/templates/home.ex)
- Site title "Buckyverse"
- Tagline: "R. Buckminster Fuller's works, digitized"
- Grid of book cards with title, author, year, description
- Link to each book's index page
Function signature:
@spec render(assigns :: map()) :: String.t()
def render(assigns)
Required assigns:
:books- List of Book structs
6.5 Components (lib/buckyverse/templates/components.ex)
sidebar(book, nav_tree, current_chapter):
- Book title linking to book index (
/{book_slug}/) - Chapter list in order
- Active state for current chapter (CSS class
active) - Chapter numbers/positions if available
- Simple vertical list (no nesting needed since chapters are top-level)
chapter_nav(book, prev_chapter, next_chapter):
- Previous link if exists (with chapter title)
- Next link if exists (with chapter title)
- Wrapped in
<nav class="chapter-nav">
toc(items):
- Heading list with anchor links
- Indentation by level (h2, h3, h4)
- Wrapped in
<nav class="toc"> - Title: "On this page"
book_card(book):
- Book title, subtitle, author, year
- Description (truncated to 200 chars)
- Link to book index
- Wrapped in
<article class="book-card">
6.6 URL Structure
All URLs use trailing slashes and rely on index.html:
- Home:
/→_site/index.html - Book index:
/grunch-of-giants/→_site/grunch-of-giants/index.html - Chapter:
/grunch-of-giants/fee-x-fie-x-fo-x-fum/→_site/grunch-of-giants/fee-x-fie-x-fo-x-fum/index.html - Figures:
/grunch-of-giants/figures/fig-01.svg→_site/grunch-of-giants/figures/fig-01.svg - Static:
/css/style.css→_site/css/style.css
Link generation:
- Always use absolute paths starting with
/ - Always include trailing slash for pages
- No trailing slash for assets
Phase 7: Static Site Builder
7.1 Build module (lib/buckyverse/build.ex)
run(opts \\ []) function:
- Load configuration from Application env
- Clean output directory (delete
_site/if exists) - Create
_site/directory - Build all books (pages as nested directories)
- Build home page
- Copy static assets recursively
- Build search index JSON
- Print build summary (pages built, time elapsed, errors/warnings)
For each book:
- Create
_site/{book_slug}/directory - Write
_site/{book_slug}/index.html(book index page) - For each chapter:
- Create
_site/{book_slug}/{chapter_slug}/directory - Calculate prev/next chapters using
Library.get_prev_next/2 - Get navigation tree using
Library.get_nav_tree/1 - Render chapter template with all assigns
- Wrap in layout template
- Write
_site/{book_slug}/{chapter_slug}/index.html
- Create
- Copy
figures/directory if exists:- Source:
priv/books/{book_slug}/figures/ - Destination:
_site/{book_slug}/figures/ - Preserve directory structure
- Copy all file types (SVG, PNG, JPG, etc.)
- Source:
Static assets copying:
- Source:
priv/static/ - Destination:
_site/ - Copy recursively, preserving directory structure
- Include: CSS, JS, images, fonts, etc.
Search index:
- Array of objects with:
title- Chapter titlebook- Book titlebook_slug- Book slugchapter_slug- Chapter slugurl- Absolute URL path (e.g., "/grunch-of-giants/fee-x-fie-x-fo-x-fum/")text- First 500 chars of plain text (strip HTML tags, keep readable text)headings- List of section headings in chapter
- Write to
_site/search-index.json - Minify JSON in production mode
Error handling:
- Continue build on non-critical errors (missing figures, math parse errors)
- Collect warnings and display at end
- Return
{:ok, stats}or{:error, reason}for critical failures - Critical failures: missing book YAML, invalid YAML syntax, file write errors
Build stats:
%{
books: 1,
chapters: 8,
figures: 15,
warnings: 2,
elapsed_ms: 1234
}
Options:
:clean- Clean output directory before build (default: true):verbose- Print detailed progress (default: false)
Phase 8: Development Server
8.1 Watch module (lib/buckyverse/watch.ex)
start(opts \\ []) function:
- Run initial full build
- Start FileSystem watcher on
priv/books/andpriv/static/ - Start HTTP server on port 4000 (from config) serving
_site/ - On file change events, trigger appropriate rebuild
- Log rebuild activity with timestamps
- Keep server running until interrupted
File watching strategy:
Watch paths:
priv/books/**/*- Content changespriv/static/**/*- Asset changeslib/buckyverse/templates/**/*- Template changes (optional, requires full rebuild)
Rebuild granularity:
-
LaTeX file change (
.texfiles):- Detect which book/chapter changed
- If
manuscript.texchanged: rebuild entire book - If chapter
.texchanged: rebuild only that chapter + book index - Rebuild home page if book metadata changed
- Rebuild search index
-
Static asset change (
.css,.js, images):- Copy only changed file to
_site/ - No need to rebuild HTML
- Copy only changed file to
-
Template change (
.exintemplates/):- Full rebuild (all books)
- Requires code recompilation (advanced, optional for v1)
HTTP server implementation:
Use Plug + Cowboy:
defmodule Buckyverse.DevServer do
use Plug.Router
plug Plug.Static,
at: "/",
from: "_site",
only: ~w(css js images fonts),
gzip: false
plug :match
plug :dispatch
# Serve index.html for directory requests
get "/*path" do
# Handle trailing slash, serve index.html
end
match _ do
send_resp(conn, 404, "Not found")
end
end
Server features:
- Serve
index.htmlfor directory paths - Proper MIME types for all file types
- 404 page for missing content
- CORS headers for local development
- Log all requests
Options:
:port- Server port (default: from config):open_browser- Auto-open browser on start (default: false)
Console output:
Building site...
✓ Built 1 book, 8 chapters in 2.4s
Watching for changes...
Server running at http://localhost:4000
[12:34:56] Changed: priv/books/grunch-of-giants/content/chapters/01-fee-x-fie-x-fo-x-fum.tex
[12:34:56] Rebuilding chapter: fee-x-fie-x-fo-x-fum
[12:34:56] ✓ Rebuilt in 0.5s
Phase 9: Styling
9.1 CSS (priv/static/css/style.css)
Layout:
.reader- Flex container for three columns.sidebar- Fixed position, 280px width, scrollable.content- Main area with max-width.toc- Fixed position on right
Sidebar styling:
- Book title
- Chapter headings (small caps)
- Page links with hover/active states
- Section numbers in monospace
Content styling:
- Prose typography (font sizes, line height, margins)
- Section number display
- Heading styles
Math styling:
.math-block- Centered, overflow scrollmathelement - Math font family.math-error- Red monospace for parse failures
Navigation:
.page-nav- Flex space-between for prev/next
Responsive:
- Hide TOC below 1200px
- Stack sidebar above content below 768px
Phase 10: Sample Content (Already Exists)
Note: The existing book "Grunch of Giants" in priv/books/grunch-of-giants/ serves as sample content. This is a Codexis book project with:
meta.yaml- Codexis metadata (title, author, ISBN, description, etc.)config.yaml- Codexis build configurationmanuscript.tex- Main LaTeX file with book structurecontent/chapters/*.tex- Chapter files (8 chapters)content/front-matter/*.tex- Front matter (foreword, dedication, etc.)assets/- Images and other media filesstyles/- LaTeX style files- LaTeX structure with
\chapter{},\section{}, etc. - Math equations, figures, and full LaTeX formatting
10.1 Codexis Metadata Structure (meta.yaml)
The Codexis meta.yaml file provides rich metadata:
title:
- type: main
text: Grunch of Giants
creator:
- role: author
text: R. Buckminster Fuller
description: Book description...
copyright_year: 1982
publisher: Design Science Press
identifier:
- scheme: ISBN-13
text: 978-1607027591
The static site generator should parse this metadata for book information.
10.2 Example LaTeX Chapter Structure
The existing chapters demonstrate proper LaTeX structure with:
\chapter{}commands for chapter titles\label{}commands for cross-references- Complex LaTeX environments (verse, enumerate, etc.)
- Inline and display math equations
- Text formatting commands
10.3 Testing with Existing Content
The existing Codexis project provides:
- Real-world LaTeX content to test conversion
- Complex formatting to stress-test Pandoc
- Metadata structure to parse
- Multiple chapters for navigation testing
- Front matter and back matter sections
Phase 11: Testing
11.1 Test Structure
test/
├── buckyverse/
│ ├── latex/
│ │ ├── parser_test.exs
│ │ └── mathml_test.exs
│ ├── markdown/
│ │ └── renderer_test.exs
│ ├── library_test.exs
│ └── build_test.exs
├── fixtures/
│ └── books/
│ └── test-book/
│ ├── _book.yaml
│ └── 01-chapter/
│ ├── _chapter.yaml
│ └── 01-page.md
├── support/
│ └── test_helpers.ex
└── test_helper.exs
11.2 Test Coverage
LaTeX Parser tests:
- Parse manuscript.tex structure
- Extract \title, \author, \include statements
- Parse chapter files for \chapter and \label
- Handle different LaTeX document structures
- Error cases (missing files, malformed LaTeX)
LaTeX Converter tests:
- Pandoc integration (mock or real)
- HTML output validation
- MathML generation for equations
- Handle conversion errors gracefully
- Verify temp file cleanup
HTML Post-Processor tests:
- Extract body content correctly
- Add IDs to headings
- Fix image paths
- TOC extraction from HTML
- Clean up Pandoc artifacts
Library tests:
- Book/chapter loading from LaTeX
- Slug extraction from labels and filenames
- Manuscript.tex parsing
- Navigation tree building
- Prev/next chapter calculation
- Error handling for missing files
Build tests:
- Full build produces expected files
- URL structure correctness (chapters as pages)
- Search index generation
- Static asset copying
- Figure directory copying
11.3 Test Helpers (test/support/test_helpers.ex)
defmodule Buckyverse.TestHelpers do
def fixture_path(path), do: Path.join([__DIR__, "..", "fixtures", path])
def create_test_book(attrs \\ %{}) do
# Helper to create test book structs
end
def assert_mathml(mathml, expected_tags) do
# Helper to assert MathML structure
end
end
11.4 Running Tests
# Run all tests
mix test
# Run specific test file
mix test test/buckyverse/latex/parser_test.exs
# Run with coverage
mix test --cover
# Run in watch mode (requires mix_test_watch)
mix test.watch
Build & Run Commands
# Install dependencies
mix deps.get
# Build static site
mix bucky.build
# Development server with watch
mix bucky.watch
# Preview built site
cd _site && python3 -m http.server 8000
# Test math parser in IEx
iex -S mix
IEx testing:
alias Buckyverse.Library
alias Buckyverse.LaTeX.{Parser, Converter}
# Load books
books = Library.list_books()
# Parse manuscript
{:ok, metadata} = Parser.parse_manuscript("priv/books/grunch-of-giants/manuscript.tex")
# Convert chapter
{:ok, html} = Converter.convert_file("priv/books/grunch-of-giants/content/chapters/01-fee-x-fie-x-fo-x-fum.tex")
Deployment
Production Build
# Set production environment
MIX_ENV=prod mix deps.get
MIX_ENV=prod mix bucky.build
Static output in _site/ can be deployed to:
Netlify
# Install Netlify CLI
npm install -g netlify-cli
# Deploy
netlify deploy --prod --dir=_site
netlify.toml:
[build]
command = "mix deps.get && mix bucky.build"
publish = "_site"
[[redirects]]
from = "/*"
to = "/index.html"
status = 404
Cloudflare Pages
- Connect repository
- Build settings:
- Build command:
mix deps.get && mix bucky.build - Build output directory:
_site - Root directory:
/
- Build command:
- Environment variables:
MIX_ENV=prod
GitHub Pages
# Build locally
mix bucky.build
# Deploy to gh-pages branch
git subtree push --prefix _site origin gh-pages
# Or use GitHub Actions (see below)
GitHub Actions workflow (.github/workflows/deploy.yml):
name: Deploy to GitHub Pages
on:
push:
branches: [main]
jobs:
build-deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: erlef/setup-beam@v1
with:
otp-version: '26.0'
elixir-version: '1.15.0'
- run: mix deps.get
- run: mix bucky.build
- uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./_site
AWS S3 + CloudFront
# Install AWS CLI
# Configure credentials: aws configure
# Sync to S3 bucket
aws s3 sync _site/ s3://your-bucket-name/ --delete
# Invalidate CloudFront cache
aws cloudfront create-invalidation \
--distribution-id YOUR_DIST_ID \
--paths "/*"
MathML Browser Support
Current support (2026):
- Chrome/Edge: Full support (since Chrome 109)
- Firefox: Full support
- Safari: Full support (since Safari 14.1)
Fallback for older browsers:
Add MathML polyfill in layout template (optional):
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/mml-chtml.js"></script>
Or detect and show warning:
if (!document.createElement('math').namespaceURI) {
console.warn('MathML not supported');
}
Future Enhancements
- Full-text search - Client-side with Lunr.js or server-side with Haystack
- PDF generation - Use tex4ht or Tectonic for downloadable PDFs
- Phoenix LiveView - Dynamic version with instant navigation
- Ash Framework - Structured domain layer for content
- Cross-references - Parse
\ref{}and\label{}for internal linking - Bibliography - BibTeX support with citations
- Figures - Enhanced figure handling with captions and numbering
- Comments/Annotations - User annotations with persistence
Key Implementation Notes
LaTeX Structure Parsing Tips
- Use regex for simple extractions (
\title{...},\author{...}) - Don't try to parse full LaTeX syntax - just extract what's needed
- Handle
\include{path}vs\input{path}(both are common) - Path resolution:
\include{content/chapters/01-file}may or may not have.texextension - Preserve order of
\includestatements for chapter ordering - Handle comments in LaTeX (
%to end of line)
Pandoc Integration Tips
- Always use
--mathmlflag for math rendering - Use
--standaloneto get full HTML document (easier to parse) - Use
--no-highlightto avoid Pandoc's code highlighting (we can add our own) - Consider
--base-header-level=2to make\chapterbecome<h2>instead of<h1> - Pandoc preserves LaTeX labels as IDs in HTML headings
- Check Pandoc version:
pandoc --version(need 2.19+) - Handle Pandoc errors gracefully (exit codes, stderr)
MathML Tips (from Pandoc output)
- Pandoc generates MathML with proper xmlns
- MathML is already included in HTML output
- No need to post-process math unless fixing specific issues
- Verify browser support (Chrome 109+, Firefox, Safari 14.1+)
Content Loading Tips
- Sort chapters by order in manuscript.tex (not by filename)
- Handle missing chapter files gracefully
- Cache converted HTML if performance becomes an issue
- Use
@external_resourcefor recompilation on content change (if compiling at build time) - Stream large file operations to avoid memory issues
- Use
Path.wildcard/2for efficient directory scanning
Performance Optimization
- Parallelize book builds using
Task.async_stream/3 - Cache parsed LaTeX expressions (memoization)
- Only rebuild changed content in watch mode
- Use ETS for in-memory caching in dev server
- Minimize file I/O with batch operations
- Profile with
:timer.tc/1to find bottlenecks
Error Recovery
- Never crash the build for content errors
- Collect all errors and display at end
- Provide helpful error messages with file paths and line numbers
- Render error placeholders in HTML for debugging
- Use
Loggerfor warnings and errors - Validate YAML schemas before parsing
Appendix A: Configuration Examples
Complete config/config.exs
import Config
config :buckyverse,
books_path: "priv/books", # Codexis projects directory
static_path: "priv/static",
output_path: "_site",
dev_server_port: 4000,
pandoc_path: "pandoc",
enable_tex4ht_fallback: false,
tex4ht_path: "/usr/local/bin/make4ht",
site_title: "Buckyverse",
site_description: "R. Buckminster Fuller's works, digitized",
base_url: "https://buckyverse.com"
config :logger,
level: :info
import_config "#{config_env()}.exs"
Complete mix.exs
defmodule Buckyverse.MixProject do
use Mix.Project
def project do
[
app: :buckyverse,
version: "0.1.0",
elixir: "~> 1.15",
start_permanent: Mix.env() == :prod,
deps: deps(),
aliases: aliases()
]
end
def application do
[
extra_applications: [:logger],
mod: {Buckyverse.Application, []}
]
end
defp deps do
[
{:yaml_elixir, "~> 2.9"}, # Required for Codexis meta.yaml
{:floki, "~> 0.36"},
{:slugify, "~> 1.3"},
{:plug_cowboy, "~> 2.7"},
{:plug, "~> 1.15"},
{:rambo, "~> 0.3"},
{:briefly, "~> 0.5"},
{:file_system, "~> 1.0"},
{:jason, "~> 1.4"}
]
end
defp aliases do
[
"bucky.build": ["run -e 'Buckyverse.Build.run()'"],
"bucky.watch": ["run --no-halt -e 'Buckyverse.Watch.start()'"]
]
end
end
Appendix B: File Organization Patterns
Content Organization (Codexis Projects)
priv/books/
├── grunch-of-giants/ # Codexis project
│ ├── meta.yaml # Codexis metadata (required)
│ ├── config.yaml # Codexis build config
│ ├── manuscript.tex # Main LaTeX file
│ ├── content/
│ │ ├── front-matter/
│ │ │ ├── foreword.tex
│ │ │ ├── dedication.tex
│ │ │ └── ...
│ │ └── chapters/
│ │ ├── 01-fee-x-fie-x-fo-x-fum.tex
│ │ ├── 02-astro-age-davids-sling.tex
│ │ └── ...
│ ├── assets/ # Images, fonts, etc.
│ │ └── images/
│ │ └── cover.jpg
│ └── styles/ # LaTeX style files
│ └── base.sty
└── synergetics/ # Another Codexis project
├── meta.yaml
├── manuscript.tex
└── ...
Output Organization
_site/
├── index.html # Home page
├── css/
│ └── style.css
├── js/
│ └── app.js
├── search-index.json
├── synergetics/
│ ├── index.html # Book index
│ ├── preface/
│ │ ├── index/
│ │ │ └── index.html
│ │ └── preface/
│ │ └── index.html
│ ├── universe/
│ │ ├── index/
│ │ │ └── index.html
│ │ ├── definition/
│ │ │ └── index.html
│ │ └── conceptuality/
│ │ └── index.html
│ └── figures/
│ ├── fig-100.01.svg
│ └── fig-100.02.svg
└── operating-manual/
└── ...
Appendix C: Common Issues and Solutions
Issue: Pandoc conversion fails for specific LaTeX
Solution:
- Check Pandoc version (need 2.19+)
- Try tex4ht/make4ht as alternative converter
- Simplify problematic LaTeX if possible
- Check Pandoc error messages for specific issues
- Add preprocessing step to fix common issues
Issue: Build is slow for large books
Solution:
- Enable parallel chapter conversion with
Task.async_stream/3 - Cache converted HTML with ETS
- Only rebuild changed chapters in watch mode
- Profile with
:timer.tc/1to find bottlenecks - Pandoc is relatively fast, but can be parallelized per-chapter
Issue: Images not displaying
Solution:
- Check image paths are relative to chapter directory
- Verify figures directory is copied to output
- Check URL generation includes book slug
- Validate image files exist before build
Issue: Math not rendering in browser
Solution:
- Check MathML namespace is included
- Verify browser supports MathML (Chrome 109+, Firefox, Safari 14.1+)
- Add MathJax polyfill for older browsers
- Check for malformed MathML with browser inspector
Issue: Navigation links broken
Solution:
- Verify all URLs use absolute paths starting with
/ - Check trailing slashes are consistent
- Validate prev/next calculation logic
- Ensure
index.htmlfiles are created in correct directories
Estimated Timeline
| Phase | Description | Effort |
|---|---|---|
| 0 | Configuration decisions | 30 min |
| 1 | Project setup | 1 hour |
| 2 | Data structures | 30 min |
| 3 | LaTeX conversion (Pandoc integration) | 3-4 hours |
| 4 | Content processing | (removed - integrated into Phase 3) |
| 5 | Content loading from LaTeX | 2-3 hours |
| 6 | HTML templates | 2-3 hours |
| 7 | Static builder | 1-2 hours |
| 8 | Dev server | 1-2 hours |
| 9 | CSS styling | 2-3 hours |
| 10 | Sample content | (already exists) |
| 11 | Testing | 2-3 hours |
Total: ~16-24 hours
Note: Using Pandoc for LaTeX conversion significantly reduces complexity compared to building a custom LaTeX parser. The existing "Grunch of Giants" book provides real content for testing.
Getting Started Checklist
- Phase 0: Review configuration decisions
- Phase 1: Create project and install dependencies
- Phase 2: Define data structures
- Phase 3: Build LaTeX parser (start with simple cases)
- Phase 4: Implement markdown renderer
- Phase 5: Build content loading system
- Phase 6: Create HTML templates
- Phase 7: Implement static site builder
- Phase 8: Add development server
- Phase 9: Style the site with CSS
- Phase 10: Add sample content
- Phase 11: Write tests
- Deploy to hosting platform
This plan is comprehensive and self-contained. Start with Phase 0 to make key decisions, then work through each phase sequentially. Each phase builds on the previous ones.