BUCKYVERSE

LaTeX TOC Parsing and Chapter Structure Extraction

This document explains how Codexis extracts chapter structure from LaTeX documents and generates individual HTML files for each chapter.

Overview

The LaTeX to HTML conversion process in Codexis involves several steps to properly extract and represent the document's hierarchical structure:

  1. TOC File Generation: The LaTeX document is compiled to generate a .toc file
  2. TOC Parsing: The .toc file is parsed to extract chapter, section, and subsection information
  3. Hierarchical Structure: The flat TOC entries are organized into a hierarchical structure
  4. HTML Generation: make4ht is configured to generate separate HTML files for chapters
  5. Navigation: Links are added between chapter files to enable easy navigation

TOC File Format

LaTeX generates a Table of Contents (TOC) file with the extension .toc when you compile a document. The format of this file varies depending on whether you're using standard LaTeX or the hyperref package:

Standard LaTeX Format

\contentsline {chapter}{\numberline {1}Introduction}{1}
\contentsline {section}{\numberline {1.1}Background}{2}

Hyperref Format

\contentsline {chapter}{\numberline {1}Introduction}{1}{chapter.1}
\contentsline {section}{\numberline {1.1}Background}{2}{section.1.1}

Each line consists of:

Implementation Details

1. Parsing the TOC File

The main parsing function is parse_toc_file/1 in Codexis.Builders.LaTeXToHTMLConverter, which:

  1. Reads the TOC file content
  2. Uses regular expressions to extract entry information
  3. Cleans LaTeX commands from titles
  4. Creates a data structure for each entry
# Parse a TOC file to extract document structure
defp parse_toc_file(toc_file) do
  # Read TOC file...
  # Extract entries...
  # Build hierarchical structure...
end

2. Building Hierarchical Structure

The hierarchical document structure is built using the build_hierarchical_structure/1 function, which:

  1. Maps entry types to levels (chapter = 1, section = 2, etc.)
  2. Creates a tree-like structure with parent-child relationships
  3. Assigns HTML filenames to top-level entries (typically chapters)
# Example of hierarchical structure:
[
  %{
    level: 1,
    type: "chapter",
    title: "Introduction",
    page: 1, 
    file: "chapter1.html",
    children: [
      %{
        level: 2,
        type: "section",
        title: "Background",
        page: 2,
        children: []
      }
    ]
  }
]

3. Splitting HTML by Chapter

The chapter-based HTML generation is configured through a custom Lua script for make4ht that:

  1. Identifies chapter boundaries in the generated HTML
  2. Extracts content for each chapter
  3. Creates separate HTML files for each chapter
  4. Generates a table of contents file (index.html)
  5. Adds navigation links between chapter files

Special Handling

LaTeX Commands in Titles

LaTeX titles often contain commands that need to be properly handled:

# Clean LaTeX commands from titles
defp clean_latex_title(title) do
  title
  |> String.replace(~r/\\LaTeX\b/, "LaTeX")
  |> String.replace(~r/\\textbf\{([^}]+)\}/, "\\1")
  # ... more replacements
end

Mathematical Notation

Math expressions in titles are detected and replaced with a placeholder:

"Quadratic Formula: $x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}$"  
# Becomes:
"Quadratic Formula: [math]"

Appendices

Appendices are treated as chapters but with special filename handling:

case entry.type do
  "chapter" -> "chapter#{entry.page}.html"
  "appendix" -> "appendix#{String.downcase(extract_appendix_letter(clean_title))}.html"
  # ...
end

Usage

The TOC parsing functionality is automatically used when converting LaTeX to HTML with chapter splitting enabled:

LaTeXToHTMLConverter.convert(source_dir, main_tex_file, output_dir, split_chapters: true)

Or when using the Make4HTWrapper directly:

Make4HTWrapper.convert(main_tex_file, output_dir, %{split_by_chapter: true})

Demo and Testing

You can test the TOC parsing functionality using the provided demo script:

mix run examples/toc_parser_demo.exs examples/latex_samples/complex.toc

For automated testing, see the test/codexis/builders/toc_parser_test.exs file.

Limitations and Future Improvements