Overview

This document details the workflow for digitizing books, from physical scanning through to multiple digital formats and AI processing capabilities. The workflow encompasses scanning, OCR processing, content organization, proofreading, format generation, and version control.

Setup Project

1. Create directory structure
2. Install manuscript templates
3. Collect metadata for book (Title, Author, Publishes, etc.)
4. Install tools (LaTeX, pandoc, Skim)
5. Validate environment
6. Set up WikiJS integration (copy files to local repo)

Basic Workflow

flowchart TD
    Start([Start]) --> ScanPrep[Prepare Book for CZUR ET24]
    
    subgraph Scanning["Physical to Digital"]
        ScanPrep --> Scanner[CZUR ET24 Scanning]
        Scanner --> ExportTIFF[Export TIFFs]
        Scanner --> ExportPDF[Export Searchable PDF]
        Scanner --> ExportDOCX[Export DOCX]
    end
    
    subgraph FileOrg["File Organization"]
        ExportTIFF --> TIFFDir[manuscript/assets/scans/tiffs]
        ExportPDF & ExportDOCX --> OCRDir[manuscript/assets/ocr]
    end
    
    subgraph Conversion["Format Conversion"]
        OCRDir --> ConvertTex[docx_to_tex_converter.rb]
        ConvertTex --> WipTex[manuscript/content/chapters/wip.tex]
    end
    
    subgraph ContentOrg["Content Organization"]
        WipTex --> ContentTypes[Sort Content]
        ContentTypes --> Chapters[chapters/ ]
        ContentTypes --> FrontMatter[front-matter/ ]
        ContentTypes --> BackMatter[back-matter/ ]
        ContentTypes --> Acronyms[acronyms/ ]
        ContentTypes --> Bibliography[bibliography/ ]
        ContentTypes --> Equations[equations/ ]
        ContentTypes --> Figures[figures/ ]
        ContentTypes --> Glossary[glossary/ ]
        ContentTypes --> Images[images/ ]
        ContentTypes --> References[references/ ]
        ContentTypes --> Tables[tables/ ]
    end
    
    subgraph Proofing["Quality Control"]
        Chapters & Figures & Tables & Equations & Code & References --> Proof[Proofread Content]
        Proof --> CheckLayout[Verify Layout]
        Proof --> CheckScannos[Check Scannos]
        Proof --> CheckTypography[Check Typography]
    end
    
    subgraph Building["Generate Formats"]
        CheckScannos & CheckLayout & CheckTypography --> BuildScript[build_ebooks.rb]
        BuildScript --> Outputs[/Output Formats/]
        Outputs --> PDF[PDF]
        Outputs --> HTML[HTML]
        Outputs --> EPUB[EPUB]
        Outputs --> MOBI[MOBI]
        Outputs --> MD[Markdown]
        Outputs --> RTF[RTF]
        Outputs --> DOCX[DOCX]
        Outputs --> DAISY[DAISY]
        EPUB --> KPF[KPF] --> KPX[KFX]
        MD --> WikiJS[WikiJS]
    end

Directory Structure

project/manuscript/
├── assets/
│   ├── fonts/
│   ├── ocr/
│   ├── scans/
│   │   └── jpgs/
│   │   └── tiffs/
├── content/
│   ├── acronyms/
│   ├── back-matter/
│   ├── bibliography/
│   ├── chapters/
│   ├── equations/
│   ├── figures/
│   ├── front-matter/
│   ├── glossary/
│   ├── images/
│   ├── references/
│   └── tables/
├── output/
│   ├── docx/
│   ├── epub/
│   ├── epub3_temp/      # Temporary EPUB3 build files
│   ├── epub_diff/
│   ├── html/
│   ├── kfx/             # KFX output files
│   ├── kpf/             # KPF output files
│   ├── markdown-wikijs/ # WikiJS formatted markdown (index.md)
│   ├── markdown/        # Markdown output
│   ├── mobi/            # MOBI output files
│   ├── pdf/             # PDF output files
│   └── reports/         # Reports directory
│       ├── ace_results/ # ACE validation results
│       ├── epubcheck/   # ePub validation
│       └── logs/        # Build logs
├── styles/
└── tmp/

Process Details

1. Scanning Process

• Use CZUR ET24 scanner
• Export formats:
  • TIFF files for archival
  • Searchable PDF
  • DOCX for conversion to LaTeX

2. File Organization

• Assets stored in structured directories
• Clear separation of source files and generated content
• Version control integration

3. Content Processing

• Convert DOCX to TEX
• Organize content by type
• Apply consistent formatting
• Track changes in version control

4. Quality Control

• Check for scanning errors, typos, misspellings
• Verify layout and typography
• Ensure consistent formatting
• Validate cross-references

5. Format Generation

• Multiple output formats
• Consistent styling
• Format-specific optimization
• Quality validation

PDF Generation Process

1. Environment validation

   • Check for lualatex
   • Verify output directory permissions
   • Validate LaTeX toolchain

2. Compilation steps

   • Process LaTeX source files
   • Generate initial PDF
   • Run PDF/A conversion
   • Validate output structure

3. Quality checks

   • Verify PDF structure
   • Check for missing fonts
   • Validate cross-references
   • Ensure proper page numbering
   • Standardized output directory structure:
     • Docx in output/docx
     • EPUB3 files go to output/epub/
     • KFX files go to output/kfx/
     • KPF files go to output/kpf/
     • MOBI files go to output/mobi/
     • Markdown files go to output/markdown/
     • WikiJS files go to output/markdown-wikijs/ as index.md
     • ACE validation results go to output/reports/ace_results/
   • Build logs go to output/reports/logs/

Enhanced Build System

flowchart TD
    Start([Build Start]) --> LoadTeX[Load TEX Source]
    
    subgraph Preprocessing["Content Preprocessing"]
        LoadTeX --> ParseContent[Parse Content]
        ParseContent --> ExtractText[Extract Text]
        ParseContent --> ExtractMath[Extract Math]
        ParseContent --> ExtractFigures[Extract Figures]
        ParseContent --> ExtractTables[Extract Tables]
    end
    
    subgraph ExistingFormats["Current Formats"]
        ExtractText --> PDF[PDF]
        ExtractText --> HTML[HTML]
        ExtractText --> EPUB[EPUB]
        ExtractText --> MOBI[MOBI]
        ExtractText --> KPF[KPF]
        ExtractText --> KFX[KFX]
        ExtractText --> MD[Markdown]
        ExtractText --> WikiJS[WikiJS Markdown]
    end
    
    subgraph NewFormats["New Format Support"]
        ExtractText --> DAISY[DAISY]
        DAISY --> DAISYText[DAISY Text]
        DAISY --> DAISYAudio[DAISY Audio]
        
        ExtractText --> RTF[RTF]
        ExtractText --> DOCX[DOCX]
        ExtractText --> Audio[Audio Formats]
        
        Audio --> MP3[MP3]
        Audio --> WAV[WAV]
        Audio --> AAC[AAC]
    end
    
    subgraph Publishing["Publishing Systems"]
        WikiJS --> WikiJSAPI[WikiJS GraphQL API]
        WikiJSAPI --> WikiJSPages[Pages]
        WikiJSAPI --> WikiJSAssets[Assets]
        
        PDF & EPUB --> DigitalLibrary[Digital Library API]
    end

Document Intelligence Integration

flowchart TD
    Start([Source Content]) --> Preprocessing[Document Preprocessing]
    
    subgraph DocumentAnalysis[Document Analysis]
        Preprocessing --> LayoutAnalysis[Layout Analysis]
        Preprocessing --> TextExtraction[Text Extraction]
        Preprocessing --> TableDetection[Table Detection]
        Preprocessing --> FigureDetection[Figure Detection]
        Preprocessing --> EquationDetection[Equation Detection]
        Preprocessing --> StructureIdentification[Structure Identification]
        
        LayoutAnalysis & TextExtraction & TableDetection & FigureDetection & EquationDetection & StructureIdentification --> EnrichedContent[Enriched Content]
    end
    
    subgraph RAGIntegration[RAG Integration]
        EnrichedContent --> TextChunks[Text Chunks]
        EnrichedContent --> Metadata[Metadata Extraction]
        
        TextChunks --> ChunkProps[/Chunk Properties/]
        ChunkProps --> Size["Size: 512-1024 tokens"]
        ChunkProps --> Overlap["Overlap: 10-20%"]
        ChunkProps --> Context["Context Preservation"]
        
        Metadata --> MetaProps[/Metadata Properties/]
        MetaProps --> Source["Source Information"]
        MetaProps --> Position["Position in Document"]
        MetaProps --> Relations["Related Chunks"]
        MetaProps --> StructureInfo["Structure Information"]
        
        TextChunks --> Embedding[Generate Embeddings]
        Metadata --> MetaEmbedding[Metadata Enrichment]
        
        Embedding --> VectorDB[(Vector Database)]
        MetaEmbedding --> VectorDB
    end
    
    subgraph WikiJSIntegration[WikiJS Integration]
        EnrichedContent --> WikiJSMarkdown[WikiJS Markdown]
        WikiJSMarkdown --> WikiJSAPI[WikiJS GraphQL API]
        WikiJSAPI --> WikiJSPages[Pages]
        WikiJSAPI --> WikiJSAssets[Assets]
    end

Version Control

All changes are tracked in Git with specific commit types:

• scan: Updates to scanned content
• ocr: Changes to OCR output
• proof: Proofreading corrections
• build: Build system updates

Quality Standards

1. Scanning Quality

   • Resolution: 300 DPI minimum
   • Color depth: 24-bit color
   • Format: Uncompressed TIFF
   • Validation: Automated format and resolution checks

2. OCR Quality

   • Accuracy: 99%+ for standard text
   • Complete character set support
   • Mathematical formula recognition

3. Output Quality

   • Format-specific validation
   • Cross-reference integrity
   • Consistent styling
   • Accessibility compliance

Workflow Automation

The process is automated where possible using scripts:

• Automatic file organization
• Batch processing
• Quality checks
• Format generation
• Version control integration
• Validation framework for consistent error handling
• Direct publishing to external systems

Publishing Automation

The system can automatically publish content to external platforms:

• WikiJS Publishing
  • Direct integration with WikiJS GraphQL API
  • Automatic page creation and updates
  • Asset uploading and linking
  • Version tracking and change history
  • Authentication via API tokens
  • Error handling and recovery

Validation Framework

The system uses a comprehensive validation framework that provides:

• Input validation for configuration, metadata, and files
• System requirement checks for tools and dependencies
• Standardized error messages with context and recovery hints
• Automated recovery procedures for common issues

Output Validation

ACE Accessibility Validation

The system uses DAISY Ace to validate EPUB accessibility:

• Automated checks for WCAG 2.0 compliance
• Validation of semantic structure
• Image alt text verification
• Navigation structure validation
• Results stored in output/reports/ace_results/

WikiJS Content Validation

Before publishing to WikiJS:

• Validate markdown syntax
• Ensure index.md naming convention
• Verify image paths and references
• Check internal links
• Confirm metadata headers
• Validate image dimensions and formats
• Check for broken internal references
• Verify proper heading hierarchy
• Ensure consistent formatting of code blocks
• Validate table structures

Additional Resources

• CZUR ET24 documentation
• TEX documentation
• Build script documentation
• Style guide
• Event Storming Analysis (docs/event-storm.md)