Content Consistency Test Inventory

Overview

This document provides a comprehensive inventory of all existing content consistency tests in the Codexis project. These tests verify that content is properly converted and maintained across different output formats (PDF, HTML, EPUB, DOCX, Markdown, WikiJS).

Main Test Files

1. /test/codexis/builders/content_consistency_test.exs

File Path: /Volumes/Storage/belmendo/Projects/codexis/test/codexis/builders/content_consistency_test.exs

Purpose: Primary comprehensive content consistency test module containing all major test cases for cross-format validation.

Functionality Tested:

• Module initialization and file path validation
• Chapter title extraction and cross-format comparison
• Acknowledgements content validation across all formats
• List of Figures/Tables presence verification
• Bibliography section validation with specific content checks
• Index section presence verification
• Front-matter sections (Dedication, Preface, Introduction, Foreword, Copyright)
• Back-matter sections (Appendix, Glossary)
• Table of Contents validation
• Epigraph and Dedication content validation
• Comprehensive structural validation of all book sections
• LaTeX cross-references conversion testing
• LaTeX citations conversion testing
• LaTeX figures conversion testing
• LaTeX tables conversion testing

Test Data Usage:

• Uses real manuscript.tex and associated content files
• Extracts chapter titles from manuscript/content/chapters/*.tex
• Reads actual LaTeX files for content extraction
• Uses module attributes for consistent file paths across tests
• Relies on TestHelpers module for common functionality

Test Setup Approach:

• Uses module attributes (@manuscript_chapters_dir, @pdf_path, etc.) for consistent path references
• Implements setup blocks to skip tests if manuscript files don't exist
• Uses Patch for mocking dependencies
• Includes format-specific exception handling (e.g., DOCX doesn't support List of Figures)

Coverage Scope:

• All 6 output formats: PDF, HTML, EPUB, DOCX, Markdown, WikiJS
• Complete document structure from front-matter to back-matter
• Mathematical content and special LaTeX commands
• Cross-references and citations
• Figures, tables, and media content

Manuscript.tex Usage:

• Direct dependency on actual manuscript files
• Extracts real content for comparison across formats
• Uses LaTeX-specific parsing for content extraction
• Validates that LaTeX commands are properly converted in each format

Issues/Dependencies:

• Large monolithic file (1880 lines) with multiple responsibilities
• Heavy reliance on file system state and manuscript content
• Some tests include workarounds for format limitations
• Contains duplicate code patterns across similar tests

2. /test/codexis/builders/content_consistency/ Directory

2.1 back_matter_tests.ex

File Path: /Volumes/Storage/belmendo/Projects/codexis/test/codexis/builders/content_consistency/back_matter_tests.ex

Purpose: Focused testing of back-matter sections (Bibliography, Index, Appendix, Glossary, List of Figures/Tables, Todo List).

Functionality Tested:

• Bibliography section presence and specific content validation
• Index section validation across formats
• Back-matter sections (Appendix, Glossary) validation
• List of Figures/Tables validation
• Todo List section validation with specific todo items

Test Data Usage:

• Uses FormatExtractors module for content extraction
• Uses TestHelpers for path management and normalization
• Includes hardcoded expected content (e.g., "Nine Chains to the Moon")
• Uses BuildOrchestrator for WikiJS file generation

Test Setup Approach:

• Modular approach using shared extractors and helpers
• Dynamic WikiJS file creation when needed
• Format-specific exception handling (DOCX limitations)
• Detailed debugging output for troubleshooting

2.2 chapter_tests.ex

File Path: /Volumes/Storage/belmendo/Projects/codexis/test/codexis/builders/content_consistency/chapter_tests.ex

Purpose: Testing chapter title consistency across all output formats.

Functionality Tested:

• Chapter title extraction from manuscript
• Cross-format validation of all chapter titles
• Format availability checking before testing
• Graceful handling of missing output files

Test Data Usage:

• Extracts chapter titles from real manuscript files
• Uses conditional testing based on file availability
• Provides detailed logging of format availability

Test Setup Approach:

• Defensive programming with existence checks
• Non-failing approach (warnings instead of failures in test environment)
• Comprehensive logging for debugging

2.3 front_matter_tests.ex

File Path: /Volumes/Storage/belmendo/Projects/codexis/test/codexis/builders/content_consistency/front_matter_tests.ex

Purpose: Testing front-matter sections (Acknowledgements, Dedication, Preface, etc.).

Functionality Tested:

• Acknowledgements content extraction and validation
• Front-matter sections (Dedication, Preface, Introduction)
• Foreword and Copyright sections
• Epigraph and Dedication content validation
• Table of Contents validation

Test Data Usage:

• Direct LaTeX content extraction from source files
• Uses FormatExtractors for consistent content retrieval
• Includes debugging helpers for LaTeX content analysis

2.4 media_tests.ex

File Path: /Volumes/Storage/belmendo/Projects/codexis/test/codexis/builders/content_consistency/media_tests.ex

Purpose: Testing media content (figures, tables) conversion across formats.

Functionality Tested:

• LaTeX figures conversion validation
• LaTeX tables conversion validation
• Caption preservation across formats

Test Data Usage:

• Scans manuscript files for figure and table commands
• Extracts captions using regex patterns
• Uses actual LaTeX content for validation

2.5 structural_tests.ex

File Path: /Volumes/Storage/belmendo/Projects/codexis/test/codexis/builders/content_consistency/structural_tests.ex

Purpose: Testing structural elements (cross-references, citations, comprehensive sections).

Functionality Tested:

• Complete book structure validation
• LaTeX cross-references conversion
• LaTeX citations conversion
• Format-specific exception handling

Test Data Usage:

• Comprehensive section list validation
• LaTeX command extraction and validation
• Bibliography integration testing

3. /test/support/builders/content_consistency/ Directory

3.1 format_extractors.ex

File Path: /Volumes/Storage/belmendo/Projects/codexis/test/support/builders/content_consistency/format_extractors.ex

Purpose: Centralized content extraction utilities for all output formats.

Functionality:

• PDF text extraction using PDFExtractor
• HTML content extraction with Floki parsing
• EPUB content extraction from ZIP archives
• DOCX content extraction using DocxParser
• Markdown content extraction and parsing
• WikiJS content extraction with frontmatter handling
• LaTeX source content extraction
• Format-specific section extractors

Test Data Usage:

• Uses TestHelpers for consistent path management
• Implements format-specific parsing logic
• Handles missing files gracefully
• Automatic WikiJS file generation when needed

Key Features:

• Modular extraction functions for reuse
• Error handling and fallback mechanisms
• Debug logging for troubleshooting
• BuildOrchestrator integration for dynamic content generation

3.2 test_helpers.ex

File Path: /Volumes/Storage/belmendo/Projects/codexis/test/support/builders/content_consistency/test_helpers.ex

Purpose: Common utilities and helper functions for content consistency testing.

Functionality:

• Centralized path management using module attributes
• Chapter title extraction from LaTeX files
• Section content extraction with multiple pattern matching
• Text normalization for cross-format comparison

Test Data Usage:

• Hardcoded path constants for consistency
• Regex-based LaTeX content parsing
• Flexible section extraction supporting multiple LaTeX patterns

Key Features:

• Standardized path management across all tests
• Robust text normalization for comparison
• Pattern-based content extraction from LaTeX sources

4. Related Integration Tests

4.1 /test/validation/cross_format_consistency_test.exs

File Path: /Volumes/Storage/belmendo/Projects/codexis/test/validation/cross_format_consistency_test.exs

Purpose: High-level validation test creating comprehensive test manuscript and validating consistency across formats.

Functionality Tested:

• Content structure consistency (chapters, sections)
• Mathematical content preservation
• Cross-reference consistency
• Formatting consistency (emphasis, lists)
• Special content preservation (code blocks, tables)
• Metadata consistency
• File integrity validation

Test Data Usage:

• Creates complete test manuscript programmatically
• Uses comprehensive LaTeX document with all features
• Generates metadata and configuration files
• Creates temporary test environment

Test Setup Approach:

• Complete isolation with temporary directories
• Programmatic manuscript generation
• Comprehensive build validation
• File integrity and format validation

Coverage Scope:

• End-to-end build system validation
• Format-specific validation rules
• Performance and efficiency testing
• Content preservation across all formats

4.2 /test/integration/unified_build_system_test.exs

File Path: /Volumes/Storage/belmendo/Projects/codexis/test/integration/unified_build_system_test.exs

Purpose: Integration testing for the unified build system with focus on content consistency as a byproduct.

Functionality Tested:

• Unified build system end-to-end operation
• Multi-format builds with shared auxiliary files
• Build performance and efficiency
• Error handling and recovery
• Content consistency across formats

Test Data Usage:

• Creates integration test manuscript
• Uses temporary test environment
• Includes performance benchmarking
• Tests error scenarios with invalid content

Test Setup Approach:

• Uses Patch for build orchestrator testing
• Includes performance measurement
• Tests preservation and cleanup policies
• Validates shared auxiliary file usage

Analysis and Key Insights

Strengths of Current Test Architecture

1. Comprehensive Coverage: Tests cover all aspects of content consistency from front-matter to back-matter across all 6 output formats.

2. Real Data Usage: Tests use actual manuscript.tex content rather than synthetic test data, ensuring real-world validation.

3. Modular Design: The content_consistency directory shows good separation of concerns with focused test modules.

4. Robust Extraction: FormatExtractors module provides reliable content extraction across all formats.

5. Graceful Degradation: Tests handle missing files and format limitations appropriately.

6. Debug Support: Extensive debugging output helps troubleshoot content consistency issues.

Areas for Improvement

1. Code Duplication: The main content_consistency_test.exs file contains significant duplication that's been partially addressed in the modular approach.

2. Test Data Dependencies: Heavy reliance on manuscript.tex file structure could make tests brittle.

3. Format-Specific Workarounds: Several hardcoded exceptions for format limitations (e.g., DOCX not supporting List of Figures).

4. Large Test Files: The main test file is very large (1880 lines) and could benefit from further modularization.

Model Approach for Consolidation

The modular approach in /test/codexis/builders/content_consistency/ represents the ideal model:

• Focused Responsibility: Each module tests specific aspects
• Shared Utilities: Common functionality extracted to helpers and extractors
• Consistent Interface: All modules use the same extraction and validation patterns
• Maintainable Structure: Easy to add new tests or modify existing ones

Recommendations for Consolidation

1. Complete Migration: Move remaining tests from the main file to focused modules
2. Enhance Shared Utilities: Expand FormatExtractors and TestHelpers with additional common functionality
3. Standardize Patterns: Ensure all tests follow the same structure and error handling approach
4. Improve Documentation: Add comprehensive documentation for the test architecture
5. Add Integration Layer: Create higher-level test orchestration for complete validation scenarios