Unified Test Architecture Documentation

Overview

The Codexis project has implemented a unified test architecture that consolidates builder and content consistency tests using manuscript.tex as the canonical test book. This architecture provides consistent patterns, improved maintainability, and comprehensive validation across all output formats.

Architecture Principles

1. Single Source of Truth

• Canonical Test Book: All tests use manuscript.tex as the primary test source
• Shared Manuscript: Eliminates test data duplication and ensures consistency
• Real-world Testing: Uses actual manuscript content rather than minimal test fixtures

2. Unified Patterns

• Consistent Test Structure: All builder tests follow the same architectural patterns
• Shared Utilities: Common validation functions and helper modules
• Standardized Setup/Teardown: Consistent test lifecycle management

3. Task #17 Integration

• BuildOrchestrator Integration: Uses shared temporary directory approach
• Cleanup Management: Proper handling of cleanup_temp_dir and preserve flags
• Performance Optimization: Efficient temporary directory lifecycle

Test Categories

1. Builder Tests (Format-Specific)

PDF Builder (pdf_builder_unified_test.exs)

@moduletag :pdf_builder_unified
@test_source "manuscript"

# Features tested:
- Full manuscript.tex processing with bibliography, glossaries
- Mathematical content rendering
- Image and figure handling
- Cross-reference resolution
- PDF structure validation
- Tool availability checking (lualatex, bibtex, makeglossaries)

HTML Builder (html_builder_unified_test.exs)

@moduletag :html_builder_unified
@test_source "manuscript"

# Features tested:
- Chapter splitting and navigation
- MathJax integration
- Custom styling and CSS generation
- Image processing and path rewriting
- HTML compliance validation
- Multi-file output structure

EPUB Builder (epub_builder_unified_test.exs)

@moduletag :epub_builder_unified
@test_source "manuscript"

# Features tested:
- EPUB3 structure compliance
- Bibliography and citation handling
- Chapter navigation and TOC
- Image and asset handling
- Metadata preservation
- ZIP archive validation

DOCX Builder (docx_builder_unified_test.exs)

@moduletag :docx_builder_unified
@test_source "manuscript"

# Features tested:
- Microsoft Word format generation
- ZIP structure validation
- Metadata preservation
- Mathematical content handling
- Bibliography integration
- Pandoc dependency management

WikiJS Builder (wikijs_builder_unified_test.exs)

@moduletag :wikijs_builder_unified
@test_source "manuscript"

# Features tested:
- Markdown formatting for WikiJS
- Image path rewriting
- Navigation structure
- External server validation (optional)
- Content structure preservation

HTML-to-Markdown Builder (html_to_markdown_builder_unified_test.exs)

@moduletag :html_to_markdown_unified
@test_source "manuscript"

# Features tested:
- HTML-to-Markdown conversion
- Image copying and path handling
- Table and list conversion
- Mathematical content preservation
- Multi-file processing

2. Content Consistency Tests (content_consistency_unified_test.exs)

The content consistency tests validate that content is properly preserved and converted across all output formats:

@moduletag :unified_content_consistency
@test_source "manuscript"

# Cross-format validation includes:
- Chapter titles consistency
- Front-matter sections (Dedication, Preface, Introduction, Foreword)
- Back-matter sections (Appendix, Glossary, Bibliography, Index)
- List sections (Table of Contents, List of Figures, List of Tables)
- LaTeX cross-references and citations
- Figure and table captions
- Mathematical content preservation

Test Configuration

Module Tags and Setup

All unified tests use consistent module tags and setup patterns:

defmodule Codexis.Builders.ExampleUnifiedTest do
  use ExUnit.Case
  import Patch

  alias Codexis.BuildManager.BuildOrchestrator
  alias Codexis.Core.{Shell, File, PathBuilder}

  @moduletag :example_unified
  @test_source "manuscript"
  @timeout 300_000

  setup_all do
    # Tool availability checking
    required_tools = ["lualatex", "pandoc", "make4ht"]
    missing_tools = Enum.filter(required_tools, fn tool ->
      case Shell.cmd("which", [tool]) do
        {_, 0} -> false
        _ -> true
      end
    end)

    if missing_tools != [] do
      IO.puts("Skipping tests - missing tools: #{Enum.join(missing_tools, ", ")}")
      :skip
    else
      :ok
    end
  end

  setup do
    # Use BuildOrchestrator for proper Task #17 integration
    case BuildOrchestrator.create(@test_source, preserve: false) do
      {:ok, orchestrator} ->
        manuscript_path = BuildOrchestrator.source_dir(orchestrator)
        output_path = BuildOrchestrator.output_dir(orchestrator)
        
        on_exit(fn ->
          BuildOrchestrator.cleanup(orchestrator)
        end)

        {:ok, orchestrator: orchestrator, manuscript_path: manuscript_path, output_path: output_path}
      
      {:error, reason} ->
        raise "Failed to create BuildOrchestrator: #{reason}"
    end
  end
end

Tool Availability Checking

Tests include robust tool availability checking to gracefully skip when required tools are not installed:

defp tools_available?(tools) do
  Enum.all?(tools, fn tool ->
    case Shell.cmd("which", [tool]) do
      {_, 0} -> true
      _ -> false
    end
  end)
end

defp skip_test(reason) do
  IO.puts("Skipping test: #{reason}")
  :ok
end

Cleanup and Temporary Directory Management

The unified tests properly integrate with Task #17's BuildOrchestrator:

# Test cleanup_temp_dir functionality
test "cleanup_temp_dir functionality works correctly" do
  # Test with preserve: true
  {:ok, orchestrator_preserve} = BuildOrchestrator.create(@test_source, preserve: true)
  temp_dir_preserve = orchestrator_preserve.base_dir
  
  assert File.exists?(temp_dir_preserve), "Temp directory should exist during build"
  
  BuildOrchestrator.cleanup(orchestrator_preserve)
  assert File.exists?(temp_dir_preserve), "Temp directory should be preserved when preserve: true"
  
  # Manual cleanup
  File.rm_rf(temp_dir_preserve)
  
  # Test with preserve: false (default)
  {:ok, orchestrator_cleanup} = BuildOrchestrator.create(@test_source, preserve: false)
  temp_dir_cleanup = orchestrator_cleanup.base_dir
  
  BuildOrchestrator.cleanup(orchestrator_cleanup)
  refute File.exists?(temp_dir_cleanup), "Temp directory should be cleaned up when preserve: false"
end

Validation Patterns

Content Validation

Unified tests use consistent validation patterns:

# Basic content validation
defp assert_content_contains(content, expected, message) do
  assert String.contains?(content, expected), message
end

# Lenient validation for PDF extraction variations
defp assert_content_contains_lenient(content, expected, message) do
  expected_words = String.split(expected, " ")
                  |> Enum.filter(&(String.length(&1) > 3))
                  |> Enum.take(3)
  
  if expected_words != [] do
    found_words = Enum.filter(expected_words, &String.contains?(content, &1))
    assert length(found_words) >= min(2, length(expected_words)), 
           "#{message} (checked for words: #{Enum.join(expected_words, ", ")})"
  else
    assert String.contains?(content, expected), message
  end
end

# Text normalization for consistent comparison
defp normalize_text(text) do
  text
  |> String.replace(~r/[''‛`´]/u, "'")
  |> String.replace(~r/[""«»]/u, "\"")
  |> String.replace(~r/^\d+\s*/, "")
  |> String.replace(~r/\s+/, " ")
  |> String.downcase()
  |> String.trim()
end

Format-Specific Validation

Each format has specialized validation functions:

# PDF validation
defp validate_pdf_structure(pdf_path) do
  assert File.exists?(pdf_path), "PDF file should exist"
  assert File.stat!(pdf_path).size > 50_000, "PDF should be substantial size"
  
  case Shell.cmd("pdfinfo", [pdf_path]) do
    {output, 0} ->
      assert String.contains?(output, "Pages:"), "PDF should contain page information"
      assert String.contains?(output, "PDF version:"), "PDF should have version information"
    _ ->
      IO.puts("Warning: pdfinfo not available for PDF validation")
  end
end

# EPUB validation
defp validate_epub_structure(epub_path) do
  case :zip.unzip(String.to_charlist(epub_path), [:memory]) do
    {:ok, files} ->
      file_names = Enum.map(files, fn {name, _data} -> List.to_string(name) end)
      assert "META-INF/container.xml" in file_names, "EPUB should contain container.xml"
      assert Enum.any?(file_names, &String.ends_with?(&1, ".opf")), "EPUB should contain OPF file"
      assert Enum.any?(file_names, &String.ends_with?(&1, ".xhtml")), "EPUB should contain XHTML files"
    {:error, reason} ->
      flunk("Failed to read EPUB file: #{reason}")
  end
end

# DOCX validation
defp validate_docx_structure(docx_path) do
  case :zip.unzip(String.to_charlist(docx_path), [:memory]) do
    {:ok, files} ->
      file_names = Enum.map(files, fn {name, _data} -> List.to_string(name) end)
      assert "[Content_Types].xml" in file_names, "DOCX should contain Content_Types.xml"
      assert Enum.any?(file_names, &String.starts_with?(&1, "word/")), "DOCX should contain word directory"
    {:error, reason} ->
      flunk("Failed to read DOCX file: #{reason}")
  end
end

Format Exception Handling

The unified tests handle known format limitations gracefully:

defp should_skip_section_for_format(section, format) do
  # Define format limitations
  format_exceptions = %{
    "List of Figures" => [:docx],  # Pandoc doesn't support \listoffigures for DOCX
    "List of Tables" => [:docx],   # Pandoc doesn't support \listoftables for DOCX
    "Index" => [:docx]             # DOCX doesn't fully support index generation
  }
  
  exceptions = Map.get(format_exceptions, section, [])
  format in exceptions
end

Performance Considerations

Efficient Testing Strategies

1. Tool Availability: Check once in setup_all to avoid repeated checks
2. Shared Temp Directory: Use BuildOrchestrator for efficient directory management
3. Graceful Degradation: Skip tests when tools are unavailable rather than failing
4. Minimal Test Manuscripts: Support for smaller test documents when needed

Timeout Management

@timeout 300_000  # 5 minutes for complex builds

@tag timeout: @timeout
test "complex format generation", context do
  # Test implementation
end

Test Execution

Running Individual Test Suites

# Run specific unified test suite
mix test test/codexis/builders/pdf_builder_unified_test.exs

# Run all unified tests
mix test --only unified

# Run content consistency tests
mix test test/codexis/builders/content_consistency_unified_test.exs

# Run with specific timeout
mix test test/codexis/builders/content_consistency_unified_test.exs --timeout 600000

Test Tags and Filtering

# Run only PDF builder tests
mix test --only pdf_builder_unified

# Run all unified builder tests
mix test --only unified_builder

# Run content consistency tests
mix test --only unified_content_consistency

# Exclude slow integration tests
mix test --exclude integration

Migration from Legacy Tests

Legacy Test Files (To Be Deprecated)

The following legacy test files have been replaced by unified versions:

test/codexis/builders/
├── pdf_builder_test.exs              📝 Legacy (213 lines) - replaced by pdf_builder_unified_test.exs
├── html_builder_test.exs             📝 Legacy (114 lines) - replaced by html_builder_unified_test.exs
├── epub_builder_test.exs             📝 Legacy (192 lines) - replaced by epub_builder_unified_test.exs
├── docx_builder_test.exs             📝 Legacy - replaced by docx_builder_unified_test.exs
├── wikijs_builder_test.exs           📝 Legacy - replaced by wikijs_builder_unified_test.exs
├── html_to_markdown_builder_test.exs 📝 Legacy - replaced by html_to_markdown_builder_unified_test.exs
└── content_consistency_test.exs      📝 Legacy (1,880 lines) - replaced by content_consistency_unified_test.exs

Migration Benefits

1. Code Reduction: ~30-35% reduction in test code duplication
2. Improved Maintainability: Consistent patterns and shared utilities
3. Enhanced Validation: More comprehensive testing with real manuscript content
4. Better Integration: Proper Task #17 BuildOrchestrator integration
5. Performance: Efficient temporary directory management

Best Practices

1. Test Structure

• Use consistent module tags and setup patterns
• Implement proper tool availability checking
• Include timeout configurations for long-running tests
• Use BuildOrchestrator for temporary directory management

2. Validation

• Normalize text content for consistent comparison
• Use lenient matching for PDF extraction variations
• Handle format-specific limitations gracefully
• Validate both structure and content

3. Error Handling

• Gracefully skip tests when tools are unavailable
• Provide informative error messages
• Handle external dependencies (e.g., servers) gracefully
• Include cleanup in error scenarios

4. Documentation

• Document test purpose and scope clearly
• Include examples of expected behavior
• Document known limitations and workarounds
• Maintain up-to-date tool requirements

Future Enhancements

Potential Improvements

1. Parallel Test Execution: Optimize test execution time with parallel builds
2. Enhanced Validation: More sophisticated content validation algorithms
3. Coverage Analysis: Automated test coverage reporting
4. Performance Benchmarking: Automated performance regression detection
5. CI/CD Integration: Enhanced continuous integration testing

Extension Points

The unified architecture is designed to be extensible:

1. New Format Builders: Easy addition of new output formats
2. Enhanced Validation: Pluggable validation strategies
3. Custom Test Manuscripts: Support for domain-specific test content
4. Advanced Metrics: Integration with testing and performance metrics

Conclusion

The unified test architecture provides a robust, maintainable, and comprehensive testing strategy for the Codexis project. By using manuscript.tex as the canonical test book and implementing consistent patterns across all tests, the architecture ensures reliable validation of all output formats while reducing code duplication and improving maintainability.

The integration with Task #17's BuildOrchestrator ensures efficient temporary directory management, while the comprehensive validation strategies provide confidence in the quality and consistency of generated outputs across all supported formats.

Standard for Temporary File and Directory Management in Tests: Using Briefly

Overview

All tests must use the Briefly library (or the project-standard wrappers in Codexis.TestHelpers) for creating and managing temporary files and directories. Direct use of File.mkdir_p!, File.write!, or hardcoded temp/test directory paths is prohibited for temporary resources. Persistent test data (e.g., manuscript/, test/fixtures/) must be treated as read-only and only copied for test use.

Rationale

• Ensures test isolation and automatic cleanup of temp resources
• Prevents permission errors and test pollution across runs
• Enforces portability and reliability in CI and local environments
• Aligns with ADR-0035: Use Briefly for Temporary Files and Directories in Tests

Required Patterns

• Always use Briefly.create/1 or the provided helpers in Codexis.TestHelpers for temp files/dirs
• Never write to or modify files in manuscript/ or test/fixtures/ during tests
• Copy persistent data to a temp location before modification
• Remove all manual cleanup logic (on_exit) where Briefly is used

Example: Creating a Temp Directory

setup :create_temp_dir

test "my test", %{temp_dir: temp_dir} do
  # Use temp_dir for test files
end

Example: Creating a Temp File with Content

setup context do
  create_temp_file_with_content(context, "test content")
end

test "my test", %{temp_file: temp_file} do
  assert File.read!(temp_file) == "test content"
end

Example: Copying Canonical Manuscript for Test Use

setup context do
  copy_to_temp_dir(context, "manuscript")
end

test "my test", %{temp_dir: temp_dir} do
  # Use temp_dir/manuscript as a writable copy
end

Helper Functions

See test/support/test_helpers.ex for:

• create_temp_dir/1
• create_temp_file/1
• create_temp_file_with_content/2
• create_temp_dir_with_subdirs/2
• copy_to_temp_dir/2

Migration Notes

• All legacy tests have been migrated to use Briefly and helpers
• No test writes to persistent data; all modifications occur in temp dirs
• Manual cleanup logic has been removed where Briefly is used
• See migration-mox-to-patch.md for related test modernization

Enforcement

• New tests must follow this standard
• CI and code review will reject tests that do not use Briefly for temp resources
• Violations may be flagged by future linter or static analysis tools