Unified Build System Testing Strategy

This document outlines the testing strategy for the Codexis unified build system, which uses a shared temporary directory for all output formats to improve build efficiency.

Testing Goals

The testing strategy aims to verify:

1. Functionality: Ensure the unified build system works correctly for all output formats
2. Efficiency: Measure and validate performance improvements
3. Correctness: Verify outputs match those from the traditional build approach
4. Resilience: Test error handling and recovery mechanisms
5. Compatibility: Ensure compatibility with existing workflows and configurations

Test Categories

1. Unit Tests

Unit tests focus on individual components of the unified build system:

• BuildOrchestrator Tests (build_orchestrator_test.exs):

  • Directory creation and structure
  • File copying and permissions
  • Directory accessors
  • Cleanup functionality
  • Concurrent orchestrator creation
  • Temp directory listing and management

• FileManager Tests (file_manager_test.exs):

  • File and directory copying
  • Directory creation
  • File reading and writing
  • File deletion
  • Directory structure creation
  • Source-to-temp copying
  • Output copying

2. Integration Tests

Integration tests verify the interaction between different components:

• Multiple Format Builds (unified_build_system_test.exs):

  • Building multiple formats (PDF, HTML, EPUB) using a shared temporary directory
  • Verifying auxiliary files are reused across formats
  • Testing concurrent builds with separate orchestrators

• Error Handling (unified_build_system_test.exs):

  • Testing how the system handles missing source directories
  • Testing how the system handles build failures

3. Performance Tests

Performance tests measure the efficiency improvements:

• Build Time Comparison (unified_build_system_test.exs):
  • Comparing build times between traditional and unified approaches
  • Measuring resource utilization (memory, disk I/O)
  • Testing with large documents to verify scalability

4. Validation Tests

Validation tests compare outputs from the unified system with those from the traditional approach:

• Output Comparison (build_validation_test.exs):
  • Comparing PDF outputs
  • Comparing HTML outputs
  • Verifying content integrity and formatting consistency

5. Specific Test Cases

Tests with specific document types:

• Complex Documents (build_validation_test.exs):
  • Documents with tables, figures, and equations
  • Documents with different languages and character sets

6. Regression Tests

Tests to ensure compatibility with existing functionality:

• Command-line Options (build_validation_test.exs):
  • Testing with various combinations of options
  • Verifying cross-reference handling

Test Execution

Running the Tests

To run all tests for the unified build system:

mix test test/codexis/build_manager/build_orchestrator_test.exs test/codexis/core/file_manager_test.exs test/codexis/integration/unified_build_system_test.exs test/codexis/integration/build_validation_test.exs

To run only integration tests:

mix test --only integration

To run only validation tests:

mix test --only validation

To run only performance tests:

mix test --only performance

Test Fixtures

The tests use the following fixtures:

• test/fixtures/test_manuscript: A simple test manuscript
• test/fixtures/complex_manuscript: A manuscript with complex elements (tables, figures, equations)
• test/fixtures/multilingual_manuscript: A manuscript with content in different languages

Expected Results

Unit Tests

All unit tests should pass, verifying that individual components work as expected.

Integration Tests

Integration tests should demonstrate:

• Multiple formats can be built using a shared temporary directory
• Auxiliary files are generated only once and reused across formats
• Concurrent builds work correctly with separate orchestrators
• Error handling mechanisms work as expected

Performance Tests

Performance tests should show:

• The unified build system is faster than the traditional approach
• The improvement is more significant for multiple format builds
• Resource utilization is more efficient

Validation Tests

Validation tests should confirm:

• Outputs from the unified system match those from the traditional approach
• Complex documents are handled correctly
• Multilingual content is processed properly

Regression Tests

Regression tests should verify:

• Compatibility with existing command-line options
• Cross-references work correctly

Test Results Documentation

After running the tests, document the results including:

1. Pass/fail status for each test category
2. Performance metrics (build times, improvement percentages)
3. Any issues or edge cases discovered
4. Recommendations for further improvements

Continuous Integration

These tests should be integrated into the CI/CD pipeline to ensure the unified build system continues to work correctly after future changes.