BUCKYVERSE

Incremental Migration Plan for Task #18

Overview

This document outlines a safe, incremental migration strategy for consolidating builder and content consistency tests around manuscript.tex as the canonical test book. The plan minimizes disruption to CI/CD pipelines while ensuring comprehensive test coverage throughout the transition.

Migration Principles

1. Zero-Downtime Migration

2. Risk Mitigation

3. Quality Assurance

Migration Phases

Phase 1: Foundation Setup (Days 1-10)

Day 1-2: Environment Preparation

Objectives: Prepare development environment for incremental migration

Tasks:

  1. Create Migration Branch

    git checkout -b task-18-test-consolidation
git checkout -b phase-1-foundation

  2. Setup Migration Tracking

    # Create migration status file
touch migration_status.json
# Initialize with current test inventory

  3. Backup Current Test State

    # Create snapshot of current test directory
cp -r test/ test_backup_$(date +%Y%m%d)

  4. Validate Current Test Suite

    # Run full test suite to establish baseline
mix test --cover
# Document current coverage and performance metrics

Success Criteria:

Day 3-5: Core Utility Implementation

Objectives: Implement core shared utilities without disrupting existing tests

Tasks:

  1. Create Shared Utility Directory

    mkdir -p test/support/builders/unified

  2. Implement ManuscriptFixtures Module

    • Create test/support/builders/unified/manuscript_fixtures.ex
    • Implement core manuscript access functions
    • Add comprehensive testing for the module itself

  3. Implement UnifiedTestConfig Module

    • Create test/config/unified_test_config.ex
    • Implement configuration management
    • Add environment variable support

  4. Create Foundation Tests

    # Test the new utilities themselves
mkdir -p test/support/unified
touch test/support/unified/manuscript_fixtures_test.exs
touch test/support/unified/unified_test_config_test.exs

Success Criteria:

Day 6-8: TestOrchestrator Implementation

Objectives: Implement build orchestration without affecting existing tests

Tasks:

  1. Implement TestOrchestrator Module

    • Create test/support/builders/unified/test_orchestrator.ex
    • Implement build coordination functions
    • Add integration with Task #17's BuildOrchestrator

  2. Create BuildManager Integration

    • Create test/support/builders/unified/build_manager.ex
    • Implement shared temporary directory support
    • Add cleanup policy management

  3. Test Orchestrator Validation

    # Create isolated test for orchestrator
touch test/support/unified/test_orchestrator_test.exs

  4. Integration Testing

    # Test integration with existing build system
mix test test/support/unified/ --only integration

Success Criteria:

Day 9-10: Enhanced Utilities

Objectives: Enhance existing utilities and create validation suite

Tasks:

  1. Enhance FormatExtractors

    • Extend existing test/support/builders/content_consistency/format_extractors.ex
    • Add new extraction functions without breaking existing ones
    • Implement caching and performance optimizations

  2. Enhance TestHelpers

    • Extend existing test/support/builders/content_consistency/test_helpers.ex
    • Add new helper functions while preserving existing interface
    • Implement enhanced assertion helpers

  3. Create ValidationSuite

    • Create test/support/builders/unified/validation_suite.ex
    • Implement comprehensive validation functions
    • Add cross-format consistency validation

  4. Foundation Validation

    # Run comprehensive test of all new utilities
mix test test/support/unified/ --cover

Success Criteria:

Phase 2: Parallel Implementation (Days 11-25)

Day 11-13: First Pilot Migration (PDF Tests)

Objectives: Create first unified test module alongside existing tests

Tasks:

  1. Create Unified Test Structure

    mkdir -p test/codexis/builders/unified/format_generation
mkdir -p test/codexis/builders/unified/content_validation
mkdir -p test/codexis/builders/unified/integration
mkdir -p test/codexis/builders/unified/specialized

  2. Implement PDF Generation Tests

    • Create test/codexis/builders/unified/format_generation/pdf_generation_tests.ex
    • Use manuscript.tex as canonical source
    • Implement comprehensive content validation
    • Preserve all functionality from existing PDF builder test

  3. Parallel Execution Setup

    # Configure CI to run both old and new PDF tests
# Add new test to test suite without removing old one

  4. Validation and Comparison

    # Run both old and new PDF tests
mix test test/codexis/builders/pdf_builder_test.exs
mix test test/codexis/builders/unified/format_generation/pdf_generation_tests.exs
# Compare coverage and performance

Success Criteria:

Day 14-16: Content Validation Migration

Objectives: Migrate content consistency tests to unified structure

Tasks:

  1. Migrate Modular Content Tests

    • Copy test/codexis/builders/content_consistency/ to test/codexis/builders/unified/content_validation/
    • Update imports to use new unified utilities
    • Enhance with new ValidationSuite capabilities

  2. Update Test References

    • Update all internal references to use new utilities
    • Preserve existing test interface for backward compatibility
    • Add enhanced validation capabilities

  3. Parallel Validation

    # Run both old and new content consistency tests
mix test test/codexis/builders/content_consistency/
mix test test/codexis/builders/unified/content_validation/

  4. Performance Comparison

    # Compare execution time and resource usage
time mix test test/codexis/builders/content_consistency/
time mix test test/codexis/builders/unified/content_validation/

Success Criteria:

Day 17-19: HTML Generation Migration

Objectives: Migrate HTML builder tests (already using real manuscript)

Tasks:

  1. Migrate HTML Builder Tests

    • Create test/codexis/builders/unified/format_generation/html_generation_tests.ex
    • Update to use TestOrchestrator for consistency
    • Integrate with ValidationSuite
    • Preserve chapter splitting and navigation tests

  2. Integration Enhancement

    • Add cross-format consistency validation
    • Implement shared temporary directory usage
    • Add performance monitoring

  3. Validation

    # Run parallel tests
mix test test/codexis/builders/html_builder_test.exs
mix test test/codexis/builders/unified/format_generation/html_generation_tests.exs

Success Criteria:

Day 20-22: EPUB and DOCX Migration

Objectives: Migrate remaining major format builders

Tasks:

  1. Migrate EPUB Builder Tests

    • Create test/codexis/builders/unified/format_generation/epub_generation_tests.ex
    • Replace mock content with manuscript.tex usage
    • Integrate with shared build system
    • Preserve EPUB-specific technical validation

  2. Migrate DOCX Builder Tests

    • Create test/codexis/builders/unified/format_generation/docx_generation_tests.ex
    • Replace inline content with manuscript.tex
    • Add comprehensive content validation
    • Preserve dependency checking and error handling

  3. Cross-Format Integration

    # Test all migrated formats together
mix test test/codexis/builders/unified/format_generation/

Success Criteria:

Day 23-25: Specialized Test Migration

Objectives: Migrate converter and infrastructure tests

Tasks:

  1. Migrate Converter Tests

    • Consolidate various converter tests in test/codexis/builders/unified/specialized/
    • Create latex_processing_tests.ex, converter_tests.ex
    • Preserve targeted feature testing while adding manuscript integration

  2. Migrate Infrastructure Tests

    • Create infrastructure_tests.ex for temp cleanup, directory management
    • Integrate with shared temporary directory system
    • Add comprehensive error handling tests

  3. Create Integration Test Suite

    • Create test/codexis/builders/unified/integration/unified_build_tests.ex
    • Implement end-to-end testing of complete build pipeline
    • Add performance and error handling tests

Success Criteria:

Phase 3: Validation and Transition (Days 26-35)

Day 26-28: Comprehensive Validation

Objectives: Validate complete unified test suite against existing tests

Tasks:

  1. Full Test Suite Comparison

    # Run complete old test suite
time mix test test/codexis/builders/ --exclude unified --cover

# Run complete new test suite
time mix test test/codexis/builders/unified/ --cover

# Compare results

  2. Coverage Analysis

    # Generate detailed coverage reports
mix test --cover --export-coverage old_coverage
mix test test/codexis/builders/unified/ --cover --export-coverage new_coverage

# Compare coverage reports
mix test.coverage --export-coverage old_coverage new_coverage

  3. Performance Analysis

    # Run performance comparison
mix test.bench test/codexis/builders/ --exclude unified
mix test.bench test/codexis/builders/unified/

  4. Gap Analysis

    • Identify any missing functionality
    • Validate all test scenarios preserved
    • Check cross-format consistency improvements

Success Criteria:

Day 29-31: CI/CD Integration

Objectives: Integrate unified test suite into CI/CD pipeline

Tasks:

  1. Update CI Configuration

    # Update .github/workflows/test.yml
- name: Run unified test suite
  run: |
    mix test test/codexis/builders/unified/
    
- name: Run legacy tests (parallel)
  run: |
    mix test test/codexis/builders/ --exclude unified

  2. Add Performance Monitoring

    - name: Performance comparison
  run: |
    echo "Old test suite:" 
    time mix test test/codexis/builders/ --exclude unified
    echo "New test suite:"
    time mix test test/codexis/builders/unified/

  3. Add Coverage Reporting

    - name: Coverage comparison
  run: |
    mix test --cover --export-coverage old_coverage test/codexis/builders/ --exclude unified
    mix test --cover --export-coverage new_coverage test/codexis/builders/unified/
    mix test.coverage --compare old_coverage new_coverage

  4. Add Artifact Preservation

    - name: Preserve test artifacts on failure
  if: failure()
  run: |
    find /tmp -name "codexis_test_*" -type d -exec cp -r {} test-artifacts/ \;

Success Criteria:

Day 32-35: Gradual Cutover

Objectives: Begin transition from old to new test suite

Tasks:

  1. Feature Flag Implementation

    # Add environment variable to control test execution
export CODEXIS_USE_UNIFIED_TESTS=true

  2. Gradual Format Transition

    # Day 32: Transition PDF tests
# Disable old PDF tests, rely on new ones

# Day 33: Transition HTML and content consistency tests
# Day 34: Transition EPUB and DOCX tests  
# Day 35: Transition specialized tests

  3. Monitoring and Validation

    # Monitor each transition day
mix test test/codexis/builders/unified/ --cover
# Validate no regressions introduced

  4. Rollback Preparation

    # Maintain ability to quickly rollback if issues found
git tag pre-unified-tests

Success Criteria:

Phase 4: Cleanup and Finalization (Days 36-42)

Day 36-38: Remove Legacy Tests

Objectives: Remove old test files and cleanup redundant code

Tasks:

  1. Archive Legacy Tests

    # Move old tests to archive directory
mkdir -p test/archived/pre_unified/
mv test/codexis/builders/pdf_builder_test.exs test/archived/pre_unified/
mv test/codexis/builders/epub_builder_test.exs test/archived/pre_unified/
mv test/codexis/builders/docx_builder_test.exs test/archived/pre_unified/
mv test/codexis/builders/wikijs_builder_test.exs test/archived/pre_unified/
mv test/codexis/builders/content_consistency_test.exs test/archived/pre_unified/

  2. Remove Redundant Code

    # Remove duplicate validation logic
# Remove unused test utilities
# Remove redundant test data files

  3. Update Test References

    # Update any remaining references to old test files
# Update documentation references
# Update script references

  4. Cleanup Validation

    # Run full test suite to ensure no broken references
mix test
# Validate all functionality still working

Success Criteria:

Day 39-40: Documentation Updates

Objectives: Update all documentation to reflect new test architecture

Tasks:

  1. Update Test Documentation

    • Update test/README.md
    • Create docs/testing-architecture.md
    • Update CLAUDE.md test section

  2. Create Migration Documentation

    • Document migration process completed
    • Create troubleshooting guide
    • Document new test patterns

  3. Update Development Guides

    • Update contributor guidelines
    • Update testing best practices
    • Update CI/CD documentation

  4. API Documentation

    • Add comprehensive module documentation
    • Create usage examples
    • Document configuration options

Success Criteria:

Day 41-42: Final Validation

Objectives: Complete final validation and optimization

Tasks:

  1. Comprehensive Test Run

    # Run complete test suite multiple times
for i in {1..5}; do
  echo "Test run $i:"
  time mix test --cover
done

  2. Performance Optimization

    # Profile test execution
mix test --trace
# Optimize any slow tests identified

  3. Coverage Verification

    # Generate final coverage report
mix test --cover
# Verify coverage meets or exceeds original baseline

  4. Final Documentation Review

    • Review all documentation for accuracy
    • Validate code examples work
    • Check for any missing information

Success Criteria:

Risk Management

High-Risk Scenarios and Mitigation

Scenario 1: CI/CD Pipeline Failure

Risk: New tests break CI/CD pipeline, blocking development

Mitigation:

Recovery Process:

# Quick rollback if CI fails
git revert <last-migration-commit>
# or
export CODEXIS_USE_UNIFIED_TESTS=false

Scenario 2: Performance Regression

Risk: New test suite significantly slower than original

Mitigation:

Recovery Process:

# If performance regression detected
mix test.bench --compare baseline.bench current.bench
# Identify slow tests and optimize
# Revert specific changes if necessary

Scenario 3: Test Coverage Loss

Risk: Migration results in reduced test coverage

Mitigation:

Recovery Process:

# If coverage loss detected
mix test --cover --export-coverage current
mix test.coverage --compare baseline.coverage current.coverage
# Identify missing coverage and add tests

Scenario 4: Functionality Loss

Risk: Migration loses important test functionality

Mitigation:

Recovery Process:

# If functionality loss detected
# Identify missing functionality from mapping document
# Implement missing functionality in new test structure
# Validate with side-by-side comparison

Monitoring and Alerts

CI/CD Monitoring

# Add to CI pipeline
- name: Monitor test performance
  run: |
    TEST_TIME=$(time mix test 2>&1 | grep real | awk '{print $2}')
    if [[ $TEST_TIME > "10m" ]]; then
      echo "WARNING: Test execution time exceeded 10 minutes"
      exit 1
    fi

- name: Monitor test coverage
  run: |
    COVERAGE=$(mix test --cover | grep "Total coverage" | awk '{print $3}' | sed 's/%//')
    if [[ $COVERAGE < 85 ]]; then
      echo "WARNING: Test coverage below 85%"
      exit 1
    fi

Automated Alerts

Success Metrics

Quantitative Metrics

Test Execution Performance

Test Coverage

Code Reduction

Maintenance Effort

Qualitative Metrics

Code Quality

Developer Experience

System Reliability

Communication Plan

Stakeholder Updates

Daily Standups

Weekly Reports

Phase Completion Reports

Documentation Updates

Real-time Updates

Phase Completion Documentation

Rollback Strategy

Immediate Rollback (Emergency)

# If critical issue discovered
git checkout main
git revert --no-edit $(git log --oneline task-18-test-consolidation..HEAD | awk '{print $1}')

Selective Rollback (Specific Issues)

# Rollback specific test module
git checkout main -- test/codexis/builders/old_module_test.exs
# Temporarily disable new module
mv test/codexis/builders/unified/new_module_tests.ex test/disabled/

Gradual Rollback (Performance Issues)

# Gradually revert to old tests while keeping infrastructure
export CODEXIS_USE_UNIFIED_TESTS=false
# Keep new utilities but use old test structure temporarily

This incremental migration plan provides a comprehensive, low-risk approach to consolidating the test suites while maintaining full CI/CD functionality and preserving all existing test capabilities throughout the transition process.