BUCKYVERSE

Pipeline Migration Testing Guide

This document provides guidelines and best practices for testing the migration from direct builders to the pipeline architecture, ensuring feature parity and compatibility.

Testing Approach

The pipeline migration framework requires thorough testing to ensure a smooth transition. The testing approach includes:

  1. Feature Parity Testing: Comparing features between direct builders and pipeline
  2. Compatibility Testing: Ensuring outputs are equivalent between both systems
  3. Performance Testing: Comparing processing times and resource usage
  4. Integration Testing: Verifying that both systems work with existing APIs and interfaces
  5. Regression Testing: Ensuring new pipeline components don't introduce regressions

Testing Tools

Compatibility Test Harness

The framework includes a dedicated compatibility test harness:

# Test all formats
mix codexis.test_pipeline_compatibility

# Test a specific format
mix codexis.test_pipeline_compatibility --format=pdf

# Save report to a specific location
mix codexis.test_pipeline_compatibility --output=reports/my_report.json

# Update the feature registry with test results
mix codexis.test_pipeline_compatibility --update-registry

# Enable verbose output
mix codexis.test_pipeline_compatibility --verbose

Feature Matrix Tool

The feature matrix tool helps identify feature gaps:

# Show feature matrix for a format
mix codexis.pipeline_migration feature-matrix --format=epub

Report Generation

Generate comprehensive reports on migration status:

# Generate feature parity report
mix codexis.pipeline_migration generate-report

Setting Up Test Cases

Creating Test Documents

Test documents should cover a range of complexity levels and features:

  1. Simple Documents: Basic formatting, minimal features
  2. Moderate Documents: Tables, images, links, footnotes
  3. Complex Documents: All supported features, edge cases
  4. Real-World Documents: Actual user documents from production

Sample Test Document Structure

# In test/fixtures/migration_test_docs/
defmodule Codexis.TestDocuments do
  def simple_document do
    %{
      content_path: "test/fixtures/simple",
      metadata: %{
        title: "Simple Test Document",
        author: "Test Suite",
        language: "en"
      },
      options: [
        format: :pdf,
        include_toc: true
      ]
    }
  end
  
  def complex_document do
    %{
      content_path: "test/fixtures/complex",
      metadata: %{
        title: "Complex Test Document",
        author: "Test Suite",
        language: "en"
      },
      options: [
        format: :pdf,
        include_toc: true,
        toc_depth: 3,
        optimize_assets: true
      ]
    }
  end
end

Validation Strategies

Output Comparison

Compare outputs from both systems using:

  1. Binary Comparison: For exact binary matches (PDF, EPUB)
  2. Semantic Comparison: For text-based formats (HTML, Markdown)
  3. Structure Comparison: For checking document structure regardless of minor formatting differences

Example Validation Function

defmodule Codexis.Testing.OutputValidator do
  def compare_outputs(builder_output, pipeline_output, format) do
    case format do
      # For binary formats, use checksums
      format when format in [:pdf, :epub, :mobi] ->
        builder_checksum = calculate_checksum(builder_output)
        pipeline_checksum = calculate_checksum(pipeline_output)
        {builder_checksum == pipeline_checksum, %{similarity: similarity_score(format, builder_output, pipeline_output)}}
        
      # For text formats, use semantic comparison
      format when format in [:html, :markdown, :text] ->
        similarity = semantic_similarity(builder_output, pipeline_output)
        {similarity > 0.95, %{similarity: similarity}}
        
      # For other formats, use custom validators
      _ ->
        custom_validation(format, builder_output, pipeline_output)
    end
  end
  
  # Implementation details...
end

Feature Support Validation

Validate that specific features are supported in both systems:

  1. Create a feature checklist for each format
  2. Test each feature individually in both systems
  3. Document any feature gaps or differences
  4. Update the feature registry with results

Automated Testing

CI/CD Integration

Integrate migration testing into your CI/CD pipeline:

# In .github/workflows/pipeline_migration.yml
name: Pipeline Migration Tests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v2
    
    - name: Set up Elixir
      uses: erlef/setup-beam@v1
      with:
        elixir-version: '1.14.2'
        otp-version: '25.1'
        
    - name: Restore dependencies cache
      uses: actions/cache@v2
      with:
        path: deps
        key: ${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }}
        restore-keys: ${{ runner.os }}-mix-
        
    - name: Install dependencies
      run: mix deps.get
      
    - name: Run migration compatibility tests
      run: mix codexis.test_pipeline_compatibility
      
    - name: Generate compatibility report
      run: mix codexis.pipeline_migration generate-report
      
    - name: Upload test results
      uses: actions/upload-artifact@v2
      with:
        name: compatibility-reports
        path: output/migration_reports/

Scheduled Tests

Schedule regular compatibility tests to track migration progress:

# In lib/codexis/pipeline/migration/scheduled_tests.ex
defmodule Codexis.Pipeline.Migration.ScheduledTests do
  use GenServer
  require Logger
  
  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end
  
  @impl true
  def init(opts) do
    # Schedule the first test
    schedule_next_test()
    {:ok, opts}
  end
  
  @impl true
  def handle_info(:run_tests, state) do
    Logger.info("Running scheduled migration compatibility tests")
    
    # Run tests and update registry
    Codexis.Pipeline.Migration.TestHarness.run_migration_test_suite(update_registry: true)
    
    # Schedule the next test
    schedule_next_test()
    {:noreply, state}
  end
  
  defp schedule_next_test do
    # Run tests daily at a specified time
    Process.send_after(self(), :run_tests, 24 * 60 * 60 * 1000)
  end
end

Performance Testing

Benchmark Both Systems

Compare performance metrics between direct builders and pipeline:

# In lib/codexis/pipeline/migration/performance_benchmarks.ex
defmodule Codexis.Pipeline.Migration.PerformanceBenchmarks do
  def run_benchmarks(format, document, metadata, opts) do
    # Warm up
    run_builder(format, document, metadata, opts)
    run_pipeline(format, document, metadata, opts)
    
    # Measure builder performance
    {builder_time, builder_result} = :timer.tc(fn -> 
      run_builder(format, document, metadata, opts)
    end)
    
    # Measure pipeline performance
    {pipeline_time, pipeline_result} = :timer.tc(fn -> 
      run_pipeline(format, document, metadata, opts)
    end)
    
    # Compare memory usage
    builder_memory = measure_memory_usage(:builder, format)
    pipeline_memory = measure_memory_usage(:pipeline, format)
    
    # Return benchmark results
    %{
      format: format,
      time_comparison: %{
        builder_time: builder_time / 1_000_000,
        pipeline_time: pipeline_time / 1_000_000,
        difference_percentage: ((pipeline_time - builder_time) / builder_time) * 100
      },
      memory_comparison: %{
        builder_memory: builder_memory,
        pipeline_memory: pipeline_memory,
        difference_percentage: ((pipeline_memory - builder_memory) / builder_memory) * 100
      },
      results: %{
        builder: builder_result,
        pipeline: pipeline_result
      }
    }
  end
  
  # Implementation details...
end

Performance Thresholds

Define performance requirements for the pipeline architecture:

Progress Monitoring

Migration Dashboard

Use the web interface to monitor migration progress:

  1. Access /pipeline-migration in your browser
  2. Review format-specific parity percentages
  3. Monitor phase transitions over time
  4. Track feature gap closures

Migration Reports

Generate and compare migration reports over time:

# Generate baseline report
mix codexis.pipeline_migration generate-report --output=reports/baseline.json

# After development iterations, generate comparison report
mix codexis.pipeline_migration generate-report --output=reports/current.json

# Compare reports to track progress
mix codexis.pipeline_migration compare-reports --baseline=reports/baseline.json --current=reports/current.json

Testing Best Practices

  1. Start Small: Begin with the simplest document formats and features
  2. Incremental Testing: Test one feature at a time before moving to more complex scenarios
  3. Document Gaps: Clearly document any feature gaps or compatibility issues
  4. Prioritize Fixes: Focus on fixing high-impact issues first
  5. Automate Where Possible: Create automated test suites for regression testing
  6. Test in All Phases: Ensure all migration phases work correctly
  7. Test Fallback Mechanisms: Verify that Phase 2 fallback functionality works properly
  8. User Acceptance Testing: Have actual users test both systems with real documents

Conclusion

A thorough testing strategy is essential for a successful migration from direct builders to the pipeline architecture. By following this guide, you can ensure feature parity, maintain compatibility, and provide a smooth transition for users.