Pipeline Architecture Migration Framework

This document provides comprehensive information about the migration framework used to transition from direct builders to the new pipeline architecture while maintaining backward compatibility and feature parity.

Overview

The Pipeline Migration Framework provides a controlled, phased approach to transitioning from the direct builder architecture to the new pipeline architecture. The framework includes:

• Configurable migration phases
• Format-specific migration controls
• Feature parity tracking
• Fallback mechanisms
• Web and CLI interfaces for migration management
• Comprehensive testing tools

Migration Phases

The framework implements a 3-phase migration approach:

Phase 1: Parallel Operation (Default)

• Both direct builders and pipeline architecture operate in parallel
• Direct builders are used by default
• Pipeline is used only for formats that are explicitly marked as ready
• Useful for initial testing and validation

Phase 2: Primary Pipeline with Fallback

• Pipeline architecture becomes the primary implementation
• Falls back to direct builders if errors occur
• Provides safety net during transition
• Recommended for formats with at least 70% feature parity

Phase 3: Pipeline-Only Operation

• Pipeline architecture is used exclusively
• No fallback to direct builders
• Requires comprehensive feature parity (90%+)
• Final stage of migration

Additionally, two special modes are available:

• Pipeline Only: Forces exclusive use of the pipeline architecture (similar to Phase 3)
• Builder Only: Forces exclusive use of direct builders (bypasses migration)

Configuration

Application Configuration

Configure default migration settings in your application configuration:

# In config/config.exs
config :codexis,
  initial_pipeline_phase: :phase1,
  pipeline_ready_formats: [:pdf, :html, :markdown],
  pipeline_formats_with_issues: [:mobi]

Runtime Configuration

Migration phases can be set at runtime through:

1. Web Interface: /pipeline-migration dashboard
2. CLI Commands: Using mix tasks (see below)
3. Programmatic API: Through the Manager module

Format-Specific Overrides

Each format can have its own phase setting, which overrides the global setting:

# Set a format-specific phase
Codexis.Pipeline.Migration.Manager.set_phase(:epub, :phase2)

# Set global phase
Codexis.Pipeline.Migration.Manager.set_global_phase(:phase2)

Usage

Web Interface

The web interface provides a dashboard for managing migration phases:

1. Access /pipeline-migration in your browser
2. View current migration status for all formats
3. Update global phase or individual format phases
4. Generate feature parity reports
5. View format-specific feature matrices

CLI Interface

The framework provides a comprehensive CLI for managing migration:

# Show current migration status
mix codexis.pipeline_migration status

# Set phase for a specific format
mix codexis.pipeline_migration set-phase --format=pdf --phase=phase2

# Set global migration phase
mix codexis.pipeline_migration set-global-phase --phase=phase1

# Generate feature parity report
mix codexis.pipeline_migration generate-report --output=reports/parity.json

# Show feature matrix for a format
mix codexis.pipeline_migration feature-matrix --format=epub

Testing Compatibility

The framework includes a dedicated testing tool:

# Test all formats
mix codexis.test_pipeline_compatibility

# Test a specific format
mix codexis.test_pipeline_compatibility --format=pdf

# Save report to a specific location
mix codexis.test_pipeline_compatibility --output=reports/my_report.json

# Update the feature registry with test results
mix codexis.test_pipeline_compatibility --update-registry

Implementation Details

Manager Module

The Codexis.Pipeline.Migration.Manager module serves as the central control point for the migration, providing functions to:

• Determine whether to use pipeline or direct builders
• Route build requests to the appropriate system
• Manage migration phases
• Handle fallback mechanisms

Key functions:

# Check if pipeline should be used for a format
Manager.use_pipeline?(:pdf)

# Get current phase for a format
phase = Manager.current_phase(:html)

# Update phase for a specific format
Manager.set_phase(:epub, :phase2)

# Update global phase
Manager.set_global_phase(:phase3)

Adapter Interface

The Codexis.Pipeline.Migration.PipelineAdapter module provides a compatibility layer that allows the pipeline architecture to be used with the existing direct builder interface:

# Call pipeline through the adapter
{:ok, output_path} = PipelineAdapter.build(document, metadata, opts)

# Validate output
:ok = PipelineAdapter.validate(output_path, metadata, opts)

Configuration Mapping

The Codexis.Pipeline.Migration.ConfigMapper module translates configuration options between direct builders and pipeline:

# Convert builder options to pipeline configuration
{:ok, pipeline_config} = ConfigMapper.builder_to_pipeline(:pdf, opts)

# Convert pipeline configuration to builder options
{:ok, builder_opts} = ConfigMapper.pipeline_to_builder(pipeline_config)

Feature Registry

The Codexis.Pipeline.Migration.FeatureRegistry module tracks feature support across both architectures:

# Get feature registry
registry = FeatureRegistry.get_registry()

# Check feature parity percentage for a format
percentage = FeatureRegistry.parity_percentage(:pdf)

# Get feature gaps
gaps = FeatureRegistry.feature_gaps(:epub)

Decision Logic

The framework uses the following logic to determine which system to use:

1. Check for explicit overrides (force_pipeline or disable_pipeline options)
2. Check format-specific phase setting
3. Check global phase setting
4. Apply phase-specific logic:
   • Phase 1: Use pipeline only for ready formats
   • Phase 2: Use pipeline by default, with fallback to builders
   • Phase 3: Use pipeline exclusively

Troubleshooting

Common Issues

1. Format using unexpected system

   • Check format-specific overrides
   • Verify format is in the pipeline_ready_formats list
   • Check if format is in the pipeline_formats_with_issues list

2. Pipeline failures with fallback

   • Enable verbose logging to see detailed error reports
   • Check compatibility test reports for feature gaps
   • Consider lowering the format to Phase 1 until issues are resolved

3. Migration phase not persisting

   • Ensure configuration is properly stored
   • Verify application is not resetting configuration on restart

Debugging

Enable verbose output for detailed information:

# Web debug mode
PIPELINE_DEBUG=true mix phx.server

# CLI verbose mode
mix codexis.pipeline_migration status --verbose
mix codexis.test_pipeline_compatibility --verbose

Monitoring and Metrics

The framework emits telemetry events for monitoring:

• :codexis, :pipeline, :migration, :build
• :codexis, :pipeline, :migration, :validate
• :codexis, :pipeline, :migration, :build_document
• :codexis, :pipeline, :migration, :validate_document
• :codexis, :pipeline, :migration, :fallback

These events can be used to track migration progress, system usage, and fallback rates.

Conclusion

The Pipeline Migration Framework provides a robust, controlled approach to transitioning from direct builders to the pipeline architecture. By using the phased migration approach, organizations can migrate at their own pace, ensuring compatibility and feature parity while minimizing disruption to existing workflows.