Pipeline Migration User Guide

Important Notice: Accelerated Migration

Since this system is not yet in production, we've accelerated the migration process to use the pipeline architecture by default for all document formats. The migration manager has been updated to bypass the phased approach and always use the pipeline implementation.

This change means:

• All document formats will now use the pipeline architecture by default
• The phased approach is still available in the code if needed in the future
• The pipeline implementation is now the primary path for all document formats
• You can still use the direct builders by explicitly setting disable_pipeline: true in your options

The remainder of this guide explains the original phased approach which has been preserved in the codebase for reference.

Getting Started

The Codexis pipeline architecture migration is designed to be gradual and controlled. As a developer, you can:

1. Use direct builders (legacy approach)
2. Use pipeline architecture (new approach)
3. Use a combination based on migration phase settings

Checking Current Configuration

Before working with the system, check the current migration status:

# Via CLI
mix codexis.pipeline_migration status

# Via web interface
# Navigate to /pipeline-migration in your browser

This will show you:

• Current global phase
• Format-specific phases
• Feature parity percentages
• Recommended phases based on readiness

Working with Documents

Building Documents

When building documents, the system will automatically choose between direct builders and pipeline based on the current configuration:

# Standard build approach - automatically routes to appropriate system
{:ok, results} = Codexis.BuildManager.build([:pdf, :html], opts)

# Force pipeline usage regardless of phase settings
{:ok, results} = Codexis.BuildManager.build([:pdf], Keyword.put(opts, :force_pipeline, true))

# Force direct builder usage regardless of phase settings
{:ok, results} = Codexis.BuildManager.build([:pdf], Keyword.put(opts, :disable_pipeline, true))

Checking Which System Was Used

To check which system was used for a build:

# Enable debug logging
Logger.configure(level: :debug)

# Build document
{:ok, results} = Codexis.BuildManager.build([:pdf], opts)

# Check logs for "Processing document with pipeline architecture" or
# "Processing document with direct builder"

Alternatively, use the telemetry events to track which system is being used.

Managing Migration Phases

Via CLI

# Set phase for a specific format
mix codexis.pipeline_migration set-phase --format=pdf --phase=phase2

# Set global migration phase
mix codexis.pipeline_migration set-global-phase --phase=phase1

Via Web Interface

1. Navigate to /pipeline-migration in your browser
2. Use the "Set Global Phase" dropdown to update the global phase
3. Use the "Set Phase" dropdown for each format to update format-specific phases

Via Code

# Set format-specific phase
Codexis.Pipeline.Migration.Manager.set_phase(:epub, :phase2)

# Set global phase
Codexis.Pipeline.Migration.Manager.set_global_phase(:phase1)

Choosing the Right Phase

Each migration phase has a specific purpose:

Phase 1: Parallel Operation

Use when:

• You're just beginning migration
• You want to test the pipeline with minimal risk
• You have formats with low feature parity (<70%)

Behavior:

• Direct builders are used by default
• Pipeline is used only for formats explicitly marked as ready

Phase 2: Primary Pipeline with Fallback

Use when:

• You have established good feature parity (>70%)
• You want to start using the pipeline primarily
• You need a safety net for edge cases

Behavior:

• Pipeline is used by default
• Falls back to direct builders if pipeline processing fails

Phase 3: Pipeline-Only Operation

Use when:

• You have excellent feature parity (>90%)
• You're confident in pipeline reliability
• You want to complete the migration

Behavior:

• Pipeline is used exclusively
• No fallback to direct builders

Testing and Validation

Running Compatibility Tests

# Test all formats
mix codexis.test_pipeline_compatibility

# Test a specific format
mix codexis.test_pipeline_compatibility --format=pdf

# Update the feature registry with test results
mix codexis.test_pipeline_compatibility --update-registry

Examining Feature Support

# Show feature matrix for a format
mix codexis.pipeline_migration feature-matrix --format=epub

Generating Reports

# Generate feature parity report
mix codexis.pipeline_migration generate-report --output=reports/my-report.json

Handling Issues

Pipeline Failures in Phase 2

In Phase 2, the system will automatically try the direct builder if the pipeline fails. You'll see log entries like:

[info] Attempting fallback with builder for format pdf

To diagnose recurring failures:

1. Enable verbose logging:

   mix codexis.pipeline_migration status --verbose
   

2. Check for feature gaps:

   mix codexis.pipeline_migration feature-matrix --format=<format>
   

3. Run compatibility tests:

   mix codexis.test_pipeline_compatibility --format=<format> --verbose
   

Common Problems and Solutions

Problem: Unexpected System Used

Cause: Migration phase configuration not as expected
Solution: Check current phases with mix codexis.pipeline_migration status

Problem: Persistent Pipeline Failures

Cause: Missing feature implementation or bugs
Solution: Identify gaps using feature-matrix and run specific tests to isolate the issue

Problem: Performance Issues

Cause: Inefficient pipeline implementation for specific formats
Solution: Run benchmarks to identify bottlenecks and optimize pipeline stages

Problem: Configuration Not Persisting

Cause: Application restart resetting configuration
Solution: Update default configuration in config.exs

Best Practices

1. Start with Phase 1: Begin migration with Phase 1 to minimize risk
2. Test Thoroughly: Use the compatibility test suite before advancing phases
3. Incremental Migration: Advance formats to higher phases one at a time
4. Monitor Usage: Track telemetry events to ensure systems are behaving as expected
5. Validate Outputs: Regularly compare outputs from both systems
6. Update Documentation: Document any format-specific considerations or workarounds
7. Communicate Changes: Keep the team informed of phase changes and migration progress

Example Workflow

Here's a typical workflow for migrating a format:

1. Assessment:

   mix codexis.pipeline_migration feature-matrix --format=epub
   mix codexis.test_pipeline_compatibility --format=epub
   

2. Phase 1 Testing:

   mix codexis.pipeline_migration set-phase --format=epub --phase=phase1
   

   Test with force_pipeline: true to explicitly use pipeline for select documents

3. When parity reaches ~70%:

   mix codexis.pipeline_migration set-phase --format=epub --phase=phase2
   

   Monitor fallback events in logs

4. When parity reaches ~90%:

   mix codexis.pipeline_migration set-phase --format=epub --phase=phase3
   

   Thoroughly test with real-world documents

5. Complete migration:

   mix codexis.pipeline_migration set-phase --format=epub --phase=pipeline_only
   

Reference

Migration Phases

Command Reference

Getting Help

If you encounter issues not covered in this guide:

1. Check the detailed documentation in docs-site/docs/migration/pipeline-migration-framework.md
2. Review the testing guide in docs-site/docs/migration/pipeline-testing-guide.md
3. Look for error messages in the application logs
4. Contact the pipeline migration team for assistance