BUCKYVERSE

Date: 2025-02-25

Status

Accepted

Context

Our ebook conversion system deals with complex workflows involving multiple external tools, file operations, and conversion processes. Errors can occur at various stages, and our current error handling approach focuses primarily on reporting errors rather than recovering from them.

As identified in ADR-0009 (Implement validation strategy), we need to enhance our system with:

The replatforming to Elixir provides an opportunity to implement more robust error recovery mechanisms, but we need to establish patterns that can be applied to our current Ruby codebase and later migrated to Elixir.

Decision

We will implement a comprehensive error recovery strategy with the following components:

1. Recovery Strategy Pattern

We will create a Recovery Strategy pattern that defines how to recover from specific error types:

module ErrorRecovery
  class Strategy
    def can_recover?(error)
      # Determine if this strategy can handle the given error
    end
    
    def recover(error, context)
      # Implement recovery logic
      # Return {:ok, result} or {:error, new_error}
    end
    
    def recovery_instructions(error)
      # Return human-readable recovery instructions
    end
  end
end

2. Retry Mechanism

For transient errors, especially those related to external tools or network operations:

module ErrorRecovery
  def self.with_retry(max_attempts: 3, backoff: :exponential, &block)
    attempt = 0
    
    begin
      attempt += 1
      yield
    rescue => error
      if attempt < max_attempts && retryable?(error)
        sleep calculate_backoff(attempt, backoff)
        retry
      else
        raise error
      end
    end
  end
end

3. Circuit Breaker Pattern

For protecting the system from cascading failures when external dependencies fail:

module ErrorRecovery
  class CircuitBreaker
    def initialize(failure_threshold: 5, reset_timeout: 30)
      # Initialize circuit breaker state
    end
    
    def execute(&block)
      # Check circuit state and execute or fail fast
    end
  end
end

4. Automatic Recovery for Common Issues

Implement automatic fixes for common issues:

module ErrorRecovery
  class AutoFix
    def self.register(error_class, &fix)
      # Register an automatic fix for an error type
    end
    
    def self.attempt_fix(error, context)
      # Try to automatically fix the error
    end
  end
end

5. Transaction-like Operations

For multi-step operations that need to be rolled back on failure:

module ErrorRecovery
  def self.with_rollback(context, &block)
    snapshot = take_snapshot(context)
    
    begin
      result = yield
      {:ok, result}
    rescue => error
      restore_snapshot(context, snapshot)
      {:error, error}
    end
  end
end

Implementation Strategy

  1. Phased Implementation:

    • Phase 1: Implement core recovery patterns
    • Phase 2: Add retry mechanisms for external tools
    • Phase 3: Implement circuit breakers for critical dependencies
    • Phase 4: Add automatic fixes for common errors
    • Phase 5: Implement transaction-like operations

  2. Integration with Existing Code:

    • Wrap critical operations with recovery mechanisms
    • Enhance error messages with recovery instructions
    • Add context information to errors for better recovery
    • Implement state preservation for long-running operations

  3. Elixir Migration Path:

    • Map Ruby recovery patterns to Elixir/OTP equivalents
    • Use Elixir's supervision trees for more robust recovery
    • Leverage GenServer restart strategies
    • Implement process isolation for better fault tolerance

Consequences

Positive

  1. Improved reliability: The system can recover from many errors without manual intervention
  2. Better user experience: Clear recovery instructions when automatic recovery isn't possible
  3. Reduced operational overhead: Fewer support requests for common issues
  4. System resilience: Protection against cascading failures
  5. State preservation: Less work lost when errors occur
  6. Clearer error handling: Standardized approach to error recovery
  7. Migration readiness: Patterns that map well to Elixir/OTP concepts

Negative

  1. Increased complexity: More sophisticated error handling logic
  2. Testing challenges: Recovery scenarios need comprehensive testing
  3. Performance overhead: Some recovery mechanisms add latency
  4. Development overhead: Implementing recovery logic takes time
  5. Potential masking of issues: Automatic recovery might hide underlying problems

Mitigation Strategies

  1. Comprehensive logging: Log all recovery attempts, even successful ones
  2. Telemetry: Track recovery statistics to identify recurring issues
  3. Circuit breaker monitoring: Alert on frequently tripped circuit breakers
  4. Recovery testing: Explicitly test recovery scenarios
  5. Documentation: Clear documentation of recovery mechanisms
  6. Timeout controls: Ensure recovery attempts don't block indefinitely
  7. Fallback degradation: Define acceptable degraded functionality

References