BUCKYVERSE

Date: 2025-03-01

Status

Accepted

Context

As part of our migration from Ruby to Elixir, we need to design a robust supervision strategy that leverages OTP's fault tolerance capabilities. Our current Ruby implementation lacks structured error recovery and process isolation, which has led to several issues:

  1. Cascading failures: Errors in one component can affect the entire system
  2. Manual recovery: Operators must manually restart failed processes
  3. Limited visibility: Difficult to track component health and state
  4. Inconsistent error handling: Different approaches across components
  5. Resource leaks: Failed operations may leave resources in inconsistent states

The Elixir/OTP platform provides built-in supervision capabilities that can address these issues through structured process management, isolation, and automatic recovery.

Decision

We will implement a comprehensive supervision strategy with the following structure:

1. Top-Level Application Supervisor

BookProcessing.Supervisor (one_for_one)
│
├── BookProcessing.Configuration.Supervisor (one_for_one)
├── BookProcessing.Conversion.Supervisor (one_for_one)
├── BookProcessing.Validation.Supervisor (rest_for_one)
├── BookProcessing.Build.Supervisor (one_for_all)
├── BookProcessing.ErrorHandling.Supervisor (one_for_one)
└── BookProcessing.Telemetry.Supervisor (one_for_one)

2. Context-Specific Supervisors

Each bounded context will have its own supervisor with an appropriate strategy:

Configuration Supervisor (one_for_one)

Conversion Supervisor (one_for_one)

Validation Supervisor (rest_for_one)

Build Supervisor (one_for_all)

ErrorHandling Supervisor (one_for_one)

Telemetry Supervisor (one_for_one)

3. Supervision Strategies

We will use different supervision strategies based on the nature of each component:

4. Process Types

We will use appropriate OTP behaviors based on component requirements:

5. Restart Strategies

We will configure appropriate restart strategies for different components:

6. Circuit Breakers

We will implement circuit breakers for external dependencies:

defmodule BookProcessing.CircuitBreaker do
  use GenServer
  
  # Circuit breaker implementation
  # ...
end

Implementation Example

defmodule BookProcessing.Application do
  use Application

  def start(_type, _args) do
    children = [
      {BookProcessing.Configuration.Supervisor, []},
      {BookProcessing.Conversion.Supervisor, []},
      {BookProcessing.Validation.Supervisor, []},
      {BookProcessing.Build.Supervisor, []},
      {BookProcessing.ErrorHandling.Supervisor, []},
      {BookProcessing.Telemetry.Supervisor, []}
    ]

    opts = [strategy: :one_for_one, name: BookProcessing.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

defmodule BookProcessing.Conversion.Supervisor do
  use Supervisor

  def start_link(init_arg) do
    Supervisor.start_link(__MODULE__, init_arg, name: __MODULE__)
  end

  @impl true
  def init(_init_arg) do
    children = [
      {BookProcessing.Conversion.LaTeXConverter, []},
      {BookProcessing.Conversion.DocxConverter, []},
      {BookProcessing.Conversion.PandocConverter, []},
      {BookProcessing.Conversion.FormatRegistry, []}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end
end

Consequences

Positive

  1. Improved fault tolerance: Failures are isolated to specific components
  2. Automatic recovery: Failed processes are automatically restarted
  3. Clear process boundaries: Each component has well-defined responsibilities
  4. Visibility: Process state and health can be monitored
  5. Resource management: Resources are properly cleaned up on process termination
  6. Scalability: Components can be distributed across nodes if needed
  7. Simplified error handling: Let-it-crash philosophy simplifies error handling

Negative

  1. Increased complexity: More sophisticated architecture to understand
  2. Learning curve: Team needs to learn OTP concepts
  3. Process overhead: Each process adds some memory overhead
  4. Message passing overhead: Communication between processes adds latency
  5. State management challenges: Need to carefully design stateful components

Mitigation Strategies

  1. Documentation: Thoroughly document supervision tree and component responsibilities
  2. Training: Provide team training on OTP concepts and best practices
  3. Monitoring: Implement comprehensive monitoring of process health
  4. Testing: Create tests specifically for supervision and recovery scenarios
  5. Gradual implementation: Start with core components and expand supervision tree incrementally

References