Date: 2025-03-05

Status

Accepted

Context

The ebook conversion system relies heavily on several external command-line tools for critical functionality:

• LaTeX/XeLaTeX: For PDF generation from TeX sources
• Calibre (ebook-convert): For converting between ebook formats, particularly EPUB to MOBI/KFX
• tex4ebook: For converting TeX to EPUB format
• pandoc: For general format conversions
• ImageMagick: For image processing and optimization
• epubcheck: For validating EPUB output
• Kindle Previewer: For creating and validating Kindle formats

In our current Ruby implementation, these tools are invoked through direct system calls with limited error handling, output capturing, and process management. This approach has led to several issues:

1. Unreliable error detection: Difficult to distinguish between tool errors and execution failures
2. Resource leaks: Abandoned processes and temporary files when system calls fail unexpectedly
3. Brittle output parsing: Custom regex-based parsing of tool output is fragile and maintenance-heavy
4. Inconsistent timeout handling: No standardized approach to handling tool hangs or long-running processes
5. Limited concurrency control: No mechanism to limit concurrent external tool executions
6. Poor cross-platform compatibility: System calls behave differently across operating systems
7. Security concerns: Direct system call execution can pose security risks without proper sanitization

As we migrate to Elixir, we need to design a robust approach to external tool integration that addresses these issues while leveraging Elixir/OTP's strengths.

Decision

We will implement a Port-based Command Execution Framework with the following components:

1. Command Port Abstraction

We will create a behavior-based abstraction for command execution:

defmodule Codexis.Port.CommandExecutor do
  @callback execute(command :: String.t(), args :: [String.t()], opts :: Keyword.t()) ::
              {:ok, result :: term()} | {:error, reason :: term()}

  @callback execute_with_timeout(
              command :: String.t(),
              args :: [String.t()],
              timeout :: non_neg_integer(),
              opts :: Keyword.t()
            ) ::
              {:ok, result :: term()} | {:error, :timeout} | {:error, reason :: term()}
end

2. Elixir Port Implementation

The primary implementation will use Elixir Ports for process management:

defmodule Codexis.Port.ElixirPort do
  @behaviour Codexis.Port.CommandExecutor

  @impl true
  def execute(command, args, opts \\ []) do
    port_opts = port_options(opts)
    command_with_args = [command | args]

    # Start port and monitor for exit
    port = Port.open({:spawn_executable, command}, port_opts ++ [args: args])
    ref = Process.monitor(port)

    # Collect output
    try do
      collect_output(port, ref, %{stdout: "", stderr: ""}, opts)
    after
      # Ensure we clean up
      Port.close(port)
    end
  end

  @impl true
  def execute_with_timeout(command, args, timeout, opts \\ []) do
    task = Task.async(fn -> execute(command, args, opts) end)
    
    case Task.yield(task, timeout) || Task.shutdown(task) do
      {:ok, result} -> result
      nil -> {:error, :timeout}
    end
  end

  # Private helper functions for port options, output collection, etc.
  defp port_options(opts) do
    [
      :binary,
      :exit_status,
      :hide,
      :use_stdio,
      {:stderr_to_stdout, Keyword.get(opts, :stderr_to_stdout, false)}
    ]
  end
  
  defp collect_output(port, ref, output, opts) do
    # Implementation for collecting output from port
  end
end

3. Tool-specific Adapters

We will create specific adapters for each external tool that encapsulate tool-specific behavior:

defmodule Codexis.Tools.LaTeX do
  # LaTeX-specific functionality
  
  def compile(tex_file, output_dir, opts \\ []) do
    engine = Keyword.get(opts, :engine, "xelatex")
    args = [
      "-interaction=nonstopmode",
      "-output-directory=#{output_dir}",
      tex_file
    ]
    
    Codexis.Port.ElixirPort.execute_with_timeout(engine, args, :timer.minutes(5))
    |> handle_latex_result(tex_file)
  end
  
  defp handle_latex_result({:ok, %{exit_status: 0, stdout: output}}, _tex_file) do
    {:ok, parse_latex_output(output)}
  end
  
  defp handle_latex_result({:ok, %{exit_status: status, stdout: output}}, tex_file) do
    errors = extract_latex_errors(output)
    {:error, %{exit_status: status, errors: errors, file: tex_file}}
  end
  
  defp handle_latex_result({:error, _} = error, _tex_file), do: error
  
  # LaTeX-specific output parsing
  defp parse_latex_output(output) do
    # Parse LaTeX output to extract warnings, page count, etc.
  end
  
  defp extract_latex_errors(output) do
    # Extract error information from LaTeX output
  end
end

# Similar adapters for other tools: Calibre, tex4ebook, etc.

4. Command Registry and Discovery

We will implement a registry for command discovery and path resolution:

defmodule Codexis.Tools.Registry do
  use GenServer
  
  def start_link(opts \\ []) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end
  
  def init(_opts) do
    # Initialize with empty registry
    {:ok, %{commands: %{}}}
  end
  
  def register(command, path) do
    GenServer.call(__MODULE__, {:register, command, path})
  end
  
  def lookup(command) do
    GenServer.call(__MODULE__, {:lookup, command})
  end
  
  def handle_call({:register, command, path}, _from, state) do
    updated_commands = Map.put(state.commands, command, path)
    {:reply, :ok, %{state | commands: updated_commands}}
  end
  
  def handle_call({:lookup, command}, _from, state) do
    case Map.fetch(state.commands, command) do
      {:ok, path} -> {:reply, {:ok, path}, state}
      :error -> {:reply, discover_command(command, state), state}
    end
  end
  
  defp discover_command(command, state) do
    # Try to find command in PATH
    case System.find_executable(command) do
      nil -> {:error, :not_found}
      path ->
        # Update registry and return path
        updated_commands = Map.put(state.commands, command, path)
        Process.send(self(), {:update_commands, updated_commands}, [])
        {:ok, path}
    end
  end
  
  def handle_info({:update_commands, commands}, state) do
    {:noreply, %{state | commands: commands}}
  end
end

5. Resource Management

We will implement a resource manager to control concurrent tool executions:

defmodule Codexis.Tools.ResourceManager do
  use GenServer
  
  def start_link(opts \\ []) do
    max_concurrent = Keyword.get(opts, :max_concurrent, System.schedulers_online())
    GenServer.start_link(__MODULE__, %{max_concurrent: max_concurrent}, name: __MODULE__)
  end
  
  def init(state) do
    {:ok, Map.put(state, :current_count, 0)}
  end
  
  def acquire() do
    GenServer.call(__MODULE__, :acquire)
  end
  
  def release() do
    GenServer.cast(__MODULE__, :release)
  end
  
  def handle_call(:acquire, from, %{current_count: count, max_concurrent: max} = state) do
    if count < max do
      {:reply, :ok, %{state | current_count: count + 1}}
    else
      # Queue the request
      {:noreply, enqueue_request(from, state)}
    end
  end
  
  def handle_cast(:release, %{current_count: count, queue: queue} = state) do
    case :queue.out(queue) do
      {{:value, from}, new_queue} ->
        # Respond to queued request
        GenServer.reply(from, :ok)
        {:noreply, %{state | queue: new_queue}}
      
      {:empty, _} ->
        # Just decrease count
        {:noreply, %{state | current_count: count - 1}}
    end
  end
  
  defp enqueue_request(from, %{queue: queue} = state) do
    %{state | queue: :queue.in(from, queue)}
  end
end

6. Command Execution Service

We will provide a high-level service for tool execution:

defmodule Codexis.Tools.Executor do
  def execute_tool(tool, args, opts \\ []) do
    # Check if tool exists
    with {:ok, tool_path} <- Codexis.Tools.Registry.lookup(tool),
         # Acquire resource
         :ok <- Codexis.Tools.ResourceManager.acquire(),
         # Execute command
         result <- do_execute_tool(tool_path, args, opts) do
      # Release resource
      Codexis.Tools.ResourceManager.release()
      result
    else
      {:error, _} = error ->
        # Release resource on error if acquired
        Codexis.Tools.ResourceManager.release()
        error
    end
  end
  
  defp do_execute_tool(tool_path, args, opts) do
    timeout = Keyword.get(opts, :timeout, :timer.minutes(5))
    Codexis.Port.ElixirPort.execute_with_timeout(tool_path, args, timeout, opts)
  end
end

7. Temporary File Management

For handling temporary files created during tool execution:

defmodule Codexis.Tools.TempManager do
  def with_temp_dir(prefix, func) do
    temp_dir = create_temp_dir(prefix)
    try do
      func.(temp_dir)
    after
      cleanup_temp_dir(temp_dir)
    end
  end
  
  defp create_temp_dir(prefix) do
    unique_id = :crypto.strong_rand_bytes(8) |> Base.encode16(case: :lower)
    temp_path = Path.join(System.tmp_dir!(), "#{prefix}_#{unique_id}")
    File.mkdir_p!(temp_path)
    temp_path
  end
  
  defp cleanup_temp_dir(dir) do
    File.rm_rf!(dir)
  end
end

Consequences

Positive

1. Improved error handling: Structured approach to error detection and reporting
2. Automatic resource cleanup: Port-based execution ensures proper process termination
3. Reliable output processing: Clean separation of output collection and parsing
4. Consistent timeout handling: Uniform approach to timeouts across all tools
5. Concurrency control: Ability to limit concurrent external process execution
6. Cross-platform compatibility: Port abstraction normalizes behavior across operating systems
7. Enhanced security: Proper command argument handling reduces security risks
8. Observability: Easier to monitor and log external tool execution

Negative

1. Increased complexity: More sophisticated architecture compared to direct system calls
2. Performance overhead: Slight overhead from additional abstraction layers
3. Learning curve: Team needs to understand port-based process execution
4. Error translation complexity: Parsing tool-specific error messages remains challenging
5. Initial development cost: More upfront work to implement the framework

Mitigation Strategies

1. Comprehensive documentation: Document tool interface patterns and error handling
2. Test fixtures: Create test fixtures for common tool outputs to validate parsers
3. Graceful degradation: Provide fallback mechanisms when tools behave unexpectedly
4. Progressive enhancement: Start with basic integration and enhance over time
5. Automated validation: Regularly test command execution in CI environment
6. Platform-specific test suite: Test on all target platforms to ensure compatibility
7. Monitoring: Monitor execution times and resource usage to identify bottlenecks

References

1. Elixir documentation on Ports: https://hexdocs.pm/elixir/Port.html
2. "Designing for Scalability with Erlang/OTP" by Francesco Cesarini and Steve Vinoski
3. "Elixir in Action" by Saša Jurić (Chapter on Port-based communication)
4. OTP Design Principles documentation: https://erlang.org/doc/design_principles/des_princ.html
5. "Secure command execution in Elixir" - ElixirConf presentation
6. "Working with external processes in Elixir" - ElixirCasts
7. Porcelain library documentation: https://github.com/alco/porcelain
8. System module documentation: https://hexdocs.pm/elixir/System.html