24. Build Pipeline Architecture

Date: 2025-03-05
Updated: 2025-06-02

Status

Accepted

Context

The Codexis book building system must transform LaTeX source content and assets into multiple output formats (PDF, EPUB, HTML, Markdown, etc.) with high reliability, extensibility, and observability. The legacy approach used direct builder modules for each format, but this led to code duplication, inconsistent error handling, and difficulty in adding new features or formats.

To address these issues, the system has migrated to a canonical pipeline architecture for all document builds. The pipeline is now the default and only supported build mechanism, except when explicitly overridden for legacy fallback. This architecture enables modular, staged processing, robust error handling, and easy extensibility.

Decision

We have adopted a formal pipeline framework as the core of the build system. All document builds are orchestrated through this pipeline, which is composed of ordered, composable stages and processors. Direct builder usage is deprecated and only available via disable_pipeline: true in build options.

Pipeline Architecture Overview

Pipeline: Orchestrates execution of a sequence of stages, manages context, error propagation, and flow control.
Stage: Implements a standard behaviour (name/0, run/1), responsible for a discrete build step (e.g., scanno detection, format conversion).
Processor: Encapsulates reusable logic, can be composed within stages or used as standalone steps.
Connector: Handles data flow between stages or external systems (e.g., file I/O, remote APIs).
Registry: Central registry for pipelines, stages, and processors, supporting dynamic configuration and extension.
Config: Centralized configuration system supporting YAML and programmatic options, with validation and schema.
Execution Engine: Manages stage order, concurrency, error handling, telemetry, and hooks for pre/post execution.

Architecture Diagram

Pipeline
  ├── Stage (ScannoDetection)
  ├── Stage (ContentProcessing)
  ├── Stage (AssetManagement)
  ├── Stage (FormatGeneration)
  │     ├── Processor (PDFBuilder)
  │     ├── Processor (EPUBBuilder)
  │     ├── Processor (HTMLBuilder)
  │     └── ...
  ├── Stage (Validation)
  └── Stage (Finalize)

Example Module Layout

lib/codexis/pipeline/
  pipeline.ex           # Pipeline orchestration logic
  stage.ex              # Stage behaviour and base implementation
  processor.ex          # Processor abstraction
  connector.ex          # Connector abstraction
  registry.ex           # PipelineRegistry for registration/config
  config.ex             # PipelineConfig and validation
  executor.ex           # PipelineExecutor for running pipelines
  hooks.ex              # Pre/post execution hooks
  context_utils.ex      # Context manipulation helpers
  error.ex              # Error types and helpers
  telemetry.ex          # Telemetry and logging
  test_helpers.ex       # Test fixtures and mocks

Registry and Configuration

All pipelines, stages, and processors are registered in a central registry.
Configuration is loaded from YAML, application env, and runtime overrides, with schema validation.
Extension points allow developers to register custom stages/processors.

Migration and Compatibility

The migration to the pipeline is complete; direct builder usage is deprecated except for explicit fallback (disable_pipeline: true).
The migration manager and adapters ensure backward compatibility and phased rollout.
See migration documentation:
- docs/migration/pipeline-migration-index.md
- docs/migration/pipeline-migration-framework.md
- docs/migration/pipeline-migration-user-guide.md
- docs/migration/pipeline-testing-guide.md

Testing and Documentation

Comprehensive unit, integration, compatibility, and performance tests are required for all pipeline components.
Developer documentation, migration guides, and usage examples are maintained alongside the code.
Test helpers and fixtures are provided for pipeline testing.

Supervision and Fault Tolerance

The pipeline and its components are supervised for reliability and fault isolation.
Error handling is standardized and supports recovery and diagnostics.

Consequences

Positive

Modularity: Clear separation of concerns and easy extension.
Parallelization: Concurrent processing where possible.
Extensibility: New formats and processors can be added with minimal effort.
Reliability: Robust error handling, recovery, and supervision.
Observability: Comprehensive telemetry and logging.
Testability: Unified test infrastructure for all pipeline components.
Backward Compatibility: Legacy builds supported via explicit fallback.

Negative

Initial Migration Overhead: Required significant refactoring and migration effort.
Learning Curve: Developers must learn the new pipeline abstractions and configuration.