BUCKYVERSE

Date: 2025-02-24

Status

Accepted

Context

The replatforming of our ebook conversion system from Ruby to Elixir introduces several challenges that affect our testing approach:

  1. Complex domain logic: Our ebook conversion system involves intricate workflows with multiple formats, tools, and validation steps that must be thoroughly tested.

  2. Concurrent and distributed processing: Elixir/OTP enables concurrent format generation and distributed build capabilities that require specialized testing approaches.

  3. External tool integration: The system interacts with various external tools (LaTeX, Calibre, tex4ebook) that need to be properly mocked or integrated in tests.

  4. Build environment dependencies: Tests must account for the potential variability of build environments and dependency versions.

  5. Quality assurance requirements: Our migration goals explicitly include enhanced validation and quality control, necessitating a robust testing strategy.

  6. System reliability needs: As we migrate to a new platform, we need to ensure the reliability and fault tolerance of our system through comprehensive testing.

The migration process itself also presents challenges in ensuring feature parity and avoiding regressions. A methodical, test-driven approach would help address these concerns.

Decision

We will adopt Test-Driven Design (TDD) as a core practice for our Elixir replatforming effort. This approach will involve:

1. Test-First Development

2. Test Hierarchy and Organization

We will implement a comprehensive test hierarchy aligned with our domain-driven architecture:

3. Testing Tools and Frameworks

We will leverage Elixir's testing ecosystem:

4. Test Environment Management

5. Continuous Integration

6. Test Data Management

Implementation Approach

Unit Testing Strategy

defmodule Codexis.BuildManagerTest do
  use ExUnit.Case, async: true
  
  describe "verify_environment/0" do
    test "returns valid environment when all dependencies are present" do
      # Arrange
      setup_mock_dependencies()
      
      # Act
      result = Codexis.BuildManager.verify_environment()
      
      # Assert
      assert {:ok, env} = result
      assert env.latex != nil
      assert env.calibre != nil
      assert env.tex4ebook != nil
    end
    
    test "returns error when LaTeX is missing" do
      # Arrange
      setup_mock_missing_latex()
      
      # Act
      result = Codexis.BuildManager.verify_environment()
      
      # Assert
      assert {:error, :latex_missing} = result
    end
  end
end

Integration Testing Strategy

defmodule Codexis.WorkflowIntegrationTest do
  use Codexis.DataCase
  
  setup do
    {:ok, workflow} = Codexis.Workflow.Pipeline.new([
      :prepare_environment,
      :process_manuscript,
      :generate_formats
    ])
    
    %{workflow: workflow}
  end
  
  test "complete workflow execution with PDF generation", %{workflow: workflow} do
    # Arrange
    manuscript = create_test_manuscript()
    
    # Act
    {:ok, completed_workflow} = Codexis.Workflow.execute(workflow, manuscript)
    
    # Assert
    assert completed_workflow.status == :completed
    assert Map.has_key?(completed_workflow.artifacts, :pdf)
    assert get_in(completed_workflow.artifacts, [:pdf, :status]) == :success
  end
end

Property-Based Testing Example

defmodule Codexis.Validation.RulesTest do
  use ExUnit.Case, async: true
  use PropCheck
  
  property "validation rules always return valid result structure" do
    forall content <- content_generator() do
      result = Codexis.Validation.validate_content(content, default_rules())
      is_validation_result(result)
    end
  end
  
  defp is_validation_result(result) do
    is_tuple(result) and
    (elem(result, 0) == :ok or elem(result, 0) == :error) and
    (if elem(result, 0) == :error, do: is_list(elem(result, 1)), else: true)
  end
end

System Testing Example

defmodule Codexis.SystemTest do
  use ExUnit.Case
  
  @tag :system
  @tag timeout: 300_000 # 5 minutes
  test "complete build pipeline with all formats" do
    # Arrange
    setup_test_environment()
    manuscript_path = create_test_manuscript()
    
    # Act
    result = Codexis.UnifiedCLI.main(["build", "--all-formats", manuscript_path])
    
    # Assert
    assert result.exit_code == 0
    assert_all_formats_generated(result.output_dir)
  end
end

Consequences

Positive

  1. Higher code quality: TDD promotes cleaner, more modular code with better separation of concerns
  2. Better documentation: Tests serve as executable documentation of system behavior
  3. Faster feedback loops: Immediate feedback on code changes reduces debugging time
  4. Safer refactoring: Comprehensive test coverage enables confident code evolution
  5. Incremental progress: TDD encourages breaking down complex features into testable increments
  6. Improved design: Test-first approach often leads to more modular, loosely coupled designs
  7. Regression prevention: Test suite catches unintended side effects of changes
  8. Confidence in migration: Tests validate that the new Elixir system matches Ruby system behavior

Negative

  1. Initial development slowdown: Writing tests first requires more upfront time
  2. Learning curve: Team needs to adapt to TDD practices and Elixir testing tools
  3. Test maintenance overhead: Tests need to be maintained alongside production code
  4. Mocking complexity: External tool dependencies make some tests complex to set up
  5. System test performance: Full system tests can be slow to execute
  6. Test environment management: Ensuring consistent test environments adds complexity

Mitigation Strategies

  1. Prioritize test coverage: Focus most thorough testing on core domain logic
  2. Use test doubles judiciously: Create clear interfaces to external systems for easier mocking
  3. Containerize test environments: Use Docker to ensure consistent test environments
  4. Parallel test execution: Leverage ExUnit's async testing capabilities
  5. Test categorization: Tag tests to allow selective running of faster test suites
  6. Shared test utilities: Develop helper functions and fixtures to reduce test code duplication
  7. Regular test maintenance: Schedule time for keeping tests clean and up-to-date
  8. TDD workshops: Conduct training sessions to build team skill with TDD in Elixir

References