37. Standardize Test Book Fixture for All Tests

Date: 2025-06-03

Status

Accepted

Context

Historically, tests in the Codexis project have used a variety of manuscript sources, temporary directories, and ad-hoc test content. This led to:

• Inconsistent test results and coverage
• Difficulty reproducing bugs and verifying fixes
• Redundant or conflicting test data
• Increased maintenance burden when updating test content or structure

To address these issues and ensure reliable, reproducible, and maintainable tests, we need a single, canonical test book fixture that all tests use as their manuscript source.

Decision

All tests in the Codexis project must use the canonical test book fixture located at test/support/fixtures/test_manuscript as their manuscript source.

• No test should generate, copy, or manipulate manuscripts in random or temporary locations for the purpose of testing core build, pipeline, or builder logic.
• All builder, pipeline, and integration tests must reference this fixture directly for any manuscript_dir or content_path arguments.
• If additional test data is needed, it should be added to the canonical fixture in a way that does not break existing tests.
• Any new test suites or modules must follow this rule.

Consequences

• Consistency: All tests run against the same, well-understood manuscript, making failures easier to diagnose and fixes easier to verify.
• Reproducibility: Developers and CI environments will always have the same test data, eliminating "works on my machine" issues.
• Maintainability: Updates to the test book fixture automatically benefit all tests, and new test cases can be added by extending the fixture.
• Reduced Redundancy: No more duplicated or conflicting test manuscripts scattered throughout the codebase.
• Test Coverage: The canonical fixture can be expanded to cover new edge cases as needed, improving overall test coverage.

Risks:

• If the canonical fixture is changed in a breaking way, it may cause many tests to fail at once. Changes should be reviewed carefully and coordinated.
• Some highly specialized tests may still require custom fixtures, but these should be rare and well-justified in their own ADRs.

Rule:

All tests must use test/support/fixtures/test_manuscript as the manuscript source unless a specific ADR justifies an exception.