Codexis Test Suite Analysis

This document presents the analysis of overlaps, redundancies, and coverage gaps between builder and content consistency tests, as required by Task 18.2.

Cross-Reference Matrix: Test Coverage by Area

Coverage Area	Builder Tests	Content Consistency Tests	Notes
Chapters/Headings	Most builder tests, especially unified	chapter_tests, unified_content_consistency, cross_format_consistency	Significant overlap; both test chapter preservation
Sections/Structure	Most builder tests, structural_tests, unified	structural_tests, unified_content_consistency	Overlap; both test section structure
Math Content	Most builder tests, especially unified	unified_content_consistency, cross_format_consistency, validation	Overlap; both test math preservation
Metadata	Some builder tests, unified	unified_content_consistency, validation	Overlap; both test metadata presence
Special Content (tables, code, etc.)	Most builder tests, especially unified	unified_content_consistency, structural_tests	Overlap; both test special content
Error Handling	All builder tests	Some content consistency tests	Mostly builder tests
Temp Dir Handling	Unified builder tests, build_manager	Some content consistency tests	Unified tests use BuildOrchestrator
Cleanup Flag	Unified builder tests	Some content consistency tests	Unified tests use cleanup_temp_dir
Multi-format Output	Unified builder tests, integration	unified_content_consistency, cross_format_consistency	Overlap; both test cross-format output
Performance	performance/build_performance_test	N/A	Only in performance test

Overlaps & Redundancies

Chapters, Sections, Math, Metadata, Special Content:
- Both builder and content consistency tests validate these areas, often with similar assertions and test data.
- Unified builder tests and unified content consistency tests are especially redundant.
Temp Dir and Cleanup Handling:
- Unified builder tests and some content consistency tests both check temp dir and cleanup logic.
Multi-format Output:
- Both builder and content consistency tests validate output across multiple formats.
Legacy vs Unified:
- Legacy builder/content consistency tests often duplicate scenarios now covered by unified tests.
Migration Progress:
- As of 2025-05-28, all builder test suites (Markdown, HTML, PDF, WikiJS, EPUB, DOCX) have been unified. Legacy and unified test files for PDF, WikiJS, EPUB, and DOCX builders have been merged into builders/pdf_builder_test.exs, builders/wikijs_builder_test.exs, builders/epub_builder_test.exs, and builders/docx_builder_test.exs respectively. All canonical builder tests now use manuscript/manuscript.tex via the shared fixture and the unified temp dir/cleanup pattern. Only the MathJax injection assertion remains for the HTML builder. These unified tests serve as reference implementations for future migrations.

Gaps & Missing Coverage

Edge Cases in manuscript.tex:
- Some special LaTeX features or rare edge cases in the canonical test book may not be explicitly tested.
Error Handling in Content Consistency:
- Most error handling is only tested in builder tests, not in content consistency tests.
Performance Under Edge Conditions:
- Only performance tests cover build speed/large docs; may need more edge case scenarios.
CI/CD Integration:
- Not all tests are run with both cleanup_temp_dir=true and false in CI.

Recommendations

Consolidate Redundant Tests:
- Merge builder and content consistency tests for chapters, sections, math, metadata, and special content into unified modules using shared helpers.
- Remove or refactor legacy tests that duplicate unified test logic.
Centralize Test Data:
- Ensure all tests use manuscript/manuscript.tex via the shared fixture, except for justified performance cases.
- All future migrations should follow the canonical example set by builders/markdown_builder_test.exs.
Expand Edge Case Coverage:
- Add tests for any untested features or edge cases present in the canonical manuscript.
Improve Error Handling Coverage:
- Add error handling scenarios to content consistency tests where appropriate.
CI/CD Improvements:
- Ensure all tests are run with both cleanup_temp_dir=true and false in CI pipelines.
Documentation:
- Update documentation to reflect the unified approach and migration plan.

This analysis will guide the next steps in the migration and consolidation of the Codexis test suite. The migration is underway, and the Markdown builder test is the reference for future work.

Unification Rationale:

Merging legacy and unified test files eliminates redundancy, ensures all tests follow the canonical fixture and temp dir pattern, and simplifies maintenance.
All builder tests now provide comprehensive coverage for build, error, and edge cases using the canonical test book and shared infrastructure.