Unified Build System Troubleshooting Guide

This guide helps diagnose and resolve issues with the Codexis Unified Build System.

Quick Diagnostic Commands

Check Build System Status

# Test basic build functionality
./codexis build --format pdf --source manuscript --output /tmp/test_output

# Test with debugging enabled
./codexis build --all --preserve-temp-dir --show-temp-dir --source manuscript --output /tmp/test_output

Verify Temp Directory Access

# Check temp directory permissions and space
df -h /tmp
ls -la /tmp/codexis_build_*

# Test temp directory creation
mkdir /tmp/codexis_test_$(date +%s) && echo "OK" || echo "FAILED"

Common Issues and Solutions

1. Permission Denied Errors

Symptoms:

** (File.Error) could not make directory (with -p) "/tmp/codexis_build_1748057072_abc123de": permission denied

Diagnosis:

# Check temp directory permissions
ls -ld /tmp
# Should show: drwxrwxrwt (1777) for /tmp

# Check available space
df -h /tmp

Solutions:

1. Fix temp directory permissions:

   sudo chmod 1777 /tmp
   

2. Use alternative temp location:

   ./codexis build --all --temp-dir-root ~/temp --source manuscript --output output
   mkdir -p ~/temp  # Ensure directory exists
   

3. Run with specific user permissions:

   # Ensure user has write access to temp directory
   sudo chown $USER:$USER /tmp/codexis_build_*
   

2. Disk Space Issues

Symptoms:

** (File.Error) could not copy file: no space left on device

Diagnosis:

# Check available space
df -h /tmp
df -h /var/tmp

# Check for large temp directories
du -sh /tmp/codexis_build_* 2>/dev/null || echo "No temp directories found"

Solutions:

1. Clean up old temp directories:

   # Remove old build directories (older than 1 day)
   find /tmp -name "codexis_build_*" -type d -mtime +1 -exec rm -rf {} \;
   

2. Use location with more space:

   # Check available locations
   df -h /var/tmp /home/$USER/tmp ~/temp
   
   # Use alternative location
   mkdir -p ~/temp
   ./codexis build --all --temp-dir-root ~/temp --source manuscript --output output
   

3. Enable automatic cleanup:

   # Ensure cleanup happens (default behavior)
   ./codexis build --all --source manuscript --output output
   # Do NOT use --preserve-temp-dir unless debugging
   

3. LaTeX Processing Errors

Symptoms:

LaTeX compilation failed with exit code 1

Diagnosis:

# Preserve temp directory to examine LaTeX logs
./codexis build --format pdf --preserve-temp-dir --show-temp-dir --source manuscript --output output

# Navigate to temp directory (path shown in output)
cd /tmp/codexis_build_1748057072_abc123de

# Check LaTeX log files
cat aux/manuscript.log | grep -i error
cat logs/pdf_build.log

Solutions:

1. Check LaTeX syntax:

   # Examine the source files in temp directory
   cd source/manuscript_source
   
   # Try manual LaTeX compilation for debugging
   lualatex manuscript.tex
   

2. Check missing packages:

   # Look for package errors in log
   grep "LaTeX Error" aux/manuscript.log
   
   # Install missing packages (Ubuntu/Debian)
   sudo apt-get install texlive-latex-extra texlive-fonts-recommended
   
   # Install missing packages (macOS with MacTeX)
   sudo tlmgr install <package-name>
   

3. Verify source file integrity:

   # Compare source files in temp vs original
   diff -r source/manuscript_source/ ../manuscript/
   

4. Format-Specific Build Failures

Symptoms:

HTML build failed: make4ht compilation error
EPUB build failed: missing content.opf

Diagnosis:

# Enable debugging and check format-specific logs
./codexis build --all --preserve-temp-dir --show-temp-dir --source manuscript --output output

# Check format-specific logs
cd /tmp/codexis_build_1748057072_abc123de
cat logs/html_build.log
cat logs/epub_build.log

Solutions:

1. HTML build issues:

   # Check make4ht availability
   which make4ht
   make4ht --version
   
   # Check auxiliary files exist
   ls -la aux/
   
   # Test manual HTML conversion
   cd source/manuscript_source
   make4ht manuscript.tex
   

2. EPUB build issues:

   # Verify HTML build succeeded first
   ls -la output/html/
   
   # Check EPUB structure
   ls -la output/epub/
   
   # Test EPUB validity
   epubcheck output/epub/manuscript.epub  # if epubcheck installed
   

3. Missing auxiliary files:

   # Ensure PDF build completed successfully
   ls -la aux/manuscript.*
   
   # Check that all required aux files exist
   test -f aux/manuscript.aux && echo "AUX OK" || echo "AUX MISSING"
   test -f aux/manuscript.toc && echo "TOC OK" || echo "TOC MISSING"
   

5. Concurrent Build Issues

Symptoms:

** (ArgumentError) cannot build multiple formats concurrently: resource contention

Diagnosis:

# Check system resources
ps aux | grep codexis
top -p $(pgrep -d',' codexis)

# Check for multiple simultaneous builds
pgrep -f "codexis build" | wc -l

Solutions:

1. Wait for current builds to finish:

   # Wait for existing builds
   while pgrep -f "codexis build" > /dev/null; do sleep 5; done
   
   # Then start new build
   ./codexis build --all --source manuscript --output output
   

2. Sequential format building:

   # Build formats one at a time if concurrency issues persist
   ./codexis build --format pdf --source manuscript --output output
   ./codexis build --format html --source manuscript --output output
   ./codexis build --format epub --source manuscript --output output
   

6. File Locking Issues

Symptoms:

** (File.Error) could not copy file: resource temporarily unavailable

Diagnosis:

# Check for file locks
lsof +D /tmp/codexis_build_* 2>/dev/null || echo "No locks found"

# Check filesystem type
df -T /tmp

Solutions:

1. Wait for locks to clear:

   # Wait a moment and retry
   sleep 10
   ./codexis build --all --source manuscript --output output
   

2. Use different temp location:

   # Some filesystems handle locks better
   ./codexis build --all --temp-dir-root /var/tmp --source manuscript --output output
   

Advanced Diagnostics

Build Performance Analysis

# Time the build process
time ./codexis build --all --source manuscript --output output

# Monitor resource usage during build
# Terminal 1: Start build
./codexis build --all --preserve-temp-dir --show-temp-dir --source manuscript --output output

# Terminal 2: Monitor resources
watch -n 1 'ps aux | grep codexis; df -h /tmp'

Memory Usage Investigation

# Check system memory
free -h

# Monitor build memory usage
./codexis build --all --preserve-temp-dir --show-temp-dir --source manuscript --output output &
PID=$!

# Monitor specific process
while kill -0 $PID 2>/dev/null; do
    ps -p $PID -o pid,vsz,rss,comm --no-headers
    sleep 5
done

Network Dependencies

# Check if build requires network access
# Disconnect network temporarily and test
sudo ifconfig en0 down  # macOS
# or
sudo ip link set eth0 down  # Linux

./codexis build --format pdf --source manuscript --output output

# Reconnect
sudo ifconfig en0 up  # macOS
# or  
sudo ip link set eth0 up  # Linux

Environment-Specific Issues

CI/CD Environments

Common Issues:

• Limited /tmp space
• Different user permissions
• Missing LaTeX packages

Solutions:

# Use workspace for temp files
./codexis build --all --temp-dir-root $WORKSPACE/temp --source manuscript --output output

# Install required packages in CI
apt-get update && apt-get install -y texlive-full

# Set explicit permissions
chmod -R 755 manuscript/

Docker Containers

Common Issues:

• Mount point permissions
• Limited container storage

Solutions:

# In Dockerfile
RUN mkdir -p /app/temp && chmod 1777 /app/temp

# Use container temp directory
WORKDIR /app
RUN ./codexis build --all --temp-dir-root /app/temp --source manuscript --output output

Network File Systems (NFS)

Common Issues:

• File locking behavior
• Permission synchronization

Solutions:

# Use local temp directory instead of NFS mount
./codexis build --all --temp-dir-root /var/tmp --source manuscript --output output

# Check NFS mount options
mount | grep nfs

Recovery Procedures

Cleaning Stale Temp Directories

# Find and remove old temp directories
find /tmp -name "codexis_build_*" -type d -mtime +1 -print
find /tmp -name "codexis_build_*" -type d -mtime +1 -exec rm -rf {} \;

# Clean up by size (remove largest first)
du -sh /tmp/codexis_build_* 2>/dev/null | sort -rh | head -5

Resetting Build Environment

# Complete reset
rm -rf /tmp/codexis_build_*
rm -rf output/*

# Clean build
./codexis build --all --clean --source manuscript --output output

Emergency Recovery

# If build system is completely broken, try minimal build
./codexis build --format pdf --source manuscript --output /tmp/emergency_output

# Check if basic LaTeX works
cd manuscript
lualatex manuscript.tex

Getting Help

Collecting Debug Information

# Generate comprehensive debug report
echo "=== System Information ===" > debug_report.txt
uname -a >> debug_report.txt
df -h >> debug_report.txt
free -h >> debug_report.txt

echo "=== Codexis Version ===" >> debug_report.txt
./codexis version >> debug_report.txt

echo "=== LaTeX Installation ===" >> debug_report.txt
which lualatex >> debug_report.txt
lualatex --version >> debug_report.txt

echo "=== Failed Build with Debug ===" >> debug_report.txt
./codexis build --all --preserve-temp-dir --show-temp-dir --source manuscript --output output 2>&1 >> debug_report.txt

Support Resources

1. GitHub Issues: Report bugs with debug information
2. Documentation: Check architecture decision records (ADRs)
3. Test Suite: Run tests to verify installation

   mix test test/codexis/build_manager/build_orchestrator_test.exs
   

Filing Bug Reports

Include the following information:

1. Codexis version: ./codexis version
2. Operating system: uname -a
3. Error message: Full error output
4. Debug report: Output from debug collection above
5. Reproducible test case: Minimal manuscript that fails
6. Temp directory contents: If using --preserve-temp-dir