BENCHMARKING.md

Benchmarking Guide for Image Color Palette Extractor

This document explains how to run and interpret performance benchmarks for the Image Color Palette Extractor WebAssembly library.

Prerequisites

Before running benchmarks, ensure you have:

1. Rust - Install from rustup.rs
2. Criterion - Added as dev dependency (automatically installed)

Benchmark Suites

This project includes two comprehensive benchmark suites:

1. Full Benchmarks (benches/palette_extraction.rs)

Comprehensive performance testing covering:

• Size Scaling: Performance across different image sizes (32x32 to 512x512)
• Color Count Impact: How different k values (1-32 colors) affect performance
• Pattern Complexity: Performance with different image patterns (solid, gradient, random, etc.)
• Configuration Impact: Effect of max_iterations and convergence settings
• Utility Functions: Performance of color distance, sorting, and filtering
• Memory Usage: Memory allocation patterns
• Regression Testing: Consistent baseline measurements

2. Micro Benchmarks (benches/micro_benchmarks.rs)

Quick performance checks for core operations:

• Color creation and conversion
• Distance calculations
• Small palette extractions
• Format conversions (hex, RGB strings)

Running Benchmarks

Option 1: Using the Benchmark Scripts

Windows (PowerShell)

.\run-benchmarks.ps1

Linux/macOS (Bash)

chmod +x run-benchmarks.sh
./run-benchmarks.sh

Option 2: Manual Commands

# Run all benchmarks
cargo bench

# Run only full benchmarks
cargo bench --bench palette_extraction

# Run only micro benchmarks  
cargo bench --bench micro_benchmarks

# Run specific benchmark groups
cargo bench palette_extraction_by_size
cargo bench utility_functions
cargo bench micro_benchmarks

Benchmark Structure

benches/
├── palette_extraction.rs    # Comprehensive benchmarks
└── micro_benchmarks.rs     # Quick performance checks

target/criterion/           # Generated benchmark results
├── report/                 # HTML reports
└── */base/                # Raw benchmark data

Understanding Results

Criterion Output

Criterion provides detailed statistics:

palette_extraction_by_size/random_image/128x128
                        time:   [45.234 ms 45.892 ms 46.621 ms]
                        thrpt:  [351.88 Kelem/s 357.42 Kelem/s 362.54 Kelem/s]

Key Metrics:

• Time: How long the operation takes
• Throughput: Elements processed per second
• Confidence Intervals: Statistical reliability ranges

Performance Categories

🟢 Excellent Performance

• Small images (32x32): < 1ms
• Medium images (128x128): < 50ms
• Large images (256x256): < 200ms

🟡 Good Performance

• Small images: 1-5ms
• Medium images: 50-100ms
• Large images: 200-500ms

🔴 Needs Optimization

• Small images: > 5ms
• Medium images: > 100ms
• Large images: > 500ms

Benchmark Categories

1. Size Scaling Benchmarks

cargo bench palette_extraction_by_size

Tests how performance scales with image size.

Expected Results:

• Linear scaling with pixel count
• Throughput should remain relatively stable

2. Color Count Impact

cargo bench palette_extraction_by_k_colors

Tests performance with different numbers of extracted colors.

Expected Results:

• Slight increase in time with more colors
• K=1 (dominant color) should be fastest

3. Pattern Complexity

cargo bench palette_extraction_by_pattern

Tests how image complexity affects performance.

Expected Results:

• Solid colors: Fastest (early convergence)
• Random patterns: Slowest (requires full iterations)
• Gradients/bands: Medium performance

4. Configuration Impact

cargo bench extractor_configuration

Tests how algorithm settings affect performance.

Key Insights:

• Higher max_iterations = longer runtime
• Lower convergence threshold = more iterations
• Find optimal balance for your use case

5. Utility Functions

cargo bench utility_functions

Tests performance of helper functions.

Critical Functions:

• color_distance_rgb: Should be sub-microsecond
• sort_colors_by_luminance: Scales with O(n log n)
• remove_similar_colors: Scales with O(n²)

Performance Optimization Tips

For Developers

1. Batch Processing: Process multiple images together
2. Optimal K Values: Use k=5-8 for best balance
3. Configuration Tuning:
   • max_iterations: 20 (default is good)
   • convergence: 5.0 (default is good)
4. Pre-filtering: Remove similar colors before palette extraction

For Users

1. Image Size: Resize large images before processing
2. Color Count: Request fewer colors for faster processing
3. Caching: Cache results for identical images
4. Progressive Enhancement: Start with dominant color, add more as needed

Regression Testing

Setting Baselines

# Save current performance as baseline
cargo bench -- --save-baseline before_optimization

# After changes, compare to baseline
cargo bench -- --baseline before_optimization

Automated Performance Monitoring

# In CI/CD, fail if performance degrades by >10%
cargo bench -- --baseline main --threshold 10

Interpreting HTML Reports

Criterion generates detailed HTML reports in target/criterion/report/index.html.

Key Report Sections:

1. Summary: Overall performance overview
2. Individual Benchmarks: Detailed timing analysis
3. Plots: Visual performance trends
4. Comparisons: Before/after analysis
5. Statistics: Confidence intervals and outliers

Important Plots:

• PDF Plot: Distribution of measurement times
• Regression Plot: Performance over iterations
• Mean Plot: Average performance with confidence intervals

CI/CD Integration

GitHub Actions Example

name: Performance Benchmarks

on: [push, pull_request]

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Install Rust
      uses: actions-rs/toolchain@v1
      with:
        toolchain: stable
    - name: Run benchmarks
      run: cargo bench --bench micro_benchmarks
    - name: Upload results
      uses: actions/upload-artifact@v2
      with:
        name: benchmark-results
        path: target/criterion/

Memory Profiling

For detailed memory analysis:

# Install valgrind (Linux) or Instruments (macOS)
cargo bench --bench palette_extraction -- --profile-time=10

# Or use cargo instruments on macOS
cargo instruments -t "Time Profiler" --bench palette_extraction

Performance Targets

Target Benchmarks (Release Mode)

Operation	Image Size	Target Time	Notes
Dominant Color	128x128	< 10ms	Single color extraction
5-Color Palette	128x128	< 50ms	Standard use case
8-Color Palette	256x256	< 200ms	Large image processing
Color Distance	Any	< 1μs	Utility function
Hex Conversion	Any	< 100ns	Format conversion

Scaling Expectations

• Linear with pixels: 2x pixels ≈ 2x time
• Sub-linear with colors: 2x colors ≈ 1.5x time
• Constant utilities: Independent of input size

Troubleshooting

Slow Benchmarks

1. Check CPU load: Ensure system isn't under load
2. Disable power management: Use high-performance mode
3. Close other applications: Minimize background processes
4. Run multiple times: Average results over several runs

Inconsistent Results

1. System noise: Use --warm-up-time and --measurement-time
2. Thermal throttling: Monitor CPU temperature
3. Memory pressure: Ensure sufficient RAM
4. Background tasks: Disable automatic updates

Build Issues

# Clean and rebuild
cargo clean
cargo bench

# Update dependencies
cargo update

Contributing Performance Improvements

When submitting performance improvements:

1. Run full benchmark suite before and after changes
2. Include performance comparison in PR description
3. Document any trade-offs (speed vs accuracy)
4. Test across different scenarios (image sizes, patterns)
5. Verify memory usage doesn't increase significantly

Advanced Benchmarking

Custom Benchmarks

To add new benchmarks, create functions in benches/palette_extraction.rs:

fn bench_your_feature(c: &mut Criterion) {
    let mut group = c.benchmark_group("your_feature");
    
    group.bench_function("test_case", |b| {
        b.iter(|| {
            // Your benchmark code here
            black_box(your_function(black_box(input)))
        });
    });
    
    group.finish();
}

// Add to criterion_group! macro
criterion_group!(benches, bench_your_feature, /* ... other benchmarks */);

Statistical Analysis

For deeper analysis, export raw data:

# Export to CSV
cargo bench -- --output-format csv > results.csv

# Custom analysis with external tools
python analyze_performance.py results.csv