Skip to content

README.md

Latest commit

 

History

History
69 lines (49 loc) · 2.85 KB

README.md

File metadata and controls

69 lines (49 loc) · 2.85 KB
Raw

Reproducing the PRISMA experiments

The standalone runner executes the two experiments formerly implemented as disabled manual JUnit tests:

  • exp1: PRISMA order-preserving substitution, with LLM-generated pool data;
  • exp2: direct single-call Ollama substitution baseline;
  • all: exp1, then exp2, then comparison of exactly the CSV files generated by those phases.

Paper configuration

paper.env records the benchmark parameters explicitly. These values match the former scripts instead of relying on runner defaults:

Setting Value
Model qwen2.5:7b
Exp1 warmup runs 3 per scenario
Exp1 measured runs 10 per scenario
Exp2 repetitions 5 per scenario
Exp1 pool strategy llm, with its cache disabled by the runner
Temperature 0.7
Seed unset (none)
Ollama timeout 150 seconds
Scenarios the explicit eight-scenario list in paper.env

Exp2 always uses the deterministic instant strategy for scenario setup and Ollama for the direct substitution call. POOL_STRATEGY controls exp1.

Runner image reference

The experiment runner is distributed as the following public container image, referenced by its immutable OCI digest:

ghcr.io/inlabfib/prisma/prisma-experiments:paper-v1

The image digest is already fixed in docker-compose.experiments.yml; no registry variable or GitHub authentication is required. Ollama uses docker.io/ollama/ollama:latest from the main docker-compose.yml.

PowerShell:

docker compose `
  -f docker-compose.yml `
  -f docker-compose.experiments.yml `
  --profile experiments `
  run --rm experiments all

Bash:

docker compose \
  -f docker-compose.yml \
  -f docker-compose.experiments.yml \
  --profile experiments \
  run --rm experiments all

Ollama downloads qwen2.5:7b on the first run and stores it in the existing ollama_data volume. Ensure sufficient disk space before starting.

Output

The generated CSV files are written to results/. The runner does not generate metadata JSON files.

  • Exp1 produces one summary CSV.
  • Exp2 produces one raw CSV containing every attempted run and one summary CSV.
  • all additionally produces the combined comparison CSV.

Reproducibility boundary

The original scripts fixed the model name and generation parameters but did not record the Ollama binary version or model digest. The experiment setup intentionally continues to use the latest Ollama image from the main Compose file. Consequently, it is not possible to confirm or guarantee that the LLM will behave exactly the same way across executions: generated responses, substitutions, constraint violations, and latency measurements may differ. The reproduction package is intended to reproduce the experimental procedure and configuration, not byte-identical CSV files or identical LLM outputs.