harden Qwen3.8 nightly scheduling and output benchmark

[groxaxo]

What changed

• pins the autonomous schedule to an explicit IANA timezone (Pacific/Auckland by default)
• calculates the real elapsed budget to the morning stop with DST-safe timestamp arithmetic
• installs separate local-midnight start and 07:00 stop timers, with the runner enforcing the same boundary independently
• stops the complete systemd control group and retains four-layer chunk-boundary resumability
• adds an exclusive flock so manual and timer invocations cannot run concurrently
• replaces marker-exists completion logic with a v2 marker bound by SHA-256 to the exact final manifest and benchmark report
• makes the post-build benchmark fail closed on artifact incompatibility, context OOM/runtime errors, missed long-context needles, malformed generated Python/JSON, weak diagnostic output, or failed Spanish output
• validates the final 64-layer/g64/max-quality/400-module assembly before model loading
• records load time, peak VRAM, TTFT proxy, end-to-end throughput, decode proxy, exact-token context cases, quality transcripts, and explicit failure reasons
• adds focused unit tests for Auckland DST transitions, completion-marker integrity, artifact metadata, exact context construction, and quality validators
• updates the Qwen3.8 operator documentation and installation commands

Review findings fixed

The earlier benchmark could record a failed 32K case and still return success because only empty quality responses affected its exit code. That could incorrectly create the final done marker. The earlier wall-clock helper also did not explicitly protect elapsed-time calculations across Auckland daylight-saving transitions.

This PR changes completion to require all of:

artifact_validation.passed = true
all_context_cases_passed   = true
all_quality_smoke_passed   = true
overall_passed             = true

No passing benchmark means no benchmark.done.json.

Nightly behavior

• start: 00:00 in NIGHTLY_TIMEZONE
• graceful runner deadline: NIGHTLY_STOP_TIME minus STOP_GRACE_SECONDS
• independent systemd stop: exactly NIGHTLY_STOP_TIME
• restart unit: four decoder layers
• completed chunks: preserved
• active interrupted chunk: repeated the next night
• completed model + valid strict benchmark: later invocations are no-ops
• modified/stale report or manifest: benchmark is regenerated

Validation performed

python -m py_compile \
  scripts/qwen38_nightly_state.py \
  scripts/benchmark_qwen38_gptqpro.py

bash -n \
  scripts/qwen38_nightly_runner.sh \
  scripts/install_qwen38_nightly_systemd.sh

pytest -q \
  tests/test_qwen38_nightly_state.py \
  tests/test_qwen38_benchmark_helpers.py
# 6 passed

systemd-analyze calendar '*-*-* 00:00:00 Pacific/Auckland'
systemd-analyze calendar '*-*-* 07:00:00 Pacific/Auckland'

The DST tests explicitly verify:

• Auckland fall-back night: midnight to 07:00 is 8 elapsed hours
• Auckland spring-forward night: midnight to 07:00 is 6 elapsed hours

Target-machine validation before merge

This remains a draft until the Ubuntu/RTX 3090 host runs the installer and one real service/benchmark smoke test:

bash scripts/install_qwen38_nightly_systemd.sh
systemctl --user status \
  gptq-pro-qwen38-nightly.timer \
  gptq-pro-qwen38-nightly-stop.timer
journalctl --user -u gptq-pro-qwen38-nightly.service -f

A full multi-night 27B quantization is not required to validate the schedule, but the assembled model benchmark must ultimately run on the target GPU before deleting the BF16 source or publishing the artifact.