Skip to content

Add recipe for O2 GPU CI tests #6264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ktf merged 5 commits into from
Aug 18, 2026
Merged

Add recipe for O2 GPU CI tests #6264

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e60699f
Add recipe for O2 GPU CI tests
fweig Jun 23, 2026
dd72bf2
Add package to test GPU builds in deterministic mode
fweig Jul 17, 2026
ab2711b
gpu ci tests: Use alibuild-generate-module
fweig Jul 21, 2026
02c2c5b
Set NO_FTZ in deterministic mode
fweig Jul 22, 2026
357736e
Cleanup
fweig Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions o2-gpu-deterministic-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
package: O2-GPU-deterministic-test
version: "1.0"
requires:
- Vc
- boost
- fmt
- ms_gsl
- TBB
- ROOT
- ONNXRuntime
- GLFW
- gpu-system
build_requires:
- CMake
- Clang
- ninja
- alibuild-recipe-tools
license: GPL-3.0
force_rebuild: true
---
#!/bin/bash -e

if [[ -n ${GPU_SYSTEM_ROOT:-} && -f $GPU_SYSTEM_ROOT/etc/gpu-features-available.sh ]]; then
source $GPU_SYSTEM_ROOT/etc/gpu-features-available.sh
fi

if [[ -n ${O2GPUCI_BACKENDS:-} ]]; then
read -r -a GPU_BACKENDS <<< "${O2GPUCI_BACKENDS//,/ }"
else
GPU_BACKENDS=()
[[ ${O2_GPU_CUDA_AVAILABLE:-0} == 1 ]] && GPU_BACKENDS+=(CUDA)
[[ ${O2_GPU_ROCM_AVAILABLE:-0} == 1 ]] && GPU_BACKENDS+=(HIP)
fi

if [[ ${#GPU_BACKENDS[@]} == 0 ]]; then
echo "O2-GPU-deterministic-test: no GPU backend selected or detected." >&2
echo "Set O2GPUCI_BACKENDS='CUDA,HIP' in CI to require both production GPU backends." >&2
exit 1
fi

# HACK to find O2 sources without depending on O2 as a dependency (and potentially building all of it as a consequence)
O2_SOURCEDIR=${O2GPUCI_O2_SOURCEDIR:-}
for SOURCE_CANDIDATE in "$WORK_DIR/../O2" "$ALIBUILD_CONFIG_DIR/../AliceO2"; do
if [[ -z $O2_SOURCEDIR && -f $SOURCE_CANDIDATE/GPU/GPUTracking/Standalone/CMakeLists.txt ]]; then
O2_SOURCEDIR=$SOURCE_CANDIDATE
fi
done
if [[ -z $O2_SOURCEDIR && -d $WORK_DIR/SOURCES/O2 ]]; then
O2_SOURCEDIR=$(find "$WORK_DIR/SOURCES/O2" -mindepth 2 -maxdepth 2 -type d \
-exec test -f '{}/GPU/GPUTracking/Standalone/CMakeLists.txt' \; -print -quit 2>/dev/null || true)
fi
if [[ ! -f $O2_SOURCEDIR/GPU/GPUTracking/Standalone/CMakeLists.txt ]]; then
echo "O2-GPU-deterministic-test: could not find the O2 source tree." >&2
echo "Set O2GPUCI_O2_SOURCEDIR to the AliceO2 checkout used for this build." >&2
exit 1
fi

rm -Rf "$BUILDDIR/gpu-standalone-test"
mkdir -p "$BUILDDIR/gpu-standalone-test/build" "$BUILDDIR/gpu-standalone-test/install"
pushd "$BUILDDIR/gpu-standalone-test/build"

cp "$O2_SOURCEDIR/GPU/GPUTracking/Standalone/cmake/config.cmake" .
cat >> config.cmake <<'EoF'
set(GPUCA_BUILD_EVENT_DISPLAY 0)
Comment thread
davidrohr marked this conversation as resolved.
set(GPUCA_DETERMINISTIC_MODE GPU)
set(GPUCA_DETERMINISTIC_NO_FTZ 1)
EoF

cmake -DCMAKE_INSTALL_PREFIX=../install "$O2_SOURCEDIR/GPU/GPUTracking/Standalone"
cmake --build . --target install -- ${JOBS:+-j $JOBS}

for BACKEND in "${GPU_BACKENDS[@]}"; do
../install/ca --noEvents -g --gpuType "${BACKEND^^}"
done

popd
rm -Rf "$BUILDDIR/gpu-standalone-test"

# Dummy modulefile
mkdir -p "$INSTALLROOT/etc/modulefiles"
alibuild-generate-module > "$INSTALLROOT/etc/modulefiles/$PKGNAME"
43 changes: 43 additions & 0 deletions o2-gpu-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
package: O2-GPU-test
version: "1.0"
requires:
- O2
- gpu-system
build_requires:
- alibuild-recipe-tools
license: GPL-3.0
force_rebuild: true
---
#!/bin/bash -e

if [[ -n ${GPU_SYSTEM_ROOT:-} && -f $GPU_SYSTEM_ROOT/etc/gpu-features-available.sh ]]; then
source $GPU_SYSTEM_ROOT/etc/gpu-features-available.sh
fi

rm -Rf $BUILDDIR/gpu-test
mkdir $BUILDDIR/gpu-test
pushd $BUILDDIR/gpu-test

if [[ -n ${O2GPUCI_BACKENDS:-} ]]; then
read -r -a GPU_BACKENDS <<< "${O2GPUCI_BACKENDS//,/ }"
else
GPU_BACKENDS=()
[[ ${O2_GPU_CUDA_AVAILABLE:-0} == 1 ]] && GPU_BACKENDS+=(CUDA)

@davidrohr davidrohr Jun 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could add also OpenCL here, at least without processing events, but to check that it can start it up.

@fweig fweig Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need PoCL for that? Or is the OpenCL runtime shipped with ROCm or CUDA enough to start the binary?

@davidrohr davidrohr Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't compile for a device, any 3.x OpenCL runtime should be good enough.

@fweig fweig Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried running with --noEvents with both OpenCL runtimes that we have in the container, but both raise an error. Didn't investigate further. I would postpone this for now, until we have the CI running.

[[ ${O2_GPU_ROCM_AVAILABLE:-0} == 1 ]] && GPU_BACKENDS+=(HIP)
fi

if [[ ${#GPU_BACKENDS[@]} == 0 ]]; then
echo "O2-GPU-test: no GPU backend selected or detected." >&2
exit 1
fi

for BACKEND in "${GPU_BACKENDS[@]}"; do
o2-gpu-standalone-benchmark --noEvents -g --gpuType "${BACKEND^^}"
done

@davidrohr davidrohr Jun 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking a bit. We can use the o2-gpu-standalone-benchmark for performance tests.

But we also need a build with deterministic mode to compare CPU and GPU output.
I think to get that, we should:

  • Use

    alidist/o2.sh

    Line 287 in b16586e

    if [[ ( "$ALIBOT_PR_REPO" == "AliceO2Group/AliceO2" || "$ALIBOT_PR_REPO" == "alisw/alidist" ) && $ALIBUILD_O2_FORCE_GPU == 1 ]]; then
  • Change that to $ALIBUILD_O2_FORCE_GPU == "ci" (The ci string did not exist, when we set that up).
  • Add a third CI build with DETERMINISTIC mode enabled to GPU/GPUTracking/Standalone/cmake/build.sh if GPUCA_STANDALONE_CI=1
  • Change o2.sh and / or build.sh to actually install the standalone builds. Or at least if we add a determinisstic build as last build, we need to install at least that one, don't care for the others for now., to $O2_ROOT/share/GPU/standalone_builds/deterministic_build
  • The standalone builds use RPATH for the GPU libraries: https://github.com/AliceO2Group/AliceO2/blob/cd36608b9783132c889097648b6f2ad44afb5416/GPU/GPUTracking/Standalone/CMakeLists.txt#L21
    • This needs to be relocated, we probably need to talk to Giulio how to do that, but we already do it for the normal installation anyway. The point is that this RPATH is needed, to avoid that standalone libraries and O2 libraries from LD_LIBRARY_PATH can collide, and also in case we want to create multiple standalone builds later. So I think we should keep it.

@fweig fweig Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a new recipe that builds the standalone binary in deterministic mode. It's kind of hacky because I'm compiling the binary in the test script and I have to guess the location of the O2 sources. But this avoids the RPATH issue, because it has it's own installation folder.

@davidrohr davidrohr Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a new recipe that builds the standalone binary in deterministic mode. It's kind of hacky because I'm compiling the binary in the test script and I have to guess the location of the O2 sources. But this avoids the RPATH issue, because it has it's own installation folder.

But this will be faulure-prone. From where do you take the O2 sources? From the O2 build? But it is not guaranteed that the O2 build happens before, it can for instance download a precompiled O2 tarball.
I think we should do it properly and use patchelf to fix the RPATH.

@davidrohr davidrohr Jul 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually thought about this again. Because the standalone test builds are only done in the CI, they will not be done for the tarballs we create, so actually O2 tarballs will not contain the standalone builds, even if we install them to the share folder in o2.sh.
Thus indeed we need a way to access the correct O2 sources. Perhaps, what you have done is the easiest solution for now. We should discuss with @ktf when he is back.

@ktf ktf Jul 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, let's assume the sources are there for now. We will have some way of depending on sources at some point soon and then we can change.


popd
rm -Rf $BUILDDIR/gpu-test

# Dummy modulefile
mkdir -p $INSTALLROOT/etc/modulefiles
alibuild-generate-module > $INSTALLROOT/etc/modulefiles/$PKGNAME
20 changes: 20 additions & 0 deletions o2gpuci.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
package: O2GPUCI
version: "1.0.0"
tag: "O2GPUCI-1.0.0"
requires:
- O2
- O2-GPU-test:(.*x86-64)
- O2-GPU-deterministic-test:(.*x86-64)
build_requires:
- alibuild-recipe-tools
license: GPL-3.0
valid_defaults:
- o2
- o2-epn
- ali
---
#!/bin/bash -ex

# Modulefile
mkdir -p $INSTALLROOT/etc/modulefiles
alibuild-generate-module > $INSTALLROOT/etc/modulefiles/$PKGNAME