Add recipe for O2 GPU CI tests

[fweig]

Add new recipes for O2 GPU CI tests.

Structure follows the full CI recipe:

• Added a meta recipe for GPU tests
• Added a test recipe that runs o2-gpu-standalone-benchmark

Test script currently only runs o2-gpu-standalone-benchmark --noEvents for available architectures to test that GPUs are present in container.

[davidrohr]

Nice, convenient that this is so straight forward. So for the actual tests, I would like to test at least 2 things:

• Run the o2-gou-standalone-benchmark on a Pb-Pb reference dataset on CPU and GPU, and compare the processing time to a reference time, and give a warning if we get 5% slower than the reference.
• Run the standalone compilation in deterministic mode on CPU, GPU, and GPU with RTC on a reference data set (can be smaller than the PbPb data set, that we use for performance), and check that the dump files are identical. (In the long run, we should add a feature that we do not write huge dump files but just create an SHA512 hash of the file on the fly.

With the latest version, you can use ./ca --recreateTrivialCalibObjects to create default calibration objects for the current software version. I.e. we need only the event and settings dump file from the data, and we can create the rest on the fly.

2 detailed comments inline.

[davidrohr]

We could add also OpenCL here, at least without processing events, but to check that it can start it up.

[fweig]

Do we need PoCL for that? Or is the OpenCL runtime shipped with ROCm or CUDA enough to start the binary?

[davidrohr]

If we don't compile for a device, any 3.x OpenCL runtime should be good enough.

[fweig]

I tried running with --noEvents with both OpenCL runtimes that we have in the container, but both raise an error. Didn't investigate further. I would postpone this for now, until we have the CI running.

[davidrohr]

I was thinking a bit. We can use the o2-gpu-standalone-benchmark for performance tests.

But we also need a build with deterministic mode to compare CPU and GPU output.
I think to get that, we should:

• Use

  alidist/o2.sh

  Line 287 in b16586e

  if [[ ( "$ALIBOT_PR_REPO" == "AliceO2Group/AliceO2" || "$ALIBOT_PR_REPO" == "alisw/alidist" ) && $ALIBUILD_O2_FORCE_GPU == 1 ]]; then

• Change that to $ALIBUILD_O2_FORCE_GPU == "ci" (The ci string did not exist, when we set that up).
• Add a third CI build with DETERMINISTIC mode enabled to GPU/GPUTracking/Standalone/cmake/build.sh if GPUCA_STANDALONE_CI=1
• Change o2.sh and / or build.sh to actually install the standalone builds. Or at least if we add a determinisstic build as last build, we need to install at least that one, don't care for the others for now., to $O2_ROOT/share/GPU/standalone_builds/deterministic_build
• The standalone builds use RPATH for the GPU libraries: https://github.com/AliceO2Group/AliceO2/blob/cd36608b9783132c889097648b6f2ad44afb5416/GPU/GPUTracking/Standalone/CMakeLists.txt#L21
  • This needs to be relocated, we probably need to talk to Giulio how to do that, but we already do it for the normal installation anyway. The point is that this RPATH is needed, to avoid that standalone libraries and O2 libraries from LD_LIBRARY_PATH can collide, and also in case we want to create multiple standalone builds later. So I think we should keep it.

[fweig]

I've added a new recipe that builds the standalone binary in deterministic mode. It's kind of hacky because I'm compiling the binary in the test script and I have to guess the location of the O2 sources. But this avoids the RPATH issue, because it has it's own installation folder.

[davidrohr]

I've added a new recipe that builds the standalone binary in deterministic mode. It's kind of hacky because I'm compiling the binary in the test script and I have to guess the location of the O2 sources. But this avoids the RPATH issue, because it has it's own installation folder.

But this will be faulure-prone. From where do you take the O2 sources? From the O2 build? But it is not guaranteed that the O2 build happens before, it can for instance download a precompiled O2 tarball.
I think we should do it properly and use patchelf to fix the RPATH.

[davidrohr]

I actually thought about this again. Because the standalone test builds are only done in the CI, they will not be done for the tarballs we create, so actually O2 tarballs will not contain the standalone builds, even if we install them to the share folder in o2.sh.
Thus indeed we need a way to access the correct O2 sources. Perhaps, what you have done is the easiest solution for now. We should discuss with @ktf when he is back.

[ktf]

As discussed, let's assume the sources are there for now. We will have some way of depending on sources at some point soon and then we can change.

[davidrohr]

Looks good in general, just 2 more minor comments from my side.
Please fix all the comments, then we can merge it.