Adding asan sanitizer support for hip by kab163 · Pull Request #795 · llnl/Umpire

kab163 · 2022-11-23T23:19:39Z

Cmake command on rzvernal:

cmake -DROCM_ROOT_DIR=/opt/rocm-5.3.0 -DHIP_ROOT_DIR=/opt/rocm-5.3.0/hip -DHIP_PATH=/opt/rocm-5.3.0/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_COMPILER=/opt/rocm-5.3.0/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-5.3.0/llvm/bin/amdclang -DCMAKE_CXX_FLAGS="-fsanitize=address" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES=gfx90a:xnack+ ../

Warning while compiling: clang-15: warning: ignoring '-fsanitize=address' option for offload arch 'gfx90a' as it is not currently supported there. Use it with an offload arch containing 'xnack+' instead [-Woption-ignored]

Output of sanitizer_tests.cpp

[belcher6@rzvernal11:build]$ ./bin/sanitizer_tests QuickPool write
data[INDEX] = 100
=================================================================
==79729==ERROR: AddressSanitizer: use-after-poison on address 0x154d16201530 at pc 0x0000004f5eb4 bp 0x7fffffff8b10 sp 0x7fffffff8b08
WRITE of size 8 at 0x154d16201530 thread T0
    #0 0x4f5eb3 in test_write_after_free() (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5eb3)
    #1 0x4f67cb in main (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f67cb)
    #2 0x15555267acf2 in __libc_start_main (/lib64/libc.so.6+0x3acf2) (BuildId: 64aa558dcdda2d8b0d7b04cef33ddbb2d9d8b8b4)
    #3 0x404e4d in _start (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x404e4d)

Address 0x154d16201530 is a wild pointer inside of access range of size 0x000000000008.
SUMMARY: AddressSanitizer: use-after-poison (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5eb3) in test_write_after_free()
Shadow bytes around the buggy address:
  0x02aa22c38250: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38260: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38270: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38280: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38290: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
=>0x02aa22c382a0: f7 f7 f7 f7 f7 f7[f7]f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382b0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382d0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382e0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c382f0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==79729==ABORTING
[belcher6@rzvernal11:build]$ ./bin/sanitizer_tests QuickPool read
data[INDEX] = 100
=================================================================
==79736==ERROR: AddressSanitizer: use-after-poison on address 0x154d16200800 at pc 0x0000004f5c22 bp 0x7fffffff8b30 sp 0x7fffffff8b28
READ of size 8 at 0x154d16200800 thread T0
    #0 0x4f5c21 in test_read_after_free() (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5c21)
    #1 0x4f679e in main (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f679e)
    #2 0x15555267acf2 in __libc_start_main (/lib64/libc.so.6+0x3acf2) (BuildId: 64aa558dcdda2d8b0d7b04cef33ddbb2d9d8b8b4)
    #3 0x404e4d in _start (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x404e4d)

Address 0x154d16200800 is a wild pointer inside of access range of size 0x000000000008.
SUMMARY: AddressSanitizer: use-after-poison (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x4f5c21) in test_read_after_free()
Shadow bytes around the buggy address:
  0x02aa22c380b0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380d0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380e0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c380f0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
=>0x02aa22c38100:[f7]f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38110: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38120: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38130: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38140: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x02aa22c38150: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==79736==ABORTING

kab163 · 2022-12-06T18:03:55Z

Updated cmake command:

cmake -DROCM_ROOT_DIR=/opt/rocm-5.4.0 -DHIP_ROOT_DIR=/opt/rocm-5.4.0/hip -DHIP_PATH=/opt/rocm-5.4.0/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_COMPILER=/opt/rocm-5.4.0/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-5.4.0/llvm/bin/amdclang -DCMAKE_CXX_FLAGS="-fsanitize=address" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx90a:xnack+" -DGPU_TARGETS="gfx90a:xnack+" -DAMDGPU_TARGETS="gfx90a:xnack+" ../

Updated Output (for hip kernel sanitizer test):

bash-4.4$ ./bin/sanitizer_tests QuickPool read
AddressSanitizer:DEADLYSIGNAL
=================================================================
==333094==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000020 (pc 0x155553a4b9cd bp 0x7fffffff7ab0 sp 0x7fffffff7a50 T0)
==333094==The signal is caused by a READ memory access.
==333094==Hint: address points to the zero page.
    #0 0x155553a4b9cd  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd59cd) (BuildId: a4e47799322466785f920489031c315137965c08)
    #1 0x155553a4bb26  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd5b26) (BuildId: a4e47799322466785f920489031c315137965c08)
    #2 0x155553a4c9fe  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd69fe) (BuildId: a4e47799322466785f920489031c315137965c08)
    #3 0x155553a0a5d6  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x945d6) (BuildId: a4e47799322466785f920489031c315137965c08)
    #4 0x155553b5de15  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x1e7e15) (BuildId: a4e47799322466785f920489031c315137965c08)
    #5 0x155553b39881  (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x1c3881) (BuildId: a4e47799322466785f920489031c315137965c08)
    #6 0x155553b3af93 in hipLaunchKernel (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0x1c4f93) (BuildId: a4e47799322466785f920489031c315137965c08)
    #7 0x543a21 in __device_stub__test_for_hip(double**, unsigned long) (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x543a21)
    #8 0x544cdf in main (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x544cdf)
    #9 0x15555267dcf2 in __libc_start_main (/lib64/libc.so.6+0x3acf2) (BuildId: 64aa558dcdda2d8b0d7b04cef33ddbb2d9d8b8b4)
    #10 0x45285d in _start (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x45285d)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/opt/rocm-5.4.0/lib/libamdhip64.so.5+0xd59cd) (BuildId: a4e47799322466785f920489031c315137965c08)
==333094==ABORTING

Note: Will not work if HSA_XNACK is set to 1, do not export... explanation TBD

kab163 · 2022-12-07T23:40:59Z

Issue: it seems like running the test with ./bin/sanitizer_tests QuickPool read HOST and ./bin/sanitizer_tests QuickPool write HOST both give an error that detects a write access....
(Whether it is build with hip support or without it)

kab163 · 2022-12-08T00:28:32Z

The way this test is ran in make test might need to be changed...

kab163 · 2022-12-21T17:22:57Z

See AMD bug report (277) - made reproducer for AMD to look in to. Seems like this does not yet work as expected in rocm 5.4

kab163 · 2023-03-02T22:49:36Z

Update: A fix is supposed to be ready in rocm 6.0

…an-poisoning-for-hip

kab163 · 2023-04-03T16:07:23Z

Edited out...

…an-poisoning-for-hip

…-1020-asan-poisoning-for-hip

kab163 · 2024-01-22T18:23:23Z

The fix is in rocm/6.0.0, but we have to add a "-Wl,-rpath=/opt/rocm-6.0.0/llvm/lib/clang/17.0.0/lib/linux/" flag. Once we update the corona/tioga jobs to rocm-6.0.0 and get that in, it should be good to go.

kab163 · 2024-01-24T16:12:28Z

This has been a long time coming! @adrienbernede - once the blt/camp releases come out and we can update umpire to rocm/6.0.0 we should add this test! (No updates yet for the reason behind why the device allocator test is failing with 6.0.0 - ticket has been "submitted to vendor", so it is being looked into at least...)

kab163 · 2025-03-13T14:52:01Z

Have you tried rocm 6.3.1? I know at least one internal rocm memory issue was fixed between rocm 6.3.0 and rocm 6.3.1.

I have not tried that, but I can try that now!

…an-poisoning-for-hip

kab163 · 2025-03-24T20:35:45Z

Had some troubles compiling all of a sudden, but was able to compile again after adding a new flag. Updated cmake:

cmake -DROCM_ROOT_DIR=/opt/rocm-6.3.1 -DHIP_ROOT_DIR=/opt/rocm-6.3.1/hip -DHIP_PATH=/opt/rocm-6.3.1/llvm/bin -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DCMAKE_CXX_COMPILER=/opt/rocm-6.3.1/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-6.3.1/llvm/bin/amdclang -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_FLAGS="-fsanitize-recover -fsanitize=address -shared-libsan" -DCMAKE_EXE_LINKER_FLAGS="-L/opt/rocm-6.3.1/lib/asan/ -L/opt/rocm-6.3.1/llvm/lib/asan -Wl,-rpath,/opt/rocm-6.3.1/lib/asan/:/opt/rocm-6.3.1/llvm/lib/asan:/opt/rocm-6.3.1/lib/llvm/lib/clang/18/lib/linux -lasan" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx942:xnack+" -DGPU_TARGETS="gfx942:xnack+" -DAMDGPU_TARGETS="gfx942:xnack+" -DCAMP_ENABLE_TESTS=Off ../

adayton1 · 2025-06-23T17:58:11Z

+__global__ void test_write_for_hip(double* data_ptr, std::size_t INDEX)
+{
+  if (threadIdx.x == 0) {
+    data_ptr[INDEX] = 256;


Are two writes needed?

adayton1 · 2025-06-23T18:01:40Z


-  if (strategy.find("QuickPool") != std::string::npos) {
-    auto pool = rm.makeAllocator<umpire::strategy::QuickPool>("test_allocator", rm.getAllocator("HOST"));
+  if ((resource_type.find("DEVICE") != std::string::npos) || (resource_type.find("UM") != std::string::npos)) {


I think we might actually want == instead of find.

adayton1 · 2025-06-23T18:06:06Z

+    run_sanitizer_test('DynamicPoolList', 'read', 'HOST')
+    run_sanitizer_test('DynamicPoolList', 'write', 'HOST')
+    run_sanitizer_test('QuickPool', 'read', 'HOST')
+    run_sanitizer_test('QuickPool', 'write', 'HOST')


Any device tests?

adayton1 · 2025-06-24T22:56:09Z

What's the current status of this merge request? I'm testing out asan and finding it works for simple tests. I'm working on enabling it in my application, but until this goes in, I'll have to disable memory pools.

Whether I'm running on rzvernal or rzadams, I see tests fail with memory leaks detected. So for example, on rzvernal:

[belcher6@rzvernal11:build]$ ctest -T test -R memory_resource_traits_tests --output-on-failure
   Site: rzvernal11
   Build name: Linux-amdclang++
Test project /g/g0/belcher6/Umpire/build
    Start 28: memory_resource_traits_tests
1/1 Test #28: memory_resource_traits_tests .....***Failed    0.51 sec
Running main() from /g/g0/belcher6/Umpire/blt/thirdparty_builtin/googletest/googletest/src/gtest_main.cc
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from MemoryResourceTraits/MemoryResourceTraitsTest
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/0
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/0 (0 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/1
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/1 (1 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/2
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/2 (0 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/3
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/3 (0 ms)
[----------] 4 tests from MemoryResourceTraits/MemoryResourceTraitsTest (1 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (1 ms total)
[  PASSED  ] 4 tests.

=================================================================
==1444299==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 648 byte(s) in 9 object(s) allocated from:
    #0 0x5ea53d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/bin/memory_resource_traits_tests+0x5ea53d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 384 byte(s) in 8 object(s) allocated from:
    #0 0x5ac189 in calloc (/g/g0/belcher6/Umpire/build/bin/memory_resource_traits_tests+0x5ac189)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 1032 byte(s) leaked in 17 allocation(s).

or on rzadams:

[belcher6@rzadams1060:build]$ ctest -T test -R blt_hip_smoke --output-on-failure
   Site: rzadams1001
   Build name: Linux-amdclang++
Test project /g/g0/belcher6/Umpire/build
    Start 3: blt_hip_smoke
1/1 Test #3: blt_hip_smoke ....................***Failed    0.50 sec

=================================================================
==1973496==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 360 byte(s) in 5 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 192 byte(s) in 4 object(s) allocated from:
    #0 0x3071a9 in calloc (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x3071a9)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 552 byte(s) leaked in 9 allocation(s).

I did not see any leaks with rocm 6.4.1 on rzadams with blt_hip_smoke or resource_manager_tests!

kab163 · 2025-07-03T19:17:22Z

What's the current status of this merge request? I'm testing out asan and finding it works for simple tests. I'm working on enabling it in my application, but until this goes in, I'll have to disable memory pools.

Whether I'm running on rzvernal or rzadams, I see tests fail with memory leaks detected. So for example, on rzvernal:

[belcher6@rzvernal11:build]$ ctest -T test -R memory_resource_traits_tests --output-on-failure
   Site: rzvernal11
   Build name: Linux-amdclang++
Test project /g/g0/belcher6/Umpire/build
    Start 28: memory_resource_traits_tests
1/1 Test #28: memory_resource_traits_tests .....***Failed    0.51 sec
Running main() from /g/g0/belcher6/Umpire/blt/thirdparty_builtin/googletest/googletest/src/gtest_main.cc
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from MemoryResourceTraits/MemoryResourceTraitsTest
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/0
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/0 (0 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/1
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/1 (1 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/2
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/2 (0 ms)
[ RUN      ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/3
[       OK ] MemoryResourceTraits/MemoryResourceTraitsTest.ResourceTraitTest/3 (0 ms)
[----------] 4 tests from MemoryResourceTraits/MemoryResourceTraitsTest (1 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (1 ms total)
[  PASSED  ] 4 tests.

=================================================================
==1444299==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 648 byte(s) in 9 object(s) allocated from:
    #0 0x5ea53d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/bin/memory_resource_traits_tests+0x5ea53d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 384 byte(s) in 8 object(s) allocated from:
    #0 0x5ac189 in calloc (/g/g0/belcher6/Umpire/build/bin/memory_resource_traits_tests+0x5ac189)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 1032 byte(s) leaked in 17 allocation(s).

or on rzadams:

[belcher6@rzadams1060:build]$ ctest -T test -R blt_hip_smoke --output-on-failure
   Site: rzadams1001
   Build name: Linux-amdclang++
Test project /g/g0/belcher6/Umpire/build
    Start 3: blt_hip_smoke
1/1 Test #3: blt_hip_smoke ....................***Failed    0.50 sec

=================================================================
==1973496==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 360 byte(s) in 5 object(s) allocated from:
    #0 0x34555d in operator new(unsigned long) (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x34555d)
    #1 0x1555495c67e3  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x617e3) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

Indirect leak of 192 byte(s) in 4 object(s) allocated from:
    #0 0x3071a9 in calloc (/g/g0/belcher6/Umpire/build/tests/blt_hip_smoke+0x3071a9)
    #1 0x15554966fc66  (/opt/rocm-6.3.0/lib/libhsa-runtime64.so.1+0x10ac66) (BuildId: 6687448ccff9be158c2b9761b2293a2551b830e1)

SUMMARY: AddressSanitizer: 552 byte(s) leaked in 9 allocation(s).

I did not see any leaks with rocm 6.4.1 on rzadams with blt_hip_smoke or resource_manager_tests!

How did you build it?

kab163 · 2026-03-12T21:40:08Z

Latest build:

cmake \
>     -DROCM_ROOT_DIR=/opt/rocm-7.2.0 \
>     -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off \
>     -DCMAKE_CXX_COMPILER=/opt/rocm-7.2.0/llvm/bin/amdclang++ \
>     -DCMAKE_C_COMPILER=/opt/rocm-7.2.0/llvm/bin/amdclang \
>     -DUMPIRE_ENABLE_ASAN=On -DUMPIRE_ENABLE_SANITIZER_TESTS=ON \
>     -DCMAKE_C_FLAGS="-fsanitize=address" \
>     -DCMAKE_CXX_FLAGS="-fsanitize=address" \
>     -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address -Wl,-rpath,/opt/rocm-7.2.0/llvm/lib/clang/18/lib/linux" \
>     -DCMAKE_HIP_ARCHITECTURES="gfx942:xnack+" \
>     -DCAMP_ENABLE_TESTS=Off \
>     ../

…an-poisoning-for-hip

kab163 · 2026-04-02T20:37:07Z

Latest updates.... need to use rocm 6.4.2 only. With

cmake -DROCM_ROOT_DIR=/opt/rocm-6.4.2 -DHIP_ROOT_DIR=/opt/rocm-6.4.2/include/hip -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DCMAKE_CXX_COMPILER=/opt/rocm-6.4.2/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-6.4.2/llvm/bin/amdclang -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_FLAGS="-fsanitize=address -shared-libsan" -DCMAKE_EXE_LINKER_FLAGS="-Wl,-rpath=/opt/rocm-6.4.2/llvm/lib/clang/19/lib/linux/ -Wl,-rpath=/opt/rocm-6.4.2/lib/asan -fsanitize=address -shared-libsan" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx942:xnack+" -DGPU_TARGETS="gfx942:xnack+" -DAMDGPU_TARGETS="gfx942:xnack+" -DCAMP_ENABLE_TESTS=Off ../

I can run the sanitizer tests with:

HSA_XNACK=1 ./bin/sanitizer_tests QuickPool read DEVICE

and get:

 Conducting sanitizer test with QuickPool strategy, read test type, and the DEVICE resource.
data[INDEX] = 100
data[256] = 0

or run with:

HSA_XNACK=1 ./bin/sanitizer_tests QuickPool write DEVICE

and get:

Conducting sanitizer test with QuickPool strategy, write test type, and the DEVICE resource.
data[INDEX] = 100
data[INDEX] = -1

kab163 · 2026-04-06T21:17:58Z

Latest updates.... need to use rocm 6.4.2 only. With

cmake -DROCM_ROOT_DIR=/opt/rocm-6.4.2 -DHIP_ROOT_DIR=/opt/rocm-6.4.2/include/hip -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DCMAKE_CXX_COMPILER=/opt/rocm-6.4.2/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-6.4.2/llvm/bin/amdclang -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_FLAGS="-fsanitize=address -shared-libsan" -DCMAKE_EXE_LINKER_FLAGS="-Wl,-rpath=/opt/rocm-6.4.2/llvm/lib/clang/19/lib/linux/ -Wl,-rpath=/opt/rocm-6.4.2/lib/asan -fsanitize=address -shared-libsan" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx942:xnack+" -DGPU_TARGETS="gfx942:xnack+" -DAMDGPU_TARGETS="gfx942:xnack+" -DCAMP_ENABLE_TESTS=Off ../

I can run the sanitizer tests with:

HSA_XNACK=1 ./bin/sanitizer_tests QuickPool read DEVICE

and get:

 Conducting sanitizer test with QuickPool strategy, read test type, and the DEVICE resource.
data[INDEX] = 100
data[256] = 0

or run with:

HSA_XNACK=1 ./bin/sanitizer_tests QuickPool write DEVICE

and get:

Conducting sanitizer test with QuickPool strategy, write test type, and the DEVICE resource.
data[INDEX] = 100
data[INDEX] = -1

Though it should be noted that the expected output should look something like the first comment on this branch, just with the correct output....

…an-poisoning-for-hip

kab163 · 2026-05-12T19:46:30Z

Update:
Build with:

cmake -DROCM_ROOT_DIR=/opt/rocm-7.2.1 -DHIP_ROOT_DIR=/opt/rocm-7.2.1/include/hip -DENABLE_HIP=On -DENABLE_OPENMP=Off -DENABLE_CUDA=Off -DCMAKE_CXX_COMPILER=/opt/rocm-7.2.1/llvm/bin/amdclang++ -DCMAKE_C_COMPILER=/opt/rocm-7.2.1/llvm/bin/amdclang -DUMPIRE_ENABLE_ASAN=On -DCMAKE_CXX_FLAGS="-fsanitize=address -shared-libsan" -DCMAKE_EXE_LINKER_FLAGS="-Wl,-rpath=/opt/rocm-7.2.1/llvm/lib/clang/22/lib/linux/ -Wl,-rpath=/opt/rocm-7.2.1/lib/asan -fsanitize=address -shared-libsan" -DUMPIRE_ENABLE_SANITIZER_TESTS=ON -DCMAKE_HIP_ARCHITECTURES="gfx942:xnack+" -DGPU_TARGETS="gfx942:xnack+" -DAMDGPU_TARGETS="gfx942:xnack+" -DCAMP_ENABLE_TESTS=Off ../

then run ./bin/sanitizer_tests QuickPool read DEVICE from build dir. I get this output:

 Conducting sanitizer test with QuickPool strategy, read test type, and the DEVICE resource.
data[INDEX] = 100
AddressSanitizer:DEADLYSIGNAL
=================================================================
==1420934==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x155552cf11c0 bp 0x7fffffff76f0 sp 0x7fffffff75e0 T0)
==1420934==The signal is caused by a READ memory access.
==1420934==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
    #0 0x155552cf11c0 in hip::ihipLaunchKernel_validate(ihipModuleSymbol_t*, amd::LaunchParams const&, void**, void**, int, unsigned int) (/opt/rocm-7.2.1/lib/asan/libamdhip64.so.7+0x76f1c0) (BuildId: 4c644848567b30f08eba4633df00f1973f35636b)
    #1 0x155552cf2559 in hip::ihipModuleLaunchKernel(ihipModuleSymbol_t*, amd::LaunchParams&, ihipStream_t*, void**, void**, ihipEvent_t*, ihipEvent_t*, unsigned int, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, unsigned int) (/opt/rocm-7.2.1/lib/asan/libamdhip64.so.7+0x770559) (BuildId: 4c644848567b30f08eba4633df00f1973f35636b)
    #2 0x155552d98e36 in hip::ihipLaunchKernel(void const*, dim3, dim3, void**, unsigned long, ihipStream_t*, ihipEvent_t*, ihipEvent_t*, int) (/opt/rocm-7.2.1/lib/asan/libamdhip64.so.7+0x816e36) (BuildId: 4c644848567b30f08eba4633df00f1973f35636b)
    #3 0x155552d0508e in hip::hipLaunchKernel_common(void const*, dim3, dim3, void**, unsigned long, ihipStream_t*) (/opt/rocm-7.2.1/lib/asan/libamdhip64.so.7+0x78308e) (BuildId: 4c644848567b30f08eba4633df00f1973f35636b)
    #4 0x155552d05da9 in hip::hipLaunchKernel(void const*, dim3, dim3, void**, unsigned long, ihipStream_t*) (/opt/rocm-7.2.1/lib/asan/libamdhip64.so.7+0x783da9) (BuildId: 4c644848567b30f08eba4633df00f1973f35636b)
    #5 0x000000406dd3 in __device_stub__test_read_for_hip(double*, unsigned long) (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x406dd3)
    #6 0x000000407000 in sanitizer_test(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>) (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x407000)
    #7 0x000000407f48 in main (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x407f48)
    #8 0x1555518b7864 in __libc_start_main (/lib64/libc.so.6+0x3a864) (BuildId: 7c09fa3f08c6f94e991ac98739f42a735d92a475)
    #9 0x00000040595d in _start (/g/g0/belcher6/Umpire/build/bin/sanitizer_tests+0x40595d)

==1420934==Register values:
rax = 0xb5e850c7834807f9  rbx = 0x00007fffffff75e0  rcx = 0x16bd0a18f06900ff  rdx = 0x0000115543df7c98  
rdi = 0x000000000040f910  rsi = 0x00001325456e5a80  rbp = 0x00007fffffff76f0  rsp = 0x00007fffffff75e0  
 r8 = 0x0000000000000009   r9 = 0x0000022b287b7240  r10 = 0x0000000000642e00  r11 = 0x0000115543df9200  
r12 = 0x0000000000000000  r13 = 0x0000022aa87bef94  r14 = 0x0000115543df9200  r15 = 0x0000115543df7ca0  
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/opt/rocm-7.2.1/lib/asan/libamdhip64.so.7+0x76f1c0) (BuildId: 4c644848567b30f08eba4633df00f1973f35636b) in hip::ihipLaunchKernel_validate(ihipModuleSymbol_t*, amd::LaunchParams const&, void**, void**, int, unsigned int)
==1420934==ABORTING

…an-poisoning-for-hip

adayton1 · 2026-08-04T17:47:49Z

Any updates on this?

adding asan sanitizer support for hip

a7e72b3

kab163 requested a review from davidbeckingsale November 23, 2022 23:20

github-actions and others added 3 commits November 23, 2022 23:20

Apply style updates

f90a7af

WIP adding a hip kernel sanitizer test

00e3f43

Apply style updates

ac5cd62

kab163 and others added 7 commits December 7, 2022 11:39

adding usage checkers to sanitizer test

522a4a9

Apply style updates

3ac6a7c

more updates to sanitizer test for error checking

a4f007a

merging...

a1c0180

Apply style updates

8b9e194

adding hip define macros

6617aba

Apply style updates

4c17280

getting rid of double pointer

5379807

kab163 and others added 2 commits December 21, 2022 09:31

updating sanitizer test

4e4804f

Apply style updates

4b92502

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

276f15b

…an-poisoning-for-hip

kab163 added 2 commits September 18, 2023 08:06

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

986c5a1

…an-poisoning-for-hip

Merge branch 'develop' of https://github.com/LLNL/Umpire into task/UM…

3858c0f

…-1020-asan-poisoning-for-hip

kab163 added 3 commits January 22, 2024 10:40

adding flag for hip asan

e9bf9f6

updates for rocm test

0232f2c

had to update the test runner to handle new paramter

f154c08

kab163 marked this pull request as ready for review January 24, 2024 16:10

kab163 added 2 commits March 13, 2025 07:52

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

749cd46

…an-poisoning-for-hip

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

9b95a36

…an-poisoning-for-hip

adayton1 added 3 commits June 20, 2025 16:05

Resolve merge conflicts

3aeb7f2

Update to 6.4.1 for hip asan job

c0cc484

Move back to rocm 6.3.1

1e42362

adayton1 reviewed Jun 23, 2025

View reviewed changes

kab163 added 3 commits October 6, 2025 12:57

merge with develop

3073c9a

merging with develop and updates

a520bc9

fixing warnings

2f4b7c5

Merge branch 'develop' of github.com:LLNL/Umpire into task/UM-1020-as…

d152553

…an-poisoning-for-hip

Merge branch 'develop' of github.com:llnl/Umpire into task/UM-1020-as…

e0d2d18

…an-poisoning-for-hip

kab163 requested a review from adayton1 May 12, 2026 19:47

adayton1 reviewed May 12, 2026

View reviewed changes

Comment thread .gitlab/jobs/tioga.yml Outdated

kab163 added 6 commits June 3, 2026 15:07

Merge branch 'develop' of github.com:llnl/Umpire into task/UM-1020-as…

c9b76d4

…an-poisoning-for-hip

pull in radiuss-spack-config updates and update jobs

3f1bde5

Merge branch 'develop' of github.com:llnl/Umpire into task/UM-1020-as…

6aa099c

…an-poisoning-for-hip

testing if the radiuss-spack-configs change works

3088281

Merge branch 'develop' of github.com:llnl/Umpire into task/UM-1020-as…

f5ee2a1

…an-poisoning-for-hip

pulling in radiuss spack config updates

2d56b42

Uh oh!

Conversation

kab163 commented Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Dec 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Dec 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Dec 8, 2022

Uh oh!

kab163 commented Dec 21, 2022

Uh oh!

kab163 commented Mar 2, 2023

Uh oh!

kab163 commented Apr 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Jan 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Jan 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Mar 13, 2025

Uh oh!

kab163 commented Mar 24, 2025

Uh oh!

adayton1 Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

adayton1 Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

adayton1 Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

adayton1 commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Jul 3, 2025

Uh oh!

kab163 commented Mar 12, 2026

Uh oh!

kab163 commented Apr 2, 2026

Uh oh!

kab163 commented Apr 6, 2026

Uh oh!

kab163 commented May 12, 2026

Uh oh!

Uh oh!

adayton1 commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kab163 commented Nov 23, 2022 •

edited

Loading

kab163 commented Dec 6, 2022 •

edited

Loading

kab163 commented Dec 7, 2022 •

edited

Loading

kab163 commented Apr 3, 2023 •

edited

Loading

kab163 commented Jan 22, 2024 •

edited

Loading

kab163 commented Jan 24, 2024 •

edited

Loading

adayton1 commented Jun 24, 2025 •

edited

Loading