Memory corruption when a session requests a larger `hw_buff_sz` than the first session in the process

[asonje]

Symptom

A process that compresses once at the default hw_buff_sz (64 KiB) and later sets up a session with a larger, in-spec hw_buff_sz (up to QZ_HW_BUFF_MAX_SZ = 512 KiB) dies with SIGSEGV after main returns, inside the library's own ELF destructor: exitFunc → cleanUpInstMem → qzFree → free, on a pointer that is no longer a pointer. The faulting address is literally the caller's payload: the reproducer fills its input with a recognizable i % 256 pattern, and the pointer being freed is a run of bytes from that pattern.

Both qzSetupSessionDeflateExt() and both qzCompress() calls return QZ_OK first. Nothing reports an error at the point of damage.

Reproducer attached: repro_hw_buff_latch.c.

$ gcc -O0 -g -Wall -Wextra -o repro_hw_buff_latch repro_hw_buff_latch.c -lqatzip -lz
$ ./repro_hw_buff_latch ; echo "exit=$?"
SMALL_FIRST=1 (expect SIGSEGV at exit)

session A: latches the process-wide buffers
  hw_buff_sz=65536   payload=1024    -> 330 bytes, rc=QZ_OK
  session A torn down and closed

session B: requests a LARGER hw_buff_sz than the latch
  hw_buff_sz=524288  payload=524288  -> 2390 bytes, rc=QZ_OK
  session B torn down and closed

returning from main; the library destructor runs after this
Segmentation fault (core dumped)
exit=139

$ SMALL_FIRST=0 ./repro_hw_buff_latch ; echo "exit=$?"     # control: large first
...
exit=0

Judge the reproducer by exit status, not by whether it prints done — the heap is already corrupt while both compressions report success.

Root cause

Everything below — line numbers, the build under test, and the SIGSEGV

1. The pinned instance buffers are process-wide and sized from the first session to compress. getInstMem() takes the size straight from the session params and allocates each source buffer with it:

/* src/qatzip.c:964 */
/*  WARN: this will mean the first sess will setup down the inst
*   buffer size, if it's very small, then we can't make then
*   any larger in the whole process. please refer test mode 17
*/
src_sz = params->hw_buff_sz;

/* src/qatzip.c:1062 */
g_process.qz_inst[i].src_buffers[j]->pBuffers->pData = (Cpa8U *)
        qzMalloc(src_sz, QZ_AUTO_SELECT_NUMA_NODE, PINNED_MEM);

and then latches at src/qatzip.c:1111:

g_process.qz_inst[i].mem_setup = 1;

src_count / dest_count are latched from the same value too (src/qatzip.c:1017-1021: NUM_BUFF_8K when hw_buff_sz <= 8K, else NUM_BUFF) — so a fix has two latched quantities to consider, not one.

2. Nothing releases those buffers short of process exit. qzClose() (src/qatzip.c:2777) is a no-op. qzTeardownSession() (:2702) frees only
per-session state and never touches g_process.qz_inst[i]. cleanUpInstMem() (:849) is the only code that clears mem_setup after
allocation (:932; the other assignment at :786 is initial zeroing during instance discovery in qzInit), and outside the two allocation-failure macros it is reachable only from exitFunc() (:514), the destructor. qzUpdateCpaSession() reallocates cpaSess alone. So on any successful run the buffers survive every teardown at the size the first compress chose.

3. A later, larger hw_buff_sz is accepted anyway. QZ_HW_BUFF_MAX_SZ is 512*1024 (include/qatzip.h:584) and qzCheckParams* validates the request (src/qatzip_utils.c:453, :506). The value lands in the session params, and doCompressIn() chunks against the session's value with no reference to what was allocated:

/* src/qatzip.c:1523*/
hw_buff_sz  = qz_sess->sess_params.hw_buff_sz;   /* 512 KiB */
...
src_send_sz = (remaining < hw_buff_sz) ? remaining : hw_buff_sz;

4. The copy into the undersized buffer is unbounded. compBufferSetup() passes two source lengths where QZ_MEMCPY expects a destination capacity and a source length:

/* src/qatzip_utils.c:1134*/
QZ_MEMCPY(g_process.qz_inst[i].src_buffers[j]->pBuffers->pData,
          src_ptr,
          src_send_sz,      /* <- a source length, in the dest_sz position */
          src_remaining);

/* include/qatzip.h:377*/
#define QZ_MEMCPY(dest, src, dest_sz, src_sz) \
        memcpy((void *)(dest), (void *) (src), (size_t)MIN(dest_sz, src_sz))

MIN(src_send_sz, src_remaining) is src_send_sz whenever a full block is available, so the MIN bounds nothing against the real 64 KiB capacity and the copy overruns by up to 448 KiB. There is no field anywhere in QzInstance_T (src/qatzip_internal.h:173-196) recording the size the buffers were allocated with, so no existing code could perform that check.

What the overrun lands on is not merely allocator metadata but the neighbouring buffer descriptors: with src_count = 32, the copy past src_buffers[0]'s data buffer overwrites src_buffers[1]'s pPrivateMetaData and pBuffers pointers with payload bytes. cleanUpInstMem() then walks those descriptors at exit and free()s pointers assembled from caller data — which is the crash in the symptom above.

Note this is USDM memory (qzMalloc → qaeMemAllocNUMA), not the glibc heap, so ASAN does not help: there are no redzones around these buffers, an ASAN build fails at the same destructor free() with no extra detail, and the overflowing write itself is never observed.

Why a caller cannot work around this

• The value is documented, in range, and validated — the library accepts it and then overflows.
• No API releases the instance buffers, so a caller cannot reset the latch.
• No API exposes the latched size, so a caller cannot even query what is safe before setting up a session.
• The trigger threshold is one 1 KB compress at the default size. In a shared runtime (our case: a JVM where unrelated components each build their own session) which session compresses first is not under any one caller's control.

The in-tree WARN at qatzip.c:964 shows the constraint is already known - the report is that it is documented but unenforced, and that violating it corrupts memory rather than failing.

One caveat on silently clamping

One possible fix that looks locally attractive and has a consequence worth stating up front, because we hit it: making the oversized request silently fall back to the latched size.

That is appealing since it needs no API change and no reallocation. The problem is that hw_buff_sz is not only an allocation size — it is the compression blocking factor. The hardware terminates a complete deflate/zlib stream at every hw_buff_sz boundary, so quietly reducing the effective value changes the format of the output, not just its performance:

A caller that asked for 512 KiB and got 64 KiB blocking receives a sequence of concatenated streams rather than the single stream it expected and could have consequences. If clamping is the direction, the effective value would need to be observable and documented so a caller can tell that its requested framing was not honored.

[asonje]

Reproducer — save as repro_hw_buff_latch.c:

/*
 * Reproducer: heap overflow when a session requests a larger hw_buff_sz than
 * the first session that compressed in the process.
 *
 * getInstMem() sizes the process-wide pinned instance buffers from the FIRST
 * session's hw_buff_sz and latches them (mem_setup = 1). Nothing releases them
 * short of process exit: qzClose() is a no-op, qzTeardownSession() only frees
 * per-session state, and cleanUpInstMem() -- the sole place clearing mem_setup
 * -- runs only from allocation-failure paths or the library's ELF destructor.
 *
 * A later session may still request a larger hw_buff_sz. It is accepted:
 * QZ_HW_BUFF_MAX_SZ is 512K and qzCheckParamsDeflateExt validates the value.
 * compBufferSetup() then copies up to that much into the smaller buffers,
 * because QZ_MEMCPY's dest-size argument receives a source length, so its
 * MIN() bounds nothing.
 *
 * The corruption is silent. Both compressions report success; the process dies
 * later in the destructor freeing the overwritten chunks, so JUDGE THIS
 * REPRODUCER BY ITS EXIT STATUS, not by whether it prints "done".
 *
 * Build and run:
 *   gcc -O0 -g -o repro_hw_buff_latch repro_hw_buff_latch.c -lqatzip -lz
 *   ./repro_hw_buff_latch ; echo "exit=$?"
 *
 * exit status 139 (SIGSEGV), faulting in
 * free() called from qzFree <- cleanUpInstMem <- exitFunc, the library's
 * __attribute__((destructor)).
 *
 * The pointer being freed is the payload itself -- a run of bytes from the i%256
 * fill below. The copy past src_buffers[0]'s data buffer lands on the adjacent
 * CpaBufferList structs, overwriting src_buffers[1]'s pPrivateMetaData and
 * pBuffers pointers, and cleanUpInstMem() then free()s them.
 *
 * ASAN does NOT localize this to the memcpy. These buffers come from
 * qaeMemAllocNUMA (USDM driver memory), not the glibc heap, so there are no
 * redzones around them and the sanitizer never observes the overflowing write;
 * an ASAN build fails at the same destructor free() with no extra detail.
 *
 * Pass SMALL_FIRST=0 to run the 512K session first, which is the control: the
 * buffers are then latched large, the 64K session fits, and the process exits 0.
 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <qatzip.h>

#define SMALL_HW_BUFF_SZ (64 * 1024)  /* QZ_HW_BUFF_SZ, the default */
#define LARGE_HW_BUFF_SZ (512 * 1024) /* QZ_HW_BUFF_MAX_SZ, the documented max */

#define SMALL_PAYLOAD (1024)          /* one small compress is enough to latch */
#define LARGE_PAYLOAD (512 * 1024)

/* Abort loudly rather than pass quietly: a reproducer that silently took the
 * software path proves nothing about hardware buffer handling. */
#define MUST(cond, ...)                       \
    do {                                      \
        if (!(cond)) {                        \
            fprintf(stderr, "FATAL: ");       \
            fprintf(stderr, __VA_ARGS__);     \
            fprintf(stderr, "\n");            \
            exit(2);                          \
        }                                     \
    } while (0)

/* Fill with a recognizable pattern, so that when these bytes are later freed as
 * a pointer the fault address is visibly the payload rather than random junk. */
static void fill_pattern(unsigned char *buf, unsigned int len)
{
    for (unsigned int i = 0; i < len; i++) {
        buf[i] = (unsigned char)(i % 256);
    }
}

/* One compress on its own session at the given hw_buff_sz, then tear the
 * session down. Teardown is deliberate: it shows that releasing every session
 * still does not release the latched instance buffers. */
static void compress_at(unsigned int hw_buff_sz, unsigned int payload_sz)
{
    QzSession_T sess;
    QzSessionParamsDeflateExt_T params;
    unsigned char *src = NULL;
    unsigned char *dst = NULL;
    unsigned int src_len = payload_sz;
    unsigned int dst_len = 0;
    int rc;

    memset(&sess, 0, sizeof(sess));

    /* sw_backup = 0 so a missing or busy device fails loudly here instead of
     * quietly running in software and reproducing nothing. */
    rc = qzInit(&sess, 0);
    MUST(rc == QZ_OK || rc == QZ_DUPLICATE,
         "qzInit returned %d -- no QAT hardware available, so this reproducer "
         "cannot exercise the pinned-buffer path. Check the device is up "
         "(qat_service status) and that this user may access it.", rc);

    memset(&params, 0, sizeof(params));
    rc = qzGetDefaultsDeflateExt(&params);
    MUST(rc == QZ_OK, "qzGetDefaultsDeflateExt returned %d", rc);

    params.deflate_params.common_params.hw_buff_sz = hw_buff_sz;
    params.deflate_params.common_params.strm_buff_sz = hw_buff_sz;
    params.deflate_params.common_params.sw_backup = 0;

    rc = qzSetupSessionDeflateExt(&sess, &params);
    MUST(rc == QZ_OK, "qzSetupSessionDeflateExt(hw_buff_sz=%u) returned %d "
         "-- the value is within QZ_HW_BUFF_MIN_SZ..QZ_HW_BUFF_MAX_SZ and a "
         "power of 2K, so it should be accepted", hw_buff_sz, rc);

    src = malloc(payload_sz);
    MUST(src != NULL, "malloc(%u) failed", payload_sz);
    fill_pattern(src, payload_sz);

    dst_len = qzMaxCompressedLength(payload_sz, &sess);
    MUST(dst_len > 0, "qzMaxCompressedLength(%u) returned 0", payload_sz);
    dst = malloc(dst_len);
    MUST(dst != NULL, "malloc(%u) failed", dst_len);

    printf("  hw_buff_sz=%-7u payload=%-7u ", hw_buff_sz, payload_sz);
    fflush(stdout);

    rc = qzCompress(&sess, src, &src_len, dst, &dst_len, 1);
    MUST(rc == QZ_OK, "qzCompress returned %d", rc);
    MUST(src_len == payload_sz, "qzCompress consumed %u of %u bytes",
         src_len, payload_sz);

    /* Note this reports success even in the overflowing case. */
    printf("-> %u bytes, rc=QZ_OK\n", dst_len);

    free(src);
    free(dst);

    rc = qzTeardownSession(&sess);
    MUST(rc == QZ_OK, "qzTeardownSession returned %d", rc);
    rc = qzClose(&sess);
    MUST(rc == QZ_OK, "qzClose returned %d", rc);
}

int main(void)
{
    const char *env = getenv("SMALL_FIRST");
    int small_first = (env == NULL) || (strcmp(env, "0") != 0);

    /* Line-buffer stdout. glibc flushes stdio (_IO_cleanup) only AFTER running
     * the ELF destructors (_dl_fini), so with the default block buffering the
     * crash in QATzip's destructor discards everything printed after the last
     * newline -- making it look as though the process died inside qzCompress
     * rather than at exit. Without this, the output misdiagnoses the bug. */
    setvbuf(stdout, NULL, _IOLBF, 0);

    printf("SMALL_FIRST=%d (%s)\n\n", small_first,
           small_first ? "expect SIGSEGV at exit" : "control, expect clean exit");

    if (small_first) {
        /* Latch the instance buffers at 64K. One small compress suffices. */
        printf("session A: latches the process-wide buffers\n");
        compress_at(SMALL_HW_BUFF_SZ, SMALL_PAYLOAD);
        printf("  session A torn down and closed\n\n");

        /* In spec, accepted, and overflows the 64K buffers by ~448K. */
        printf("session B: requests a LARGER hw_buff_sz than the latch\n");
        compress_at(LARGE_HW_BUFF_SZ, LARGE_PAYLOAD);
        printf("  session B torn down and closed\n\n");
    } else {
        printf("session A: latches the buffers at the larger size\n");
        compress_at(LARGE_HW_BUFF_SZ, LARGE_PAYLOAD);
        printf("  session A torn down and closed\n\n");

        printf("session B: requests a SMALLER hw_buff_sz (fits, wasteful only)\n");
        compress_at(SMALL_HW_BUFF_SZ, SMALL_PAYLOAD);
        printf("  session B torn down and closed\n\n");
    }

    /* Reaching this line means nothing. The heap is already corrupt in the
     * SMALL_FIRST case; QATzip's destructor now runs cleanUpInstMem() -> qzFree()
     * -> free() over the overwritten chunks, and that is where it dies. Check
     * the exit status. */
    printf("returning from main; the library destructor runs after this\n");
    return 0;
}