Skip to content

[Bug] List<T> reader fails with "Inconsistent sample sizes" on nullable outer rows #1401

Description

@0lai0

What

When reading a nullable List<T> column, a null outer row makes the entire read fail with an InvalidInput consistency error, instead of being treated as a missing sample.

The List sample-size validation calls Arrow's ListArray::value_length(i) without an is_null(i) guard. For a null outer row, value_length(i) returns 0; the reader treats this as a length-0 list, which trips the sample_size consistency check. After a non-null row it fails with expected N, got 0; if the null row is row 0, it seeds sample_size to 0 and breaks every subsequent row. FillZero (NullHandling::FillZero) does not help — the failure is at the list-length level, before any value-filling runs.

Reproduce by reading a List<Float64> column with the values [[1, 2], null, [3, 4]]:

InvalidInput("Inconsistent sample sizes: expected 2, got 0")

Why

  • Real-world impact. Nullable List columns with null rows are common in practice, so this is hit by ordinary data, not just edge cases.
  • No coverage. There is currently no test for a null outer row.
  • Pre-existing, but worth fixing. This is not introduced by [Feature][QDP]ParquetReader generic over f32/f64 with Arrow cast #1393 — the previous per-row code behaved the same way. The PR is fine to merge; this is a follow-up.
  • Affects multiple readers. Three readers share the same gap (see How), so the bug surfaces across both batch and streaming paths and the IPC path.

How

The fix is to add an is_null(i) guard to the List sample-size validation so a null row is not counted as a length-0 list (including the row-0 seed). Affected code, all in qdp/qdp-core/src/readers/:

  • ParquetReaderparquet.rs:294–306. Loop over all rows calls value_length(i) at line 295 with no is_null(i) check; a null row 0 locks sample_size to 0.
  • ParquetStreamingReaderparquet.rs. Row-0 seed at line 564 (no null check) and validation loop at lines 568–573, same gap.
  • ArrowIPCReaderarrow_ipc.rs:154. Uses list_array.value(i).len() with no is_null(i) guard; same problem.

The intended semantics of a null row (skip vs. fill per null policy) should be decided first, since it determines how the guard behaves.

Tasks

  • Add the is_null(i) guard to all three readers (ParquetReader, ParquetStreamingReader, ArrowIPCReader).
  • Add a regression test for a nullable List column with a null outer row, covering both null-in-the-middle and null-at-row-0.
  • (Optional) Harden written / self.sample_size.unwrap_or(1) at parquet.rs:490 against Some(0): unwrap_or(1) guards None but not Some(0). Unreachable today (the consistency check errors first), but a cheap guard keeps the division safe against a future sample_size == 0 path.

Follow-up to #1393.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions