You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When reading a nullable List<T> column, a nullouter row makes the entire read fail with an InvalidInput consistency error, instead of being treated as a missing sample.
The List sample-size validation calls Arrow's ListArray::value_length(i) without an is_null(i) guard. For a null outer row, value_length(i) returns 0; the reader treats this as a length-0 list, which trips the sample_size consistency check. After a non-null row it fails with expected N, got 0; if the null row is row 0, it seeds sample_size to 0 and breaks every subsequent row. FillZero (NullHandling::FillZero) does not help — the failure is at the list-length level, before any value-filling runs.
Reproduce by reading a List<Float64> column with the values [[1, 2], null, [3, 4]]:
Affects multiple readers. Three readers share the same gap (see How), so the bug surfaces across both batch and streaming paths and the IPC path.
How
The fix is to add an is_null(i) guard to the List sample-size validation so a null row is not counted as a length-0 list (including the row-0 seed). Affected code, all in qdp/qdp-core/src/readers/:
ParquetReader — parquet.rs:294–306. Loop over all rows calls value_length(i) at line 295 with no is_null(i) check; a null row 0 locks sample_size to 0.
ParquetStreamingReader — parquet.rs. Row-0 seed at line 564 (no null check) and validation loop at lines 568–573, same gap.
ArrowIPCReader — arrow_ipc.rs:154. Uses list_array.value(i).len() with no is_null(i) guard; same problem.
The intended semantics of a null row (skip vs. fill per null policy) should be decided first, since it determines how the guard behaves.
Tasks
Add the is_null(i) guard to all three readers (ParquetReader, ParquetStreamingReader, ArrowIPCReader).
Add a regression test for a nullable List column with a null outer row, covering both null-in-the-middle and null-at-row-0.
(Optional) Harden written / self.sample_size.unwrap_or(1) at parquet.rs:490 against Some(0): unwrap_or(1) guards None but not Some(0). Unreachable today (the consistency check errors first), but a cheap guard keeps the division safe against a future sample_size == 0 path.
What
When reading a nullable
List<T>column, anullouter row makes the entire read fail with anInvalidInputconsistency error, instead of being treated as a missing sample.The List sample-size validation calls Arrow's
ListArray::value_length(i)without anis_null(i)guard. For anullouter row,value_length(i)returns0; the reader treats this as a length-0 list, which trips thesample_sizeconsistency check. After a non-null row it fails withexpected N, got 0; if thenullrow is row0, it seedssample_sizeto0and breaks every subsequent row.FillZero(NullHandling::FillZero) does not help — the failure is at the list-length level, before any value-filling runs.Reproduce by reading a
List<Float64>column with the values[[1, 2], null, [3, 4]]:Why
Listcolumns with null rows are common in practice, so this is hit by ordinary data, not just edge cases.How
The fix is to add an
is_null(i)guard to the List sample-size validation so anullrow is not counted as a length-0 list (including the row-0 seed). Affected code, all inqdp/qdp-core/src/readers/:ParquetReader—parquet.rs:294–306. Loop over all rows callsvalue_length(i)at line 295 with nois_null(i)check; a null row 0 lockssample_sizeto0.ParquetStreamingReader—parquet.rs. Row-0 seed at line 564 (no null check) and validation loop at lines 568–573, same gap.ArrowIPCReader—arrow_ipc.rs:154. Useslist_array.value(i).len()with nois_null(i)guard; same problem.The intended semantics of a null row (skip vs. fill per null policy) should be decided first, since it determines how the guard behaves.
Tasks
is_null(i)guard to all three readers (ParquetReader,ParquetStreamingReader,ArrowIPCReader).Listcolumn with anullouter row, covering both null-in-the-middle and null-at-row-0.written / self.sample_size.unwrap_or(1)atparquet.rs:490againstSome(0):unwrap_or(1)guardsNonebut notSome(0). Unreachable today (the consistency check errors first), but a cheap guard keeps the division safe against a futuresample_size == 0path.Follow-up to #1393.