feat: write value stats for append data files

[QuakeWang]

Purpose

Rust append writes currently emit empty _VALUE_STATS, so manifest-level stats pruning cannot skip data files written by Rust even when min/max/null-count information is available.

Brief change log

• Accumulate min/max/null-count value stats across batches in DataFileWriter.
• Persist stats in existing DataFileMeta.value_stats and dense value_stats_cols.
• Skip unsupported variable-length value stats and add nanosecond timestamp extraction support.
• Add tests for value stats writing, timestamp(9), skipped variable-length columns, and scan pruning.

Tests

• cargo fmt --all -- --check
• cargo clippy --all-targets --workspace --features fulltext,vortex,mosaic -- -D warnings
• cargo test -p paimon --all-targets --features fulltext,vortex,mosaic

API and Format

Documentation

[JingsongLi]

In Java Side, the performance of the accumulator is poor, and the only truly useful link is the one that extracts stats from Parquet. Perhaps we should optimize its implementation instead of the accumulator.

[QuakeWang]

In Java Side, the performance of the accumulator is poor, and the only truly useful link is the one that extracts stats from Parquet. Perhaps we should optimize its implementation instead of the accumulator.

@JingsongLi Thanks, addressed.

I changed the implementation to avoid per-row stats accumulation. Append value stats are now extracted from Parquet writer metadata on close, aligned with the Java ParquetSimpleStatsExtractor path.

Added/updated tests for append value stats, stats pruning, and timestamp stats.

[JingsongLi]

Just return stats from parquet? It is not good to expose ParquetMetaData in core.

[JingsongLi]

We need to support default stats mode: truncate(16), otherwise the stats are too big.

[QuakeWang]

@JingsongLi Addressed, thanks. Conflicts are resolved, ParquetMetaData is no longer exposed to core, and string value stats now use default truncate(16) with test coverage.

[JingsongLi]

Thanks for the update. I think there are still a couple of Java compatibility gaps here.

1. The manifest stats mode is currently hard-coded to the default truncate(16) behavior. In Java, stats collectors are created from metadata.stats-mode, fields.<name>.stats-mode, and metadata.stats-keep-first-n-columns. With this implementation, a table configured with metadata.stats-mode = none/counts/full or a field-level override would still get truncate(16)-style stats in Rust-written manifests. Could we thread the table stats options into this extraction path and apply the same collector semantics as Java?

2. supports_manifest_value_stats currently skips unsupported min/max types completely. Java does not do that: for unsupported Parquet min/max types such as BINARY/VARBINARY, ParquetSimpleStatsExtractor still returns SimpleColStats(null, null, nullCount), and dense stats storage keeps the column when the null count is known. The current Rust behavior drops those columns from value_stats_cols, so their null counts are lost. Also, Java supports TIME_WITHOUT_TIME_ZONE through IntStatistics, while Rust's conversion code has Time support but the support list excludes DataType::Time. Could we keep null counts for unsupported min/max columns and include Time to match Java?

[leaves12138]

The existing stats-mode, null-count, timestamp, pruning, and MERGE-focused tests pass locally. I also added a minimal compatibility regression with metadata.stats-dense-store=false; it fails because the produced file has value_stats_cols = Some(["id"]) instead of None.

Java threads metadata.stats-dense-store into SimpleStatsConverter. In non-dense mode it stores full-width stats and leaves valueStatsCols null. In dense mode it filters disabled columns, but still leaves valueStatsCols null when every write column is represented. This implementation never reads the option and always emits Some(column_names), so Rust-written manifests are not compatible with the configured Java storage semantics. Please thread the dense-store flag into extraction/serialization and add coverage for false plus the all-columns-present dense case.

[leaves12138]

Parsing the per-column stats modes is not sufficient here because Java also passes metadata.stats-dense-store (default true) into SimpleStatsConverter. Rust currently ignores that option and extract_value_stats always returns dense, filtered stats with Some(column_names). With metadata.stats-dense-store=false, Java requires full-width stats and valueStatsCols = null; with dense storage enabled, Java also uses null when all write columns are present. Please thread this flag through and add compatibility tests. A local one-column counts regression with dense store disabled currently returns Some(["id"]) instead of None.