feat(encryption) [15/N] write encrypted parquet data files by aarushigupta132 · Pull Request #2701 · apache/iceberg-rust

aarushigupta132 · 2026-06-23T23:34:56Z

Which issue does this PR close?

Working towards #2034.

What changes are included in this PR?

Adds support for writing Iceberg data files with Parquet Modular Encryption.

When a table is configured for encryption, ParquetWriterBuilder::from_table_properties derives the data-encryption key size from the table properties. ParquetWriter then mints a fresh per-file DEK, writes the file encrypted, and records the encoded key metadata on the resulting DataFile so readers can decrypt.

Are these changes tested?

Existing ParquetWriter tests pass; encryption round-trip coverage lands later in the series.

xanderbailey

Thanks carrying the flame on the encryption work. Have made a couple of comments to get us started.

aarushigupta132 · 2026-07-15T23:51:09Z

    type R = ParquetWriter;

    async fn build(&self, output_file: OutputFile) -> Result<Self::R> {
+        let key_metadata = self


Confirmed that in rolling_writer as well, we build this again with new dek for each new parquet file

aarushigupta132 · 2026-07-15T23:53:02Z

    /// When writing into an existing Iceberg table, prefer
    /// [`Self::from_table_properties`], which derives `WriterProperties` from
    /// the table's `write.parquet.*` properties.


curious to know if we should deprecate new and new_with_match_mode in favor of from_table_properties. Or update the comment to prefer the latter in all cases?

I would add a comment in the doc here that encrypted writing is not supported with this constructor. I would honestly be in favour of removing this constructor since it's not used in this crate and nor should it be. I don't think we should do that in this PR but it would be a good follow-up IMO.

xanderbailey

Few more comments and a test for the writer would be great. Maybe also something that shows datafusion writer works but that can be a follow up

xanderbailey · 2026-07-16T13:55:18Z

    /// When writing into an existing Iceberg table, prefer
    /// [`Self::from_table_properties`], which derives `WriterProperties` from
    /// the table's `write.parquet.*` properties.


I would add a comment in the doc here that encrypted writing is not supported with this constructor. I would honestly be in favour of removing this constructor since it's not used in this crate and nor should it be. I don't think we should do that in this PR but it would be a good follow-up IMO.

mbutrovich

First pass, thanks @aarushigupta132!

mbutrovich · 2026-07-28T16:39:12Z

+        let data_encryption_key_size = table_props
+            .encryption_key_id
+            .is_some()
+            .then(|| AesKeySize::from_key_length(table_props.encryption_data_key_length))
+            .transpose()?;


This derives whether to encrypt purely from table_props.encryption_key_id.is_some(), independent of whether the Table this came from actually has an EncryptionManager.

Traced this through: today the two can't disagree at the point a Table is built. EncryptionManager::from_table_metadata (crates/iceberg/src/encryption/manager.rs:122-136) returns an error if encryption.key-id is set on the table's metadata but no KeyManagementClient was supplied to TableBuilder. So a freshly-built Table always has encryption_manager() in sync with table.metadata().table_properties().encryption_key_id.

But Table::with_metadata() (crates/iceberg/src/table.rs:212-215) swaps in new metadata without recomputing encryption_manager (that field is only set once, in TableBuilder::build()). If a commit turns on encryption.key-id for the first time and the caller keeps using the same in-memory Table handle afterward (via with_metadata) rather than reloading from the catalog, ParquetWriterBuilder::from_table_properties would see the new encryption_key_id and start minting DEKs and encrypting data files, while transaction/snapshot.rs's manifest writer (which branches on self.table.encryption_manager(), e.g. crates/iceberg/src/transaction/snapshot.rs:257) would still take the unencrypted path off the stale cached value. That would leave a freshly generated, unwrapped per-file DEK sitting in a plaintext manifest, the thing the two-layer envelope design is built to avoid.

This is in table.rs, outside this PR's diff, so it isn't this PR's bug to fix, and it's possible normal usage always reloads the Table from the catalog after a property-changing commit and never hits this. Worth either confirming that path is genuinely unreachable, or filing an issue to make with_metadata keep encryption_manager in sync (or recompute it lazily from current metadata + a stored kms_client, rather than caching once at construction).

Yeah I thought about this earlier this week. Fortunately in the case of encryption.key-id the spec notes that it should not be changed or removed for the lifetime of the table. So it's existence is immutable per the spec. https://iceberg.apache.org/docs/nightly/encryption/#catalog-security-requirements .

I need to think a little more deeply about the case where you have encryption keys added in memory in the manager and then you refresh the table before the commit which risks loosing those keys but I think we're safe from that today.

That being said I'll have a slightly deeper think about how we might improve our robustness here a little.

mbutrovich · 2026-07-28T16:39:41Z

    /// parquet-rs defaults.
-    pub fn from_table_properties(table_props: &TableProperties, schema: SchemaRef) -> Self {
+    ///
+    /// When `encryption.key-id` is set, records the DEK length.


Doc comment above this says "When encryption.key-id is set, records the DEK length." Accurate, but might be worth a second line noting that the DEK itself is generated per-file in build() (line ~138), not here: from_table_properties only records the size to use later. Minor, only raising it because it took a moment to trace where the actual key material gets minted while checking comment above.

xanderbailey · 2026-07-28T18:08:53Z

+            table.metadata().current_schema(),
+        )?;
+
+        // The planner encrypted the file.


Not sure what this means

aarushigupta132 changed the title ~~Feat/encrypted parquet data files~~ feat(encryption): write encrypted parquet data files Jun 23, 2026

aarushigupta132 marked this pull request as ready for review June 23, 2026 23:45

xanderbailey reviewed Jun 25, 2026

View reviewed changes

Comment thread crates/iceberg/src/writer/file_writer/parquet_writer.rs Outdated

Comment thread crates/iceberg/src/writer/file_writer/parquet_writer.rs Outdated

aarushigupta132 changed the title ~~feat(encryption): write encrypted parquet data files~~ feat(encryption) [15/N] write encrypted parquet data files Jul 15, 2026

aarushigupta132 force-pushed the feat/encrypted-parquet-data-files branch 2 times, most recently from 58df8f8 to 0783948 Compare July 15, 2026 16:55

aarushigupta132 closed this Jul 15, 2026

aarushigupta132 force-pushed the feat/encrypted-parquet-data-files branch from 0783948 to b80ebbf Compare July 15, 2026 17:38

aarushigupta132 reopened this Jul 15, 2026

aarushigupta132 force-pushed the feat/encrypted-parquet-data-files branch 3 times, most recently from cc5fca7 to 74c54db Compare July 15, 2026 23:46

aarushigupta132 commented Jul 15, 2026

View reviewed changes

xanderbailey reviewed Jul 16, 2026

View reviewed changes

aarushigupta132 force-pushed the feat/encrypted-parquet-data-files branch 2 times, most recently from 5a0ea6f to 9832f50 Compare July 19, 2026 21:53

feat(encryption): write encrypted parquet data files

2ba0884

aarushigupta132 force-pushed the feat/encrypted-parquet-data-files branch from 9832f50 to 2ba0884 Compare July 19, 2026 21:59

aarushigupta132 added 2 commits July 20, 2026 11:22

Merge branch 'main' into feat/encrypted-parquet-data-files

492f453

Merge branch 'main' into feat/encrypted-parquet-data-files

cb302f2

xanderbailey mentioned this pull request Jul 27, 2026

Update purge encryption test to use PME #2917

Closed

Merge branch 'main' into feat/encrypted-parquet-data-files

b67a0bb

mbutrovich reviewed Jul 28, 2026

View reviewed changes

mbutrovich mentioned this pull request Jul 28, 2026

[EPIC] Deletion vector read support (deletion-vector-v1 Puffin blobs) #2792

Open

3 tasks

xanderbailey reviewed Jul 28, 2026

View reviewed changes

Uh oh!

Conversation

aarushigupta132 commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

xanderbailey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xanderbailey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mbutrovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aarushigupta132 commented Jun 23, 2026 •

edited

Loading