[Protocol RFC] File data type by dejankrak-db · Pull Request #7148 · delta-io/delta

dejankrak-db · 2026-07-06T23:44:32Z

Which Delta project/connector is this regarding?

Description

This adds a protocol RFC for a new file data type. A file value stores a reference to a range of bytes that is located either inline in the value or in an external file at an absolute URI. It targets file-inventory, manifest, and unstructured-data use cases (e.g. images/audio in object storage) that are increasingly common with AI/ML workloads.

The type follows the Parquet FILE logical type (introduced in apache/parquet-format#585, now merged): the field set (uri, offset, size, content_type, checksum, inline) and the physical encoding are defined by the Parquet spec, so a Delta file column round-trips through Parquet without loss. Delta adds a single restriction on top: a uri must be absolute. The RFC does not restate the Parquet spec; it links to it and focuses on the Delta schema representation and feature interactions.

Note: the Parquet FILE type is inline-or-external only. An earlier revision also allowed a self-reference (a byte range within the containing data file); that form was removed from the specification in apache/parquet-format#603, so it is not a Delta-specific restriction. This RFC has been updated accordingly.

Design doc: [DESIGN] Introducing a new "File" logical type to Parquet

The RFC introduces:

a new fileType table feature (Reader Version 3 / Writer Version 7, readers and writers);
a new file primitive type name in the schema serialization format;
the mapping to the Parquet FILE logical type (a group with the six optional fields), by reference to the Parquet specification; in a Delta table a value resolves either inline or from an external absolute uri, and offset/size designate a byte range only within the referenced file;
per-leaf statistics — nullCount on each leaf field and minValues/maxValues on the comparable, skipping-useful leaves (uri, offset, size, content_type), excluding inline and checksum — enabling data skipping on file-inventory tables filtered by URI prefix;
time travel and Change Data Feed semantics — inline bytes live in the data file and time-travel with the table, while externally-referenced (uri) bytes live outside the transaction log and carry no guarantee;
compatibility notes with partitioning, clustering (on comparable leaf fields), generated columns, CHECK constraints, default values (a file column must default to NULL), Change Data Feed, Iceberg Compatibility V1/V2 (a file column is not permitted), Type Widening (no change to/from file), and map keys (file cannot be a map key; array element / map value are allowed).

This is a documentation-only change adding protocol_rfcs/file-type.md and listing it in protocol_rfcs/README.md.

see #7147

Add a protocol RFC for a new `file` data type that stores a reference to a range of bytes located inline, elsewhere in the same data file, or in an external file. Aligned with the Parquet FILE logical type proposed in apache/parquet-format#585. Introduces the `fileType` table feature (Reader v3 / Writer v7), the `file` primitive type name, Parquet physical encoding, per-leaf statistics, and compatibility notes with other Delta features. See delta-io#7147 Co-authored-by: Isaac

ilicmarkodb · 2026-07-07T14:17:04Z

+Partition Columns | **Supported:** A `file` column is allowed to be a non-partitioned column of a partitioned table. <br/> **Unsupported:** A `file` value is a group and cannot be serialized to a partition-value string, so a `file` column cannot be a partition column.
+Clustered Tables | **Supported:** A `file` column is allowed to be a non-clustering column of a clustered table. <br/> **Unsupported:** A `file` value is a group and is not a comparable data type as a whole, so a `file` column cannot be a clustering column.
+Delta Column Statistics | **Supported:** A `file` column supports the `nullCount` statistic, and `minValues` / `maxValues` on its comparable leaf fields. See [Statistics for File Columns](#statistics-for-file-columns). <br/> **Unsupported:** The `file` column as a whole is not a comparable data type, and the `inline` field does not support `minValues` / `maxValues`.
+Generated Columns | **Supported:** A `file` column is allowed to be used as a source in a generated column expression, as long as the `file` type is not the result type of the generated column expression. <br/> **Unsupported:** The `file` data type is not allowed to be the result type of a generated column expression.


why wouldn't we have FILE generated columns?

Fair question — that restriction was carried over from the Variant RFC. I've reframed this as an open question for discussion on the issue.

…-columns question - Expand the self-reference bullet in the intro with how the bytes are located within the same data file, linking to Byte Resolution. - Reframe the generated-column result-type clause as an open question for discussion rather than a hard restriction inherited from the variant RFC. Co-authored-by: Isaac

…on-goals - Add a "Time Travel and Change Data Feed" section clarifying that Delta time- travels and CDFs the reference, not the referenced bytes, which live outside the transaction log and may be overwritten or deleted independently. - Add a "Non-Goals" section scoping out lifecycle/GC of referenced bytes (including any Delta-managed notion and its VACUUM interaction) and access brokering, keeping the type reference-only and aligned with Parquet/Iceberg. Co-authored-by: Isaac

Standardize the representation of a file column's storage mode (MANAGED/EXTERNAL/UNKNOWN) in the __FILE_TYPE_MODE schema-metadata key, a path-keyed map on the nearest ancestor StructField, mirroring the __COLLATIONS representation and DBR's FileType mode (databricks-eng/runtime#213587). This lets the qualifier round-trip through the Delta log without placing it inside the file value (which must round-trip through the fixed Parquet FILE field set). Only the representation is standardized; MANAGED vs EXTERNAL carries no Delta behavior yet, and lifecycle/GC remains a non-goal. Co-authored-by: Isaac

…a-io#585) Align with the latest apache/parquet-format#585: - size must be set whenever offset is set; a self-reference (no path) must set offset, and therefore size. Drop the now-invalid [offset, EOF) and [0, size) self-reference modes and the external [offset, EOF) mode from the resolution table; add explicit invalid rows. - Define "set" (present, non-null, non-empty for strings) and allow sparse group definitions (a group need only define the fields it uses); add an inline-only example group. - Fields matched case-sensitively by name; field IDs "if they exist". - Readers should ignore unknown checksum algorithms. Follows the consistent prose intent of PR delta-io#585; note its resolution table still lists an [offset, EOF) row that contradicts its own validation section (offset requires size) -- to be raised on the Parquet PR. Co-authored-by: Isaac

…@ 90147dc) Align with the latest apache/parquet-format#585 (commit 90147dc): - path is now a URI-reference per RFC 3986 (absolute or relative), not an opaque location string. Resolves the path/uri debate: name stays `path`, semantics become URI. - checksum form changes from <algorithm>:base64(<digest>) to <algorithm>:<digest> with per-algorithm encoding (lowercase hex for MD5/CRC32/CRC32C/SHA-256, opaque for ETAG); add RFC references and an Encoding column. - content_type: MIME per RFC 2046, defaults to application/octet-stream when unset. - offset must not be negative. - Note that implementations are not expected to treat empty strings as null. Co-authored-by: Isaac

…@ 92228a1) Align with the latest apache/parquet-format#585 (commit 92228a1): - Update checksum RFC citations: MD5 -> RFC 1321, CRC32 -> RFC 2083, CRC32C -> RFC 3385 (SHA-256 unchanged at RFC 6234). - Readers may return a null file value for a row whose reference is invalid. Co-authored-by: Isaac

…o#585 @ 4223744) Align with apache/parquet-format#585 (commit 4223744): the FILE locator field was renamed from `path` to `uri`. Rename the field throughout the field table, resolution table, Parquet examples, writer/reader requirements, statistics, and prose. Generic uses of "path" (field path, absolute path) are unchanged, as is the DBR-specific __FILE_TYPE_MODE storage-mode key. Co-authored-by: Isaac

…@ f241be8) Align with apache/parquet-format#585 (commit f241be8): add the modular-encryption restriction -- data files containing self-references must not use Parquet modular encryption, since self-referenced byte ranges are not encryption modules and cannot be encrypted/authenticated independently. Co-authored-by: Isaac

…rged) apache/parquet-format#585 is now merged. The FILE spec content is unchanged from the last sync (commit f241be8), so no semantic updates are needed. Update references to point at the stable LogicalTypes.md#file spec and drop the "proposed" framing now that the type is part of the Parquet format. Co-authored-by: Isaac

brkyvz

Very excited about this! Left some comments

brkyvz · 2026-08-04T19:57:32Z

+# File Data Type
+**Associated Github issue for discussions: https://github.com/delta-io/delta/issues/7147**
+
+This protocol change adds support for the `file` data type.


Can we call it FILE EXTERNAL please?

We're keeping the plain file type here, to mirror the Parquet FILE type exactly. FILE EXTERNAL implies an additional qualifier which may suggest that there is another flavor (MANAGED), which isn't exposed in the OSS protocol for the time being.

This doesn't prevent adding support for managed later: a future managed capability could be added as a separate/additive type with an optional mode marker — an unqualified file is external, so nothing would need renaming or reinterpreting. Given that, a single unqualified file reads cleaner than FILE EXTERNAL whose counterpart doesn't exist in OSS.

Please let me know if you still think we should call it FILE EXTERNAL, and we can revisit/iterate. Thanks!

…-file details Per review (brkyvz) and internal decision: - Remove all DBR-specific managed-file details (Storage Mode section, __FILE_TYPE_MODE column-metadata key, FILE MANAGED example, managed clause in Non-Goals). Keep the plain Parquet `file` type; do not expose a managed/external qualifier. A future managed capability, if any, can be added later as a separate additive feature without renaming or breaking this type. - Stop restating the Parquet FILE spec (fields, checksum encoding, byte resolution). Link to the Parquet FILE specification and keep the RFC focused on the Delta schema representation and feature interactions. - Rephrase the partition/clustering compatibility rows to lead with the restriction; allow clustering on a file column's comparable leaf fields (e.g. size, content_type), consistent with per-leaf statistics. - Clarify time travel/CDF: inline and self-reference bytes live in the data file and time-travel with the table; only external (uri) bytes live outside the log. Co-authored-by: Isaac

dtenedor

Thanks for working on this, the new specification will help Delta Lake users!

…uri, stats/leaf addressing Address dtenedor's review comments: - Disallow self-references in Delta tables: they break silently when Delta rewrites data files (OPTIMIZE, MERGE/UPDATE/DELETE, REORG PURGE, CDF), which relocate rows without relocating referenced byte ranges. v1 supports inline + external only. Inline is kept (reader-must, writer-may) since it travels with the row and the DBR layout already includes it. - Require uri to be absolute; relative URIs have no defined resolution base under SHALLOW/DEEP CLONE and OPTIMIZE. - Define leaf-addressing / stats key spelling: <file column>.<field>, since the FILE group's field names are fixed literals not subject to column mapping. Used consistently by statistics, clustering columns, and CHECK constraints. - Per-leaf nullCount (with example); min/max on uri/offset/size/content_type only (exclude checksum and inline); each indexed leaf counts toward dataSkippingNumIndexedCols. - file columns must default to NULL (no file literal), as with Variant. - Tighten reader behavior: invalid value -> must return null or fail (not may). - Add compatibility rows: Iceberg Compat V1/V2 (blocked), Type Widening (unsupported), Map keys (unsupported; array element / map value fine). Co-authored-by: Isaac

dtenedor

Did another round of review, mostly LGTM now after some remaining slight/minor updates.

…e provenance, IcebergCompatV1, stats counts Co-authored-by: Isaac <no-reply@databricks.com>

…ype-rfc # Conflicts: # protocol_rfcs/README.md

…uet-format#603) Self-references are being removed from the Parquet FILE type, so they are no longer a Delta-specific restriction. FILE is now inline-or-external only, with an absolute uri as Delta's sole added restriction. Co-authored-by: Isaac <no-reply@databricks.com>

…l, no guessed base) A relative uri is valid Parquet but not permitted in Delta; a conforming writer never emits one. Define reader behavior for a non-conforming input: return null or fail, and never resolve against a guessed base. Co-authored-by: Isaac <no-reply@databricks.com>

…format#603) Co-authored-by: Isaac <no-reply@databricks.com>

anoopj · 2026-08-20T01:05:54Z

+Delta CHECK Constraints | A `file` column may be used in a CHECK constraint expression through its leaf fields, addressed by logical name (for example, `f.size > 0`). Because a FILE leaf is not a struct field of the Delta schema, this is an explicit carve-out from the usual requirement that referenced columns exist in the schema — see the leaf-addressing rules in [Statistics for File Columns](#statistics-for-file-columns).
+Default Column Values | A `file` column must default to `NULL`. There is no Delta-defined way to construct a non-null `file` literal as a default expression, so `NULL` is the only permitted default (as with the Variant type).
+Change Data Feed | **Supported:** A table using the `file` data type is allowed to enable the Delta Change Data Feed. A `file` value flows through Change Data Feed and time travel like any other column value. See [Time Travel and Change Data Feed](#time-travel-and-change-data-feed) for the distinction between the reference and the referenced bytes.
+Iceberg Compatibility V1 / V2 | **Unsupported:** Under [IcebergCompatV2](#writer-requirements-for-icebergcompatv2) a `file` column is already blocked, because its type allow-list does not include `file`. [IcebergCompatV1](#writer-requirements-for-icebergcompatv1) has no type allow-list (it only blocks `Map`/`Array`/`Void`), so this RFC adds the requirement that a `file` column is not permitted in an IcebergCompatV1 table either. Iceberg has no equivalent type today; interaction with the in-flight IcebergCompatV3 RFC is out of scope for this RFC.


Consider clarifying that this won't be supported with icebergCompatV3 and icebergNativeV4 is coming soon.

Good call — reworked the row. It now states that file is unsupported in currently-released Iceberg versions (no equivalent type): blocked under IcebergCompatV2 by the type allow-list, and this RFC adds the same block for IcebergCompatV1 (which has no allow-list, only Map/Array/Void), and the same holds for the in-flight IcebergCompatV3 (#4574). It also notes that support is targeted for Iceberg V4 via the IcebergNativeV4 RFC (#7374), with the precise interaction left out of scope here. d05ec6f.

anoopj · 2026-08-20T01:08:32Z

+
+**Leaf addressing.** A FILE leaf is named by extending Delta's [field path](#field-path) formalism — "the ordered sequence of field names along that path" — by one final segment naming the literal `FILE` field (`uri`, `offset`, `size`, `content_type`, `checksum`, or `inline`). The FILE group's inner field names are fixed literals: they are **not** subject to [Column Mapping](#column-mapping) (the Parquet spec requires that they not be renamed) and have no assigned physical name, and they are **not** [struct fields](#struct-field) of the Delta schema. This one logical leaf path is *encoded differently at each site* where a leaf is referenced:
+
+- **Per-file statistics** are nested JSON objects keyed by physical names, so a leaf statistic is keyed by the file column's physical name followed by the literal FILE field name — for example `minValues.<physical name of the file column>.uri` (see the example below).


With AMT, this is changing. Stats will be keyed off field IDs.

Thanks — you're right that under the Adaptive Metadata Tree, content_stats are keyed by column-mapping field ID rather than physical name. The leaf addressing in this section describes the current add.stats format (JSON keyed by physical name). I've added a note that alignment with field-ID-keyed statistics per the in-flight Iceberg V4 Adaptive Metadata Tree RFC (#6640) is out of scope for this RFC and will follow that specification — the FILE leaves aren't schema fields, so how field IDs extend to them is something to settle in that work. d05ec6f.

anoopj · 2026-08-20T01:11:08Z

+- **inline** — the bytes are stored directly in the value (the `inline` field), or
+- **external** — the bytes are stored in a separate file at an absolute `uri` (optionally a byte range within it, via `offset`/`size`).
+
+These are the only two forms the Parquet `FILE` type provides: `offset`/`size` designate a byte range **within the file referenced by `uri`**, and there is no form that addresses a byte range in the data file that physically contains the value. (An earlier revision of the Parquet type allowed such a *self-reference*; it was removed from the specification in [apache/parquet-format#603](https://github.com/apache/parquet-format/pull/603).)


It says "it was removed from the specification" but apache/parquet-format#603 is not merged yet.

Yes, this is intentional, I synced up with folks driving apache/parquet-format#603 and they confirmed that this is the direction that they agreed on, following feedback from the community, and that we can adapt accordingly.
Of course, I will wait for that PR to be merged first and then make any additional adjustments in this PR if needed.

anoopj · 2026-08-20T01:18:53Z

+Clustered Tables | A `file` column cannot itself be chosen as a clustering column (a `file` value is a group and is not a comparable data type as a whole), but it can be used as a non-clustering data column of a clustered table. Its comparable leaf fields (for example, `size` or `content_type`) may be used as clustering columns, addressed by the leaf path defined in [Statistics for File Columns](#statistics-for-file-columns) — encoded in the `clusteringColumns` list as a path-segment array (for example `[["<physical name of the file column>", "size"]]`), the same logical leaf path used for its required per-column statistics.
+Delta Column Statistics | **Supported:** `nullCount` on the `file` column's leaf fields, and `minValues` / `maxValues` on its comparable, skipping-useful leaf fields (`uri`, `offset`, `size`, `content_type`). See [Statistics for File Columns](#statistics-for-file-columns). <br/> **Unsupported:** The `file` value as a whole is not a comparable data type; and `minValues` / `maxValues` are not collected for `inline` or `checksum`.
+Generated Columns | **Supported:** A `file` column is allowed to be used as a source in a generated column expression, via its leaf fields addressed by logical name (see the leaf-addressing carve-out in [Statistics for File Columns](#statistics-for-file-columns)). <br/> **Open question:** Whether `file` may be the *result* type of a generated column expression (for example, constructing a `file` reference from other columns) is left open for discussion on the associated issue, and is not specified by this RFC.
+Delta CHECK Constraints | A `file` column may be used in a CHECK constraint expression through its leaf fields, addressed by logical name (for example, `f.size > 0`). Because a FILE leaf is not a struct field of the Delta schema, this is an explicit carve-out from the usual requirement that referenced columns exist in the schema — see the leaf-addressing rules in [Statistics for File Columns](#statistics-for-file-columns).


It's unclear whether f.inline and f.checksum are referenceable

Clarified. inline is not a public field — it holds the raw referenced bytes and is not referenceable (in CHECK constraints or generated columns) and not collected in statistics. The other five fields, including checksum, are public and exposed. The RFC now states this explicitly in the statistics/leaf-addressing section and the CHECK/Generated-Columns rows, and inline is dropped from the indexed-leaf count (now five public leaves, not six). d05ec6f.

…target, AMT stats note - inline is not a public field: not referenceable in CHECK/generated columns and not collected in statistics; only the 5 public fields are exposed (5 indexed leaves). - Iceberg: blocked in V1/V2/V3 (no equivalent type); support targeted for Iceberg V4 (IcebergNativeV4, delta-io#7374). - Note that field-ID-keyed statistics (Iceberg V4 AMT RFC) are out of scope here. Co-authored-by: Isaac <no-reply@databricks.com>

dejankrak-db marked this pull request as ready for review July 7, 2026 09:39

ilicmarkodb reviewed Jul 7, 2026

View reviewed changes

dejankrak-db added 9 commits July 7, 2026 23:10

brkyvz reviewed Aug 4, 2026

View reviewed changes

dtenedor reviewed Aug 13, 2026

View reviewed changes

dtenedor approved these changes Aug 18, 2026

View reviewed changes

dejankrak-db and others added 5 commits August 18, 2026 20:08

[Protocol RFC] Address Daniel's 2nd review: leaf-path encoding, inlin…

7f69bfc

…e provenance, IcebergCompatV1, stats counts Co-authored-by: Isaac <no-reply@databricks.com>

Merge remote-tracking branch 'origin/master' into dejankrak-db/file-t…

5fc240a

…ype-rfc # Conflicts: # protocol_rfcs/README.md

[Protocol RFC] Word self-reference removal as merged (apache/parquet-…

d55f210

…format#603) Co-authored-by: Isaac <no-reply@databricks.com>

anoopj reviewed Aug 20, 2026

View reviewed changes


		Leaf addressing. A FILE leaf is named by extending Delta's [field path](#field-path) formalism — "the ordered sequence of field names along that path" — by one final segment naming the literal `FILE` field (`uri`, `offset`, `size`, `content_type`, `checksum`, or `inline`). The FILE group's inner field names are fixed literals: they are not subject to [Column Mapping](#column-mapping) (the Parquet spec requires that they not be renamed) and have no assigned physical name, and they are not [struct fields](#struct-field) of the Delta schema. This one logical leaf path is encoded differently at each site where a leaf is referenced:

		- Per-file statistics are nested JSON objects keyed by physical names, so a leaf statistic is keyed by the file column's physical name followed by the literal FILE field name — for example `minValues.<physical name of the file column>.uri` (see the example below).

Uh oh!

Conversation

dejankrak-db commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which Delta project/connector is this regarding?

Description

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brkyvz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dejankrak-db Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dtenedor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dtenedor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dejankrak-db commented Jul 6, 2026 •

edited

Loading

dejankrak-db Aug 13, 2026 •

edited

Loading