Skip to content

fix(spec): reject non-primitive primary-key and partition-key types - #976

Open
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix/reject-non-primitive-key-field-types
Open

jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix/reject-non-primitive-key-field-types

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

validate_key_field_types rejected only VECTOR, and only for primary keys
and an explicit bucket-key — partition keys were never checked. Java
SchemaValidation.validateOnlyContainPrimitiveType forbids MAP, ARRAY,
ROW, MULTISET, VECTOR, and VARIANT for both primary keys and partition
keys. We therefore accepted tables Java rejects: such a key has no ordering,
and a partition value is encoded into a directory path, so bucket/merge and
partition-path behavior is undefined and the table is unreadable across
engines. Reject the full set for primary, partition, and bucket keys, and add
create-time tests.

`validate_key_field_types` rejected only `VECTOR`, and only for primary keys
and an explicit `bucket-key` — partition keys were never checked. Java
`SchemaValidation.validateOnlyContainPrimitiveType` forbids `MAP`, `ARRAY`,
`ROW`, `MULTISET`, `VECTOR`, and `VARIANT` for both primary keys and partition
keys. We therefore accepted tables Java rejects: such a key has no ordering,
and a partition value is encoded into a directory path, so bucket/merge and
partition-path behavior is undefined and the table is unreadable across
engines. Reject the full set for primary, partition, and bucket keys.
@JingsongLi

Copy link
Copy Markdown
Contributor

[P2] Do not apply the primary/partition VARIANT restriction to hash bucket keys

At crates/paimon/src/spec/schema.rs:1499-1502, the shared reject closure now prohibits VARIANT bucket keys as well as primary/partition keys. The default append-table bucket route hashes a BinaryRow; it does not require key ordering, and this codebase already encodes VARIANT values into that row.

I verified a Parquet append table with bucket=2, bucket-key=v, and a VARIANT column v: on the exact baseline, it writes, commits, scans and reads two original JSON values successfully, with valid bucket IDs. On this head, schema creation fails with The VARIANT type of bucket key field 'v' is unsupported.

The upgrade also affects existing tables. Loading the persisted baseline schema succeeds, but a filesystem Catalog alter_table that only changes the ordinary id column's comment now fails with that same bucket-key error. The identical ALTER passes on the baseline. This makes unrelated metadata maintenance unavailable for tables which were working before this change.

Please separate bucket restrictions from primary/partition restrictions and retain the supported VARIANT hash-key path. Java's primitive-key validation applies to primary/partition keys; its separate nested bucket check lists ARRAY/MULTISET/MAP/ROW, not VARIANT. I am not claiming Java's full VARIANT write path is supported—the reproduced regression is in the existing Rust write/read and ALTER paths.

Validation on head c7504c1663e51cad3f9ea3efb28e72c06042c1e7: 127 schema tests passed. Both additional creation and existing-table ALTER scenarios fail on head; full write/commit/read and the ALTER scenario pass with schema.rs restored to the exact baseline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants