Skip to content

feat(writer): cascade delta like Rust's DeltaScheme, edition-gated (issue #410) - #439

Merged
dfa1 merged 1 commit into
mainfrom
feat/410-delta-cascade
Oct 3, 2026
Merged

dfa1 merged 1 commit into
mainfrom
feat/410-delta-cascade

Conversation

@dfa1

@dfa1 dfa1 commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Fixes #440.
Closes #410: Rust-parity port 4/4, after RunEnd (#435), ZigZag (#436) and RLE (#437, plus float RLE in #438).

Delta

  • DeltaEncodingEncoder.encodeCascade: bases and deltas (Rust ids 0/1, also our wire order) become open children, with Rust's DeltaScheme exclusion (Delta on both, so data is never delta-encoded twice).
  • Arrays shorter than one FastLanes chunk are skipped (Rust's MIN_DELTA_LEN = 1024).
  • Edition-gated like Rust: Rust registers DeltaScheme, but notes "no edition includes fastlanes.delta yet, so the session's enabled editions decide". It is offered as a cascade candidate only when an enabled edition contains it (unstable2025.05.0+, the same gate as OnPair). Default writes never emit it.
  • Rust's ancestor rule (no Delta on Dict codes) is moot: Delta joins the cascade only via an unstable edition, and Dict's CODES_EXCLUDED already bars Sequence there. Noted for a follow-up if we want it literal.
  • The shared delta computation moves into one Deltas helper used by encode and encodeCascade; the terminal output is unchanged.
  • Rust's ratio-callback estimate (1.25× bar, 0.95 penalty) is not ported. Our cascade measures by sampling, so this is a selection-heuristic difference, not a format one.

Bug fix found on the way (edition guard)

editionExcluded only covered the encoders the writer itself lists. SparseEncodingEncoder keeps a private candidate list for compressing a validity bitmap's patch indices, and it offers fastlanes.delta. Plain delta always lost there, so this never surfaced. Once delta cascades it wins, and a default-edition write then failed the backstop guard (fastlanes.delta: outside the configured edition(s) [core2026.08.0]). This reproduced in 3 FileSizeComparison tests. editionExcluded now covers every well-known id outside the enabled editions. That is safe by construction: anything it removes would have failed the backstop anyway.

Tests

  • WriterEditionGuardTest: with default editions, cascading never emits delta; the unstable edition picks it on jittered timestamps and reads back exactly; and the private-candidate-list regression (fails without the fix, verified with a clean build).
  • DeltaEncodingEncoderTest.Cascade: children equal the terminal buffers, exclusions, and the under-one-chunk skip.
  • Interop javaWriter_jniReader_delta_cascading_unstableEdition: vortex-jni reads the cascaded delta file on a full scan, a mid-chunk row range and a zone-pruned filter.
  • Default-edition sizes are byte-identical. Green: writer suite plus 370 integration tests.

Docs: docs/compatibility.md delta row, plus the CHANGELOG.

🤖 Generated with Claude Code

…ssue #410)

Parity port 4/4 for #410. fastlanes.delta overrides encodeCascade: bases
and deltas (Rust ids 0/1, also our wire order) become open children with
Rust's exclusion (Delta on both), and arrays under one FastLanes chunk are
skipped (Rust's MIN_DELTA_LEN). Like Rust, Delta is registered as a cascade
candidate but only offered when an enabled edition contains it
(unstable2025.05.0+); default writes never emit it. On jittered timestamps
with the unstable edition the cascade now picks it.

Fix found on the way: the edition exclusion set only covered the encoders
the writer itself lists, so a candidate kept in an encoder's private list
(Sparse's bool patch-index cascade offers delta) escaped it. Cascading
delta then won there and a default-edition write failed the backstop
guard. editionExcluded now covers every well-known id outside the enabled
editions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dfa1
dfa1 force-pushed the feat/410-delta-cascade branch from 1fcc907 to 9882bc6 Compare October 3, 2026 07:30
@dfa1
dfa1 merged commit 309dee1 into main Oct 3, 2026
6 checks passed
@dfa1
dfa1 deleted the feat/410-delta-cascade branch October 3, 2026 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant