feat(writer): cascade run-end ends and values (issue #410) - #435
Merged
Merged
Conversation
RunEnd was a terminal encoding: its ends and values were stored raw, so the upstream RunEndScheme exclusion rules had nothing to attach to. It now overrides encodeCascade and hands both buffers to the compressor as open child slots, with Rust's per-child exclusions: - ends (strictly increasing, all distinct): Dict, RunEnd, RLE, Sparse barred - values (adjacent always differ): RunEnd barred Same [ends, values] wire shape as the terminal path. FileSizeComparison drops 1,174,359 -> 1,161,559 bytes (-1.09%, Java/JNI 0.95x -> 0.94x) and GlobalDictComparison 2,487,928 -> 2,466,808 (-0.85%); vortex-jni reads the cascaded files (all interop suites green). Also adds PrimitiveArrays.fromLongsArray, the inverse of toLongs, replacing RunEnd's private widen/narrow and Patched's private fromLongs copies. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
force-pushed
the
feat/410-runend-cascade
branch
from
October 3, 2026 06:37
d3d1820 to
a4327e9
Compare
This was referenced Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #410: the "should RunEnd/RLE/ZigZag/Delta cascade?" follow-up, starting with RunEnd.
RunEndEncodingEncoderwas terminal, so its ends and values were always stored raw. It now overridesencodeCascadeand hands both to the compressor as openChildSlots, with Rust'sRunEndSchemeexclusions declared on each slot:vortex.dict,vortex.runend,fastlanes.rle,vortex.sparsevortex.runendSame
[ends, values]wire shape as the terminalencode.Rust's ancestor rule (no RunEnd on a Dict's values pool) has nothing to attach to here: primitive Dict keeps its values as an owned buffer, and the Utf8 pool never reaches RunEnd, which accepts integers only.
Effect (
FileSizeComparisonIntegrationTest,main→ this PR):vortex-jni reads the cascaded files: every interop suite is green (367 integration tests), plus 1,922 writer and 1,592 reader unit tests.
Dedup: adds
PrimitiveArrays.fromLongsArray, the inverse oftoLongs. It replaces RunEnd's privatewiden/narrow(widenwas a copy oftoLongs) and Patched's privatefromLongs.Tests:
RunEndEncodingEncoderTest.Cascade(run split, per-slot exclusions, U8 carrier narrowing, empty input) andPrimitiveArraysTest(fromLongsArrayinvertstoLongsat every integer width's extremes).🤖 Generated with Claude Code