GH-48701: [C++][Parquet] Add ALPpd encoding by prtkgaur · Pull Request #48345 · apache/arrow

prtkgaur · 2025-12-05T00:23:45Z

Co-authored-by: Dhirhan Kanesalingam dhirhan17@gmail.com

Rationale for this change

Adaptive Lossless Floating-Point (ALP)
is designed for floating-point data that commonly represents decimal values.
For these workloads, ALP can provide better compression and faster decoding
than general-purpose compression or existing Parquet encodings.

This PR adds ALP support for Parquet FLOAT and DOUBLE columns in the Arrow
C++ implementation.

Specification

Original ALP specification
Parquet AlpEncoding.md
(merged through Add ALP support proposal parquet-format#539; currently in Preview)

What changes are included in this PR?

This PR adds:

The core ALP compression and decompression implementation.
Sampling logic for selecting encoding parameters.
Page metadata serialization, validation, and decoding.
Parquet encoders and decoders for FLOAT and DOUBLE.
Incremental decoding of ALP pages one vector at a time.
CMake and Meson build integration.
C++ documentation describing how to use the encoding.

ALP is opt-in on the write path. It is used only when the writer explicitly
selects Encoding::ALP and dictionary encoding is disabled. Arrow does not
currently select ALP automatically based on the input data.

Are these changes tested?

Yes. Test coverage includes:

Interoperability with the ALP conformance file in parquet-testing.
FLOAT and DOUBLE round trips.
Decimal-like, constant, random, and exception-heavy inputs.
Empty and all-null pages.
Configurable vector sizes.
Truncated or malformed page metadata.
Invalid offsets, element counts, bit widths, and exception positions.
Batched and incremental decoding.

Are there any user-facing changes?

Yes. Arrow C++ can read Parquet pages encoded with ALP. Writers can explicitly
select ALP for supported floating-point columns.

ALP requires reader support and is not selected automatically, so existing
writer behavior remains unchanged unless users opt in.

GitHub Issue: [ALP][Parquet] Add C++ implementation of ALPpd encoder/decoder #48701

github-actions · 2025-12-05T00:24:08Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

alamb · 2025-12-08T14:07:48Z

Thanks @prtkgaur -- it is super exciting to see this movement.

Unfortunately, I am not familiar with the C/C++ codebase to give this a realistic review.

I started the CI checks on this PR and had some comments about the testing.

prtkgaur · 2025-12-08T23:31:26Z

+    std::string tarball_path = std::string(__FILE__);
+    tarball_path = tarball_path.substr(0, tarball_path.find_last_of("/\\"));
+    tarball_path = tarball_path.substr(0, tarball_path.find_last_of("/\\"));
+    tarball_path += "/arrow/cpp/submodules/parquet-testing/data/floatingpoint_data.tar.gz";


@Reviewer the data sits in the parquet-testing submodule
apache/parquet-testing#100

prtkgaur · 2025-12-08T23:41:27Z


+  // Unsafe resize without initialization - use only when you will immediately
+  // overwrite the memory (e.g., before memcpy). Only safe for POD types.
+  void UnsafeResize(size_t n) {


Using this over resize gave us around 2-3% performance improvement

emkornfield · 2025-12-09T21:32:31Z

Talked offline and wanted to capture notes on high-level changes:

For headers, lets try to reduce duplication with values already in the parquet header.
For remaining items in headers, lets try to be parsimonious with values (i.e. 4 bytes is probably overkill for enums)
Naming convention on files is off (use snake_case).
Given description of ALP, we probably want a top level encoding enum value for the 2 different modes of ALP.

prtkgaur · 2025-12-16T01:24:55Z

Talked offline and wanted to capture notes on high-level changes:

For headers, lets try to reduce duplication with values already in the parquet header.

For remaining items in headers, lets try to be parsimonious with values (i.e. 4 bytes is probably overkill for enums)

Naming convention on files is off (use snake_case).

Given description of ALP, we probably want a top level encoding enum value for the 2 different modes of ALP.

Thanks for the feedback @emkornfield. We have addressed

Reduce duplication of fields between page header and alp header
Other fields have been updated to use 1 byte. Header is now just 8 bytes compared to 40 bytes earlier.
Naming of files has been updated.
We do have the top level enums describing the mode and layout structure.
enum class AlpBitPackLayout { kNormal }; and enum class AlpMode { kAlp };

github-actions · 2026-01-01T20:58:34Z

⚠️ GitHub issue #48701 has been automatically assigned in GitHub to PR creator.

prtkgaur · 2026-01-02T05:56:08Z

+
+    // Slow path: partial read - decode to intermediate buffer
+    // ALP Bit unpacker needs batches of 64
+    if (needs_decode_) {


TODO(prateek) : check with Antoine and other reviewers if there is a way to relax this constraint. Though this has negligible impact on performance.

no action needed

prtkgaur · 2026-01-02T06:00:11Z

Please check cpp/src/arrow/util/alp/ALP_Encoding_Specification_terse.md for a more terse spec of the encoding.

Also this file will be removed once the spec in parquet format repository is merged.

prtkgaur · 2026-01-02T06:01:08Z

+
+## 2. Data Layout
+
+ALP encoding consists of a page-level header followed by one or more encoded vectors. Each vector contains up to 1024 elements.


Replace 1024 with the constant specified in AlpConstant file.

clang cannot attach a tparam to a member template declared in its class, so the doc build rejects it.

The marker on a class does not reach a member template, so the Windows link could not find the instantiations.

CompressVector takes int32_t, so passing a size_t narrows and clang rejects it with -Wshorten-64-to-32 under -Werror.

Their definitions live in implementation files, so a separate test executable cannot reach them across a shared library without the marker.

The index is 64-bit, so MSVC warns that a 32-bit shift may have been meant as 64-bit, and the CI build treats that as an error.

arrow_reader_writer_test.cc has a DoRoundtrip with defaulted trailing parameters, so a four-argument call matched both helpers once a unity build compiled the two files together.

AlpEncodedVectorInfo is exported, so binding a reference to its static constexpr kStoredSize needs an out-of-line definition the library does not provide, and the Windows GCC link fails. GetStoredSize returns it by value. The FOR info class is a template and is instantiated locally, so it is unaffected.

An empty vector's data() may be null, and memcpy and memset must not be passed a null pointer even for a zero length. A vector with no exceptions, or with bit width zero, reached that case, which the sanitizer build reports as fatal. The encoded form and decoded values are unchanged.

The bounds admitted the largest float below 2^31 and the largest double below 2^63. Fast rounding can carry a value up one ulp, to exactly 2^31 or 2^63, which the integer types cannot hold. Those two values now become exceptions and still round-trip exactly.

The ALP conformance file is already on parquet-testing main, so Arrow picks it up on any routine bump. The tests skip until the pin moves.

The pipeline flowchart in alp_internal.h becomes a paragraph, the page layout is drawn once instead of three times, subscripts and arrows are spelled in ASCII, and the test section headings use Arrow's single rule. Comments only.

Some comments described what the code used to do rather than what the test pins, such as the misaligned exception arrays that LoadView copies into aligned storage. Others restated the code below them. Comments only.

Split the ALP implementation into constants, metadata, compression, sampler, and codec units. Simplify the internal APIs and update the CMake/Meson build integration. Remove the separate writer opt-in flag. ALP is selected explicitly with Encoding::ALP while dictionary encoding is disabled. Harden encoding and decoding: - validate page headers, vector metadata, offset chains, bit widths, element counts, and exception positions; - reject pages that leave ALP values unconsumed after all levels are read; - reuse pool-backed scratch buffers and cache partially decoded vectors; - use typed power-of-ten constants for exact decode semantics; - emit a valid header-only page for all-null input. Rework the ALP tests around the production APIs, remove redundant cases, and add malformed-input, boundary, exception, and all-null coverage. Move the end-to-end tests to arrow_encoding_test.cc and enable real parquet-testing interoperability coverage. Update the submodule pin, benchmarks, and C++ documentation.

The layout tables for AlpInfo, AlpForInfo and the serialized vector were lost when alp_internal.h was split. They now sit next to the classes they describe. The old ForInfo table gave 6 and 10 bytes; the real sizes are 5 and 9, a frame of reference plus a bit width with no padding.

AlpEncoder's static_assert fires only when the template is instantiated, and for an unsupported physical type the factories throw before that, so test the throw for the six other types. Also restores the note about falling back to PLAIN, since ALP expands a few of the paper's datasets and the writer never declines it.

No write path selects ALP, so a column carries it only where encoding() names it.

Move the decode target restriction from requires clauses to static assertions, so an invalid target stays a compile-time error without the constraint appearing in exported symbol names. GCC can then match the explicit instantiations, and Clang shared builds resolve the same symbols.

prtkgaur · 2026-10-07T17:59:46Z

Since this is now approved by @wgtmac are there any concerns if we merge it? If not, I propose we do so tomorrow @prtkgaur it looks like there is a conflict too

Done. The new failure looks unrelated to my change.

    @        0x110aafac3  absl::lts_20250127::log_internal::LogMessage::PrepareToDie()
    @        0x110aaf3e8  absl::lts_20250127::log_internal::LogMessage::SendToLog()
    @        0x110aaea47  absl::lts_20250127::log_internal::LogMessage::Flush()
    @        0x110aafd15  absl::lts_20250127::log_internal::LogMessageFatal::~LogMessageFatal()
    @        0x110aafd45  absl::lts_20250127::log_internal::LogMessageFatal::~LogMessageFatal()
    @        0x10f819561  grpc::internal::CallOpSet<>::ContinueFillOpsAfterInterception()
    @        0x10034e2be  grpc::internal::InterceptorBatchMethodsImpl::ProceedClient()

wgtmac · 2026-10-08T02:49:51Z

@github-actions crossbow submit -g cpp

github-actions · 2026-10-08T02:52:37Z

Revision: 2fb0c5f

Submitted crossbow builds: ursacomputing/crossbow @ actions-1ee1f44348

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

wgtmac · 2026-10-08T03:28:09Z

The two failed crossbow builds are related. Could you please fix them @prtkgaur?

kStoredSize is int64_t, so subtracting it from a size_t yields int64_t wherever size_t is 32 bits. Inside the span's braced initializer that is a narrowing conversion, which broke the i386 and Emscripten builds.

prtkgaur · 2026-10-08T20:07:06Z

The two failed crossbow builds are related. Could you please fix them @prtkgaur?

Hi @wgtmac took a stab at it. Quick question, should I trigger this myself "@github-actions crossbow submit -g cpp"

emkornfield · 2026-10-08T20:08:52Z

@github-actions crossbow submit -g cpp

emkornfield · 2026-10-08T20:09:23Z

@prtkgaur I tried totrigger it but if it still comes up with failures might be worth a try to see if you have permissions.

github-actions · 2026-10-08T20:11:34Z

Revision: e2275a1

Submitted crossbow builds: ursacomputing/crossbow @ actions-f3bf07b14a

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

prtkgaur · 2026-10-08T22:15:29Z

@prtkgaur I tried totrigger it but if it still comes up with failures might be worth a try to see if you have permissions.

Thanks @emkornfield. Don't see that failure in the latest report.

The only failing thing now

110/110 Test  #81: arrow-s3fs-test ..............................***Timeout 300.04 sec

Is unrelated to this change.

github-actions Bot added Component: Parquet Component: C++ awaiting review Awaiting review labels Dec 5, 2025

prtkgaur force-pushed the gh540-alp-pseudoDecimal-encoding branch 3 times, most recently from 1b78a5c to d563ce0 Compare December 7, 2025 15:46

alamb reviewed Dec 8, 2025

View reviewed changes

Comment thread cpp/src/arrow/util/alp/data/floatingpoint_data.tar.gz Outdated

alamb reviewed Dec 8, 2025

View reviewed changes

Comment thread cpp/src/parquet/types.h

alamb mentioned this pull request Dec 8, 2025

[Parquet] Prototype ALP encoding apache/arrow-rs#8748

Closed

prtkgaur changed the title ~~[Gh540] Add ALPpd encoding to parquet~~ [Gh539] Add ALPpd encoding to parquet Dec 8, 2025

prtkgaur commented Dec 8, 2025

View reviewed changes

prtkgaur changed the title ~~[Gh539] Add ALPpd encoding to parquet~~ [Gh539][Encoding] Add ALPpd encoding to parquet Dec 8, 2025

prtkgaur changed the title ~~[Gh539][Encoding] Add ALPpd encoding to parquet~~ [Gh-539][Encoding] Add ALPpd encoding to parquet Dec 8, 2025

prtkgaur force-pushed the gh540-alp-pseudoDecimal-encoding branch from 0c035b7 to 1cb0852 Compare December 8, 2025 23:48

prtkgaur changed the title ~~[Gh-539][Encoding] Add ALPpd encoding to parquet~~ [Gh-539][ParquetEncoding][c++] Add ALPpd encoding to parquet Dec 9, 2025

prtkgaur force-pushed the gh540-alp-pseudoDecimal-encoding branch from 35f1ad7 to 0908342 Compare December 15, 2025 21:28

github-actions Bot added the Component: Documentation label Dec 31, 2025

prtkgaur changed the title ~~[Gh-539][ParquetEncoding][c++] Add ALPpd encoding to parquet~~ [Gh-48701][ParquetEncoding][c++] Add ALPpd encoding to parquet Dec 31, 2025

kou changed the title ~~[Gh-48701][ParquetEncoding][c++] Add ALPpd encoding to parquet~~ GH-48701: [C++][Parquet] Add ALPpd encoding Jan 1, 2026

prtkgaur commented Jan 2, 2026

View reviewed changes

prtkgaur force-pushed the gh540-alp-pseudoDecimal-encoding branch from 1b08599 to f5f5011 Compare January 12, 2026 16:00

prtkgaur marked this pull request as ready for review January 13, 2026 18:03

prtkgaur requested a review from wgtmac as a code owner January 13, 2026 18:03

alamb mentioned this pull request Jan 14, 2026

GH-533: Adaptive Lossless Floating-Point (ALP) Encoding apache/parquet-format#548

Closed

1 task

sfc-gh-pgaur and others added 19 commits October 7, 2026 14:44

Describe the decode target type in prose

aa2e431

clang cannot attach a tparam to a member template declared in its class, so the doc build rejects it.

Export the ALP decode member templates

4219b5b

The marker on a class does not reach a member template, so the Windows link could not find the instantiations.

Cast the element count the ALP test passes

d9fa958

CompressVector takes int32_t, so passing a size_t narrows and clang rejects it with -Wshorten-64-to-32 under -Werror.

Export the internal ALP classes for tests

4df7226

Their definitions live in implementation files, so a separate test executable cannot reach them across a shared library without the marker.

Widen the shift the ALP test takes a modulus with

c32fd46

The index is 64-bit, so MSVC warns that a 32-bit shift may have been meant as 64-bit, and the CI build treats that as an error.

Rename the ALP roundtrip helper

9f2b73e

arrow_reader_writer_test.cc has a DoRoundtrip with defaulted trailing parameters, so a four-argument call matched both helpers once a unity build compiled the two files together.

Drop the parquet-testing submodule bump

df83dfb

The ALP conformance file is already on parquet-testing main, so Arrow picks it up on any routine bump. The tests skip until the pin moves.

Trim the ALP comments

e0bcc57

The pipeline flowchart in alp_internal.h becomes a paragraph, the page layout is drawn once instead of three times, subscripts and arrows are spelled in ASCII, and the test section headings use Arrow's single rule. Comments only.

State the constraints the ALP tests check

892b195

Some comments described what the code used to do rather than what the test pins, such as the misaligned exception arrays that LoadView copies into aligned storage. Others restated the code below them. Comments only.

Restore ALP wire layout diagrams

c0dd888

Document that ALP is never chosen automatically

dff2ea5

No write path selects ALP, so a column carries it only where encoding() names it.

Fix ALP builds across compilers

642ee4e

prtkgaur force-pushed the gh540-alp-pseudoDecimal-encoding branch from 619c39b to 2fb0c5f Compare October 7, 2026 16:03

Fix ALP vector metadata bounds on 32-bit targets

e2275a1

kStoredSize is int64_t, so subtracting it from a size_t yields int64_t wherever size_t is 32 bits. Inside the span's braced initializer that is a narrowing conversion, which broke the i386 and Emscripten builds.


		## 2. Data Layout

		ALP encoding consists of a page-level header followed by one or more encoded vectors. Each vector contains up to 1024 elements.

Conversation

prtkgaur commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Specification

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

alamb commented Dec 8, 2025

Uh oh!

Uh oh!

prtkgaur Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

prtkgaur Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

emkornfield commented Dec 9, 2025

Uh oh!

prtkgaur commented Dec 16, 2025

Uh oh!

github-actions Bot commented Jan 1, 2026

Uh oh!

prtkgaur Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

prtkgaur Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

prtkgaur Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

prtkgaur May 7, 2026

Choose a reason for hiding this comment

Uh oh!

prtkgaur Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

prtkgaur commented Oct 7, 2026

Uh oh!

wgtmac commented Oct 8, 2026

Uh oh!

github-actions Bot commented Oct 8, 2026

Uh oh!

wgtmac commented Oct 8, 2026

Uh oh!

prtkgaur commented Oct 8, 2026

Uh oh!

emkornfield commented Oct 8, 2026

Uh oh!

emkornfield commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 8, 2026

Uh oh!

prtkgaur commented Oct 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

prtkgaur commented Dec 5, 2025 •

edited

Loading

emkornfield commented Oct 8, 2026 •

edited

Loading