[Bug]: WAL replay of an entry referencing a deleted data file crashes InitPhase2 (exit 139), still on main with #3429 — is a supported way to discard unreplayable entries possible?

[Gisselle-Gonzalez]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Related: #3435 (same reporter, same instance — WAL files are never recycled and manual cleanup by timestamp/size crashes InitPhase2). This issue reports a related but different failure mode, and specifically tests whether #3429 covers it. As far as we can measure, it does not.

Version or Commit ID

Two builds, tested side by side on identical fresh copies of the same data volume:

• Control: v0.7.3-0-d755c5a (image infiniflow/infinity:v0.7.3-x64-v3, build 2026-08-06) — the build our production instance runs.
• Treatment: main@-0-eca7266 (image infiniflow/infinity:nightly-x64-v3, digest sha256:9203ca30ac080e62f1386bb18fa9912d72906abced1ba59c7d4ec4963473bcb7, build 2026-09-16 13:14), which includes 807af1b05 (Fix corrupt WAL handling: out-of-bounds reads and over-aggressive purge #3429, merged 2026-09-04).

Other environment information

Infinity deployment: Docker, standalone mode, via RAGFlow v0.27.1
Data layout: default /var/infinity layout on a named Docker volume (wal/, data/, persistence/, catalog/)
Same instance as #3435: at the time of this test, wal/ held 58,742 files (1.6 GB), 51,147 of them exactly 82 bytes
Config: stock infinity_conf.toml, default paths, log_level=info

Actual behavior and How to reproduce it

Context — how the volume got into this state

The engine crashed during a drop/cleanup operation. The WAL entries written after the last checkpoint reference data files that the crashed operation had already deleted. Since then, every boot of this volume crashes; the only recovery we have found is manually quarantining wal.log* files (moving, not deleting), which forfeits the entire post-checkpoint tail.

What happens on boot (6/6 runs, deterministic)

1. Startup reads the catalog fine and finds the latest checkpoint: Find and set checkpoint max commit ts: 499869.
2. WAL replay phases 1–2 complete: 15 entries are found after the checkpoint and parse without any checksum/size error — the WAL stream itself is intact.
3. While executing the replay of one of those entries (a WalCmdAppendV2), the engine fails to load the data file the entry references:

[critical] GetObjCache Failed to find object for local path db_1/tbl_11/seg_0/blk_0/col_3_out: Key not found: pm|object|db_1/tbl_11/seg_0/blk_0/col_3_out
[critical] Error: Failed to find object for local path /var/infinity/data/db_1/tbl_11/seg_0/blk_0/col_3_out@src/storage/buffer/file_worker/file_worker_impl.cpp:284
...
[critical] Error: TerminateHandler: Unhandled Exception: Failed to find object for local path /var/infinity/data/db_1/tbl_11/seg_0/blk_0/col_3_out@src/storage/buffer/file_worker/file_worker_impl.cpp:284

4. The terminate handler logs the unhandled exception and the process dies with exit code 139 (SIGSEGV) ~60 seconds into every start. Stack trace from the nightly build:

  0# infinity::PrintStacktrace@infinity_core(...) at /infinity/src/common/utility/exception_impl.cpp:48
  1# infinity::UnrecoverableError@infinity_core(...) at /infinity/src/common/utility/exception_impl.cpp:85
  2# infinity::FileWorker@infinity_core::GetFilePathInner[abi:cxx11](bool) at /infinity/src/storage/buffer/file_worker/file_worker_impl.cpp:284
  3# infinity::FileWorker@infinity_core::ReadFromFile(bool) at /infinity/src/storage/buffer/file_worker/file_worker_impl.cpp:207
  4# infinity::BufferObj@infinity_core::LoadNoLock(bool) at /infinity/src/storage/buffer/buffer_obj_impl.cpp:117
  5# infinity::BufferObj@infinity_core::Load(bool) at /infinity/src/storage/buffer/buffer_obj_impl.cpp:178
  6# infinity::VarBufferManager@infinity_core::GetInnerNoLock() at /infinity/src/storage/column_vector/var_buffer_impl.cpp:187
  7# infinity::VarBufferManager@infinity_core::TotalSize() at /infinity/src/storage/column_vector/var_buffer_impl.cpp:211
  8# infinity::NewTxn@infinity_core::AppendInColumn(...) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:1016
  9# infinity::NewTxn@infinity_core::AppendInBlock(...) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:948
 10# infinity::NewTxn@infinity_core::CommitBottomAppend(infinity::WalCmdAppendV2@infinity_core*) at /infinity/src/storage/new_txn/new_txn_data_impl.cpp:1882
 11# infinity::NewTxn@infinity_core::CommitBottom() at /infinity/src/storage/new_txn/new_txn_impl.cpp:4578
 12# infinity::NewTxn@infinity_core::CommitReplay() at /infinity/src/storage/new_txn/new_txn_impl.cpp:2151
 13# infinity::NewTxnManager@infinity_core::CommitReplayTxn(...) at /infinity/src/storage/new_txn/new_txn_manager_impl.cpp:327
 14# infinity::WalManager@infinity_core::ReplayWalEntries(...) at /infinity/src/storage/wal/wal_manager_impl.cpp:719
 15# infinity::Storage@infinity_core::AdminToWriter() at /infinity/src/storage/storage_impl.cpp:326
 16# infinity::Storage@infinity_core::SetStorageMode(...) at /infinity/src/storage/storage_impl.cpp:835
 17# infinity::InfinityContext@infinity_core::ChangeServerRole(...) at /infinity/src/main/infinity_context_impl.cpp:132
 18# infinity::InfinityContext@infinity_core::InitPhase2(bool) at /infinity/src/main/infinity_context_impl.cpp:90
 19# main at /infinity/src/bin/infinity_main.cpp:285

Why this looks like #3429 territory but isn't covered by it

#3429 fixes the WAL reading layer (out-of-bounds reads on corrupt entries, over-aggressive purge) — and in our repro the reading layer works: all 15 post-checkpoint entries are found and parsed cleanly on both builds. The crash is in the execution of replay — CommitReplay → CommitBottomAppend → append into a column whose backing file no longer exists. We verified this empirically with a controlled A/B:

• Two fresh identical clones of the poisoned volume (cp -a into disposable volumes; source mounted read-only; one clone per arm, so the control run could not contaminate the treatment run).
• Control arm — v0.7.3-0-d755c5a: 3/3 boots → exit 139.
• Treatment arm — main@-0-eca7266 (with Fix corrupt WAL handling: out-of-bounds reads and over-aggressive purge #3429): 3/3 boots → exit 139, identical critical line (file_worker_impl.cpp:284) and the same stack (the only difference between builds is one frame's source line: new_txn_data_impl.cpp:1881 vs :1882).

Arm	Build	Boot 1	Boot 2	Boot 3
Control	v0.7.3-0-d755c5a	exit 139 (~67 s)	exit 139 (~61 s)	exit 139 (~60 s)
Treatment	main@-0-eca7266 (incl. #3429)	exit 139 (~64 s)	exit 139 (~60 s)	exit 139 (~61 s)

So this is not a one-off crash report: the same poisoned WAL that crash-loops v0.7.3 also crash-loops the current tip of main.

How we ran each arm (for reference)

docker volume create inf_rehearsal_ctrl
docker run --rm -v <poisoned_backup_volume>:/src:ro -v inf_rehearsal_ctrl:/dst \
  alpine sh -c 'cp -a /src/. /dst/'

docker create --name inf-test --network inf-test-net \
  --memory 8073741824 --restart=no \
  -v inf_rehearsal_ctrl:/var/infinity \
  infiniflow/infinity:v0.7.3-x64-v3 -f /infinity_conf.toml
docker cp infinity_conf.toml inf-test:/infinity_conf.toml
docker start inf-test          # ~60 s later: exit 139
docker start inf-test          # repeatable

(For the treatment arm, same commands with the nightly image referenced by digest. Standalone mode; --memory mirrors what our production container runs with; --restart=no, boots restarted manually to confirm determinism.)

Expected behavior

Two things we'd love your thoughts on:

1. Is there, or could there be, a supported way to selectively discard a WAL entry whose replay references a missing data file — without discarding the rest of the post-checkpoint WAL? The engine already detects the exact situation and logs it (GetObjCache Failed to find object ... Key not found) right before throwing UnrecoverableError. Skipping that one transaction — with a loud warning and some durable marker, e.g. in the catalog — would let such a volume boot with a bounded, explicit loss instead of requiring manual WAL surgery. Fix corrupt WAL handling: out-of-bounds reads and over-aggressive purge #3429's purge is intentionally conservative at file granularity (and its tests argue well for that); this would be one level more surgical: entry-level, at replay execution.
2. Is the failure mode itself intended? UnrecoverableError during replay propagates through the terminate handler and the process dies with SIGSEGV (exit 139) rather than exiting cleanly with the message. A clean non-zero exit with the same message would already make automated boot-loop detection and restart policies much easier to handle.

For our part: the operational workaround we use today (also referenced in #3435) is moving — never deleting — post-checkpoint wal.log* files into a quarantine directory. That boots the instance but forfeits the whole tail. We are not set up to contribute a fix ourselves — this report is mainly to (a) document, with a deterministic reproduction, that #3429 does not cover this path, and (b) ask whether a supported "discard unreplayable entry" path exists or is planned.

Happy to share more logs/details from the test environment if useful.

Thanks!
Gisselle