Skip to content

test_runner: do not crash on stdout that mimics a v8 frame - #66273

Open
faizanu94 wants to merge 4 commits into
nodejs:mainfrom
faizanu94:test-runner-stdout-frame-guard
Open

faizanu94 wants to merge 4 commits into
nodejs:mainfrom
faizanu94:test-runner-stdout-frame-guard

Conversation

@faizanu94

@faizanu94 faizanu94 commented Sep 25, 2026 •

Copy link
Copy Markdown

Summary

node --test can crash the parent runner when a test writes bytes to stdout that look like a v8 report frame. A single console.log of the wrong bytes aborts the whole run with an internal error that does not point at any test:

Error: Unable to deserialize cloned data.
    at #processRawBuffer (node:internal/test_runner/runner:...)

Fixes: #66164

Root cause

The child test process multiplexes framed report messages and raw user stdout on one pipe. A frame starts with the two magic bytes FF 0F followed by a 4 byte size. #processRawBuffer scans for that magic, reads the size and hands the payload to the v8 deserializer. User output can contain those same bytes, so stray stdout with a plausible size reaches the deserializer, which throws. The call had no error handling, so the exception aborted the entire run. The earlier >>> 0 fix in #64706 hardened the size read only. This case has a valid looking size and fails one step later at the deserialize.

Fix

The reporter writes each frame as a v8 header, a 4 byte size, then a second v8 header, then the serialized value. So a real frame always repeats the FF 0F header at the start of its payload. #processRawBuffer now uses that structure instead of a blanket catch:

  1. Validate before deserializing. A candidate is handed to the deserializer only when its payload is long enough and begins with the inner v8 header. Stray stdout that only mimics a frame fails this check and is emitted as stdout. A genuine frame with a valid inner header that still fails to deserialize is left to throw, so a real regression in the report protocol surfaces instead of being hidden.
  2. Resync live. When the bytes only mimic a frame, one byte is emitted as stdout and the header search restarts right away, so a real frame that follows stray bytes is reported immediately instead of waiting for shutdown.
  3. Single loop. #processRawBuffer is now one loop that inspects the head of the buffer on each pass and handles exactly one case: leading stdout, an incomplete frame, a frame that only mimics the header, or a real frame. This replaces the earlier nested loops and the mimicsFrame flag.

This turns a fatal crash into recoverable stdout, keeps any real frames that follow the stray bytes and surfaces genuine deserialize failures.

Tests

Added cases to test/parallel/test-runner-v8-deserializer.mjs:

  • stdout that mimics a frame with a plausible size becomes stdout and does not crash
  • that false frame followed by a real message: the parser resyncs live and reports the real event before drain()
  • a real message, then the false frame, then another real message: both real messages survive
  • several stray frames in a row are peeled off as stdout and a real message after them is still reported
  • the false frame split across two chunks still recovers as stdout
  • a frame whose payload is shorter than the inner header is treated as stdout
  • a genuinely corrupt frame with a valid inner header is left to throw instead of being hidden

Known limitation (possible follow-up, out of scope here)

For stray bytes to be misread now they would have to reproduce the full frame shape, the outer header, a matching size, the inner header, then a valid serialized payload. This is pre-existing and extremely unlikely. Fully removing the ambiguity needs an escape mechanism or a separate channel for user stdout, which is a larger change. This PR removes the crash, which is the reported bug.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/test_runner

@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Sep 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Welcome to Node.js, and thank you for your first contribution!

Before review, please take a moment to read:

Please make sure every commit is signed off. For a first pull request, GitHub Actions require collaborator approval and Jenkins CI must be started by a collaborator or triager, so an initial wait is normal.

@inoway46

Copy link
Copy Markdown
Contributor

This seems reasonable as a mitigation, but I'm concerned that swallowing deserialization errors could hide real regressions in the internal report protocol.

Could we make the frame validation stricter before deserializing instead? For example, checking for the expected inner V8 header after the length field may let us reject false positives as stdout without hiding genuine deserialization failures.

Separating the report stream from user stdout still seems like the cleanest long-term fix, but stricter validation might be a smaller alternative.

@codecov

codecov Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.92308% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.36%. Comparing base (6a6e09e) to head (d249668).
⚠️ Report is 19 commits behind head on main.

Files with missing lines Patch % Lines
lib/internal/test_runner/runner.js 96.92% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #66273   +/-   ##
=======================================
  Coverage   90.36%   90.36%           
=======================================
  Files         790      790           
  Lines      274787   274813   +26     
  Branches    52631    52629    -2     
=======================================
+ Hits       248323   248347   +24     
- Misses      16932    16942   +10     
+ Partials     9532     9524    -8     
Files with missing lines Coverage Δ
lib/internal/test_runner/runner.js 95.12% <96.92%> (-0.07%) ⬇️

... and 24 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@faizanu94

Copy link
Copy Markdown
Author

Thanks for the sharp review. You are right that a blanket catch could hide a real regression in the report protocol. I pushed a commit that validates the inner v8 header before deserializing, so stray stdout that only mimics a frame is rejected as output, while a genuine frame that fails to deserialize is left to surface instead of being swallowed. I also added a test for that second case. I agree that separating the report stream from user stdout is the cleanest long-term fix.

@faizanu94
faizanu94 force-pushed the test-runner-stdout-frame-guard branch 3 times, most recently from 8120db3 to 642807f Compare September 26, 2026 01:19

@inoway46 inoway46 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update! My previous concern is resolved. I left one remaining comment.

Also, #66307 has landed in main, so could you rebase onto the latest main to pick up the fix for the CI failure?

Comment on lines +217 to +220
const reported = await collectReported([
plausibleSizeFalseHeader,
...chunks,
]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this actually verify resync before drain()?

collectReported() calls fileTest.drain() after feeding all chunks, so this would still pass if the real frame stayed buffered until the child exited.

Could we assert before drain() instead, e.g.:

fileTest.parseMessage(plausibleSizeFalseHeader);
fileTest.parseMessage(chunks[0]);

assert.deepStrictEqual(reported.at(-1), reportedDiagnosticEvent);

If the inner-header check already tells us this is not a valid frame, perhaps we could consume one byte as stdout and immediately restart the header search instead of breaking.

@faizanu94
faizanu94 force-pushed the test-runner-stdout-frame-guard branch from 642807f to 2899307 Compare September 26, 2026 09:31
@faizanu94

Copy link
Copy Markdown
Author

Thanks for the review. I have addressed the remaining comment.

Recovery is now live inside #processRawBuffer. When the payload does not start with the inner v8 header it emits one byte as stdout and restarts the header search right away instead of breaking, so a real frame that follows a stray one is reported immediately.

The test now asserts before drain(). It feeds the false frame then a real message and never calls drain(). It then checks that the diagnostic is already reported. One small note: the reporter is a stream, so I flush it with end() and finished() before asserting rather than reading it synchronously. That keeps the assertion deterministic.

I also rebased onto the latest main to pick up #66307.

@faizanu94
faizanu94 requested a review from inoway46 September 26, 2026 09:35
@inoway46

Copy link
Copy Markdown
Contributor

Thanks for the update. The behavior looks better now, but #processRawBuffer() has become difficult to reason about with the nested loops and mimicsFrame state.

Could we simplify it into a single loop that re-checks the buffer head after each step?

  1. If the buffer does not start with a frame header, emit stdout and continue.
  2. If the frame is incomplete, stop and wait for more data.
  3. If it only mimics a frame, consume enough to make progress and continue.
  4. Otherwise deserialize the real frame and continue from the remaining buffer.

I think this would make the resync behavior and edge cases much easier to reason about.

The child test process sends framed report messages and raw user stdout
over one pipe, using the bytes FF 0F to mark the start of a frame. User
output can contain those same bytes, so #processRawBuffer could read a
plausible size from stray stdout and hand the bytes to the v8
deserializer. The deserializer then threw. Because the call had no error
handling, the exception aborted the whole test run.

Read the frame before advancing the buffer and wrap the deserialize in a
try/catch. When the read fails, leave the buffer untouched and stop
parsing frames so #drainRawBuffer emits the stray byte as stdout and
rescans for the next real header. This turns a fatal crash into
recoverable stdout and preserves any real frames that follow the stray
bytes.

Fixes: nodejs#66164
Signed-off-by: Muhammad Faizan Uddin <faizan.uddin94@gmail.com>
Check that a framed payload starts with the inner v8 header before
handing it to the deserializer, so stray stdout that mimics a frame is
rejected as output while genuine deserialize failures still surface.

Signed-off-by: Muhammad Faizan Uddin <faizan.uddin94@gmail.com>
Recovery from stray stdout that mimics a frame happened only at
shutdown, inside #drainRawBuffer, so a real frame that followed the
stray bytes was reported late. Move the recovery into #processRawBuffer.
When the payload does not start with the inner v8 header, emit one byte
as stdout and restart the header search right away, so the next real
frame is reported live. Assert before drain() in the test so it verifies
live recovery instead of recovery at shutdown.

Signed-off-by: Muhammad Faizan Uddin <faizan.uddin94@gmail.com>
#processRawBuffer used nested loops and a mimicsFrame flag, which made
the resync path hard to follow. Rewrite it as a single loop that
inspects the head of the buffer once per pass and handles one case:
leading stdout, an incomplete frame, a frame that only mimics the
header, or a real frame. The existing tests cover each case.

Signed-off-by: Muhammad Faizan Uddin <faizan.uddin94@gmail.com>
@faizanu94
faizanu94 force-pushed the test-runner-stdout-frame-guard branch from 2899307 to d249668 Compare September 26, 2026 20:40
@faizanu94

Copy link
Copy Markdown
Author

Thanks, that reads much better. I rewrote #processRawBuffer as a single loop that re-checks the head on each pass and handles one case at a time: leading stdout, an incomplete frame, a frame that only mimics the header, or a real frame. The nested loops and the mimicsFrame flag are gone. The existing tests cover each case.

@MoLow

MoLow commented Sep 27, 2026 •

Copy link
Copy Markdown
Member

this might break every time we upgrade v8.
also, Id expect to see some regression tests

@inoway46 inoway46 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing. LGTM!

@inoway46

Copy link
Copy Markdown
Contributor

@faizanu94 It looks like the PR description is a bit outdated. Could you update it to reflect the current implementation?

@inoway46 inoway46 added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Sep 27, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Sep 27, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@panva panva added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Sep 27, 2026
@faizanu94

Copy link
Copy Markdown
Author

@inoway46 Done. I have updated the description to match the current implementation.

@inoway46 inoway46 added the resume-ci Add this label to resume the latest eligible Jenkins CI run on a PR with an approving review. label Sep 27, 2026
@github-actions github-actions Bot removed the resume-ci Add this label to resume the latest eligible Jenkins CI run on a PR with an approving review. label Sep 27, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@inoway46

Copy link
Copy Markdown
Contributor

@MoLow Would you mind landing this manually and updating the squashed commit message to reflect the current implementation? Rewriting the first commit now would require another approval and full CI run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author ready PRs with CI started, the required approvals, and no outstanding review comments. needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_runner: parent runner still crashes on child stdout bytes that mimic an event frame (survives the #64706 fix)

5 participants