Skip to content

Make the sink-base fault-mix test clock-driven - #1085

Merged
dahlia merged 1 commit into
fedify-dev:mainfrom
seongmin36:fix/flaky-failure-runner-test
Sep 27, 2026
Merged

dahlia merged 1 commit into
fedify-dev:mainfrom
seongmin36:fix/flaky-failure-runner-test

Conversation

@seongmin36

Copy link
Copy Markdown
Contributor

Closes #951.

failureRunner - shares sink base across remote fault mix ran a closed-loop load (concurrency: 1) for a real 25 ms and asserted requests.total > 1.
runClosedLoop yields between dispatches and never calls clock.sleepUntil(), so the assertion's actual guarantee was "the host completed a second async round trip within 25 real milliseconds" — which failed once under CI scheduling load in #949's Node.js job.

This switches the scenario to a rate-based open loop (load: { rate: 100 }) and injects a Clock via RunContext.clock - the same mechanism fanoutRunner's own tests already rely on (fanout.test.ts:362-374).
scheduleArrivals() (load/arrival.ts) computes the arrival count purely from rate and duration before the loop ever runs, so with rate: 100 and duration: "25ms" the three dispatches at t=0/10/20 are fixed by arithmetic, not by how fast the host executes.

runOpenLoop calls clock.sleepUntil() per arrival, and each remote fault's drain wait (waitForRemoteFault) also polls through the same injected clock, so nothing on this path touches real wall-clock time anymore.

Checks

  • node --test --experimental-transform-types packages/cli/src/bench/scenarios/failure.test.ts — 17/17 pass.
  • Ran the target test alone 100 times in a loop (--test-name-pattern='failureRunner - shares sink base across remote fault mix') — 100/100 pass.
  • Confirmed measurement.requests.total === triggerCalls === 3 and both /inbox/0//inbox/1 still show up, so the sink-base-sharing behavior the test exists to check is unchanged.

No changelog fragment, since this is test-only work; the commit carries Changelog: none.

AI disclosure

Per AI_POLICY.md: Claude Code (claude-sonnet-5) helped trace the flakiness to runClosedLoop never calling clock.sleepUntil(), proposed switching to the open-loop rate + injected-Clock pattern already used elsewhere in this file, and ran the checks full suite once (the first check below).

I reviewed the reasoning against the actual source before accepting it, adjusted the final assertions myself, and independently re-ran the isolated test 100 times in my own terminal to confirm it no longer flakes.

The commit carries the Assisted-by: Claude Code:claude-sonnet-5 trailer.

Replace the closed-loop concurrency load with a rate-based open loop
and an injected Clock, so the sink-base fault-mix test's dispatch
count no longer depends on completing two requests inside a real
25ms window. scheduleArrivals() fixes the arrival count (3) up front
from rate/duration alone, and both the load generator and each
remote-fault drain poll consume the injected clock instead of
wall-clock time.

fedify-dev#951

Changelog: none
Assisted-by: Claude Code:claude-sonnet-5
@netlify

netlify Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for fedify-json-schema canceled.

Name Link
🔨 Latest commit c25e6b1
🔍 Latest deploy log https://app.netlify.com/projects/fedify-json-schema/deploys/6ab91dbce9811200082e44c5

@coderabbitai

coderabbitai Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 930cfe3c-85e7-40b8-a9b6-3b1c1378c288

📥 Commits

Reviewing files that changed from the base of the PR and between 140b94d and c25e6b1.

📒 Files selected for processing (1)
  • packages/cli/src/bench/scenarios/failure.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.


📝 Walkthrough

Walkthrough

The remote fault mix test now uses rate-based load and simulated time. It asserts that the measured request count equals the three trigger calls.

Changes

Remote fault mix test

Layer / File(s) Summary
Control test timing and assert request count
packages/cli/src/bench/scenarios/failure.test.ts
The test uses rate-based load at 100 requests per second and a clock that advances simulated time during sleeps. It passes the clock to the runner and asserts exactly three measured requests, matching the trigger-call count. The imports are reordered.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other · Severity of issue fixed: Low

Merge Risk: ⚪ Minimal · up to c25e6

This test-only change makes dispatch timing deterministic while retaining checks for successful requests and both inboxes. No merge-blocking risk is evident.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: making the sink-base fault-mix test clock-driven. It is concise and specific.
Description check ✅ Passed The description directly explains the test flakiness, the switch to rate-based open-loop execution, clock injection, and the reported validation results. It is fully related to the changeset.
Linked Issues check ✅ Passed The change satisfies #951. The test replaces closed-loop concurrency with rate-based load, injects a deterministic Clock, and asserts exactly three measured requests. It also asserts that the reques…
Out of Scope Changes check ✅ Passed The changes stay within #951. They modify only the targeted remote fault-mix test and reorder imports. The new assertions and clock setup directly support deterministic dispatch and the issue requirem…
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
✨ Finishing Touches 💡 1
🧪 Generate unit tests (beta)
  • Create a new PR
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@dahlia dahlia self-assigned this Sep 27, 2026
@dahlia dahlia added the component/ci CI/CD workflows and GitHub Actions label Sep 27, 2026
@dahlia dahlia added this to the Fedify 2.4 milestone Sep 27, 2026
@codecov

codecov Bot commented Sep 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.
see 38 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dahlia dahlia left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job, thanks!

@dahlia
dahlia merged commit 1756e19 into fedify-dev:main Sep 27, 2026
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/ci CI/CD workflows and GitHub Actions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make the failure runner's remote fault mix test independent of wall-clock timing

2 participants