Repository navigation
Add pineforge-feed: native Binance spot feed adapter (slice 1) - #17
Merged
Merged
Conversation
Hatchling's sdist took every tracked file, so the next Python release would have shipped native/ inside the Python package. Exclude it from the sdist and fail CI if a built sdist lists anything under native/. The wheel is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Align actions/checkout with the other workflows (v7), add ubuntu-24.04-arm to the Release matrix, install cmake on Linux runners, and give each job a 30-minute timeout. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The repository's `ruff check .` covers native/feed/tests. Format both scripts and fix the lint findings: open the harness's log files as descriptors, use itertools.pairwise, and drop an unused import and unused names. No behaviour changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A tick fill carries its print's time while the OHLC batch carries the modeled segment's clock, so the strict tick comparison could not pass. Apply the predeclared R-B2 mapping (floor(ts / script_tf) * script_tf on both sides, tick mode only), keep every other field exact, fail the run on any difference, and retire --allow-tick-ohlc-difference. The minimum live duration now follows --bar-minutes and --tick-minutes, and the tick replay gets 600 seconds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Group commit: verify every event made ready by the source messages already queued (up to 1024 messages or 4096 events), append them in one write, fsync the journal once, replace the cursor once, then publish the lines. One event per line and persist-before-publish are unchanged; a stop publishes what was already verified and a failed commit leaves the journal unusable until resume truncates its tail. - A tick minute whose own x=true kline was missed reconciles the REST kline with reconcile(kline, false) behind an explicit next-minute fence check; no kline field is filled in. - Bars mode subscribes to kline_1m only. An unknown event outside the data streams is a structured warning; the data streams stay strict (23). - REST is spaced at half of each published ceiling and pauses when the used- weight header passes half of the 1-minute limit. Retry-After accepts delta-seconds and an IMF-fixdate HTTP-date. The REST body cap matches the 1 MiB JSON parser limit. - The initial raw-trade fence lookup retries when REST lags the WebSocket. - Warmup compares its REST rows with the x=true bars seen on the WebSocket. - A matched-time regression stops with 23 instead of the gap code. - stdout flags are restored on exit; stderr is left alone and written by one thread in whole records, so a full sink drops records instead of cutting them. Signal handlers are installed first. - A fresh start takes over a directory whose only file is an empty journal. - Remove the unused reconnect hook and string_field. - Tests: tick reconnect overlap change (unit + mock), Retry-After seconds and HTTP-date, multi-page healing, matched-time regression, HTTP 418 and 403, unknown and malformed events, lagging REST fence, warmup cross-check, stdio flags and a full stderr pipe, group commit, crashed initialization; the failed cursor write now asserts that nothing was published. - README: group commit and measured throughput, run-time horizon per mode, the historical closure rule, tick mode's liquid-symbol limit, and the expected restart policy. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The batch replays warmup plus the minutes the tape proved complete. In tick mode the runner also sees the prints of the minute after the last `time` event and may act on them; those actions have no batch counterpart. Compare the actions on proven bars and report how many fell after them; the same-print replay still checks every runner action. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
pineforge-feed, a native C++17 adapter that turns an exchange's public market data into the PineForge feed protocol that the engine'spineforge-liverunner reads. This is the first slice: Binance spot, confirmed 1-minute bars and trade ticks. It lives innative/feed/, outside the Python package: the wheel is byte-identical, and the sdist excludesnative/.seqis the venue's contiguous trade id;time(completeness) event is emitted only when the closed kline's trade-id range and OHLCV reconcile exactly;Anything it cannot prove stops the feed with a distinct exit code (20 gap, 21 changed overlap or revision, 22 budget, 23 protocol or access), and a resumable cursor is kept.
--resume --output-from Npairs with the runner's--from-input N. Writes are group-committed, measured at more than 5,000 messages per second on a 16-vCPU box.warmup(REST 1m history to CSV with a manifest) andrun(feed events on stdout). Theservefan-out is a later slice.Verification
pineforge-live:Not in this slice: OKX, Bybit, Binance USD-M aggregate trades,
serve, and journal segmentation for long tick runs.🤖 Generated with Claude Code