publish: upload large files in parts, retrying each part

[smypmsa]

publish sent every file as one PutObject. A dataset_referenced release uploads its parquet beside the page, and on a project with about 4M event rows that is five files totalling 263 MiB, the largest 160 MiB. On a connection that drops a TLS record now and then, a single long upload fails as a whole (ssl3_read_bytes: bad record mac), and publish fails with it: in two runs every small file went up and not one parquet did.

What changes

• Large bodies go up in parts. Above 16 MiB, put runs a multipart upload: create, then one 16 MiB part at a time, then complete with the parts' ETags in order. One part at a time, because the body is already in memory and parallel parts would only put more of it on a connection that is dropping records. Every part but the last is the same size, which R2 requires.
• Each part is retried on its own. Up to four attempts with a short backoff, so a dropped record costs one part, not the whole file.
• A part that keeps failing aborts the upload. An upload left open keeps its parts in the bucket. The error is transient_dependency, marked retryable, and names the key.
• Conditional writes stay a single PutObject. latest.json is written with If-Match / If-None-Match, which a multipart upload cannot carry; it is small anyway.
• makeS3Ops takes an optional client and tuning so the upload path can be tested without the network. No new dependency: the multipart commands come from @aws-sdk/client-s3, already in use.

Testing

• pnpm test: 386 passed; the 3 live tests skipped.
• New tests/publish/s3multipart.test.ts (5): a small body is one PutObject; a large one splits into ordered parts of the right sizes and completes with their ETags; a failing part is retried without restarting the upload; a part that keeps failing aborts with a retryable error; a conditional write stays one PutObject whatever its size.
• Against Cloudflare R2, on the connection that failed before: three publishes of the 263 MiB release above, each with every parquet uploaded and verified (about 1 min 50 s). A fork from the public URL then rebuilt offline to the same 19 query results.

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 37e7730f-5755-4a24-9a37-839bfc760e90

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}