[mjackson]

storage.set() in the S3 backend reads the whole file with arrayBuffer() before it sends a single PUT, so storing a 500 MB file needs 500 MB of RAM even when the file is on disk or is being streamed from somewhere else. This PR streams files to S3 instead.

import { createS3FileStorage } from 'remix/file-storage/s3'

let storage = createS3FileStorage({
  accessKeyId,
  secretAccessKey,
  bucket,
  region,
  partSize: 16 * 1024 * 1024, // optional, defaults to 8 MiB
})

await storage.set('videos/intro.mp4', file) // sent in parts, without buffering the whole file

• Multipart uploads for large files: set() reads the file in fixed-size parts.
  • Blobs and Files are read with slice().arrayBuffer(). Blob#stream() on Node.js before 24.20.0 and 26.6.0 keeps every chunk in memory for the life of the process (nodejs/node#63574), so it would leak every stored file. A comment notes that this workaround can be removed once the minimum supported Node.js version is >=24.20.0.
  • Other File-shaped inputs are read with stream(). Only stream(), name, type and lastModified are used, so this also works with streaming sources whose size isn't known until they've been read.
  • Files that fit in one part still go up with a single PUT.
  • Larger files use CreateMultipartUpload, UploadPart and CompleteMultipartUpload. Each part uploads while the next one is read, so at most two parts are in memory.
  • Every part except the last is exactly partSize bytes, which providers like Cloudflare R2 require.
• New partSize option: defaults to 8 MiB, which supports files up to about 78 GiB, and must be at least 5 MiB, the S3 minimum.
• Failure handling:
  • If a part upload fails or the source stream errors, the multipart upload is aborted and the source stream is canceled.
  • A completion that S3 reports as an error inside a 200 response is treated as a failure.
  • The README recommends a lifecycle rule for incomplete uploads, in case a process exits before it can abort one.
• put() uses the same upload path. It still holds the whole file in memory because it returns a File with the stored contents, and the docs point large files to set().

Peak memory growth when storing one file read from a stream with set(), using a fetch that discards request bodies (Node.js 24.18):

File size	Before	After
100 MiB	195 MiB	149 MiB
500 MiB	1,004 MiB	183 MiB

Before, memory grows with the file. After, it stays flat, and what's left is garbage-collection lag on discarded part copies. Storing ten 20 MiB in-memory Files leaves no memory retained after garbage collection.

The unit tests cover part splitting, reading Blobs without stream(), the single-PUT cutoff, how far reading runs ahead of uploading, abort on part and stream failures, and failed completions. A new integration test runs a real multipart upload against LocalStack in CI.

This is independent of #12029 and #12030. Once #12030 lands, streaming uploads from remix/form-data-parser/future can go straight from the request to S3 through set().

[github-actions]

Preview Build Available

A preview build has been created for this PR. You can install it using:

pnpm install "remix-run/remix#preview/pr-12070&path:packages/remix"

This preview build will be updated automatically as you push new commits.

[github-actions]

The preview branch preview/pr-12070 has been deleted now that this PR is merged/closed.