Skip to content

Multipart DWAA encoding uses about 10x more peak memory in 3.3.0 than in 3.2.12 #2791

Closed

Description

@ultra-sonic

Multipart DWAA encoding uses about 10x more peak memory in 3.3.0 than in 3.2.12

Writing a synthetic 16-part EXR with DWAA compression peaks at about 6 GiB of resident memory with OpenEXR 3.3.0, compared with 604 MiB using 3.2.12. OpenEXR 3.3.2 gives almost the same result as 3.3.0. Reducing the thread count lowers memory use, but the difference remains at one thread.

The comparison uses the same C++ program linked directly to each OpenEXR build. Neither Houdini nor OpenImageIO is involved in these measured conversions.

Test setup

  • AlmaLinux 9.6, Linux x86_64, AMD EPYC 7713, 64 logical CPUs available, 235 GiB RAM.
  • OpenEXR release tags v3.2.12, v3.3.0, and v3.3.2, built from source with GCC 11.5.0.
  • C++17, CMake Release builds (-O3 -DNDEBUG), shared libraries, glibc allocator.
  • The same Imath 3.1.9 and libdeflate 1.18 builds for all three versions. Runtime library paths were checked with ldd.
  • Peak process RSS measured with /usr/bin/time -v, one run per configuration, with conversions run sequentially.

The attached C++ probe reads all input parts into interleaved HALF buffers, then opens one MultiPartOutputFile and writes each part sequentially with OutputPart::writePixels(). The pixel buffers and input file stay alive throughout the write. Only one OutputPart wrapper is in scope at a time; the multipart output file closes after all 16 parts have been written.

The requested thread count is passed to setGlobalThreadCount(), MultiPartInputFile, and MultiPartOutputFile. The probe changes the compression on the copied headers and leaves the DWA compression level at its default.

Download reproduction suite here

openexr-dwaa-reproducer.tar.gz

Synthetic source image

The source is one scanline multipart EXR containing 16 parts named layer000 through layer015. Each part is 2213 x 925 pixels with R, G, and B HALF channels, no subsampling, and a data window starting at (0, 0). The input uses ZIPS compression.

Every part contains the same vertical gradient, from RGB (1, 0.2, 0.1) at the top to (0.01, 0.3, 0.8) at the bottom. The raw HALF pixel data across all parts is about 187.4 MiB. There is no production image data in the reproducer.

The source was generated with OpenImageIO 2.5.17.0, built with OpenEXR 3.1.9, using this Bash script:

#!/usr/bin/env bash
set -euo pipefail
OIIO=${OIIO:-oiiotool}
args=(--threads 8)
for ((part=0; part<16; part++)); do
    printf -v name 'layer%03d' "$part"
    args+=(--pattern 'fill:top=1,0.2,0.1:bottom=0.01,0.3,0.8' 2213x925 3
           --attrib oiio:subimagename "$name")
done
"$OIIO" "${args[@]}" --siappendall -d half --compression zips \
    -o "${1:-synthetic-16parts.exr}"

The script in the bundle was checked against the original test input with idiff -a; the pixels match in all parts.

Results

Peak RSS, including both reading and writing:

| OpenEXR | DWAA, 64 threads | DWAA, 8 threads | DWAA, 1 thread | ZIPS, 64 threads |
|:--------|-----------------:|----------------:|---------------:|-----------------:|
| 3.2.12  |       603.61 MiB |      375.85 MiB |     220.32 MiB |       315.90 MiB |
| 3.3.0   |      6078.17 MiB |     3420.87 MiB |     599.78 MiB |       488.75 MiB |
| 3.3.2   |      6079.36 MiB |     3420.67 MiB |     600.05 MiB |       618.80 MiB |

At 64 threads, DWAA peak memory is about 10 times higher in 3.3.0 than in 3.2.12. ZIPS shows a smaller increase.

The probe also records peak RSS after reading and after writing each part. In the 3.3.0 DWAA run with 64 threads, it is about 195 MiB after reading, 573 MiB after writing the first part, and 6078 MiB after writing all 16 parts. Most of the increase occurs during writing, with the peak rising across successive parts. These measurements do not establish a memory leak.

All 12 conversions exited successfully. Each output was decoded with oiiotool -a --info -v --stats; all 16 parts had the expected compression and finite pixel values.

Reproduction

The attached bundle contains the input EXR, generator, C++ probe, CMake file, and measurement logs. With OpenEXR 3.3.0 and its dependencies installed in separate prefixes:

cmake -S . -B build-3.3.0 \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_PREFIX_PATH="/path/to/openexr-3.3.0;/path/to/deps"
cmake --build build-3.3.0

/usr/bin/time -v build-3.3.0/exr_memory_probe \
    synthetic-16parts.exr out-dwaa.exr 64 dwaa

Repeat with the 3.2.12 and 3.3.2 prefixes. Change 64 to 8 or 1 for the other DWAA runs, or change dwaa to zips for the control.

A separate test with the official OIIO 3.2.2.0 Linux wheel, which reports OpenEXR 3.5.2, also showed high DWAA memory use on this input: 6.54 GiB at 64 threads. That was an OIIO conversion, so it is not directly comparable to the C++ measurements above. I have not yet run the C++ probe against 3.5.2.

The increase is already present in the first 3.3 release, which moved the C++ API onto OpenEXRCore. I have not isolated the responsible commit. Is this expected memory use for multipart DWAA output, or a regression in how compression buffers are allocated or retained between parts?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions