Skip to content

Reduce write memory - #2809

Merged
cary-ilm merged 4 commits into
AcademySoftwareFoundation:mainfrom
kdt3rd:reduce_write_memory
Nov 3, 2026
Merged

cary-ilm merged 4 commits into
AcademySoftwareFoundation:mainfrom
kdt3rd:reduce_write_memory

Conversation

@kdt3rd

@kdt3rd kdt3rd commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Reduce accumulation of temporary buffers when writing multiple parts - after one part finished, it did not clean up temporary line buffers used during writing.

Further, in dwaa, only allocate huf table scratch buffer when needed

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>
Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>
Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>
@kdt3rd

kdt3rd commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

This is complementary to #2794 but should address #2791 for all compression types. There is more possible in future versions

@vlazar-ilm

Copy link
Copy Markdown
Contributor

Dunno how popular they are in the wild, but Tiled files suffer from the same memory accumulation issue I think.

@cary-ilm cary-ilm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cary-ilm
cary-ilm merged commit 330ecde into AcademySoftwareFoundation:main Nov 3, 2026
39 of 40 checks passed
cary-ilm added a commit that referenced this pull request Nov 5, 2026
* once finished writing, clean up lineBuffer

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>

* Fix memory tag on compressed buffer allocation

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>

* move scratch buffer alloc check to only when needed

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>

---------

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>
Co-authored-by: Cary Phillips <seabeepea@gmail.com>
cary-ilm added a commit that referenced this pull request Nov 5, 2026
* once finished writing, clean up lineBuffer

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>

* Fix memory tag on compressed buffer allocation

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>

* move scratch buffer alloc check to only when needed

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>

---------

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>
Co-authored-by: Cary Phillips <seabeepea@gmail.com>
cary-ilm added a commit that referenced this pull request Nov 5, 2026
* once finished writing, clean up lineBuffer

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>

* Fix memory tag on compressed buffer allocation

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>

* move scratch buffer alloc check to only when needed

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>

---------

Signed-off-by: Kimball Thurston <kdt3rd@gmail.com>
Co-authored-by: Cary Phillips <seabeepea@gmail.com>
@cary-ilm cary-ilm added v3.5.4 v3.4.18 Cherry-picked into the 3.4.18 release v3.3.16 Cherry-picked into the 3.3.16 release and removed v3.5 v3.4 v3.3 labels Nov 5, 2026
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Nov 15, 2026
## Version 3.5.4 (November 11, 2026)

Patch release that addresses several build and security issues, and
also fixes a regression introduced in v3.3.0 that leads to excessive
memory usage during write, especially with DWA compression (Issue
[#2791](AcademySoftwareFoundation/openexr#2791)).

* Fix a configuration issue with OpenJPH 0.33.0 that caused it to
  silently and incorrectly fall back to the vendored version (PR
  [#2813](AcademySoftwareFoundation/openexr#2813))
* The vendored OpenJPH is now built in a custom namespace to avoid
  symbol collision with other versions when built statically ([2811](AcademySoftwareFoundation/openexr#2811))
  built statically,
* The vendored OpenJPH version is now 0.33.0.
* Fix `exrmultipart -convert` on tiled and deep single-part files
* `OPENEXR_USE_INTERNAL_IMATH/OPENJPH` are are now included in
  `OpenEXRConfig.h` alongside `OPENEXR_USE_INTERNAL_DEFLATE/ZSTD`.
* 32-bit glibc builds now guarantee support for files of size 2GiB or
  more ([2812](AcademySoftwareFoundation/openexr#2812))

This release addresses the following security vulnerabilities:

* ***CVE Pending*** ([GHSA-pp92-4rv3-wfq6](GHSA-pp92-4rv3-wfq6)) OpenEXR: a signed-int data-window-origin overflow in the OpenEXRUtil Image API turns `Imf::loadImage` into an attacker-steered out-of-bounds write
* ***CVE Pending*** ([GHSA-2v77-pvhx-25c3](GHSA-2v77-pvhx-25c3)) Integer overflow in legacy OpenEXR.OutputFile.writePixels() causes heap over-read

### Merged Pull Requests

* [2828](AcademySoftwareFoundation/openexr#2828)
  Add `OPENEXR_USE_INTERNAL_IMATH/OPENJPH` to BUILD.bazel (by @cary-ilm/Cary Phillips)
* [2826](AcademySoftwareFoundation/openexr#2826)
  Find Threads in installed config before OpenJPH (by @cary-ilm/Cary Phillips)
* [2824](AcademySoftwareFoundation/openexr#2824)
  Docs: make `maxSamplesPerPixel` optional for deep parts (by @cary-ilm/Cary Phillips)
* [2823](AcademySoftwareFoundation/openexr#2823)
  Fix `exrmultipart -convert` on tiled and deep single-part files (by @cary-ilm/Cary Phillips)
* [2818](AcademySoftwareFoundation/openexr#2818)
  Remove `test_write_overflow` (by @cary-ilm/Cary Phillips)
* [2817](AcademySoftwareFoundation/openexr#2817)
  Fix integer overflow in legacy Python `OutputFile.writePixels()` (by @cary-ilm/Cary Phillips)
* [2816](AcademySoftwareFoundation/openexr#2816)
  Fix integer overflow in OpenEXRUtil Image channel base pointers (by @cary-ilm/Cary Phillips)
* [2814](AcademySoftwareFoundation/openexr#2814)
  CI: validate that the build uses the dependencies the job installed (by @cary-ilm/Cary Phillips)
* [2813](AcademySoftwareFoundation/openexr#2813)
  Accept a newer OpenJPH, and warn when an installed one is too old (by @cary-ilm/Cary Phillips)
* [2812](AcademySoftwareFoundation/openexr#2812)
  Define `_FILE_OFFSET_BITS=64` for OpenEXRCore explicitly (by @cary-ilm/Cary Phillips)
* [2811](AcademySoftwareFoundation/openexr#2811)
  Rename vendored OpenJPH namespace, and hide zstd visibility (by @cary-ilm/Cary Phillips)
* [2809](AcademySoftwareFoundation/openexr#2809)
  Reduce write memory (by @kdt3rd/Kimball Thurston)
* [2808](AcademySoftwareFoundation/openexr#2808)
  Bump OpenJPH version to 0.33.0 (by @cary-ilm/Cary Phillips)
* [2802](AcademySoftwareFoundation/openexr#2802)
  Use unaligned stores in the flat unpack fast paths (by @Ar-maan05/Armaan Sandhu)
* [2798](AcademySoftwareFoundation/openexr#2798)
  Fix heap overflow in legacy Python `InputFile.channel()` with ySampling (by @Dilshad-Tanvi)
* [2796](AcademySoftwareFoundation/openexr#2796)
  exrenvmap test: shrink output size and skip blur under QEMU emulation (by @cary-ilm/Cary Phillips)
* [2794](AcademySoftwareFoundation/openexr#2794)
  Only zero the DWA chunk header instead of the full output buffer (by @vlazar-ilm/Vlad Lazar)
* [2792](AcademySoftwareFoundation/openexr#2792)
  Fix `Python File.write()` over-reads for subsampled and RGB pixel arrays (by @Dilshad-Tanvi)
* [2758](AcademySoftwareFoundation/openexr#2758)
  Fix tile chunk extents at reduced-resolution edges (by @vera-rubin/Nathan Chandrasekar)
* [2734](AcademySoftwareFoundation/openexr#2734)
  Fix B44/B44A raw fallback when compressed size equals raw size (by @vera-rubin/Nathan Chandrasekar)

### Merged Workflow Pull Requests

* [2835](AcademySoftwareFoundation/openexr#2835)
  release-notice: add workflow_dispatch to send announcements manually (by @cary-ilm/Cary Phillips)
* [2834](AcademySoftwareFoundation/openexr#2834)
  ci: add `openjph_sha` to `ci_full/ci_premerge` with: blocks (by @cary-ilm/Cary Phillips)
* [2832](AcademySoftwareFoundation/openexr#2832)
  Bump the actions-deps group across 1 directory with 2 updates (by @app/dependabot)
* [2827](AcademySoftwareFoundation/openexr#2827)
  Bump CI's default OpenJPH to 0.33.0 (by @cary-ilm/Cary Phillips)
* [2822](AcademySoftwareFoundation/openexr#2822)
  Bump the bazel-deps group with 2 updates (by @app/dependabot)
* [2799](AcademySoftwareFoundation/openexr#2799)
  Fix missing step id breaking Slack release notifications (by @cary-ilm/Cary Phillips)
* [2797](AcademySoftwareFoundation/openexr#2797)
  Bump the bazel-deps group with 3 updates (by @app/dependabot)
* [2793](AcademySoftwareFoundation/openexr#2793)
  Bump the actions-deps group across 1 directory with 2 updates (by @app/dependabot)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v3.3.16 Cherry-picked into the 3.3.16 release v3.4.18 Cherry-picked into the 3.4.18 release v3.5.4

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants