Skip to content

kvazaar: fix NULL chroma pointer dereference when encoding monochrome/alpha images - #1915

Closed
twn39 wants to merge 1 commit into
strukturag:masterfrom
twn39:fix/kvazaar-monochrome-p400-crash
Closed

twn39 wants to merge 1 commit into
strukturag:masterfrom
twn39:fix/kvazaar-monochrome-p400-crash

Conversation

@twn39

@twn39 twn39 commented Sep 21, 2026

Copy link
Copy Markdown

Summary

Fixes a crash (NULL pointer dereference / SIGSEGV / Windows Access Violation 0xC0000005) when encoding monochrome images (heif_colorspace_monochrome) or images with an Alpha channel using the Kvazaar HEVC encoder.

Root Cause

In kvazaar_start_sequence_encoding_intern() (libheif/plugins/encoder_kvazaar.cc):

  • When encoding a monochrome image (such as an auxiliary alpha image), the code set:
    config->input_format = KVZ_FORMAT_P400;
    However, kvz_config_init() initializes config->chroma_format to KVZ_CSP_420 by default.
  • Kvazaar's internal state expects config->chroma_format to be updated (or kvz_config_parse(config, "input-format", "P400") to be invoked), which sets chroma_format = KVZ_CSP_400 and chroma_shift_w = 0, chroma_shift_h = 0.
  • Because chroma_format was never changed from KVZ_CSP_420, Kvazaar's encoder was created expecting 4:2:0 input.
  • Later, in kvazaar_encode_sequence_frame(), the picture was allocated via api->picture_alloc_csp(KVZ_CSP_400, ...). In Kvazaar's videoframe.c, KVZ_CSP_400 does not allocate U and V chroma planes, setting pic->u = NULL; pic->v = NULL;.
  • When api->encoder_encode() ran, Kvazaar's intra mode search (search_intra.c: search_intra_chroma_rough) checked state->encoder_control->cfg.chroma_format != KVZ_CSP_400. Because cfg.chroma_format was still KVZ_CSP_420, it attempted to access pic->u and pic->v, resulting in an immediate crash dereferencing NULL.

Solution

  1. In kvazaar_start_sequence_encoding_intern(), explicitly set config->chroma_format and corresponding chroma shift parameters.
  2. Call api->config_parse(config, "input-format", format_str) to ensure Kvazaar's internal configuration state is fully synchronized.
  3. Added a regression test kvazaar encodes monochrome image in tests/encode.cc.

…/alpha images

When encoding monochrome (P400) or alpha images with Kvazaar, kvazaar_start_sequence_encoding_intern() only set config->input_format = KVZ_FORMAT_P400 without updating config->chroma_format or calling api->config_parse(). Because kvz_config_init() defaults chroma_format to KVZ_CSP_420, Kvazaar's encoder internal state retained 4:2:0.

Meanwhile, kvazaar_encode_sequence_frame() allocated the picture buffer with KVZ_CSP_400, leaving pic->u and pic->v as NULL. During encoding, Kvazaar's intra search checked state->encoder_control->cfg.chroma_format != KVZ_CSP_400 and dereferenced the NULL chroma planes, causing a SIGSEGV / access violation (0xC0000005).

Fix this by setting config->chroma_format and calling api->config_parse(config, "input-format", format_str) for all supported chroma formats. Also add a regression unit test.
farindk added a commit that referenced this pull request Sep 21, 2026
kvazaar git master (cbdcc31a, 2026-07-27) keeps the encoder chroma format
in new kvz_config fields that default to 4:2:0 and are only updated by the
"input-format" parser. Writing config->input_format directly leaves them
at 4:2:0, so every monochrome encode, including every alpha auxiliary
image, dereferences the NULL chroma planes of the 4:0:0 input picture.
Released kvazaar versions derive the chroma format from input_format and
are not affected.

Set the input format through api->config_parse(), which is how kvazaar's
own CLI configures the encoder and works on every version. Chroma formats
a kvazaar build does not support are rejected by the parser and now return
an error instead of running the encoder in a mode it was not built for.

Reported by @twn39 in #1915, whose regression test this includes.
@farindk

farindk commented Sep 21, 2026

Copy link
Copy Markdown
Member

Thanks for the report and the analysis. The crash is real, but only against kvazaar git master since July 2026 (commit cbdcc31a moved chroma_format into kvz_config). The fields your patch writes do not exist in any released kvazaar, so the PR does not compile against 2.3.x. The portable fix is to configure the input format through config_parse("input-format", ...), which is what kvazaar's own CLI does and which works on every version. I've applied that variant together with your test in d3772c0.

@farindk farindk closed this Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants