Repository navigation
ZSTD_freeCCtx() frees a dictionary loaded with ZSTD_CCtx_loadDictionary() before stopping the worker threads #4862
Description
Describe the bug
ZSTD_freeCCtxContent() calls ZSTD_clearAllDicts() before ZSTDMT_freeCCtx(). With nbWorkers >= 1 and a dictionary loaded with ZSTD_CCtx_loadDictionary() (also ZSTD_CCtx_loadDictionary_byReference()), the first job of a frame holds a pointer to cctx->localDict.cdict. If the context is freed while that job is still queued or running, the calling thread frees the local CDict and the copied dictionary buffer, and only then joins the pool; the worker reads freed memory.
| v1.5.7 | dev (49cf517) | |
|---|---|---|
ZSTD_freeCCtxContent() calls ZSTD_clearAllDicts() |
lib/compress/zstd_compress.c:174 (frees at :156-157) |
:187 (frees at :169-170) |
then ZSTDMT_freeCCtx() |
zstd_compress.c:176, POOL_free() at zstdmt_compress.c:1048 |
:189, zstdmt_compress.c:1050 |
job 0 is given mtctx->cdict |
zstdmt_compress.c:1425 |
:1426 |
| and uses it | zstdmt_compress.c:729 |
:729 |
From reading the code, a context that uses ZSTD_CCtx_refCDict() or ZSTD_CCtx_refPrefix() is not affected by the free (the caller owns the CDict; the prefix's mtctx->cdictLocal is freed after the join); these were not run under AddressSanitizer.
To Reproduce (expand to see demo C code)
/* cc -O1 -g repro.c -o repro -lzstd */
#include <stdio.h>
#include <stdlib.h>
#include <zstd.h>
#define CHECK(e) do { size_t const r_ = (e); if (ZSTD_isError(r_)) { \
fprintf(stderr, "%s: %s\n", #e, ZSTD_getErrorName(r_)); return 1; } } while (0)
int main(void)
{
size_t const dictSize = 1 << 20; /* raw content dictionary */
size_t const srcSize = (1 << 20) + (64 << 10); /* a bit more than one job */
size_t const dstCapacity = ZSTD_compressBound(srcSize);
unsigned char *dict = malloc(dictSize), *src = malloc(srcSize);
unsigned char *dst = malloc(dstCapacity);
unsigned char block[1024];
unsigned seed = 1;
size_t i;
if (!dict || !src || !dst) return 1;
for (i = 0; i < sizeof(block); i++) {
seed = seed * 1103515245u + 12345u;
block[i] = (unsigned char)(seed >> 16);
}
for (i = 0; i < dictSize; i++) dict[i] = block[i % sizeof(block)];
for (i = 0; i < srcSize; i++) src[i] = block[(i + 7) % sizeof(block)];
ZSTD_CCtx *cctx = ZSTD_createCCtx();
if (!cctx) return 1;
CHECK(ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 10));
CHECK(ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, 1));
CHECK(ZSTD_CCtx_setParameter(cctx, ZSTD_c_jobSize, 1 << 20));
CHECK(ZSTD_CCtx_loadDictionary(cctx, dict, dictSize));
ZSTD_inBuffer in = { src, srcSize, 0 };
ZSTD_outBuffer out = { dst, dstCapacity, 0 };
/* Consumes all input and returns while a worker runs the first job. */
CHECK(ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_continue));
printf("consumed %zu of %zu, produced %zu\n", in.pos, in.size, out.pos);
fflush(stdout);
ZSTD_freeCCtx(cctx);
free(dict);
free(src);
free(dst);
puts("ok");
return 0;
}Linux x86-64, Ubuntu 26.04
- Against the distribution's libzstd 1.5.7 (
1.5.7+dfsg-3),cc -O1 -g repro.c -o repro -lzstd: printsconsumed 1048576 of 1114112, produce d 0, then segmentation fault in the worker thread, 10 of 10 runs. - With AddressSanitizer and the library sources compiled in, where
zstd/is the source tree of tag v1.5.7, 10 of 10 runs it crashes with a heap-use-after-free. - The same command with
zstd/atdevcommit 49cf517: same report, 10 of 10 runs (SUMMARY: AddressSanitiz er: heap-use-after-free zstd/lib/compress/zstd_compress.c:2385 in ZSTD_resetCCtx_byAttachingCDict, freed fromZSTD_clearAllDictszstd_compre
ss.c:170 <-ZSTD_freeCCtxContent:187 <-ZSTD_freeCCtx:201).
Conditions: nbWorkers >= 1, a dictionary loaded with ZSTD_CCtx_loadDictionary() or ZSTD_CCtx_loadDictionary_byReference(), and the contex
t freed while the first job of a frame is pending. Without AddressSanitizer the crash depends on the allocator and on timing.
Expected behavior
Freeing a context with unfinished multi-threaded work is safe, as it is without a dictionary or with ZSTD_CCtx_refCDict().
zstd.h gives ZSTD_freeCCtx() no precondition about pending work, documents abandoning a frame as supported (ZSTD_CCtx_reset(): the session reset is "Useful after an error, or to interrupt any ongoing compression"), and says the dictionary passed to ZSTD_CCtx_loadDictionary() "will be copied internally", so the freed CDict is owned by the context and the caller has no way to keep it alive.
Possible fix
Stop the workers before freeing the dictionaries:
static void ZSTD_freeCCtxContent(ZSTD_CCtx* cctx)
{
assert(cctx != NULL);
assert(cctx->staticSize == 0);
- ZSTD_clearAllDicts(cctx);
#ifdef ZSTD_MULTITHREAD
ZSTDMT_freeCCtx(cctx->mtctx); cctx->mtctx = NULL;
#endif
+ ZSTD_clearAllDicts(cctx);
ZSTD_cwksp_free(&cctx->workspace, cctx->customMem);
}This was only checked with the reproducer under AddressSanitizer on the v1.5.7 sources (clean, also with ZSTD_CCtx_loadDictionary_byReference( )), not with zstd's test suite, and not with ZSTD_CCtx_refThreadPool().
It does not cover the other callers of ZSTD_clearAllDicts(). With a job pending, replacing the ZSTD_freeCCtx() call in the reproducer by Z STD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters), or by ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only) followed by another ZSTD_CCtx _loadDictionary(), gives the same AddressSanitizer report on v1.5.7 and on dev, with or without the reordering above.
Activity
Shahinpar49 commentedon Oct 24, 2026
gh-160549: Fix crash when freeing a busy multi-threaded ZstdCompressor (
[3.14] gh-160549: Fix crash when freeing a busy multi-threaded ZstdCo…
[3.15] gh-160549: Fix crash when freeing a busy multi-threaded ZstdCo…