Repository navigation
Use thread local storage for frontend compile cache - #3280
Merged
Merged
Conversation
angeloskath
approved these changes
Mar 19, 2026
angeloskath
left a comment
angeloskath
left a comment
Member
There was a problem hiding this comment.
Looks great!
I wonder if people rely on compile being ran once. They shouldn't but this could be a fun bug to debug in the future.
@mx.compile
def fun(x):
call_my_run_once_and_only_once_side_effect_op()
return x**2 + 2 * x + 1
# thread 1
fun(mx.ones((10,))
# thread 2
fun(mx.ones((10,))) # oops
Member
Author
|
That would already be a problem since changing the default stream would invalidate the compile cache. In a multi-thread environment we expect users to use different streams as default streams so hopefully it would not be a problem. |
1 task done
3 of 4 tasks
rockiestar-com
pushed a commit
to RockieStar-Inc/mlx
that referenced
this pull request
Aug 25, 2026
(cherry picked from commit 70a0da6)
rockiestar-com
pushed a commit
to RockieStar-Inc/mlx
that referenced
this pull request
Aug 25, 2026
… exit; never skip handler bookkeeping N1: mlx/compile.cpp, mlx/compile_impl.h and python/src/transforms.cpp now match v0.32.0 exactly for the compile cache (CompilerCache::empty, compile_cache_empty, ensure_compile_cache_cleanup with its ThreadCleanup dtor, the atexit main-thread cleanup). The half-applied lambda that was never invoked is gone. N2: the pending-CPU-error throw path notes the error as reported, like the other three report paths. N3: the Scheduler singleton is leaked on every platform, not only Windows: a completion handler calls notify_task_completion, which locks its mutex, and at exit that mutex may already be destroyed. Scheduler threads are therefore no longer joined at exit; ScreenKite holds the inference gate at quit, so they are idle. N4: the commit completion handler runs its error bookkeeping, its handlers_done increment plus notify_all, its event signalling and the completion callback in four separate try blocks, so nothing a Device::synchronize waiter depends on can be skipped by a throw. The command-buffer message is built through a helper with a pre-allocated fallback string, so bad_alloc cannot escape it either. The end_encoding and eval/fence handlers were audited: no waiter bookkeeping sits after a throwing statement there. N5: commits_issued is incremented after commit() succeeds, so a throw from addCompletedHandler or commit cannot raise a target no handler will ever reach. N6: note_error_reported skips an error already registered and caps the registry at 64 entries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #2086, #3078, #3216.
The frontend compile cache uses function ID as key so it is very unlikely functions in different threads would shared cache, and making it thread local should be enough to ensure thread safety.
For backend compile cache, i.e.
Compiled::eval_cpu/eval_gpu, it is reasonable to share cache between threads and they are already guarded with locks (except for the CUDA backend which I will fix in followup PRs).