Skip to content

Race on GC enabled state under free threading #129533

Closed
Closed
Race on GC enabled state under free threading#129533
Assignees
corona10
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or error

Description

@hawkinsp
hawkinsp
opened on Jan 31, 2025

Bug report

Bug description:

The following code, when run under Python 3.13 commit 0468ea1 built with TSAN, produces a data race error:

import concurrent.futures
import functools
import gc
import threading

num_threads = 10

def closure(b):
  b.wait()
  for _ in range(100):
      gc.disable()
      gc.enable()

with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
  for i in range(100):
    b = threading.Barrier(num_threads)
    for _ in range(num_threads):
      executor.submit(functools.partial(closure, b))

TSAN report:

WARNING: ThreadSanitizer: data race (pid=3441855)
  Write of size 4 at 0x5576617e4e34 by thread T10:
    #0 PyGC_Disable /usr/local/google/home/phawkins/p/cpython/Python/gc_free_threading.c:1508:22 (python3.13+0x444e39) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #1 gc_disable_impl /usr/local/google/home/phawkins/p/cpython/Modules/gcmodule.c:51:5 (python3.13+0x4d8cbf) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #2 gc_disable /usr/local/google/home/phawkins/p/cpython/Modules/clinic/gcmodule.c.h:45:12 (python3.13+0x4d8cbf)
    #3 cfunction_vectorcall_NOARGS /usr/local/google/home/phawkins/p/cpython/Objects/methodobject.c:484:24 (python3.13+0x289ed9) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #4 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ead4a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #5 PyObject_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c:327:12 (python3.13+0x1ead4a)
    #6 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e271b) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #7 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #8 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #9 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #10 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x5723a2) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #11 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x5723a2)
    #12 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #13 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #14 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #15 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4902) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #16 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #17 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #18 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #19 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef38f) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #20 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef38f)
    #21 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #22 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #23 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #24 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564a82) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #25 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bdd87) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)

  Previous write of size 4 at 0x5576617e4e34 by thread T9:
    #0 PyGC_Enable /usr/local/google/home/phawkins/p/cpython/Python/gc_free_threading.c:1499:22 (python3.13+0x444dc9) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #1 gc_enable_impl /usr/local/google/home/phawkins/p/cpython/Modules/gcmodule.c:37:5 (python3.13+0x4d8c9f) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #2 gc_enable /usr/local/google/home/phawkins/p/cpython/Modules/clinic/gcmodule.c.h:27:12 (python3.13+0x4d8c9f)
    #3 cfunction_vectorcall_NOARGS /usr/local/google/home/phawkins/p/cpython/Objects/methodobject.c:484:24 (python3.13+0x289ed9) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #4 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ead4a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #5 PyObject_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c:327:12 (python3.13+0x1ead4a)
    #6 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e271b) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #7 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #8 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #9 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #10 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x5723a2) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #11 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x5723a2)
    #12 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #13 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #14 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #15 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4902) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #16 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #17 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #18 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #19 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef38f) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #20 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef38f)
    #21 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)
    #22 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #23 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 64364346a930c8e649018fed7b43c1cc0e8920e5)

This is my attempt to reproduce a race I saw in the JAX tsan CI, which is similar but not identical:

WARNING: ThreadSanitizer: data race (pid=145998)
  Read of size 4 at 0x55cb2fa81e34 by thread T64 (mutexes: read M0):
    #0 gc_should_collect /__w/jax/jax/cpython/Python/gc_free_threading.c:1062:59 (python3.13+0x44732b) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #1 record_allocation /__w/jax/jax/cpython/Python/gc_free_threading.c:1086:13 (python3.13+0x44732b)
    #2 gc_alloc /__w/jax/jax/cpython/Python/gc_free_threading.c:1708:5 (python3.13+0x44732b)
    #3 _PyObject_GC_New /__w/jax/jax/cpython/Python/gc_free_threading.c:1720:20 (python3.13+0x447101) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #4 tuple_iter /__w/jax/jax/cpython/Objects/tupleobject.c:1110:10 (python3.13+0x2e0700) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #5 PyObject_GetIter /__w/jax/jax/cpython/Objects/abstract.c:2860:25 (python3.13+0x1c1f12) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
...

  Previous write of size 4 at 0x55cb2fa81e34 by thread T70 (mutexes: read M0):
    #0 PyGC_Disable /__w/jax/jax/cpython/Python/gc_free_threading.c:1508:22 (python3.13+0x444e39) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #1 __Pyx_PyType_Ready _traversal.c (_traversal.cpython-313t-x86_64-linux-gnu.so+0x7b16) (BuildId: 0e187ad70a2faa86eeb3f1292897a0491466b409)
    #2 exec_builtin_or_dynamic /__w/jax/jax/cpython/Python/import.c:815:12 (python3.13+0x4658d0) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #3 _imp_exec_dynamic_impl /__w/jax/jax/cpython/Python/import.c:4756:12 (python3.13+0x4658d0)
    #4 _imp_exec_dynamic /__w/jax/jax/cpython/Python/clinic/import.c.h:513:21 (python3.13+0x4658d0)
    #5 cfunction_vectorcall_O /__w/jax/jax/cpython/Objects/methodobject.c:512:24 (python3.13+0x28a135) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #6 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
...

i.e., it looks like there might be a race between gc_should_collect and PyGC_Disable as well.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

Activity

  1. hawkinsp
    added
    type-bugAn unexpected behavior, bug, or error
    on Jan 31, 2025
  2. Eclips4
    added
    topic-free-threading
    on Jan 31, 2025
  3. hawkinsp
    added 2 commits that reference this issue on Jan 31, 2025

    Add additional tsan suppressions for races.

    ac2cffd

    Add additional tsan suppressions for races.

    d4c6657
  4. picnixz
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Jan 31, 2025
  5. corona10
    self-assigned this
    on Feb 2, 2025

  6. corona10 commented on Feb 2, 2025

    @corona10
    corona10
    on Feb 2, 2025
    Member

    I am working on

  7. corona10
    added 5 commits that reference this issue on Feb 2, 2025

    pythongh-129533: Update PyGC_Enable/Disable/IsEnabled to use CAS operâ€Ķ

    1f335ca

    pythongh-129533: Update PyGC_Enable/Disable/IsEnabled to use CAS operâ€Ķ

    62cccf8

    gh-129533: Update PyGC_Enable/Disable/IsEnabled to use atomic operatâ€Ķ

    b184abf

    pythongh-129533: Update PyGC_Enable/Disable/IsEnabled to use atomic â€Ķ

    aea5951

    [3.13] gh-129533: Update PyGC_Enable/Disable/IsEnabled to use atomicâ€Ķ

    f7af8bc

  8. corona10 commented on Feb 7, 2025

    @corona10
    corona10
    on Feb 7, 2025
    Member

    @hawkinsp It's also fixed for 3.13, You can check the fix at the next release of 3.13 :)

  9. corona10
    closed this as completedon Feb 7, 2025
  10. srinivasreddy
    added a commit that references this issue on Feb 7, 2025

    pythongh-129533: Update PyGC_Enable/Disable/IsEnabled to use atomic â€Ķ

    9e29b4f
  11. cmaloney
    added a commit that references this issue on Feb 8, 2025

    pythongh-129533: Update PyGC_Enable/Disable/IsEnabled to use atomic â€Ķ

    8df34e9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions