Skip to content

Data race between _PyMonitoring_RegisterCallback and _Py_call_instrumentation_2args in instrumentation.c #131141

Closed

Description

@vfdev-5

Bug report

Bug description:

I built cpython 3.13 from source with TSAN and running the following code:

import sys
import concurrent.futures
import threading


if __name__ == "__main__":
    num_workers = 20
    num_runs = 100

    barrier = threading.Barrier(num_workers)

    tool_id = 3
    event_id = sys.monitoring.events.CALL

    sys.monitoring.use_tool_id(tool_id, "test_tool_1")
    sys.monitoring.set_events(tool_id, event_id)


    def closure():
        barrier.wait()

        def my_callback(code, instruction_offset, callable, arg0):
            pass

        sys.monitoring.register_callback(tool_id, event_id, my_callback)

        def example_function():
            a = 1
            b = 2
            c = a + b

        for _ in range(num_runs):
            example_function()

    with concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) as executor:
        futures = []
        for i in range(num_workers):
            futures.append(executor.submit(closure))
        assert len(list(f.result() for f in futures)) == num_workers

TSAN reports the following data race: https://gist.github.com/vfdev-5/e80f9b4528993eb1543e45a9216350e3#file-repro-log

cpython version:

Python 3.13.2+ experimental free-threading build (heads/3.13:9e0fce413a9, Mar 12 2025, 00:48:15) [Clang 18.1.3 (1ubuntu1)]

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

Activity

  1. kumaraditya303 commented on Mar 12, 2025

    @kumaraditya303
    Contributor

    On current main TSAN report:

    ==================
    WARNING: ThreadSanitizer: data race (pid=76663)
      Atomic write of size 8 at 0x55dfbb90bfc0 by thread T1:
        #0 _Py_atomic_exchange_ptr Include/cpython/pyatomic_gcc.h:195 (python+0x3ea0d4) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #1 exchange_callables Python/instrumentation.c:3014 (python+0x3ea0d4)
        #2 _PyMonitoring_RegisterCallback Python/instrumentation.c:3044 (python+0x3ef67d) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #3 monitoring_register_callback_impl Python/instrumentation.c:2289 (python+0x3efb95) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #4 monitoring_register_callback Python/clinic/instrumentation.c.h:152 (python+0x3efcbc) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #5 cfunction_vectorcall_FASTCALL Objects/methodobject.c:436 (python+0x1f7699) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #6 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x171175) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #7 PyObject_Vectorcall Objects/call.c:327 (python+0x1712be) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #8 _PyEval_EvalFrameDefault Python/generated_cases.c.h:6141 (python+0x355edd) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #9 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x372960) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #10 _PyEval_Vector Python/ceval.c:1843 (python+0x372be6) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #11 _PyFunction_Vectorcall Objects/call.c:413 (python+0x170d1f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #12 _PyVectorcall_Call Objects/call.c:273 (python+0x17312e) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #13 _PyObject_Call Objects/call.c:348 (python+0x173559) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #14 PyObject_Call Objects/call.c:373 (python+0x1735bd) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #15 _PyEval_EvalFrameDefault Python/generated_cases.c.h:6302 (python+0x356fa8) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #16 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x372960) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #17 _PyEval_Vector Python/ceval.c:1843 (python+0x372be6) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #18 _PyFunction_Vectorcall Objects/call.c:413 (python+0x170d1f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #19 _PyVectorcall_Call Objects/call.c:273 (python+0x17312e) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #20 _PyObject_Call Objects/call.c:348 (python+0x173559) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #21 PyObject_Call Objects/call.c:373 (python+0x1735bd) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #22 _PyEval_EvalFrameDefault Python/generated_cases.c.h:6302 (python+0x356fa8) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #23 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x372960) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #24 _PyEval_Vector Python/ceval.c:1843 (python+0x372be6) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #25 _PyFunction_Vectorcall Objects/call.c:413 (python+0x170d1f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #26 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x1754c4) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #27 method_vectorcall Objects/classobject.c:72 (python+0x1757e4) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #28 _PyVectorcall_Call Objects/call.c:273 (python+0x17312e) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #29 _PyObject_Call Objects/call.c:348 (python+0x173559) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #30 PyObject_Call Objects/call.c:373 (python+0x1735bd) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #31 thread_run Modules/_threadmodule.c:351 (python+0x4e448f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #32 pythread_wrapper Python/thread_pthread.h:242 (python+0x43ac31) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    
      Previous read of size 8 at 0x55dfbb90bfc0 by main thread:
        #0 call_one_instrument Python/instrumentation.c:977 (python+0x3e9036) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #1 call_instrumentation_vector Python/instrumentation.c:1171 (python+0x3ea8cc) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #2 _Py_call_instrumentation_2args Python/instrumentation.c:1226 (python+0x3eca2c) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #3 _PyEval_EvalFrameDefault Python/generated_cases.c.h:6074 (python+0x355de0) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #4 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x19cd2a) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #5 gen_send_ex2 Objects/genobject.c:255 (python+0x19d307) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #6 gen_iternext Objects/genobject.c:629 (python+0x19f367) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #7 list_extend_iter_lock_held Objects/listobject.c:1230 (python+0x1c21fb) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #8 _list_extend Objects/listobject.c:1403 (python+0x1c2343) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #9 list___init___impl Objects/listobject.c:3448 (python+0x1c263c) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #10 list_vectorcall Objects/listobject.c:3472 (python+0x1c27db) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #11 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x171175) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #12 PyObject_Vectorcall Objects/call.c:327 (python+0x1712be) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #13 _PyEval_EvalFrameDefault Python/generated_cases.c.h:6141 (python+0x355edd) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #14 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x372960) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #15 _PyEval_Vector Python/ceval.c:1843 (python+0x372be6) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #16 PyEval_EvalCode Python/ceval.c:771 (python+0x372d5b) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #17 run_eval_code_obj Python/pythonrun.c:1362 (python+0x418fa0) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #18 run_mod Python/pythonrun.c:1433 (python+0x419217) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #19 pyrun_file Python/pythonrun.c:1290 (python+0x419c7a) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #20 _PyRun_SimpleFileObject Python/pythonrun.c:518 (python+0x41bbd4) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #21 _PyRun_AnyFileObject Python/pythonrun.c:78 (python+0x41bdd4) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #22 pymain_run_file_obj Modules/main.c:394 (python+0x45659b) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #23 pymain_run_file Modules/main.c:413 (python+0x45670f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #24 pymain_run_python Modules/main.c:679 (python+0x45753f) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #25 Py_RunMain Modules/main.c:760 (python+0x457899) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #26 pymain_main Modules/main.c:790 (python+0x457950) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #27 Py_BytesMain Modules/main.c:814 (python+0x457aaf) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
        #28 main Programs/python.c:15 (python+0x85b3b) (BuildId: 3cf7871e49ad73b7cfa6dc8c3d839e908ed6ea3c)
    
      Location is global '_PyRuntime' of size 377920 at 0x55dfbb8b4400 (python+0x7ebfc0)
    

    The data race is because the callback is being read here without atomic whereas it was exchanged with atomics.

    PyObject *instrument = interp->monitoring_callables[tool][event];

  2. added 3 commits that reference this issue on Mar 12, 2025
  3. added a commit that references this issue on Mar 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.13only security fixes3.14bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions