Skip to content

test_socket: testCmsgTrunc0() deadlock on ARM Raspbian 3.x #110167

Closed
Closed
test_socket: testCmsgTrunc0() deadlock on ARM Raspbian 3.x#110167
Labels
testsTests in the Lib/test dir

Description

@vstinner
vstinner
opened on Oct 1, 2023

test_socket adds a lock on addCleanup:

class ThreadSafeCleanupTestCase:
    """Subclass of unittest.TestCase with thread-safe cleanup methods.

    This subclass protects the addCleanup() and doCleanups() methods
    with a recursive lock.
    """

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._cleanup_lock = threading.RLock()

    def addCleanup(self, *args, **kwargs):
        with self._cleanup_lock:
            return super().addCleanup(*args, **kwargs)

    def doCleanups(self, *args, **kwargs):
        with self._cleanup_lock:
            return super().doCleanups(*args, **kwargs)

Problem: what happens if a thread calls addCleanup() while the main thread is calling doCleanups()? Well, a deadlock.

ARM Raspbian 3.x:

0:40:31 load avg: 3.05 [337/467/1] test_socket worker non-zero exit code (Exit code 1) -- running (1): test_math (1 min 4 sec)
Timeout (0:40:00)!
Thread 0xf5dea440 (most recent call first):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_socket.py", line 230 in addCleanup
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_socket.py", line 3576 in newFDs
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_socket.py", line 3611 in createAndSendFDs
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_socket.py", line 3841 in _testCmsgTrunc0
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_socket.py", line 402 in clientRun

Thread 0xf7acb040 (most recent call first):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/threading.py", line 348 in wait
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/threading.py", line 648 in wait
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/unittest/case.py", line 597 in _callCleanup
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/unittest/case.py", line 673 in doCleanups
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_socket.py", line 235 in doCleanups
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/unittest/case.py", line 640 in run
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/unittest/case.py", line 692 in __call__
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/unittest/suite.py", line 122 in run
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/unittest/suite.py", line 122 in run
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/unittest/suite.py", line 122 in run
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/unittest/runner.py", line 240 in run
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/support/__init__.py", line 1155 in _run_suite
  ...

build: https://buildbot.python.org/all/#/builders/424/builds/5065

Linked PRs

Activity

  1. vstinner
    added
    testsTests in the Lib/test dir
    on Oct 1, 2023
  2. vstinner
    added a commit that references this issue on Oct 5, 2023

    pythongh-110167: Increase support.LOOPBACK_TIMEOUT to 10 seconds

    11ce425

  3. vstinner commented on Oct 5, 2023

    @vstinner
    vstinner
    on Oct 5, 2023
    MemberAuthor

    Patch to trigger the bug:

    diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
index 09605f7e77..d3638bccc2 100644
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -215,6 +215,8 @@ def setUp(self):
         self.serv = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDPLITE)
         self.port = socket_helper.bind_port(self.serv)
 
+CLEANUP_EVENT = threading.Event()
+
 class ThreadSafeCleanupTestCase:
     """Subclass of unittest.TestCase with thread-safe cleanup methods.
 
@@ -232,6 +234,7 @@ def addCleanup(self, *args, **kwargs):
 
     def doCleanups(self, *args, **kwargs):
         with self._cleanup_lock:
+            CLEANUP_EVENT.set()
             return super().doCleanups(*args, **kwargs)
 
 class SocketCANTest(unittest.TestCase):
@@ -2818,7 +2821,7 @@ class SendrecvmsgBase(ThreadSafeCleanupTestCase):
 
     # Time in seconds to wait before considering a test failed, or
     # None for no timeout.  Not all tests actually set a timeout.
-    fail_timeout = support.LOOPBACK_TIMEOUT
+    fail_timeout = 1e-9
 
     def setUp(self):
         self.misc_event = threading.Event()
@@ -3573,6 +3576,7 @@ def newFDs(self, n):
         fds = []
         for i in range(n):
             fd, path = tempfile.mkstemp()
+            CLEANUP_EVENT.wait()
             self.addCleanup(os.unlink, path)
             self.addCleanup(os.close, fd)
             os.write(fd, str(i).encode())
  4. vstinner
    added 4 commits that reference this issue on Oct 5, 2023

    pythongh-110167: Fix test_socket deadlock in doCleanups()

    005c9c5

    pythongh-110167: Fix test_socket deadlock in doCleanups()

    90caab2

    pythongh-110167: Fix test_socket deadlock in doCleanups()

    98ac898

    gh-110167: Fix test_socket deadlock in doCleanups() (#110416)

    318f5df
  5. miss-islington
    added 2 commits that reference this issue on Oct 5, 2023

    pythongh-110167: Fix test_socket deadlock in doCleanups() (pythonGH-1…

    4877266

    pythongh-110167: Fix test_socket deadlock in doCleanups() (pythonGH-1…

    1eb2ced
  6. vstinner
    added 2 commits that reference this issue on Oct 5, 2023

    gh-110167: Increase support.LOOPBACK_TIMEOUT to 10 seconds (#110413)

    0db2f14

    pythongh-110167: Increase support.LOOPBACK_TIMEOUT to 10 seconds (pyt…

    a46969d
  7. vstinner
    closed this as completedon Oct 5, 2023
  8. vstinner
    added 4 commits that reference this issue on Oct 5, 2023

    pythongh-110167: Increase support.LOOPBACK_TIMEOUT to 10 seconds (pyt…

    d401794

    [3.11] gh-110167: Fix test_socket deadlock in doCleanups() (GH-110416) (

    8da3367

    [3.12] gh-110167: Fix test_socket deadlock in doCleanups() (GH-110416) (

    2ab41c8

    [3.12] gh-110167: Increase support.LOOPBACK_TIMEOUT to 10 seconds (#1…

    350d89b
  9. miss-islington
    added a commit that references this issue on Oct 5, 2023

    [3.12] pythongh-110167: Increase support.LOOPBACK_TIMEOUT to 10 secon…

    649add4
  10. vstinner
    added a commit that references this issue on Oct 5, 2023

    [3.11] [3.12] gh-110167: Increase support.LOOPBACK_TIMEOUT to 10 seco…

    331d90f
  11. Glyphack
    added 2 commits that reference this issue on Sep 2, 2024

    pythongh-110167: Fix test_socket deadlock in doCleanups() (python#110416

    ee2e555

    pythongh-110167: Increase support.LOOPBACK_TIMEOUT to 10 seconds (pyt…

    38fcf0d

  12. encukou commented on Aug 11, 2026

    @encukou
    encukou
    on Aug 11, 2026
    Member

    The PR changed LOOPBACK_TIMEOUT to 10 seconds, but the docs still say 5.

  13. encukou
    reopened this on Aug 11, 2026
  14. StanFromIreland
    added 3 commits that reference this issue on Aug 11, 2026

    gh-110167: Document the increase of `test.support.LOOPBACK_TIMEOUT` t…

    3444ef9

    [3.13] gh-110167: Document the increase of `test.support.LOOPBACK_TIM…

    4e37cb5

    [3.14] gh-110167: Document the increase of `test.support.LOOPBACK_TIM…

    c611e80
  15. hugovk
    added a commit that references this issue on Aug 12, 2026

    [3.15] gh-110167: Document the increase of `test.support.LOOPBACK_TIM…

    5fc9651
  16. hugovk
    closed this as completedon Aug 12, 2026
  17. mbeijen
    added a commit that references this issue on Aug 14, 2026

    pythongh-110167: Document the increase of `test.support.LOOPBACK_TIME…

    3579b6c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dir

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions