Repository navigation
#138473 backports to 3.13 & 3.14 break running programs on Python version upgrade (multiprocessing) #142206
Description
Bug report
Bug description:
#138473 changed the format used to pass data between the parent process and subprocesses in multiprocessing. This change has been backported as-is to Python 3.14.1 and FWICS 3.13.10. Unfortunately, this means that if Python is upgraded to a newer patch version while applications are running using this Python version, newly started subprocesses suddenly start using the new data format that is incompatible with the parent process. I think such a change is acceptable for 3.15 now, but this is a major issue with 3.13 & 3.14, because it literally means we would have to tell our users "please restart all programs after upgrading Python, or things may silently break".
I'm sorry but I don't have a traceback handy, but it broke Gentoo's package manager for me mid-upgrade, with the parent process starting to throw JSONDecodeErrors at the non-JSON data provided by children.
Kudos to @thesamesam for finding the offending change.
CPython versions tested on:
3.13, 3.14
Operating systems tested on:
Linux
Linked PRs
- gh-142206: multiprocessing.resource_tracker: Decode messages using older protocol #142215
- [3.14] gh-142206: multiprocessing.resource_tracker: Decode messages using older protocol (GH-142215) #142285
- [3.13] gh-142206: multiprocessing.resource_tracker: Decode messages using older protocol (GH-142215) #142287
Activity
gpshead commentedon Dec 3, 2025
FWIW there are a lot of problems with upgrading a Python installation across patch releases in-place that have happened over the years. I don't think this is nearly as common of a thing to do as it was 10-20y ago given the worlds use of containers.
We don't have regression tests for this concept (and the test matrix would be quite huge as it'd need to test .N to any .N+M patch upgrade).
Gentoo with I presume emerge still being a Python application (sorry it's been too long since i've run gentoo) got bitten hard I guess?
mgorny commentedon Dec 3, 2025
Yep. The package manager uses
multiprocessingto install packages, so the top process couldn't communicate with running installs anymore.gpshead commentedon Dec 3, 2025
the old IPC message format prior to the PR that got backported was
msg = f"{cmd}:{name}:{rtype}\n".encode("ascii")- we could still use that format and parse names that have ':' in them by splitting/partitioning only on the first and last colon without reinventing the format. just brainstorming; that'd probably work as a fix for 3.13 and 3.14 (no need to change main/3.15)eli-schwartz commentedon Dec 3, 2025
Yes, the issue as I understand it was that the communication format was changed in a point release, and the new subprocesses didn't have code to understand the old parent process.
It's a standard mechanism for compatible updates, to simply support reading both formats but only emit one of them. So that should work fine. (And something like this is useful to keep in mind for consideration whenever backporting changes into a previously released version as a bugfix.)
pythongh-142206: multiprocessing.resource_tracker: Decode messages us…
encukou commentedon Dec 3, 2025
Sorry for the issue. I've sent a PR to make this more robust.
[-]#138473 backports to 3.13 & 3.14 break running programs on Python version upgrade[/-][+]#138473 backports to 3.13 & 3.14 break running programs on Python version upgrade (multiprocessing)[/+]gh-142206: multiprocessing.resource_tracker: Decode messages using ol…
00470: pythongh-142206: multiprocessing.resource_tracker: Decode mess…
radi2015 commentedon Dec 4, 2025
/usr/local/lib/python3.14/multiprocessing/resource_tracker.py:369: UserWarning: resource_tracker: There appear to be 8 leaked semaphore objects to clean up at shutdown: {'/mp-p5mvdjr_', '/mp-eod2j56x', '/mp-259nug50', '/mp-kao00ric', '/mp-ajx2y7ng', '/mp-kmq460kb', '/mp-7__h1ds3', '/mp-ld509vj2'}
warnings.warn(
[3.14] pythongh-142206: multiprocessing.resource_tracker: Decode mess…
[3.14] gh-142206: multiprocessing.resource_tracker: Decode messages u…
[3.14] pythongh-142206: multiprocessing.resource_tracker: Decode mess…
[3.13] gh-142206: multiprocessing.resource_tracker: Decode messages u…
mgorny commentedon Dec 5, 2025
Thanks a lot!
pythongh-142206: multiprocessing.resource_tracker: Decode messages us…
therealsickonechase-bit commentedon Dec 12, 2025
Simonkx36 commentedon Jan 2, 2026
dragon081 commentedon Jan 6, 2026
hugovk commentedon Jan 6, 2026
@dragon081 reported to GH.