Repository navigation
ProcessPoolExecutor should not share one BrokenProcessPool exception among all futures #101267
Description
Summary
For Python 3.9.x through what's on main now,
When a worker process terminates abruptly, concurrent.futures.ProcessPoolExecutor creates a single BrokenProcessPool exception to share among all futures (code). This is unsafe because exceptions are mutable.
As a consequence, calling Future.result() on all failed futures will result in malformed tracebacks. Each traceback will be the concatenation of all prior tracebacks with the current one.
This issue can be resolved by creating a separate exception instance for each failed future.
Note that in the example below, the size of the malformed traceback is limited. In real-world applications, it can easily be thousands of lines.
Example Code
from concurrent.futures import ProcessPoolExecutor
import time
import traceback
def main():
with ProcessPoolExecutor() as e:
futures = [e.submit(time.sleep, 10) for _ in range(3)]
# Get one of the processes, and terminate (kill) it.
p = next(iter(e._processes.values()))
p.terminate()
for i, fut in enumerate(futures):
try:
fut.result()
except Exception as e:
print(f"\nException {i}:")
traceback.print_exception(e)
if __name__ == "__main__":
main()Current (Incorrect) Output
Notice that each consecutive traceback appends to the previous one.
Exception 0:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Exception 1:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Exception 2:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.New (Correct) Output
Exception 0:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Exception 1:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Exception 2:
Traceback (most recent call last):
File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
fut.result()
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.Linked PRs
- gh-101267: [bug] ProcessPoolExecutor should not share one BrokenProcessPool exception among all failed futures #101268
- [3.15] gh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcessPool exception among all failed futures (GH-101268)Â #151430
- [3.14] gh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcessPool exception among all failed futures (GH-101268)Â #151431
- [3.13] gh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcessPool exception among all failed futures (GH-101268) (GH-151431)Â #151435
Metadata
Metadata
Assignees
Labels
Projects
- Status
Activity
ProcessPoolExecutor should not share one BrokenProcessPool exception âĶ
ProcessPoolExecutor should not share one BrokenProcessPool exception âĶ
gh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcessPool eâĶ
pythongh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcessâĶ
pythongh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcessâĶ
gpshead commentedon Jun 13, 2026
good catch & thanks!
[3.15] gh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcesâĶ
[3.14] gh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcesâĶ
[3.13] gh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcesâĶ
pythongh-101267: ProcessPoolExecutor no longer shares 1 BrokenProcessâĶ