Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
8b7c3ee
Add raise_exceptions parameter to multiprocessing.set_forkserver_preload
gpshead Nov 23, 2025
e8836d5
Skip forkserver preload tests on platforms without fork support
gpshead Nov 23, 2025
5ce91ba
Skip all forkserver tests on platforms without fork support
gpshead Nov 23, 2025
75495cc
Refactor set_forkserver_preload to use on_error parameter
gpshead Nov 23, 2025
84c9e5b
Fix unused import and make __notes__ test more robust
gpshead Nov 23, 2025
a399218
Fix warn mode to work when warnings are configured as errors
gpshead Nov 23, 2025
9c3ba84
Change __main__ warning message from 'import' to 'preload'
gpshead Nov 23, 2025
70c05d8
Refactor set_forkserver_preload to use _handle_preload helper
gpshead Nov 23, 2025
045be92
Simplify temporary file handling in tests
gpshead Nov 23, 2025
6d4c521
Remove obvious comments and improve import style in tests
gpshead Nov 23, 2025
30c2cf8
Fix warn mode to work when warnings are configured as errors
gpshead Nov 23, 2025
bad9691
Add comments explaining exception catching strategy
gpshead Nov 23, 2025
9d8125f
Use double quotes for string values in documentation
gpshead Nov 23, 2025
2f8edb8
Fix warn mode to work when warnings are configured as errors
gpshead Nov 23, 2025
622345d
Add Gregory P. Smith to NEWS entry contributors
gpshead Nov 23, 2025
42e8eb1
Simplify comments and exception note message
gpshead Nov 23, 2025
64ca5a0
Update _send_value docstring to explain pickling requirement
gpshead Nov 23, 2025
5a8bfc6
Merge main into forkserver on_error branch
gpshead Jan 14, 2026
128e7f0
Fix test hang by restoring __main__ state in TestHandlePreload
gpshead Jan 14, 2026
54f9336
Add stderr capture and assertions to forkserver preload tests
gpshead Jan 14, 2026
6ffe214
Refactor preload error handling into _handle_import_error helper
gpshead Jan 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Refactor set_forkserver_preload to use on_error parameter
Changed from a boolean raise_exceptions parameter to a more flexible
on_error parameter that accepts 'ignore', 'warn', or 'fail'.

- 'ignore' (default): silently ignores import failures
- 'warn': emits ImportWarning from forkserver subprocess
- 'fail': raises exception, causing forkserver to exit

Also improved error messages by adding .add_note() to connection
failures when on_error='fail' to guide users to check stderr.

Updated both spawn.import_main_path() and module __import__()
failure paths to support all three modes using match/case syntax.

Co-authored-by: aggieNick02 <nick@pcpartpicker.com>
Co-authored-by: Claude (Sonnet 4.5) <noreply@anthropic.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
  • Loading branch information
3 people committed Nov 23, 2025
commit 75495cc9e25f7bdeda7b260321100c2ca7325a0c
17 changes: 9 additions & 8 deletions Doc/library/multiprocessing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1211,7 +1211,7 @@ Miscellaneous
.. versionchanged:: 3.11
Accepts a :term:`path-like object`.

.. function:: set_forkserver_preload(module_names, *, raise_exceptions=False)
.. function:: set_forkserver_preload(module_names, *, on_error='ignore')

Set a list of module names for the forkserver main process to attempt to
import so that their already imported state is inherited by forked
Expand All @@ -1221,20 +1221,21 @@ Miscellaneous
For this to work, it must be called before the forkserver process has been
launched (before creating a :class:`Pool` or starting a :class:`Process`).

By default, any :exc:`ImportError` when importing modules is silently
ignored. If *raise_exceptions* is ``True``, :exc:`ImportError` exceptions
will be raised in the forkserver subprocess, causing it to exit. The
exception traceback will appear on stderr, and subsequent attempts to
create processes will fail with :exc:`EOFError` or :exc:`ConnectionError`.
Use *raise_exceptions* during development to catch import problems early.
The *on_error* parameter controls how :exc:`ImportError` exceptions during
module preloading are handled: ``'ignore'`` (default) silently ignores
failures, ``'warn'`` causes the forkserver subprocess to emit an
:exc:`ImportWarning` to stderr, and ``'fail'`` causes the forkserver
subprocess to exit with the exception traceback on stderr, making
subsequent process creation fail with :exc:`EOFError` or
:exc:`ConnectionError`.

Only meaningful when using the ``'forkserver'`` start method.
See :ref:`multiprocessing-start-methods`.

.. versionadded:: 3.4

.. versionchanged:: next
Added the *raise_exceptions* parameter.
Added the *on_error* parameter.

.. function:: set_start_method(method, force=False)

Expand Down
10 changes: 5 additions & 5 deletions Lib/multiprocessing/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -177,15 +177,15 @@ def set_executable(self, executable):
from .spawn import set_executable
set_executable(executable)

def set_forkserver_preload(self, module_names, *, raise_exceptions=False):
def set_forkserver_preload(self, module_names, *, on_error='ignore'):
'''Set list of module names to try to load in forkserver process.

If raise_exceptions is True, ImportError exceptions during preload
will be raised instead of being silently ignored. Such errors will
break all use of the forkserver multiprocessing context.
The on_error parameter controls how import failures are handled:
'ignore' (default) silently ignores failures, 'warn' emits warnings,
and 'fail' raises exceptions breaking the forkserver context.
'''
from .forkserver import set_forkserver_preload
set_forkserver_preload(module_names, raise_exceptions=raise_exceptions)
set_forkserver_preload(module_names, on_error=on_error)

def get_context(self, method=None):
if method is None:
Expand Down
62 changes: 49 additions & 13 deletions Lib/multiprocessing/forkserver.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ def __init__(self):
self._inherited_fds = None
self._lock = threading.Lock()
self._preload_modules = ['__main__']
self._raise_exceptions = False
self._preload_on_error = 'ignore'

def _stop(self):
# Method used by unit tests to stop the server
Expand All @@ -65,17 +65,22 @@ def _stop_unlocked(self):
self._forkserver_address = None
self._forkserver_authkey = None

def set_forkserver_preload(self, modules_names, *, raise_exceptions=False):
def set_forkserver_preload(self, modules_names, *, on_error='ignore'):
'''Set list of module names to try to load in forkserver process.

If raise_exceptions is True, ImportError exceptions during preload
will be raised instead of being silently ignored. Such errors will
break all use of the forkserver multiprocessing context.
The on_error parameter controls how import failures are handled:
'ignore' (default) silently ignores failures, 'warn' emits warnings,
and 'fail' raises exceptions breaking the forkserver context.
'''
if not all(type(mod) is str for mod in modules_names):
raise TypeError('module_names must be a list of strings')
if on_error not in ('ignore', 'warn', 'fail'):
raise ValueError(
f"on_error must be 'ignore', 'warn', or 'fail', "
f"not {on_error!r}"
)
self._preload_modules = modules_names
self._raise_exceptions = raise_exceptions
self._preload_on_error = on_error

def get_inherited_fds(self):
'''Return list of fds inherited from parent process.
Expand Down Expand Up @@ -114,6 +119,15 @@ def connect_to_new_process(self, fds):
wrapped_client, self._forkserver_authkey)
connection.deliver_challenge(
wrapped_client, self._forkserver_authkey)
except (EOFError, ConnectionError, BrokenPipeError) as exc:
# Add helpful context if forkserver likely crashed during preload
if (self._preload_modules and
self._preload_on_error == 'fail'):
exc.add_note(
"Forkserver process may have crashed during module "
"preloading. Check stderr for ImportError traceback."
)
raise
finally:
wrapped_client._detach()
del wrapped_client
Expand Down Expand Up @@ -159,8 +173,8 @@ def ensure_running(self):
main_kws['sys_path'] = data['sys_path']
if 'init_main_from_path' in data:
main_kws['main_path'] = data['init_main_from_path']
if self._raise_exceptions:
main_kws['raise_exceptions'] = True
if self._preload_on_error != 'ignore':
main_kws['on_error'] = self._preload_on_error

with socket.socket(socket.AF_UNIX) as listener:
address = connection.arbitrary_address('AF_UNIX')
Expand Down Expand Up @@ -206,7 +220,7 @@ def ensure_running(self):
#

def main(listener_fd, alive_r, preload, main_path=None, sys_path=None,
*, authkey_r=None, raise_exceptions=False):
*, authkey_r=None, on_error='ignore'):
"""Run forkserver."""
if authkey_r is not None:
try:
Expand All @@ -224,15 +238,37 @@ def main(listener_fd, alive_r, preload, main_path=None, sys_path=None,
process.current_process()._inheriting = True
try:
spawn.import_main_path(main_path)
except Exception as e:
match on_error:
case 'fail':
raise
case 'warn':
import warnings
warnings.warn(
f"Failed to import __main__ from {main_path!r}: {e}",
ImportWarning,
stacklevel=2
)
case 'ignore':
pass
finally:
del process.current_process()._inheriting
for modname in preload:
try:
__import__(modname)
except ImportError:
if raise_exceptions:
raise
pass
except ImportError as e:
match on_error:
case 'fail':
raise
case 'warn':
import warnings
warnings.warn(
f"Failed to preload module {modname!r}: {e}",
ImportWarning,
stacklevel=2
)
case 'ignore':
pass

# gh-135335: flush stdout/stderr in case any of the preloaded modules
# wrote to them, otherwise children might inherit buffered data
Expand Down
51 changes: 38 additions & 13 deletions Lib/test/test_multiprocessing_forkserver/test_preload.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import multiprocessing
import multiprocessing.forkserver
import unittest
import warnings


class TestForkserverPreload(unittest.TestCase):
Expand All @@ -23,9 +24,9 @@ def _send_value(conn, value):
"""Helper to send a value through a connection."""
conn.send(value)

def test_preload_raise_exceptions_false_default(self):
def test_preload_on_error_ignore_default(self):
"""Test that invalid modules are silently ignored by default."""
# With raise_exceptions=False (default), invalid module is ignored
# With on_error='ignore' (default), invalid module is ignored
self.ctx.set_forkserver_preload(['nonexistent_module_xyz'])

# Should be able to start a process without errors
Expand All @@ -40,9 +41,9 @@ def test_preload_raise_exceptions_false_default(self):
self.assertEqual(result, 42)
self.assertEqual(p.exitcode, 0)

def test_preload_raise_exceptions_false_explicit(self):
"""Test that invalid modules are silently ignored with raise_exceptions=False."""
self.ctx.set_forkserver_preload(['nonexistent_module_xyz'], raise_exceptions=False)
def test_preload_on_error_ignore_explicit(self):
"""Test that invalid modules are silently ignored with on_error='ignore'."""
self.ctx.set_forkserver_preload(['nonexistent_module_xyz'], on_error='ignore')

# Should be able to start a process without errors
r, w = self.ctx.Pipe(duplex=False)
Expand All @@ -56,27 +57,45 @@ def test_preload_raise_exceptions_false_explicit(self):
self.assertEqual(result, 99)
self.assertEqual(p.exitcode, 0)

def test_preload_raise_exceptions_true_breaks_context(self):
"""Test that invalid modules with raise_exceptions=True breaks the forkserver."""
self.ctx.set_forkserver_preload(['nonexistent_module_xyz'], raise_exceptions=True)
def test_preload_on_error_warn(self):
"""Test that invalid modules emit warnings with on_error='warn'."""
self.ctx.set_forkserver_preload(['nonexistent_module_xyz'], on_error='warn')

# Should still be able to start a process, warnings are in subprocess
r, w = self.ctx.Pipe(duplex=False)
p = self.ctx.Process(target=self._send_value, args=(w, 123))
p.start()
w.close()
result = r.recv()
r.close()
p.join()

self.assertEqual(result, 123)
self.assertEqual(p.exitcode, 0)

def test_preload_on_error_fail_breaks_context(self):
"""Test that invalid modules with on_error='fail' breaks the forkserver."""
self.ctx.set_forkserver_preload(['nonexistent_module_xyz'], on_error='fail')

# The forkserver should fail to start when it tries to import
# The exception is raised during p.start() when trying to communicate
# with the failed forkserver process
r, w = self.ctx.Pipe(duplex=False)
try:
p = self.ctx.Process(target=self._send_value, args=(w, 42))
with self.assertRaises((EOFError, ConnectionError, BrokenPipeError)):
with self.assertRaises((EOFError, ConnectionError, BrokenPipeError)) as cm:
p.start() # Exception raised here
# Verify that the helpful note was added
self.assertIn('Forkserver process may have crashed', str(cm.exception.__notes__[0]))
finally:
# Ensure pipes are closed even if exception is raised
w.close()
r.close()

def test_preload_valid_modules_with_raise_exceptions(self):
"""Test that valid modules work fine with raise_exceptions=True."""
# Valid modules should work even with raise_exceptions=True
self.ctx.set_forkserver_preload(['os', 'sys'], raise_exceptions=True)
def test_preload_valid_modules_with_on_error_fail(self):
"""Test that valid modules work fine with on_error='fail'."""
# Valid modules should work even with on_error='fail'
self.ctx.set_forkserver_preload(['os', 'sys'], on_error='fail')

r, w = self.ctx.Pipe(duplex=False)
p = self.ctx.Process(target=self._send_value, args=(w, 'success'))
Expand All @@ -89,6 +108,12 @@ def test_preload_valid_modules_with_raise_exceptions(self):
self.assertEqual(result, 'success')
self.assertEqual(p.exitcode, 0)

def test_preload_invalid_on_error_value(self):
"""Test that invalid on_error values raise ValueError."""
with self.assertRaises(ValueError) as cm:
self.ctx.set_forkserver_preload(['os'], on_error='invalid')
self.assertIn("on_error must be 'ignore', 'warn', or 'fail'", str(cm.exception))


if __name__ == '__main__':
unittest.main()

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Add an ``on_error`` keyword-only parameter to
:func:`multiprocessing.set_forkserver_preload` to control how import failures
during module preloading are handled. Accepts ``'ignore'`` (default, silent),
``'warn'`` (emit :exc:`ImportWarning`), or ``'fail'`` (raise exception).
Contributed by Nick Neumann.
Loading