gh-135335: flush stdout/stderr in forkserver after preloading modules by duaneg · Pull Request #135338 · python/cpython

duaneg · 2025-06-10T09:45:07Z

If a preloaded module writes to stdout or stderr, and the stream is buffered, child processes will inherit the buffered data after forking. Attempt to prevent this by flushing the streams after preload.

Issue: Fork server doesn't flush stdout/stderr after preloading modules, potentially leaving buffered data to be inherited by child processes #135335

…odules If a preloaded module writes to stdout or stderr, and the stream is buffered, child processes will inherit the buffered data after forking. Attempt to prevent this by flushing the streams after preload.

...or at least produce more useful output if they fail

Lib/test/mp_preload_flush.py

… if the defaults change elsewhere in the future)

Lib/test/_test_multiprocessing.py

…use a not-completely-awful name for the module to test.

Lib/test/_test_multiprocessing.py

Co-authored-by: Mikhail Efimov <efimov.mikhail@gmail.com>

efimov-mikhail

LGTM

Lib/test/_test_multiprocessing.py

Lib/test/mp_preload_flush.py

Co-authored-by: Victor Stinner <vstinner@python.org>

Lib/test/_test_multiprocessing.py

Misc/NEWS.d/next/Library/2025-06-10-21-42-04.gh-issue-135335.WnUqb_.rst

Co-authored-by: Victor Stinner <vstinner@python.org>

…nUqb_.rst Co-authored-by: Victor Stinner <vstinner@python.org>

vstinner

LGTM. I just have a last suggestion.

Lib/test/mp_preload_flush.py

efimov-mikhail

LGTM

miss-islington-app · 2025-06-18T12:17:31Z

Thanks @duaneg for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington-app · 2025-06-18T12:17:31Z

Thanks @duaneg for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

…odules (pythonGH-135338) If a preloaded module writes to stdout or stderr, and the stream is buffered, child processes will inherit the buffered data after forking. Attempt to prevent this by flushing the streams after preload. (cherry picked from commit 9877d19) Co-authored-by: Duane Griffin <duaneg@dghda.com> Co-authored-by: Mikhail Efimov <efimov.mikhail@gmail.com> Co-authored-by: Victor Stinner <vstinner@python.org>

bedevere-app · 2025-06-18T12:17:40Z

GH-135670 is a backport of this pull request to the 3.14 branch.

…odules (pythonGH-135338) If a preloaded module writes to stdout or stderr, and the stream is buffered, child processes will inherit the buffered data after forking. Attempt to prevent this by flushing the streams after preload. (cherry picked from commit 9877d19) Co-authored-by: Duane Griffin <duaneg@dghda.com> Co-authored-by: Mikhail Efimov <efimov.mikhail@gmail.com> Co-authored-by: Victor Stinner <vstinner@python.org>

bedevere-app · 2025-06-18T12:17:45Z

GH-135671 is a backport of this pull request to the 3.13 branch.

vstinner · 2025-06-18T12:19:05Z

Thanks for the bug report and the fix @duaneg.

duaneg · 2025-06-18T12:22:45Z

Thanks for the reviews, folks.

…modules (GH-135338) (#135671) gh-135335: flush stdout/stderr in forkserver after preloading modules (GH-135338) If a preloaded module writes to stdout or stderr, and the stream is buffered, child processes will inherit the buffered data after forking. Attempt to prevent this by flushing the streams after preload. (cherry picked from commit 9877d19) Co-authored-by: Duane Griffin <duaneg@dghda.com> Co-authored-by: Mikhail Efimov <efimov.mikhail@gmail.com> Co-authored-by: Victor Stinner <vstinner@python.org>

…modules (GH-135338) (#135670) gh-135335: flush stdout/stderr in forkserver after preloading modules (GH-135338) If a preloaded module writes to stdout or stderr, and the stream is buffered, child processes will inherit the buffered data after forking. Attempt to prevent this by flushing the streams after preload. (cherry picked from commit 9877d19) Co-authored-by: Duane Griffin <duaneg@dghda.com> Co-authored-by: Mikhail Efimov <efimov.mikhail@gmail.com> Co-authored-by: Victor Stinner <vstinner@python.org>

efimov-mikhail · 2025-06-18T13:24:13Z

Thanks for thorough working on this PR!
Looking forward for the next one!

duaneg · 2025-06-18T22:14:41Z

Thanks for thorough working on this PR! Looking forward for the next one!

Well, as it happens, I would love #132574 to get a review 🙂

It is a very simple bug and fix: an off-by-one error, that allows an out-of-bounds read, which affects basic string operations like replace(), and has been in our code base ~forever.

efimov-mikhail · 2025-06-18T22:24:22Z

Ok, I'll take a look.

…odules (python#135338) If a preloaded module writes to stdout or stderr, and the stream is buffered, child processes will inherit the buffered data after forking. Attempt to prevent this by flushing the streams after preload. Co-authored-by: Mikhail Efimov <efimov.mikhail@gmail.com> Co-authored-by: Victor Stinner <vstinner@python.org>

pythongh-135335: flush stdout/stderr in forkserver after preloading m…

d59b229

…odules If a preloaded module writes to stdout or stderr, and the stream is buffered, child processes will inherit the buffered data after forking. Attempt to prevent this by flushing the streams after preload.

duaneg requested a review from gpshead as a code owner June 10, 2025 09:45

bedevere-app bot added the awaiting review label Jun 10, 2025

bedevere-app bot mentioned this pull request Jun 10, 2025

Fork server doesn't flush stdout/stderr after preloading modules, potentially leaving buffered data to be inherited by child processes #135335

Closed

duaneg added 2 commits June 10, 2025 23:01

Attempt to fix MacOS test failures

552266f

...or at least produce more useful output if they fail

Check stderr first to get details of assertion failures

6e36eea

efimov-mikhail requested changes Jun 11, 2025

View reviewed changes

Lib/test/mp_preload_flush.py Outdated Show resolved Hide resolved

bedevere-app bot added awaiting core review and removed awaiting review labels Jun 11, 2025

duaneg added 2 commits June 12, 2025 13:53

Further minor diagnostic improvements upon test failure

33f9956

Set start method explicitly to "forkserver" so it works on MacOS (and…

9944968

… if the defaults change elsewhere in the future)

duaneg requested a review from efimov-mikhail June 16, 2025 09:06

efimov-mikhail reviewed Jun 17, 2025

View reviewed changes

Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved

vstinner reviewed Jun 17, 2025

View reviewed changes

Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved

Address reviewer feedback: introduce some intermediate variables and …

1523e74

…use a not-completely-awful name for the module to test.

efimov-mikhail reviewed Jun 17, 2025

View reviewed changes

Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved

efimov-mikhail reviewed Jun 17, 2025

View reviewed changes

Lib/test/_test_multiprocessing.py Show resolved Hide resolved

duaneg and others added 2 commits June 17, 2025 23:27

Update Lib/test/_test_multiprocessing.py

2125574

Co-authored-by: Mikhail Efimov <efimov.mikhail@gmail.com>

Additional review feedback: add comment and align ordering

db298cb

efimov-mikhail approved these changes Jun 17, 2025

View reviewed changes

vstinner reviewed Jun 17, 2025

View reviewed changes

Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved

vstinner reviewed Jun 17, 2025

View reviewed changes

Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved

vstinner reviewed Jun 17, 2025

View reviewed changes

Lib/test/mp_preload_flush.py Outdated Show resolved Hide resolved

duaneg and others added 2 commits June 18, 2025 00:44

Update Lib/test/mp_preload_flush.py

f9bab0a

Co-authored-by: Victor Stinner <vstinner@python.org>

Show default diff on unexpected stdout/err

4ffa138

vstinner reviewed Jun 17, 2025

View reviewed changes

Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved

Lib/test/_test_multiprocessing.py Outdated Show resolved Hide resolved

Misc/NEWS.d/next/Library/2025-06-10-21-42-04.gh-issue-135335.WnUqb_.rst Outdated Show resolved Hide resolved

duaneg and others added 4 commits June 18, 2025 10:35

Update Lib/test/_test_multiprocessing.py

4d57d74

Co-authored-by: Victor Stinner <vstinner@python.org>

Update Misc/NEWS.d/next/Library/2025-06-10-21-42-04.gh-issue-135335.W…

e73ea34

…nUqb_.rst Co-authored-by: Victor Stinner <vstinner@python.org>

Remove redundant check

3f07ff7

Don't write an newline to avoid any implicit flushes that might happen

da4df44

vstinner approved these changes Jun 18, 2025

View reviewed changes

Lib/test/mp_preload_flush.py Outdated Show resolved Hide resolved

bedevere-app bot added awaiting merge and removed awaiting core review labels Jun 18, 2025

Be consistent and do not rely on asserts being enabled

fdb30e1

efimov-mikhail approved these changes Jun 18, 2025

View reviewed changes

vstinner merged commit 9877d19 into python:main Jun 18, 2025
39 checks passed

bedevere-app bot removed the awaiting merge label Jun 18, 2025

vstinner added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Jun 18, 2025

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Jun 18, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Jun 18, 2025

Uh oh!

Conversation

duaneg commented Jun 10, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efimov-mikhail left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

efimov-mikhail left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app bot commented Jun 18, 2025

Uh oh!

miss-islington-app bot commented Jun 18, 2025

Uh oh!

bedevere-app bot commented Jun 18, 2025

Uh oh!

bedevere-app bot commented Jun 18, 2025

Uh oh!

vstinner commented Jun 18, 2025

Uh oh!

duaneg commented Jun 18, 2025

Uh oh!

efimov-mikhail commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

duaneg commented Jun 18, 2025

Uh oh!

efimov-mikhail commented Jun 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

duaneg commented Jun 10, 2025 •

edited by bedevere-app bot

Loading

efimov-mikhail commented Jun 18, 2025 •

edited

Loading