bpo-34602: Quadruple stack size on macOS when compiling with UBSAN by ambv · Pull Request #27309 · python/cpython

ambv · 2021-07-23T15:49:07Z

When running ./configure with --with-undefined-behavior-sanitizer, the currently used stack size on macOS is insufficient to pass the test suite.

https://bugs.python.org/issue34602

ambv · 2021-07-23T15:59:49Z

Ideally we could only quadruple the stack size if --with-undefined-behavior-sanitizer is used but I'm not sure how to do that.

ammaraskar

Ideally we could only quadruple the stack size if --with-undefined-behavior-sanitizer is used but I'm not sure how to do that.

I think you should just be able to do a conditional where LINKFORSHARED is set with

if test "$with_undefined_behavior_sanitizer" != "no"; then

configure

ambv · 2021-08-10T16:46:21Z

Closed by mistake. I intend to add the quadrupling to pydebug builds only.

ambv · 2021-08-27T14:20:16Z

It works now only when --with-undefined-behavior-sanitizer is passed.

With:

checking LINKFORSHARED... -Wl,-stack_size,4000000  -framework CoreFoundation

Without:

checking LINKFORSHARED... -Wl,-stack_size,1000000  -framework CoreFoundation

Also, as @ammaraskar pointed out, this should be in sync with THREAD_STACK_SIZE so now it is.

configure

configure.ac

pyconfig.h.in

…passed

ambv · 2021-08-27T20:35:11Z

I rebased on current main and added Blurb.

To demonstrate why this is necessary:

Without this patch the test suite fails miserably with UBSAN

== Tests result: FAILURE ==

393 tests OK.

19 tests failed:
    test_exceptions test_fileio test_io test_isinstance test_json
    test_largefile test_lib2to3 test_logging test_os test_pickle
    test_pickletools test_plistlib test_richcmp test_runpy test_sys
    test_threading test_traceback test_typing test_xml_etree

Command terminated abnormally.
     1490.89 real      5134.98 user       222.52 sys

There's plenty of clear stack issues like:

(click to expand) Fatal Python error: Aborted

Current thread 0x000000010db75dc0 (most recent call first): File "/Volumes/RAMDisk/cpy2/Lib/test/libregrtest/setup.py", line 82 in _test_audit_hook File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ File "/Volumes/RAMDisk/cpy2/Lib/xml/etree/ElementTree.py", line 178 in __repr__ ...

Extension modules: _testcapi (total: 1)

With the patch tests almost pass

411 tests OK.

1 test failed:
    test_threading

The threading failure is another thing we'll have to get to work for UBSAN on macOS but it's unrelated to this bug.

ronaldoussoren

The patch looks good in general, as mentioned below I'd use AC_DEFINE_UNQUOTED(THREAD_STACK_SIZE, ...) even if non using sanitisers and remove setting THREAD_STACK_SIZE for macOS from thread_pthread.h.

configure.ac

configure

bedevere-bot · 2021-08-28T09:37:49Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

ambv · 2021-08-28T10:52:59Z

I have made the requested changes; please review again.

Using --with-undefined-behavior-sanitizer:

>>> import sysconfig
>>> sysconfig.get_config_var('LINKFORSHARED')
'-Wl,-stack_size,4000000  -framework CoreFoundation'
>>> sysconfig.get_config_var('THREAD_STACK_SIZE')
'0x4000000'

No --with-undefined-behavior-sanitizer:

>>> import sysconfig
>>> sysconfig.get_config_var('LINKFORSHARED')
'-Wl,-stack_size,1000000  -framework CoreFoundation'
>>> sysconfig.get_config_var('THREAD_STACK_SIZE')
'0x1000000'

bedevere-bot · 2021-08-28T10:53:01Z

Thanks for making the requested changes!

@ronaldoussoren: please review the changes made to this pull request.

ambv · 2021-09-03T07:32:09Z

Merging this since I have some UBSAN-related work for next week.

miss-islington · 2021-09-03T07:32:21Z

Thanks @ambv for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

miss-islington · 2021-09-03T07:32:24Z

Sorry, @ambv, I could not cleanly backport this to 3.10 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker be9de8721d63b9d8e032d508069daf88c06542c6 3.10

ambv · 2021-09-03T07:33:01Z

I'll deal with the conflict today.

ambv · 2021-09-03T16:56:49Z

The conflict is because the 3.10 branch used autoconf 2.71 for its configure file whereas main (like the other branches) uses 2.69.

…BSAN (pythonGH-27309) (cherry picked from commit be9de87) Co-authored-by: Łukasz Langa <lukasz@langa.pl>

bedevere-bot · 2021-09-10T20:38:49Z

GH-28280 is a backport of this pull request to the 3.10 branch.

…BSAN (GH-27309) (GH-28280) (cherry picked from commit be9de87) Co-authored-by: Łukasz Langa <lukasz@langa.pl>

ambv added the OS-mac label Jul 23, 2021

ambv requested a review from ned-deily July 23, 2021 15:49

bedevere-bot added the awaiting core review label Jul 23, 2021

the-knights-who-say-ni added the CLA signed label Jul 23, 2021

ammaraskar reviewed Jul 27, 2021

View reviewed changes

configure Outdated Show resolved Hide resolved

ned-deily requested a review from ronaldoussoren August 1, 2021 22:34

ambv closed this Aug 10, 2021

ambv deleted the quadruple-stack-memory branch August 10, 2021 16:40

ambv restored the quadruple-stack-memory branch August 10, 2021 16:45

ambv reopened this Aug 10, 2021

ambv commented Aug 27, 2021

View reviewed changes

configure Outdated Show resolved Hide resolved

configure.ac Outdated Show resolved Hide resolved

pyconfig.h.in Outdated Show resolved Hide resolved

ambv added 3 commits August 27, 2021 17:00

Quadruple stack size to be able to run full test suite with sanitizers

a92ce66

Only quadruple stack size if --with-undefined-behavior-sanitizer was …

188315e

…passed

add Blurb

5d734c3

ambv force-pushed the quadruple-stack-memory branch from 9f21de0 to 5d734c3 Compare August 27, 2021 17:53

ronaldoussoren requested changes Aug 28, 2021

View reviewed changes

configure.ac Outdated Show resolved Hide resolved

configure Outdated Show resolved Hide resolved

bedevere-bot added awaiting changes and removed awaiting core review labels Aug 28, 2021

Respond to Ronald's review

7044e8f

bedevere-bot removed the awaiting changes label Aug 28, 2021

bedevere-bot added the awaiting change review label Aug 28, 2021

bedevere-bot requested a review from ronaldoussoren August 28, 2021 10:53

ambv added the needs backport to 3.10 only security fixes label Aug 28, 2021

ambv changed the title ~~bpo-34602: Quadruple stack size to be able to run full test suite with sanitizers~~ bpo-34602: Quadruple stack size on macOS when compiling with UBSAN Aug 28, 2021

ambv merged commit be9de87 into python:main Sep 3, 2021

bedevere-bot removed the awaiting change review label Sep 3, 2021

ambv deleted the quadruple-stack-memory branch September 3, 2021 07:32

miss-islington assigned ambv Sep 3, 2021

ambv added a commit to ambv/cpython that referenced this pull request Sep 10, 2021

[3.10] bpo-34602: Quadruple stack size on macOS when compiling with U…

d5b6249

…BSAN (pythonGH-27309) (cherry picked from commit be9de87) Co-authored-by: Łukasz Langa <lukasz@langa.pl>

bedevere-bot removed the needs backport to 3.10 only security fixes label Sep 10, 2021

ambv added a commit that referenced this pull request Sep 15, 2021

[3.10] bpo-34602: Quadruple stack size on macOS when compiling with U…

2563dd2

…BSAN (GH-27309) (GH-28280) (cherry picked from commit be9de87) Co-authored-by: Łukasz Langa <lukasz@langa.pl>

marche147 mannequin mentioned this pull request Apr 10, 2022

python3 resource.setrlimit strange behaviour under macOS #78783

Closed

Uh oh!

Conversation

ambv commented Jul 23, 2021 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ambv commented Jul 23, 2021

Uh oh!

ammaraskar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ambv commented Aug 10, 2021

Uh oh!

ambv commented Aug 27, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ambv commented Aug 27, 2021

Without this patch the test suite fails miserably with UBSAN

With the patch tests almost pass

Uh oh!

ronaldoussoren left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented Aug 28, 2021

Uh oh!

ambv commented Aug 28, 2021

Uh oh!

bedevere-bot commented Aug 28, 2021

Uh oh!

ambv commented Sep 3, 2021

Uh oh!

miss-islington commented Sep 3, 2021

Uh oh!

miss-islington commented Sep 3, 2021

Uh oh!

ambv commented Sep 3, 2021

Uh oh!

ambv commented Sep 3, 2021

Uh oh!

bedevere-bot commented Sep 10, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ambv commented Jul 23, 2021 •

edited by bedevere-bot

Loading