Skip to content

gh-109649: Use os.process_cpu_count()#110165

Merged
vstinner merged 4 commits intopython:mainfrom
vstinner:use_process_count
Oct 1, 2023
Merged

gh-109649: Use os.process_cpu_count()#110165
vstinner merged 4 commits intopython:mainfrom
vstinner:use_process_count

Conversation

@vstinner
Copy link
Member

@vstinner vstinner commented Sep 30, 2023 •
edited by github-actions bot
Loading

Replace os.cpu_count() with os.process_cpu_count() in modules:

  • compileall
  • concurrent.futures
  • multiprocessing

Replace os.cpu_count() with os.process_cpu_count() in programs:

  • _decimal deccheck.py test
  • freeze.py
  • multissltests.py
  • python -m test (regrtest)
  • wasm_build.py

Other changes:

  • test.pythoninfo logs os.process_cpu_count().
  • regrtest gets os.process_cpu_count() / os.cpu_count() in headers.

📚 Documentation preview 📚: https://cpython-previews--110165.org.readthedocs.build/

@vstinner
pythongh-109649: Use os.process_cpu_count()
6b15e90 
Replace os.cpu_count() with os.process_cpu_count() in modules:

* compileall
* concurrent.futures
* multiprocessing

Replace os.cpu_count() with os.process_cpu_count() in programs:

* _decimal deccheck.py test
* freeze.py
* multissltests.py
* python -m test (regrtest)
* wasm_build.py

Other changes:

* test.pythoninfo logs os.process_cpu_count().
* regrtest gets os.process_cpu_count() / os.cpu_count() in headers.
@bedevere-app bedevere-app bot mentioned this pull request Sep 30, 2023
Closed
@vstinner
Copy link
Member Author

vstinner commented Sep 30, 2023

Example: CPU count: 2 (process) / 12 (system)

vstinner@mona$ taskset -c 0-1 ./python -m test 
== CPython 3.13.0a0 (heads/use_process_count:a8514200b8, Oct 1 2023, 01:31:10) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)]
== Linux-6.4.15-200.fc38.x86_64-x86_64-with-glibc2.37 little-endian
== Python build: debug
== cwd: /home/vstinner/python/main/build/test_python_worker_2132029æ
== CPU count: 2 (process) / 12 (system)
(...)

@vstinner
Copy link
Member Author

vstinner commented Sep 30, 2023

Replace os.cpu_count() with os.process_cpu_count() in programs:

  • _decimal deccheck.py test
  • freeze.py
  • multissltests.py
  • python -m test (regrtest)
  • wasm_build.py

multissltests.py and wasm_build.py use the code below to keep support for old Python versions:

    if hasattr(os, 'process_cpu_count'):
        jobs = os.process_cpu_count()
    else:
        jobs = os.cpu_count()

@vstinner vstinner merged commit a46e960 into python:main Oct 1, 2023
@vstinner vstinner deleted the use_process_count branch October 1, 2023 01:14
@bedevere-bot
Copy link

bedevere-bot commented Oct 1, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x SLES 3.x has failed when building commit a46e960.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/540/builds/6704) and take a look at the build logs.
  4. Check if the failure is related to this commit (a46e960) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/540/builds/6704

Failed tests:

  • test.test_concurrent_futures.test_thread_pool

Failed subtests:

  • test_idle_thread_reuse - test.test_concurrent_futures.test_thread_pool.ThreadPoolExecutorTest.test_idle_thread_reuse

Summary of the results of the build (if available):

==

Click to see traceback logs 
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_concurrent_futures/test_thread_pool.py", line 49, in test_idle_thread_reuse
    self.assertEqual(len(executor._threads), 1)
AssertionError: 2 != 1

@bedevere-bot
Copy link

bedevere-bot commented Oct 1, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Debian 3.x has failed when building commit a46e960.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/49/builds/6739) and take a look at the build logs.
  4. Check if the failure is related to this commit (a46e960) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/49/builds/6739

Failed tests:

  • test.test_multiprocessing_forkserver.test_processes

Summary of the results of the build (if available):

==

Click to see traceback logs 
remote: Enumerating objects: 40, done.        
remote: Counting objects:   2% (1/40)        
remote: Counting objects:   5% (2/40)        
remote: Counting objects:   7% (3/40)        
remote: Counting objects:  10% (4/40)        
remote: Counting objects:  12% (5/40)        
remote: Counting objects:  15% (6/40)        
remote: Counting objects:  17% (7/40)        
remote: Counting objects:  20% (8/40)        
remote: Counting objects:  22% (9/40)        
remote: Counting objects:  25% (10/40)        
remote: Counting objects:  27% (11/40)        
remote: Counting objects:  30% (12/40)        
remote: Counting objects:  32% (13/40)        
remote: Counting objects:  35% (14/40)        
remote: Counting objects:  37% (15/40)        
remote: Counting objects:  40% (16/40)        
remote: Counting objects:  42% (17/40)        
remote: Counting objects:  45% (18/40)        
remote: Counting objects:  47% (19/40)        
remote: Counting objects:  50% (20/40)        
remote: Counting objects:  52% (21/40)        
remote: Counting objects:  55% (22/40)        
remote: Counting objects:  57% (23/40)        
remote: Counting objects:  60% (24/40)        
remote: Counting objects:  62% (25/40)        
remote: Counting objects:  65% (26/40)        
remote: Counting objects:  67% (27/40)        
remote: Counting objects:  70% (28/40)        
remote: Counting objects:  72% (29/40)        
remote: Counting objects:  75% (30/40)        
remote: Counting objects:  77% (31/40)        
remote: Counting objects:  80% (32/40)        
remote: Counting objects:  82% (33/40)        
remote: Counting objects:  85% (34/40)        
remote: Counting objects:  87% (35/40)        
remote: Counting objects:  90% (36/40)        
remote: Counting objects:  92% (37/40)        
remote: Counting objects:  95% (38/40)        
remote: Counting objects:  97% (39/40)        
remote: Counting objects: 100% (40/40)        
remote: Counting objects: 100% (40/40), done.        
remote: Compressing objects:  16% (1/6)        
remote: Compressing objects:  33% (2/6)        
remote: Compressing objects:  50% (3/6)        
remote: Compressing objects:  66% (4/6)        
remote: Compressing objects:  83% (5/6)        
remote: Compressing objects: 100% (6/6)        
remote: Compressing objects: 100% (6/6), done.        
remote: Total 40 (delta 34), reused 40 (delta 34), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to 'a46e96076898d126c9f276aef1934195aac34b4e'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at a46e960768 gh-109649: Use os.process_cpu_count() (#110165)
Switched to and reset branch 'main'

make: *** [Makefile:2018: buildbottest] Error 5

@bedevere-bot
Copy link

bedevere-bot commented Oct 1, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora 3.x has failed when building commit a46e960.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/223/builds/4583) and take a look at the build logs.
  4. Check if the failure is related to this commit (a46e960) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/223/builds/4583

Failed tests:

  • test.test_concurrent_futures.test_shutdown

Failed subtests:

  • test_interpreter_shutdown - test.test_concurrent_futures.test_shutdown.ProcessPoolForkserverProcessPoolShutdownTest.test_interpreter_shutdown

Summary of the results of the build (if available):

==

Click to see traceback logs 
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 50, in test_interpreter_shutdown
    self.assertEqual(out.strip(), b"apple")
AssertionError: b'' != b'apple'

Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@vstinner @Glyphack
pythongh-109649: Use os.process_cpu_count() (python#110165)
c3bc035 
Replace os.cpu_count() with os.process_cpu_count() in modules:

* compileall
* concurrent.futures
* multiprocessing

Replace os.cpu_count() with os.process_cpu_count() in programs:

* _decimal deccheck.py test
* freeze.py
* multissltests.py
* python -m test (regrtest)
* wasm_build.py

Other changes:

* test.pythoninfo logs os.process_cpu_count().
* regrtest gets os.process_cpu_count() / os.cpu_count() in headers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@corona10 corona10 corona10 approved these changes

@brettcannon brettcannon Awaiting requested review from brettcannon

@rhettinger rhettinger Awaiting requested review from rhettinger

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@vstinner @bedevere-bot @corona10