Skip to content

gh-117151: increase default buffer size of shutil.copyfileobj() to 256k. #119783

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gpshead merged 2 commits into from
Oct 4, 2024
Merged

gh-117151: increase default buffer size of shutil.copyfileobj() to 256k. #119783

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cbfaad6
gh-117151: increase default buffer size of shutil.copyfileobj() to 256k.
rmmancom Apr 30, 2024
47c9fa7
add news
rmmancom Oct 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
gh-117151: increase default buffer size of shutil.copyfileobj() to 256k. 
it was set to 16k in the 1990s.
it was raised to 64k in 2019. the discussion at the time mentioned another 5% improvement by raising to 128k and settled for a very conservative setting.

it's 2024 now, I think it should be revisited to match modern hardware. I am measuring 0-15% performance improvement when raising to 256k on various types of disk. there is no downside as far as I can tell.

this function is only intended for sequential copy of full files (or file like objects). it's the typical use case that benefits from larger operations.

for reference, I came across this function while trying to profile pip that is using it to copy files when installing python packages.
  • Loading branch information
@rmmancom
rmmancom committed May 30, 2024
commit cbfaad642045f1d55b8d177bd16646d042b311a9
2 changes: 1 addition & 1 deletion Lib/shutil.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
else:
_winapi = None

COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 64 * 1024
COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 256 * 1024
# This should never be removed, see rationale in:
# https://bugs.python.org/issue43743#msg393429
_USE_CP_SENDFILE = (hasattr(os, "sendfile")
Expand Down