Skip to content

bpo-46845: Reduce dict size when all keys are Unicode.#31564

Merged
methane merged 9 commits intopython:mainfrom
methane:dict-stringkeys
Mar 1, 2022
Merged

bpo-46845: Reduce dict size when all keys are Unicode.#31564
methane merged 9 commits intopython:mainfrom
methane:dict-stringkeys

Conversation

@methane
Copy link
Member

@methane methane commented Feb 25, 2022 •
edited by bedevere-bot
Loading

@methane methane requested a review from markshannon as a code owner February 25, 2022 05:58
@methane
Copy link
Member Author

methane commented Feb 25, 2022 •
edited
Loading

pyperformance (with --enable-optimizations --with-lto):

Result
Benchmark main-opt compact-opt
2to3 294 ms 292 ms: 1.01x faster
chameleon 7.78 ms 7.66 ms: 1.02x faster
chaos 81.0 ms 82.3 ms: 1.02x slower
crypto_pyaes 93.1 ms 94.3 ms: 1.01x slower
deltablue 4.59 ms 4.73 ms: 1.03x slower
dulwich_log 86.6 ms 85.3 ms: 1.02x faster
fannkuch 433 ms 426 ms: 1.02x faster
float 88.0 ms 86.5 ms: 1.02x faster
go 160 ms 158 ms: 1.01x faster
hexiom 7.40 ms 7.45 ms: 1.01x slower
json_dumps 13.8 ms 13.6 ms: 1.01x faster
json_loads 27.3 us 26.9 us: 1.02x faster
logging_format 7.38 us 7.33 us: 1.01x faster
logging_silent 118 ns 119 ns: 1.00x slower
logging_simple 6.72 us 6.66 us: 1.01x faster
mako 11.8 ms 11.5 ms: 1.02x faster
meteor_contest 116 ms 115 ms: 1.01x faster
nbody 101 ms 103 ms: 1.02x slower
pickle 13.1 us 13.0 us: 1.00x faster
pickle_dict 36.4 us 37.9 us: 1.04x slower
pickle_list 5.28 us 5.22 us: 1.01x faster
pidigits 216 ms 217 ms: 1.01x slower
pyflate 488 ms 490 ms: 1.00x slower
python_startup 28.5 ms 28.1 ms: 1.01x faster
python_startup_no_site 6.89 ms 6.80 ms: 1.01x faster
raytrace 347 ms 355 ms: 1.02x slower
regex_compile 157 ms 157 ms: 1.00x faster
regex_dna 233 ms 234 ms: 1.01x slower
regex_effbot 3.51 ms 3.46 ms: 1.01x faster
regex_v8 26.0 ms 25.5 ms: 1.02x faster
richards 54.8 ms 54.2 ms: 1.01x faster
scimark_fft 365 ms 368 ms: 1.01x slower
scimark_lu 130 ms 126 ms: 1.03x faster
scimark_monte_carlo 79.2 ms 77.7 ms: 1.02x faster
scimark_sor 137 ms 136 ms: 1.01x faster
scimark_sparse_mat_mult 5.44 ms 5.37 ms: 1.01x faster
spectral_norm 115 ms 116 ms: 1.01x slower
sqlalchemy_declarative 159 ms 158 ms: 1.01x faster
sqlalchemy_imperative 24.4 ms 24.2 ms: 1.01x faster
sqlite_synth 2.81 us 2.75 us: 1.02x faster
sympy_expand 566 ms 563 ms: 1.01x faster
sympy_integrate 23.9 ms 24.0 ms: 1.01x slower
sympy_sum 197 ms 196 ms: 1.00x faster
sympy_str 347 ms 344 ms: 1.01x faster
telco 7.50 ms 7.36 ms: 1.02x faster
unpack_sequence 52.3 ns 51.9 ns: 1.01x faster
unpickle 16.0 us 15.1 us: 1.06x faster
unpickle_list 5.42 us 5.28 us: 1.03x faster
unpickle_pure_python 274 us 271 us: 1.01x faster
xml_etree_generate 86.9 ms 86.3 ms: 1.01x faster
xml_etree_process 61.8 ms 61.6 ms: 1.00x faster
Geometric mean (ref) 1.01x faster

Benchmark hidden because not significant (8): django_template, html5lib, nqueens, pathlib, pickle_pure_python, tornado_http, xml_etree_parse, xml_etree_iterparse

@methane
Copy link
Member Author

methane commented Feb 25, 2022

sys.getsizeof(dict.fromkeys("abcdefg")) becomes 272 bytes from 352 bytes on 64bit platform.

maxrss(KB) of /usr/bin/time python -c 'import flask,sqlalchemy,fastapi'

  main compact -  ratio
default 46716 45300 1416 96.97%
-OO 43476 40844 2632 93.95%
-OO + nodebugrange 41424 39632 1792 95.67%

@methane methane mentioned this pull request Feb 25, 2022
Closed
@methane
Copy link
Member Author

methane commented Feb 25, 2022

This PR includes fixing dict_merge, dict_keys, dict_values, and dict_items don't respect order of split table.
They just use _PyDict_Next() for now.

@methane methane merged commit 9833bb9 into python:main Mar 1, 2022
@methane methane deleted the dict-stringkeys branch March 1, 2022 23:09
@methane methane mentioned this pull request Mar 3, 2022
Merged
@markshannon
Copy link
Member

markshannon commented Mar 3, 2022

Thanks for fixing the regressions in split table ordering.

@markshannon markshannon mentioned this pull request Mar 3, 2022
Merged
@gvanrossum gvanrossum mentioned this pull request Sep 12, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@methane @markshannon @the-knights-who-say-ni @bedevere-bot