Skip to content

Re-implement 'mccabe' to reduce supply chain risks and optimize check#10551

Draft
Pierre-Sassoulas wants to merge 16 commits intomainfrom
vendored-mccabe
Draft

Re-implement 'mccabe' to reduce supply chain risks and optimize check#10551
Pierre-Sassoulas wants to merge 16 commits intomainfrom
vendored-mccabe

Conversation

@Pierre-Sassoulas
Copy link
Copy Markdown
Member

@Pierre-Sassoulas Pierre-Sassoulas commented Sep 14, 2025

Type of Changes

Type
✨ New feature
🔨 Refactoring

Description

Draft following #9667. Seems like a lot of optimization can be done by having the code directly in pylint. typing, and removing inheritance, and edge's name handling in particular. Because we're using a tool that permits to draw graph and we don't want to draw anything, we just want to get the damn mccabe metric.

@Pierre-Sassoulas Pierre-Sassoulas added this to the 4.0.0 milestone Sep 14, 2025
@Pierre-Sassoulas Pierre-Sassoulas added Maintenance Discussion or action around maintaining pylint or the dev workflow Needs decision 🔒 Needs a decision before implemention or rejection labels Sep 14, 2025
@codecov
Copy link
Copy Markdown

codecov bot commented Sep 14, 2025

Codecov Report

❌ Patch coverage is 98.26087% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.16%. Comparing base (712ee39) to head (b693390).

Files with missing lines Patch % Lines
pylint/testutils/_primer/primer_compare_command.py 96.03% 4 Missing ⚠️

❌ Your patch check has failed because the patch coverage (98.26%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main   #10551      +/-   ##
==========================================
- Coverage   96.16%   96.16%   -0.01%     
==========================================
  Files         177      177              
  Lines       19617    19725     +108     
==========================================
+ Hits        18865    18968     +103     
- Misses        752      757       +5     
Files with missing lines Coverage Δ
pylint/extensions/mccabe.py 100.00% <100.00%> (+1.68%) ⬆️
pylint/testutils/_primer/primer_compare_command.py 94.04% <96.03%> (-2.55%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

This comment has been minimized.

@Pierre-Sassoulas Pierre-Sassoulas changed the title [mccabe] Vendor mccabe to reduce supply chain risks and optimize analysis [mccabe] Vendor 'mccabe' to reduce supply chain risks and optimize analysis later Sep 14, 2025
@Pierre-Sassoulas Pierre-Sassoulas changed the title [mccabe] Vendor 'mccabe' to reduce supply chain risks and optimize analysis later Vendor 'mccabe' to reduce supply chain risks and optimize analysis later Sep 14, 2025
@github-actions

This comment has been minimized.

Pierre-Sassoulas added a commit that referenced this pull request Sep 21, 2025
## Type of Changes

<!-- Leave the corresponding lines for the applicable type of change:
-->

|     | Type                   |
| --- | ---------------------- |
| ✓   | 📜 Docs          |

## Description

Refs #10551
@Pierre-Sassoulas Pierre-Sassoulas marked this pull request as draft September 21, 2025 15:32
@github-actions

This comment has been minimized.

@Pierre-Sassoulas Pierre-Sassoulas modified the milestones: 4.0.0, 4.1.0 Oct 11, 2025
@Pierre-Sassoulas Pierre-Sassoulas changed the title Vendor 'mccabe' to reduce supply chain risks and optimize analysis later Re-implement 'mccabe' to reduce supply chain risks and optimize check Mar 9, 2026
@Pierre-Sassoulas Pierre-Sassoulas added performance Enhancement ✨ Improvement to a component and removed Needs decision 🔒 Needs a decision before implemention or rejection labels Mar 9, 2026
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

When a message changes slightly (e.g. McCabe rating 18→17, or a
function moves a few lines), the primer used to show it as "message
no longer emitted" + "new message emitted", which is misleading.

Now, after the exact-match pass, residual messages are fuzzy-matched
by (symbol, path, obj) identity and difflib similarity.  Matched
pairs are shown in a "messages have been changed" section with a
unified diff of the changed fields only.

Also fix a bug where the "no longer emitted" closing </details> tag
was always emitted even when there were no missing messages (the
condition checked the dict itself, which is always truthy).
The counter-based approach (complexity_score += 1) couldn't replicate
the original mccabe library's graph formula (E - N + 2). This caused
188 functions to have different scores on astroid alone, almost all
under-counted because the counter misses how paths merge at try/except
blocks and other constructs.

Replace with a proper control flow graph that tracks edge and node
counts, then computes E - N + 2. Only 2 expected differences remain
vs the original mccabe library: match/case (unsupported in mccabe
v0.7.0) and template strings (Python 3.14), both improvements.

Key design: no PathNode objects needed — just integer counters for E
and N. Simple statements are skipped since collapsing linear chains
doesn't change E - N + 2. All performance optimizations retained
(dict dispatch, skip expressions, walk astroid directly).
The dispatch dict mapping node types to visitor methods is now created
once at module level using unbound methods, instead of being rebuilt
for each PathGraphingAstVisitor instance. Callers pass self explicitly.

Eliminates ~18K dict allocations (12 entries each) per ansible run.
Speeds up attribute access for _num_nodes, _num_edges, _tail, _active
and graphs by using C-level struct offsets instead of __dict__ lookups.
These attributes are read/written millions of times during traversal.
Eliminates ~187K method calls per ansible run by inlining the subgraph
parse logic directly into the two callers. For if/for/while (no except
handlers), the extra_blocks loop is dead code and removed entirely.
Astroid nodes are hashable by identity, so we can use them directly as
dict keys. This eliminates ~107K id() builtin calls per ansible run
and simplifies the graphs dict from {int: (result, node)} to
{node: result}.
merge _visit_classdef/_visit_with, use _walk_body from checker

- Store complexity as plain int instead of _ComplexityResult object,
  eliminating ~107K object allocations and method calls.
- Cache self._walk_body as local variable in hot methods to avoid
  repeated bound-method creation via descriptor protocol.
- Merge _visit_classdef and _visit_with into _visit_body since both
  just recurse into node.body.
- Use visitor._walk_body(module.body) from the checker instead of
  a dispatch loop, saving ~18K dispatch calls per run.
…elling

- Add top-level `is_toplevel` handling to `_visit_match` (was an early
  return, inconsistent with `_visit_subgraph` and `_visit_try`)
- Add functional test cases for module-level try/except and match
- Fix spelling issues in comments and docstrings flagged by spell checker
@github-actions
Copy link
Copy Markdown
Contributor

🤖 Effect of this PR on checked open source code: 🤖

Effect on astropy:
The following messages are now emitted:

Details
  1. too-complex:
    'value' is too complex. The McCabe rating is 17
    https://github.com/astropy/astropy/blob/d40c06077eb2557549d86407532da47a5ef8b34a/astropy/io/fits/card.py#L306
  2. too-complex:
    '_make_parser' is too complex. The McCabe rating is 30
    https://github.com/astropy/astropy/blob/d40c06077eb2557549d86407532da47a5ef8b34a/astropy/coordinates/angles/formats.py#L83
  3. too-complex:
    'discretize_model' is too complex. The McCabe rating is 18
    https://github.com/astropy/astropy/blob/d40c06077eb2557549d86407532da47a5ef8b34a/astropy/convolution/utils.py#L88
  4. too-complex:
    '_parser' is too complex. The McCabe rating is 23
    https://github.com/astropy/astropy/blob/d40c06077eb2557549d86407532da47a5ef8b34a/astropy/units/format/cds.py#L125
  5. too-complex:
    '_parser' is too complex. The McCabe rating is 25
    https://github.com/astropy/astropy/blob/d40c06077eb2557549d86407532da47a5ef8b34a/astropy/units/format/ogip.py#L145

The following messages are no longer emitted:

Details
  1. too-complex:
    '_make_parser' is too complex. The McCabe rating is 31
    https://github.com/astropy/astropy/blob/d40c06077eb2557549d86407532da47a5ef8b34a/astropy/coordinates/angles/formats.py#L83
  2. too-complex:
    'discretize_model' is too complex. The McCabe rating is 19
    https://github.com/astropy/astropy/blob/d40c06077eb2557549d86407532da47a5ef8b34a/astropy/convolution/utils.py#L88
  3. too-complex:
    '_parser' is too complex. The McCabe rating is 24
    https://github.com/astropy/astropy/blob/d40c06077eb2557549d86407532da47a5ef8b34a/astropy/units/format/cds.py#L125
  4. too-complex:
    '_parser' is too complex. The McCabe rating is 29
    https://github.com/astropy/astropy/blob/d40c06077eb2557549d86407532da47a5ef8b34a/astropy/units/format/ogip.py#L145

Effect on coverage:
The following messages are now emitted:

Details
  1. too-complex:
    'update' is too complex. The McCabe rating is 14
    https://github.com/nedbat/coveragepy/blob/2a53705f5fe588158b8a8d37ff3beee86388b9e4/coverage/misc.py#L163

The following messages are no longer emitted:

Details
  1. too-complex:
    'update' is too complex. The McCabe rating is 15
    https://github.com/nedbat/coveragepy/blob/2a53705f5fe588158b8a8d37ff3beee86388b9e4/coverage/misc.py#L163

Effect on music21:
The following messages are now emitted:

Details
  1. too-complex:
    'elementToMidiEventList' is too complex. The McCabe rating is 11
    https://github.com/cuthbertLab/music21/blob/65fe74f6584bdeed8421dc57b0452b4d364e4159/music21/midi/translate.py#L1358

The following messages are no longer emitted:

Details
  1. too-complex:
    'elementToMidiEventList' is too complex. The McCabe rating is 12
    https://github.com/cuthbertLab/music21/blob/65fe74f6584bdeed8421dc57b0452b4d364e4159/music21/midi/translate.py#L1358

Effect on ansible:
The following messages are now emitted:

Details
  1. too-complex:
    '_ensure_type' is too complex. The McCabe rating is 38
    https://github.com/ansible/ansible/blob/3f10c2c0a910b15675534891ca567dcde355ad8e/lib/ansible/config/manager.py#L122
  2. too-complex:
    '_fixup_perms2' is too complex. The McCabe rating is 25
    https://github.com/ansible/ansible/blob/3f10c2c0a910b15675534891ca567dcde355ad8e/lib/ansible/plugins/action/__init__.py#L597
  3. too-complex:
    'run' is too complex. The McCabe rating is 12
    https://github.com/ansible/ansible/blob/3f10c2c0a910b15675534891ca567dcde355ad8e/lib/ansible/plugins/lookup/config.py#L95
  4. too-complex:
    '_launch_ssh_agent' is too complex. The McCabe rating is 13
    https://github.com/ansible/ansible/blob/3f10c2c0a910b15675534891ca567dcde355ad8e/lib/ansible/_internal/_ssh/_agent_launch.py#L25
  5. too-complex:
    '_visit' is too complex. The McCabe rating is 17
    https://github.com/ansible/ansible/blob/3f10c2c0a910b15675534891ca567dcde355ad8e/lib/ansible/_internal/_json/__init__.py#L145

The following messages are no longer emitted:

Details
  1. too-complex:
    '_ensure_type' is too complex. The McCabe rating is 39
    https://github.com/ansible/ansible/blob/3f10c2c0a910b15675534891ca567dcde355ad8e/lib/ansible/config/manager.py#L122
  2. too-complex:
    '_fixup_perms2' is too complex. The McCabe rating is 26
    https://github.com/ansible/ansible/blob/3f10c2c0a910b15675534891ca567dcde355ad8e/lib/ansible/plugins/action/__init__.py#L597
  3. too-complex:
    'run' is too complex. The McCabe rating is 13
    https://github.com/ansible/ansible/blob/3f10c2c0a910b15675534891ca567dcde355ad8e/lib/ansible/plugins/lookup/config.py#L95
  4. too-complex:
    '_launch_ssh_agent' is too complex. The McCabe rating is 14
    https://github.com/ansible/ansible/blob/3f10c2c0a910b15675534891ca567dcde355ad8e/lib/ansible/_internal/_ssh/_agent_launch.py#L25
  5. too-complex:
    '_visit' is too complex. The McCabe rating is 18
    https://github.com/ansible/ansible/blob/3f10c2c0a910b15675534891ca567dcde355ad8e/lib/ansible/_internal/_json/__init__.py#L145

Effect on django:
The following messages are no longer emitted:

Details
  1. too-complex:
    'normalize_choices' is too complex. The McCabe rating is 11
    https://github.com/django/django/blob/ad5ea292748246b2f07f3e379a250985c1ebcba5/django/utils/choices.py#L72

Effect on pandas:
The following messages are now emitted:

Details
  1. too-complex:
    '_split_by_backtick' is too complex. The McCabe rating is 13
    https://github.com/pandas-dev/pandas/blob/ec30af13057b4e6a66fe2b8ba66b4265cbd94493/pandas/core/computation/parsing.py#L145

The following messages are no longer emitted:

Details
  1. too-complex:
    '_split_by_backtick' is too complex. The McCabe rating is 14
    https://github.com/pandas-dev/pandas/blob/ec30af13057b4e6a66fe2b8ba66b4265cbd94493/pandas/core/computation/parsing.py#L145

Effect on pytest:
The following messages are now emitted:

Details
  1. too-complex:
    'run' is too complex. The McCabe rating is 17
    https://github.com/pytest-dev/pytest/blob/77dceaa1fd1598293a7a39a837f859620a07002f/src/_pytest/assertion/rewrite.py#L692

The following messages are no longer emitted:

Details
  1. too-complex:
    'run' is too complex. The McCabe rating is 18
    https://github.com/pytest-dev/pytest/blob/77dceaa1fd1598293a7a39a837f859620a07002f/src/_pytest/assertion/rewrite.py#L692

Effect on sentry:
The following messages are now emitted:

Details
  1. too-complex:
    'on_autofix_update' is too complex. The McCabe rating is 13
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/seer/entrypoints/slack/entrypoint.py#L179
  2. too-complex:
    'create_uptime_detector' is too complex. The McCabe rating is 12
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/uptime/subscriptions/subscriptions.py#L255
  3. too-complex:
    'convert_rule_condition_to_snuba_condition' is too complex. The McCabe rating is 14
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/rules/conditions/event_frequency.py#L796
  4. too-complex:
    'literal' is too complex. The McCabe rating is 12
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/replays/lib/eap/snuba_transpiler.py#L725
  5. too-complex:
    'as_log_message' is too complex. The McCabe rating is 43
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/replays/usecases/summarize.py#L333
  6. too-complex:
    'as_trace_item_context' is too complex. The McCabe rating is 43
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/replays/usecases/ingest/event_parser.py#L426
  7. too-complex:
    'get_explore_urls' is too complex. The McCabe rating is 25
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/api/serializers/models/dashboard.py#L134
  8. too-complex:
    'convert_filter_to_snuba_condition' is too complex. The McCabe rating is 21
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/workflow_engine/handlers/condition/event_frequency_query_handlers.py#L201
  9. too-complex:
    'filter_detectors' is too complex. The McCabe rating is 12
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/workflow_engine/endpoints/organization_detector_index.py#L160

The following messages are no longer emitted:

Details
  1. too-complex:
    'on_autofix_update' is too complex. The McCabe rating is 14
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/seer/entrypoints/slack/entrypoint.py#L179
  2. too-complex:
    'create_uptime_detector' is too complex. The McCabe rating is 13
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/uptime/subscriptions/subscriptions.py#L255
  3. too-complex:
    'process_result_internal' is too complex. The McCabe rating is 11
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/uptime/consumers/results_consumer.py#L380
  4. too-complex:
    'convert_rule_condition_to_snuba_condition' is too complex. The McCabe rating is 15
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/rules/conditions/event_frequency.py#L796
  5. too-complex:
    'literal' is too complex. The McCabe rating is 13
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/replays/lib/eap/snuba_transpiler.py#L725
  6. too-complex:
    'as_log_message' is too complex. The McCabe rating is 44
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/replays/usecases/summarize.py#L333
  7. too-complex:
    'as_trace_item_context' is too complex. The McCabe rating is 44
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/replays/usecases/ingest/event_parser.py#L426
  8. too-complex:
    'get_explore_urls' is too complex. The McCabe rating is 26
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/api/serializers/models/dashboard.py#L134
  9. too-complex:
    'convert_filter_to_snuba_condition' is too complex. The McCabe rating is 22
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/workflow_engine/handlers/condition/event_frequency_query_handlers.py#L201
  10. too-complex:
    'filter_workflows' is too complex. The McCabe rating is 11
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/workflow_engine/endpoints/organization_workflow_index.py#L148
  11. too-complex:
    'filter_detectors' is too complex. The McCabe rating is 13
    https://github.com/getsentry/sentry/blob/791ed66d9edf42d08c9e656ce396a6dbe85347e8/src/sentry/workflow_engine/endpoints/organization_detector_index.py#L160

Effect on home-assistant:
The following messages are now emitted:

Details
  1. too-complex:
    '_process_search_results' is too complex. The McCabe rating is 11
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/music_assistant/media_browser.py#L515
  2. too-complex:
    '_update_stream_source' is too complex. The McCabe rating is 13
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/go2rtc/__init__.py#L359
  3. too-complex:
    'process_update' is too complex. The McCabe rating is 22
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/onkyo/media_player.py#L369
  4. too-complex:
    'event_listener' is too complex. The McCabe rating is 22
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/home_connect/coordinator.py#L279
  5. too-complex:
    '_update_from_device' is too complex. The McCabe rating is 19
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/matter/climate.py#L267
  6. too-complex:
    '_reduce_statistics' is too complex. The McCabe rating is 16
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/recorder/statistics.py#L1121
  7. too-complex:
    '_get_max_mean_min_statistic_in_sub_period' is too complex. The McCabe rating is 12
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/recorder/statistics.py#L1439

The following messages are no longer emitted:

Details
  1. too-complex:
    '_get_media_types_from_query' is too complex. The McCabe rating is 11
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/music_assistant/media_browser.py#L464
  2. too-complex:
    '_process_search_results' is too complex. The McCabe rating is 12
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/music_assistant/media_browser.py#L515
  3. too-complex:
    '_update_stream_source' is too complex. The McCabe rating is 14
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/go2rtc/__init__.py#L359
  4. too-complex:
    'process_update' is too complex. The McCabe rating is 23
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/onkyo/media_player.py#L369
  5. too-complex:
    'event_listener' is too complex. The McCabe rating is 23
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/home_connect/coordinator.py#L279
  6. too-complex:
    '_update_from_device' is too complex. The McCabe rating is 20
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/matter/climate.py#L267
  7. too-complex:
    '_reduce_statistics' is too complex. The McCabe rating is 17
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/recorder/statistics.py#L1121
  8. too-complex:
    '_get_max_mean_min_statistic_in_sub_period' is too complex. The McCabe rating is 14
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/recorder/statistics.py#L1439
  9. too-complex:
    '_get_max_mean_min_statistic' is too complex. The McCabe rating is 11
    https://github.com/home-assistant/core/blob/003ee5a6994f4999a8083ff24e4cd4f4c257fad5/homeassistant/components/recorder/statistics.py#L1493

This comment was generated for commit b693390

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Enhancement ✨ Improvement to a component Maintenance Discussion or action around maintaining pylint or the dev workflow performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant