[`perflint`] implement quick-fix for `manual-list-comprehension` (`PERF401`) #13919

w0nder1ng · 2024-10-24T23:29:38Z

Summary

I implemented a fix to rewrite for-loops where PERF401 applies into list.extends. Since the target is guaranteed to be a list, and the lint (seemingly) guarantees that the iterator doesn't reference itself, this should be valid for all for-loops where the lint is applicable. It would be nice to implement this to rewrite the entire for-loop/variable assignment into a single list comprehensions, but I thought of several cases where this would affect how the code works, so this would need more consideration to write.

Test Plan

I tried this on local files and it seemed to work, but I couldn't find where tests for ruff check --fix lints go. I'd be happy to add tests if someone points me in the right direction.

MichaReiser · 2024-10-25T07:21:35Z

Nice, thank you.
Would you mind adding a few tests showing the fix? Are there cases where a comprehension would be preferred over
list.extend. If so, then I don't think we should provide a code fix.

…teral

w0nder1ng · 2024-10-25T22:23:02Z

Again, I'm not sure where to add tests for fixes.

MichaReiser · 2024-10-26T07:52:29Z

You can extend the existing tests. The testing framework runs your lint rule against the file and snapshots all created diagnostics with their fixes.

ruff/crates/ruff_linter/src/rules/perflint/mod.rs

Line 20 in b34278e

#[test_case(Rule::ManualListComprehension, Path::new("PERF401.py"))]

github-actions · 2024-10-28T17:04:48Z

`ruff-ecosystem` results

Linter (stable)

ℹ️ ecosystem check detected linter changes. (+49 -49 violations, +0 -0 fixes in 3 projects; 51 projects unchanged)

apache/airflow (+31 -31 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --no-preview --select ALL

- dev/breeze/src/airflow_breeze/commands/ci_commands.py:374:25: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/commands/ci_commands.py:374:25: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/commands/release_management_commands.py:3058:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/commands/release_management_commands.py:3058:9: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/commands/sbom_commands.py:973:13: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/commands/sbom_commands.py:973:13: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/params/shell_params.py:385:13: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/params/shell_params.py:385:13: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/utils/exclude_from_matrix.py:32:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/exclude_from_matrix.py:32:9: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/utils/packages.py:326:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/packages.py:326:9: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/utils/reproducible.py:126:17: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/reproducible.py:126:17: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/utils/selective_checks.py:1077:17: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/selective_checks.py:1077:17: PERF401 Use a list extend to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:103:9: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:103:9: PERF401 Use a list extend to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:111:9: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:111:9: PERF401 Use a list extend to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:119:9: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:119:9: PERF401 Use a list extend to create a transformed list
- providers/src/airflow/providers/amazon/aws/hooks/s3.py:486:21: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/hooks/s3.py:486:21: PERF401 Use a list extend to create a transformed list
- providers/src/airflow/providers/amazon/aws/hooks/s3.py:669:21: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/hooks/s3.py:669:21: PERF401 Use a list extend to create a transformed list
- providers/src/airflow/providers/amazon/aws/log/s3_task_handler.py:122:17: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/log/s3_task_handler.py:122:17: PERF401 Use a list extend to create a transformed list
- providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:1210:17: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:1210:17: PERF401 Use a list extend to create a transformed list
- providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:379:17: PERF401 Use a list comprehension to create a transformed list
... 31 additional changes omitted for project

apache/superset (+11 -11 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --no-preview --select ALL

- scripts/benchmark_migration.py:128:21: PERF401 Use a list comprehension to create a transformed list
+ scripts/benchmark_migration.py:128:21: PERF401 Use a list extend to create a transformed list
- superset/db_engine_specs/base.py:107:9: PERF401 Use a list comprehension to create a transformed list
+ superset/db_engine_specs/base.py:107:9: PERF401 Use a list extend to create a transformed list
- superset/tasks/cache.py:192:17: PERF401 Use a list comprehension to create a transformed list
+ superset/tasks/cache.py:192:17: PERF401 Use a list extend to create a transformed list
- superset/utils/core.py:1184:17: PERF401 Use a list comprehension to create a transformed list
+ superset/utils/core.py:1184:17: PERF401 Use a list extend to create a transformed list
- tests/integration_tests/charts/api_tests.py:351:13: PERF401 Use a list comprehension to create a transformed list
+ tests/integration_tests/charts/api_tests.py:351:13: PERF401 Use a list extend to create a transformed list
- tests/integration_tests/charts/api_tests.py:465:13: PERF401 Use a list comprehension to create a transformed list
... 11 additional changes omitted for project

bokeh/bokeh (+7 -7 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --no-preview --select ALL

- src/bokeh/layouts.py:475:25: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/layouts.py:475:25: PERF401 Use a list extend to create a transformed list
- src/bokeh/plotting/_figure.py:479:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/plotting/_figure.py:479:17: PERF401 Use a list extend to create a transformed list
- src/bokeh/plotting/_figure.py:485:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/plotting/_figure.py:485:17: PERF401 Use a list extend to create a transformed list
- src/bokeh/server/contexts.py:310:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/server/contexts.py:310:17: PERF401 Use a list extend to create a transformed list
- src/bokeh/settings.py:780:21: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/settings.py:780:21: PERF401 Use a list extend to create a transformed list
... 4 additional changes omitted for project

Changes by rule (1 rules affected)

code	total	+ violation	- violation	+ fix	- fix
PERF401	98	49	49	0	0

Linter (preview)

ℹ️ ecosystem check detected linter changes. (+49 -49 violations, +0 -0 fixes in 3 projects; 51 projects unchanged)

apache/airflow (+31 -31 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL

- dev/breeze/src/airflow_breeze/commands/ci_commands.py:374:25: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/commands/ci_commands.py:374:25: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/commands/release_management_commands.py:3058:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/commands/release_management_commands.py:3058:9: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/commands/sbom_commands.py:973:13: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/commands/sbom_commands.py:973:13: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/params/shell_params.py:385:13: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/params/shell_params.py:385:13: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/utils/exclude_from_matrix.py:32:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/exclude_from_matrix.py:32:9: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/utils/packages.py:326:9: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/packages.py:326:9: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/utils/reproducible.py:126:17: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/reproducible.py:126:17: PERF401 Use a list extend to create a transformed list
- dev/breeze/src/airflow_breeze/utils/selective_checks.py:1077:17: PERF401 Use a list comprehension to create a transformed list
+ dev/breeze/src/airflow_breeze/utils/selective_checks.py:1077:17: PERF401 Use a list extend to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:103:9: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:103:9: PERF401 Use a list extend to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:111:9: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:111:9: PERF401 Use a list extend to create a transformed list
- docs/exts/docs_build/fetch_inventories.py:119:9: PERF401 Use a list comprehension to create a transformed list
+ docs/exts/docs_build/fetch_inventories.py:119:9: PERF401 Use a list extend to create a transformed list
- providers/src/airflow/providers/amazon/aws/hooks/s3.py:486:21: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/hooks/s3.py:486:21: PERF401 Use a list extend to create a transformed list
- providers/src/airflow/providers/amazon/aws/hooks/s3.py:669:21: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/hooks/s3.py:669:21: PERF401 Use a list extend to create a transformed list
- providers/src/airflow/providers/amazon/aws/log/s3_task_handler.py:122:17: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/log/s3_task_handler.py:122:17: PERF401 Use a list extend to create a transformed list
- providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:1210:17: PERF401 Use a list comprehension to create a transformed list
+ providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:1210:17: PERF401 Use a list extend to create a transformed list
- providers/src/airflow/providers/amazon/aws/operators/sagemaker.py:379:17: PERF401 Use a list comprehension to create a transformed list
... 31 additional changes omitted for project

apache/superset (+11 -11 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL

- scripts/benchmark_migration.py:128:21: PERF401 Use a list comprehension to create a transformed list
+ scripts/benchmark_migration.py:128:21: PERF401 Use a list extend to create a transformed list
- superset/db_engine_specs/base.py:107:9: PERF401 Use a list comprehension to create a transformed list
+ superset/db_engine_specs/base.py:107:9: PERF401 Use a list extend to create a transformed list
- superset/tasks/cache.py:192:17: PERF401 Use a list comprehension to create a transformed list
+ superset/tasks/cache.py:192:17: PERF401 Use a list extend to create a transformed list
- superset/utils/core.py:1184:17: PERF401 Use a list comprehension to create a transformed list
+ superset/utils/core.py:1184:17: PERF401 Use a list extend to create a transformed list
- tests/integration_tests/charts/api_tests.py:351:13: PERF401 Use a list comprehension to create a transformed list
+ tests/integration_tests/charts/api_tests.py:351:13: PERF401 Use a list extend to create a transformed list
- tests/integration_tests/charts/api_tests.py:465:13: PERF401 Use a list comprehension to create a transformed list
... 11 additional changes omitted for project

bokeh/bokeh (+7 -7 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL

- src/bokeh/layouts.py:475:25: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/layouts.py:475:25: PERF401 Use a list extend to create a transformed list
- src/bokeh/plotting/_figure.py:479:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/plotting/_figure.py:479:17: PERF401 Use a list extend to create a transformed list
- src/bokeh/plotting/_figure.py:485:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/plotting/_figure.py:485:17: PERF401 Use a list extend to create a transformed list
- src/bokeh/server/contexts.py:310:17: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/server/contexts.py:310:17: PERF401 Use a list extend to create a transformed list
- src/bokeh/settings.py:780:21: PERF401 Use a list comprehension to create a transformed list
+ src/bokeh/settings.py:780:21: PERF401 Use a list extend to create a transformed list
... 4 additional changes omitted for project

Changes by rule (1 rules affected)

code	total	+ violation	- violation	+ fix	- fix
PERF401	98	49	49	0	0

w0nder1ng · 2024-10-28T17:07:27Z

One thing that I wasn't sure how to handle is cases where there is control flow between the binding and the loop.

def f(early_return):
    result = []
    if early_return:
        return
    for i in range(10):
        result.append(i+1)

Currently, this fix offers to rewrite the code:

def f(early_return):
    result = [i+1 for i in range(10)]
    if early_return:
        return

How do you think this case ought to be handled?

MichaReiser

This looks great. Thanks for working on this fix.

I have a few smaller code improvements that are up to you. The one change we should make is to gate the fix behind preview to give us a chance to catch any errors before rolling it out to all users.

MichaReiser · 2024-10-29T17:29:19Z

crates/ruff_linter/src/rules/perflint/rules/manual_list_comprehension.rs

    #[derive_message_formats]
    fn message(&self) -> String {
-        let ManualListComprehension { is_async } = self;
+        let ManualListComprehension { is_async, .. } = self;


Let's adjust the message to make use of the new comprehension_type as well to avoid inconsistency between the message and the fix title.

crates/ruff_linter/src/rules/perflint/rules/manual_list_comprehension.rs

...er/src/rules/perflint/snapshots/ruff_linter__rules__perflint__tests__PERF401_PERF401.py.snap

w0nder1ng · 2024-11-03T21:49:43Z

I've modified the code to preserve comments and not leave a newline when removing the for loop. The tests seem to be failing because the fix is now gated behind a preview--is there a way to allow the test to set the preview flag, or should I temporarily change the fix availability to "sometimes"?

dhruvmanila · 2024-11-04T03:20:01Z

The tests seem to be failing because the fix is now gated behind a preview--is there a way to allow the test to set the preview flag, or should I temporarily change the fix availability to "sometimes"?

@w0nder1ng You'll need to add a new test case where the preview flag is enabled. Refer to the following as an example:

ruff/crates/ruff_linter/src/rules/ruff/mod.rs

Lines 386 to 402 in 8d7dda9

    
           #[test_case(Rule::ZipInsteadOfPairwise, Path::new("RUF007.py"))] 
        
           fn preview_rules(rule_code: Rule, path: &Path) -> Result<()> { 
        
               let snapshot = format!( 
        
                   "preview__{}_{}", 
        
                   rule_code.noqa_code(), 
        
                   path.to_string_lossy() 
        
               ); 
        
               let diagnostics = test_path( 
        
                   Path::new("ruff").join(path).as_path(), 
        
                   &settings::LinterSettings { 
        
                       preview: PreviewMode::Enabled, 
        
                       ..settings::LinterSettings::for_rule(rule_code) 
        
                   }, 
        
               )?; 
        
               assert_messages!(snapshot, diagnostics); 
        
               Ok(()) 
        
           }

w0nder1ng · 2024-11-04T16:00:40Z

How should I deal with the existing test case?

MichaReiser · 2024-11-05T07:16:18Z

How should I deal with the existing test case?

You would have to mark the rule as sometimes fixable. I suggest to add a TODO comment to the preview check that says that the rule should be changed to always fixable when promoting the fix to stable.

MichaReiser

This is great

crates/ruff_linter/src/rules/perflint/rules/manual_list_comprehension.rs

w0nder1ng · 2024-11-07T01:54:51Z

I changed the diagnostic message, does the new one make more sense? Also, do you know why the tests aren't running?

dhruvmanila · 2024-11-07T04:06:28Z

@w0nder1ng there are merge conflicts which needs to be resolved to get the CI running

implement quick fix for manual list comprehensions

98c14e3

MichaReiser added the fixes Related to suggested fixes for violations label Oct 25, 2024

convert into list comprehension when the variable is an empty list li…

d89950f

…teral

w0nder1ng added 2 commits October 28, 2024 11:28

update tests and only offer fix when binding is a single assignment

f3867ee

update test output

13592b4

w0nder1ng added 2 commits October 28, 2024 14:02

fix false negative when binding statement did not exist

27f8ba8

use slice instead of .name()

0aece9c

w0nder1ng changed the title ~~[PERF401] implement quick-fix for manual-list-comprehension (PERF401)~~ [perflint] implement quick-fix for manual-list-comprehension (PERF401) Oct 28, 2024

MichaReiser requested changes Oct 30, 2024

View reviewed changes

w0nder1ng added 6 commits November 3, 2024 15:29

modify fix to preserve comments

1f7f4c0

modify fix to use direct string modification instead of generation

499bdca

use LineRanges trait

158b2e2

update snapshot

4d441c3

remove leftover indentation from for loop

5dbc764

uncomment preview gate

a3b42e6

add preview test case

aa36f1b

w0nder1ng added 2 commits November 5, 2024 14:34

change fix to FixAvailability::Sometimes

0078d62

uncomment LineRanges import

9aa5627

MichaReiser approved these changes Nov 6, 2024

View reviewed changes

fix nits and improve diagnostic message

64f0f56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[`perflint`] implement quick-fix for `manual-list-comprehension` (`PERF401`) #13919

[`perflint`] implement quick-fix for `manual-list-comprehension` (`PERF401`) #13919

w0nder1ng commented Oct 24, 2024

MichaReiser commented Oct 25, 2024 •

edited

Loading

w0nder1ng commented Oct 25, 2024

MichaReiser commented Oct 26, 2024

github-actions bot commented Oct 28, 2024 •

edited

Loading

w0nder1ng commented Oct 28, 2024

MichaReiser left a comment

MichaReiser Oct 29, 2024

w0nder1ng commented Nov 3, 2024

dhruvmanila commented Nov 4, 2024

w0nder1ng commented Nov 4, 2024

MichaReiser commented Nov 5, 2024

MichaReiser left a comment

w0nder1ng commented Nov 7, 2024

dhruvmanila commented Nov 7, 2024

[perflint] implement quick-fix for manual-list-comprehension (PERF401) #13919

Are you sure you want to change the base?

[perflint] implement quick-fix for manual-list-comprehension (PERF401) #13919

Conversation

w0nder1ng commented Oct 24, 2024

Summary

Test Plan

MichaReiser commented Oct 25, 2024 • edited Loading

w0nder1ng commented Oct 25, 2024

MichaReiser commented Oct 26, 2024

github-actions bot commented Oct 28, 2024 • edited Loading

ruff-ecosystem results

Linter (stable)

Linter (preview)

w0nder1ng commented Oct 28, 2024

MichaReiser left a comment

Choose a reason for hiding this comment

MichaReiser Oct 29, 2024

Choose a reason for hiding this comment

w0nder1ng commented Nov 3, 2024

dhruvmanila commented Nov 4, 2024

w0nder1ng commented Nov 4, 2024

MichaReiser commented Nov 5, 2024

MichaReiser left a comment

Choose a reason for hiding this comment

w0nder1ng commented Nov 7, 2024

dhruvmanila commented Nov 7, 2024

[`perflint`] implement quick-fix for `manual-list-comprehension` (`PERF401`) #13919

[`perflint`] implement quick-fix for `manual-list-comprehension` (`PERF401`) #13919

MichaReiser commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 28, 2024 •

edited

Loading

`ruff-ecosystem` results