replace pyupgrade+isort+autoflake with ruff #105

jakkdl · 2024-03-05T14:22:05Z

Take 3 after
master...use-ruff and #102

fixes #101

Notes:

The one feature we're losing with replacing autoflake is not exploding star imports. (There's no test for that functionality.)
when looking for tests I traversed git blame, so bfb380e is now ignored. Unfortunately the commit[s] when @Zac-HD added a ton of carriage returns are in the same commits as adding a bunch of other stuff, so can't ignore them :P
ruff removing pass also removes unnecessary ..., hence change in pattern-matching.txt.

remaining TODOs:

replace usage of autoflake.is_python_file
- I can probably just inline it.
replace black with ruff format (could leave that for a separate PR)
- figure out a way to not rely on black for autodetecting target version
figure out minimum ruff version
open issue about git worktrees
- I made the test xfail for now
remove any now-redundant autofixers
- ~fully done. I'm now working on documenting the remaining ones properly, partly because wtf-why-isn't-that-done-already, and partly so the ruff folks can consider implementing them. But that could be for a different PR.

… isort), skip test_guesses_shed_is_first_party inside worktrees

…r fork

… handle falsey literals

jakkdl · 2024-03-06T16:31:01Z

libcst 1.2 dropped support for 3.8. Resolving it by generating deps.txt on 3.8, but could also do it by limiting libcst to <1.2. Ooooor we can drop 3.8 😉

jakkdl · 2024-03-06T16:37:06Z

ugh that broke 3.12 somehow. Seems like we might actually have to drop 3.8?

Zac-HD

Thanks again @jakkdl, this is looking great!

(especially the documentation so that ruff can pick up more of the codemods... extremely valuable and would have sat on my 'todo someday' list ~forever)

CODEMODS.md

Zac-HD · 2024-03-10T20:05:48Z

src/shed/__init__.py

+    # TODO: I guess this required frozenset because of isort compatibility
+    # but we can probably relax that to an iterable[str]
    assert isinstance(first_party_imports, frozenset)


I think it was more because I prefer strict validation unless there's a specific reason not to, and there's the 'str is an iterable of str' footgun.

ugh, right. I suppose I could change it to Iterable[str] and assert not isinstance(first_party_imports, str)?

and then convert it to a frozenset? Sounds good.

It was previously passed to isort as a frozenset, but now we only use it as a parameter to ruff - where I currently do (the equivalent of) str(list(first_party_imports)) for formatting it. So if we'd only accept a list[str] we don't need to convert the type... not that it really matters though.

src/shed/__init__.py

Zac-HD · 2024-03-10T20:10:01Z

src/shed/_cli.py

+        # TODO: isort.place_module is horrendously complicated, but we only use
+        # a fraction of the functionality. So best approach, if we still need
+        # the ability to exclude stdlib modules here, is probably to generate a list of
+        # known stdlib modules - either dynamically or store in a file.
+        if p.isidentifier()  # and place_module(p) != "STDLIB"


Yeah, having a known list seems fine, and we can just take the union of stdlib modules in all non-EOL Python versions.

Oh, on >3.10 there's https://docs.python.org/3/library/sys.html#sys.stdlib_module_names so only need to generate static lists for 3.8 and 3.9

or well, depends on how thorough we want to be + the level of support for running-version-being-different-from-checked-version

https://pypi.org/project/stdlibs/ sounds pretty great if we want to be maximum thorough

Let's just use that on 3.10+, and vendor the 3.10 list for older versions 👍

src/shed/_codemods.py

tests/recorded/comprehensions/C413.txt

tests/recorded/issue75.txt

jakkdl · 2024-03-11T11:48:49Z

ugh that broke 3.12 somehow. Seems like we might actually have to drop 3.8?

Oh, can just add libcst<1.2; python_version < "3.9" to deps/check.in.

For funsies I also tried running 3.13 locally, but encountered HypothesisWorks/hypothesis#3916

Mostly unrelated to this PR:

Looking at https://github.com/Zac-HD/shed/actions/runs/8175453287/job/22352606988#step:5:75 says

git apply .hypothesis/patches/2024-03-06--c3414d12.patch to add failing examples to your code.

Not that it matters much here (I think? kinda weird error...), but is it possible to get at that patch in any way when it's run in CI? If not, does hypothesis have a flag for printing that patch in a different way to the log so a user could make use of it?

Added/modified comments

…py' instead of 'python'

jakkdl · 2024-03-11T14:27:05Z

The CI check previously failed because tox -e check runs shed, auto-detecting all files in the git repository, which includes CODEMODS.md, and ruff then wants to modify the "input code" in two of the examples. I managed to get around it by using '''py instead of '''python (except with backticks) where ruff tries to butt in. Github still seems to display those correctly, but shed doesn't care for them (which it maybe should?)

Zac-HD · 2024-03-11T17:58:20Z

Mostly unrelated to this PR:

Looking at https://github.com/Zac-HD/shed/actions/runs/8175453287/job/22352606988#step:5:75 says

git apply .hypothesis/patches/2024-03-06--c3414d12.patch to add failing examples to your code.

Not that it matters much here (I think? kinda weird error...), but is it possible to get at that patch in any way when it's run in CI? If not, does hypothesis have a flag for printing that patch in a different way to the log so a user could make use of it?

If you've configured artifact upload for that directory, yes. It's useful to print them though, because if there's a printable example we already show them - in this case here with source_code='#\rµ',.

Zac-HD

I think this is a nice improvement over the status quo, so merging now.

If you want to make a follow-up PR with the stdlib list and optionally the iterable/frozenset handling, plus a new version number, I'll release shortly 🎉

jakkdl added 3 commits March 5, 2024 15:11

replace isort&autoflake with ruff

5533619

remove autoflake, remove last vestige of isort (autoflake depended on…

b50fba0

… isort), skip test_guesses_shed_is_first_party inside worktrees

""""fix"""" coverage

94859af

jakkdl mentioned this pull request Mar 6, 2024

C414 --fix is also unsafe for sorted(reversed(iterable)), and other issues encountered when comparing with shed astral-sh/ruff#10245

Open

jakkdl added 9 commits March 6, 2024 13:23

a bunch of codemods replaced with ruff

4a02434

add commented-out error codes for all checks that zac enabled in thei…

61d63e3

…r fork

enable B011, simplify ShedFixers.leave_Assert

91bbf5b

comments after researching PT018

cfd5c83

comments after researching unnecessary len()/bool()

b2828f0

comments after researching SIM117 vs removed_nested_with codemod

7e91f6f

replace collapse_isinstance_checks with SIM101

fa359a5

fix --check fail, revert leave_Assert changes since ruff also doesn't…

33e934b

… handle falsey literals

pragma: no cover

bac8ede

jakkdl marked this pull request as ready for review March 6, 2024 16:19

jakkdl changed the title ~~replace black+isort+autoflake with ruff~~ replace [black]+isort+autoflake with ruff Mar 6, 2024

jakkdl added 2 commits March 6, 2024 17:23

oops. Though 3.8 seems totes broke dependency-wise.

bae73ab

generate deps on 3.8, since libcst==1.2 requires 3.9+

5b671ba

jakkdl added 2 commits March 8, 2024 16:28

add first draft of CODEMODS.md

2e51c6b

minor fixes for CODEMODS.md

5b9e863

Zac-HD reviewed Mar 10, 2024

View reviewed changes

jakkdl added 5 commits March 11, 2024 12:50

changes after review

f59d4da

remove pyupgrade, enable rules in ruff instead.

6569599

Added/modified comments

update README.md

0667eae

cheat ruff by using 'py' instead of 'python' to mark code blocks

5e49913

reformat some code blocks, hide some from shed by marking them with '…

87e80cf

…py' instead of 'python'

jakkdl changed the title ~~replace [black]+isort+autoflake with ruff~~ replace pyupgrade+isort+autoflake with ruff Mar 11, 2024

Zac-HD approved these changes Mar 11, 2024

View reviewed changes

Zac-HD merged commit b03fad6 into Zac-HD:master Mar 11, 2024
7 checks passed

jakkdl mentioned this pull request Mar 13, 2024

version bump, detect stdlib import #106

Merged

This was referenced Mar 13, 2024

update autoflake min version to 2.1.0 #94

Closed

Use ruff to replace autoflake, pyupgrade, isort and black #102

Closed

After next autoflake release, grab the perf improvement #88

Closed

This was referenced Mar 13, 2024

Add custom rewrites, using e.g. libcst #22

Open

Enable more ruff autofixes #107

Open

jakkdl deleted the use_ruff branch March 13, 2024 21:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

replace pyupgrade+isort+autoflake with ruff #105

replace pyupgrade+isort+autoflake with ruff #105

jakkdl commented Mar 5, 2024 •

edited

Loading

jakkdl commented Mar 6, 2024

jakkdl commented Mar 6, 2024 •

edited

Loading

Zac-HD left a comment

Zac-HD Mar 10, 2024

jakkdl Mar 11, 2024

Zac-HD Mar 11, 2024

jakkdl Mar 13, 2024

Zac-HD Mar 10, 2024

jakkdl Mar 11, 2024

jakkdl Mar 11, 2024

jakkdl Mar 11, 2024

Zac-HD Mar 11, 2024

jakkdl commented Mar 11, 2024 •

edited

Loading

jakkdl commented Mar 11, 2024

Zac-HD commented Mar 11, 2024

Mostly unrelated to this PR:

Zac-HD left a comment

replace pyupgrade+isort+autoflake with ruff #105

replace pyupgrade+isort+autoflake with ruff #105

Conversation

jakkdl commented Mar 5, 2024 • edited Loading

jakkdl commented Mar 6, 2024

jakkdl commented Mar 6, 2024 • edited Loading

Zac-HD left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakkdl commented Mar 11, 2024 • edited Loading

Mostly unrelated to this PR:

jakkdl commented Mar 11, 2024

Zac-HD commented Mar 11, 2024

Mostly unrelated to this PR:

Zac-HD left a comment

Choose a reason for hiding this comment

jakkdl commented Mar 5, 2024 •

edited

Loading

jakkdl commented Mar 6, 2024 •

edited

Loading

jakkdl commented Mar 11, 2024 •

edited

Loading