Glob behavior is inconsistent across libraries

[jaraco]

the [Python] docs for pathlib.glob are unclear on the meaning of **

You are right, and it gets even more confusing:

• glob.glob('**') returns dirs and files, non-recursively, not including .
• glob.glob('**', recursive=True) returns dirs and files, recursively, not including .
• list(pathlib.Path('.').glob('**')) returns only dirs, recursively, and also includes .

I think there should be a Python upstream issue to document all of that, including the differences between glob and pathlib.

Then zipp could implement what's documented for pathlib.

Repro

Python 3.10.11

mkdir -p dir/subdir ; touch toplevel.txt dir/indir.txt

creates:

toplevel.txt
dir/
  indir.txt
  subdir/

>>> import glob, pathlib

>>> glob.glob('**')
['dir', 'toplevel.txt']

>>> glob.glob('**', recursive=True)
['dir', 'dir/subdir', 'dir/indir.txt', 'toplevel.txt']

>>> list(pathlib.Path('.').glob('**'))
[PosixPath('.'), PosixPath('dir'), PosixPath('dir/subdir')]

Originally posted by @nh2 in jaraco/zipp#102 (comment)

Linked PRs

• GH-106747: Improve Path.glob() expectations in pathlib tests #112365
• GH-106747: Make pathlib ABC globbing more consistent with glob.glob() #115056
• GH-106747: Prepare pathlib globbing for dir_fd support. #116319
• GH-106747: Document another difference between glob and pathlib. #116518

[jaraco]

Without having thought about it too much, I'd argue pathlib.Path.glob('**') should return dirs and files recursively, not including ., same as glob.glob(recursive=True). I'd also argue that glob.glob('**') should imply recursive=True. After all, what's the meaning of ** if not recursive?

[jaraco]

@barneygale Do you have any insight on the concern? Any opinions?

[jaraco]

Another point of reference, ls on Unix also treats ** as recursive without the dot:

# ls **
dir/indir.txt  toplevel.txt

dir:
indir.txt  subdir

dir/subdir:

[barneygale]

Yeah, it's a mess

the [Python] docs for pathlib.glob are unclear on the meaning of **

Logged as #101112, stale-ish PR up at #101223

• glob.glob('**', recursive=True) returns dirs and files, recursively, not including .

Try glob.glob('tests/**', recursive=True) - you'll find that tests is included in the results, and so ** is capable of consuming zero path segments. The fact that it doesn't emit . seems like the bug to me! Also conflicts with the docs.

• list(pathlib.Path('.').glob('**')) returns only dirs, recursively, and also includes .

Logged as #70303, PR up as #105413.

[jaraco]

I'm going to be nitpicky because this issue demands it. Understand my matter of fact tone below may come off as hostile, but it's not intended that way. I'm merely trying to lay out the facts and disentangle the concerns.

you'll find that tests is included in the results

More precisely, tests/ is included in the results:

 cpython main @ glob.glob('Lib/test/**', recursive=True)[0]
'Lib/test/'

Note I'm using xonsh shell, which is why you see python interleaved with a shell prompt in the cpython repo.

Interestingly, tests/ is not included in the results if recursive=False.

The fact that it doesn't emit . seems like the bug to me! Also conflicts with the docs.

That's the opposite of my intuition. It's almost never the case that a user wants . in their globs (any more than they want ..). When I do glob('mydir/*'), I want to find all of the contents of "mydir", and "mydir" itself is not contained in "mydir".

Even pathlib only emits the . when a recursive glob is indicated:

 cpython main @ next(pathlib.Path().glob('*'))
PosixPath('PC')
 cpython main @ next(pathlib.Path().glob('**'))
PosixPath('.')

I'm now confused. These two statements seem contradictory:

tests is included in the results

it doesn't emit .

In the case of glob.glob('tests/**', recursive=True), tests/ is emitted and that's equivalent to tests/.. Are you saying you think it's a bug that tests/ is emitted without a dot?

Also, I'm unsure how it conflicts with the docs. Can you elaborate on where the conflict lies?

[barneygale]

That's the opposite of my intuition. It's almost never the case that a user wants . in their globs (any more than they want ..). When I do glob('mydir/*'), I want to find all of the contents of "mydir", and "mydir" itself is not contained in "mydir".

Even pathlib only emits the . when a recursive glob is indicated:

 cpython main @ next(pathlib.Path().glob('*'))
PosixPath('PC')
 cpython main @ next(pathlib.Path().glob('**'))
PosixPath('.')

I was talking about **, not *. The behaviour of standalone * pattern segments is well-defined: it always consumes one path segment.

In the case of glob.glob('tests/**', recursive=True), tests/ is emitted and that's equivalent to tests/.. Are you saying you think it's a bug that tests/ is emitted without a dot?

I'm saying that these two function calls should return the same result, as they do in pathlib:

>>> def myglob(pattern, root_dir=''):
...     for path in glob.glob(pattern, root_dir=root_dir, recursive=True):
...         yield os.path.join(root_dir, path)
... 
>>> list(myglob('testdir/**', ''))
['testdir/', 'testdir/a', 'testdir/b']
>>> list(myglob('**', 'testdir'))
['testdir/a', 'testdir/b']

Also, I'm unsure how it conflicts with the docs. Can you elaborate on where the conflict lies?

** is described as matching "any files and zero or more directories". You can see that, in the testdir/** example above, it can consume zero files and zero directories. But in the ** example, it doesn't consume the current directory.

[barneygale]

FWIW, there is no single standard for how ** should work. Different shells and programming languages implement it differently. See https://stackoverflow.com/questions/70613723/recursive-globs-or-globstar-different-on-zsh-bash-python-and-ruby

[barneygale]

FYI, I'm looking to add a new sep argument to fnmatch.translate() which would unify its pattern-matching with pathlib.PurePath.match() and pathlib.Path.glob(). In doing so it would make fnmatch.translate() more compatible with glob.glob(), but there would still be differences, such as the above-noted difference in how ** works, but also the default treatment of hidden files/directories. PR here:

• GH-72904: Add glob.translate() function #106703

[jaraco]

I'm very interested in reviewing the proposed change. I'm particularly interested to see if it obviates or conflicts with zipp.glob.

[barneygale]

Per the discussion in this thread, pathlib can't be made to preserve trailing slashes, and so pathlib.Path.glob() can't match glob.glob() behaviour completely, as far as I can tell.

[novas0x2a]

Path.glob and PurePath.match are also inconsistent with each other (test run is on linux)

$ find a -type f
a/thing0.a
a/lib/thing1.a
a/lib/sub/thing2.a

given this script:

#!/usr/bin/env python3

from pathlib import Path

pat = "a/**/*.a"
m = list(Path(".").glob(pat))
for p in m:
    print(p, p.match(pat))

$ python3.12  ./testglob.py
a/thing0.a False
a/lib/thing1.a True
a/lib/sub/thing2.a False

This is probably due to PurePath.match not officially implementing recursive glob support (i.e. ** is being interpreted as *) but it's confusing having some glob methods implement ** and some not. It says it implements a "glob-style pattern" which I took to mean "same implementation as python's other glob patterns" which seems to be incorrect.

[barneygale]

That'll be fixed in 3.13! #73435

[barneygale]

The bug where Path.glob() patterns ending ** only match directories is now fixed: #114684

The lack of documentation will be addressed shortly: #114030

Differences in trailing slash handling can't be addressed due to backwards-compatibility concerns (see my earlier comment)

I think the only other issue raised in this bug is the behaviour of glob.glob('**'), and so I'm going to remove the topic-pathlib label.

[barneygale]

(feel free to re-add it if there's something I've missed)

[barneygale]

I've just realised that the glob tests special-case "**" and "**/" patterns:

cpython/Lib/test/test_glob.py

Lines 72 to 75 in 3383d6a

	if pattern == '**' or pattern == '**' + os.sep:
	expected = res[1:]
	else:
	expected = res

In these two cases but no others, ** effectively means **/*, i.e. the returned paths must have at least one segment. I'm increasingly convinced that this is wrong.

Also different: glob.glob("", root_dir=path) returns nothing, but path.glob("") raises ValueError. I think they're both wrong, and instead the path should be yielded if it's a directory.

[serhiy-storchaka]

Let resolve some misunderstandings.

• glob.glob('**') returns dirs and files, non-recursively, not including .

** without recursive=True is interpreted as * (two subsequent *s, but this is the same). This is not an example for **.

Another point of reference, ls on Unix also treats ** as recursive without the dot:

ls does not interpolate patterns. The shell does this. By default ** is equivalent to *, which is expanded to dir toplevel.txt. ls dir produces indir.txt subdir, and ls toplevel.txt produces toplevel.txt. I do not know how did you get your result, except if you use very non-standard shell and ls.

In these two cases but no others, ** effectively means **/*, i.e. the returned paths must have at least one segment. I'm increasingly convinced that this is wrong.

The patch cannot have less than one component.

Also different: glob.glob("", root_dir=path) returns nothing, but path.glob("") raises ValueError. I think they're both wrong, and instead the path should be yielded if it's a directory.

No file or directory name matches pattern "", therefore the glob.glob() result is correct. As for path.glob("") it is a matter of question how much we want it to differ from glob.glob(). They cannot be completely equivalent for objective reasons, so we should decide what differences are deliberate. Raising an error is safer than producing a different result.

[barneygale]

Thanks for the clarifications

The patch cannot have less than one component.

pathlib.Path('.') is a zero-component path in pathlib. Its .parts is an empty tuple, whereas Path('foo').parts is a 1-element tuple.

os.path.relpath() needs to count components in order to generate the correct number of ../ components, and counts empty/dot paths as having zero components:

>>> os.path.relpath('foo', start='.')
'foo'
>>> os.path.relpath('foo', start='bar')
'../foo'

[barneygale]

FWIW, most of these differences highlighted in this thread are now either resolved or documented.

I'll open a PR to document that glob.glob(root_dir=root_dir) never returns an empty path (that could represent the root_dir), whereas pathlib.Path(root_dir).glob() can return the root_dir. I think that'll cover the remaining differences.