Content of deleted directory still visible

[yuyichao]

ArchLinux, x86_64, kernel 5.14.7, latest git version of fuse-overlayfs

mkdir -p {overlay,mount,upper,lower/v/A}
fuse-overlayfs -o "lowerdir=lower,upperdir=upper,workdir=overlay" "mount"

rm -rf "mount/v/"
mkdir -p "mount/v/"
ls -alF "mount/v/"
stat mount/v/A

The last stat is supposed to error since mount/v should be empty, as verified by the ls yet running the script above prints something like,

total 0
drwxr-xr-x 2 yuyichao yuyichao 40 Sep 28 07:24 ./
drwxr-xr-x 3 yuyichao yuyichao  2 Sep 28 07:24 ../
  File: mount/v/A
  Size: 0               Blocks: 0          IO Block: 4096   directory
Device: 0,73    Inode: 46969979    Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 1000/yuyichao)   Gid: ( 1000/yuyichao)
Access: 2021-09-28 07:24:46.247284663 -0400
Modify: 2021-09-28 07:24:46.243951278 -0400
Change: 2021-09-28 07:24:46.243951278 -0400
 Birth: -

which shows that the ls indeed see an empty directory but stat can still see something in it.

[yuyichao]

Output with -d

uid=unchanged
gid=unchanged
upperdir=/home/yuyichao/workspace/overlay/upper
workdir=overlay
lowerdir=lower
mountpoint=mount
plugins=<none>
fsync=enabled
FUSE library version: 3.10.5
unique: 2, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.34
flags=0x33fffffb
max_readahead=0x00020000
   INIT: 7.31
   flags=0x0051f069
   max_readahead=0x00020000
   max_write=0x00100000
   max_background=0
   congestion_threshold=0
   time_gran=1
   unique: 2, success, outsize: 80
unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 1849
ovl_getattr(ino=1)
   unique: 4, success, outsize: 120
unique: 6, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 1849
   unique: 6, success, outsize: 96
unique: 8, opcode: LOOKUP (1), nodeid: 1, insize: 42, pid: 1206011
ovl_lookup(parent=1, name=v)
   unique: 8, success, outsize: 144
unique: 10, opcode: OPENDIR (27), nodeid: 94061144948480, insize: 48, pid: 1206011
ovl_opendir(ino=94061144948480)
   unique: 10, success, outsize: 32
unique: 12, opcode: READDIRPLUS (44), nodeid: 94061144948480, insize: 80, pid: 1206011
ovl_readdirplus(ino=94061144948480, size=4096, offset=0)
   unique: 12, success, outsize: 496
unique: 14, opcode: RELEASEDIR (29), nodeid: 94061144948480, insize: 64, pid: 0
ovl_releasedir(ino=94061144948480)
   unique: 14, success, outsize: 16
unique: 16, opcode: OPENDIR (27), nodeid: 94061144948480, insize: 48, pid: 1206011
ovl_opendir(ino=94061144948480)
   unique: 16, success, outsize: 32
unique: 18, opcode: GETATTR (3), nodeid: 94061144948480, insize: 56, pid: 1206011
ovl_getattr(ino=94061144948480)
   unique: 18, success, outsize: 120
unique: 20, opcode: READDIRPLUS (44), nodeid: 94061144948480, insize: 80, pid: 1206011
ovl_readdirplus(ino=94061144948480, size=4096, offset=0)
   unique: 20, success, outsize: 496
unique: 22, opcode: READDIR (28), nodeid: 94061144948480, insize: 80, pid: 1206011
ovl_readdir(ino=94061144948480, size=4096, offset=3)
   unique: 22, success, outsize: 16
unique: 24, opcode: OPENDIR (27), nodeid: 94061144917680, insize: 48, pid: 1206011
ovl_opendir(ino=94061144917680)
   unique: 24, success, outsize: 32
unique: 26, opcode: READDIRPLUS (44), nodeid: 94061144917680, insize: 80, pid: 1206011
ovl_readdirplus(ino=94061144917680, size=4096, offset=0)
   unique: 26, success, outsize: 336
unique: 28, opcode: READDIR (28), nodeid: 94061144917680, insize: 80, pid: 1206011
ovl_readdir(ino=94061144917680, size=4096, offset=2)
   unique: 28, success, outsize: 16
unique: 30, opcode: RELEASEDIR (29), nodeid: 94061144917680, insize: 64, pid: 0
ovl_releasedir(ino=94061144917680)
   unique: 30, success, outsize: 16
unique: 32, opcode: RMDIR (11), nodeid: 94061144948480, insize: 42, pid: 1206011
ovl_rmdir(parent=94061144948480, name=A)
   unique: 32, success, outsize: 16
unique: 34, opcode: FORGET (2), nodeid: 94061144917680, insize: 48, pid: 0
ovl_forget(ino=94061144917680, nlookup=2)
unique: 36, opcode: RELEASEDIR (29), nodeid: 94061144948480, insize: 64, pid: 0
ovl_releasedir(ino=94061144948480)
   unique: 36, success, outsize: 16
unique: 38, opcode: RMDIR (11), nodeid: 1, insize: 42, pid: 1206011
ovl_rmdir(parent=1, name=v)
   unique: 38, success, outsize: 16
unique: 40, opcode: FORGET (2), nodeid: 94061144948480, insize: 48, pid: 0
ovl_forget(ino=94061144948480, nlookup=1)
unique: 42, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 1206029
ovl_getattr(ino=1)
   unique: 42, success, outsize: 120
unique: 44, opcode: LOOKUP (1), nodeid: 1, insize: 42, pid: 1206029
ovl_lookup(parent=1, name=v)
   unique: 44, success, outsize: 144
unique: 46, opcode: MKDIR (9), nodeid: 1, insize: 50, pid: 1206029
ovl_mkdir(ino=1, name=v, mode=511)
   unique: 46, success, outsize: 144


unique: 48, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 1206045
ovl_getattr(ino=1)
   unique: 48, success, outsize: 120
unique: 50, opcode: GETXATTR (22), nodeid: 94061144915408, insize: 72, pid: 1206045
ovl_getxattr(ino=94061144915408, name=system.posix_acl_access, size=4096)
   unique: 50, error: -61 (No data available), outsize: 16
unique: 52, opcode: GETXATTR (22), nodeid: 94061144915408, insize: 73, pid: 1206045
ovl_getxattr(ino=94061144915408, name=system.posix_acl_default, size=4096)
   unique: 52, error: -61 (No data available), outsize: 16
unique: 54, opcode: OPENDIR (27), nodeid: 94061144915408, insize: 48, pid: 1206045
ovl_opendir(ino=94061144915408)
   unique: 54, success, outsize: 32
unique: 56, opcode: READDIRPLUS (44), nodeid: 94061144915408, insize: 80, pid: 1206045
ovl_readdirplus(ino=94061144915408, size=4096, offset=0)
   unique: 56, success, outsize: 336
unique: 58, opcode: GETXATTR (22), nodeid: 1, insize: 72, pid: 1206045
ovl_getxattr(ino=1, name=system.posix_acl_access, size=4096)
   unique: 58, error: -61 (No data available), outsize: 16
unique: 60, opcode: GETXATTR (22), nodeid: 1, insize: 73, pid: 1206045
ovl_getxattr(ino=1, name=system.posix_acl_default, size=4096)
   unique: 60, error: -61 (No data available), outsize: 16
unique: 62, opcode: READDIR (28), nodeid: 94061144915408, insize: 80, pid: 1206045
ovl_readdir(ino=94061144915408, size=4096, offset=2)
   unique: 62, success, outsize: 16
unique: 64, opcode: RELEASEDIR (29), nodeid: 94061144915408, insize: 64, pid: 0
ovl_releasedir(ino=94061144915408)
   unique: 64, success, outsize: 16


unique: 66, opcode: LOOKUP (1), nodeid: 94061144915408, insize: 42, pid: 1206063
ovl_lookup(parent=94061144915408, name=A)
   unique: 66, success, outsize: 144

I've inserted two blank lines in the log before running the ls and stat commands.

[yuyichao]

Oh, also just realized that if I didn't do a ls (i.e. readdir) then it's fine. This was probably also why it was a bit hard for me to debug this initially since I occasionally couldn't reproduce it...

[yuyichao]

In the version that works, do_lookup_file returned a NULL node whereas in the broken version it returns a non-NULL node ->whiteout == 0...

[bowlofeggs]

I can confirm this issue - I experience it frequently. I use podman and I have a gentoo container that frequently fails with error messages about how portage tried to delete folders but couldn't.