[BUG] lsof stuck in docker container for minutes #281

Janhouse · 2023-03-15T10:15:30Z

Describe the bug
I am unable to use lsof from within docker container (tried with ubuntu and fedora) on Archlinux host with latest stable kernel (tested with 6.2.6 and 6.2.5). If I run it within the container from root user, it gets stuck for at least a few minutes. After a few minutes it provides output as expected.

Not sure if this is because of Arch host, latest kernel or latest docker version but on Arch with latest kernel and docker I managed to reproduce it on different machines. Tested with ancient debian host and it worked fine there.

My guess it has something to do with kernel version and cgroups but that is just a guess. 🙂

On the host itself lsof works fine.

To Reproduce
Steps to reproduce the behavior:

Run docker run -it ubuntu or docker run -it fedora
Then install and run lsof with apt-get update && apt-get install lsof or in case of fedora yum install lsof
Run lsof from container as root without any parameters and see it get stuck and provide no output.

Expected behavior
It should not get stuck for minutes.

Program output
After a few minutes you get correct output.

Environment (please complete the following information):

Kernel: Linux 6.2.6
OS: Archlinux host
lsof Version: 4.98.0
Origin: Tried both installing from package manager and building from source

Additional context
Possibly related to #147

Running it with strace it seems to go into an endless loop of trying to close some handler and getting EBADF (Bad file descriptor)

strace lsof
execve("/usr/bin/lsof", ["lsof"], 0x7ffdb62088b0 /* 12 vars */) = 0
brk(NULL)                               = 0x561efcae3000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffd8e094d20) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f414a8ee000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=7951, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 7951, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f414a8ec000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=176800, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 181896, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a8bf000
mprotect(0x7f414a8c5000, 143360, PROT_NONE) = 0
mmap(0x7f414a8c5000, 110592, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f414a8c5000
mmap(0x7f414a8e0000, 28672, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7f414a8e0000
mmap(0x7f414a8e8000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f414a8e8000
mmap(0x7f414a8ea000, 5768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f414a8ea000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libtirpc.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=197280, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 195416, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a88f000
mmap(0x7f414a897000, 114688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f414a897000
mmap(0x7f414a8b3000, 36864, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24000) = 0x7f414a8b3000
mmap(0x7f414a8bc000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2c000) = 0x7f414a8bc000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320v\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=2224288, ...}, AT_EMPTY_PATH) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1953104, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a6b2000
mmap(0x7f414a6d8000, 1400832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f414a6d8000
mmap(0x7f414a82e000, 339968, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17c000) = 0x7f414a82e000
mmap(0x7f414a881000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1ce000) = 0x7f414a881000
mmap(0x7f414a887000, 32080, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f414a887000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=642072, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 639536, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a615000
mprotect(0x7f414a618000, 622592, PROT_NONE) = 0
mmap(0x7f414a618000, 442368, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f414a618000
mmap(0x7f414a684000, 176128, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6f000) = 0x7f414a684000
mmap(0x7f414a6b0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9a000) = 0x7f414a6b0000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libgssapi_krb5.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=359648, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 355096, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a5be000
mprotect(0x7f414a5ca000, 294912, PROT_NONE) = 0
mmap(0x7f414a5ca000, 241664, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) = 0x7f414a5ca000
mmap(0x7f414a605000, 49152, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x47000) = 0x7f414a605000
mmap(0x7f414a612000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x53000) = 0x7f414a612000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libkrb5.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=916064, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 902256, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a4e1000
mprotect(0x7f414a504000, 696320, PROT_NONE) = 0
mmap(0x7f414a504000, 438272, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f414a504000
mmap(0x7f414a56f000, 253952, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8e000) = 0x7f414a56f000
mmap(0x7f414a5ae000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xcc000) = 0x7f414a5ae000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libk5crypto.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=99976, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f414a4df000
mmap(NULL, 98416, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a4c6000
mmap(0x7f414a4cb000, 53248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f414a4cb000
mmap(0x7f414a4d8000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12000) = 0x7f414a4d8000
mmap(0x7f414a4dc000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f414a4dc000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=24496, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 24640, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a4bf000
mmap(0x7f414a4c1000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f414a4c1000
mmap(0x7f414a4c3000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f414a4c3000
mmap(0x7f414a4c4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f414a4c4000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libkrb5support.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=67312, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 66192, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a4ae000
mmap(0x7f414a4b2000, 32768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f414a4b2000
mmap(0x7f414a4ba000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) = 0x7f414a4ba000
mmap(0x7f414a4bd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x7f414a4bd000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libkeyutils.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=24704, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 24584, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a4a7000
mmap(0x7f414a4a9000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f414a4a9000
mmap(0x7f414a4ab000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f414a4ab000
mmap(0x7f414a4ac000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f414a4ac000
mmap(0x7f414a4ad000, 8, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f414a4ad000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libcrypto.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=4455088, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 4367384, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a07c000
mprotect(0x7f414a129000, 3284992, PROT_NONE) = 0
mmap(0x7f414a129000, 2453504, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xad000) = 0x7f414a129000
mmap(0x7f414a380000, 827392, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x304000) = 0x7f414a380000
mmap(0x7f414a44b000, 364544, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3ce000) = 0x7f414a44b000
mmap(0x7f414a4a4000, 9240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f414a4a4000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=67824, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 76360, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a069000
mprotect(0x7f414a06c000, 49152, PROT_NONE) = 0
mmap(0x7f414a06c000, 36864, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f414a06c000
mmap(0x7f414a075000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) = 0x7f414a075000
mmap(0x7f414a078000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x7f414a078000
mmap(0x7f414a07a000, 6728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f414a07a000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f414a067000
openat(AT_FDCWD, "/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=103296, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 102408, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f414a04d000
mmap(0x7f414a050000, 57344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f414a050000
mmap(0x7f414a05e000, 28672, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11000) = 0x7f414a05e000
mmap(0x7f414a065000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f414a065000
mmap(0x7f414a066000, 8, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f414a066000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f414a04b000
arch_prctl(ARCH_SET_FS, 0x7f414a04bc80) = 0
set_tid_address(0x7f414a04bf50)         = 66
set_robust_list(0x7f414a04bf60, 24)     = 0
rseq(0x7f414a04c5a0, 0x20, 0, 0x53053053) = 0
mprotect(0x7f414a881000, 16384, PROT_READ) = 0
mprotect(0x7f414a065000, 4096, PROT_READ) = 0
mprotect(0x7f414a078000, 4096, PROT_READ) = 0
mprotect(0x7f414a44b000, 352256, PROT_READ) = 0
mprotect(0x7f414a4ac000, 4096, PROT_READ) = 0
mprotect(0x7f414a6b0000, 4096, PROT_READ) = 0
mprotect(0x7f414a8e8000, 4096, PROT_READ) = 0
mprotect(0x7f414a4bd000, 4096, PROT_READ) = 0
mprotect(0x7f414a4c4000, 4096, PROT_READ) = 0
mprotect(0x7f414a4dc000, 8192, PROT_READ) = 0
mprotect(0x7f414a5ae000, 57344, PROT_READ) = 0
mprotect(0x7f414a612000, 8192, PROT_READ) = 0
mprotect(0x7f414a8bc000, 8192, PROT_READ) = 0
mprotect(0x561efc505000, 4096, PROT_READ) = 0
mprotect(0x7f414a923000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7f414a8ec000, 7951)            = 0
statfs("/sys/fs/selinux", 0x7ffd8e094ce0) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7ffd8e094ce0)      = -1 ENOENT (No such file or directory)
getrandom("\x8d\x51\x39\x84\x3d\x7e\xc7\x52", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0x561efcae3000
brk(0x561efcb04000)                     = 0x561efcb04000
openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(3, "nodev\tsysfs\nnodev\ttmpfs\nnodev\tbd"..., 1024) = 403
read(3, "", 1024)                       = 0
close(3)                                = 0
access("/etc/selinux/config", F_OK)     = -1 ENOENT (No such file or directory)
prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1073741816, rlim_max=1073741816}) = 0
close(3)                                = -1 EBADF (Bad file descriptor)
close(4)                                = -1 EBADF (Bad file descriptor)
close(5)                                = -1 EBADF (Bad file descriptor)
close(6)                                = -1 EBADF (Bad file descriptor)
close(7)                                = -1 EBADF (Bad file descriptor)
close(8)                                = -1 EBADF (Bad file descriptor)
close(9)                                = -1 EBADF (Bad file descriptor)
close(10)                               = -1 EBADF (Bad file descriptor)
close(11)                               = -1 EBADF (Bad file descriptor)
close(12)                               = -1 EBADF (Bad file descriptor)
close(13)                               = -1 EBADF (Bad file descriptor)
close(14)                               = -1 EBADF (Bad file descriptor)
close(15)                               = -1 EBADF (Bad file descriptor)
close(16)                               = -1 EBADF (Bad file descriptor)
close(17)                               = -1 EBADF (Bad file descriptor)
close(18)                               = -1 EBADF (Bad file descriptor)
close(19)                               = -1 EBADF (Bad file descriptor)
close(20)                               = -1 EBADF (Bad file descriptor)
close(21)                               = -1 EBADF (Bad file descriptor)
close(22)                               = -1 EBADF (Bad file descriptor)
close(23)                               = -1 EBADF (Bad file descriptor)
close(24)                               = -1 EBADF (Bad file descriptor)
close(25)                               = -1 EBADF (Bad file descriptor)
close(26)                               = -1 EBADF (Bad file descriptor)
close(27)                               = -1 EBADF (Bad file descriptor)
close(28)                               = -1 EBADF (Bad file descriptor)
close(29)                               = -1 EBADF (Bad file descriptor)
close(30)                               = -1 EBADF (Bad file descriptor)
close(31)                               = -1 EBADF (Bad file descriptor)
close(32)                               = -1 EBADF (Bad file descriptor)
close(33)                               = -1 EBADF (Bad file descriptor)
close(34)                               = -1 EBADF (Bad file descriptor)
close(35)                               = -1 EBADF (Bad file descriptor)
close(36)                               = -1 EBADF (Bad file descriptor)
close(37)                               = -1 EBADF (Bad file descriptor)
close(38)                               = -1 EBADF (Bad file descriptor)
close(39)                               = -1 EBADF (Bad file descriptor)
close(40)                               = -1 EBADF (Bad file descriptor)
close(41)                               = -1 EBADF (Bad file descriptor)
close(42)                               = -1 EBADF (Bad file descriptor)
close(43)                               = -1 EBADF (Bad file descriptor)
close(44)                               = -1 EBADF (Bad file descriptor)
close(45)                               = -1 EBADF (Bad file descriptor)
close(46)                               = -1 EBADF (Bad file descriptor)
close(47)                               = -1 EBADF (Bad file descriptor)
close(48)                               = -1 EBADF (Bad file descriptor)
close(49)                               = -1 EBADF (Bad file descriptor)
close(50)                               = -1 EBADF (Bad file descriptor)
close(51)                               = -1 EBADF (Bad file descriptor)
close(52)                               = -1 EBADF (Bad file descriptor)
close(53)                               = -1 EBADF (Bad file descriptor)
close(54)                               = -1 EBADF (Bad file descriptor)
close(55)                               = -1 EBADF (Bad file descriptor)
close(56)                               = -1 EBADF (Bad file descriptor)
close(57)                               = -1 EBADF (Bad file descriptor)
close(58)                               = -1 EBADF (Bad file descriptor)
close(59)                               = -1 EBADF (Bad file descriptor)
close(60)                               = -1 EBADF (Bad file descriptor)
close(61)                               = -1 EBADF (Bad file descriptor)
close(62)                               = -1 EBADF (Bad file descriptor)
close(63)                               = -1 EBADF (Bad file descriptor)
close(64)                               = -1 EBADF (Bad file descriptor)
close(65)                               = -1 EBADF (Bad file descriptor)
close(66)                               = -1 EBADF (Bad file descriptor)
close(67)                               = -1 EBADF (Bad file descriptor)
close(68)                               = -1 EBADF (Bad file descriptor)
close(69)                               = -1 EBADF (Bad file descriptor)
close(70)                               = -1 EBADF (Bad file descriptor)
close(71)                               = -1 EBADF (Bad file descriptor)
close(72)                               = -1 EBADF (Bad file descriptor)
close(73)                               = -1 EBADF (Bad file descriptor)
close(74)                               = -1 EBADF (Bad file descriptor)
close(75)                               = -1 EBADF (Bad file descriptor)
close(76)                               = -1 EBADF (Bad file descriptor)
close(77)                               = -1 EBADF (Bad file descriptor)
close(78)                               = -1 EBADF (Bad file descriptor)
close(79)                               = -1 EBADF (Bad file descriptor)
close(80)                               = -1 EBADF (Bad file descriptor)
close(81)                               = -1 EBADF (Bad file descriptor)
close(82)                               = -1 EBADF (Bad file descriptor)
close(83)                               = -1 EBADF (Bad file descriptor)
close(84)                               = -1 EBADF (Bad file descriptor)
close(85)                               = -1 EBADF (Bad file descriptor)
close(86)                               = -1 EBADF (Bad file descriptor)
close(87)                               = -1 EBADF (Bad file descriptor)
close(88)                               = -1 EBADF (Bad file descriptor)
close(89)                               = -1 EBADF (Bad file descriptor)

The text was updated successfully, but these errors were encountered:

jiegec · 2023-03-16T00:24:39Z

I cannot reproduce the bug running in Ubuntu 22.04 docker. Can you provide full strace log?

jiegec · 2023-03-16T02:30:55Z

The EBADF error is expected:

lsof/main.c

Lines 115 to 132 in 6292bfc

    
               if ((MaxFd = (int)GET_MAX_FD()) < 53) 
        
                   MaxFd = 53; 
        
           #if defined(HAS_CLOSEFROM) 
        
               (void)closefrom(3); 
        
           #else /* !defined(HAS_CLOSEFROM) */ 
        
           #    if defined(SYS_close_range) 
        
               if (MaxFd > 3 && syscall(SYS_close_range, 3, MaxFd - 1, 0) == 0) 
        
                   goto closed; 
        
           #    endif 
        
               for (i = 3; i < MaxFd; i++) 
        
                   (void)close(i); 
        
           #    if defined(SYS_close_range) 
        
           closed: 
        
           #    endif 
        
           #endif /* !defined(HAS_CLOSEFROM) */ 
        
               while (((i = open("/dev/null", O_RDWR, 0)) >= 0) && (i < 2))

x-yuri · 2023-03-29T06:40:45Z

@jiegec I never waited until it finishes. Probably because I have bigger pid_max. Steps to reproduce:

$ docker run --rm nginx:alpine
$ docker exec --privileged -it `docker ps -q | head -1` sh
/ # apk add strace lsof
/ # strace -fp1 lsof
strace: Process 1 attached
[pid     1] rt_sigsuspend([], 8 <unfinished ...>
[pid   100] execve("/usr/bin/lsof", ["lsof"], 0x7fff9ca1d308 /* 9 vars */) = 0
[pid   100] arch_prctl(ARCH_SET_FS, 0x7f19d2cc8b48) = 0
[pid   100] set_tid_address(0x7f19d2cc8fb0) = 100
[pid   100] brk(NULL)                   = 0x55791ae48000
[pid   100] brk(0x55791ae4a000)         = 0x55791ae4a000
[pid   100] mmap(0x55791ae48000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x55791ae48000
[pid   100] mprotect(0x7f19d2cc5000, 4096, PROT_READ) = 0
[pid   100] mprotect(0x557919b6d000, 4096, PROT_READ) = 0
[pid   100] prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1073741816, rlim_max=1073741816}) = 0
[pid   100] close_range(3, 1073741815, 0) = 0
[pid   100] open("/dev/null", O_RDWR|O_LARGEFILE) = 3
[pid   100] close(3)                    = 0
[pid   100] umask(000)                  = 022
[pid   100] getpid()                    = 100
[pid   100] getgid()                    = 0
[pid   100] getegid()                   = 0
[pid   100] geteuid()                   = 0
[pid   100] getuid()                    = 0
[pid   100] mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19d2c2b000
[pid   100] stat("/dev", {st_mode=S_IFDIR|0755, st_size=340, ...}) = 0
[pid   100] open("/", O_RDONLY|O_LARGEFILE) = 3
[pid   100] lseek(3, 1, SEEK_SET)       = 1
[pid   100] lstat("/proc/100/fd/3", {st_mode=S_IFLNK|0500, st_size=64, ...}) = 0
[pid   100] open("/proc/100/fdinfo/3", O_RDONLY|O_LARGEFILE) = 4
[pid   100] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19d2c2a000
[pid   100] read(4, "pos:\t1\nflags:\t0100000\nmnt_id:\t35"..., 1024) = 47
[pid   100] lseek(4, -40, SEEK_CUR)     = 7
[pid   100] close(4)                    = 0
[pid   100] munmap(0x7f19d2c2a000, 4096) = 0
[pid   100] close(3)                    = 0
[pid   100] open("/proc/mounts", O_RDONLY|O_LARGEFILE) = 3
[pid   100] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19d2c2a000
[pid   100] read(3, "overlay / overlay rw,relatime,lo"..., 4088) = 2115
[pid   100] pipe([4, 5])                = 0
[pid   100] pipe([6, 7])                = 0
[pid   100] rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [], 8) = 0
[pid   100] rt_sigprocmask(SIG_BLOCK, ~[], ~[KILL STOP RTMIN RT_1 RT_2], 8) = 0
[pid   100] fork(strace: Process 101 attached
)                      = 101
[pid   101] gettid()                    = 101
[pid   100] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1 RT_2],  <unfinished ...>
[pid   101] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1 RT_2],  <unfinished ...>
[pid   100] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid   101] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid   100] rt_sigprocmask(SIG_SETMASK, [],  <unfinished ...>
[pid   101] rt_sigprocmask(SIG_SETMASK, [],  <unfinished ...>
[pid   100] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid   101] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid   100] close(4 <unfinished ...>
[pid   101] close(0 <unfinished ...>
[pid   100] <... close resumed>)        = 0
[pid   101] <... close resumed>)        = 0
[pid   100] close(7 <unfinished ...>
[pid   101] close(1 <unfinished ...>
[pid   100] <... close resumed>)        = 0
[pid   101] <... close resumed>)        = 0
[pid   100] rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2],  <unfinished ...>
[pid   101] close(2 <unfinished ...>
[pid   100] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid   101] <... close resumed>)        = 0
[pid   100] rt_sigaction(SIGALRM, {sa_handler=0x557919b5f4b6, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f19d2c78acd},  <unfinished ...>
[pid   101] close(3 <unfinished ...>
[pid   100] <... rt_sigaction resumed>{sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
[pid   101] <... close resumed>)        = 0
[pid   100] setitimer(ITIMER_REAL, {it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=15, tv_usec=0}},  <unfinished ...>
[pid   101] close(5 <unfinished ...>
[pid   100] <... setitimer resumed>{it_interval={tv_sec=0, tv_usec=0}, it_value={tv_sec=0, tv_usec=0}}) = 0
[pid   101] <... close resumed>)        = 0
[pid   100] write(5, "\315\364\265\31yU\0\0", 8 <unfinished ...>
[pid   101] close(6 <unfinished ...>
[pid   100] <... write resumed>)        = 8
[pid   101] <... close resumed>)        = 0
[pid   100] write(5, "\2\0\0\0", 4 <unfinished ...>
[pid   101] close(8 <unfinished ...>
[pid   100] <... write resumed>)        = 4
[pid   101] <... close resumed>)        = -1 EBADF (Bad file descriptor)
[pid   100] write(5, "/\0", 2 <unfinished ...>
[pid   101] close(9 <unfinished ...>
[pid   100] <... write resumed>)        = 2
[pid   101] <... close resumed>)        = -1 EBADF (Bad file descriptor)
[pid   100] write(5, "\0\20\0\0", 4 <unfinished ...>
[pid   101] close(10 <unfinished ...>
[pid   100] <... write resumed>)        = 4
[pid   101] <... close resumed>)        = -1 EBADF (Bad file descriptor)
[pid   100] read(6,  <unfinished ...>
[pid   101] close(11)                   = -1 EBADF (Bad file descriptor)
[pid   101] close(12)                   = -1 EBADF (Bad file descriptor)
[pid   101] close(13)                   = -1 EBADF (Bad file descriptor)
[pid   101] close(14)                   = -1 EBADF (Bad file descriptor)
[pid   101] close(15)                   = -1 EBADF (Bad file descriptor)
[pid   101] close(16)                   = -1 EBADF (Bad file descriptor)
[pid   101] close(17)                   = -1 EBADF (Bad file descriptor)
[pid   101] close(18)                   = -1 EBADF (Bad file descriptor)
[pid   101] close(19)                   = -1 EBADF (Bad file descriptor)
...
[pid   101] close(1141630)              = -1 EBADF (Bad file descriptor)
[pid   101] close(1141631)              = -1 EBADF (Bad file descriptor)
[pid   101] close(1141632)              = -1 EBADF (Bad file descriptor)
[pid   101] close(1141633)              = -1 EBADF (Bad file descriptor)
[pid   101] close(1141634)              = -1 EBADF (Bad file descriptor)
[pid   101] close(1141635)              = -1 EBADF (Bad file descriptor)
[pid   101] close(1141636)              = -1 EBADF (Bad file descriptor)
[pid   101] close(1141637)              = -1 EBADF (Bad file descriptor)
[pid   101] close(1141638)              = -1 EBADF (Bad file descriptor)
[pid   101] close(1141639strace: Process 1 detached
strace: Process 100 detached
strace: Process 101 detached
 <detached ...>

I'm running Arch Linux. lsof -b helps. Can you possibly explain why it tries to close all descriptors above 2? And why doesn't it seem to do so with -b? At first glance, it doesn't seem like -b should affect it.

jiegec · 2023-03-29T06:54:15Z

@x-yuri Nice! Actually I dunno why either.. The code was written decades ago. Anyways, this should be fixed.

x-yuri · 2023-03-29T07:05:47Z

Let's just drop it, while nobody's looking then? :) Joking aside, I guess it tries to avoid affecting the output, although I'm not sure I can say anything more specific on the matter (some specific cases).

The other mystery is what -b has to do with it?.. I now see, that w/ -b it doesn't start a child, and only the child seems to loop over the pids. Or so it looks.

jiegec · 2023-03-29T07:17:30Z

I tried building locally in alpine docker, here is the strace output:

execve("./lsof", ["./lsof"], 0x7ffd1d4dd3b0 /* 7 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x7fa55a45fb48) = 0
set_tid_address(0x7fa55a45ffb0)         = 3605
brk(NULL)                               = 0x5630462ef000
brk(0x5630462f1000)                     = 0x5630462f1000
mmap(0x5630462ef000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x5630462ef000
mprotect(0x7fa55a45c000, 4096, PROT_READ) = 0
mprotect(0x563045560000, 4096, PROT_READ) = 0
prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0
close_range(3, 1048575, 0)              = 0

Which uses the close_range syscall instead of calling close in a loop. So the compilation environment matters.

UPDATE: There is another close loop in misc.c:

lsof/misc.c

Lines 314 to 330 in 6292bfc

    
           for (fd = 0; fd < MaxFd; fd++) { 
        
               if (fd == Pipes[0] || fd == Pipes[3]) 
        
                   continue; 
        
               (void)close(fd); 
        
               if (fd == Pipes[1]) 
        
                   Pipes[1] = -1; 
        
               else if (fd == Pipes[2]) 
        
                   Pipes[2] = -1; 
        
           } 
        
           if (Pipes[1] >= 0) { 
        
               (void)close(Pipes[1]); 
        
               Pipes[1] = -1; 
        
           } 
        
           if (Pipes[2] >= 0) { 
        
               (void)close(Pipes[2]); 
        
               Pipes[2] = -1; 
        
           }

This is the culprit.

x-yuri · 2023-03-29T07:45:56Z

I tried building locally in alpine docker, here is the strace output

Now, how did you do that?

$ docker run --rm -it alpine:3.17 sh
/ # apk add curl linux-headers build-base strace
/ # curl -L https://github.com/lsof-org/lsof/archive/4.98.0/lsof-4.98.0.tar.gz -o lsof-4.98.0.tar.gz
/ # tar xf lsof-4.98.0.tar.gz
/ # cd lsof-4.98.0
/lsof-4.98.0 # ./Configure -n linux
/lsof-4.98.0 # make
/lsof-4.98.0 # strace -f ./lsof
...
[pid   249] close(10)                   = -1 EBADF (Bad file descriptor)
[pid   249] close(11)                   = -1 EBADF (Bad file descriptor)
...

Here's how they supposedly build lsof.

jiegec · 2023-03-29T08:23:00Z

@x-yuri I pushed a fix in master, however it seemed that musl does not provide closefrom() call, so an additional check to SYS_close_range might be required.

x-yuri · 2023-03-29T08:37:30Z

But that means the issue shouldn't be closed yet?

On a side note, can you possibly explain what I'm missing when building from a cloned repository?

$ apk add build-base git
$ git clone https://github.com/lsof-org/lsof
$ ./Configure -n linux
$ make
(cd lib; make DEBUG="" CFGF="-DLINUXV=61012 -DHASIPv6 -DHASNORPC_H -DNEEDS_NETINET_TCPH -DHAS_DUP2 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DHAS_STRFTIME -DLSOF_VSTR=\"6.1.12-arch1-1\"")
make[1]: Entering directory '/lsof/lib'
cc  -DLINUXV=61012 -DHASIPv6 -DHASNORPC_H -DNEEDS_NETINET_TCPH -DHAS_DUP2 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DHAS_STRFTIME -DLSOF_VSTR="6.1.12-arch1-1"  -I.. -I../include -Idialects/linux    -c -o ckkv.o ckkv.c
cc  -DLINUXV=61012 -DHASIPv6 -DHASNORPC_H -DNEEDS_NETINET_TCPH -DHAS_DUP2 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DHAS_STRFTIME -DLSOF_VSTR="6.1.12-arch1-1"  -I.. -I../include -Idialects/linux    -c -o cvfs.o cvfs.c
cvfs.c:42:10: fatal error: ../machine.h: No such file or directory
   42 | #include "../machine.h"
      |          ^~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [<builtin>: cvfs.o] Error 1
make[1]: Leaving directory '/lsof/lib'
make: *** [Makefile:96: lib/liblsof.a] Error 2

UPD Created a separate issue (#283).

jiegec · 2023-03-29T10:00:16Z

Done in commit 124ab0f

MarcT512 · 2024-05-15T16:04:23Z

The fix above does not help for platforms which do not have closefrom or close_range (eg: SuSE SLES 15).
I have submitted a patch via #316 to optimize the slow path on Linux.

jiegec closed this as completed in 44d61e8 Mar 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] lsof stuck in docker container for minutes #281

[BUG] lsof stuck in docker container for minutes #281

Janhouse commented Mar 15, 2023

jiegec commented Mar 16, 2023

jiegec commented Mar 16, 2023

x-yuri commented Mar 29, 2023 •

edited

Loading

jiegec commented Mar 29, 2023 •

edited

Loading

x-yuri commented Mar 29, 2023

jiegec commented Mar 29, 2023 •

edited

Loading

x-yuri commented Mar 29, 2023

jiegec commented Mar 29, 2023

x-yuri commented Mar 29, 2023 •

edited

Loading

jiegec commented Mar 29, 2023

MarcT512 commented May 15, 2024

[BUG] lsof stuck in docker container for minutes #281

[BUG] lsof stuck in docker container for minutes #281

Comments

Janhouse commented Mar 15, 2023

jiegec commented Mar 16, 2023

jiegec commented Mar 16, 2023

x-yuri commented Mar 29, 2023 • edited Loading

jiegec commented Mar 29, 2023 • edited Loading

x-yuri commented Mar 29, 2023

jiegec commented Mar 29, 2023 • edited Loading

x-yuri commented Mar 29, 2023

jiegec commented Mar 29, 2023

x-yuri commented Mar 29, 2023 • edited Loading

jiegec commented Mar 29, 2023

MarcT512 commented May 15, 2024

x-yuri commented Mar 29, 2023 •

edited

Loading

jiegec commented Mar 29, 2023 •

edited

Loading

jiegec commented Mar 29, 2023 •

edited

Loading

x-yuri commented Mar 29, 2023 •

edited

Loading