How to prevent xpra (python?) from hanging when NFS server unresponsive?

[AGI-chandler]

Ok we are having an emergency now ... NFS server for home directories is suddenly not responding. I need to start firefox on a computer in the same network so I can access the IPMI interface and console of the NFS server. I can log into this computer using the emergency local account, but I can't even run something like xpra list, it just hangs, I'm guessing due to the unresponsive NFS mounts. However, this local account is separate from the NFS mounts. I guess since xpra is running python, then it has something to do with python trying to access the NFS mounts? Any ideas how to prevent this so I can run xpra? Thanks

[totaam]

See --socket-dirs and remove any entries that are on NFS drives.
See also xpra showconfig | grep socket-dir.

[totaam]

I guess showconfig does some path probing, don't use it then and just use '--socket-dirs=$XDG_RUNTIME_DIR/xpra' with all your commands.
Or even /tmp if you have to.

If that works then make the setting permanent.

[AGI-chandler]

Unfortunately that hangs too, I tried all 4 of these commands (assume you meant $XDG_RUNTIME_DIR):

xpra --socket-dirs=$XDG_RUNTIME_DIR/xpra showconfig
xpra --socket-dirs=$XDG_RUNTIME_DIR/xpra list
xpra --socket-dirs=/tmp list
xpra --socket-dirs=/tmp showconfig

Is there perhaps a python equivalent to bash's set -x command that shows everything being done?

[AGI-chandler]

Oh and here XDG_RUNTIME_DIR=/run/user/1003 which is a local filesystem, and /tmp is local too...

[totaam]

Try with strace and see where it hangs.

[AGI-chandler]

Awesome, thanks!  Yes can see 2755 lines of output from xpra list command 😵  I don't think I need to include all of it?
It's indeed hanging on stat("/wing2", without even finishing that line (this is one of the unreachable NFS directories).
Not too many lines before that I see references to the xpra code, so I'll include from there until it hangs below, does it help?

$ strace xpra list

lines 2,646 to 2,755 of output

2646 stat("/home/agi/.config/xpra", {st_mode=S_IFDIR|0700, st_size=23, ...}) = 0
2647 stat("/home/agi/.config/xpra", {st_mode=S_IFDIR|0700, st_size=23, ...}) = 0
2648 stat("/home/agi/.config/xpra/conf.d", 0x7fffffffaba0) = -1 ENOENT (No such file or directory)
2649 stat("/home/agi/.config/xpra/xpra.conf", {st_mode=S_IFREG|0664, st_size=291, ...}) = 0
2650 stat("/home/agi/.config/xpra/xpra.conf", {st_mode=S_IFREG|0664, st_size=291, ...}) = 0
2651 stat("/home/agi/.config/xpra/xpra.conf", {st_mode=S_IFREG|0664, st_size=291, ...}) = 0
2652 stat("/home/agi/.config/xpra/xpra.conf", {st_mode=S_IFREG|0664, st_size=291, ...}) = 0
2653 openat(AT_FDCWD, "/home/agi/.config/xpra/xpra.conf", O_RDONLY|O_CLOEXEC) = 4
2654 fstat(4, {st_mode=S_IFREG|0664, st_size=291, ...}) = 0
2655 ioctl(4, TCGETS, 0x7ffffffface0)        = -1 ENOTTY (Inappropriate ioctl for device)
2656 lseek(4, 0, SEEK_CUR)                   = 0
2657 lseek(4, 0, SEEK_CUR)                   = 0
2658 read(4, "# xpra user configuration file\n#"..., 8192) = 291
2659 read(4, "", 8192)                       = 0
2660 close(4)                                = 0
2661 getuid()                                = 1003
2662 stat("/home/agi/.config/xpra", {st_mode=S_IFDIR|0700, st_size=23, ...}) = 0
2663 stat("/home/agi/.config/xpra", {st_mode=S_IFDIR|0700, st_size=23, ...}) = 0
2664 stat("/home/agi/.config/xpra/conf.d", 0x7fffffffad00) = -1 ENOENT (No such file or directory)
2665 stat("/home/agi/.config/xpra/xpra.conf", {st_mode=S_IFREG|0664, st_size=291, ...}) = 0
2666 stat("/home/agi/.config/xpra/xpra.conf", {st_mode=S_IFREG|0664, st_size=291, ...}) = 0
2667 stat("/usr/lib64/python3.6/site-packages/xpra/platform", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2668 stat("/usr/lib64/python3.6/site-packages/xpra/platform/features.py", {st_mode=S_IFREG|0644, st_size=2990, ...}) = 0
2669 stat("/usr/lib64/python3.6/site-packages/xpra/platform/features.py", {st_mode=S_IFREG|0644, st_size=2990, ...}) = 0
2670 openat(AT_FDCWD, "/usr/lib64/python3.6/site-packages/xpra/platform/__pycache__/features.cpython-36.pyc", O_RDONLY|O_CLOEXEC) = 4
2671 fstat(4, {st_mode=S_IFREG|0644, st_size=2035, ...}) = 0
2672 lseek(4, 0, SEEK_CUR)                   = 0
2673 fstat(4, {st_mode=S_IFREG|0644, st_size=2035, ...}) = 0
2674 read(4, "3\r\r\n\367T8c\256\v\0\0\343\0\0\0\0\0\0\0\0\0\0\0\0\23\0\0\0@\0\0"..., 2036) = 2035
2675 read(4, "", 1)                          = 0
2676 close(4)                                = 0
2677 stat("/usr/lib64/python3.6/site-packages/xpra/platform/xposix", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2678 stat("/usr/lib64/python3.6/site-packages/xpra/platform/xposix/features.py", {st_mode=S_IFREG|0644, st_size=1807, ...}) = 0
2679 stat("/usr/lib64/python3.6/site-packages/xpra/platform/xposix/features.py", {st_mode=S_IFREG|0644, st_size=1807, ...}) = 0
2680 openat(AT_FDCWD, "/usr/lib64/python3.6/site-packages/xpra/platform/xposix/__pycache__/features.cpython-36.pyc", O_RDONLY|O_CLOEXEC) = 4
2681 fstat(4, {st_mode=S_IFREG|0644, st_size=1359, ...}) = 0
2682 lseek(4, 0, SEEK_CUR)                   = 0
2683 fstat(4, {st_mode=S_IFREG|0644, st_size=1359, ...}) = 0
2684 read(4, "3\r\r\n\367T8c\17\7\0\0\343\0\0\0\0\0\0\0\0\0\0\0\0\22\0\0\0@\0\0"..., 1360) = 1359
2685 read(4, "", 1)                          = 0
2686 close(4)                                = 0
2686 close(4)                                = 0
2687 stat("/usr/lib64/python3.6/site-packages/xpra/platform", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2688 stat("/usr/lib64/python3.6/site-packages/xpra/platform/info.py", {st_mode=S_IFREG|0644, st_size=2495, ...}) = 0
2689 stat("/usr/lib64/python3.6/site-packages/xpra/platform/info.py", {st_mode=S_IFREG|0644, st_size=2495, ...}) = 0
2690 openat(AT_FDCWD, "/usr/lib64/python3.6/site-packages/xpra/platform/__pycache__/info.cpython-36.pyc", O_RDONLY|O_CLOEXEC) = 4
2691 fstat(4, {st_mode=S_IFREG|0644, st_size=2174, ...}) = 0
2692 lseek(4, 0, SEEK_CUR)                   = 0
2693 fstat(4, {st_mode=S_IFREG|0644, st_size=2174, ...}) = 0
2694 read(4, "3\r\r\n\367T8c\277\t\0\0\343\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0@\0\0"..., 2175) = 2174
2695 read(4, "", 1)                          = 0
2696 close(4)                                = 0
2697 stat("/usr/lib64/python3.6/site-packages/xpra/platform/xposix", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2698 stat("/usr/lib64/python3.6/site-packages/xpra/platform/xposix/info.py", {st_mode=S_IFREG|0644, st_size=219, ...}) = 0
2699 stat("/usr/lib64/python3.6/site-packages/xpra/platform/xposix/info.py", {st_mode=S_IFREG|0644, st_size=219, ...}) = 0
2700 openat(AT_FDCWD, "/usr/lib64/python3.6/site-packages/xpra/platform/xposix/__pycache__/info.cpython-36.pyc", O_RDONLY|O_CLOEXEC) = 4
2701 fstat(4, {st_mode=S_IFREG|0644, st_size=111, ...}) = 0
2702 lseek(4, 0, SEEK_CUR)                   = 0
2703 fstat(4, {st_mode=S_IFREG|0644, st_size=111, ...}) = 0
2704 read(4, "3\r\r\n\350T8c\333\0\0\0\343\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0@\0\0"..., 112) = 111
2705 read(4, "", 1)                          = 0
2706 close(4)                                = 0
2707 getuid()                                = 1003
2708 fstat(3, {st_mode=S_IFREG|0664, st_size=9253600, ...}) = 0
2709 stat("/etc/xpra", {st_mode=S_IFDIR|0755, st_size=290, ...}) = 0
2710 stat("/etc/os-release", {st_mode=S_IFREG|0644, st_size=333, ...}) = 0
2711 openat(AT_FDCWD, "/etc/os-release", O_RDONLY|O_CLOEXEC) = 4
2712 fstat(4, {st_mode=S_IFREG|0644, st_size=333, ...}) = 0
2713 ioctl(4, TCGETS, 0x7fffffffa330)        = -1 ENOTTY (Inappropriate ioctl for device)
2714 lseek(4, 0, SEEK_CUR)                   = 0
2715 lseek(4, 0, SEEK_CUR)                   = 0
2716 fstat(4, {st_mode=S_IFREG|0644, st_size=333, ...}) = 0
2717 read(4, "NAME=\"CentOS Linux\"\nVERSION=\"8\"\n"..., 334) = 333
2718 read(4, "", 1)                          = 0
2719 close(4)                                = 0
2720 uname({sysname="Linux", nodename="EagI", ...}) = 0
2721 pipe2([4, 5], O_CLOEXEC)                = 0
2722 fstat(4, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
2723 ioctl(4, TCGETS, 0x7fffffff9740)        = -1 ENOTTY (Inappropriate ioctl for device)
2724 lseek(4, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
2725 pipe2([6, 7], O_CLOEXEC)                = 0
2726 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x155555532e50) = 3351209
2727 close(7)                                = 0
2728 close(5)                                = 0
2729 brk(NULL)                               = 0x555555976000
2730 brk(0x555555998000)                     = 0x555555998000
2731 read(6, "", 50000)                      = 0
2732 close(6)                                = 0
2733 ioctl(4, TCGETS, 0x7fffffff9e80)        = -1 ENOTTY (Inappropriate ioctl for device)
2734 lseek(4, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
2735 lseek(4, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
2736 fstat(4, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
2737 read(4, "x86_64\n", 8192)               = 7
2738 read(4, "", 8185)                       = 0
2739 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3351209, si_uid=1003, si_status=0, si_utime=0, si_stime=0} ---
2740 close(4)                                = 0
2741 wait4(3351209, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 3351209
2742 stat("/usr/libexec/Xorg", {st_mode=S_IFREG|0755, st_size=2486352, ...}) = 0
2743 stat("/usr/libexec/Xorg", {st_mode=S_IFREG|0755, st_size=2486352, ...}) = 0
2744 stat("/usr/libexec/Xorg", {st_mode=S_IFREG|0755, st_size=2486352, ...}) = 0
2745 stat("/usr/libexec/Xorg", {st_mode=S_IFREG|0755, st_size=2486352, ...}) = 0
2746 stat("/usr/libexec/Xorg", {st_mode=S_IFREG|0755, st_size=2486352, ...}) = 0
2747 stat("/usr/libexec/Xorg", {st_mode=S_IFREG|0755, st_size=2486352, ...}) = 0
2748 openat(AT_FDCWD, "/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
2749 fstat(4, {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
2750 getdents64(4, 0x55555596bc30 /* 34 entries */, 32768) = 896
2751 stat("/bin", {st_mode=S_IFDIR|0555, st_size=77824, ...}) = 0
2752 stat("/lib", {st_mode=S_IFDIR|0555, st_size=12288, ...}) = 0
2753 stat("/sbin", {st_mode=S_IFDIR|0555, st_size=24576, ...}) = 0
2754 stat("/lib64", {st_mode=S_IFDIR|0555, st_size=110592, ...}) = 0
2755 stat("/wing2",

[totaam]

Is /wing2 on the $PATH?
Why else would it go looking there?
You may also want to set $HOME=/tmp if $HOME is on NFS.
Try adding the environment variable XPRA_ALL_DEBUG=1 to strace to see where xpra is at when this happens.

[AGI-chandler]

Is /wing2 on the $PATH?

Nope!

Why else would it go looking there?

Dunno man!

You may also want to set $HOME=/tmp if $HOME is on NFS.

$HOME is normally on NFS but since that server is dead at the moment, the account I'm using xrpa with is a backup account under /home which is part of the local root partition.

Try adding the environment variable XPRA_ALL_DEBUG=1 to strace to see where xpra is at when this happens.

I ran strace -E XPRA_ALL_DEBUG=1 xpra list but don't see much difference in the output or an indication of "where xpra is"
There's 2,684 lines of output.  If I just run strace xpra list again there's now 2,701 lines of output.  If I save the outputs to files and diff the two, just seeing minor differences such as read(5, "", 1) instead of read(4, "", 1) or different address locations such as 0x55555593e770 instead of 0x7fffffffcc98...

I ran strace -E XPRA_ALL_DEBUG=1 xpra list on my fully functioning workstation to see what differences there are.  2,958 lines of output and I can see the message debug enabled for xpra.os_util / ('util',) printed on stdout and in the strace output.  So it seems xpra is hanging on the other computer before it can even print any of the debugging messages.

[totaam]

Then I am afraid that it is probably the python interpreter that is looking there and so there is nothing we can do.
You could try python3 -c from xpra import os_util, and so on until you find the import that's causing the hang.

[AGI-chandler]

Okay, kind of lost at this point... python3 -c from xpra import os_util runs fine with exit 0.  How do I figure out what to try to run next?  searching for debugging python indicates I can run things such as python -m trace --trace YOURSCRIPT.py.  Is there a way I could incorporate that into /usr/bin/xpra for example?

[totaam]

How do I figure out what to try to run next?

I assume that the main entry point fails:
python3 -c "from xpra.scripts import main
So you would have to break it down and try all the imports until you find the problematic one:
https://github.com/Xpra-org/xpra/blob/master/xpra/scripts/main.py

[AGI-chandler]

Actually that also runs fine but the next line is where it hangs:

$ python3
Python 3.6.8 (default, Nov  2 2021, 13:01:57) 
[GCC 8.4.1 20200928 (Red Hat 8.4.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> from xpra.platform import init, set_default_name
>>> set_default_name("Xpra")
>>> init()
>>> from xpra.scripts.main import main
>>> sys.exit(main(sys.argv[0], sys.argv))
^C^C^C^C

[totaam]

Then you can just follow by hand:

xpra/xpra/scripts/main.py

Line 93 in 64b5d54

def main(script_file, cmdline):

onwards.

[AGI-chandler]

Welp, had to umount -l the NFS mounts due to them causing other problems, so can't test this anymore, xpra runs fine now... Thanks very much for your insights either way. I have a new question for you soon 😃