[Remote-SSH Bug]:

[xiangjuner]

Is there an existing issue for this bug?

• I have searched the existing issues

Required Troubleshooting Steps

• I have followed these troubleshooting steps
• I have tried both values of the remote.SSH.useLocalServer setting

Connect Locally

It connects successfully

->

Hello experts, this is my first time to post my problem here. it maybe not formal.

My question is it seems that after the last update (about 1 week), I got an error after "Waiting for server log..." although it seems I have connected to the remote server. The problem can be solved by removing the .vscode-server. But it can only work for one time. If I restarted the vscode to connect the remote server , the same problem will arise again.

Could you please give me some advice, thank you very much!

The log is shown below :

Expected Behavior

No response

Steps To Reproduce

No response

Remote-SSH Log

[16:14:48.151] Log Level: 2
[16:14:48.161] VS Code version: 1.94.2
[16:14:48.161] Remote-SSH version: [email protected]
[16:14:48.161] darwin arm64
[16:14:48.162] SSH Resolver called for "ssh-remote+lxplus-gpu.cern.ch", attempt 1
[16:14:48.162] "remote.SSH.useLocalServer": true
[16:14:48.162] "remote.SSH.useExecServer": true
[16:14:48.162] "remote.SSH.path": undefined
[16:14:48.162] "remote.SSH.configFile": undefined
[16:14:48.162] "remote.SSH.useFlock": true
[16:14:48.162] "remote.SSH.lockfilesInTmp": true
[16:14:48.162] "remote.SSH.localServerDownload": auto
[16:14:48.162] "remote.SSH.remoteServerListenOnSocket": false
[16:14:48.162] "remote.SSH.showLoginTerminal": false
[16:14:48.162] "remote.SSH.defaultExtensions": []
[16:14:48.162] "remote.SSH.loglevel": 2
[16:14:48.162] "remote.SSH.enableDynamicForwarding": true
[16:14:48.162] "remote.SSH.enableRemoteCommand": false
[16:14:48.163] "remote.SSH.serverPickPortsFromRange": {}
[16:14:48.163] "remote.SSH.serverInstallPath": {}
[16:14:48.163] "remote.SSH.permitPtyAllocation": false
[16:14:48.163] "remote.SSH.preferredLocalPortRange: undefined
[16:14:48.163] "remote.SSH.useCurlAndWgetConfigurationFiles: false
[16:14:48.166] SSH Resolver called for host: lxplus-gpu.cern.ch
[16:14:48.166] Setting up SSH remote "lxplus-gpu.cern.ch"
[16:14:48.170] Acquiring local install lock: /var/folders/c5/sz58p_mj03s_htbv9_3t68qr0000gn/T/vscode-remote-ssh-26866061-install.lock
[16:14:48.172] Looking for existing server data file at /Users/xiangjun/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-26866061-384ff7382de624fb94dbaf6da11977bba1ecd427-0.115.0-es/data.json
[16:14:48.172] No existing data file
[16:14:48.173] Using commit id "384ff7382de624fb94dbaf6da11977bba1ecd427" and quality "stable" for server
[16:14:48.174] Script variables:
{
"InstallExitCode.24": "AlreadyInProgress",
"InstallExitCode.25": "ServerDownloadFailed",
"InstallExitCode.26": "NoDownloaderAvailable",
"InstallExitCode.27": "UnsupportedArch",
"InstallExitCode.28": "StatusCheckFailed",
"InstallExitCode.29": "NeedInsidersArch",
"InstallExitCode.30": "NoDownloaderAvailableForStatusCheck",
"InstallExitCode.31": "ServerTransferFailed",
"InstallExitCode.32": "ServerFailedToStart",
"InstallExitCode.33": "NeedInsidersWindows",
"InstallExitCode.34": "CreateInstallDirFailed",
"InstallExitCode.35": "UnsupportedPlatform",
"InstallExitCode.36": "ServerTerminatedCVE20201416",
"InstallExitCode.37": "UnpackFailed",
"InstallExitCode.38": "ChangeDirFailed",
"InstallExitCode.AlreadyInProgress": "24",
"InstallExitCode.ServerDownloadFailed": "25",
"InstallExitCode.NoDownloaderAvailable": "26",
"InstallExitCode.NoDownloaderAvailableForStatusCheck": "30",
"InstallExitCode.UnsupportedArch": "27",
"InstallExitCode.StatusCheckFailed": "28",
"InstallExitCode.NeedInsidersArch": "29",
"InstallExitCode.ServerTransferFailed": "31",
"InstallExitCode.ServerFailedToStart": "32",
"InstallExitCode.NeedInsidersWindows": "33",
"InstallExitCode.CreateInstallDirFailed": "34",
"InstallExitCode.UnsupportedPlatform": "35",
"InstallExitCode.ServerTerminatedCVE20201416": "36",
"InstallExitCode.UnpackFailed": "37",
"InstallExitCode.ChangeDirFailed": "38",
"InstallUnpackCode.Success": "success",
"InstallUnpackCode.Error": "error",
"InstallUnpackCode.MissingFiles": "missingFiles",
"uuid": "f0572d34014e",
"startMarker": "f0572d34014e: running",
"commitId": "384ff7382de624fb94dbaf6da11977bba1ecd427",
"quality": "stable",
"token": "11a1a1a1-11a1-11a1-a11a-aaa111aa1a1a",
"vscodeAgentFolder": "$HOME/.vscode-server",
"allowClientDownload": "1",
"forceClientDownload": "0",
"cliNameInArchive": "code",
"ignoreWgetConfigFlag": " --no-config ",
"ignoreCurlConfigFlag": " --disable ",
"wgetTriesSegment": "--tries=1",
"listenArgs": "--on-host=127.0.0.1 --on-port",
"getDownloadServerStartTrigger": "f0572d34014e:trigger_server_download",
"getDownloadServerEndTrigger": "f0572d34014e:trigger_server_download_end",
"getProgressDownloading": "f0572d34014e%%1%%",
"getProgressInstalling": "f0572d34014e%%2%%"
}
[16:14:48.175] Install and start server if needed
[16:14:48.177] PATH: /opt/anaconda3/bin:/opt/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/opt/homebrew/Cellar/[email protected]/3.10.9/bin/python3.10/bin:/bin:/jre/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Library/Apple/usr/bin
[16:14:48.178] Checking ssh with "ssh -V"
[16:14:48.183] > OpenSSH_9.0p1, LibreSSL 3.3.6

[16:14:48.185] askpass server listening on /var/folders/c5/sz58p_mj03s_htbv9_3t68qr0000gn/T/vscode-ssh-askpass-b24d9a955621f17081d797cbe5465a7e7ce24662.sock
[16:14:48.185] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/c5/sz58p_mj03s_htbv9_3t68qr0000gn/T/vscode-ssh-askpass-3cab7ae17ec1749b7f4e005e9f7a366b5fb9c31c.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","57242","-o","ConnectTimeout=15","lxplus-gpu.cern.ch"],"serverDataFolderName":".vscode-server","dataFilePath":"/Users/xiangjun/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-26866061-384ff7382de624fb94dbaf6da11977bba1ecd427-0.115.0-es/data.json"}
[16:14:48.185] Local server env: {"SSH_AUTH_SOCK":"/private/tmp/com.apple.launchd.IYJjPDzhZi/Listeners","SHELL":"/bin/bash","DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin)","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"","VSCODE_SSH_ASKPASS_MAIN":"/Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/c5/sz58p_mj03s_htbv9_3t68qr0000gn/T/vscode-ssh-askpass-b24d9a955621f17081d797cbe5465a7e7ce24662.sock"}
[16:14:48.186] Spawned 96886
[16:14:48.186] Using connect timeout of 17 seconds
[16:14:48.255] > local-server-1> Running ssh connection command: ssh -v -T -D 57242 -o ConnectTimeout=15 lxplus-gpu.cern.ch
[16:14:48.257] > local-server-1> Spawned ssh, pid=96896
[16:14:48.260] stderr> OpenSSH_9.0p1, LibreSSL 3.3.6
[16:14:49.057] stderr> debug1: Server host key: ssh-ed25519 SHA256:xOtCWmpDSZenkpkC7zNvTW0l8eh3kXaStmqZC3UVcAk
[16:14:49.765] stderr> no credential for 8E1465DD-A14E-441F-B376-49C03494C2D5
[16:14:49.765] stderr>
[16:14:49.765] stderr>
[16:14:49.766] stderr> unknown mech-code 0 for mech 1 2 752 43 14 2
[16:14:49.766] stderr>
[16:14:49.766] stderr>
[16:14:49.766] stderr> unknown mech-code 0 for mech 1 3 6 1 5 5 14
[16:14:49.766] stderr>
[16:14:49.766] stderr>
[16:14:49.789] stderr> unknown mech-code 2 for mech 1 3 6 1 4 1 311 2 2 10
[16:14:49.789] stderr>
[16:14:49.789] stderr>
[16:14:49.789] stderr> unknown mech-code 0 for mech 1 3 5 1 5 2 7
[16:14:49.789] stderr>
[16:14:49.789] stderr>
[16:14:49.790] stderr> unknown mech-code 0 for mech 1 3 6 1 5 2 5
[16:14:49.790] stderr>
[16:14:49.790] stderr>
[16:14:50.310] Got askpass request: {"request":" ([email protected]) Password: "}
[16:14:50.310] Showing password prompt
[16:14:50.311] Listening for interwindow password on /var/folders/c5/sz58p_mj03s_htbv9_3t68qr0000gn/T/vscode-ssh-askpass-6b461d83eb3d5f8df519de9074c8d04021798ec8.sock
[16:14:50.311] Writing password prompt to globalState
[16:14:58.781] Got password response
[16:14:58.782] Interactor gave response: ************
[16:14:58.783] Cleaning up other-window auth server
[16:14:58.785] Using connect timeout of 17 seconds
[16:14:59.236] stderr> Authenticated to lxplus-gpu.cern.ch ([188.184.195.72]:22) using "keyboard-interactive".
[16:15:00.068] > * ********************************************************************

• Welcome to lxplus909.cern.ch, Red Hat Enterprise Linux release 9.4 (Plow)
• Archive of news is available in /etc/motd-archive
• Reminder: you have agreed to the CERN
• computing rules, in particular OC5. CERN implements
• the measures necessary to ensure compliance.
• https://cern.ch/ComputingRules
• Puppet environment: qa, Roger state: production
• Foreman hostgroup: lxplus/nodes/gpu
• Availability zone: none
• LXPLUS Public Login Service - http://lxplusdoc.web.cern.ch/
• ************** INFO INFO ****************
  lxplus909.cern.ch is a special lxplus node with GPUs.
  It is not part of the general lxplus.cern.ch alias. For
  any support requests please mention that this node is member
  of lxplus-gpu.cern.ch

---

• Please read LXPLUS Privacy Notice in http://cern.ch/go/TpV7

---

• October 10th: LxPlus will switch to requiring 2fa during 2025
• Test your access from now https://cern.ch/otg0152605

Last failed login: Sat Oct 12 14:05:43 CEST 2024 from 202.119.45.243 on ssh:notty
There were 3 failed login attempts since the last successful login.
[16:15:00.264] > ready: f0572d34014e
[16:15:00.268] stderr> /afs/cern.ch/user/z/zgao/.bashrc: line 22: show: command not found
[16:15:00.450] > Linux 5.14.0-427.37.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 13 12:41:50 EDT 2024
[16:15:00.451] Platform: linux
[16:15:00.634] > /bin/bash
[16:15:00.635] Parent Shell: bash
[16:15:00.635] Parent Shell pid: 96886
[16:15:00.820] > 3895501
[16:15:00.820] Waiting for pid of spawned 'sh' subshell: '3895501'...
[16:15:00.867] > 3895501
[16:15:00.921] > 3895501
[16:15:00.977] > 3895501
[16:15:01.007] > f0572d34014e: running
Script executing under PID: 3895501
[16:15:01.021] > Found existing installation at /afs/cern.ch/user/z/zgao/.vscode-server...
Starting VS Code CLI...
printenv:
[16:15:01.022] > SHELL=/bin/bash
EOS_MGM_URL=root://eosatlas.cern.ch
LCG_GFAL_INFOSYS=lcg-bdii.cern.ch:2170
HISTCONTROL=ignoredups
CASTOR_HOME=/castor/cern.ch/user/z/zgao
HOSTNAME=lxplus909.cern.ch
HISTSIZE=1000
OS_REGION_NAME=cern
OS_PROJECT_DOMAIN_ID=default
GUESTFISH_OUTPUT=\e[0m
ALRB_TutorialData=/cvmfs/atlas.cern.ch/repo/tutorials/asg/cern-jun2022
EDITOR=/bin/nano -w
KRB5CCNAME=FILE:/run/user/152320/krb5cc
PWD=/afs/cern.ch/user/z/zgao
LOGNAME=zgao
XDG_SESSION_TYPE=tty
MODULESHOME=/usr/share/Modules
MANPATH=/usr/share/man::/opt/puppetlabs/puppet/share/man
MESA_GLSL_CACHE_DIR=/run/user/152320
MYPROXY_SERVER=myproxy.cern.ch
OS_AUTH_URL=https://keystone.cern.ch/v3
GUESTFISH_RESTORE=\e[0m
MOTD_SHOWN=pam
__MODULES_SHARE_MANPATH=:1
HOME=/afs/cern.ch/user/z/zgao
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
LANG=en_US.UTF-8
GUESTFISH_PS1=[\e[1;32m]>[\e[0;31m]
TMPDIR=/tmp/zgao
SSH_CONNECTION=202.119.42.79 43408 188.184.195.72 22
KUBECACHEDIR=/run/user/152320/kubecache
XDG_SESSION_CLASS=user
SELINUX_ROLE_REQUESTED=
CPLUS_INCLUDE_PATH=:/cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.24.06/x86_64-centos7-gcc48-opt/include
LESSOPEN=||/usr/bin/lesspipe.sh %s
USER=zgao
CONDA_SHLVL=0
MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD
OS_VOLUME_API_VERSION=3
LOADEDMODULES=
SELINUX_USE_CURRENT_RANGE=
OS_IDENTITY_PROVIDER=sssd
SHLVL=2
GUESTFISH_INIT=\e[1;34m
CVS_RSH=ssh
XDG_SESSION_ID=780
OS_AUTH_TYPE=v3fedkerb
LD_LIBRARY_PATH=/usr/local/cuda-12.4/lib64:
XDG_RUNTIME_DIR=/run/user/152320
OS_MUTUAL_AUTH=disabled
S_COLORS=auto
SSH_CLIENT=202.119.42.79 43408 22
__MODULES_LMINIT=module use --append /usr/share/Modules/modulefiles:module use --append /etc/modulefiles:module use --append /usr/share/modulefiles
OS_PROJECT_NAME=Personal zgao
which_declare=declare -f
KDEDIRS=/usr
OS_PROTOCOL=kerberos
SITE_NAME=CERN-PROD
PATH=/afs/cern.ch/atlas/scripts:/usr/sue/bin:/opt/rocm/bin:/usr/share/Modules/bin:/usr/local/cuda-12.4/bin:/usr/condabin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/afs/cern.ch/user/z/zgao/.local/bin:/afs/cern.ch/user/z/zgao/bin
SELINUX_LEVEL_REQUESTED=
MODULEPATH=/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/152320/bus
MAIL=/var/spool/mail/zgao
MESA_SHADER_CACHE_DIR=/run/user/152320
OS_COMPUTE_API_VERSION=2.72
RUCIO_ACCOUNT=zgao
OS_IDENTITY_API_VERSION=3
FRONTIER_SERVER=(serverurl=http://atlasfrontier-local.cern.ch:8000/atlr)(serverurl=http://atlasfrontier-ai.cern.ch:8000/atlr)(proxyurl=http://ca-proxy-atlas.cern.ch:3128)(proxyurl=http://ca-proxy-meyrin.cern.ch:3128)(proxyurl=http://ca-proxy.cern.ch:3128)(proxyurl=http://atlasbpfrontier.cern.ch:3127)
MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl
BASH_FUNC_ml%%=() { module ml "$@"
}
BASH_FUNC_which%%=() { ( alias;
eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}
BASH_FUNC_module%%=() { local _mlredir=0;
if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then
if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then
_mlredir=0;
else
if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then
_mlredir=1;
fi;
fi;
fi;
case " $@ " in
' --no-redirect ')
_mlredir=0
;;
' --redirect ')
_mlredir=1
;;
esac;
if [ $_mlredir -eq 0 ]; then
_module_raw "$@";
else
_module_raw "$@" 2>&1;
fi
}
BASH_FUNC_scl%%=() { if [ "$1" = "load" -o "$1" = "unload" ]; then
eval "module $@";
else
/usr/bin/scl "$@";
fi
}
BASH_FUNC__module_raw%%=() { eval "$(/usr/bin/tclsh '/usr/share/Modules/libexec/modulecmd.tcl' bash "$@")";
_mlstatus=$?;
return $_mlstatus
}
_=/usr/bin/printenv
[16:15:01.024] > Removing old logfile at /afs/cern.ch/user/z/zgao/.vscode-server/.cli.384ff7382de624fb94dbaf6da11977bba1ecd427.log
[16:15:01.030] > Spawned remote CLI: 3895524
[16:15:01.052] > Waiting for server log...
[16:15:01.068] > Waiting for server log...
[16:15:01.103] > Waiting for server log...
[16:15:01.137] > Waiting for server log...
[16:15:01.173] > Waiting for server log...
[16:15:01.208] > Waiting for server log...
[16:15:01.243] > Waiting for server log...
[16:15:01.279] > Waiting for server log...
[16:15:01.314] > Waiting for server log...
[16:15:01.349] > Waiting for server log...
[16:15:01.384] > Waiting for server log...
[16:15:01.420] > Waiting for server log...
[16:15:01.458] > Waiting for server log...
[16:15:01.492] > Waiting for server log...
[16:15:01.528] > Waiting for server log...
[16:15:01.562] > f0572d34014e: start
listeningOn====
osReleaseId==rhel==
arch==x86_64==
vscodeArch==x64==
bitness==64==
tmpDir==/run/user/152320==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
serverStartTime==537==
execServerToken==11a1a1a1-11a1-11a1-a11a-aaa111aa1a1a==
platformDownloadPath==cli-alpine-x64==
f0572d34014e: end
[16:15:01.562] Received install output:
listeningOn====
osReleaseId==rhel==
arch==x86_64==
vscodeArch==x64==
bitness==64==
tmpDir==/run/user/152320==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
serverStartTime==537==
execServerToken==11a1a1a1-11a1-11a1-a11a-aaa111aa1a1a==
platformDownloadPath==cli-alpine-x64==

[16:15:01.563] Failed to parse remote port from server output
[16:15:01.563] Terminating local server
[16:15:01.567] Resolver error: Error:
at v.Create (/Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/extension.js:2:493431)
at t.handleInstallOutput (/Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/extension.js:2:490753)
at e (/Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/extension.js:2:545236)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/extension.js:2:567561
at async t.withShowDetailsEvent (/Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/extension.js:2:571256)
at async /Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/extension.js:2:541941
at async T (/Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/extension.js:2:539992)
at async t.resolveWithLocalServer (/Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/extension.js:2:541481)
at async P (/Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/extension.js:2:564693)
at async t.resolve (/Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/extension.js:2:568667)
at async /Users/xiangjun/.vscode/extensions/ms-vscode-remote.remote-ssh-0.115.0/out/extension.js:2:839059
[16:15:01.572] ------

[16:15:01.575] Local server exit: 15

Anything else?

No response

[abhishekLHCb-CERN]

I am somehow, experiencing the same problem since last week.

[abhishekLHCb-CERN]

I have solved the issue on my end. Although I am not entirely sure what caused the problem for you, I suggest running a few checks on both your VS Code and LXPLUS server to ensure they communicate properly.

Steps to follow:

1. Run checks on the LXPLUS server:

   • Open the terminal and log in to the LXPLUS server.
   • Once logged in, run the following command to clean up any previous VS Code server installations:

     rm -rf ~/.vscode-server

2. Check VS Code Remote-SSH settings:

   • Go to Settings > Remote - SSH > Show Settings.
   • Ensure that "remote.SSH.useLocalServer" is set to false (it's true in your logs). Disable it and test again.
   • Optionally, increase "remote.SSH.loglevel" to 3 for more detailed logs if needed.

3. Restart VS Code and try connecting again.

Let me know if this resolves your issue!

[xiangjuner]

Hi ,

Thank you very much! It works. :)

[joshspicer]

Thanks for the details in your issue, and glad to hear disabling useLocalServer is helping.

I see the shared /afs file system on your remote machine, which sounds like the same issue captured in #10445. I'll continue tracking in that issue

/duplicate #10445

[vs-code-engineering]

Thanks for creating this issue! We figured it's covering the same as another one we already have. Thus, we closed this one as a duplicate. You can search for similar existing issues. See also our issue reporting guidelines.

Happy Coding!