barman-wal-restore : ssh connection with or without keys

[greg42300]

Hello,
I try to restore a streaming backup from my "barman" server to my "pg2" server, without success.
There is a ssh problem with keys. It seems the barman-wal-restore not find or user ssh-keys of the postgres user.
You can see the ssh log of the remote server pg2 at the end of the post.
hosts : Debian 11 uptodate, postgres-13 + repmgr 5.2.0 + barman 2.21
pg2 : 192.168.0.21 (slave)
barman : 192.168.0.25

Here is my tests:

From host pg2 to barman server with postgres system user:
barman-wal-restore -t -U barman barman SIG DUMMY DUMMY
[email protected]'s password:
Ready to retrieve WAL files from the server SIG

Hummm, a password is required when one would expect an automatic connection by key given the configuration

The ssh connection tests are ok:

on the other pg2 hosts:
.ssh/config de la sorte :
Host barman
User barman
IdentityFile ~/.ssh/id_rsa_postgres

on the barman host :
.ssh/config :
Host pg2
User postgres
IdentityFile ~/.ssh/id_rsa_barman

From the pg2 host to barman serveur, with postgres user:
#ssh barman@barman : login ok, with no passphrase and no password

And From the barman host to pg1 et pg2, with the barman user:
#ssh postgres@pg1 : login ok, with no passphrase and no password
#ssh postgres@pg2 : login ok, with no passphrase and no password

/etc/postgresql/13/main/postgresql.conf
#restore_command=''

any hints or help.
Best regards

ssh log on barman host after comand barman-wal-restore -t -U barman barman SIG DUMMY DUMMY on the pg2 host:
Oct 14 20:13:57 barman sshd[4052]: debug1: match: OpenSSH_8.4p1 Debian-5+deb11u3 pat OpenSSH* compat 0x04000000
Oct 14 20:13:57 barman sshd[4052]: debug1: permanently_set_uid: 106/65534 [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: SSH2_MSG_KEXINIT received [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: kex: algorithm: curve25519-sha256 [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: kex: client->server cipher: [email protected] MAC: compression: none [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: kex: server->client cipher: [email protected] MAC: compression: none [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: ssh_packet_send2_wrapped: resetting send seqnr 3 [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: rekey out after 134217728 blocks [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: Sending SSH2_MSG_EXT_INFO [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: ssh_packet_read_poll2: resetting read seqnr 3 [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: rekey in after 134217728 blocks [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: KEX done [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: userauth-request for user barman service ssh-connection method none [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: attempt 0 failures 0 [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: PAM: initializing for "barman"
Oct 14 20:13:57 barman sshd[4052]: debug1: PAM: setting PAM_RHOST to "192.168.0.21"
Oct 14 20:13:57 barman sshd[4052]: debug1: PAM: setting PAM_TTY to "ssh"
Oct 14 20:13:57 barman sshd[4052]: debug1: userauth-request for user barman service ssh-connection method password [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: attempt 1 failures 0 [preauth]
Oct 14 20:13:57 barman sshd[4052]: Failed none for barman from 192.168.0.21 port 52562 ssh2
Oct 14 20:13:57 barman sshd[4052]: debug1: userauth-request for user barman service ssh-connection method password [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: attempt 2 failures 1 [preauth]
Oct 14 20:13:57 barman sshd[4052]: Failed password for barman from 192.168.0.21 port 52562 ssh2
Oct 14 20:13:57 barman sshd[4052]: debug1: userauth-request for user barman service ssh-connection method password [preauth]
Oct 14 20:13:57 barman sshd[4052]: debug1: attempt 3 failures 2 [preauth]
Oct 14 20:13:57 barman sshd[4052]: Failed password for barman from 192.168.0.21 port 52562 ssh2
Oct 14 20:13:57 barman sshd[4052]: Connection closed by authenticating user barman 192.168.0.21 port 52562 [preauth]

[greg42300]

I have found an other similar issue: #920
I have check permissions on ssh files:
/var/lib/postgres/.ssh/
drwx------ 2 postgres postgres 4096 Oct 18 15:06 .
-rw-r--r-- 1 postgres postgres 394 Oct 12 19:49 id_rsa_postgres.pub
-rw------- 1 postgres postgres 1823 Oct 12 19:49 id_rsa_postgres
-rw-r--r-- 1 postgres postgres 1110 Oct 13 10:14 known_hosts
-rw-r--r-- 1 postgres postgres 1183 Oct 13 15:32 authorized_keys
-rw------- 1 postgres postgres 169 Oct 18 15:06 config

[martinmarques]

Hard to help here. I would say that a skilled sysadmin would be able to root out where the problem is (it seems some sshd configuration that is not barman-wal-restore get the WAL via rsync). Have you tried using rsync manually?

One other thing: Does barman-wal-archive work in the other direction? (from pg2 to barman)