SSH fails to connect from one computer but works from other computers

Problem

I have an external VPS server running Ubuntu 18.04 with openssh-server 7.6p1. I can connect via ssh to this server from many different networks except from one particular client. This particular client runs on Ubuntu 16.04 with openssh 7.2p2.

Relevant configurations and logs

• Server SSHD config / $ cat /etc/ssh/sshd_config | egrep -v "^$|^#"

Port 41232
LogLevel DEBUG3
AuthorizedKeysFile  .ssh/authorized_keys
Subsystem   sftp    /usr/libexec/sftp-server

• Client SSH config / $cat /etc/ssh/ssh_config | egrep -v "^$|^#"

Host *
    StrictHostKeyChecking no
    SendEnv LANG LC_*
    NoHostAuthenticationForLocalhost yes

• The SSH command (server IP address changed)

ssh -vvv -F /dev/null [email protected] -p 41232

produces the following log:

OpenSSH_7.2p2 Ubuntu-4ubuntu2.10, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /dev/null
debug2: resolving "12.215.24.089" port 41232
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 12.215.24.089 [12.215.24.089] port 41232.
debug1: Connection established.
debug1: identity file /home/saikat/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/saikat/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/saikat/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/saikat/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/saikat/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/saikat/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/saikat/.ssh/id_ed25519 type 4
debug1: key_load_public: No such file or directory
debug1: identity file /home/saikat/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
ssh_exchange_identification: read: Connection reset by peer

The /var/log/auth.log on the server shows the following during the connection reset:

Jul 23 19:47:49 rohini sshd[3336]: debug3: fd 5 is not O_NONBLOCK
Jul 23 19:47:49 rohini sshd[3336]: debug1: Forked child 3725.
Jul 23 19:47:49 rohini sshd[3336]: debug3: send_rexec_state: entering fd = 8 config len 197
Jul 23 19:47:49 rohini sshd[3336]: debug3: ssh_msg_send: type 0
Jul 23 19:47:49 rohini sshd[3336]: debug3: send_rexec_state: done
Jul 23 19:47:49 rohini sshd[3725]: debug3: oom_adjust_restore
Jul 23 19:47:49 rohini sshd[3725]: debug1: Set /proc/self/oom_score_adj to 0
Jul 23 19:47:49 rohini sshd[3725]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Jul 23 19:47:49 rohini sshd[3725]: debug1: inetd sockets after dupping: 3, 3
Jul 23 19:47:49 rohini sshd[3725]: Connection from 124.56.232.23 port 10400 on 12.215.24.089 port 41232
Jul 23 19:47:49 rohini sshd[3725]: debug1: Client protocol version 2.0; client software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
Jul 23 19:47:49 rohini sshd[3725]: debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.10 pat OpenSSH* compat 0x04000000
Jul 23 19:47:49 rohini sshd[3725]: debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
Jul 23 19:47:49 rohini sshd[3725]: debug2: fd 3 setting O_NONBLOCK
Jul 23 19:47:49 rohini sshd[3725]: debug3: ssh_sandbox_init: preparing seccomp filter sandbox
Jul 23 19:47:49 rohini sshd[3725]: debug2: Network child is on pid 3726
Jul 23 19:47:49 rohini sshd[3725]: debug3: preauth child monitor started
Jul 23 19:47:49 rohini sshd[3725]: debug3: privsep user:group 109:65534 [preauth]
Jul 23 19:47:49 rohini sshd[3725]: debug1: permanently_set_uid: 109/65534 [preauth]
Jul 23 19:47:49 rohini sshd[3725]: debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
Jul 23 19:47:49 rohini sshd[3725]: debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
Jul 23 19:47:49 rohini sshd[3725]: debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Jul 23 19:47:49 rohini sshd[3725]: debug3: send packet: type 20 [preauth]
Jul 23 19:47:49 rohini sshd[3725]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Jul 23 19:47:49 rohini sshd[3725]: Connection reset by 124.56.232.23 port 10400 [preauth]
Jul 23 19:47:49 rohini sshd[3725]: debug1: do_cleanup [preauth]
Jul 23 19:47:49 rohini sshd[3725]: debug1: monitor_read_log: child log fd closed
Jul 23 19:47:49 rohini sshd[3725]: debug3: mm_request_receive entering
Jul 23 19:47:49 rohini sshd[3725]: debug1: do_cleanup
Jul 23 19:47:49 rohini sshd[3725]: debug1: Killing privsep child 3726
Jul 23 19:47:49 rohini sshd[3725]: debug1: audit_event: unhandled event 12

• The tcpdump -i any -n -vvv src 124.56.232.23 during the SSH attempt:

tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
11:35:00.268722 IP (tos 0x0, ttl 52, id 17296, offset 0, flags [DF], proto TCP (6), length 60)
    124.56.232.23.10600 > 12.215.24.089.41232: Flags [S], cksum 0x1f12 (correct), seq 406486862, win 64240, options [mss 1460,sackOK,TS val 822851129 ecr 0,nop,wscale 7], length 0
11:35:00.276593 IP (tos 0x0, ttl 52, id 17297, offset 0, flags [DF], proto TCP (6), length 52)
    124.56.232.23.10600 > 12.215.24.089.41232: Flags [.], cksum 0x9554 (correct), seq 406486863, ack 2004122924, win 502, options [nop,nop,TS val 822851138 ecr 1494508470], length 0
11:35:00.277541 IP (tos 0x0, ttl 52, id 17298, offset 0, flags [DF], proto TCP (6), length 94)
    124.56.232.23.10600 > 12.215.24.089.41232: Flags [P.], cksum 0x6ecc (correct), seq 0:42, ack 1, win 502, options [nop,nop,TS val 822851139 ecr 1494508470], length 42
11:35:00.297736 IP (tos 0x0, ttl 117, id 18028, offset 0, flags [none], proto TCP (6), length 40)
    124.56.232.23.10600 > 12.215.24.089.41232: Flags [R], cksum 0x07df (correct), seq 406486905, win 24862, length 0
^C
4 packets captured
5 packets received by filter
0 packets dropped by kernel

Pastebin

What I have tried / checked

• Restart the server
• Restart the client
• No entry in the /etc/hosts.deny on the server
• Forced IPV4, but did not help ssh -F /dev/null -4 [email protected] -p 41232
• Tried using the flag -i /dev/null
• One possibility is MTU fragmentation, nice discussion here. I tried several answers from the discussion, but it did not solve the issue. To be particular:
  • I tried adding the Ciphers, MACs, HostKeyAlgorithms and other flags in /etc/ssh/ssh_config
  • Tried changing MTU of client to 576, 1000, and 1280. The MTU of ens3 on server was 1500.
  • Also tried adding IPQoS lowdelay throughput to /etc/ssh/ssh_config
• I also cannot connect to the server via PuTTY.

Firewall

Following the comment from @kenster, here are the software firewall of the client and the server.

• Server settings / sudo ufw status verbose

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
41232/tcp                  ALLOW IN    Anywhere                   # OpenSSH
80/tcp (Nginx HTTP)        ALLOW IN    Anywhere                  
443/tcp (Nginx HTTPS)      ALLOW IN    Anywhere                  
1313/tcp                   ALLOW IN    Anywhere                   # Hugo Server
41232/tcp (v6)             ALLOW IN    Anywhere (v6)              # OpenSSH
80/tcp (Nginx HTTP (v6))   ALLOW IN    Anywhere (v6)             
443/tcp (Nginx HTTPS (v6)) ALLOW IN    Anywhere (v6)             
1313/tcp (v6)              ALLOW IN    Anywhere (v6)              # Hugo Server

• Client settings / sudo ufw status verbose

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

References (similar problems and their solutions)

• Same problem for one computer on the same LAN
• SSH hangs due to MTU
• Other threads:
  [1]
  [2]
  [3]
  [4]