SSH Client sporadically hangs for few seconds

On Ubuntu 14.04 LTS when I use SSH client, it hangs sporadically for few seconds every minute or so. no input is lost is just delayed.

SSH -V

OpenSSH_6.6.1p1 Ubuntu-2ubuntu2, OpenSSL 1.0.1f 6 Jan 2014

.ssh/config

Host *
  ServerAliveInterval 240

/etc/ssh/ssh_config

Host *
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication yes
GSSAPIDelegateCredentials no

Same machine has Ubuntu 12.04 LTS with the same configuration, works fine. other machine works fine with the same server as well.

SSH -V

OpenSSH_5.9p1 Debian-5ubuntu1.4, OpenSSL 1.0.1f 6 Jan 2014