Proving DNS/Network issues on client network

Problem

1 of ~150 server clients (in different locations and with different network setups) is not redirected through my apache service. I need to know where the problem is but can’t figure it out.

All clients access a virtual host and send the same requests against the proxy:

  <VirtualHost *:80>
  ServerName update.***.tld
  ServerAdmin [email protected]

  CustomLog /var/log/apache2/update.***.tld_access.log combined
  ErrorLog  /var/log/apache2/update.***.tld_error.log

  # redirect all http request to https
  RewriteEngine on
  Options +FollowSymLinks
  RewriteCond %{SERVER_PORT} !^443$
  RewriteRule ^.*$ https://%{SERVER_NAME}%{REQUEST_URI} [L,R]
  </VirtualHost>

  <VirtualHost *:443>
  ServerName update.***.tld

  LogLevel warn
  SSLEngine on
  SSLCertificateFile /etc/ssl/certs/wildcard.***.tld.cert
  SSLCertificateKeyFile /etc/ssl/private/wildcard.***.tld.key
  SSLCertificateChainFile /etc/ssl/certs/wildcard.***.tld.combined.cert

  CustomLog /var/log/apache2/update.***.tld_access.log combined
  ErrorLog  /var/log/apache2/update.***.tld_error.log

  TimeOut 3600
  KeepAlive On

  AddDefaultCharset UTF-8

  SSLProxyEngine on
  ProxyPreserveHost Off
  SetEnv force-proxy-request-1.0 1
  SetEnv proxy-nokeepalive 1
  ProxyTimeout 15

  ProxyRequests Off
  ProxyPass /         https://***-***-prod.aws.tld/
  ProxyPassReverse /  https://***-***-prod-prod.aws.tld/

  <Proxy *>
     AddDefaultCharset UTF-8
     Require all granted
  </Proxy>
  </VirtualHost>

Debug

• Changed the client to a static IP setup with Google DNS server
• Checked the log files
  • It seems that all requests from that special client are not redirected correctly
  • The requests from that client are logged in default_access.log and dont reach my custom log so I guess the forwarding is not working but why it is for 150 other clients ..
  • Default access log: clientipaddress – – [23/Jul/2020:20:23:17 +0200] "GET /api/agent/ping HTTP/1.1" 400 301 "-" "-"
  • When I send a wget from the client to my proxy on Port 443 it is correctly forwarded to the virtual host and logged in my custom log
  • I have checked the tcpdump and noticed that the client tries to send to the correct server with the correct port (443)

tcpdump on corrupt client:

tcpdump -i eth0 -vvv host update.***.tld  > dump

https://gist.github.com/herz0g/e02ef883688c904667164a175955ecc0

Conclusion

I guess it is a problem on the customer side network otherwise it would not work for 150 other clients but I am not sure how to prove that or what could be debugged further.

[Fri Jul 24 12:33:11.463639 2020] [ssl:info] [pid 9792:tid 139651482162944] [client clientip:26294] AH01964: Connection to child 441 established (server default.virtual.host:443)
[Fri Jul 24 12:33:11.463917 2020] [ssl:debug] [pid 9792:tid 139651482162944] ssl_engine_kernel.c(2096): [client clientip:26294] AH02043: SSL virtual host for servername update.***.tld found
[Fri Jul 24 12:33:11.612839 2020] [core:debug] [pid 9792:tid 139651482162944] protocol.c(1158): [client clientip:26294] AH02427: Request header value is malformed: TOKEN ******r
[Fri Jul 24 12:33:11.612873 2020] [core:debug] [pid 9792:tid 139651482162944] protocol.c(1318): [client clientip:26294] AH00567: request failed: error reading the headers