TransWikia.com

Can't make vsftpd on kubernetes (minikube) with load balancer AND ssl work

Server Fault Asked by Bstorm on December 9, 2021

CONTEXT

School project : to host several services on a kubernetes node minikube including a ftps server on port 21 using a load balancer. School ask us to make our own images based on Alpine.

What I did so far

I setup metallb as loadbalancer.

I build an image in which I setup vsftpd and generate certificate and keys :

openssl req -x509 -subj "/C=FR/ST=PARIS/L=Paris/O=42_Corp/OU=myname/CN=mywebsite" -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/vsftpd.key -out /etc/ssl/certs/vsftpd.crt

side note : I use supervisor in order to run telegraf in the same container to get a dashboard.

I deploy this image with this yml:

metadata:
  name: ftps
spec:
  serviceName: "ftps"
  selector:
    matchLabels:
      app: ftps
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: ftps
    spec:
      containers:
      - name: ftps
        image: ft_service/ftps
        imagePullPolicy: Never
        volumeMounts:
        - name: ftps-data
          mountPath: /mnt/ftps_data
      volumes:
      - name: ftps-data
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: ftps
  name: ftps
  namespace: default
spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  ports:
  - name: port-20
    port: 20
    protocol: TCP
    targetPort: 20
  - name: port-21
    port: 21
    protocol: TCP
    targetPort: 21
  selector:
    app: ftps

Then I have a conf file for vsftp which contain this :

local_enable=YES
write_enable=YES
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
chroot_local_user=YES
allow_writeable_chroot=YES
listen=YES


rsa_cert_file=/etc/ssl/certs/vsftpd.crt
rsa_private_key_file=/etc/ssl/private/vsftpd.key
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
require_ssl_reuse=NO
ssl_ciphers=HIGH
seccomp_sandbox=NO

(the last "seccomp_sandbox=NO" seemed to resolve some issues I had a the begining)

Which gives me…

Once I deploy everything, I get my pods :

default          pod/ftps-1                             1/1     Running   0          7m55s 

I get my service:

default       service/ftps         LoadBalancer   10.104.132.127   192.168.1.241   20:31268/TCP,21:31581/TCP   8m43s

Then when I run minikube service ftps --url,
I get
http://172.17.0.2:31268
http://172.17.0.2:31581

WHERE I HAVE AN ISSUE

I use lftp (I tried few other with pretty much the same result, so I ruled out the client issue… but I pretty new to the whole ftp thing so… I let you judge)

1. Using lftp with ftps://

lftp -u user,password ftps://172.17.0.2 -p 31581
then ls

ls: Fatal error: gnutls_handshake: An unexpected TLS packet was received

Here is the very long version using debug (I truncated many "GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65" lines because of post limitations) :

lftp [email protected]:~> debug 99999999
lftp [email protected]:~> ls
FileCopy(0x563eb3eed780) enters state INITIAL
FileCopy(0x563eb3eed780) enters state DO_COPY
---- dns cache hit
---- attempt number 1 (max_retries=1000)
---- Connecting to 172.17.0.2 (172.17.0.2) port 31581
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65

[...]

GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: common.c[x509_read_value]:698
GNUTLS: ASSERT: common.c[x509_read_value]:698
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: REC[0x563eb42050a0]: Allocating epoch #0
GNUTLS: ASSERT: constate.c[_gnutls_epoch_get]:600
GNUTLS: REC[0x563eb42050a0]: Allocating epoch #1
GNUTLS: HSK[0x563eb42050a0]: Adv. version: 3.3
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_256_GCM_SHA384 (C0.2C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_CAMELLIA_256_GCM_SHA384 (C0.87)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_CHACHA20_POLY1305 (CC.A9)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_256_CCM (C0.AD)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_256_CBC_SHA1 (C0.0A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_128_GCM_SHA256 (C0.2B)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_CAMELLIA_128_GCM_SHA256 (C0.86)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_128_CCM (C0.AC)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_128_CBC_SHA1 (C0.09)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_3DES_EDE_CBC_SHA1 (C0.08)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_256_GCM_SHA384 (C0.30)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_CAMELLIA_256_GCM_SHA384 (C0.8B)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_CHACHA20_POLY1305 (CC.A8)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_256_CBC_SHA1 (C0.14)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_128_GCM_SHA256 (C0.2F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_CAMELLIA_128_GCM_SHA256 (C0.8A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_128_CBC_SHA1 (C0.13)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_3DES_EDE_CBC_SHA1 (C0.12)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_256_GCM_SHA384 (00.9D)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_256_GCM_SHA384 (C0.7B)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_256_CCM (C0.9D)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_256_CBC_SHA1 (00.35)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_256_CBC_SHA1 (00.84)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_128_GCM_SHA256 (00.9C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_128_GCM_SHA256 (C0.7A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_128_CCM (C0.9C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_128_CBC_SHA1 (00.2F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_128_CBC_SHA1 (00.41)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_3DES_EDE_CBC_SHA1 (00.0A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_256_GCM_SHA384 (00.9F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_256_GCM_SHA384 (C0.7D)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CHACHA20_POLY1305 (CC.AA)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_256_CCM (C0.9F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_256_CBC_SHA1 (00.39)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_256_CBC_SHA1 (00.88)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_128_GCM_SHA256 (00.9E)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_128_GCM_SHA256 (C0.7C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_128_CCM (C0.9E)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_128_CBC_SHA1 (00.33)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_128_CBC_SHA1 (00.45)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_3DES_EDE_CBC_SHA1 (00.16)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Extended Master Secret (0 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Encrypt-then-MAC (0 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension OCSP Status Request (5 bytes)
GNUTLS: HSK[0x563eb42050a0]: sent server name: '172.17.0.2'
GNUTLS: EXT[0x563eb42050a0]: Sending extension Server Name Indication (15 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Safe Renegotiation (1 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Session Ticket (0 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Supported curves (12 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Supported ECC Point Formats (2 bytes)
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (4.1) RSA-SHA256
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (4.3) ECDSA-SHA256
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (5.1) RSA-SHA384
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (5.3) ECDSA-SHA384
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (6.1) RSA-SHA512
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (6.3) ECDSA-SHA512
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (3.1) RSA-SHA224
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (3.3) ECDSA-SHA224
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (2.1) RSA-SHA1
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (2.3) ECDSA-SHA1
GNUTLS: EXT[0x563eb42050a0]: Sending extension Signature Algorithms (22 bytes)
GNUTLS: HSK[0x563eb42050a0]: CLIENT HELLO was queued [220 bytes]
GNUTLS: REC[0x563eb42050a0]: Preparing Packet Handshake(22) with length: 220 and min pad: 0
GNUTLS: ENC[0x563eb42050a0]: cipher: NULL, MAC: MAC-NULL, Epoch: 0
GNUTLS: REC[0x563eb42050a0]: Sent Packet[1] Handshake(22) in epoch 0 and length: 225
GNUTLS: ASSERT: buffers.c[get_last_packet]:1160
GNUTLS: ASSERT: buffers.c[_gnutls_io_read_buffered]:587
GNUTLS: ASSERT: buffers.c[get_last_packet]:1160
GNUTLS: ASSERT: buffers.c[_gnutls_io_read_buffered]:587
GNUTLS: ASSERT: buffers.c[get_last_packet]:1160
GNUTLS: REC[0x563eb42050a0]: SSL 48.48 Unknown Packet packet received. Epoch 0, length: 8271
GNUTLS: ASSERT: record.c[check_recv_type]:580
GNUTLS: Received record packet of unknown type 53
GNUTLS: ASSERT: record.c[recv_headers]:1093
GNUTLS: ASSERT: record.c[_gnutls_recv_in_buffers]:1175
GNUTLS: ASSERT: buffers.c[_gnutls_handshake_io_recv_int]:1415
GNUTLS: ASSERT: handshake.c[_gnutls_recv_handshake]:1474
GNUTLS: ASSERT: handshake.c[handshake_client]:2841
**** gnutls_handshake: An unexpected TLS packet was received.
GNUTLS: REC[0x563eb42050a0]: Start of epoch cleanup
GNUTLS: REC[0x563eb42050a0]: End of epoch cleanup
GNUTLS: REC[0x563eb42050a0]: Epoch #0 freed
GNUTLS: REC[0x563eb42050a0]: Epoch #1 freed
---- Closing control socket
ls: Fatal error: gnutls_handshake: An unexpected TLS packet was received.
lftp [email protected]:~> exit

2. Using lftp with ftp://

If I remove the s from ftp in the lftp command above, I get :

➜  repo git:(master) ✗ lftp -u user,password ftp://172.17.0.2 -p 31581
lftp [email protected]:~> set ssl:verify-certificate false
lftp [email protected]:~> ls
Interrupt
lftp [email protected]:/> debug 9
lftp [email protected]:/> ls
---- Connecting to 172.17.0.2 (172.17.0.2) port 31581
<--- 220 (vsFTPd 3.0.3)
---> FEAT
<--- 211-Features:
<---  AUTH TLS
<---  EPRT
<---  EPSV
<---  MDTM
<---  PASV
<---  PBSZ
<---  PROT
<---  REST STREAM
<---  SIZE
<---  TVFS
<---  UTF8
<--- 211 End
---> AUTH TLS
<--- 234 Proceed with negotiation.
---> OPTS UTF8 ON
Certificate: C=FR,ST=PARIS,L=Paris,O=42_Corp,OU=user,CN=email
 Issued by: C=FR,ST=PARIS,L=Paris,O=42_Corp,OU=user,CN=email
WARNING: Certificate verification: Not trusted (28:51:50:4B:83:40:96:E7:2C:9A:1E:9B:38:C5:25:46:B4:CC:4C:24)
WARNING: Certificate verification: certificate common name doesn't match requested host name ‘172.17.0.2’ (28:51:50:4B:83:40:96:E7:2C:9A:1E:9B:38:C5:25:46:B4:CC:4C:24)
<--- 200 Always in UTF8 mode.
---> USER ftp42
<--- 331 Please specify the password.
---> PASS 42
<--- 230 Login successful.
---> PBSZ 0
<--- 200 PBSZ set to 0.
---> PROT P
<--- 200 PROT now Private.
---> PASV
<--- 227 Entering Passive Mode (172,18,0,6,194,222).
---- Connecting data socket to (172.18.0.6) port 49886
Interrupt
---> LIST
---> ABOR
---- Closing aborted data socket
lftp [email protected]:/>
<--- 425 Failed to establish connection.
<--- 225 No transfer to ABOR.
lftp [email protected]:/>

TO SUM UP

I guess my issue is in the ssl configuration, but where ?… What am I missing ?
Thanks for reading this hidiously long post 🙂

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP