Context

I have a systemd service. (I wrote the contrail-vrouter-dpdk.service file myself, included below.)

When I run sudo systemctl is-active contrail-vrouter-dpdk it says “deactivating”.

When I run sudo systemctl status contrail-vrouter-dpdk is says:

Active: deactivating (stop-sigterm) since Fri 2019-04-26 15:28:49 AEST; 39min ago

(Full output below)

When I run sudo systemctl kill contrail-vrouter-dpdk it returns straight away, return code zero, nothing printed. But the status remains the same.

When I run systemctl start (or restart) it hangs forever, no output.

When I try to reboot the machine, the shut-down hangs, waiting for this service to deactivate. I have to do a forcefull power down to actually make it restart.

Question

How can I restart a systemd process if it is stuck on deactivating? Why doesn’t kill actually kill the process?

Full detail

systemctl status

● contrail-vrouter-dpdk.service - Contrail vrouter dpdk service
   Loaded: loaded (/etc/systemd/system/contrail-vrouter-dpdk.service; enabled; vendor preset: enabled)
   Active: deactivating (stop-sigterm) since Fri 2019-04-26 15:28:49 AEST; 39min ago
 Main PID: 5517 (contrail-vroute)
    Tasks: 8 (limit: 12287)
   CGroup: /system.slice/contrail-vrouter-dpdk.service
           ├─5517 [contrail-vroute]
           └─5534 /home/ubuntu/src/vrouter/build/production/vrouter/dpdk/contrail-vrouter-dpdk --no-daemon --socket-mem 1024 1024

Apr 26 15:26:14 dell01 contrail-vrouter-dpdk[5517]: 2019-04-26 15:26:14,428 VROUTER:     lcore 10 RX from HW queue 1
Apr 26 15:28:49 dell01 systemd[1]: Stopping Contrail vrouter dpdk service...
Apr 26 15:28:49 dell01 contrail-vrouter-dpdk[5517]: 2019-04-26 15:28:49,315 VROUTER: Got signal 15 on lcore 0, stopping...
Apr 26 15:28:49 dell01 contrail-vrouter-dpdk[5517]: 2019-04-26 15:28:49,339 VROUTER: Releasing KNI devices...
Apr 26 15:28:49 dell01 contrail-vrouter-dpdk[5517]: 2019-04-26 15:28:49,339 VROUTER: Releasing TAP devices...
Apr 26 15:28:49 dell01 contrail-vrouter-dpdk[5517]: 2019-04-26 15:28:49,339 VROUTER: Closing eth devices...
Apr 26 15:28:49 dell01 contrail-vrouter-dpdk[5517]: 2019-04-26 15:28:49,354 VROUTER: Got signal 15 on lcore 0, stopping...
Apr 26 15:53:50 dell01 contrail-vrouter-dpdk[5517]: 2019-04-26 15:53:50,790 VROUTER: Got signal 15 on lcore 4294967295, stopping...

contrail-vrouter-dpdk.service

[Unit]
Description=Contrail vrouter dpdk service
After=network.target
#Wants=contrail-vrouter-agent.service

[Service]
Type=simple
ExecStart=/home/ubuntu/src/vrouter/build/production/vrouter/dpdk/contrail-vrouter-dpdk --no-daemon --socket-mem 1024,1024
TimeoutStartSec=240
TimeoutStopSec=0
Restart=always
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=yes
ProtectHome=no
ExecStartPre=/opt/contrail/bin/vrouter-pre-start.sh
ExecStartPost=/opt/contrail/bin/vrouter-post-start.sh
ExecStopPost=/opt/contrail/bin/vrouter-post-stop.sh
ReadWriteDirectories=-/var/log/contrail
ReadWriteDirectories=-/var/lib/contrail
ReadWriteDirectories=-/var/crashes
ReadWriteDirectories=-/var/run
ReadWriteDirectories=-/dev
LimitCORE=infinity
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=contrail-vrouter-dpdk
User=root
CPUAffinity=2,3

[Install]
WantedBy=multi-user.target

# systemctl status logstash.service
● logstash.service - logstash
   Loaded: loaded 
  Drop-In: /etc/systemd/system/logstash.service.d
   Active: deactivating (stop-sigterm)
 Main PID: 20410 (java)
   CGroup: /system.slice/logstash.service
           └─20410 /usr/bin/java

# kill -9 20410