Can't run fsck or format device - device is in use

today I crashed my boot partition. While is normally no problem due backups, I can’t write to the partition anymore.

root@hades ~ # touch /boot/a
touch: cannot touch '/boot/a': Structure needs cleaning

dmesg shows me:

[317873.920810] EXT4-fs error (device md2): ext4_find_dest_de:1932: inode #2: block 518: comm touch: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0, size=1024

Even after successfully umount the device, I can’t fsck

root@hades ~ # umount /boot
root@hades ~ # e2fsck /dev/md2
e2fsck 1.45.5 (07-Jan-2020)
/dev/md2 is in use.
e2fsck: Cannot continue, aborting.


root@hades ~ # fsck.ext4 /dev/md2
e2fsck 1.45.5 (07-Jan-2020)
/dev/md2 is in use.
e2fsck: Cannot continue, aborting.

I’m unable to reformat or destroy the partition:

root@hades ~ # wipefs /dev/md2
DEVICE OFFSET TYPE UUID                                 LABEL
md2    0x438  ext4 2112045a-2e84-4c69-8c67-d76e8ad7a13a boot
root@hades ~ # wipefs -a /dev/md2
wipefs: error: /dev/md2: probing initialization failed: Device or resource busy
root@hades ~ # mkfs.ext4 -L boot /dev/md2
mke2fs 1.45.5 (07-Jan-2020)
/dev/md2 contains a ext4 file system labelled 'boot'
    last mounted on /boot on Fri Sep 18 21:33:34 2020
Proceed anyway? (y,N) y
/dev/md2 is apparently in use by the system; will not make a filesystem here!

I also can’t stop the mdam array to set the while raid array to fail:

root@hades ~ # mdadm --stop /dev/md2 --force
mdadm: Cannot get exclusive access to /dev/md2:Perhaps a running process, mounted filesystem or active volume group?


root@hades ~ # mdadm /dev/md2 --fail /dev/nvme0n1p2 --remove /dev/nvme0n1p2
mdadm: set /dev/nvme0n1p2 faulty in /dev/md2
mdadm: hot removed /dev/nvme0n1p2 from /dev/md2
root@hades ~ # mdadm /dev/md2 --fail /dev/nvme1n1p2 --remove /dev/nvme1n1p2
mdadm: set device faulty failed for /dev/nvme1n1p2:  Device or resource busy

mdadm --add /dev/md2 /dev/nvme0n1p2

root@hades ~ # mdadm /dev/md2 --fail /dev/nvme1n1p2 --remove /dev/nvme1n1p2
mdadm: set /dev/nvme1n1p2 faulty in /dev/md2
mdadm: hot removed /dev/nvme1n1p2 from /dev/md2
root@hades ~ # mdadm /dev/md2 --fail /dev/nvme0n1p2 --remove /dev/nvme0n1p2
mdadm: set device faulty failed for /dev/nvme0n1p2:  Device or resource busy

fuser or lsof does now shoe any process except kernel threads:

root@hades ~ # lsof | grep md2
md2_raid1     262                              root  cwd       DIR                9,4     4096          2 /
md2_raid1     262                              root  rtd       DIR                9,4     4096          2 /
md2_raid1     262                              root  txt   unknown                                        /proc/262/exe
jbd2/md2-     599                              root  cwd       DIR                9,4     4096          2 /
jbd2/md2-     599                              root  rtd       DIR                9,4     4096          2 /
jbd2/md2-     599                              root  txt   unknown                                        /proc/599/exe
root@hades ~ # fuser -v /dev/md2
root@hades ~ # fuser -v /dev
                     USER        PID ACCESS COMMAND
/dev:                root     kernel mount /dev
                     root         57 .rc.. kdevtmpfs

I’m not using LVM, no userspace is installed:

root@hades ~ # vgs

Command 'vgs' not found, but can be installed with:

apt install lvm2

The mdadm RAID looks clean:

root@hades ~ # mdadm -D /dev/md2
/dev/md2:
           Version : 1.2
     Creation Time : Wed Jul 17 22:22:07 2019
        Raid Level : raid1
        Array Size : 523712 (511.44 MiB 536.28 MB)
     Used Dev Size : 523712 (511.44 MiB 536.28 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Tue Sep 22 14:30:52 2020
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : rescue:boot
              UUID : b8cce71f:11cb0913:e75f8623:0846cd86
            Events : 176

    Number   Major   Minor   RaidDevice State
       3     259        2        0      active sync   /dev/nvme0n1p2
       2     259        9        1      active sync   /dev/nvme1n1p2

I’m running Ubuntu 20.04.

Linux hades 5.4.0-47-generic #51-Ubuntu SMP Fri Sep 4 19:50:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linu

I can’t restart the server, because the system isn’t bootable now. Currently I do not have physical access to the machine.

I already shutdown all services except systemd, ssh, polkit, rsyslog and mdadm monitor.

How I could identify the source which is using the device?