RESOLVED: EXT4-fs (md1): unable to read superblock - really want to avoid data loss

my level of Linux knowledge is very light (I’m learning) and I’ve got a problem that’s beyond my level of Google-Fu. I’m well out of my depth and I’m hoping somebody might be able to help me…

I’m running OpenMediaVault 5.5.3-1 (Usul) in my homelab and have two software RAID arrays /dev/md0 and /dev/md1. When I logged on this morning I found that /dev/md1, a four-disk RAID5 set (aka /dev/disk/by-label/RAID5) was not available. It holds a lot of data I’d really prefer not to lose.

What I’ve done so far:

1. I checked the OpenMediaVault GUI which told me that raidset was "Missing"

2. I determined that one of the four disks that make up the Raidset wasn’t showing up at BIOS level. I reseated the cables and all four drives (sdc, sdd, sde and sdh) are now visible to the OS.

3. Upon reboot, the raidset is still showing as "missing" and there’s a message coming up in dmesg

   EXT4-fs (md1): unable to read superblock
   

4. /proc/mdstat confirms that md1 is inactive:

   root@OpenMediaTower:~# cat /proc/mdstat
   Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
   md1 : inactive sdh[3](S) sde[2](S) sdd[1](S) sdc[0](S)
         7813529952 blocks super 1.2
   
   md0 : active raid1 sdf[1] sda[0]
         3906886464 blocks super 1.2 [2/2] [UU]
         bitmap: 0/30 pages [0KB], 65536KB chunk
   
   unused devices: <none>
   

5. Through google I found a previous post that dealt with a somewhat similar problem and from that I’ve had a look at mdadm, but unforunately am insufficiently knowledgable to use the detail to dignose and address the issue

Output from mdadm –detail

root@OpenMediaTower:~# mdadm --detail --scan
ARRAY /dev/md0 metadata=1.2 name=OpenMediaTower:Mirror UUID=f7b1c667:3df80c11:975d87ad:126b5401
INACTIVE-ARRAY /dev/md1 metadata=1.2 name=RAID5 UUID=c813cb15:a9d26d51:7faada85:9b76b36d

Attempt to reassemble the raidset

root@OpenMediaTower:~# mdadm --stop
/dev/md1 mdadm: stopped /dev/md1

root@OpenMediaTower:~# mdadm --assemble --scan
mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.

Output from mdadm –examine

root@OpenMediaTower:~# mdadm --examine /dev/sd[dehc]
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : c813cb15:a9d26d51:7faada85:9b76b36d
           Name : RAID5
  Creation Time : Sun May 17 14:49:46 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : clean
    Device UUID : 42088567:13765c92:e2a5503b:c30355d0

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Jul 22 08:35:02 2020
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : dd641c67 - correct
         Events : 218937

         Layout : left-symmetric
     Chunk Size : 512K

    Device Role : Active device 0
    Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : c813cb15:a9d26d51:7faada85:9b76b36d
           Name : RAID5
  Creation Time : Sun May 17 14:49:46 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : active
    Device UUID : 69f98cc0:b818da43:b883695d:2246b3ab

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Jul 19 03:32:33 2020
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : e55cb645 - correct
         Events : 30843

         Layout : left-symmetric
     Chunk Size : 512K

    Device Role : Active device 1
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : c813cb15:a9d26d51:7faada85:9b76b36d
           Name : RAID5
  Creation Time : Sun May 17 14:49:46 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : active
    Device UUID : 04ec1c61:a7f1bb11:ee13bfe0:7153e38a

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jul 21 21:59:53 2020
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : b6579d04 - correct
         Events : 216993

         Layout : left-symmetric
     Chunk Size : 512K

    Device Role : Active device 2
    Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : c813cb15:a9d26d51:7faada85:9b76b36d
           Name : RAID5
  Creation Time : Sun May 17 14:49:46 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : clean
    Device UUID : 2af17abb:48930a97:31ce1fa5:850ac7d0

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Jul 22 08:35:02 2020
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : a0495199 - correct
         Events : 218937

         Layout : left-symmetric
     Chunk Size : 512K

    Device Role : Active device 3
    Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)

6. I ran e2fsck which reports a problem with the superblock. At this stage I have not attempted

   root@OpenMediaTower:~# e2fsck /dev/md1
   e2fsck 1.45.5 (07-Jan-2020)
   e2fsck: Invalid argument while trying to open /dev/md1
   
   The superblock could not be read or does not describe a validext2/ext3/ext4
   filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
   filesystem (and not swap or ufs or something else), then the superblock
   is corrupt, and you might try running e2fsck with an alternate superblock:
       e2fsck -b 8193 <device>
    or
       e2fsck -b 32768 <device>
   

7. I tried to reassemble the raidset, discretely naming each of the disks

   root@OpenMediaTower:~# mdadm --verbose --assemble /dev/md1 /dev/sdc /dev/sdd /dev/sde /dev/sdh
   mdadm: looking for devices for /dev/md1
   mdadm: /dev/sdc is identified as a member of /dev/md1, slot 0.
   mdadm: /dev/sdd is identified as a member of /dev/md1, slot 1.
   mdadm: /dev/sde is identified as a member of /dev/md1, slot 2.
   mdadm: /dev/sdh is identified as a member of /dev/md1, slot 3.
   mdadm: added /dev/sdd to /dev/md1 as 1 (possibly out of date)
   mdadm: added /dev/sde to /dev/md1 as 2 (possibly out of date)
   mdadm: added /dev/sdh to /dev/md1 as 3
   mdadm: added /dev/sdc to /dev/md1 as 0
   mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.
   

mdadm --stop /dev/md1
mdadm --assemble /dev/md1 --force /dev/mapper/sdc /dev/mapper/sde /dev/mapper/sdh