TransWikia.com

ZFS send/receive fails consistently with "Input/output error"

Server Fault Asked by Pryo on December 11, 2021

I have a ZFS disk with deduplication turned on. I have copied data from many other old hard disks onto this ZFS disk, ultimately with the intention of getting rid of all the original disks. I need to make a backup copy of the ZFS disk though, but I’m having terrible trouble getting this to work.

For the backup disk, I prepared a second disk of similar size to the source disk and have plugged it into the same machine. They’re both USB external drives. I created a snapshot on the source disk and then used the following instruction to send it to the backup disk:

zfs send srcpool/dedup@snap1 | zfs recv backuppool/dedup

But this would never complete (for reasons I can’t quite remember). To try to rule out some silly configuration or command syntax mistake, I have since been trying to use syncoid instead. However, this fails consistently too, each time after a day or two of copying time, like so:

$ sudo syncoid srcpool/dedup backuppool/dedup                                                                                                                                                       
INFO: Sending oldest full snapshot srcpool/dedup@snap1 (~ 3227.7 GB) to new target filesystem:
warning: cannot send 'srcpool/dedup@snap1': Input/output error=============================================>                                                                                         ] 46% ETA 21:18:35
1.45TiB 18:15:37 [23.2MiB/s] [=========================================================================>                                                                                         ] 46%             
cannot receive new filesystem stream: checksum mismatch or incomplete stream.
Partially received snapshot is saved.
A resuming stream can be generated on the sending system by running:
    zfs send -t 1-b12345678-b8-1234567890
CRITICAL ERROR:  zfs send  'srcpool/dedup'@'snap1' | mbuffer  -q -s 128k -m 16M 2>/dev/null | pv -p -t -e -r -b -s 3465678004032 |  zfs receive  -s -F 'backuppool/dedup' failed: 256 at /usr/sbin/syncoid line 474.

When then trying to resume, I get the following instantly and consistently:

$ sudo syncoid srcpool/dedup backuppool/dedup                                                                                                                                                   
Resuming interrupted zfs send/receive from srcpool/dedup to backuppool/dedup (~ 1738.2 GB remaining):
warning: cannot send 'srcpool/dedup@snap1': Input/output error
 264KiB 0:00:00 [ 745KiB/s] [>                                                                                                                                                                    ]  0%            
cannot receive resume stream: checksum mismatch or incomplete stream.
Partially received snapshot is saved.
A resuming stream can be generated on the sending system by running:
    zfs send -t 1-b87654321-b8-0987654321
CRITICAL ERROR:  zfs send  -t 1-b12345678-b8-1234567890 | mbuffer  -q -s 128k -m 16M 2>/dev/null | pv -p -t -e -r -b -s 1866392479776 |  zfs receive  -s -F 'backuppool/dedup' 2>&1 failed: 256 at /usr/sbin/syncoid line 559.

Neither exporting/reimporting the disks nor restarting the whole machine lets me get passed this point. It will continuously say cannot receive resume stream: checksum mismatch or incomplete stream while trying to resume.

So I have instead destroyed backuppool (on the backup disk) and tried again, a few times. Each time the process eventually fails with the "Input/output error" as above, roughly mid-way through the process.

I am currently running the process yet again, with the syncoid "–debug" option, but it’ll no doubt be another day and a half until it fails again with (hopefully) more info.

Any hints for how I can debug or resolve this would be greatly appreciated! Thanks.

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP