TrueNAS Scale data recovery

My TrueNAS Scale server recently started throwing the following zfs module kernel panic ¹ on boot:

PANIC at space_map.c:405:space_map_load_callback()

What follows are the steps I took to retrieve the data from the corrupted zfs pool.

Getting setup

• boot from a Ubuntu 22.04 Live USB disk. Ubuntu (unlike Fedora) supports zfs without needing to compile kernel modules.
• install the zfsutils-linux package:

$ apt install zfsutils-linux

• import the pool as readonly (attempting to import read/write causes the kernel to panic!)

$ zpool import -o readonly=on -f Seagate4TB -R /mnt

Pool file systems are now available readonly under /mnt. In my case, I had a file system that I was using for NFS storage. Volumes have to be accessed a different way - see below.

Using a new pool

In my case, the data I really needed was inside a zvol which had an XFS filesystem on it. Attempting to mount that directly from the readonly pool resulted in an XFS error due to it being read-only. The solution was to copy the zvol to a new pool.

The new pool needs to reside somewhere with sufficient free space to hold data exported from the readonly pool. I used a spare external USB hard disk I had lying around. Instead of using the USB disk as a block device for the pool, I created a sparse file on the USB disk and used that for the pool’s storage.

• create a 100G sparse file

$ dd if=/dev/zero of=/external-usb/testpool bs=1 count=0 seek=100G

• create the new pool on the sparse file

$ zpool create testpool /external-usb/testpool

• export the zvol from the readonly pool to the newly created pool

$ zfs send Seagate4TB/Fedora-home | zfs receive testpool/Fedora-home

• mount the zvol from the new pool to a temporary location

mount /dev/zvol/testpool/Fedora-home /mnt2

• retrieve the needed files from /mnt2