Surviving rm -rf *: How ZFS Copy-on-Write Saved My Home Directory

I've been using a special .bashrc config to sync command history across all open terminal sessions in real-time. It was a massive workflow upgrade. Yesterday, I told my workmate about it. Today? That exact feature betrayed me. I pressed the up arrow to recall a command I had just run in a junk directory, hit Enter without looking, and realized with creeping horror what I had just done.

I had executed rm -rf * directly inside my /home/maciej/ directory.

Years of scripts, configurations, and a brand-new project I had just started pouring my soul into: gone in about two seconds. Of course, I had backups. They contained absolutely everything, except the new project. If I were using ext4 or NTFS, I would be writing a very different, much sadder post right now. But because my /home was sitting on a ZFS pool, I actually had a time machine. I immediately turned my server off and started thinking. So, here is the ultimate manual on how to recover deleted files from ZFS without snapshots using Transaction Groups (TXG).

Why The Data Isn't Actually Gone

ZFS is a Copy-on-Write (CoW) filesystem. When you delete files, ZFS doesn't overwrite those sectors. It simply updates its metadata, the filesystem's "table of contents" to unlink those files and mark the blocks they occupy as free for future use.

Every few seconds, ZFS bundles all these changes into a Transaction Group (TXG) and writes a new "Uberblock" (a master pointer) to the disks. ZFS keeps a history of the last 128 Uberblocks. This means you have a rolling window of about 10 minutes where the old table of contents still exists on the disk, pointing to your "deleted" files.

The Recovery Manual: Step-by-Step

0. The Preparation

The very first thing you must do is stop the bleeding. Shut down your server immediately to prevent your OS from writing any new data (logs, cache, updates) over your newly freed blocks.

To perform the recovery safely, you need to boot your machine from a different drive. I highly recommend preparing an Ubuntu Live USB. Ubuntu is perfect for this because it comes with ZFS tools pre-installed (or they are just a quick sudo apt update && sudo apt install zfsutils-linux away in the live RAM environment).

Boot into the Live OS, open the terminal, and do not import your pool normally. Leave your disks completely untouched. We are going to strictly control how the system accesses them.

1. Find Your TXG "Time Machine"

First, you need to find the ID of an Uberblock from before you ran the fatal command. Use the ZFS debugger tool zdb on one of your raw drives (e.g., /dev/sda (yes, just one drive from your pool)):

sudo zdb -ul /dev/sda | less

Look at the timestamps and pick a txg number from before the disaster.

2. The Read-Only "Extreme Rewind" Import

This is the magic command. You are going to import the pool into a temporary recovery folder, forcing it to use the old TXG, and strictly enforcing read-only mode so you don't accidentally overwrite the current state.

sudo zpool import -N -f -R /mnt/recovery -o readonly=on -T <YOUR_TXG_NUMBER> <POOL_NAME>

• -N: Do not mount the filesystem automatically. (Crucial for avoiding chaos).
• -f: Force the import (bypasses "pool was mounted somewhere else" errors).
• -R /mnt/recovery: Uses an alternate root path so it doesn't conflict with your live system.
• -o readonly=on: Absolutely essential. Do not skip this! We don't want to overwrite our data.
• -T: Specifies the exact Transaction Group to roll back to.

3. The Waiting Game

If you are using NVMe drives, this takes minutes. If you are like me, running spinning rust (HDDs), prepare to wait.

My import took hours. I was watching iostat -k 2 and basic iostat (very cool to see the total amount of data already read) like a hawk. I saw my drives pushing 150,000 kB/s with 200-300 TPS. ZFS was painstakingly reconstructing the old metadata tree in RAM, forcing the HDD read heads to jump all over the platters. Be patient. Do not press CTRL+C. Let it finish.

I chose the oldest txg available (2373763). I waited 7 hours and... failed:

cannot import 'HDD': one or more devices is currently unavailable

That really scared me. I was practically screaming, "OH NO, MY DATA!" I frantically scoured literaly every article on the web about txg rollback, and it seemed like everyone was hitting the exact same wall. At one point, I read a forum comment asking, "Has anyone actually ever done this successfully?" - that completely froze me.

But I couldn't give up. I tried one newer TXG (2373764). And what?

cannot import 'HDD': one or more devices is currently unavailable

Now I was really shitting my pants. I tried once again with an even newer TXG and... Success!

4. Mount the Dataset and Extract

Once the terminal prompt returns, your pool is loaded in the background. Now, mount just the specific dataset you need:

sudo zfs mount <POOL_NAME>/home

Navigate to /mnt/recovery/home. Seeing my deleted files sitting right there felt like witnessing a miracle.

Plug in an external drive and use rsync to pull the data out safely:

sudo rsync -avhHAX --numeric-ids --progress /mnt/recovery/home/ /media/external_drive/backup/

This is the ultimate 1:1 rsync copy with correct attributes, ownerships, etc. I recommend running it with --exclude='node_modules' and other dirs with lots of small files, to make transfers faster.

You can see the damage on files using zpool status. Check the CKSUM column before and after reading data.

	NAME                                          STATE     READ WRITE CKSUM
	HDD                                           ONLINE       0     0     0
	  raidz1-0                                    ONLINE       0     0     0
	    ata-yourdisk                              ONLINE       0     0     0
	    ata-yourdisk                              ONLINE       0     0     0
	    ata-yourdisk                              ONLINE       0     0     0

If your value right after import is high-it's bad (metadata is damaged). If it stays at 0 but grows during the rsync copy-it means some specific file data was already overwritten. In that case, you can actually see what files are damaged! Just run zpool status -v and you'll get a list. In my case, a few node_modules files and one .git object file were damaged. For me, it was completely fine. If your files are critical, you can try to unmount the dataset, export the pool, and try again with a slightly newer txg until you are satisfied.

5. Clean Up and Return to Reality

Once you have safely backed up all your recovered data, clean up the recovery session:

sudo zfs unmount <POOL_NAME>/home
sudo zpool export <POOL_NAME>

Finally, import your pool normally (without the -T or -o readonly flags) to boot back into the present, and copy your recovered files back into your /home folder.

Final Tips

1. Don't Panic. Stop writing to the disk immediately. Every new megabyte you download or log you write could overwrite your unlinked blocks.
2. RAM is your friend. Do not restart your computer or clear your cache during this process. Keeping the metadelta in your RAM makes iterating through TXGs lightning fast. (Just beware: if you run a massive rsync right after an import, it will flush this cache, and your next import will be slow again!)
3. Use Snapshots! I survived this purely because of the underlying architecture of ZFS, but this was a highly stressful, multi-hour ordeal. A simple zfs-auto-snapshot cron job would have turned this 4-hour heart attack into a 5-second zfs rollback command. Set it up today.