Server Upgrade / FreeBSD / ZFS and More

Today, Esther and I drove to Amsterdam to replace the old Dell R310 named ‘Pandora’ (when all the shit is out the box, only hope remains…) with the new Pandora, a Supermicro XEON-Beastly machine which I bought from Giorgio.

This post will be some form of write-up of what I did, so you can learn from my failure :’-)

The New Pandora…


So the Pandora runs FreeBSD 12, which I simply love. 4 years ago I started the Pandora with ZFS (Zetta File System from Sun Microsystems), which is a marvelous filesystem to work with.

Partitioning on the Pandora was as follows:

# gpart show
=> 34 250069613 ada0 GPT (119G)
34 1024 1 freebsd-boot (512K)
1058 4194304 2 freebsd-swap (2.0G)
4195362 245874285 3 freebsd-zfs (117G)

The ZFS-layout was the normal layout of FreeBSD:

# zfs list
tweede 8.56M 1.76T 88K /tweede
tweede/vms 88K 1.76T 88K /tweede/vms
zroot 49.3G 164G 88K none
zroot/ROOT 11.5G 164G 88K none
zroot/ROOT/default 11.5G 164G 10.7G /
zroot/poudriere 2.19G 164G 88K none
zroot/poudriere/jails 1.23G 164G 88K none
zroot/poudriere/jails/freebsd_12-1x64 1.23G 164G 1.23G /usr/local/poudriere/jails/freebsd_12-1x64
zroot/poudriere/ports 983M 164G 88K none
zroot/poudriere/ports/HEAD 983M 164G 900M /usr/local/poudriere/ports/HEAD
zroot/tmp 8.29G 164G 8.29G /tmp
zroot/usr 17.4G 164G 88K /usr
zroot/usr/home 13.0G 164G 13.0G /usr/home
zroot/usr/ports 4.46G 164G 4.46G /usr/ports
zroot/usr/src 88K 164G 88K /usr/src
zroot/var 9.58G 164G 88K /var
zroot/var/crash 88K 164G 88K /var/crash
zroot/var/log 1.04G 164G 1.04G /var/log
zroot/var/mail 8.54G 164G 8.53G /var/mail
zroot/var/tmp 1.09M 164G 1016K /var/tmp
zroot/vms 328M 164G 328M /vms

Except for Poudriere (And “tweede”, Second One), a straight-through layout in ZFS right?

ZFS Snaps, Sends and Receives

ZFS is a marvelous system. I wanted to make sure that in every case, the Old Pandora would stay alive (I do have backups with Tarsnap, but I don’t want to rely on those…). So snapping, sending and restoring the filesystem would solve this issue, right?

So I started installing FreeBSD on the new Pandora, which got me the partition table like I mentioned earlier. Then, I removed the pool (zpool destroy), and made the new zpool with ‘zpool create mirror zroot /dev/da0 /dev/da1’ (this command might be somewhat different).

Or, how to overwrite your own partition-table. #Fail

After I found this out, I did remove all the datapools in ZFS (zfs destroy $ALLTHETHINGS) on NewPandora, and start snapping, sending and receiving from the OldPandora. Aka:

zfs snap zroot/var/mail@bla
zfs send zroot/var/mail@bla > thisWouldBeYourSnap
gzip thisWouldBeYourSnap

and then using scp to copy the file from the OldPandora to the NewPandora

At the NewPandora I did the following:

gunzip thisWouldBeYourSnap.gz
zfs recv -f root/var/mail < thisWouldBeYourSnap

Works. So lets do this for all partitions. ZFS List, snap all, send all, gzip all, ssh all and then unzip everything, recv, right?

Wrong. Of course you need to learn the hard way that you can mount a Zpool in a /tmp directory so if you mount your zroot you would not be able to do anything left.

Restart from USB-disk:

mkdir /tmp/zroot
zpool import -f -R /tmp/zroot zroot

Gives you the full partitions on /tmp/zroot. Fail 2

From there on, I started snapping the zfs-datapools to the other side. You need to tell ZFS where the directory with the boot loader is, otherwise you will get a /boot/kernel/kernel that borks. Solution:

zpool get all
zroot bootfs zroot/ROOT/default local

Set the zpool:

zpool set bootfs=zroot/ROOT/default zroot

WORKING! (And is this a fail? Wellllll.. $failallthethings…

For some reason I couldn’t get /usr full at the other side. It remains a mystery for me until know WHY. Fail 3 (and a ego that is hurt…)

From there on I decided to have a bald move. Lets snap all the things recursively, and restore everything:

zfs snap -r zroot
zfs send -Rv zroot > SendAllTheThings
gzip $bla

you probably know the drill by now. Send the file, unpack it, and then restore everything. Set the bootfs correctly…


It has been some harsh weeks on learning and failing in ZFS (mainly). I got a sparkling machine running, and I must say that I’m very happy to learn every day.

I hope that this writeup gives you just the edge that I needed. If so, please let me know 🙂


Dit bericht is geplaatst in Daily stuff met de tags , , , . Bookmark de permalink.