I am a Data Hoarder or: How to Create a Btrfs RAID on Multiple Encrypted Disks
This is what my computer must feel like.
Original image from Alan Denney on Flickr. Some rights reserved: by-nc-sa.
I finally gave in to my data hoarding tendencies and bought another hard drive for my media server. I told myself last time "three terabytes is plenty, nobody could use up all that space." How wrong I was. I bought a 4 TB drive this time, but I'm sure that it too will be full before long.
Anyway, since I've already got a substantial data hoard on my existing 3 TB drive, I thought I'd try out the LVM and RAID features in Btrfs to use my new hard drive as part of a larger 7 TB array. I also started encrypting all of my hard drives about two years ago, but I haven't been able to encrypt my video hoard since there's never anywhere big enough for it to sit while I encrypt its home. That problem is solved with the addition of a larger disk, so I'll be doing that today as well.
Setting Up the New Disk
The first step, of course, is to set up my new hard drive. This involves creating an encrypted partition and formatting it as Btrfs, both pretty straightforward things. I've covered disk encryption in my previous post about installing Arch on an encrypted drive, but this time it's a little different because my media server is headless, meaning that it needs to be able to mount the encrypted partitions without any input. For that to happen, I have to use a key file rather than a password.
There are two big disadvantages to using key files: anybody with the key file can decrypt the partitions and if I lose the key file, the data is basically gone. Fortunately, LUKS allows multiple keys to be used to decrypt a partition, so I'll add a password to one of the other key-slots so that I can get to my data if I lose the key file.
Before doing anything though, the new disk needs a partition table. I like using cfdisk for this, but any partitioner will do. I won't bother explaining how to do this since I just made one big partition for the encrypted container, which is a pretty simple operation.
Generating a Key File
The next thing to do, then, is to create a key file. I'll just take 256 bytes of randomness and dump it into a file, giving me a 2048-bit key.
dd if=/dev/urandom of=~/keyfile bs=1 count=256
Creating the LUKS Container and Btrfs File System
My server is running Ubuntu Server 14.04, so we'll have to install the crypto and Btrfs packages before continuing.
apt-get install cryptsetup btrfs-tools
The new hard drive is /dev/sdc, so let's create a new container there, first with the key file and then with a password in the next key slot.
cryptsetup --key-file=keyfile luksFormat /dev/sdc1
cryptsetup --key-file=keyfile -y luksAddKey /dev/sdc1
cryptsetup --key-file=keyfile luksOpen /dev/sdc1 newdrive
Now that we have our encrypted partition available at /dev/mapper/newdrive, we can create the file system and mount it.
mkfs.btrfs /dev/mapper/newdrive
mount /dev/mapper/newdrive /media/newdrive
Adding the Old Disk to the New Btrfs Pool
Since I want to keep all of my files, I'll copy them from the old hard drive before encrypting and reformatting it.
cp -a /srv/media/* /media/newdrive/
After this (very, very long) process is complete, we're ready to set up the old hard drive (/dev/sdb) and add it to our storage pool.
umount /srv/media
cryptsetup --key-file=keyfile luksFormat /dev/sdb1
cryptsetup --key-file=keyfile -y luksAddKey /dev/sdb1
cryptsetup --key-file=keyfile luksOpen /dev/sdb1 oldmedia
btrfs device add /dev/mapper/oldmedia /media/newdrive
With no options, Btrfs uses RAID 0 for data and RAID 1 for the metadata, which means that while the data is split across both drives, each drive has a complete copy of the metadata. At this point then, df should show a 7 TB partition where before there were two: a 3 TB one and a 4 TB one. Before using this fancy new array however, we need to re-balance the file system so that the chunks are evenly distributed across the two disks. To do this, we just issue this command:
btrfs filesystem balance /media/enc-media
The balance will take a long time if you have a lot of data to move around, so just sit quietly and wait until it finishes. Once that's done, everything is ready to be used, but mounting the disks is still pretty cumbersome:
cryptsetup --key-file=keyfile luksOpen /dev/sdb1 enc1
cryptsetup --key-file=keyfile luksOpen /dev/sdc1 enc2
mount /dev/mapper/enc1 /srv/media
Much worse than with a single unencrypted partition. Fortunately, the cryptsetup package lets us decrypt the partitions at boot time if we tell it some details, like where they keys are or which devices are encrypted.
Mounting the Pool at Boot
This part of the process caused me the most headache by far. I ran into a few race conditions between the decryption and the system trying to mount the Btrfs array, but I finally worked it out. The first step is to add the new array to /etc/crypttab so both disks get decrypted at boot.
# Encrypted media disks
enc2 UUID=8621e822-7baa-4760-a1bb-0a6d6fa57aab /root/keyfile luks
enc1 UUID=5073eafb-441f-4813-b622-6657d7295c31 /root/keyfile luks
The part that took me a while to work out is what options /etc/fstab requires, but here's what works for me:
/dev/mapper/enc1 /srv/media btrfs defaults,device=/dev/mapper/enc1,device=/dev/mapper/enc2 0 0
The important piece is device=/dev/mapper/enc1,device=/dev/mapper/enc2. Without this, mount can't figure out how to mount the partition without running btrfs device scan, which it doesn't do at boot.
One drawback to this approach is that adding more disks in the future is a bit of a pain since they'll have to be added to both /etc/crypttab and /etc/fstab, but hopefully I won't be doing that more than once every year or so if I can get my addiction under control.