If you're in a hurry, download and read the bash script, as well as the kernel update script. You'll also need an authorized_keys files in the same directory as the previous two scripts.
Just two language related notes before we go any further
Let me explain what kind of risk this setup is supposed to mitigate, in other words what the threat model is. If someone breaks into your place and steals your server where you've stored personal data, that sucks. That person could simply take out the micro SD card in your server, put it into a computer and read everything on it. If you encrypt the root partition, this becomes practically impossible. As soon as power is removed from the server, it will shut down, and reading anything from it will require the passphrase to unlock the encrypted partition.
Some people are of the opinion that encrypting the root partition on a server doesn't matter, because the main attack surface is via the services exposed to the internet. I agree this is the most important attack surface, but I also want to guard against someone physically picking up the server and just reading all the data stored on it.
Now I also don't want this encrypted root to be too much of a bother. For example, if the power goes out and the server reboots, it will need you to unlock the encrypted partition before it can finish booting. That's where unlocking over SSH is useful: you don't need to be where the server is physically located to type in the passphrase, you can just ssh in and enter the passphrase. If your server is open to the internet, you can do this from anywhere you have internet access.
You can take this a step further and use a backup power supply to your server, so it doesn't reboot during power cut and you won't need to remotely unlock it. But this is beyond the scope of this post.
Here I will go through the bash installation script step-by-step and explain what each part does
First, to get started you should be in a directory which has three files: the bash installation script, the kernel update script, and an authorized_key file which contains the public keys that can log in to the server. This file should have one or several entries that look like `ssh-rsa LongPublicKeyString user@host`. Note I believe the SSH server used to unlock the root partition during boot (dropbear) currently does not support ed25519 keys, so you should use rsa ssh keys.
Second, you should download a copy of raspberry pi OS from here, choose Raspberry Pi OS lite buster, in the "legacy" section. You can try with the newer bullseye release, but I haven't personally tested it, so it may or may not work. Install the OS to a micro SD card with `dd if=raspios.img of=/dev/sdX bs=4M status=progress`, replacing X with the device that corresponds to your micro SD card. Watch out! If by mistake you specify your computer's internal storage this command will overwrite whatever is there, likely rendering your computer unusable. Double check you've got the right micro SD with the `lsblk` command. Repeat the process with the second microSD card.
Put either one of the freshly imaged microSDs in the pi and boot it up. Login with `pi` and password `raspberry` (the defaults). Once it's booted up and you've checked it works, do the same thing with the second microSD card. Enable ssh with `systemctl enable --now ssh`. Connect the pi to the network, transfer the bash installation script, kernel update script, `authorized_key` file, and log in via SSH.
Put the other microSD card in a USB microSD reader and plug it in to the pi.
#!/bin/bash
set -e
This immediately stops the script if one command throws an error. Good if you forgot something which makes the script fail.
# Port setup for ssh and dropbear. Dropbear unlocks root partition
during boot
SSH_PORT=22
DROPBEAR_PORT=22
These are the ssh ports used to unlock the encrypted partition during boot, and then to login normally once the system is booted. Ideally, change the SSH and dropbear ports to some random number above 10'000.
# Device to partition
DEVICE=/dev/sdb
BOOT_PART="${DEVICE}1"
ROOT_PART="${DEVICE}2"
ROOT_PATH=vgroup
CRYPT_NAME=cryptroot
This defines where the encrypted file system will be installed. Again, check with `lsblk` that '/dev/sdb' is indeed the microSD card in the USB adapter.
# Install needed packages
apt install -y cryptsetup lvm2 busybox
# Encrypted root partition
echo "Choose a password for unlocking the encrypted root partition"
cryptsetup luksFormat $ROOT_PART
echo "Enter previous password to unlock encrypted root partition"
cryptsetup open $ROOT_PART ${CRYPT_NAME}
# LVM setup
# not sure why these lvm commands say '/dev/sda open failed: No medium found',
# doesn't seem to be an issue
pvcreate /dev/mapper/${CRYPT_NAME}
vgcreate ${ROOT_PATH} /dev/mapper/${CRYPT_NAME}
# Calculate lvm size to fill root partition
LVM_SIZE="$(blockdev --getsz $ROOT_PART)"
let LVM_SIZE=LVM_SIZE/2/1024/1024
echo "size of lvm is ${LVM_SIZE}"
lvcreate -L ${LVM_SIZE}G ${ROOT_PATH} -n root
mkfs.ext4 /dev/mapper/${ROOT_PATH}-root
BOOT_UUID="$(blkid -o value -s UUID $BOOT_PART)"
echo "BOOT_UUID: $BOOT_UUID"
ROOT_UUID="$(blkid -o value -s UUID $ROOT_PART)"
echo "ROOT_UUID: $ROOT_UUID"
This does all the partitioning and creation of encrypted and LVM partitions. You will have to choose a passphrase for the encrypted root partition at this point.
# Syncing the root fs to the 2nd uSD
mount /dev/mapper/${ROOT_PATH}-root /mnt/
mkdir /mnt/boot
mount $BOOT_PART /mnt/boot
rsync -a --info=progress2 --del --exclude '/dev/*' --exclude '/proc/*'
--exclude '/sys/*' --exclude '/tmp/*' --exclude '/run/*' --exclude '/mnt/*'
--exclude '/media/*' --exclude '/lost+found' / /mnt/
sync
This part mounts the microSD in the adapter and copies over the root filesystem to it. This can take a few minutes, so be patient.
# Changing boot parameters for the encrypted root
sed -i 's/root=\S\+/cryptdevice=${ROOT_UUID}:${ROOT_PATH}-root
root=\/dev\/mapper\/'${ROOT_PATH}-root'/' /mnt/boot/cmdline.txt
echo "initramfs initramfs.gz followkernel" >> /mnt/boot/config.txt
This section adds kernel parameters indicating where the root device is, and that it is an encrypted partition. The second line also tells the kernel to boot from an `initramfs.gz` file. This part is raspberry pi specific. So it won't work if you're installing it on a regular desktop, laptop, or a single board computer with Armbian.
# Prepare chroot
mount -o rbind /dev /mnt/dev/
mount -t proc /proc /mnt/proc
mount -t sysfs /sys /mnt/sys
# Change SSH port
sed -i 's/#Port .*/Port '$SSH_PORT'/' /mnt/etc/ssh/sshd_config
# Install, configure dropbear and copy over authorized ssh keys
chroot /mnt/ apt install -y dropbear-initramfs cryptsetup-initramfs
sed -i 's/#CRYPTSETUP=.*/CRYPTSETUP=Y/' /mnt/etc/cryptsetup-initramfs/conf-hook
sed -i 's/.*DROPBEAR_OPTIONS.*/DROPBEAR_OPTIONS="-p '$DROPBEAR_PORT'"/' /mnt/etc/dropbear-initramfs/config
cp ./authorized_keys /mnt/etc/dropbear-initramfs/
This part prepares the chroot, changes the default SSH port to what was specified in the beginning, and installs and configures dropbear, which is the SSH daemon that runs during boot before the encrypted root partition is unlocked.
# Setup unlocking and mounting of root partition
echo "${CRYPT_NAME} UUID=${ROOT_UUID} none luks" >> /mnt/etc/crypttab
sed -i 's/PARTUUID=.*\/\s/\/dev\/mapper\/'${ROOT_PATH}-root' \/ /' /mnt/etc/fstab
# Add mounting boot partition to fstab
sed -i 's/PARTUUID=.* \/boot/UUID='$BOOT_UUID' \/boot/' /mnt/etc/fstab
This sections adds the relevant entries to the `crypttab` and `fstab` files. The `crypttab` entry is for the proper unlocking of the encrypted root partition. The `fstab` entries are for the proper mounting of the unlocked root partition and boot partition.
# Rebuild initramfs in /boot
chroot /mnt/ mkinitramfs -o /boot/initramfs.gz
# Add script for initramfs rebuild following kernel upgrades
cp initramfs-rebuild.sh /mnt/etc/kernel/postinst.d/initramfs-rebuild
chmod +x /mnt/etc/kernel/postinst.d/initramfs-rebuild
# Flush filesystem buffers
sync
This final section rebuilds the kernel image, so that unlocking of the encrypted root partition via dropbear works properly. The `initramfs-rebuild` file is raspberry specific and rebuilds the kernel whenever it is upgraded.
There, you should now have a raspberry pi with an encrypted root partition which you can unlock remotely.
Those with a keen an eye may notice that there are a few ways to improve this setup. Namely, the part with using two microSD cards and the use of LVM.
It's a little bit tedious to write the image to two microSD cards and then boot up both of them. What's even more inefficient is that one of them is first flashed with raspberry pi OS, just to be formatted by the installation script directly afterwards. This could be avoided by copying the bootloader from the first microSD card to the second, as this is the only part we actually keep from the raspberry pi OS image. The bootloader is located on the first few blocks of the microSD card, and can be copied with the `dd` command. I just haven't had the time or need to investigate this method.
LVM is great for making consistent backups of a running system. It works by taking a snapshot of the root filesystem, so that any changes made to the filesystem during the backup process are disregarded. However, the drawback of this is performance. When you make an LVM snapshot, it needs to keep track of every change to the underlying ext4 filesystem, potentially slowing down other processes running on the server. This isn't the end of the world, as the snapshot can be deleted once the backup is done, removing the performance penalty of the LVM snapshot.
BTRFS is a modern copy-on-write filesystem which has snapshot functionality built in. Taking a BTRFS snapshot has zero performance penalty on the filesystem. Again, this is a potential solution if you find that IO performance isn't satisfactory when you're runnig backups. In my daily usage I haven't found this to be an issue, so I haven't looked into switching to BTRFS. Another way I mitigate this problem is simply running backups when I'm not using the server for some intensive task.
Hope you got something useful out of this article, happy hacking!