snapshot all the things –
Let’s take a sneak ZFS peek under the hood of Ubuntu Eoan Ermine’s latest build.
Yesterday brought exciting news on the ZFS and Ubuntu fronts — experimental ZFS root support in the installer for Ubuntu’s upcoming interim release, Eoan Ermine. The featureappearedin the 2019 – 10 – 09 daily build of Eoan — it’s not in the regular beta release, and in fact wasn’t even in the “current daily” when we first went to download it. It’s that new! (Readers wanting to play with the new functionality can find it intodaydaily build, availablehere.)
For the ZFS newbies
If you’re new to theZFShype train, you might wonder why a new filesystem option in an OS installer is a big deal. So here’s a quick explanation: ZFS is a copy-on-write filesystem, which can takeatomic snapshotsof entire filesystems. This looks like sheer magic if you’re not used to it — a snapshot of a 10 TB filesystem can be taken instantly without interrupting any system process in the slightest. Once the snapshot is taken, it’s an immutable record of the exact, block-for-block condition of the filesystem at the moment in time the snapshot was taken.
When a snapshot is first taken, it consumes no additional disk space. As time goes by and changes are made to the filesystem, the space required to keep the snapshot grows by the amount of data that’s been deleted or altered. So let’s say you snapshot a 10 TB filesystem: the snapshot completes instantly, requiring no additional room. Then you delete a 5MB JPEG file — now the snapshot consumes 5MB of disk space, because it still has the JPEG you deleted. Then you change 5MB of data in a database, and the snapshot takes 10 MB — 5MB for the JPEG you deleted and another 5MB for the data that you altered in the database.
That’s just one awesome ZFS feature. There’s also the ability to manage multiple disks in a native RAID-like system, inline compression with selectable algorithms, rapid asynchronous incrementalreplication, and more. But we’re going to focus mostly on the snapshots here, because oneotherthing you can do with a snapshot is roll it back.
Although there isn’t any support built into Eoan’s apt package manager for automatically taking snapshots yet, we can demonstrate a snapshot — oops — rollback moment manually. In the above gallery, first we take a ZFS snapshot. Eoan has split our root filesystem into tons of little datasets (more on that later), so we use the- R
option forZFS Snapshot
to recursively take snapshots throughout the entire tree.
Now that we’ve insured ourselves against mistakes, we do something we’re going to regret. For the purposes of this demo, we’re just removing Firefox — but we could really recover from anything up to and including anrm -rf --no-preserve-root /
this way with a little extra legwork. After removing Firefox, we need to roll back our snapshots to restore the system to its original condition.
Since the root filesystem is scattered through abunchof individual datasets, we need to roll them all back individually. Although this is a pain for the casual user without additional tooling, it does make it possible to do more granular restore operations if we’re feeling picky — like rolling back the root filesystem without rolling back / home. Ubuntu will undoubtedly eventually have tooling to make this easier, but for the moment, we do a bit of sysadmin-fu and pipezfs list
togrep (to) AWK
toxargs
, oh my.
The command line acrobatics might have been obnoxious, but the rollback itself was instantaneous, and Firefox has returned. It still doesn’t work quite right, though, due toorphaned filehandles– we rolled back a live mounted root filesystem, which is kind of a cowboy thing to do. To make things entirely right, a reboot is necessary — but after the reboot, everything’s the way it once was, and without the need to wait through any lengthy Windows Restore Point styleGrovelingover the filesystem.
For the ZFS enthusiasts
In this section, we’re going to take a detailed look at just how Ubuntu is carving up the filesystems in Eoan’s experimental installer. The version in our daily build is 0.8.1, so this is great news for the ZFS fans among us, even without the experimental root installer — assuming the final version of Eoan follows this one, we’ll get native encryption, TRIM, device removal, and zpool checkpoints. Thesefeatureshave been in the ZFS on Linuxmastersince 0.8, but this is the first time they’ve shown up in Ubuntu’s native ZFS.
So far — remember, this is alpha software in a daily build — the installer doesn’t give you any control over how it carves up the disk when you select a ZFS install; it just does what it wants to do. The Eoan VM I created has a single 20 GB virtual disk. Eoan’s installer carved this into one primary partition and two logical — a small UEFI boot partition and partitions for two separate ZFS storage pools, named (bpool) andRpool
.
Bpool is pretty boring; it’s just where the system’s / boot directory gets mounted. Eoan made this pool 2GB, which is twice what a conservative/ boot
is normally provisioned to; this is probably to allow headroom to maintain a fairly deep archive of snapshots in the future.rpool
gets all the remaining disk space after the UEFI and (bpool) partitions are created; it’s where all the fun stuff goes, including your root filesystem, home directory, and so forth.
Beneathrpool
, you’ll find a pretty bewildering array of small datasets, all of which correspond to particular vital areas in what would normally be a single root filesystem. This appears to us to be an inherited BSD-ism — most Linux distributions left the concept of heavily partitioned disks with multiple filesystems behind twenty years ago, but FreeBSD — which has had root ZFS options in its installer for many years now — was a lot more stubborn about it.
The good thing about carving up the root filesystem into so many separate datasets is that you can snapshot and roll them back individually. In some cases, this is great — there’s an obvious, clear, and useful distinction between rolling back the root filesystem as a whole and rolling back your own home directory, for example. Most users — even pretty competent sysadmin types — will be a lot more confused about how and why you might want to roll back/ USR
without rolling back, say,/ var / lib / AccountServices
, though. It’s nice that youcanif you really want to, but we’re not so sure the capability outweighs the utility of a simpler approach.
Peeking a little deeper, we can see that Eoan isn’t setting any significant per-dataset properties on all these separate datasets. It is setting compression=lz4 on across the entire pool, though. This is a good thing — many people worry that compressed filesystems are slow filesystems, but LZ4 stream compression is so lightweight that it’s effectively “free.” We’ve done extensive testing across years of ZFS experience and have never seen a situation where LZ4 wasn’t a good idea. Even a $ 50 tinkertoy APU from several years ago can compress and decompress LZ4 faster than a pair of fast SSDs can keep up, with no significant CPU utilization.
We did spot a bug pretty quickly while looking over the pools and datasets — both (bpool) ***************************************************** (itself and) bpool / BOOT / ubuntu_oalrlu(we think the string of random-seeming characters is intended to be a unique system identifier) have/ boot
set as their mountpoint. This clearly isn’t causing any significant problems right now, and we’re sure it will get ironed out well before Eoan goes live.
Although Eoan created datasets automatically for both my real user account homedir and root’s, theAdduser
command didn ‘ t create one for a new test user. This is something we also expect to get ironed out reasonably quickly — even ifadduser
itself never takes those steps, the GUI for adding new users likely will, if it doesn’t already. This is also pretty simple to do manually; in the above example, where new user test isnotlogged in, we could upgrade test to a zfs dataset homedir like this:
root @ eoan: ~ $ zfs create rpool / ROOT / ubuntu_oal4lu / test_twm 547 root @ eoan: ~ $ rsync -ha / home / test / / rpool / ROOT / ubuntu_oal4lu / test_twm 547 / root @ eoan: ~ $ rm / home / test root @ eoan: ~ $ chown test.test / rpool / ROOT / ubuntu_oal4lu / test_twm 547 root @ eoan: ~ $ zfs set mountpoint=/ home / test rpool / ROOT / ubuntu_oal4lu / test_twm 547 root @ eoan: ~ $ zfs mount rpool / ROOT / ubuntu_oal4lu / test_twm 547
… and that would be that.
The next big thing we looked for was a mechanism for automatically taking snapshots. You can’t roll back to a snapshot you never took, so a safe ZFS system should automatically take snapshots pretty regularly. There’s nothing in Eoan to take snapshots for you yet — the only cron job is the standard one which scrubs the pool once per month — but there are a few general purpose ZFS snapshot orchestration tools readily available; these includezfs-auto-snapshotand my ownsanoid.
Alpha software is alpha!
In conclusion, we want to remind readers that while ZFS itself is very stable, Ubuntu’s ZFS-enabled installer and use of it as a root filesystem are stillalphaquality. We do not recommend that you attempt to use the new ZFS installer on systems you care deeply about until the installer makes it past alpha, past beta, and all the way to full release quality. This also means you should be kind about any bugs you find playing with it in the meantime — again, this isalphasoftware and bugs are not only possible, they’re to be expected.
With all that said, we’re extremely excited about ZFS on root making visible progress in Ubuntu — and we hope these features and more will make it into Eoan Ermine’s expected end-of-the-month release.
Listing image by Jim Salter
GIPHY App Key not set. Please check settings