How Containers Work: Overlayfs, Hacker News

I wrote a comic about overlay filesystems for a potential future containerzinethis morning, and then I got excited about the topic and wanted to write a blog post with more details. Here’s the comic, to start out:

container images are big

Container images can be pretty big (though some are really small, likealpine linux is 2.5MB). Ubuntu 16. 04 is about 27 MB, andthe Anaconda Python distribution is 800 MB to 1.5GB.

Every container you start with an image starts out with the same blank slate, as if it made a copy of the image just for that container to use. But for big container images, like that 800 MB Anaconda image, making a copy would be both a waste of disk space and pretty slow. So Docker doesn’t make copies – instead it uses anoverlay.

how overlays work

Overlay filesystems, also known as “union filesystems” or “union mounts” let you mount a filesystem using 2 directories: a “lower” directory, and an “upper” directory.

Basically:

thelowerdirectory of the filesystem is read-only
theupperdirectory of the filesystem can be both read to and written from

When a processreadsa file, the overlayfs filesystem driver looks in the upper directory and reads the file from there if it’s present. Otherwise, it looks in the lower directory.

When a processwritesa file, overlayfs will just write it to the upper directory.

let’s make an overlay with`mount`!

That was all a little abstract, so let’s make an overlay filesystem and try it out! This is just going to have a few files in it: I’ll make upper and lower directories, and amergeddirectory to mount the combined filesystem into:

$ mkdir upper lower merged work $ echo "I'm from lower!">lower / in_lower.txt $ echo "I'm from upper!">upper / in_upper.txt $ # `in_both` is in both directories $ echo "I'm from lower!">lower / in_both.txt $ echo "I'm from upper!">upper / in_both.txt

Combining the upper and lower directories is pretty easy: we can just do it withmount!

$ sudo mount -t overlay overlay     -o lowerdir=/ home / bork / test / lower, upperdir=/ home / bork / test / upper, workdir=/ home / bork / test / work     / home / bork / test / merged

There’s was an extremely annoying error message I kept getting while doing this, that saidmount: / home / bork / test / merged: special device overlay does not. This message is a lie, and actually just means that one of the directories I specified was missing (I’d written~ / test / mergedbut it wasn’t being expanded).

Okay, let’s try to read one of the files from the overlay filesystem! The filein_both.txtexists in bothlower /andupper /, so it should read the file from theupper / directory.


$ cat merged / in_both.txt "I'm from upper!
It worked!
And the contents of our directories are what we’d expect:
find lower / upper / merged / lower / lower / in_lower.txt lower / in_both.txt upper / upper / in_upper.txt upper / in_both.txt merged / merged / in_lower.txt merged / in_both.txt merged / in_upper.txt
what happens when you create a new file?
$ echo 'new file'>merged / new_file $ ls -l * / new_file -rw-r - r-- 1 bork bork 9 Nov 18 14: 24 merged / new_file -rw-r - r-- 1 bork bork 9 Nov 18 14: 24 upper / new_file
That makes sense, the new file gets created in theupper directory.









what happens when you delete a file?
Reads and writes seem pretty straightforward. But what happens with deletes? Let’s do it!
$ rm merged / in_both.txt
What happened? Let's look withls:
ls - l upper / in_both.txt lower / lower1.txt merged / lower1.txt ls: cannot access 'merged / in_both.txt': No such file or directory -rw-r - r-- 1 bork bork 6 Nov 18 14: 09 lower / in_both.txt c --------- 1 root root 0, 0 Nov 18 14: 19 upper / in_both.txt
So:

in_both.txtis still in thelowerdirectory, and it's unchanged
it's not in themergeddirectory. So far this is all what we expected.
But what happened inupperis a little strange: there's a file calledupper / in_both.txt, but it's a… character device? I guess this is how the overlayfs driver represents a file being deleted.

What happens if we try to copy this weird character device file?
$ sudo cp upper / in_both.txt upper / in_lower.txt cp: cannot open 'upper / in_both.txt' for reading: No such device or address
Okay, that seems reasonable, being able to copy this weird deletion signal file doesn’t really make sense.
you can mount multiple “lower” directories
Docker images are often composed of like 25 “layers”. Overlayfs supports having multiple lower directories, so you can run
mount - t overlay overlay       -o lowerdir: / dir1: / dir2: / dir3: ...: / dir 25, upperdir=...
So I assume that’s how containers with many Docker layers work, it just unpacks each layer into a separate directory and then asks overlayfs to combine them all together together with an empty upper directory that the container will write its changes to it.
docker can also use btrfs snapshots
Right now I’m using ext4, and Docker uses overlayfs snapshots to run containers. But I used to use btrfs, and then Docker would use btrfs copy-on-write snapshots instead. (Here's a list of when Docker uses whichstorage drivers)
Using btrfs snapshots this way had some interesting consequences - at some point last year I was running hundreds of short-lived Docker containers on my laptop, and this resulted in me running out of btrfs metadata space (likethis person). This was really confusing because I’d never heard of btrfs metadata before and it was tricky to figure out how to clean up my filesystem so I could run Docker containers again. (this docker github issuedescribes a similar problem with Docker and btrfs)
It's fun to try out container features in a simple way!
I think containers often seem like they’re doing “complicated” things and I think it's fun to break them down like this - you can just run onemountincantation without actually doing anything else related to containers at all and see how overlays work!