QubesOS provides a desktop operating system made up of multiple virtual machines, running under Xen. To protect against buggy network drivers, the physical network hardware is accessed only by a dedicated (and untrusted) “NetVM”, which is connected to the rest of the system via a separate (trusted) “FirewallVM”. This firewall VM runs Linux, processing network traffic with code written in C.
In this blog post, I replace the Linux firewall VM with a MirageOS unikernel . The resulting VM uses safe (bounds-checked, type-checked) OCaml code to process network traffic, Uses less than a tenth of the memory of the default FirewallVM, boots several times faster, and should be much simpler to audit or extend.
is a security-focused desktop operating system that uses virtual machines to isolate applications from each other. The screenshot below shows my current desktop. The windows with green borders are running Fedora in my “comms” VM, which I use for gmail and similar trusted sites (with NoScript). The blue windows are from a Debian VM which I use for software development. The red windows are another Fedora VM, which I use for general browsing (with flash, etc) and running various untrusted applications:
Another Fedora VM (“dom0”) runs the window manager and drives most of the physical hardware (mouse, keyboard, screen, disks, etc).
Networking is a particularly dangerous activity, since attacks can come from anywhere in the world and handling network hardware and traffic is complex. Qubes therefore uses two extra VMs for networking:
NetVM drives the physical network device directly. It runs network-manager and provides the system tray applet for configuring the network.
FirewallVM sits between the application VMs and NetVM. It implements a firewall and router.
The full system looks something like this:
The lines between VMs in the diagram above represent network connections. If NetVM is compromised (e.g. by exploiting a bug in the kernel module driving the wifi card) then the system as a whole can still be considered secure – the attacker is still outside the firewall.
Besides traditional networking, all VMs can communicate with dom0 via some Qubes-specific protocols. These are used to display window contents, tell VMs about their configuration, and provide direct channels between VMs where appropriate.
Qubes networking
There are three IP networks in the default configuration:
NAT , so packets from “comms” appear to NetVM to have been sent by the firewall, and packets from the firewall appear to my house router to have come from NetVM.
, the netfront code that processes responses from netback uses the request ID quoted by netback as an index into an array without even checking if it’s in range ( they have fixed this in their fork).
What can an attacker do once they’ve exploited FirewallVM’s trusting netfront driver? Presumably they now have complete control of FirewallVM. At this point, they can simply reuse the same exploit to take control of the client VMs, which are running the same trusting netfront code!
A Unikernel Firewall
I decided to see whether I could replace the default firewall (“sys-firewall”) with a MirageOS unikernel. A Mirage unikernel is an OCaml program compiled to run as an operating system kernel. It pulls in just the code it needs, as libraries. For example, my firewall doesn’t require or use a hard disk, so it doesn’t contain any code for dealing with block devices.
If you want to follow along, my code is on GitHub in my qubes-mirage-firewall
(this installation mechanism is obviously not ideal; hopefully future versions of Qubes will be more unikernel-friendly)
You can run mirage-firewall alongside your existing sys-firewall and you can choose which AppVMs use which firewall using the GUI. For example, to configure “untrusted” to use mirage-firewall:
You can view the unikernel’s log output from the GUI, or with (sudo xl console mirage-firewall) in dom0 if you want to see live updates.
If you want to explore (the code
) but don’t know OCaml, a good tip is that most modules ( .ml (files) have a corresponding . mli (interface file which describes the module’s public API) is a bit like a . h (file in C). It’s usually worth reading those interface files first.
I initially tested with Qubes 3.0 and have just upgraded to the 3.1 alpha. Both seem to work.
Booting a Unikernel on Qubes
Qubes runs on Xen and a Mirage application can be compiled to a Xen kernel image using mirage configure –xen . However, Qubes expects a VM to provide three Qubes-specific services and doesn’t consider the VM to be running until it has connected to each of them. They are (qrexec) (remote command execution), (gui) (displaying windows on the dom0 desktop) and QubesDB (a key-value store).
I wrote a little library, mirage-qubes
, to implement enough of these three protocols for the firewall (the GUI does nothing except handshake with dom0, since the firewall has no GUI).
Here’s the full boot code in my firewall, showing how to connect the agents:
The first directory is monitored by the firewall (domain 4). When it sees the new entry, it knows it has a new network connection to domain 5, interface 0. It writes to the directory information about what features it supports and sets the state to 2 ( init-wait ).
The second directory will be seen by the new domain 5 when it boots. It tells it that is has a network connection to dom 4. The client looks in the dom 4's backend directory and waits for the state to change to (init-wait) , the checks the supported features. It allocates memory to share with the firewall, tells Xen to grant access to dom 4, and writes the ID for the grant to the XenStore directory. It sets its own state to 4 (
connected .
When the firewall sees the client is connected, it reads the grant refs, tells Xen to map those pages of memory into its own address space, and sets its own state to (connected) too. The two VMs can now use the shared memory to exchange messages (blocks of data up to (KB).
The reason I had to find out about all this is that the
mirage-net-xen library only implemented the netfront side of the protocol. Luckily, Dave Scott had already started adding support for netback and I was able to complete that work.
Getting this working with a Mirage client was fairly easy, but I spent a long time trying to figure out why my code was making Linux VMs kernel panic. It turned out to be an amusing bug
in my netback serialization code, which only worked with Mirage by pure luck.
However, this did alert me to a second bug in the Linux netfront driver: even if the ID netback sends is within the array bounds, that entry isn’t necessarily valid. Sending an unused ID would cause netfront to try to unmap someone else's grant-ref. Not exploitable, perhaps, but another good reason to replace this code!
(The Ethernet layer)
It might seem like we’re nearly done: we want to send IP (Internet Protocol) packets between VMs, and we have a way to send blocks of data. However, we must now take a little detour down Legacy Lane ...
Operating systems don’t expect to send IP packets directly. Instead, they expect to be connected to an Ethernet network, which requires each IP packet to be wrapped in an Ethernet “frame”. Our virtual network needs to emulate an Ethernet network.
When a client VM wants to send an IP packet, it first broadcasts an Ethernet
ARP
request, asking for the MAC address of the target machine. The target machine responds with its MAC address. The client then transmits an Ethernet frame addressed to this MAC address, containing the IP packet inside.
If we were building our system out of physical machines, we’d connect everything via an Ethernet switch, like this:
The (Netback.listen) at the end runs a loop that communicates with the netfront driver in the client. Each time a frame arrives, we check the type and dispatch to either the ARP handler or, for IPv4 packets, the firewall code. We don’t support IPv6, since Qubes doesn’t either.
You can use any template, and make it standalone or not. It doesn’t matter, since we don’t use the hard disk. 
sys-firewall 
). Click (OK) to create the VM.
Booting a Unikernel on Qubes is the empty tuple (called “unit”). It’s used for functions that don’t take arguments, or return nothing useful. ~ foo