What a Linux container actually is

Since dotCloud open-sourced Docker at PyCon in March, I’ve had three different people ask me whether containers are “the new VMs”. The question makes sense. The demos look like magic: one command and you have an isolated Ubuntu environment that boots in under a second.

I’ve spent a good chunk of my career babysitting VMware hosts, so I went and read up on what’s actually going on underneath. They’re not VMs, and the difference matters for what you can trust them with.

The picture

A virtual machine fakes a whole computer. The hypervisor presents virtual hardware, and each guest boots its own kernel on it. That’s strong isolation: a guest can crash its own kernel without touching its neighbors. It’s also heavy, because you’re running a complete OS for every app, with its own memory overhead and boot time.

A container on the other hand is just ordinary Linux processes running on the host’s kernel. What makes them a “container” is that the kernel has been told to show them a restricted view of the world. There’s no guest kernel. Starting a container is about as expensive as starting a process, because that’s what it is.

The three ingredients

1. Namespaces: what a process can see. A namespace wraps a global resource so that processes inside it get their own private copy. Linux has six of them now:

• PID: the container has its own process tree, and its first process is PID 1. It can’t see or signal processes outside.
• Mount: its own view of the filesystem, so / can be a different directory tree.
• Network: its own interfaces, routing table and ports. Two containers can both listen on port 80.
• UTS: its own hostname.
• IPC: its own shared memory and semaphores.
• User: its own user IDs, so root inside can map to an unprivileged user outside. This one only became complete with Linux 3.8 in February, so it’s very new.

You can play with these directly, unshare and ip netns are on most recent distros.

2. Control groups: what a process can use. Cgroups put limits on, and keep accounts of, resources for a group of processes: this group gets at most 512 MB of RAM, this share of CPU, and this much disk I/O. Without cgroups, one runaway container could starve all the others. Google contributed a lot of the cgroups work to the kernel, and by all accounts they’ve been running their own workloads this way internally for years.

3. A layered filesystem: what a process starts with. This is the part Docker adds that makes it pleasant. Images are built from read-only layers stacked with a union filesystem (AUFS, today). A base Ubuntu layer, your packages on top, your app on top of that, and a thin writable layer for the running container. Ten containers from the same image share the bottom layers on disk, which is why they start instantly and use so little space.

Namespaces and cgroups aren’t new. LXC has packaged them up for years, and Docker currently uses LXC under the hood. FreeBSD jails and Solaris zones did similar things long before that. Docker’s real contribution is the workflow: a simple file that describes how to build an image, and a standard way to ship that image anywhere.

So should you use it?

Density and speed are the obvious wins. You can run dozens of containers on a box that would hold a handful of VMs, they start in milliseconds, and “works on my machine” gets a lot less painful when the machine ships with the app.

The thing I’d be careful about is isolation. Every container shares one kernel, and the kernel has a huge attack surface, so a kernel exploit inside a container is a kernel exploit on the host. Root inside a container (without user namespaces, which basically nobody has yet) is uncomfortably close to root outside. I’d happily use containers to package and deploy my own apps. I would not, today, make them the only wall between my app and code from a stranger. For that I still want a VM boundary in there somewhere.

Also, Docker is at version 0.3 and the README says in so many words not to use it in production yet. Believe them. But do try it. It could be how a lot of us ship software in a few years, or it could be another thing I install, play with for an afternoon and forget. We’ll see.