As part of my work at Sojourn Labs (more about this at a later time), I’ve recently been working with creating a secure private cloud based on CoreOS, Docker (basically, a lightweight virtualization system), and some decade-old (or older) hardware. Different applications within the cloud need to communicate with each other securely. For example, our wiki needs to communicate with our database server. To link two Docker containers on the same machine together, one need only instruct Docker to create a link between the two. Things get more interesting when the communicating containers reside on different machines.
The first interesting bit of business is how to get packets from one container to another over the network since Docker’s links don’t handle this. The accepted solution is the ambassador pattern: two extra containers are run. One is bound to the server and the other to the client, both via Docker links. These two containers then connect to each other via a network. A popular implementation of the ambassador pattern is Sven Dowideit’s ambassador. This implementation is easy to grok, simple, and lightweight.
Things get a bit hairy when a computer running a service goes down and a new one takes its place. A new IP means the ambassadors need to be reconfigured etcd cluster to store necessary information for establishing links. Wow! This is all pretty amazing.
But what if you don’t trust the network your infrastructure is running on? For example, Sojourn Labs is running on a shoestring budget and we are pretty liberal with our network access which also happens to run our servers (shhh). Obviously, not only is encryption required, but peer authentication. For the rest of this post, I’m going to assume you actually want to set up secure links between Docker containers and you are familiar with OpenSSL. If the latter isn’t true, vapr is gradually being built which will aid in setting up your own, secure private cloud, right down to securely bootstrapping a computer over the public Internet. More on this, too, at a later time.
The way I’ve implemented this ambassador is to use a pair of busybox Docker image to implement the ambassador pattern. The server’s ambassador uses the Docker remote API via a UNIX socket to determine which ports the server has exposed and the mapping of its private ports to public ports (thus allowing an arbitrary port mapping of exposed ports on the ambassador to the exposed ports on the server). It then posts the mapping to etcd. Next, the ambassador server does some secure port forwarding. I chose socat with OpenSSL rather than SSH since certificate generation/distribution using a private (or public) certificate authority is much easier than trying to securely distribute SSH public keys to each client ambassador. etcd could be used, but a rogue etcd client (due to a compromised container, for example) then becomes much more powerful.
The client ambassador uses etcd to fetch the host ports and IP address of the ambassador serving the the desired container. Using socat and its own certificates, it sets up a tunnel to intended container. Lastly, the client container is linked to the client ambassador.
All in all, the Docker image is just over 7MiB. You can audit/download the source from https://github.com/SojournLabs/ambassador or use the Docker image at https://registry.hub.docker.com/u/sojournlabs/ambassador. It requires a minimal amount of configuration and should be fleet and systemd friendly.