kamal-proxy deep dive

I’ve started experimenting with Kamal in order to deploy web applications to my DigitalOcean VPS. It is a pretty opinionated tool that sits on top of Docker and allows you to deploy without downtime. It relies on a proxy named kamal-proxy to interface with the web application, which is what allows the tool to perform gapless deploys and rollbacks. I’ve read the bare minimum amount of documentation in order to deploy my application, but thought it would be an interesting exercise to inspect what kamal-proxy does under the hood. I will be using my currently deployed application as we perform the deep dive.

Exploration

The first thing to notice is that kamal-proxy runs as a container side-by-side to our application:

Networking

Our application exposes port 8080 while kamal-proxy binds to every interface on ports 80 and 443.

We can use docker inspect kamal-proxy to learn more about the container configuration and confirm our thought process. For example, what is the command being executed by the container? Which ports is it binding to?

Notice how the kamal-proxy container is running a binary called kamal-proxy. This will help us in the future. Another interesting thing to confirm is that both our web application and kamal-proxy share the same network ID:

We can inspect this network by doing:

Both containers are on the same private Docker bridge network called kamal. This means that kamal-proxy doesn’t reach our web application through our host’s public IP or localhost, instead it uses the internal network using Docker’s built-in DNS. This is the reason our web application doesn’t need to publish port 8080 on the host.

Through ip link show we can see that the bridge network exists at the kernel level:

Running ip addr show with the bridge’s interface confirms that it acts as the gateway for both containers with the associated IP being 172.18.0.1/16.

Each container should be “plugged” into the bridge. This happens through veth pairs (virtual ethernet cables). Let’s inspect that:

Two things of notice. First, both veth pairs are @if2, which means they are connected to the second interface in the namespace container network stack. Second, see that master is br-ee31959bc247? That’s our bridge! Let’s check the other side of one of our veth pairs inside kamal-proxy container namespace by using nsenter:

See how our second interface has the IP 172.18.0.2? That’s exactly what the output from docker network inspect printed. Let’s also inspect the routes within our kamal-proxy container to see how it is linked to the gateway:

See how the default gateway is 172.18.0.1? That’s our bridge again! How does a packet coming from the internet make it to the bridge though? That’s the job of the routing table!

This can be pretty overwhelming, so let’s break it down:

1. PREROUTING handles packets as they arrive, rule 1 sends any packet destined for a local address into the DOCKER chain for inspection.
2. We make it to DOCKER chain, here we have two rules:
   • Any packets arriving at port 80 gets DNAT’d (Destination NAT) to 172.18.0.2:80, which is kamal-proxy’s IP.
   • The same thing happens, but for port 443

DNAT rewrites the destination IP of the packet before it’s forwarded. So a packet arriving from the internet destined for 67.205.158.151:443 gets its destination rewritten to 172.18.0.2:443 and sent into the bridge network. The original sender never knows this happened.

What about packets leaving the container?

1. POSTROUTING handles packets just before leaving the host:
   • In 1, any packet from 172.18.0.0/16 going anywhere gets MASQUERADE'd, as in, its source IP is rewritten to the host’s public IP. This is how container traffic appears to come from the VPS when it reaches the internet.

New deploys

We know how the network is configured, but how does kamal-proxy handle a new deploy without downtime though? The first part of the magic is that during deploys the container for kamal-proxy remains stable, but how does it know to reach the newly deployed container rather than the old one?

In order to do this I’ve triggered a deploy and dug through logs, because why not? The first thing to notice is the new output for docker ps:

Armed with the container ID, we can investigate our logs:

Notice the "target":"dfdee429978c:8080". That’s not an IP address, it’s a container ID prefix and a port. Docker DNS resolves dfdee429978c to whatever IP that container has on the kamal network. Let’s look inside the kamal-proxy network namespace to see how that DNS interception actually works:

In order to understand these rules it’s important to know that Docker’s embedded DNS resolver runs at 127.0.0.11 inside every container’s network namespace. Let’s see what these rules are doing:

DOCKER_OUTPUT intercepts outgoing DNS queries:

• TCP port 53 → redirected to 127.0.0.11:39291
• UDP port 53 → redirected to 127.0.0.11:57361

Any process inside this namespace that tries to query DNS on port 53 gets silently redirected to dockerd’s actual listening ports. The process never knows.

DOCKER_POSTROUTING is the return path, it rewrites the source port back to 53 on the way out:

• Responses from 127.0.0.11:39291 → source rewritten to port 53
• Responses from 127.0.0.11:57361 → source rewritten to port 53

But this still doesn’t explain how kamal-proxy knows about the new container! Let’s inspect the binary itself:

Turns out kamal-proxy is told about the container instead of figuring it out!

See how our target is the existing container? Pretty cool! So who tells kamal-proxy about the container and how? A common pattern in these systems is to use a unix socket to communicate with a process, do we have one of those?

Bingo! proc/<pid>/net/ shows the network state from inside the process’s network namespace, including sockets. Since we are interested in unix sockets, we ran /proc/<pid>/net/unix to find it. This is the path as seen from inside the container’s filesystem. To access it from the host we need to find it through the overlay filesystem:

There we have it! Can we use something like socat to intercept the traffic going through it? In order to do it we will have to put a proxy in front of Kamal’s proxy and then trigger a re-deploy. How fun is that?!

It seems like kamal-proxy communicates through RPC. More importantly, can you spot the new container ID in there? Here it is! 13e21e3cada6:8080. And if we run docker ps again:

The container IDs match! Very neat.

We are getting to the end here. The image used by Kamal is deterministic since in my case every deploy uses git rev-parse --short HEAD as a tag:

See how it matches kaeros/xps:98043a3? Kamal uses the git SHA as the image tag, then after starting the new container it gets the container ID from Docker’s API and passes that to kamal-proxy. We can apply the same socat interception technique to the Docker socket itself in order to learn which commands Kamal is running since it SSHs into our server:

And this clears any doubt of what is happening:

1. Pull the new image

2. Ensure infrastructure exists

3. Find and rename the old container

4. Create and start the new container

5. Tell kamal-proxy about the new container

6. Stop the old container

7. Cleanup

Before quitting, let’s quickly see how kamal-proxy stores its state.

Mounts

Let’s dig into these:

That kamal-proxy.state looks interesting!

active_targets has our new container ID! Let’s re-deploy and see how this file changes:

So this is how kamal-proxy keeps the current state of the system.

This was an interesting journey with lots of dead ends. As the reader you have the privilege of walking on the paved path instead of blindly stumbling through the infrastructure like I did. If you feel like I missed something or things are unclear, give me a shout! See you next time.