Firecracker is the easy part

Every coding agent I run wants to execute code it just wrote. Containers are the default answer and the wrong one: a shared kernel is a shared attack surface, and the code running inside was written by a model thirty seconds ago. MicroVMs fix that. Firecracker boots a real Linux kernel in a hardware-isolated VM in well under a second, and every sandbox vendor is built on it or on something like it.

There are a dozen tutorials on booting a Firecracker VM. There is almost nothing on what it takes to run a fleet of them behind an API. So I built one. This is what I learned, with numbers and the actual terminal output.

The code is at github.com/avirajkhare00/sandboxd. One Go binary, two dependencies, about 650 lines, Apache 2.0.

What it does

The service exposes four endpoints an agent harness calls: create a sandbox, run a command in it, upload a file, destroy it. Here is a full session against a running instance:

$ curl -s -XPOST localhost:8080/sandboxes
{"id":"674e8d71"}

$ curl -s -XPOST localhost:8080/sandboxes/674e8d71/exec \
    -d '{"cmd":["python3","-c","print(1+1)"]}'
{"stdout":"2\n","stderr":"","exit_code":0}

$ curl -s -XPOST localhost:8080/sandboxes/674e8d71/exec \
    -d '{"cmd":["sh","-c","node -v; git --version; curl -sI -m 3 http://8.8.8.8 || echo egress-blocked"]}'
{"stdout":"v18.20.4\ngit version 2.39.5\negress-blocked\n","stderr":"","exit_code":0}

$ curl -s -XPUT localhost:8080/sandboxes/674e8d71/files/a.py \
    --data-binary 'import sys; print("hello from file", sys.version_info[:2])'
$ curl -s -XPOST localhost:8080/sandboxes/674e8d71/exec -d '{"cmd":["python3","a.py"]}'
{"stdout":"hello from file (3, 11)\n","stderr":"","exit_code":0}

$ curl -s -XDELETE -w '%{http_code}\n' localhost:8080/sandboxes/674e8d71
204

Under the hood each sandbox is:

• a Firecracker microVM restored from a golden snapshot, so it comes up with Python, Node, and git already loaded
• running inside its own network namespace with a tap device, so every VM sees the same interface name and IP
• rooted on a copy-on-write clone of a base ext4 image
• reachable over vsock, where a 100-line agent inside the guest accepts newline-delimited JSON and runs commands
• egress-filtered by nftables to an IP allowlist, deny by default
• killed after a maximum lifetime, with its namespace, chroot, and overlay removed

A warm pool keeps N restored VMs idle so create returns in about a millisecond. The pool is a buffered Go channel. When you take one out, a goroutine restores another.

type Pool struct {
	ready chan *VM // buffered, size N
	make  func() (*VM, error)
}

func (p *Pool) Get(ctx context.Context) (*VM, error) {
	select {
	case vm := <-p.ready:
		go p.fill()
		return vm, nil
	case <-ctx.Done():
		return nil, ctx.Err()
	}
}

That is the whole warm pool.

The host

I did this on a Google Cloud n2-standard-4 with nested virtualization enabled: 4 vCPUs, 16 GB, Debian 13. Nested KVM adds overhead, so treat every number here as a ceiling. Bare metal would be faster. I chose it anyway because it costs twenty cents an hour and no provider I tried had bare metal in a region near me.

The numbers

Guest: 1 vCPU, 512 MB, Debian bookworm rootfs, kernel 6.1, Firecracker 1.13.

The daemon logs the time from Firecracker start to the first successful command over vsock. Cold boot on the left, snapshot restore on the right, same host, minutes apart:

2026/09/18 18:53:43 vm 674e8d71 ready in 1.432s      2026/09/18 18:58:26 vm 939217ad ready in 76ms
2026/09/18 18:53:43 vm 072c2c2d ready in 1.433s      2026/09/18 18:58:26 vm 184c9132 ready in 94ms
2026/09/18 18:53:48 vm 4a64b1ed ready in 1.308s      2026/09/18 18:58:31 vm 2eb50a61 ready in 88ms

The restore number is the one vendors quote and it is real. A 512 MB guest comes back with its userspace intact in under a hundred milliseconds. The memory file is mmapped, so a restored VM that touches little memory costs the host 28 MB, not 512.

The 4 ms exec round trip surprised me. That is an HTTP request to the daemon, a Unix socket connect to Firecracker's vsock proxy, a handshake, a JSON message, fork and exec inside the guest, and the response back. Nested virtualization included.

$ ./loadtest -n 1 -m 50 -cmd true
n=1 m=50 wall=382ms errs=0
create: n=1 p50=1ms p90=1ms p99=1ms max=1ms
exec:   n=50 p50=4ms p90=6ms p99=9ms max=13ms

The slowest thing in the system was cp

The first time I started the pool with 16 slots and threw 20 concurrent creates at it, this happened:

$ ./loadtest -n 20 -m 5
n=20 m=5 wall=1m5.599s errs=0
create: n=20 p50=33.818s p90=1m3.547s p99=1m3.776s max=1m5.239s
exec:   n=100 p50=48ms p90=233ms p99=431ms max=531ms

Thirty-three seconds to create a sandbox. The restore log said every VM was ready in 200 to 500 ms. Something else was eating the time, and my log line was not measuring it.

It was the rootfs overlay. Each VM needs a writable copy of the 2 GB base image. On ext4, cp is a full copy, about 680 MB of real data per VM. Sixteen of those at once on a throughput-limited cloud disk queue behind each other. Once I added the overlay step to the log line:

ready in 334ms (overlay 24.399s)
ready in 320ms (overlay 35.673s)
ready in 312ms (overlay 37.08s)
ready in 272ms (overlay 37.499s)

I formatted a loop image as XFS with reflink=1, moved the state directory onto it, and the same cp --reflink=auto became a metadata operation:

ready in 177ms (overlay 112ms)
ready in 231ms (overlay 127ms)
ready in 248ms (overlay 120ms)
ready in 266ms (overlay 109ms)

The restore path never changed. The vendor benchmark was accurate and irrelevant.

The lesson generalizes: measure from the API call to the first useful response, not from the point where the exciting technology starts. My "ready in" log line originally started its clock after the overlay was created. It was lying by omission.

What oversubscription does to exec latency

With reflink in place I ran a sweep: N sandboxes created at once, five commands each.

Two things fall out of this.

Create latency is bimodal. It is a pool hit at a millisecond or a pool miss at the refill rate, and refills serialize on CPU at 300 to 500 ms each under load. Pool size is your create SLO, and it costs 28 MB per slot. Size it for your burst, not your average.

Exec latency is about host cores. Twenty single-vCPU guests on four host cores is five-to-one oversubscription, and exec p99 went from 250 ms to 1.4 s. The rough rule from this box: one host core for every four or five mostly-idle sandboxes if you want exec p99 under half a second. Nested virt inflates this; I would expect metal to tolerate more.

Zero errors across every run. The thing is stable. It is just slower than you want if you starve it.

Fourteen things that broke

Firecracker booted on the first try. These are the things that did not, in the order they happened. Every one of them cost more time than the Firecracker part.

Building the rootfs. My first attempt ran cp -a / /out inside a container to dump its filesystem. /out is under /. It copies itself forever. Use docker export.

The kernel URL. Firecracker's CI bucket has vmlinux-6.1.141, not vmlinux-6.1.bin. My 314-byte kernel was an S3 error page.

nftables keywords. fwd, nat, and snat are reserved words. Naming a chain after one produces a syntax error that points at the wrong token.

Error: syntax error, unexpected fwd, expecting string or last
add chain inet sandboxd fwd { type filter hook forward priority 0; policy drop; }
                        ^^^

Interface name length. Linux caps interface names at 15 characters. My sb-<12 hex>h was 16.

ip link add sb-72994e2c7186h type veth peer name veth0 netns sb-72994e2c7186
Error: Attribute failed policy validation.

That message tells you nothing. Shorter IDs.

cgroup v2. Debian 13 is v2 only. The jailer defaults to v1 and dies with CgroupHierarchyMissing. One config field.

The SDK links drives for you. The Go SDK's jailer support hardlinks every drive into the chroot. I had already placed the overlay there, so it failed with "file exists." Put the overlay one directory up and let the SDK move it.

The jailer drops privileges. Firecracker runs as an unprivileged user inside the jail. A root-owned overlay gave "Permission denied" when attaching the block device. chown it.

Go does not know vsock. The guest agent is PID 1. It opens an AF_VSOCK socket and hands it to net.FileListener, which fails, so PID 1 exits, so the kernel panics. The whole story is in the console:

[    1.136813] Run /sbin/init as init process
2026/09/18 18:27:50 file file+net vsock: getsockname: address family not supported by protocol
[    1.171027] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[    1.173016] CPU: 0 PID: 252 Comm: agent Not tainted 6.1.141 #1

The agent has to accept connections with raw unix.Accept and wrap the descriptor itself.

init has no PATH. It starts with an empty environment. Every exec came back with exec: "python3": executable file not found in $PATH. Set it yourself.

My firewall broke Docker. I gave my forward chain a default drop policy. That is the same hook Docker's containers go through. apt-get inside every container on the host hung for the rest of the evening until I noticed. Filter by ingress interface only.

Docker's firewall broke me. Docker sets its own FORWARD chain to drop. In nftables, a drop in any table wins. My allowlisted egress was silently blackholed. Two rules fix it:

iptables -I FORWARD -i sb0 -j ACCEPT
iptables -I FORWARD -o sb0 -j ACCEPT

Snapshots and the jailer do not compose in the SDK. Snapshot mode stats the memory file on the host, but the path is relative to the chroot. It re-adds the vsock device that the snapshot already restores. And it drops the handler that links the rootfs into the chroot. Three edits to the handler list after machine creation fix it. None of this is documented.

m.Handlers.Validation = m.Handlers.Validation.Remove(firecracker.ValidateLoadSnapshotCfgHandlerName)
m.Handlers.FcInit = m.Handlers.FcInit.Remove(firecracker.AddVsocksHandlerName)
if !m.Handlers.FcInit.Has(firecracker.LinkFilesToRootFSHandlerName) {
	m.Handlers.FcInit = m.Handlers.FcInit.AppendAfter(firecracker.CreateLogFilesHandlerName,
		firecracker.LinkFilesHandler(filepath.Base(cfg.Kernel)))
}

pkill matches its own sudo wrapper. sudo pkill sandboxd kills the sudo process whose command line contains "sandboxd" and exits before reaching the daemon. Use pkill -x or kill by PID.

Restored guests have a frozen clock. Every VM restored from my snapshot thought it was still the moment I took it:

# snapshot taken at 18:55. daemon log, a few minutes later:
2026/09/18 18:58:26 vm 939217ad ready in 76ms
# inside that guest:
$ date -u
Fri Sep 18 18:55:56 UTC 2026

TLS certificate validation eventually notices. The fix was to stamp the host's wall clock onto every request and have the agent call clock_settime when they disagree by more than a second. Restored guests now match the host to the second.

Use it from Claude Code

The repo ships an MCP server, sandboxd-mcp, that wraps the four endpoints as tools. Point Claude Code at it and every shell command the model wants to run happens inside a microVM instead of on your machine:

{
  "mcpServers": {
    "sandbox": {
      "command": "sandboxd-mcp",
      "env": { "SANDBOXD_ADDR": "http://127.0.0.1:8080" }
    }
  }
}

The model sees sandbox_run, sandbox_put_file, sandbox_create, and sandbox_destroy. The first call lazily creates a session sandbox so the model never has to manage IDs.

What I would tell someone starting this

Pick a reflink filesystem for the state directory before you measure anything. XFS or btrfs. It changes the shape of the whole system.

Instrument the entire create path from the first line. My worst hour came from a log line that started its timer too late.

Run it on a host with nothing else on it, or learn nftables well enough to coexist with Docker. Two of my fourteen failures were the two firewalls fighting.

Budget for the SDK. The Firecracker Go SDK is good for boot-from-scratch and thin for snapshots under the jailer. You will read its source.

And the actual Firecracker part really is the easy five percent. It booted a kernel in 1.4 seconds the first time I asked. Everything else is Linux.

What is next

A rerun on bare metal for un-nested numbers. A real agent workload through the MCP server instead of print(1+1). And the comparison against just paying E2B or Modal, which is the question this whole exercise exists to answer.