우병수

Posted on May 11 • Originally published at techdigestor.com

Building a Docker-like Container From Scratch: What Actually Happens When You Run `docker run`

#docker #ai #devops #machinelearning

TL;DR: I was three hours deep into a Docker networking debug session — containers couldn't reach each other, docker network inspect was giving me nothing useful — and I had this uncomfortable realization: I was treating Docker like magic. I knew the commands.

📖 Reading time: ~41 min

What's in this article

Why I Built This (And Why You Should Too)
The Four Linux Primitives Docker is Built On
Step 1 — Isolating a Process With Namespaces
Step 2 — Building a Minimal Root Filesystem With debootstrap\
Step 3 — Pivoting the Root With chroot\ (and Why pivot\_root\ Is Better)
Step 4 — Limiting Resources With cgroups v2
Step 5 — Network Isolation With a veth Pair
Putting It All Together — A ~80 Line Shell Script That Actually Works

Why I Built This (And Why You Should Too)

I was three hours deep into a Docker networking debug session — containers couldn't reach each other, docker network inspect was giving me nothing useful — and I had this uncomfortable realization: I was treating Docker like magic. I knew the commands. I had no idea what was actually running beneath them. That frustration is what pushed me to build a minimal container from scratch, and honestly, it's one of the better decisions I've made as a systems engineer.

Here's what surprised me: there's no secret sauce. Docker, containerd, Podman — they all sit on top of the same Linux kernel primitives that have been there since kernel 3.8. Namespaces, cgroups, pivot_root. Once you've wired those together yourself in maybe 80 lines of Go or C, the next time a container networking issue bites you, you'll actually know what layer to look at. That alone makes this exercise worth a Saturday afternoon.

By the time you're done with this walkthrough, you'll have a working mini-container that does three real things:

Process isolation — your containerized process has its own PID namespace, so ps aux inside shows only what you put there
Filesystem isolation — a separate root filesystem via chroot or pivot_root, so the process can't see your host's /etc/passwd
Network isolation — its own network namespace, optionally wired up with a veth pair so it can actually talk to the outside world

Prerequisites are minimal and I mean that literally. You need a Linux machine — I tested everything here on Ubuntu 22.04 with kernel 5.15, though anything from 5.4 onwards behaves the same for our purposes. You need root access because namespace operations require it. And you need to be comfortable enough in a terminal that running unshare --pid --fork --mount-proc /bin/bash doesn't make you flinch. That's the bar. No prior kernel knowledge required.

One thing I want to be blunt about: this is not a production runtime. We're not implementing seccomp filters, we're not handling user namespace mapping properly for rootless operation, and we're definitely not building an OCI-compliant image puller. If you want that, Podman and containerd already exist and they're excellent. This is purely a learning exercise — the equivalent of building a toy compiler to understand how GCC works. The goal is demystification, not deployment. For a broader look at developer productivity tools and workflow automation, check out our guide on Productivity Workflows.

The Four Linux Primitives Docker is Built On

The thing that surprised me most when I first looked under Docker's hood: there's no special "container runtime" magic happening. A container is just a Linux process — what makes it a container is a handful of kernel flags you set before exec(). Docker, containerd, Podman — they're all just orchestrating these same four kernel features. If you understand these, you understand containers.

Namespaces: The Kernel's Blinders

A namespace is just a flag you pass to clone() or unshare() that tells the kernel: "this process should see its own version of X." There are six you care about:

PID — the process gets its own PID 1. From inside the container, it can't see host processes. From outside, you can still see the container process with ps aux.
Network — private network stack: own loopback, own IP, own routing table. This is why you have to explicitly port-forward with -p 8080:80.
Mount — own filesystem view. Mounts inside don't leak to the host, and vice versa.
UTS — own hostname and domain name. This is why your container can have hostname webapp-prod while the host is node-42.
IPC — isolates System V IPC and POSIX message queues. Mostly matters if you're running apps that use shared memory between processes.
User — maps container UIDs to host UIDs. UID 0 inside the container can map to an unprivileged UID on the host. Rootless containers depend entirely on this one.

You can prove any of this yourself without writing a single line of Go. Run unshare --pid --fork --mount-proc bash and you get a shell where ps aux shows only two processes. That's a container, basically — minus the filesystem isolation and resource limits. The --mount-proc flag remounts /proc inside the new PID namespace so tools like ps don't read the host's process list.

# This gives you a shell with its own PID namespace
# Your shell becomes PID 1 inside it
unshare --pid --fork --mount-proc /bin/bash

# Now run this inside — you'll only see 2 processes
ps aux

cgroups: Where Resource Limits Actually Get Enforced

Namespaces give a process restricted vision — cgroups give it restricted access. These are two different things and it's easy to mix them up. A process in a PID namespace still competes for real CPU cycles until you put it in a cgroup. The kernel exposes cgroups through a pseudo-filesystem, currently at /sys/fs/cgroup if you're on a system running cgroups v2 (which is pretty much everything post-kernel 5.10).

Docker does this automatically when you pass --memory or --cpus. But you can do it manually to see exactly what's happening:

# Create a cgroup for memory limiting (cgroups v2)
mkdir /sys/fs/cgroup/mytest

# Limit to 50MB RAM
echo 52428800 > /sys/fs/cgroup/mytest/memory.max

# Put the current shell's PID into this cgroup
echo $$ > /sys/fs/cgroup/mytest/cgroup.procs

# Now anything this shell spawns is also memory-limited
# Try running something memory-hungry and watch it get OOM-killed

CPU limits work differently than most people expect. --cpus=0.5 in Docker doesn't pin your container to half a core — it sets a CPU quota using cpu.max in the cgroup. The default period is 100ms, so 0.5 CPUs means the container gets 50ms of CPU time per 100ms window. It can burst during the window then get throttled. I/O limits work similarly through io.max. These aren't soft suggestions — the kernel enforcer is real and will OOM-kill your process if you hit the memory limit without a swap allowance.

OverlayFS: Why Layers Are Genius

Every Docker image is a stack of read-only layers. When you run a container, the kernel mounts them together using OverlayFS and adds one writable layer on top. The lower layers are shared between every container using that image — they're not copied. This is why docker pull ubuntu:22.04 doesn't re-download the base if another image already pulled it: the layers are content-addressed by SHA256 and shared on disk.

# OverlayFS mount syntax — this is what Docker does under the hood
mount -t overlay overlay \
  -o lowerdir=/layer2:/layer1:/layer0,\
     upperdir=/container-writes,\
     workdir=/overlay-work \
  /merged

# lowerdir: read-only image layers (colon-separated, top to bottom)
# upperdir: where container writes land — this is what gets committed if you docker commit
# workdir: internal OverlayFS scratch space, must be on same filesystem as upperdir
# /merged: the unified view the container process sees

The trade-off worth knowing: OverlayFS has real performance costs on write-heavy workloads. If your container is doing thousands of small file writes — like a database — you absolutely want to use a bind mount or a Docker volume instead of writing to the container layer. The copy-on-write overhead adds up fast. Check /proc/mounts inside a running container and you'll see the actual overlay mount listed.

Capabilities and Seccomp: The Security Layer Most Tutorials Skip

By default, Docker doesn't run containers as fully privileged root even if the user inside is UID 0. It drops a specific set of Linux capabilities. Capabilities break the all-or-nothing root vs. non-root model — instead of needing full root to bind port 80, you just need CAP_NET_BIND_SERVICE. Docker drops around 14 capabilities by default, keeping only what most apps need. The dangerous ones it drops include CAP_SYS_ADMIN (basically root in disguise), CAP_NET_ADMIN, and CAP_SYS_PTRACE.

Seccomp (secure computing mode) is a layer on top of that. It's a BPF filter that runs on every syscall and either allows it or kills the process. Docker ships a default seccomp profile that blocks around 44 syscalls — things like keyctl, ptrace, kexec_load. You can inspect Docker's default profile at /usr/share/docker/seccomp.json on most systems, or pull it from the Moby repo. When people run --privileged, they're disabling both the capability drops and the seccomp filter — which is why that flag is a pretty serious security hole you shouldn't use in production unless you have a specific reason.

# See what capabilities a running container has
docker run --rm ubuntu:22.04 cat /proc/self/status | grep Cap

# Decode the hex capability bitmask on the host
capsh --decode=00000000a80425fb

# Add a capability back (e.g., if your app needs net_admin)
docker run --cap-add NET_ADMIN myimage

# Check which syscalls are blocked by inspecting seccomp on a process
# (requires kernel 5.8+ for this specific interface)
cat /proc/$(pgrep containerd)/status | grep Seccomp

The Mental Model That Makes Everything Click

A container is a process (or a process tree) that has been given its own namespace context, placed into a cgroup, shown a merged filesystem view via OverlayFS, and had its syscall surface trimmed by seccomp + capability drops. That's the complete picture. Nothing runs inside a hypervisor. There's no kernel boundary between the container and the host — which is why containers boot in milliseconds and why a container escape vulnerability is significantly more serious than a VM escape. The process is genuinely on your host kernel, just wearing blinders. That distinction matters when you're making decisions about multi-tenant security, because two containers on the same host share one kernel — and a kernel CVE affects all of them simultaneously.

Step 1 — Isolating a Process With Namespaces

The first time I ran ps aux inside an isolated namespace and saw only two processes staring back at me, I genuinely had to double-check I hadn't accidentally SSH'd into a different machine. That's the moment namespaces click — not from reading about them, but from seeing your terminal lie to a process in real time.

The command that produces that moment is this:

# --fork: spawn a child process before entering the namespace (critical — more on this below)
# --pid: create a new PID namespace so processes see a fresh PID table
# --mount-proc: remount /proc so tools like ps read from the new namespace, not the host
sudo unshare --fork --pid --mount-proc /bin/bash

Once you're inside that shell, run ps aux. You'll see exactly two entries: bash at PID 1 and ps at PID 2. On your host in another terminal, run the same command and you'll see the full process tree — hundreds of entries, the unshare process itself, everything. Same kernel. Same hardware. Two completely different realities. That gap is the entire point of container isolation.

The --fork flag is where people get burned. Skip it and run sudo unshare --pid --mount-proc /bin/bash instead — your shell will open but ps aux still shows host processes, or you'll get weird errors about /proc not mounting cleanly. The reason is subtle: without --fork, the unshare process itself becomes PID 1 in the new namespace. But unshare isn't designed to be an init process, so signal handling breaks and /proc remounting gets confused. The man page mentions this in passing but doesn't spell out the symptom — you just get a namespace that half-works and spend 20 minutes blaming your kernel version.

UTS namespaces are a cleaner intro for understanding namespace isolation without the /proc complexity. Run this:

# UTS = Unix Timesharing System — controls hostname and NIS domain name
sudo unshare --uts /bin/bash
hostname mycontainer   # set it inside the namespace
hostname               # returns: mycontainer

Then, without closing that shell, open a second terminal on the host and run hostname. It still shows your original hostname. The change is fully contained. This is exactly how Docker sets the per-container hostname you define in docker run --hostname — it's not a config file swap, it's a UTS namespace. Knowing this also tells you why hostname-based service discovery inside containers works without touching the host's /etc/hostname.

One thing worth testing early: namespace isolation is not security isolation by itself. If your isolated bash shell runs as root (which it does under sudo unshare), it still has broad capabilities on the host filesystem unless you layer in mount namespaces and drop capabilities explicitly. PID isolation hides the process table from the process — it does not prevent that process from affecting shared kernel resources. That distinction matters a lot when you move from "cool demo" to "I want to run untrusted code."

Step 2 — Building a Minimal Root Filesystem With `debootstrap\`

The namespace setup from Step 1 is deceptively incomplete. Your process is isolated in terms of PID, UTS, and mount namespaces — but ls / inside that namespace still shows your host's entire filesystem. Every binary, every config file, every secret your host has. That's not a container; that's just a process with identity confusion. The rootfs is what makes it a real container.

Installing debootstrap

On Ubuntu or Debian, this is one line:

sudo apt install debootstrap

If you're on Arch or Fedora, the package exists in AUR and dnf respectively, but honestly the experience is smoother on Debian-based hosts. debootstrap is essentially a shell script that fetches a minimal Debian/Ubuntu system from an archive mirror and installs it into a directory. No virtualization, no special kernel support needed.

Creating the rootfs

sudo debootstrap --arch=amd64 jammy /tmp/mycontainer-root http://archive.ubuntu.com/ubuntu

That command bootstraps Ubuntu 22.04 (jammy) into /tmp/mycontainer-root. The thing that catches people off guard: there is no progress bar during the package download phase. You'll see a line like Retrieving packages... and then nothing for potentially 3–5 minutes on a slow or throttled connection. It's not hung. The tool is silently fetching and unpacking around 100+ packages. On a fast connection it takes under 2 minutes; on a capped VPS or hotel WiFi I've watched it sit for 12 minutes. Don't Ctrl+C it.

What actually lands in /tmp/mycontainer-root

ls /tmp/mycontainer-root
# bin  boot  dev  etc  home  lib  lib64  media  mnt  opt
# proc  root  run  sbin  srv  sys  tmp  usr  var

It looks like a real Linux system root because it is one — just stripped down. A few specific things worth knowing:

/bin and /sbin are symlinks to /usr/bin and /usr/sbin on modern Ubuntu — same as your host, no surprise there.
/etc/resolv.conf will exist but might be empty or point at nothing useful. You'll need to handle DNS separately when you actually pivot into this root.
/proc and /sys are empty directories. They only populate when you bind-mount or remount them inside the namespace — which is exactly what you'll do in Step 3.
/dev has a few static device nodes but none of the dynamic ones. No /dev/null populated by udev here.

The total size comes out to roughly 300–350MB. That's the "minimal" Ubuntu experience — still heavy compared to a Alpine-based container image, but it gives you a full apt ecosystem to work with, which matters for learning this stuff without fighting missing libraries.

The faster alternative: docker export

If you already have Docker installed and just want a rootfs without waiting on debootstrap, this trick is worth knowing:

# Create a container from any image (no need to run it)
docker create --name temp-export ubuntu:22.04

# Export the entire filesystem as a tarball
docker export temp-export -o /tmp/ubuntu-rootfs.tar

# Unpack into your target directory
mkdir -p /tmp/mycontainer-root
tar -xf /tmp/ubuntu-rootfs.tar -C /tmp/mycontainer-root

# Clean up
docker rm temp-export

This is significantly faster because Docker pulls a pre-built layer cache rather than bootstrapping from package archives. The trade-off: the rootfs you get is whatever the Docker image maintainer decided to include, not a raw debootstrap base. For this exercise that doesn't matter — the directory structure is identical and your namespace + chroot code won't know the difference. I actually use this method most of the time when prototyping container tooling because the iteration loop is faster.

One gotcha with the docker export path: it flattens all layers into a single tarball. That's actually what you want here, but if you're building something that needs to understand image layers (like a container registry or a build cache), you'd use docker save instead, which gives you the OCI layer format. For our purposes, the flat tarball from export is perfect.

Step 3 — Pivoting the Root With `chroot\` (and Why `pivot\_root\` Is Better)

The thing that surprised me most when I first ran chroot was how fast it works — and how little it actually protects you. One command and you're "inside" a different root filesystem. Feels like Docker. It's not.

# First, pull a minimal rootfs to play with
mkdir -p /tmp/mycontainer-root
# I use Alpine's minirootfs — it's ~3MB and has a real /bin/sh
curl -o /tmp/alpine.tar.gz \
  https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/x86_64/alpine-minirootfs-3.19.1-x86_64.tar.gz
tar -xzf /tmp/alpine.tar.gz -C /tmp/mycontainer-root

# Drop into it
sudo chroot /tmp/mycontainer-root /bin/sh

You're now in a shell where / points to /tmp/mycontainer-root. Running ls / shows the Alpine tree, not your host. Satisfying. But here's the problem: if you're root inside this chroot (and you are, because sudo), you can escape it. The classic trick is chdir("../../..") in C, or just calling chroot(".") twice with the right directory manipulation. Security researchers documented this decades ago. chroot was never designed as a security boundary — it's a filesystem view change, full stop. Docker does not use it alone, and neither should you.

The /proc problem hits you immediately

Run ps aux inside your chroot and you'll get nothing, or an error. That's because /proc is a virtual filesystem the kernel populates dynamically — it doesn't exist as real files on disk, so it didn't get included in your Alpine tarball extraction. You have to mount it explicitly before entering the chroot, or from inside after mounting:

# From outside, before entering chroot
sudo mount -t proc proc /tmp/mycontainer-root/proc

# Also /dev, otherwise tools like ls will throw fits about missing devices
sudo mount --bind /dev /tmp/mycontainer-root/dev
sudo mount --bind /dev/pts /tmp/mycontainer-root/dev/pts

# /sys is needed for some tools too
sudo mount -t sysfs sysfs /tmp/mycontainer-root/sys

Skip the /dev bind-mount and you'll see errors like ls: cannot access '/dev/null': No such file or directory immediately. Some programs check for /dev/urandom or /dev/zero at startup. Binding the host /dev is fine for experimentation, but in production runtimes they use devtmpfs and populate only the specific device nodes the container actually needs — that's a deliberate security decision.

Why pivot_root exists and what it requires

pivot_root swaps the root mount of the current mount namespace — it makes your new rootfs the actual mount namespace root, and stashes the old one somewhere you can unmount it afterward. This means the host filesystem isn't even visible as a mount point from inside the container, which chroot never guarantees. The catch: pivot_root requires you to be inside a mount namespace. You can't call it on your host's namespace. This is why every real container runtime — runc, crun, containerd — always creates a new mount namespace first, then calls pivot_root. The two are inseparable.

#!/bin/bash
# container.sh — combines unshare + pivot_root for a real-ish container
# Requires: util-linux >= 2.36, run as root

ROOTFS=/tmp/mycontainer-root
OLD_ROOT=$ROOTFS/old_root

# Mount the rootfs as a bind mount on itself — pivot_root needs the
# new root to be a mount point, not just a directory
mount --bind "$ROOTFS" "$ROOTFS"

mkdir -p "$OLD_ROOT"

# pivot_root: new_root old_root
# After this, / is $ROOTFS and the old / is at /old_root
pivot_root "$ROOTFS" "$OLD_ROOT"

# Fix PATH to find Alpine's binaries now that we're in the new root
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# Mount proc in the new root — /old_root still points to the host here
mount -t proc proc /proc

# Unmount the old root so the host filesystem is gone
umount -l /old_root
rmdir /old_root

exec /bin/sh

# The outer invocation — this is what you actually run
sudo unshare --mount --uts --ipc --pid --fork bash container.sh

The --fork flag on unshare is something I missed the first time. Without it, PID namespace isolation doesn't work correctly because the unshare process itself becomes PID 1, which causes fork() to behave unexpectedly with signal handling. With --fork, unshare forks a child that becomes PID 1 inside the namespace, which is how real init processes work. Also notice the mount --bind "$ROOTFS" "$ROOTFS" line — pivot_root will flat-out refuse to run if the new root isn't already a mount point. That bind-mount-to-self trick is the standard workaround and it's not obvious from the man page.

When chroot is actually fine

I still use plain chroot for cross-compilation environments and build toolchains — situations where I own the host, I'm the one entering the chroot, and isolation isn't the goal. If you're setting up an ARM cross-compile environment with QEMU binfmt and a Debian rootfs, chroot is exactly the right tool. The mistake is thinking it equals container security. For anything where untrusted code runs, or where you need the process to genuinely believe it's in its own system, you need the namespace + pivot_root combination above.

Step 4 — Limiting Resources With cgroups v2

The thing that tripped me up the hardest here wasn't the concept — it was that every tutorial I found was written for cgroups v1, and I'm running Ubuntu 22.04 which uses v2 by default. The syntax is completely different. If you're following an old guide and nothing is working, that's almost certainly why. Before you touch anything, confirm which version your system actually uses:

stat -fc %T /sys/fs/cgroup/
# cgroup2fs  ← you want this on Ubuntu 22.04+
# tmpfs      ← this means you're on v1, stop and find a v1 guide

If you got cgroup2fs, you're good to follow along. Now create a cgroup for your container process. On v2 this is just a directory under /sys/fs/cgroup/ — the kernel populates it with control files automatically the moment you create it:

mkdir /sys/fs/cgroup/mycontainer
ls /sys/fs/cgroup/mycontainer
# cgroup.controllers  cgroup.max.depth  cgroup.procs  cgroup.subtree_control
# cgroup.threads      cpu.stat          memory.current  memory.max  ...

Setting a memory limit is a single write to memory.max. The value is in bytes, so 64MB looks like this:

# 64 * 1024 * 1024 = 67108864
echo '67108864' > /sys/fs/cgroup/mycontainer/memory.max

# confirm it stuck
cat /sys/fs/cgroup/mycontainer/memory.max
# 67108864

Now assign your container process (or any process, really) to this cgroup. Once you write a PID to cgroup.procs, that process and everything it forks is subject to your limits:

# Replace $PID with the actual PID of your unshare'd process
echo $PID > /sys/fs/cgroup/mycontainer/cgroup.procs

# Verify the process is in the cgroup
cat /sys/fs/cgroup/mycontainer/cgroup.procs
# 94312

To actually verify the limit fires, run a memory hog inside your container and watch the OOM killer do its job. A quick Python one-liner works fine for this:

# Inside your namespaced process:
python3 -c "x = ' ' * 200_000_000"
# Killed

# On the host, check dmesg to confirm the OOM kill happened:
dmesg | grep -i oom | tail -5
# [12043.882] oom-kill:constraint=CONSTRAINT_MEMCG,task=python3,pid=94312
# [12043.882] Memory cgroup out of memory: Killed process 94312 (python3)

A few gotchas worth calling out explicitly: First, on v2 you can only set controllers on a cgroup if the parent cgroup has that controller listed in cgroup.subtree_control. If writing to memory.max gives you a Permission denied error even as root, check that the root cgroup has memory enabled:

cat /sys/fs/cgroup/cgroup.subtree_control
# cpuset cpu io memory hugetlb pids rdma misc  ← memory needs to be here

# If memory is missing, add it:
echo '+memory' > /sys/fs/cgroup/cgroup.subtree_control

Second, you also get CPU throttling almost for free — just write to cpu.max using the format quota period. Something like 50000 100000 limits the process to 50% of one CPU core. No extra setup needed once the cgroup exists. That's one of the genuinely nice things about v2 — the unified hierarchy is cleaner once you understand it, even if the migration from v1 docs is painful.

Step 5 — Network Isolation With a veth Pair

The thing that trips most people up here isn't the veth pair itself — it's that you can do everything right and still have no connectivity because of a single missing kernel switch. IP forwarding is disabled by default on most Linux installs. Your packets just vanish silently. I'll get to that, but keep it in mind as you follow along.

A veth pair is exactly what it sounds like: two virtual ethernet interfaces that are wired directly to each other. Whatever you send into one end comes out the other. You're going to put one end on your host and shove the other end into the network namespace your container is running in. At that point the container has its own interface, its own IP, and no idea it's living inside a namespace on your machine.

Create the pair first:

# veth0 stays on the host, veth1 goes into the container
sudo ip link add veth0 type veth peer name veth1

# Confirm both exist on the host right now
ip link show veth0
ip link show veth1

Now move veth1 into your container's network namespace. You need the PID of the process running inside the namespace — whatever you stored as $CONTAINER_PID when you called clone() or unshare:

sudo ip link set veth1 netns $CONTAINER_PID

After this command, veth1 disappears from ip link on the host. That's correct — it now only exists inside the container's namespace. To configure it, you need to run commands inside that namespace:

# On the HOST — configure the host-side interface
sudo ip addr add 172.20.0.1/24 dev veth0
sudo ip link set veth0 up

# Inside the container namespace — use nsenter to get in there
sudo nsenter --net=/proc/$CONTAINER_PID/ns/net -- ip addr add 172.20.0.2/24 dev veth1
sudo nsenter --net=/proc/$CONTAINER_PID/ns/net -- ip link set veth1 up
sudo nsenter --net=/proc/$CONTAINER_PID/ns/net -- ip link set lo up

# Set the default route inside the container so traffic knows where to go
sudo nsenter --net=/proc/$CONTAINER_PID/ns/net -- ip route add default via 172.20.0.1

At this point the container can ping 172.20.0.1 (the host) and vice versa. But it can't reach the internet yet. For that you need two things: IP forwarding enabled on the host kernel, and a NAT masquerade rule so outbound packets get the host's real IP slapped on them before they leave.

# Without this, packets routed through veth0 just get dropped — no error, nothing
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward

# The NAT rule — any packet from our container subnet gets masqueraded
sudo iptables -t nat -A POSTROUTING -s 172.20.0.0/24 -j MASQUERADE

# Verify the rule landed
sudo iptables -t nat -L POSTROUTING -n -v

The /proc/sys/net/ipv4/ip_forward write is ephemeral — it resets on reboot. If you want it permanent, add net.ipv4.ip_forward = 1 to /etc/sysctl.conf and run sudo sysctl -p. The other gotcha worth knowing: if you have a restrictive default iptables FORWARD policy (check with sudo iptables -L FORWARD), your packets will still get dropped even with masquerade in place. Add sudo iptables -A FORWARD -i veth0 -j ACCEPT if you see this. Docker sets this up automatically which is why most people never encounter it — building this yourself strips away all those defaults.

Putting It All Together — A ~80 Line Shell Script That Actually Works

The Full Script — All Five Steps in One Place

Everything we've covered — namespaces, pivot_root, cgroups, network setup — fits into about 80 lines of bash. I was surprised how readable the final result is. No magic, no abstraction layers hiding what's happening. Here it is:

#!/usr/bin/env bash
# container.sh — a minimal container runtime for learning purposes
# Usage: sudo bash container.sh /path/to/rootfs /bin/bash
# Requires: util-linux (unshare, nsenter), iproute2, coreutils
# Tested on: Ubuntu 22.04 / 24.04, kernel 5.15+

set -euo pipefail

ROOTFS="${1:?Usage: $0  }"
CMD="${2:?Usage: $0  }"
CONTAINER_ID="ctr-$$"           # unique per invocation using PID
VETH_HOST="veth-host-$$"
VETH_CONT="veth-cont-$$"
BRIDGE="br-containers"
CONTAINER_IP="10.88.0.$((RANDOM % 200 + 10))/24"
CGROUP_PATH="/sys/fs/cgroup/${CONTAINER_ID}"

# ── STEP 1: Cgroup setup (do this before unshare) ──────────────────────────
# We write limits from host-side; the container process inherits them.
setup_cgroups() {
  mkdir -p "${CGROUP_PATH}"
  # 256MB memory limit — tweak this for your needs
  echo $((256 * 1024 * 1024)) > "${CGROUP_PATH}/memory.max"
  # 50% of one CPU core across any scheduling period
  echo "50000 100000"          > "${CGROUP_PATH}/cpu.max"
  # pids.max stops fork bombs dead
  echo "64"                    > "${CGROUP_PATH}/pids.max"
  echo $$ > "${CGROUP_PATH}/cgroup.procs"
}

# ── STEP 2: Network setup — bridge + veth pair ─────────────────────────────
setup_network() {
  # Create bridge if it doesn't exist already
  if ! ip link show "${BRIDGE}" &>/dev/null; then
    ip link add "${BRIDGE}" type bridge
    ip addr add 10.88.0.1/24 dev "${BRIDGE}"
    ip link set "${BRIDGE}" up
    # NAT so the container can reach the internet
    iptables -t nat -A POSTROUTING -s 10.88.0.0/24 -j MASQUERADE
    echo 1 > /proc/sys/net/ipv4/ip_forward
  fi

  ip link add "${VETH_HOST}" type veth peer name "${VETH_CONT}"
  ip link set "${VETH_HOST}" master "${BRIDGE}"
  ip link set "${VETH_HOST}" up
  # The container-side veth gets moved into the new netns inside pivot_root
}

# ── STEP 3: Pivot into the rootfs ─────────────────────────────────────────
pivot_into_rootfs() {
  local rootfs="$1"
  local old_root="${rootfs}/.old_root"

  mount --bind "${rootfs}" "${rootfs}"   # bind-mount so pivot_root is happy
  mkdir -p "${old_root}"
  pivot_root "${rootfs}" "${old_root}"

  # Remount proc fresh — host's /proc leaks into the new root otherwise
  mount -t proc proc /proc
  mount -t sysfs sysfs /sys
  mount -t tmpfs tmpfs /tmp

  # Now drop the old root — we don't need it anymore
  umount -l /.old_root
  rmdir /.old_root
}

# ── STEP 4: Network config inside the container namespace ──────────────────
configure_container_network() {
  ip link set lo up
  # VETH_CONT was passed in via env since we're in a new netns
  ip link set "${VETH_CONT}" up
  ip addr add "${CONTAINER_IP}" dev "${VETH_CONT}"
  ip route add default via 10.88.0.1
}

# ── CLEANUP on exit ────────────────────────────────────────────────────────
cleanup() {
  ip link del "${VETH_HOST}" 2>/dev/null || true
  rmdir "${CGROUP_PATH}" 2>/dev/null || true
}
trap cleanup EXIT

# ── ENTRYPOINT ────────────────────────────────────────────────────────────
setup_cgroups
setup_network

# Move the container-side veth into the network namespace we're about to create.
# unshare --net creates a new netns; we grab its fd via /proc after the fact.
unshare \
  --mount \
  --uts \
  --ipc \
  --pid \
  --net \
  --fork \
  --mount-proc \
  bash -c "
    export VETH_CONT=${VETH_CONT}
    export CONTAINER_IP=${CONTAINER_IP}
    # Move our veth into this netns (host side knows the new netns PID)
    ip link set ${VETH_CONT} netns \$\$
    $(declare -f pivot_into_rootfs)
    $(declare -f configure_container_network)
    pivot_into_rootfs '${ROOTFS}'
    configure_container_network
    hostname '${CONTAINER_ID}'
    exec ${CMD}
  "

Walking Through the Key Sections

The ordering matters more than the code itself. Cgroups come first, before unshare, because we write limits into the host cgroup hierarchy and the child process inherits them. If you do it the other way around — try to assign cgroups from inside the new namespace — you'll hit permission errors in cgroupv2 unless you've done the delegation dance with cgroup.subtree_control. Skip that complexity for now and just do it host-side.

pivot_root is the part that trips people up. It's not chroot — it actually changes the root mount for the entire mount namespace, not just the process. The trick is that pivot_root requires the new root to be a mount point, which is why we do the mount --bind rootfs rootfs step first. Without that bind mount, you get EINVAL and no useful error message. The old root goes into .old_root temporarily, then we lazily unmount it with umount -l. After that, the container process has zero visibility into the host filesystem.

The veth pair handoff to the new network namespace is the trickiest coordination point. We create the pair on the host, set one end on the bridge, then move the other end into the container's netns using its PID. The container then configures its own IP from inside. The ip_forward + iptables MASQUERADE combo is the minimum viable setup for outbound internet access — same thing Docker does under the hood, just with more error handling and rule deduplication.

Running It

First, get a rootfs. The fastest way is to export one from Docker if you have it around:

# Pull a minimal alpine rootfs — ~3MB decompressed
docker export $(docker create alpine) | tar -C /tmp/mycontainer-root -xf -

# Or with skopeo + umoci if you're going Docker-free:
skopeo copy docker://alpine:3.19 oci:/tmp/alpine-oci:latest
umoci unpack --image /tmp/alpine-oci:latest /tmp/mycontainer-root

Then run the script:

sudo bash container.sh /tmp/mycontainer-root /bin/sh

# You should see something like:
/ # hostname
ctr-94821
/ # cat /proc/self/cgroup
0::/
/ # ip addr
1: lo:  ...
2: veth-cont-94821:  ... 10.88.0.47/24
/ # cat /proc/meminfo | grep MemTotal
# Will reflect host total — but writes beyond 256MB will get OOM-killed

The /proc/self/cgroup output showing 0::/ is normal — it means the container thinks it's at the root of its own cgroup hierarchy, which is exactly what you want. Same behavior you see with real Docker containers. To verify the memory limit is actually enforced, run cat /sys/fs/cgroup/ctr-${PID}/memory.max from the host while the container is alive.

The Parallels to Docker Become Obvious

Once you run this and poke around inside, the Docker mental model snaps into place. The docker run --memory 256m flag? That's our memory.max write. The bridge network Docker creates (docker0 by default)? Same veth + bridge architecture we built — Docker just names it differently and manages veth lifetimes automatically. The thing that surprised me most: docker inspect on a running container shows a SandboxKey which is literally a path to a network namespace file in /var/run/docker/netns/. You can nsenter into it directly and it behaves exactly like our container's netns.

Where to Go Next

The logical next stop is the runc source code on GitHub. runc is the reference OCI runtime — every major container tool (Docker, containerd, Podman) shells out to it or embeds it. The libcontainer package inside runc does exactly what our script does, just in Go with proper error recovery, seccomp filter setup, capability dropping, and user namespace support. Start with libcontainer/container_linux.go — the newInitProcess function is where namespace creation happens and it maps almost 1:1 to our unshare call. Reading production code after building the toy version is one of the more effective ways I've found to stop feeling lost in a large codebase.

Two concrete extensions worth trying before you move on: add --user namespace support with --map-root-user (rootless containers), and replace the iptables MASQUERADE rule with nftables — that's the direction the Linux networking stack is heading and Podman already defaults to nftables on Fedora 38+. Neither is hard once you've internalized the five-step flow this script implements.

What Docker Adds On Top (That We Skipped)

The thing that surprised me most when I first dug into this: Docker's actual container runtime is maybe 20% of what Docker does. The other 80% is image management, networking plumbing, and a daemon that coordinates all of it. What you just built is that 20% — and understanding it makes the rest of Docker's architecture obvious rather than mysterious.

Image Layers and OverlayFS

Our rootfs is a flat directory we unpacked from a tarball. Docker's approach is fundamentally different — every RUN, COPY, and ADD instruction in a Dockerfile creates a separate read-only layer. At runtime, those layers are stacked using OverlayFS, which is a union filesystem built into the Linux kernel since 3.18. The container gets a writable layer on top, but the base layers are shared across every container running from the same image. This is why pulling a second container from the same base image is nearly instant — you already have the layers.

# What OverlayFS actually looks like under the hood
# Docker sets this up for you, but you can do it manually:

mkdir upper lower work merged

# lower = read-only base (your image layers, merged)
# upper = writable layer (container's changes go here)
# work  = required scratch dir for overlayfs internals

mount -t overlay overlay \
  -o lowerdir=lower,upperdir=upper,workdir=work \
  merged

# Now 'merged' shows both, writes go to 'upper' only
# After container exits, 'upper' is the diff you committed

Our scratch container used a plain bind mount for the rootfs — writes go straight to disk, nothing is isolated, and you can't snapshot it. The overlay approach is why docker commit works at all and why you can spin up 50 containers from the same image without 50x the disk usage.

containerd, runc, and the OCI Spec

Docker doesn't call clone() and unshare() directly anymore. That code was extracted into runc, which implements the OCI Runtime Spec. containerd sits above that — it manages image pulls, snapshot storage, and lifecycle (start/stop/kill). Docker Engine sits above containerd. So the actual call chain for docker run is: Docker CLI → Docker daemon → containerd → runc → your process.

The OCI Runtime Spec is just a JSON file called config.json that describes namespaces, cgroups, the root filesystem path, environment variables, and capability sets. runc reads it and does exactly what our shell script did, except with 500 lines of Go, proper error handling, and support for the full spec. You can generate and inspect this yourself:

# Generate a spec skeleton — this is what runc actually reads
runc spec

# You'll get a config.json with sections like:
# "namespaces": [{"type": "pid"}, {"type": "network"}, ...]
# "cgroupsPath": "/sys/fs/cgroup/runc/mycontainer"
# "process": {"args": ["/bin/sh"], "env": [...]}

# Run it directly without Docker:
runc run mycontainer

The reason this API layer exists is operational, not technical. Multiple container runtimes (containerd, CRI-O, kata-containers) need to interoperate with Kubernetes and each other. Without a spec, every runtime would have its own calling convention and you couldn't swap them. The spec turns "how to start a container" into a boring JSON config problem.

seccomp Profiles and Capability Dropping

Our container runs with whatever capabilities the calling process has, and every syscall is available. Docker's default seccomp profile blocks 44 syscalls — things like keyctl, add_key, request_key, mbind, mount, reboot, kexec_load. The full list is in Docker's source at profiles/seccomp/default.json and it's worth reading once — you can see exactly what attack surface they're cutting off.

# Docker also drops these capabilities by default (--cap-drop=ALL is common):
# CAP_NET_ADMIN, CAP_SYS_ADMIN, CAP_SYS_PTRACE, CAP_SYS_MODULE
# This means: can't modify routing tables, can't load kernel modules,
# can't ptrace arbitrary processes, can't mount filesystems

# Check what caps your container actually has:
docker run --rm alpine cat /proc/1/status | grep Cap
# CapPrm: 00000000a80425fb
# Decode it:
capsh --decode=00000000a80425fb

Our scratch container runs as root with full capabilities because we never dropped them. In practice this means a process that escapes our container's PID/mount namespace isolation could do real damage. Docker's hardening defaults aren't optional niceties — they're the actual security boundary. If you're running a scratch container in production for learning purposes, at minimum add a seccomp profile via the --security-opt seccomp=profile.json flag on unshare's equivalent.

Networking: Bridge, Host, Overlay

We left our container in a network namespace but didn't wire it up to anything. Docker's bridge networking does the heavy lifting: it creates a virtual Ethernet pair (veth), puts one end in the container's namespace and one end on the docker0 bridge interface, assigns IPs from a private subnet (default 172.17.0.0/16), and sets up iptables NAT rules so outbound traffic looks like it's coming from the host. Port mapping is just a DNAT rule: traffic hitting host port 8080 gets rewritten to the container IP on port 80.

# What Docker actually creates for a bridged container — you can see it live:
ip link show type veth
# veth3a91b2c@if8:  ...

# The iptables rule that makes -p 8080:80 work:
iptables -t nat -L DOCKER -n --line-numbers
# DNAT  tcp  --  !docker0  *  0.0.0.0/0  0.0.0.0/0
#       tcp dpt:8080 to:172.17.0.2:80

# Overlay networking (Swarm/multi-host) adds VXLAN tunneling on top —
# traffic is encapsulated in UDP packets between hosts on port 4789.

Host networking (--network=host) skips all of this — the container just uses the host's network namespace directly, which is exactly what happens in our scratch build. It's faster and simpler but means port conflicts are your problem and you lose isolation. The bridge model is where most production single-host containers run.

The Real Takeaway

Docker is a UX layer. A very good, very well-engineered one — the image format, the layer caching, the networking model, the security defaults — all of it is real engineering work that took years to get right. But the core primitive you built (namespaces + cgroups + a rootfs) is identical to what runc executes. When Docker does something surprising — slow image builds, unexpected network behavior, a capability error you can't explain — you now have the mental model to go one layer deeper and read the actual system calls, mount points, and iptables rules rather than cargo-culting flags until something works.

Gotchas I Hit That The Tutorials Don't Mention

The thing that cost me the most time when first building container primitives wasn't the namespace setup or the cgroup math — it was a cascade of silent failures that left me staring at "operation not permitted" with zero useful context. Here's what actually bit me, in roughly the order it'll bite you.

User namespaces might just be off

On Ubuntu (and anything based on older Debian defaults), unprivileged user namespaces are disabled at the kernel level. Your rootless container code will fail with a cryptic permission error, and nothing in the error message will point you at the actual fix.

# Check if it's disabled — 0 means off
cat /proc/sys/kernel/unprivileged_userns_clone

# Enable it for the current session
echo 1 | sudo tee /proc/sys/kernel/unprivileged_userns_clone

# Make it survive reboots
echo 'kernel.unprivileged_userns_clone=1' | sudo tee /etc/sysctl.d/99-userns.conf
sudo sysctl --system

This is specific to kernels where Ubuntu (or the distro) has applied the Debian hardening patch. Vanilla upstream kernels on Arch or Fedora usually have this on by default. If you're on Ubuntu 22.04 or earlier and wondering why unshare --user works as root but fails for your normal user, this is it.

Unmount /proc or you'll haunt yourself

Every container tutorial tells you to bind-mount /proc into the new rootfs. Almost none of them tell you what happens when you forget to unmount it before tearing things down. The mount sticks. It survives your script exit. On some setups, it survives a reboot because systemd lazily re-reads mount state from /proc/mounts.

# What you probably wrote:
mount -t proc proc "$ROOTFS/proc"
# ... do container stuff ...
rm -rf "$ROOTFS"  # ← disaster waiting to happen

# What you should write:
mount -t proc proc "$ROOTFS/proc"
# ... do container stuff ...
umount "$ROOTFS/proc"   # explicit unmount first
rm -rf "$ROOTFS"

If you already have phantom mounts, findmnt --list | grep deleted will show them. You can clean them with umount -l (lazy unmount) if the path is already gone. Add this cleanup to your trap handler — more on that next.

Always add a trap handler for cgroup cleanup

cgroups are kernel objects. If your script crashes or you Ctrl-C mid-run, the cgroup directory you created doesn't disappear. The next run tries to create the same cgroup, finds it already exists, and either fails silently or inherits stale resource limits. I've seen containers get OOM-killed at 128MB because a previous failed run left a cgroup with a memory limit still attached.

#!/bin/bash
CGROUP_PATH="/sys/fs/cgroup/my-container-$$"

cleanup() {
  # Kill any processes still in the cgroup before removing it
  if [ -f "$CGROUP_PATH/cgroup.procs" ]; then
    cat "$CGROUP_PATH/cgroup.procs" | xargs -r kill -9 2>/dev/null
  fi
  umount "$ROOTFS/proc" 2>/dev/null
  rmdir "$CGROUP_PATH" 2>/dev/null
  echo "Cleaned up cgroup and mounts"
}

# This fires on exit, Ctrl-C (SIGINT), and unhandled errors
trap cleanup EXIT INT TERM ERR

mkdir -p "$CGROUP_PATH"
echo "50000 100000" > "$CGROUP_PATH/cpu.max"   # 50% CPU limit (cgroups v2)
echo "134217728" > "$CGROUP_PATH/memory.max"   # 128MB

Using $$ in the cgroup path is a quick way to make each run's cgroup unique to the process ID, so parallel runs don't stomp on each other. Clean up with rmdir not rm -rf — the kernel doesn't let you forcibly delete a cgroup with active PIDs, and that's actually useful behavior you want to respect rather than work around.

clone() vs unshare() — the ergonomic difference matters

clone() is the raw syscall that creates a new process with new namespaces in one shot. unshare() is a syscall that detaches the calling process from its current namespaces. From a namespace-isolation standpoint, both get you the same end state. The practical difference is in how you use them.

# Shell scripts use the unshare(1) utility, which wraps the unshare() syscall:
unshare --pid --mount --net --uts --ipc --fork bash

# Go/Rust container runtimes use clone() via syscall directly:
# In Go (what runc does under the hood):
cmd := exec.Command("/proc/self/exe", "child")
cmd.SysProcAttr = &syscall.SysProcAttr{
    Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID |
                syscall.CLONE_NEWNS  | syscall.CLONE_NEWNET,
}

The shell unshare command is great for quick experiments. The issue is PID namespace isolation — when you unshare a PID namespace in a shell script, your shell becomes PID 1 in the new namespace, but signals work differently than you expect and zombie reaping becomes your problem. With clone(), runc spawns a dedicated init process from the start. For a learning project, unshare is fine. For anything that runs real workloads, understand you're eventually going to want clone() semantics.

AppArmor and SELinux will block things without telling you why

This one is particularly maddening because the operations look like they should work — the namespace is set up, the cgroup exists, the binary is present in the rootfs — but you get EPERM or the process just dies. The strace output looks fine. The error is above the syscall layer: the LSM (Linux Security Module) rejected it after the kernel already said yes.

# First place to check — AppArmor denials:
sudo dmesg | grep -i apparmor | tail -20

# SELinux denials (on Fedora/RHEL):
sudo ausearch -m avc -ts recent
# or
sudo journalctl -t setroubleshoot --since "5 minutes ago"

# Quick test: temporarily put AppArmor in complain mode for your process
sudo aa-complain /path/to/your/binary

# For SELinux — check if this is the issue by putting it in permissive temporarily:
sudo setenforce 0
# Run your code — if it works now, SELinux is your problem
sudo setenforce 1

Don't leave SELinux in permissive or AppArmor in complain mode permanently. Use it to diagnose, then write the actual policy. For AppArmor, aa-genprof will watch your program run and suggest a policy. For SELinux, audit2allow converts the AVC denials into a policy module. The real mistake is assuming the absence of a useful error message means the code is wrong — sometimes the kernel said yes and the LSM said no, and you'll only find out via dmesg.