The Case Against Kubernetes at the Edge

The Case Against Kubernetes at the Edge

Kubernetes is, by a wide margin, the most successful piece of infrastructure software of the last decade. It is also, by a similarly wide margin, the wrong tool for the edge.

This is not a popular position. There is an entire sub-industry of “lightweight Kubernetes for the edge” distributions; K3s, MicroK8s, KubeEdge, OpenYurt, all of which are serious engineering efforts by serious engineers. We’ve tried most of them. We respect the work.

We still think the answer is no.

Here’s why.

What Kubernetes is actually good at

Kubernetes solves a specific problem exceptionally well: dynamically scheduling fungible workloads across a pool of fungible machines in a reliable datacentre network. Give it a fleet of identical servers with fast interconnects and a constant power supply, tell it to run 100 copies of a stateless web service, and it will do so beautifully. When a machine dies, it’ll reschedule. When a workload needs more capacity, it’ll scale. When you push a new version, it’ll roll it out with careful health checks.

Every single one of those capabilities assumes things that are not true at the edge.

The edge breaks every assumption

Devices aren’t fungible. The mini PC behind the digital signage screen in the Melbourne store is not interchangeable with the mini PC behind the screen in the Perth store. They’re physically in different places. They display different content. If one fails, you can’t “reschedule” its workload onto the other. The “workload” is “render pixels on the HDMI port attached to this specific device.”

Networks aren’t reliable. A Kubernetes control plane expects its nodes to maintain continuous, low-latency contact. At the edge, your devices are on 4G, on a flaky shop Wi-Fi, behind a captive portal, or on a diesel generator that restarts nightly. Kubelet’s response to losing contact with the API server for five minutes is “the node is NotReady, evict the pods.” At the edge, losing contact for five minutes is a Tuesday.

Scale is inverted. Datacentre Kubernetes runs thousands of pods on dozens of nodes. Edge fleets run one or two pods on tens of thousands of nodes. Every part of Kubernetes; etcd, the scheduler, the controller-manager; was designed around the first shape. “Lightweight” distributions shrink the control plane, but they don’t change the fundamental cardinality mismatch.

The failure modes are wrong. When a datacentre node goes bad, you cordon it, drain it, and replace it. When an edge device goes bad, someone has to physically drive to a warehouse in another city. “Self-healing” means something completely different when the healing involves a Toyota Hilux.

Security posture is inside-out. Kubernetes assumes a trusted network between nodes and control plane. The edge has no trusted network. Every connection must be authenticated, encrypted, and initiated outbound from the device, because there’s no inbound path. Retrofitting this onto Kubernetes is possible, but you spend all your time fighting the framework.

The “lightweight” distributions don’t fix this

K3s is a remarkable piece of engineering; it crams a working Kubernetes cluster into a single ~50MB binary. But shrinking the binary doesn’t change the architecture. You still have:

• An API server that devices need to reach
• An etcd (or SQLite) store that has to be reconciled
• A scheduler that assumes fungibility
• A kubelet that assumes connectivity
• Container images pulled from a registry with no delta compression, no layer deduplication across reboots, no A/B partitioning of the node OS itself

What you get is a system where the workload is managed through a modern orchestrator, but the device. The kernel, the bootloader, the root filesystem, the OS updates; is managed by whatever you stitched together around it. Yocto, Ansible, a custom update server, a cron job, hopes, prayers.

That seam is where the pain lives. It’s where drift happens. It’s where you spend your Friday afternoons.

What the edge actually needs

Pick up any real edge deployment; industrial IoT, digital signage, agricultural telemetry, retail, mining; and the list of actual requirements looks nothing like what Kubernetes offers:

1. One artefact that defines the entire device state, kernel to userland to workload
2. Atomic updates with automatic rollback, because you can’t drive to the site to recover a brick
3. Outbound-only connectivity because inbound is not available
4. Offline tolerance, because the link will drop and the device still has to do its job
5. Hardware identity a cryptographic bond between a specific device and its credentials
6. Fleet-wide observability with minutes-to-debug-a-random-device SLAs
7. A per-device dimension the signage player in Melbourne must show Melbourne content, and “just reschedule the pod” is not a response

None of these are things Kubernetes is bad at by accident. They are things it is not trying to do. Asking Kubernetes to do them is like asking a Formula 1 car to tow a caravan. The caravan will get there; everyone involved will hate their life.

The honest counter-arguments

There are two arguments for edge Kubernetes that we take seriously.

“My team already knows Kubernetes.” This is real. Cognitive load is a real cost. If you have a platform team deeply fluent in Kubernetes and you’re deploying ten devices in a controlled environment, the familiarity may outweigh the architectural mismatch. For a while.

“I want to run the same workload at the edge and in the cloud.” Also real. The dream of “one deployment manifest, two targets” is appealing. In practice, we find edge workloads diverge from cloud workloads quickly; they need hardware access, different networking, different lifecycle; and the unified deployment manifest becomes mostly an illusion maintained through willpower.

Both arguments weaken as the fleet grows and the environment gets harsher. They are strongest when you’re running a handful of devices in a single building. They are weakest when you have 10,000 devices across 40 countries and a 1% failure rate that would mean 100 trucks need rolling.

What we did instead

We built Admiral with a different starting premise: the device is the primitive, not the pod.

• Every device runs an immutable OS we ship as an OCI image (see Why We Ship Our OS as an OCI Container)
• Updates are atomic, signed, and rollback automatically
• The control plane is multi-region and devices connect out
• Workloads are deployed as part of the OS image, not scheduled onto it
• Identity is hardware-rooted and credentials are short-lived

This is a smaller, more constrained model than Kubernetes. It cannot dynamically reschedule workloads. It cannot auto-scale pods across nodes. It does not try to.

What it can do is take a single signed artefact and reliably, observably, verifiably put it on 50,000 devices in outbound-only networks around the world, and tell you with confidence that every one of them is running exactly that artefact right now.

That is the edge problem. That is what we built for.

When Kubernetes is still right

For completeness: if you are running a regional edge; say, a handful of beefy servers in each of a dozen metro areas, with reliable networking, in climate-controlled facilities, running fungible stateless workloads; Kubernetes is excellent. That shape of deployment is really “small datacentre” more than “edge” in the sense we mean, but the terminology is contested.

If you are running actual devices; constrained, remote, numerous, hostile-networked, single-purpose; we’d encourage you to at least try a model built for that reality.

We’ll give you five of them to start with, for free, forever.