When namespaces are not enough

I have written before about how to run several tenants on one Kubernetes cluster with a small team: namespaces for scope, RBAC so tenants cannot reach each other’s API objects, quotas so nobody starves anyone, a default-deny NetworkPolicy, and a virtual cluster when tenants genuinely do not trust each other. That post ends at a section called the ceiling of soft tenancy, and it ends there deliberately, because everything below the ceiling is cheap and everything above it is not.

This post is about above: stop sharing the network path, stop sharing the kernel, stop sharing the control plane.

The question that gets you here is usually not technical. It is a customer asking whether their data can be reached from another customer’s workload, or a security review asking what happens when a container escapes, or your own quiet realisation that “we use namespaces” has been doing more work in conversations than it can actually support. At that point you need to know what the next rungs are, what each one buys, and what it costs you to operate — because the honest answer is that each one costs considerably more than the last, and most teams need a lower rung than they think.

Where the ceiling actually is

Soft tenancy shares one kernel. That is the whole ceiling, stated in six words.

Namespaces, RBAC, quotas and network policy are all controls implemented by the cluster, enforced within the cluster, on top of a kernel every tenant’s workload is running against. They are real controls and they stop real problems — most of the incidents a small platform actually sees are a misrouted service call or a runaway deployment, and default-deny plus quotas stops both. But none of them changes the fact that a container escape is an escape onto a machine somebody else’s pods are also running on.

So the rungs above the ceiling are all variations on one move: stop sharing something. Each rung stops sharing a different thing, and the cost of each rung is proportional to how fundamental the thing is.

Rung five: stop sharing the network path

The first move up is usually not compute at all. It is giving a tenant network separation that does not depend on a policy object being correct.

A default-deny NetworkPolicy is enforcement by configuration. It works, and it is the highest-value thing on the lower half of the ladder — but it is one manifest away from being wrong, and it is enforced by the same CNI that every other tenant is using. Moving a tenant onto its own network path, with its own address space and its own egress, changes the failure mode: a policy mistake now leaks within a tenant rather than across the cluster.

What this costs is not compute. It is routing complexity, and the fact that you now have per-tenant network state to keep correct. Every additional address space is one more thing that can collide, one more route that can go stale, and one more place a debugging session has to look. I would take this rung when tenants have genuinely separate egress requirements — different upstream endpoints, different compliance stories about where traffic exits — and not merely to feel safer about east-west traffic, which default-deny already handles.

Rung six: stop sharing the kernel

This is the rung that actually answers the security review, and it is the first genuinely expensive one.

Running a tenant’s workloads in virtual machines rather than containers — on Kubernetes, that means something like KubeVirt — puts a hypervisor boundary between them and everyone else. A container escape lands the attacker inside a VM belonging to the tenant they already had access to. That is a categorically different blast radius from an escape onto a shared node, and it is the first rung where you can answer “can another customer’s workload reach our data” with something stronger than a description of your policies.

The cost is real and it is not only money. You are now running VMs, which means you are running an operating system per tenant: patching it, watching it, sizing it, and accepting that a VM which is idle still occupies memory in a way an idle pod does not. Scheduling gets coarser. Bin-packing gets worse. The thing you were buying with containers — density — is exactly the thing you are now spending.

There is a version of this argument that says “just use VMs for everything,” and it is wrong for the same reason “just use containers for everything” is wrong. The rung is worth paying for when the boundary is the product. It is waste when it is a reflex.

Rung seven: stop sharing the control plane

The top rung is a cluster of the tenant’s own — in my case, child clusters with their own API servers, scheduled onto the same underlying hardware.

What this buys is the thing no lower rung can: the tenant can have their own cluster-scoped objects. Their own CRDs, their own operators, their own admission controllers, their own version of Kubernetes on their own upgrade schedule. On a shared cluster, one tenant installing an operator that registers a CRD affects everyone, and one tenant needing to stay on an older version holds the whole platform back. A child cluster ends both of those arguments permanently.

The cost is that you now operate N clusters, and cluster count is the thing that scales your operational load fastest. Every cluster is a certificate rotation, an upgrade, a set of controllers, and an audit surface. This is the rung where a small team should be most suspicious of itself, because it is genuinely the cleanest answer and it is also the one that will quietly consume the ops capacity you were preserving by having a small team in the first place.

The caveat I have to state plainly

Everything above isolates network and data. None of it, as I run it, gives you compute fair-sharing on the shared rungs — the VM rung does separate compute, but by dedicating it rather than by sharing it fairly.

I do not run per-tenant resource quotas on my own platform, and I do not have a tenant RBAC model, because on my platform tenants do not touch the API at all — I operate it for them, which removes the threat those controls exist to address. That is a legitimate design choice for the shape of my estate. It is also a real gap in the ladder, and I would rather say so than let a diagram imply an isolation story I have not built.

So if you are reading this to work out whether a noisy neighbour can starve your workload: that is a quota question, it lives on the lower half of the ladder, and it is answered in the post about soft tenancy rather than here. The rungs above the ceiling are about reach, not about share. They are different problems that happen to be discussed under the same word.

Which rung you actually need

Most teams asking this question need rung four — the virtual cluster — and have not finished implementing rungs one to three properly. That is not a criticism. Default-deny is unglamorous, and quota manifests are the kind of work that never feels urgent until the afternoon it does.

The decision I would run:

• Do your tenants touch the Kubernetes API? If not, most of the lower ladder is answering a question you do not have, and your real risk is network reach. Start there.
• Is the boundary something you are selling? If a customer contract or a compliance regime specifies isolation, buy the rung that lets you describe it in one sentence without qualifiers. Rung six is usually that rung.
• Does a tenant need their own cluster-scoped objects, or their own upgrade schedule? That is the only question that genuinely requires rung seven, and if the answer is no, a child cluster is an expensive way to feel organised.
• Can you name the incident you are preventing? If not, you are buying a rung to settle an argument rather than to reduce a risk, and the argument is cheaper to have.

The ladder is worth having in your head as a whole, because the useful move is usually not “go up one rung.” It is realising which rung the actual risk lives on, and discovering that it is two below where the conversation had drifted.