Replacing a production platform, node by node, without losing data

The objection this answers

Almost nobody who asks me about leaving the cloud is really asking about money. They have already done that arithmetic. What they are asking is whether the move will break something — whether they will be down, whether they will lose data, whether the thing they migrate onto turns out to be worse than the thing they left.

It is a fair question and it deserves better than reassurance. So here is the version of it I did to myself, with what it actually cost.

What happened

I have run self-hosted bare metal since roughly February 2026, with production workloads on it since roughly April 2026 — about five months of production on my own hardware. The OVH era ended around 23 May 2026.

Then, on 3 June 2026, I did the frightening part: I replaced the entire platform underneath running production workloads. Cozystack out, Talos in. Not a rebuild on new hardware with a cutover weekend — a replacement of the platform on the machines that were already serving.

How it was done, and what it cost in downtime

1. Rolling, node by node. Absorb a node, wipe it, rejoin it to the new cluster. The cluster was never torn down and rebuilt as a unit, so there was never a moment when everything depended on one big step working.
2. Per-service cutover, not a big-bang switch. Workloads moved one service at a time via edge Caddy backend flips. Each flip is a single change.
3. Instant rollback on every flip. If a service misbehaved on the new platform, the flip went back and the old path kept serving. The unit of risk was one service for as long as it took to notice — not the estate.
4. Storage with two independent answers. LINSTOR at RF=2 for replication, and restic to offsite object storage for the case where replication is not the thing that saves you.

Zero data loss across the replacement.

What it cost in downtime: 77 minutes. The cutover ran in two planned windows on 8 June 2026 — one of 53 minutes and one of 24 minutes, both returning HTTP 503 while the flip completed. That is the honest answer to the question behind the objection: not "no downtime", which nobody should believe, but seventy-seven minutes, on a date, in windows I chose.

Where that number comes from: it is measured at the service endpoint by my own monitoring, running continuously since 3 May 2026 — not by a third party, and I am not going to describe it as if it were. It is a duration observed at the service, which is a different and smaller claim than an availability figure.

What broke

Two real failures, both during the period described. Neither took the cluster down, and both are worth stating plainly — a migration story with no failures in it is a story someone is not telling you all of.

• An e1000e NIC-driver hang took a worker off the network. The node dropped out; the cluster kept serving. It rejoined on 19 June 2026. This is the ordinary shape of bare-metal operation — hardware and drivers misbehave, and the design question is whether one of them leaving takes anything with it.
• An edge Caddy crash took the ingress path down for 13 minutes on 4 June 2026 — ECONNREFUSED at the edge — while the cluster stayed up. That one was unplanned, and it is the only unplanned outage in the window. Worth separating the two: the platform was healthy and unreachable. The failure domain that actually bit was the one in front, not the one everyone worries about.

The operating record

In roughly three months of steady-state Talos operation, there has been no complete cluster outage.

Here is how that is measured, because "no outage" is worth exactly as much as the method behind it. The control-plane node's kube-apiserver shows an age of 98 days with 0 restarts. A node reboot resets a static pod's age — so that single figure says both that the node has not rebooted and that the API server has not dropped since 8 June 2026. One number, checkable in one command, and it constrains both facts at once.

What I am not claiming

There is no uptime percentage here, and I am not going to give you one. I do not have a defensible number yet, and a figure I cannot show you the working for is worth less than nothing — it invites you to trust the one claim I could not support and discount the ones I could.

What I have instead is a date: external, third-party uptime measurement started on 15 September 2026. After it has run a real window there will be a verifiable figure for a stated period, measured by something that is not me. Until then the claim is the one above — no complete cluster outage in roughly three months of steady state, measured the way described.