Self-Hosting Doesn't Delete the Bill. It Moves It.

We switched Datadog off. That saved $40,000 a month, it was real, and I have written about it more than once.

Then a NAT gateway charge climbed to about $40,000 a month, in a completely different part of the bill, and nobody noticed for one to two months.

For that window, the headline saving from the migration had an approximately equal and opposite entry somewhere else on the same AWS account. Not because the saving was fake — it wasn’t, and the observability bill really did end up at $15K a month at a terabyte a day. But because moving a workload in-house does not delete its cost. It relocates it — and it relocates it out of the line item you are watching.

This is the post about the part of that migration nobody puts in the case study.

The charge that was not in the observability bill

Here is the detail that made it invisible, and it is the whole mechanism: that NAT charge was never part of the observability bill at all. It sat in EC2-Other — general infrastructure spend.

So consider who would have caught it. Someone reviewing the observability migration is looking at the observability line: Datadog, then CloudWatch, then the self-hosted stack’s compute and storage. That line was doing beautifully. It was the whole point of the project, it was the number in the update to leadership, and every week it looked better than the last.

EC2-Other is not a line anyone opens while reviewing an observability migration. It is where NAT gateway data processing lands — not under a networking heading, not under anything with “observability” in the name. The charge was in plain sight, in a bill we were reading closely, in the one section that had nothing to do with what we thought we were reading about.

So the cost was not slow to disappear. It was slow to be noticed. Once we connected it to the migration that caused it, the fix took an architecture change and it was gone. One to two months at roughly $40,000 a month was the price of the noticing, not the price of the fixing.

Why self-hosted telemetry generates that charge

A NAT gateway bills two ways: an hourly charge for existing, and $0.045 per gigabyte processed — not egressed to the internet, processed on the way through, including traffic bound for other AWS services.

Now think about what changes when telemetry comes in-house. On SaaS, your agents ship data out and somebody else’s infrastructure absorbs every hop after that. Self-hosted, every one of those hops is now yours, on your network, on your bill. The write path fans out — agent to gateway, gateway to distributor, distributor to ingesters, ingesters replicating between themselves, ingesters flushing chunks to object storage, queriers reading them back. An observability pipeline is not a system that occasionally moves data. Moving data is the entire job.

Put that behind a per-gigabyte processing meter and the number gets large quickly. At the point we found it, the platform was at five terabytes a day. (For orientation against the published figures: the teardown’s numbers are anchored at one terabyte a day at the start and six terabytes a day a year on. Five is the middle of that growth, and it is where this episode sits.)

None of that cost existed the month before. It was created by the migration, and it was created in a place the migration’s own reporting did not look.

What we built, and which cells these were

The shape we moved to was cells: self-contained copies of the stack, each one complete, each one bounded.

Two cells, one per data centre — a primary and a secondary, with failover driven by Lambda updating Route 53. That is the whole topology.

It is worth being exact, because I have written about cells before on this platform and they were not the same cells.

The gaming war story describes three cells, one per availability zone, carved out of the application estate — game servers, session caches and player databases co-located so chatty services stopped paying to talk across zones. Those cells shard live traffic, and high availability comes from having several of them.

These are a different system and a different shape. Two, not three. Bounded per data centre. And not a load-sharded fleet at all — a blue-green pair: one serving, one standing by, Route 53 moved between them when it needs to be. The application cells were prior art; I had run the pattern before, which is why reaching for it was obvious. But copying the count or the boundary across would have been wrong. The pattern is portable, the topology is not.

How the meter actually went away

Two steps, and only the first is the obvious one.

The traffic flushing chunks to object storage and reading them back did not need to pass through a NAT gateway at all. VPC gateway endpoints are free, and pointing that traffic at them instead took it off the NAT path. That is the boring, well-documented fix and it is the one to do first. It is also the fix I have pointed to before — in the gaming write-up it appears as a one-line quiet win, “free VPC endpoints eliminated NAT Gateway charges” — and for the S3-bound traffic, that is exactly what they did.

But it was the easy first slice, not the end of the meter. What eliminated the charge was the cell work. In a bounded cell, where a dependency sits and how it is reached is a property of the cell definition rather than a setting somebody has to keep getting right across a sprawling estate — so the paths still reaching out through NAT went away with the boundary, instead of being hunted down one at a time.

I am deliberately not giving you a per-path accounting of that second step, because I do not have one I trust. What I can tell you is the order and the outcome: endpoints first, cells after, and the meter went to zero.

The two other things we got wrong first

We over-built the cache before we optimised it. The NVMe cache — memcached with the extstore backend, spilling hot entries onto local NVMe instead of evicting them — ended up as one of the levers that worked. But in its first form, extstore was forcing us onto large machines to get the capacity we wanted. The saving arrived when we took complexity out, not when we added it.

Optimised, that cache held 6–7 TB on machines costing roughly $400 a month. RAM-based memcached holding the same 6–7 TB would have run about $5,000 a month — roughly 12× for the same working set. That comparison belongs in the same breath as the number, because on its own “12× cheaper cache” reads like a claim about caching in general and it is not. It is a comparison between two ways of holding one specific working set, where the access pattern tolerated NVMe latency.

We ran Cluster Autoscaler before Karpenter. Cluster Autoscaler scales pre-declared node groups reactively — survivable for steady load, poor for ours. The failure that ended the argument: a customer application running its own large-scale load tests generated a burst of log volume, and the cluster could not scale just in time to absorb it. Not a gradual degradation — a specific event where the capacity arrived after it was needed. Karpenter provisions right-sized nodes on demand from a wide instance pool and removes them when the work is gone.

The part that changed how I think about this

We did not want the tightly-bounded shape we ended up with. We wanted to spread out. Spreading a monitoring system across failure domains is the correct instinct, and if there had been no problem with the charges we would definitely have done it. The meters made it impossible — every boundary the design wanted to cross had a per-gigabyte price on it. What we built was a good design and it was also a workaround for a pricing structure — and those two facts sit together without contradiction.

I run my own infrastructure on dedicated hardware now, and what stands out is not that it is cheaper per unit. It is that within a region there are no data-centre-to-data-centre transfer charges at all. The meter that shaped a year of architecture decisions simply does not exist. Multi-zone stops being a cost negotiation and goes back to being an engineering decision made on resilience grounds, which is what it should have been all along.

That is the argument for owning your infrastructure that I find hardest to argue against. Not the monthly number — the fact that on rented infrastructure, the pricing model gets a vote on your architecture, and it votes constantly and quietly.

If you are moving a workload in-house

The saving is usually real. What catches people is the second-order cost, so plan for it explicitly.

1. Before you migrate, write down which line items the workload will newly touch — network, storage, load balancers, data processing. Not the ones it will reduce. Those you will watch anyway.
2. Check whether traffic to object storage goes through a NAT gateway or a gateway endpoint. Gateway endpoints are free. This is a configuration difference, not a project.
3. Count the hops on your highest-volume write path and multiply by your replication factor. The volume crossing a metered boundary is several times the volume you think you are sending, because it is not sent once.
4. Review the whole bill for two months after a migration, not the part the migration was about. That is the specific mistake I made, and it cost one to two months at roughly $40,000.

If you are planning a move like this and want the second-order costs mapped before you commit, that is exactly the kind of thing I look at. Book a free 15-minute bill review.