How I Cut an $80K/Month Observability Bill to $15K
At one terabyte of telemetry a day, a high-scale real-money gaming platform was paying $80,000 a month to watch its own systems. That was $40K to Datadog and another $40K to AWS CloudWatch, running side by side, neither one aware of the other. I replaced both with a self-hosted LGTM stack I built and now operate. A year later the platform was pushing six terabytes a day — six times the data — and observability still cost $15-18K a month. The volume grew 6×. The bill barely moved.
That gap is the whole story, and it is not luck. It is six specific places where an observability bill hides money, and six specific things I did to take that money back. This is the teardown. The numbers here are real — I ran this platform — and where a figure is an estimate rather than a bill I actually paid, I say so plainly.
I have written the argument for why you should own your observability separately, in the Datadog-alternative teardown and the full gaming-company war story. This post is the engineering: the six levers, what each one costs you if you ignore it, and what it gave back.
First, the honest version of the “after”
Let me get the one soft number out of the way up front, because the honesty is the point.
The $80K, the 1TB/day, the 6TB/day, and the $15-18K are all real. What I will not do is tell you Datadog and CloudWatch “would have cost X” at six terabytes a day, because we never let it get there — so no such invoice exists. What I can tell you is the mechanism: SaaS observability is metered per GB ingested. That meter is linear. Six times the data on a linear per-GB meter is roughly six times the bill, which extrapolates the $80K toward several hundred thousand dollars a month. That is an extrapolation from a real 1TB bill, not a bill anyone received. Treat it as the shape of the curve, not a quote.
The self-hosted number is not an extrapolation. $15-18K a month at 6TB/day is what the stack actually cost. Here is where every dollar of that restraint came from.
Lever 1 — Tiered storage: stop paying hot prices for cold logs
Where the money hides: most log data is written once, read never. But if all of it sits in hot S3 at $0.023/GB-month, you pay premium storage rates for data nobody will ever query again. At six terabytes a day, that back-catalogue becomes the quiet majority of your storage bill.
What I did: an S3 lifecycle policy that keeps 90 days of logs hot and fast, then moves everything older to S3 Glacier Deep Archive at roughly $0.00099/GB-month. That is a 96% cut on cold-log storage. The trade-off is honest and worth stating: retrieving an archived log takes hours, not milliseconds. For logs older than 90 days that is exactly the right trade — you almost never need them, and when you do for an audit or an incident post-mortem, a few hours is fine. Hot data stays hot; only the graveyard moves.
Lever 2 — Single-AZ: the cross-zone tax nobody reads
Where the money hides: AWS charges you to move data between availability zones, about $0.01/GB. An observability pipeline moves enormous amounts of data internally — agents to ingesters, ingesters to storage, queriers back out. Spread that across three zones for high availability and you are paying a transfer tax on every byte, twice.
What I did: pinned each cluster to a single availability zone. The naive version of this number is a trap. At a glance it looks like $0.01/GB × 6TB/day × 30 ≈ $1,800 a month — which is what a quick estimate gives you, and it is wrong, because a log does not cross the network once. It crosses at every hop of the write path — agent to gateway to distributor to ingester — and the ingesters replicate it three times, each copy landing in a different zone. Count the real cross-AZ volume — north of 20TB a day at $0.01/GB each way — and single-AZ was worth on the order of $10,000–15,000 a month, not eighteen hundred. Single-zone deployment takes it to near zero. The objection is fair — you give up zone-level HA — so I bought the resilience back a different way, with blue-green secondary clusters for failover. Multi-AZ for a monitoring system is a default most people never question, and questioning it was one of the largest single levers on the bill.
Lever 3 — ARM Graviton: the same compute, 20-40% cheaper
Where the money hides: the default instance families on AWS are x86. They are also the most expensive way to buy the compute an LGTM stack needs — and it needs a lot, because ingesters, queriers and compactors are hungry.
What I did: moved the whole stack onto ARM Graviton2 instances — the c6g, m6g and r6g families. For this workload that is a 20-40% compute saving for the same work, and Loki, Mimir, Tempo and Grafana all run on ARM without drama. This is the single least clever item on the list and one of the most effective. There is no architecture change, no trade-off to defend. It is a different instance family. Most teams simply never switch because the x86 default is invisible.
Lever 4 — NVME cache: serve queries without hammering S3
Where the money hides: when your storage backend is S3, every cache miss is an S3 GET request — and at query time, a dashboard that misses cache repeatedly turns into a storm of S3 calls, each one slower and each one billed. Cache badly and you pay twice: once in latency, once on the S3 invoice.
What I did: a memcached layer with NVME-backed extstore — hot cache entries in memory, warm ones spilling to fast local NVME disk instead of being evicted. That held an 80% cache hit rate, which cut S3 query load by about 5×. Fewer requests to S3, faster dashboards, lower bill — the same change buying two wins. The NVME comes free with the instance type, so the warm tier costs nothing extra beyond choosing the right node.
Lever 5 — Karpenter: stop paying for capacity you are not using
Where the money hides: static node groups are sized for peak and paid for at peak, all day, every day. An observability workload is spiky — ingest surges during incidents and traffic peaks, query load surges when engineers are debugging — so a fixed fleet means you are renting your worst-case headroom around the clock.
What I did: Karpenter, with more than ten specialised node pools, each matched to a specific job — memory-optimised nodes for ingesters, NVME nodes for cache, and so on. Karpenter provisions the right instance for the pending work and removes it when the work is gone. Right-sizing instead of over-provisioning. This is where “elasticity” actually earns its keep — not as a cloud marketing word, but as nodes that appear for a traffic spike and are gone an hour later so you never paid for the idle.
Lever 6 — Real multi-tenancy: one stack, not fifteen
Where the money hides: this is the big one, and it is organisational as much as technical. Fifteen teams that each stand up their own monitoring means fifteen sets of ingesters, fifteen storage backends, fifteen bills, fifteen sets of idle headroom. The waste is not any single stack — it is the multiplication.
What I did: one LGTM platform serving 15+ teams as tenants, isolated by tenant ID so each team sees only its own data, on shared infrastructure sized once for the whole. One stack instead of fifteen. This is the lever that most changes the slope of the curve, because it means new teams onboard at near-zero marginal cost instead of spinning up another full deployment. It is also the hardest to retrofit, which is exactly why it is worth designing in from the start.
Why the bill stayed flat while the data grew 6×
Put the six together and the shape becomes clear. Tiered storage means old data falls off a cost cliff instead of accumulating at hot prices. Single-AZ and Graviton cut the fixed cost of moving and computing. NVME caching means more data does not mean proportionally more S3 calls. Karpenter means you pay for load, not for headroom. And multi-tenancy means growth is shared, not multiplied.
A per-GB SaaS meter does the opposite of all six. It scales linearly and against you — every unit of success your product has, it bills you more to watch it. A stack you own scales on compute and storage, both of which you can bend with exactly the kind of engineering above. That is the entire reason the curves diverge: six terabytes a day cost me almost the same as one, because none of the six levers care very much how much data flows through them.
None of this is exotic. There is no proprietary trick, no secret AWS discount. It is six boring, well-understood decisions that nobody made because the bill grew slower than anyone was looking, and by the time it was $80K a month it felt like a fact of nature instead of a set of choices.
What your bill is probably doing right now
If you run observability on a metered SaaS at any real volume, your bill is on the wrong curve — the one that gets steeper as you grow, on a meter you do not own. You do not need to rip anything out to find that out. You need to look at where the money actually goes, which most teams have never done, because the spend was never made legible in the first place.
That is what I do in a free 15-minute cloud-bill review. You show me the shape of your bill; I tell you honestly which of these levers apply to you and roughly what they are worth. Sometimes the answer is a migration. Often it is three changes you can make next week without moving off cloud at all. Either way you leave knowing the number, which is more than most teams have.
Book it here: cal.com/rajesh-medampudi/bill-review. No pitch, no obligation — I lead with the numbers, because the numbers are the whole argument.