Where the meter hides: how to read a six-figure observability bill

When I open a cloud bill, I am not looking for savings. I am looking for the three or four line items that nobody in the company owns.

That is the whole method. Every large bill I have read has the same shape: eighty percent of it is decisions someone made deliberately and can defend, and the rest is accretion — defaults nobody chose, meters nobody read, and the same data paid for twice by two teams who never compared invoices. The accretion is where the money is, and it is invisible from the inside because each individual line looks reasonable to the person who owns it.

This post is the diagnosis, not the fix. I have written the fix elsewhere: the six levers that cut the bill, the argument for owning your observability, the architecture that replaced the SaaS, and the war story of how it started. What I have not written down before is how I actually read the invoice — which lines I go to first, what arithmetic I do on them, and the point at which I tell someone not to hire me. That is what a paid teardown produces, and this is a public one, run on a bill I know better than any client’s, because it was mine to fix.

A note on where this sits. This post is the method — how to read the bill, line by line, and what each line is actually charging you for. The cost teardown itself, with the before-and-after figures and the projection caveats stated properly, is in the Datadog-alternative post. The numbers below are that same estate; here they are working examples rather than the headline claim.

The bill

A high-scale real-money gaming platform, moving 6 TB of telemetry a day across 15-plus departments. Logs, metrics and traces from on-prem game servers and AWS workloads, retained 90 days.

The observability spend, before I touched it, was two invoices:

Line	Monthly	Basis
Datadog	~$120,000	Projected at 2024 list pricing, 90-day retention
AWS CloudWatch	~$105,000	Projected at 2024 list pricing, 90-day retention
Total	~$225,000

One caveat before we go further, because it matters more than anything else in this post. The ~$225K is a projection, not an invoice I received. It is what that telemetry volume at that retention would have cost on those two vendors at 2024 list pricing. The number I actually received, a year earlier at 1 TB/day, was $80,000 a month — $40K Datadog and $40K CloudWatch running side by side. That one is real. The self-hosted number at the end of this post, ~$25,000 a month at 6 TB/day, is also real: it is what I operated.

I label these because a consultant who blurs “would have cost” into “did cost” is a consultant whose other numbers you also cannot trust. If your teardown does not tell you which of its figures are invoices and which are models, throw it away.

Line item one: you are paying for the same data twice

The single largest recoverable amount in this bill was not a pricing inefficiency. It was duplication.

The platform ran the Datadog agent for application observability. It also, separately, shipped logs and metrics into CloudWatch — because that is what AWS services do by default, because a compliance requirement had been satisfied that way years earlier, and because the infrastructure team and the application team had each solved their own problem correctly without ever putting the two invoices on one page.

Both systems ingested substantially the same telemetry. Both charged for it. Neither team knew the other one’s number, because one arrived as a SaaS invoice through procurement and the other was buried inside a seven-figure AWS bill as a line called CloudWatch.

This is the most common finding I have, and it is worth stating plainly: double instrumentation almost never shows up as a problem in either team’s own view of the world. It only appears when someone puts both invoices side by side and asks what each one is for. That is a fifteen-minute conversation, and it is usually the most valuable fifteen minutes of the engagement.

The first question of any teardown is therefore not “what is expensive?” It is: how many systems is this byte paying for?

Line item two: CloudWatch, where the arithmetic is fully determined

I like starting on the CloudWatch line because it is the one where I can show my work completely. There are no hidden inputs. Volume and retention determine the number.

CloudWatch Logs on the Standard log class bills $0.50 per GB ingested, and archived log storage bills $0.03 per GB per month (AWS CloudWatch pricing, us-east-1, accessed August 2026).

Run it on 6 TB/day:

• Ingest: 6,000 GB/day × 30 days = 180,000 GB/month × $0.50 = $90,000/month
• Storage at 90-day retention: roughly 540,000 GB resident × $0.03 = ~$16,200/month
• Total: ~$106,000/month

That reconstruction, at today’s list pricing, lands within a few percent of the ~$105K projected back then. Two independent passes agreeing is the only reason I am comfortable publishing the figure at all.

Now look at what that arithmetic tells you. Ninety thousand dollars a month is the ingest meter alone, and the ingest meter does not care whether anybody ever queries the data. It is charged at the door. Retention adds a sixth on top, but the door is the expensive part.

So the second question of a teardown is: what fraction of this ingested volume has ever been read? On every platform I have looked at, the honest answer is a small one. Debug-level logs from healthy services, request logs from internal health checks, verbose framework chatter that was switched on during an incident in some earlier year and never switched off — all of it paying $0.50 a gigabyte to be written to a store nobody opens.

There is a cheaper class. Infrequent Access bills $0.25 per GB ingested, half the Standard rate (AWS CloudWatch pricing, accessed August 2026). Moving the never-queried streams onto it is a config change, not a migration, and on this bill it would have been a five-figure monthly line by itself. Almost nobody does it, because the class assignment is a default and defaults are not decisions.

Line item three: Datadog, where the meter hides

The Datadog side is a different problem, and I am going to be careful here.

I am not going to reconstruct the ~$120K line by line the way I did CloudWatch, because I cannot do it honestly. Datadog’s log billing is two separate meters — $0.10 per GB ingested, plus $1.70 per million log events indexed at 15-day retention — and the second meter is driven by event count and average event size, not by volume alone (Datadog pricing, accessed June 2026). To derive a number I would have to assume an average event size and an indexing ratio. I would be publishing my assumptions dressed as findings. In a client engagement I pull the real counts out of the usage dashboard; in a public post I do not have them, so the canonical figure stands as a projection and I leave it there.

What I can tell you is the mechanics, which is what you actually need to audit your own bill.

Infrastructure is billed per host, and the host count is not the number you think it is. Datadog lists $15 per host per month for Pro infrastructure monitoring on annual billing, and $31 per host per month for APM on top (Datadog pricing, accessed June 2026). The billing model meters your host count hourly, discards the top 1% of hours, and bills the entire month at the peak of what remains. On an autoscaling platform, that means a single traffic spike sets the price of the whole month. Your average host count is irrelevant to your invoice. If you are running Karpenter or any aggressive scaler, your Datadog bill is a function of your worst hour, and cost-optimising your compute by scaling harder can actively raise it.

Ingest and index are separate meters, and retention multiplies the second one. The $1.70-per-million-events figure is quoted at 15-day retention. Push retention to 90 days for a compliance requirement that was never re-examined and you are paying the index meter on a longer tail. Retention is treated as an engineering setting. It is a finance setting wearing an engineering costume, and in every teardown I ask who signed off on the number and when. The answer is usually “nobody” and “before my time.”

Custom metrics are the third rail. CloudWatch charges $0.30 per metric per month for the first 10,000 metrics (AWS CloudWatch pricing, accessed August 2026), and every vendor in this category prices some version of the same thing. The trap is that “a metric” means a unique combination of name and label values. Add a user_id label to one counter and you have not created one metric; you have created as many metrics as you have users. Cardinality is the fastest way to turn a one-line code change into a five-figure monthly line, and it is invisible in code review because the diff is three words long.

Line item four: the network charges that are really observability charges

Two findings from this platform never appear under “observability” in any cost report, and both were caused entirely by the observability stack.

S3 traffic routing through the NAT gateway. Loki and Mimir continuously push chunks and indexes to S3. That traffic was leaving through the NAT gateway, which meant paying NAT data-processing charges on every gigabyte written to S3, on top of the transfer itself. An S3 Gateway VPC endpoint is route-table based and carries no hourly or data charge. Adding it moved all S3 traffic onto the private AWS backbone and removed the NAT processing charge for S3 entirely — better security posture as a side effect. I have written the general case up separately in the NAT gateway hidden tax; the point here is that it surfaced as a networking line while being caused by a telemetry decision.

Cross-AZ transfer on intra-cluster replication — and the arithmetic trap that hides most of it. This one is worth slowing down on, because the naive reading is off by roughly an order of magnitude and the naive reading is the one almost everyone does.

Here is the tempting version. Inter-AZ transfer bills at about $0.01/GB. The platform moves 6 TB/day. Six thousand gigabytes a day, thirty days, a cent a gigabyte — about $1,800 a month. Real, but small against a $225K bill, and easy to file under nice to have.

That number is wrong, and the way it is wrong is the whole method in one example. A log does not cross the network once. It crosses at every hop of the write path — agent to gateway, gateway to distributor, distributor to ingester — and then the ingesters replicate it three times, each copy landing in a different zone. Inter-AZ is also billed per direction: $0.01/GB each way, not once.

So the quantity to multiply is not the payload. It is the real cross-AZ volume, which lands north of 20 TB a day once the hops and the replication factor are counted. At $0.01/GB each way, single-AZ was worth on the order of $10,000–15,000 a month — not eighteen hundred.

⚠ One distinction that is easy to lose while reading a transfer bill: inter-AZ is $0.01/GB per direction. The $0.09–0.10/GB figure is internet egress, a different line entirely. Mixing them inflates the estimate as badly as ignoring the hops deflates it.

That reframes the trade rather than removing it. Pinning nodes to a single zone takes the charge to near zero and gives up zonal redundancy — so the honest version is that high availability had to be rebuilt a different way, with blue-green production clusters and automated zonal failover. The point is that this is a $10–15K/month decision, not an eighteen-hundred-dollar one, which is exactly the size at which “we never questioned multi-AZ for a monitoring system” stops being a detail. The full lever is written up in the observability teardown.

Cost work that ignores what it breaks is not cost work — it is deferred incident cost. But cost work that under-counts what it found is not cost work either. The difference between $1,800 and $10–15K is not a pricing subtlety. It is the difference between reading a bill and reading a payload.

What the other side actually costs

The self-hosted replacement — a Grafana LGTM stack (Loki, Mimir, Tempo, Grafana with Alloy) across four EKS clusters, blue-green, multi-tenant by X-Scope-OrgID for 15-plus departments — ran the same 6 TB/day at 90-day retention for ~$25,000 a month. Real operated cost. That is about 79% off the Datadog line and about 89% off the duplicated whole.

The levers that got it there, with the honest numbers:

• Tiered storage, S3 to Glacier Deep Archive past the 90-day hot window: 96% off cold-log storage.
• Single AZ, with blue-green clusters restoring HA: ~$10–15K/month of cross-AZ transfer, gone.
• ARM Graviton2 for the compute: 20–40% off, at the cost of every image and agent needing to be ARM-clean, which added real build and test friction.
• Caching and right-sizing — a memcached tier that takes most reads off S3, and workload-specific node pools rather than one general-purpose one. The mechanics of both, with the instance families and the pool layout, are in the observability teardown — that post owns the levers; this one owns the bill.
• S3 Gateway endpoint removing the NAT processing charge on the S3 write path.

What broke on the way

I would not publish a cost post without this section, because a teardown that only lists wins is marketing.

Subnet IP exhaustion stopped node scale-up cold. The VPC CNI assigns an IP per pod. At this scale the worker subnets simply ran out of addresses, and Karpenter could not launch nodes — pods sat Pending reporting no nodes available, while compute limits were nowhere near their ceiling. The failure mode pointed at the wrong subsystem entirely. Fixed by re-planning subnet CIDRs and enabling prefix delegation.

Node provisioning failures had overlapping root causes that were genuinely hard to disambiguate: pool CPU and memory limits, the subnet exhaustion above, and EKS-optimised AMI IDs drifting after version bumps. Three different problems producing one identical symptom.

An EFS volume was deleted and recreated, putting its data at risk. We recovered it and used the incident to move that data onto managed MySQL on RDS, where a repeat could not lose it. Separately, EFS UID mismatches put pods into CrashLoopBackOff with permission-denied on the shared mount, fixed with an init container doing the chown before the main container mounts.

Spot was not wired up at setup. Interruption handling was not ready, so the querier and compactor pools ran on-demand longer than they needed to. That is money left on the table by my own sequencing, and I would do it earlier next time.

Six weeks of parallel running, four clusters, and a cutover that had to be zero-downtime because the platform underneath it was taking real money from real players. This is not a weekend project.

The decision tree

Here is the actual branch I walk, in order. It is short, and three of the four exits are “don’t do this.”

1. Is the same telemetry being paid for more than once? If yes, fix that first, whatever else you decide. It is the highest-return, lowest-risk change available and it requires no migration at all.
2. Is the workload steady, or spiky? Owned and self-hosted infrastructure wins on predictable load, because you are buying capacity rather than renting it by the unit. Genuinely bursty workloads belong on metered cloud, or on a hybrid. If your load profile is spiky, the honest answer is optimise in place — and I will tell you that on the call.
3. Does someone own the platform after I leave? A self-hosted LGTM stack is a system, not a purchase. It needs a person with real Kubernetes operational depth, and it needs on-call. If there is nobody, the savings will drift back within a year and you will have bought yourself an outage instead of a discount.
4. Is the recoverable amount worth the disruption? Below roughly $50,000 a month of total bill, the migration effort usually costs more than it returns. Optimise what you have: log classes, retention, Savings Plans, right-sizing, storage lifecycle. I have written the starting checklist for a bill that size.

When you should not do this

Explicitly, because the failure cases matter more than the success case:

• Your workload is genuinely bursty. Cloud is the right call for you. Owning this would cost more, not less.
• You have no platform ownership and no plan to build any. Self-hosting without an owner is a slow-motion incident.
• Your bill is under ~$50K/month. There is real waste in it, almost certainly, but the answer is optimisation, not migration.
• You are already lean. It happens. Some teams have done this work and their bill is honest. In that case there is nothing here for me to sell and I will say so.

That last one is not modesty. If I take money to tell a lean team to keep doing what they are doing, I have converted a referral into a refund.

What this costs you to find out

Everything above came from reading two invoices and doing arithmetic on them. No access to source code, no architecture review, no discovery workshop. Volume, retention, host metering, log class, cardinality, and the question of how many systems each byte is paying for. That is the first pass, and on a large bill the first pass is usually where the number is.

One thing this method deliberately does not answer. It tells you what your observability is costing and which parts of that are choices — while you stay exactly where you are. For most bills that is the right first question, because the waste is real and removing it needs no structural change at all. The second question only arrives once the bill is honest: whether the load underneath it should be running on rented infrastructure in the first place. That is a different read, and it is a different post.

If you are running a cloud and observability bill north of $100K a month and cannot say which line items are decisions and which are accretion, I will do the first pass on yours for free — 15 minutes, your actual bill, no deck. You will leave the call with a real number or with me telling you that you are already lean.

Book the bill review →