Everything that was broken looked fine

A backup bucket on my own infrastructure was flagged as seventeen days stale. That was wrong in both directions. It was over four months stale — and the reason it looked recent was a file called _writetest.txt, left behind by an offboarding task that had checked whether it could write to the bucket, confirmed that it could, and moved on.

Nobody was careless. The write-test was a sensible thing to do and the person who did it had a good reason. It just happened to update the timestamp that everything downstream was reading as “backups are arriving.” The monitoring was right that something was recent. It was completely wrong about what.

I found four more like it in the same audit. None of them was alarming from the outside. All of them were reporting success. Here they are, and then the thing they have in common.

The clusters with no backups at all

Three production PostgreSQL clusters, all running, all healthy, all serving. Backup configuration: empty. No scheduled backups, no backups, nothing. Not stale — absent.

The reason this survived is that nothing about a database with no backup looks different from a database with backups. The pods are running. The queries return. The dashboards are green, and they are green correctly, because every question they were asked was about whether the database was up. Nobody had wired a question about whether it was recoverable.

I want to be accurate about the scale rather than dramatic: these were small clusters, well under a gigabyte between them. The lesson is not that I nearly lost something enormous. It is that the absence of a backup produces no signal at all, so the size of what you nearly lost is decided by which database happens to be affected, not by anything you would have noticed.

The backup that was missing exactly one segment

A different cluster did have backups configured. Its first base backup was taken slightly before the archiver had somewhere to ship its write-ahead log, so the archive begins one segment after the backup ends.

One segment. The backup completed, reported success, and sits there looking like protection. A restore from it fails, because the chain from that snapshot to any usable state has a gap at the very start.

The rule I wrote down afterwards is short: after enabling archiving, take a second base backup and treat the first as scaffolding. The first one is a rehearsal that produces an artefact indistinguishable from the real thing.

The credentials nobody could decrypt

An integration on one of my sites depended on credentials that had been undecryptable for four months. Four months, no error surfaced to anyone, no failure anybody noticed.

It did not break because nothing had asked it to do the thing the credentials were for. The integration was configured, present in the inventory, and would have appeared in any audit of what exists as a working component. It was working in the sense that it had not yet failed, which is a different property from working, and one that lasts exactly until somebody needs it.

The folder named after what it was not

One storage folder was named for a category of thing that is re-downloadable — inputs you can always fetch again, the sort of content nobody protects because nobody needs to. That is what the name promised and that is how it had been treated.

It also held several gigabytes of build output: artefacts that took hours of machine time to produce and exist nowhere else. Backed up nowhere, because the name had done its job of telling everyone this folder was disposable.

Names are the cheapest metadata in a system and the most trusted. Judge the contents. The name was written once, by somebody who did not know what the folder would become.

And the same bucket, wrong in the other direction

Here is the part I did not expect. That first bucket — the one the write-test kept looking fresh — had genuinely not received a backup in over four months. Strip away the misleading timestamp and what you are left with looks unambiguously like a failure.

It was not one. The bucket held backups for a child cluster that had been decommissioned; its workloads had been moved elsewhere months earlier. Backups stopping was the consequence of that decision, not a symptom of anything. The silence was correct. The only action it needed was eventually deleting the bucket.

So one artefact carried two wrong readings in opposite directions. The proxy said recent, and that was reassuring and false. The raw fact underneath said four months of nothing, and that was alarming and also false. Neither reading was about the thing anybody actually wanted to know, which was whether a database that still existed could still be restored.

That is the case that stops this being a post about how monitoring lies to you. Monitoring did not lie. It answered the questions it was asked, correctly, and none of them were the question.

What these have in common

Every one of these was reporting success while doing nothing useful, and the mechanism is the same every time — five artefacts, six wrong readings, because the bucket carries two: something was measuring a proxy, and the proxy held while the real thing failed underneath it.

• The bucket’s proxy for backups are arriving was object recency. A write-test satisfied it.
• The clusters’ proxy for the database is fine was the database being up. It was up.
• The first backup’s proxy for this is protection was the backup completing. It completed.
• The integration’s proxy for this works was nothing having failed. Nothing had.
• The folder’s proxy for this is replaceable was its name. The name was accurate when it was chosen.
• And the same bucket’s proxy for something is wrong was backups having stopped. They had stopped for a good reason.

The proxy is not biased. It is simply not the thing. It can be wrong in the reassuring direction and wrong in the alarming direction, and it will be wrong in whichever direction the world happens to move. That is a harder claim to argue with than alerts lie, and it is the more useful one, because it tells you what to do: check the thing.

Proxies are not a mistake. You cannot monitor the real thing directly in most cases — “is this backup restorable” is not a check you can run continuously, so you check whether an object arrived and you infer the rest. The inference is usually correct, which is what makes it dangerous: it is right often enough that nobody re-examines it, and when it decouples from reality there is no event. Nothing fires, because from the proxy’s point of view nothing happened.

That is why these are all discovered by audit rather than by alerting. An alert can only tell you that a thing it is watching has changed. None of these changed. Two of them I can date to four months, because something eventually forced the question. For the rest I do not know how long they had been true — and not knowing is the point. A silent failure has no start date you can read off anything.

What I actually changed

Not more monitoring. More of the same kind of check would have produced more of the same silence.

I check the thing, not the signal for the thing. For the bucket, that meant looking at what the newest object actually is rather than when it arrived. That single question — what is the most recent file, and is it a backup? — would have caught the first failure on day one, and it costs nothing.

I test restores rather than trusting completions. Running the restore is the only check that examines the real property rather than a proxy for it, and it would have caught three of the five above on the day they happened.

I audit for absence, not just for errors. Every one of these was invisible to error-based monitoring, because none of them produced an error. The question that finds them is not what is failing but what should exist and does not — and that question has to be asked deliberately, because nothing will ever ask it for you.

And I distrust names. Including my own. Especially the ones that were accurate when they were written.

None of this is sophisticated. It is the difference between asking a system whether it is happy and asking it to prove something. The first question is cheap and it is answered constantly. The second one is annoying, and it is the only one that has ever told me anything I did not already believe.