The outage we caused, and the change we made after

On a Tuesday afternoon we merged a change that looked boring. It renamed a field in one of our parsers, tightened a bit of logic, and passed every test we had. We deployed it, watched the jobs go green, and moved on. The feed it touched kept running. Rows kept landing in the client's warehouse on schedule. As far as any of our systems could tell, nothing had happened at all.

Two days later the client emailed. Their downstream reports had been showing empty categories since Tuesday, and they wanted to know if something was wrong on our end. It was. Something was very wrong on our end, and we had spent two days looking straight at it without seeing it.

What actually broke

The rename was correct. The logic change was not. A conditional that used to catch a certain class of records now silently skipped them. The job did not error. It fetched, it parsed, it wrote. It just wrote a smaller slice of reality than it was supposed to, and it did so cleanly, every run, with no complaint.

Our monitoring at the time watched for failures. Failed requests, exceptions, jobs that did not finish. This job finished fine. It returned data. The volume was lower, but nothing we had running was paying attention to volume. From the outside the feed looked completely healthy. The only signal that anything was wrong lived in the client's report, and they were the ones reading it, not us.

What we told the client

We did not soften it. We wrote back within the hour and said plainly that a change we deployed on Tuesday had dropped a category of records from their feed, that the fault was entirely ours, and that they had been running on incomplete data for two days as a result. We told them we had already identified the cause and were backfilling the missing window.

Then we told them the harder part, which was that our monitoring had not caught it and would not have caught it, because we had built it to watch for errors and this was not an error. We would have kept shipping incomplete data until they told us to stop. That admission was uncomfortable to write. It was also the truest thing we could say, and pretending otherwise would have been worse than the outage.

The failure was the parser. The real problem was that we had built a safety net with a hole exactly the shape of this bug.

The backfill itself was straightforward once we knew the window. We re-ran collection for the affected range, restored the skipped records, and confirmed the counts against the source. By the end of the day their reports were whole again. The trust took longer, as it should have.

The change we made after

We stopped trusting the absence of errors to mean the presence of correctness. Those are different claims, and we had quietly been treating them as the same one.

Concretely, we changed how we monitor and how we ship:

• We alert on the absence of expected data and on volume that falls outside the normal band, so a feed that quietly shrinks pages someone the same way a feed that crashes would.
• We validate output against expectations before it publishes, so a run that produces a smaller or stranger slice than usual gets held for a look instead of flowing straight through to a client.

The volume check is the one that would have saved us. A feed that had been returning a steady count for months suddenly returned meaningfully less, and that drop, on its own, is enough to wake someone. It does not need to understand the bug. It just needs to notice that today does not look like every day before it, and ask a human whether that is expected.

Validation before publishing is the second layer. Even when volume looks fine, a run can be subtly wrong in ways a schema and a few sanity rules will catch: fields that should never be empty, values outside a plausible range, structures that do not match what the source has always given us. Holding a suspect run costs a small delay. Publishing a bad one costs a client two days of decisions made on data that was lying to them.

What we carry from it

The thing we got wrong was not the code. Code breaks, and we plan for that. The thing we got wrong was believing that green jobs meant good data. A pipeline can run perfectly and still be feeding someone the wrong answer, and if the only person positioned to notice is the client, then you have not really built monitoring. You have built a system that reports its own opinion of itself.

We watch for silence now, and for data that has quietly changed shape. That one outage taught us more about reliability than any run of clean weeks ever did.