Why the two feeds before ours died

When we took over the pricing feed, it was the third attempt. Two vendors had built it before us, and both accounts ended the same way: the client stopped trusting the numbers, then stopped paying. We asked for the logs from both eras before we wrote a line of code. They told a clearer story than any handover call would have.

Across two years and two vendors, there were 74 recorded outages. We sorted them by cause, because the causes are where the lesson is.

Where the outages came from

Forty-one percent were a site changing its markup. That is not a surprise and not really the vendors' fault. Sites change. What matters is the next number down.

Nineteen percent were silent bad data. The pipeline ran, the rows landed, and the values were wrong. Nobody noticed until a buyer spotted a price they knew was off, and by then trust in the whole feed had dropped, well beyond that one source. Twelve percent were simpler still: nobody was watching over a weekend.

The last two categories are the ones a client never sees coming, and the reason we now sell monitoring as part of the build rather than as an upsell. A scraper that breaks loudly is a nuisance. A scraper that breaks quietly is what loses you the account.

Four rules we apply now

We rebuilt the feed around a small set of rules. None of them are clever. They are just the things the previous two builds skipped.

Validate before publishing, never after

Every row is checked against what we already know before it is allowed into the feed: expected ranges, currency, the shape of the record, whether the price moved more than any real price moves in a day. A row that fails does not land. It gets held and flagged. The alternative, checking after publishing, means the bad data has already reached the client's screen.

Alert on absence, as well as errors

Most monitoring watches for errors. The dangerous failure is the one that throws no error at all: the source that quietly returns an empty page, so the pipeline runs clean and the feed just stops updating. We alert when a source goes quiet, not only when it complains.

A scraper that breaks loudly is a nuisance. A scraper that breaks quietly is what loses you the account.

A named person owns each source

Every source has a person against its name, not a rota and not a shared inbox. When something changes, one person is responsible for the fix and knows that source's quirks. Diffusion of responsibility is how a weekend outage becomes a Monday apology.

The client hears about the fix, not the outage

When a source changes, our alerts fire, the owner patches it, and we replay the gap so the history has no hole in it. In most cases the client learns about it from a note that says what broke and that it is already handled. Eleven months in, there has been no data incident that reached them cold.

What it costs to run

Clients are sometimes surprised that the collection itself is the cheap part. Roughly a third of the ongoing bill is on-call. A quarter is alerting infrastructure. About a fifth is replay, re-running a source once it is fixed so the record is complete. The scraping is close to a rounding error next to all of that.

That split is exactly why two vendors could sell the feed and still lose the account. They priced the easy part and skipped the expensive part. The expensive part is the only reason anyone keeps paying.