I found the problem while looking for something else, which by now is my favourite diagnostic method: an automated alerts channel had gone silent. Nobody misses a channel that only informs, so the silence felt like calm.
Every time a certain event happens, the system posts a notice to that channel. The data gets saved either way; the notice just saves a trip to go find it, and without the notice nobody goes looking.
The new credential went out and the system kept the old one
We rotated the credential on the messaging platform's side, the key that authorises a system to send messages. The system kept presenting the previous one, so every attempt got a 401, the response an API uses to say that key is no good anymore.
A channel that only informs fails differently from one somebody is waiting on: every rejection ends up in an error log, and an error log only gets opened when you're already hunting a problem. It's a doorbell with a cut cable: nobody complains about the silence and you assume nobody came.
A log with no reader is not monitoring
Change my mind
The deploy was the obvious culprit
I thought the sending code had broken in a deploy, because we'd touched that part not long before. The code was unchanged.
We confirmed it with a read-only endpoint, a query that returns 200 if the credential works and 401 if it doesn't, without posting any message. It answered 401. Testing it with a real send would have slipped a fake notice in among the real ones, and the silence was mess enough.
Resending the backlog without inventing the format
With the credential updated the new alerts went out again, but the events from the silent period were still unannounced. I wrote a resend script that reuses the production functions instead of rebuilding the messages, because an alert in a different format looks like it came from some other system.
I gave it three handbrakes: it runs in dry-run by default, which prints what it would send without sending anything; it refuses to start if the system isn't running in its normal mode; and it skips notices that are no longer current, because announcing something already over only confuses people. The pending alerts went out in a single run, in the usual format.
One path, and a log that counts successes
Underneath the incident there was more than one route for publishing alerts, each with its own credential. Every extra route is one more place where a key can go stale. We kept a single one: today every alert goes out through the same API, and one credential that expires is one credential to renew.
The other change was flipping the log around. Recording errors wasn't enough, because errors show up where nobody is looking; now the log also records every successful send. Monitoring by successes changes the question: instead of trusting that some error will shout, you count what did go out, and silence stops being an absence and becomes a number.
The second time we saw it right away
Weeks later the messaging platform went through a migration and the process that published these alerts dropped out of service. This time the count of successful sends stopped moving, and that gave it away. Once it resumed, the cross-check between the database records and the chat history closed at zero lost notices.
When we rotate a credential, the new one goes through the query that posts nothing first. The doorbell still doesn't complain when it goes quiet, but now someone counts how many times it rang.
