A batch of test notifications went out to the chat where the system posts its notices, all of them announcing repairs that had finished weeks earlier. The system forgets nothing, and it picks the worst moment to remember. And it wasn't the first time.
There was no new bug: the usual rule, "turn notifications off before you test and set them back to normal when you're done", breaks on its own as soon as two agents work at the same time.
Tests can fire real notifications
My AI agents verify their changes on the store's real system, not a toy copy. A test that marks a repair as ready fires the real notice. That's why the mute switch existed: without it, one careless test ends in a notice nobody asked for.
With a single agent the switch works, and I almost never work with a single agent.
The sign anyone can take down
The notification mode was one field with two values: normal or off. Agent A turns it off and starts testing. Agent B, on another branch, turns it off too, finishes fifteen minutes earlier and sets it back to normal, exactly as the instruction asks. A is still running and the system has no way to know.
I blamed the agents first, sure that one of them had ignored the rule. They followed it to the letter, every time. The field didn't store who asked for the silence or until when, so "restore" meant "cancel everyone's silence", including the silence of whoever was still inside. It was the "do not disturb" sign on a meeting room door: there was only one, and the last person out took it down even with people still working inside.
The agents ignored the instruction
They followed it to the letter
Every agent hangs its own sign
We swapped the single value for a table of temporary permits. Each agent takes its own row with its name, the reason and an expiry time: 180 minutes by default, with a hard ceiling for the forgetful. It releases only its own row and can't touch anyone else's.
One function answers what the current mode is, and every channel uses it. If at least one permit is live, it returns off; if none are left, it reads the normal setting. During tests we no longer touch that setting, so the restore step disappeared, which was the exact step that kept failing.
The expiry covers the agent that hangs halfway through: an orphan permit with no deadline would mute real notifications forever. The read also fails closed, and if the query to the table errors out the function returns off. I'd rather have a legitimate notice wait a few minutes than another batch of tests out announcing repairs from weeks ago.
The duplicate environment we didn't build
We could have set up a rotation, a queue, or a sterner paragraph in the repo policy, but instructions had already lost us several times.
We didn't stand up a duplicate test environment either, which is the textbook answer. Duplicating the whole system, with its databases and its scheduled jobs, is a project of weeks, and the risk sitting on us was solved with one table.
Who holds the sign is visible at a glance
Since the change, no test notification has gone out again. An admin screen shows live who holds a permit and why, so when a notification doesn't arrive, I look at that screen instead of asking around. The test suite takes its own permit when it starts and releases it at the end, and the repo policy requires taking one before any check that can write.
The bad case went from eternal silence to silence with an expiry time, and silence with a deadline gets noticed. The silence lifts when the last sign comes down, and repairs from weeks ago have nothing left to announce.
