Down for 44 minutes because a cable stayed plugged in
· 6 min read · Tim Thomas
The site had a backup network path the whole time and never used it, and every alert we had went out over the link that was broken. What an outage teaches you that a design review never does.
FM.video runs on a physical machine at my desk, behind a tunnel to a global edge network. I know how that sounds. It is a deliberate choice, and the reasoning is a post of its own, but the short version is that the part of the system that finds music videos needs a residential internet connection, and a machine I can walk over to is easier to reason about than a cloud I cannot.
The cost of that choice is that my desk is now infrastructure. On Saturday 23 August I was tidying it, and at 10:27 the site went down.
The cable that stayed up
The machine has two ways onto the network: a cable, and WiFi as a standby. The standby is supposed to take over if the cable goes. I had tested that by pulling the cable out of the machine, and it worked.
What I had not tested was a cable coming loose at the other end, past the switch. That is what happened. From the machine's point of view the cable was still perfectly connected: the link was up, the light was on, the lease was valid, the error counters were zero. There was just nothing behind it. So it kept sending every packet down a link that was up and led nowhere, and never once considered the WiFi, which was healthy the entire time and was receiving the router's chatter that the cable had stopped seeing.
Failover based on "is the link up" cannot see "link up, path dead". Nobody had told me that, and I had not thought to ask.
Every alarm rang in an empty room
The monitor on the box noticed within eight minutes. It correctly diagnosed "the public edge is down but the local services are healthy" and did the only thing in its repertoire: it restarted the tunnel. Three times. Restarting a connector that cannot dial out does nothing, and it had no way to know that.
Then it tried to tell me. The alert channel was email, which goes out over the same link that was broken. So the alert about the outage was itself stuck in the outage. I found out the way you find out about these things: by looking at the site.
Forty-four minutes, until a human reseated a cable.
What the site did in the meantime
This is the part that had gone right, mostly. Two weeks earlier the agent had built a shield at the edge: a worker that keeps a snapshot of every important answer the site gives, so that when the origin is unreachable the edge serves the last good copy instead of an error. During those 44 minutes the home page loaded, the charts loaded, the station list loaded, all from snapshots.
Except the one thing people actually come for. The list of what a station has been playing showed "Couldn't reach the feed", because that specific endpoint had been deliberately left out of the shield two weeks earlier to stay under a request cap that no longer applied. The reasoning had been "a stale now-playing is worthless", which is true, but the feed is not the now-playing, and the exclusion had caught both.
The three fixes, built that afternoon
I asked the agent to write up what happened and what it recommended, read it, and said "do everything you recommend". By the evening all three were live.
Failover that tests the path, not the light. A small watchdog every thirty seconds sends real traffic through each interface separately, to the gateway and then to the internet. If the cable fails three times in a row while WiFi passes, it moves the default route to WiFi; when the cable recovers, it moves back. It does nothing if both paths are dead, because failing over would not help. The agent wrote a test that walks all 22 branches of that logic on a healthy machine, so the failover can be verified without an outage.
Alerts that cannot share the failure. SMS now goes out over the mobile network, which does not care about my cable. And an external check runs on the edge network itself, probing the site from outside every five minutes, so it works even if the whole box is off. The two split cleanly: the outside check answers "is the box reachable at all", the inside monitor answers "are the services healthy".
Shield the feed. The feed now has its own path so the edge can protect it separately from the now-playing poll, with the snapshot's age passed through to the app, so during an outage the station page says "last known playlist, 20 minutes ago" instead of either lying or failing.
What I took from it
An outage is the only design review that tells the truth. Every assumption in the setup had been reasonable, and two of them were wrong in a way that only a specific, physical, slightly embarrassing event could reveal. The write-up the agent produced afterwards, with the exact timeline from the logs, is in our knowledge base with the date on it, so that the next time someone (or something) proposes "we should fail over to WiFi", the answer is already written down: to what, and how would you know?
The site has not been down since. The desk is still infrastructure. I tidy it more carefully.
Earlier in this log: DevOps in the AI World. Two days after this, a very different kind of day: the Dolly Parton tribute.