Test Whether Your Own Alarm Can Lie To You

Every Forum Eulogy Is Mastered at the Same Volume — Gus spent eleven years mastering records at a pressing plant, which means he hears a flat mix before he can name what’s wrong with it, and he ran that ear on twenty years of forum-eulogy essays instead of arguing with them. His finding: every one of them is mixed at the same volume, Usenet’s 110,000 newsgroups landing at the identical weight as a sentence about software licensing, so nothing in the piece ever gets to be the loudest thing in the room. He hands you the three-point scoring trick to run on your own drafts, and it takes about ten minutes to see why nobody’s ever been persuaded by this genre, correct as it is.

The Alarm That Can’t Ring

If your monitor needs the same connection it’s watching to tell you it’s broken, call it what it is: a hope with a cron job.

Most alerting only gets tested by the failure it was built for. Mine got tested by a different one: a dead VPN tunnel that took nineteen hours to notice, because every watchdog I had running every sixty seconds needed the internet to report that the internet was down.

It’s a design problem more than a networking one, and it’s probably sitting in whatever’s watching your site or your mail right now, silent, waiting for a failure it was never built to survive. Mine sat quiet for NINETEEN HOURS before anybody caught it. (Still can’t say that number without wanting to put my head through the monitor.) Go find yours before it finds you.

1 // Map the alert’s dependency chain

Pull up whatever fires when something breaks for you: a cron job, a monitoring service, a webhook, a text message. Trace what it needs to reach you. Does it resolve a hostname first? Call an API over HTTPS, or ride the exact same connection it’s supposed to be watching? Write the chain down, every hop, before you touch anything.

2 // Kill the exact failure it exists to catch

Pick the one thing your alert is built to notice. If it watches for “server down,” stop the server. If it watches for “unreachable,” stop the box from resolving DNS, not the whole network; point resolv.conf at a dead address for five minutes. You own the box. You touch nothing but your own test traffic.

3 // Time the silence

Start a stopwatch the second you break it. Watch for the alert to land somewhere you’d see it. If nothing arrives inside your normal detection window, you’ve found a monitor that only ever worked when nothing was wrong. Write down the exact minute it should have fired and didn’t.

4 // Build the second wire

Wire rooms used to keep a second teletype circuit on a different phone trunk for exactly this reason. One cut line couldn’t blind the whole room. Nobody remembers why until the day the first line goes dead and the backup doesn’t either, because somebody ran both circuits through the same junction box.

Same principle here: one alert channel that reaches you by raw IP instead of hostname, or off a second network entirely, a phone on cellular data instead of your office WiFi. Test it exactly like step two. One working backup is enough.

5 // Put a date on the next test

Set a recurring reminder, every ninety days, to run steps two and three again. Alerting rots. Nobody rings a bell when it does. A provider changes an API and doesn’t tell you, your own config drifts (it always drifts), and you find out on the one day it would’ve mattered.

Standard: if step three comes back silent, what you had was a green light that only ever meant nothing had broken yet. Not a monitor. Don’t trust it again until it’s made noise on command.

Kill your own DNS tonight, on purpose, before something else does it without asking.

THE TIP. The board that would’ve caught mine sooner is the same one Desk of One ships with: one pane for what’s running, not what’s supposed to be. If it’s not visible in one place, you don’t know it’s down until a customer tells you. Desk of One

DEAD AIR · FRONT DESK / GUEST REGISTER, ROOM 4

The paper register at the County Line Motor Lodge has logged every check-in by hand since 1968, same ruled ledger, same chained ballpoint at the counter, guests filling in their own plate numbers because the county never paid for a second clerk. The line for Room 4, night of March 3rd, 1979, is signed D. Purcell, Knoxville, plate in a seven-character format the state didn’t start stamping until 1994, checkout logged at 11:52 PM, check-in on the same line, same ink, same hand, at 12:01 AM. The clerk who witnessed the signature, Arlene Suggs, retired in 2003. She remembers the car, dark green, engine running the whole ten minutes he was inside, and not the plate. The guest wrote that part himself; she only signed under it. The county historical society keeps a postcard file on the old motor-court strip, one card mailed in every year since 1968 by whoever was running the place. Eleven of them show the County Line. The door to Room 4 is closed in all eleven.

Reply and tell me what your test found. I want to hear about the alert that went silent exactly when you needed it loud. ~ J.D.