The server that restarted every five hours

The worst bugs don't crash loudly. For a few releases, SendDock's server died quietly — no panic, no stack trace, just a container that kept getting replaced while users saw intermittent 502s. Every time I fixed it, it came back wearing a different disguise. Here's the whole tour, because each disguise taught me something about health checks.

Disguise #1: the health check that couldn't reach itself

On some VPS and Swarm hosts with net.ipv6.bindv6only=1, Go's default :8080 listener bound to IPv6 only. The container's HEALTHCHECK was hitting 127.0.0.1:8080 — IPv4. Every probe failed. After three retries the orchestrator marked the container unhealthy, killed it, started a new one, and the new one did exactly the same thing. A 60–90 second crash-replace loop with no panic and no error log — just intermittent Bad Gateway from the proxy while a "healthy" service quietly rotated underneath it.

The fix was two lines: bind 0.0.0.0:8080 explicitly, and point the health check at localhost (which resolves to both 127.0.0.1 and ::1). The lesson underneath was bigger: a health check that can't reach the thing it's checking is worse than no health check — it actively destroys a working server.

Disguise #2: the health check that lied

Next, /health returned {"status":"ok"} without ever touching the database. So when Postgres went zombie — an idle-in-transaction backlog, an OOM-killed worker, a network partition — the Go connection pool kept its old TCP sockets open and never noticed. Docker kept routing traffic to a container reporting perfect health; the queries behind it hung; the proxy returned 502. Restarting the app didn't help, because Postgres was the broken side.

The obvious fix — actually ping the database in /health — is where I planted the next bug.

Disguise #3: the fix that became the bug

I made /health run a synchronous PingContext with a 2-second timeout. Reasonable — until Postgres took longer than two seconds for a perfectly transient reason: a GC pause, a slow disk, a noisy neighbor on a small VPS. One slow blip, five failed probes over three minutes, container killed. I'd turned a momentary hiccup into a crash.

The real fix was to stop doing work inside the health check. A background goroutine now pings Postgres every 10s and stores the last-success timestamp atomically. /health just reads that atomic and returns 503 only if no ping has succeeded in the last 60 seconds. Single blips get absorbed; a sustained outage still surfaces. A health check should observe, not perform.

Disguise #4: every five hours, on the dot

This is the one people remember. After all of the above, the container settled into a new rhythm — it restarted roughly every five hours. Not random. Clockwork.

The rate limiter was the culprit, and it was strangling the health check. My Redis Increment did an INCR followed unconditionally by EXPIRE, which reset the key's one-minute TTL on every call. The Docker health check hit /health every 30 seconds, so the counter's key never got a chance to expire. Instead of counting ~2 requests per minute and resetting, it accumulated forever. Around 600 hits — about five hours at two per minute — it crossed the 600-req/min threshold, and /health started returning 429. wget -q … || exit 1 turned that into a failed probe, and the orchestrator did what orchestrators do.

Two fixes, both worth stating as rules:

• The counter now uses an atomic Lua script that sets the TTL only when the key is first created (count == 1). That is what a fixed-window counter actually is. INCR plus an unconditional EXPIRE is a sliding reset that never expires under steady traffic.
• /health no longer passes through the rate limiter, CORS, or body-size middleware at all. It's mounted on a separate root mux that bypasses the whole pipeline. Your health check must never be subject to the throttles you built for clients — it is not a client.

What the whole saga taught me

Four bugs, one theme: the health check kept being the thing that killed the server. That is not a coincidence. A health check sits in the most dangerous spot in the system — it has the authority to end a process, and it usually runs with the least scrutiny. When it's wrong it doesn't fail safe; it fails by executing a healthy server.

So here are the rules I now hold, paid for one restart loop at a time:

• It must bypass every client-facing middleware — no rate limits, no CORS, no auth.
• It must observe state cheaply, never do expensive work on the request path.
• It must actually check the dependency that matters — but tolerate transient blips.
• It must be reachable at the exact address the prober uses.

None of these are clever. All four cost me a production incident to learn. The quiet bugs are the expensive ones.