Make the broadcast survive a restart

The first version of broadcasts in SendDock worked on a good day. You hit send, a goroutine looped over your subscribers, and emails went out. The problem was every bad day: a deploy, a crash, an OOM — anything that ended the process mid-send — and the broadcast was simply gone. Some recipients got the email, some didn't, and there was no record of which. For a tool whose entire pitch is own your sending, "we lost half your campaign" is not an acceptable failure mode.

So I rebuilt it. The design goal was narrow and strict: a broadcast must survive the process dying at any moment — no recipient sent to twice, none silently dropped.

Durability means it lives in the database

The single-goroutine version held all its state in memory, which is exactly why a restart erased it. The fix starts by refusing to keep anything important in memory. Every recipient of a broadcast becomes a row in a broadcast_jobs table, each with a status: pending, sending, retry, sent, failed. The queue is the table. If the process dies, the queue is still sitting in Postgres, exactly where it was.

Draining the queue without stepping on itself

Five worker goroutines drain the table concurrently. The trick that makes concurrent workers safe over a shared SQL table is one clause:

SELECT ... FROM broadcast_jobs
WHERE status = 'pending'
FOR UPDATE SKIP LOCKED

FOR UPDATE locks the rows a worker claims; SKIP LOCKED tells the other workers to walk right past locked rows instead of blocking on them. No two workers ever grab the same recipient, and no worker sits idle waiting for a lock. It's the simplest correct way to turn a Postgres table into a work queue, and it needs no extra infrastructure — no broker, no separate queue service.

Retries that respect the difference between "later" and "never"

Not every failure means the same thing. A transient SMTP error — a 4xx, a network blip, a DNS hiccup — means try again later. A 5xx bounce means this address is dead; stop. Treating those the same is how you either give up too early or hammer a dead mailbox forever.

So transient failures reschedule with exponential backoff — 30s, 2m, 8m, 30m, 1h — capped at five attempts before the recipient is marked failed. A 5xx is never retried: the recipient is tagged bounced and added to the suppression list immediately, so the next campaign doesn't even try.

The part that actually earns the title

Here's the restart behavior. When the backend boots, any job left in sending — meaning a worker had claimed it but the process died before it finished — is reset to retry. Sending resumes from exactly where it stopped. Because claiming and sending are separate states, a job that truly went out is already marked sent and won't be touched; a job that was mid-flight goes back in line. No double-sends, no drops.

There was a subtle follow-on bug worth mentioning, because it shows how these changes ripple. With the new queue, Broadcast() returns in milliseconds — it only enqueues. The campaign worker used to call it and immediately mark the campaign sent with the upfront recipient count, so campaigns flashed to 213/0 sent before a single email had left. The fix was to link each campaign to its broadcast with a broadcast_id and let the real counts cascade back as the queue drains. The campaign stays sending until the work is actually done — because now "done" is a real, observable thing.

The lesson

"Durable" isn't a feature you bolt on; it's a property you get by refusing to keep important state where it can evaporate. The moment the queue moved from a goroutine's stack into a Postgres table, most of the reliability fell out for free — restart recovery, concurrency, retries, visibility. The hard part wasn't the queue. It was giving up the comfortable illusion that the process would stay alive.