Background Workers & Job Processing

Request handlers are easy to reason about: they start, they finish, the client is waiting. Background work is different. A job might take seconds or hours. The caller is long gone. The process can restart. You need ownership of the lifecycle, durability, idempotency, bounded concurrency, and a way to observe what actually happened.

This post builds directly on concurrency patterns, synchronization, context, and resiliency. We'll turn "just spin a goroutine" into something you can run in production.

Own the loop, own shutdown

The simplest reliable worker is a goroutine that owns its work loop and participates in graceful shutdown. Don't fire-and-forget.

type Worker struct {
    jobs chan Job
    wg   sync.WaitGroup
    stop chan struct{}
}

func (w *Worker) Run(ctx context.Context) {
    for {
        select {
        case job := <-w.jobs:
            w.wg.Add(1)
            go func(j Job) {
                defer w.wg.Done()
                w.process(ctx, j)
            }(job)
        case <-ctx.Done():
            return
        }
    }
}

func (w *Worker) Stop() {
    close(w.stop)
    w.wg.Wait()
}

Key points: the worker loop is the owner. Stop waits for in-flight jobs. Use the incoming context for cancellation inside process. This is the foundation before you add queues or retries.

Queues: in-memory vs durable

For low-stakes work an in-memory channel is fine. For anything that must survive restarts you need a durable queue (Redis streams, Postgres, SQS, etc.). The shape is the same:

type Queue interface {
    Enqueue(ctx context.Context, job Job) error
    Dequeue(ctx context.Context) (Job, error) // blocks or returns ctx err
    Ack(ctx context.Context, job Job) error
    Nack(ctx context.Context, job Job) error
}

The worker pulls, processes, then acks. If the process dies before ack, the job becomes visible again after a visibility timeout. That's the contract you must honor.

Idempotency is non-negotiable

Jobs will be delivered at-least-once. Make them safe to run twice.

func (w *Worker) process(ctx context.Context, j Job) error {
    if w.seen(ctx, j.ID) { // dedup by stable key
        return nil
    }
    if err := w.doWork(ctx, j); err != nil {
        return err
    }
    w.markSeen(ctx, j.ID)
    return nil
}

Use a stable business key (order ID, user ID + action, etc.), not the queue message ID. Store the "processed" marker with the same durability as your side effects.

Bounded concurrency + context everywhere

Never let background work consume the whole machine. Use a weighted semaphore (from post 6) or a worker pool with a fixed size.

sem := semaphore.NewWeighted(8)

func (w *Worker) process(ctx context.Context, j Job) {
    if err := sem.Acquire(ctx, 1); err != nil {
        return // context cancelled while waiting
    }
    defer sem.Release(1)

    ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
    defer cancel()

    // actual work here — every subcall must respect ctx
}

Retries, backoff, and the dead-letter queue

Use the same retry discipline from post 9, but inside the worker. After a configured number of attempts, move the job to a dead-letter queue instead of dropping it or looping forever.

for attempt := 0; attempt < maxAttempts; attempt++ {
    if err := doWithTimeout(ctx, j); err == nil {
        queue.Ack(ctx, j)
        return
    }
    if !isRetryable(err) || attempt == maxAttempts-1 {
        queue.MoveToDLQ(ctx, j)
        return
    }
    sleepWithJitter(ctx, backoff(attempt))
}

Record the attempt count either in the job payload or as queue metadata. Never retry non-idempotent work without a stable key that makes the second try a no-op.

Observability: you must be able to answer "what happened?"

Every job should carry a correlation ID. Log at start, on retry, on success, on permanent failure. Emit metrics for latency, attempts, and DLQ rate. A simple pattern:

slog.Info("job start", "id", j.ID, "type", j.Type, "attempt", attempt, "corr", j.CorrID)
defer func() {
    if rec := recover(); rec != nil {
        slog.Error("job panic", "id", j.ID, "panic", rec)
        queue.MoveToDLQ(ctx, j)
    }
}()

Expose a small admin endpoint or metric that shows queue depth and oldest unacked job age — this is your canary for stuck work.

Putting it together

A production worker usually looks like:

func main() {
    ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
    defer cancel()

    q := durable.NewQueue(...)
    w := &Worker{jobs: make(chan Job, 100), q: q}

    go w.Run(ctx)

    <-ctx.Done()
    w.Stop() // waits for current jobs
}

Combine with the layering from post 9: rate limit admission if you have a hot queue, bulkhead the worker pool, circuit-break external calls inside jobs, and always propagate deadlines.

The takeaway