DevPath
Tracks

Message Queues and Pub/Sub

Decouple services with queues and pub/sub, understand at-least-once delivery, make consumers idempotent, and apply backpressure and dead-letter queues.

intermediate40 min#queues#pub-sub#idempotency#backpressure#delivery-semantics

Prerequisites: A Framework for System Design, Stacks and Queues

Why it matters

When service A calls service B directly, A must wait for B, fails when B is down, and is overwhelmed if B is slow. Putting a message queue between them breaks that coupling: A drops a message and moves on, B processes at its own pace, and a traffic spike becomes a longer queue rather than an outage. Queues power email sending, image processing, payments workflows, and event streams, and interviewers use them to test whether you understand failure.

Intuition

A queue is a post office box. The sender drops a letter and leaves; the receiver collects it later. If the receiver is on holiday the letters pile up and nothing is lost. The interesting questions are all about failure. What if the receiver picks up a letter, starts reading, and falls asleep? Was it handled or not? The answer shapes the whole design.

How it works

Queue (point to point). Producers append messages; each message is delivered to one consumer among a group of competing workers. Use it for work distribution: "resize this image".

Pub/sub (fan-out). A producer publishes to a topic and every subscriber group gets its own copy. Use it for events: "order placed" is interesting to billing, shipping, and analytics independently. Many systems combine both: each subscriber group is itself a queue with competing workers.

Delivery semantics. A message can be lost or duplicated whenever the consumer, broker, or network fails between "work done" and "acknowledged".

  • At-most-once: acknowledge first, then work. A crash loses the message. No duplicates.
  • At-least-once: work first, then acknowledge. A crash before the ack means the message is delivered again. No loss, but duplicates happen. This is the usual default.
  • Exactly-once: in practice, at-least-once delivery plus consumers that make duplicates harmless (called effectively-once). True end-to-end exactly-once across independent systems is not something a broker can promise alone.

Visibility timeout. When a worker receives a message, the broker hides it for a period. If the worker acks in time, the message is deleted. If not, the message reappears and another worker gets it. That is exactly how at-least-once works.

Idempotency. An operation is idempotent if doing it twice has the same effect as doing it once. "Set balance to 50" is idempotent; "add 10 to balance" is not. Make consumers idempotent by recording processed message ids (or a business key) and skipping repeats, ideally in the same transaction as the side effect.

Backpressure. If producers outrun consumers, an unbounded queue grows until memory or disk is exhausted, and latency grows with it. Defenses: bound the queue and reject or slow producers when full, drop low-value messages, scale out consumers, and monitor queue depth and message age.

Dead-letter queue (DLQ). A poison message that always fails would loop forever. After a maximum number of attempts, move it to a DLQ where it can be inspected without blocking healthy traffic.

Ordering. A single queue partition can preserve order, but competing consumers and retries break it. If order matters per entity (all events for one order), route by that key to one partition.

 producers --> [ bounded queue ] --> consumer 1
                  |    ^            consumer 2   (competing workers)
        full? reject  | redeliver after timeout
                      +-- no ack
        too many attempts --> [ dead-letter queue ]
Producer A
Producer B
Bounded queuerejects when full (backpressure)
Consumer 1
Consumer 2
Consumer 3
Producers and consumers never talk to each other. Each message goes to just one of the competing consumers, and you can add consumers without touching the producers.

The next diagram follows message a from the code below. The worker does the job but its ack is lost, so the broker redelivers it and the idempotent consumer skips the duplicate.

ProducerQueueWorkerpublish a (amount 10)deliver a, hide for 30 sack a (lost on the way)timeout passes: deliver a againseen id a, skip work, ack a
Visibility timeout in action: no ack within 30 seconds means the message reappears. At-least-once delivery gave two deliveries, and the processed-id check made the effect happen once.
OperationTimeNotes
PublishO(1)Append; rejected when a bounded queue is full
ReceiveO(1) typicalHide message for the visibility timeout
AckO(1)Delete by id
Redeliveryafter timeoutSource of at-least-once duplicates

Code: an in-memory queue with redelivery

This simulation uses a fake clock. It shows backpressure, a lost ack causing a redelivery, a consumer crash, an idempotent handler skipping the duplicate, and a poison message ending in the DLQ.

class Clock:
    def __init__(self):
        self.now = 0
 
class Queue:
    def __init__(self, capacity, visibility, max_attempts, clock):
        self.capacity, self.visibility = capacity, visibility
        self.max_attempts, self.clock = max_attempts, clock
        self.items, self.dlq = [], []
 
    def publish(self, mid, body):
        if len(self.items) >= self.capacity:
            return False  # backpressure: tell the producer to slow down
        self.items.append({"id": mid, "body": body, "attempts": 0, "visible_at": 0})
        return True
 
    def receive(self):
        for m in list(self.items):
            if m["visible_at"] > self.clock.now:
                continue
            if m["attempts"] >= self.max_attempts:
                self.items.remove(m)
                self.dlq.append(m["id"])
                continue
            m["attempts"] += 1
            m["visible_at"] = self.clock.now + self.visibility
            return m
        return None
 
    def ack(self, mid):
        self.items = [m for m in self.items if m["id"] != mid]
 
clock = Clock()
q = Queue(capacity=3, visibility=30, max_attempts=2, clock=clock)
for mid, amount in [("a", 10), ("b", 20), ("c", 5), ("d", 99)]:
    print("publish", mid, "->", "ok" if q.publish(mid, amount) else "rejected (queue full)")
 
total, processed = 0, set()
 
def handle(msg, ack=True):
    global total
    if msg["id"] in processed:
        print("skip duplicate", msg["id"])
    else:
        total += msg["body"]
        processed.add(msg["id"])
        print("processed", msg["id"], "total =", total)
    if ack:
        q.ack(msg["id"])
 
handle(q.receive(), ack=False)        # a: work done, ack lost
handle(q.receive())                   # b: normal
c = q.receive()
print("crash while handling", c["id"])  # c: worker dies, no work, no ack
print("idle:", q.receive())
clock.now = 31
handle(q.receive())                   # a redelivered -> duplicate skipped
handle(q.receive())                   # c redelivered -> processed
print("queue empty:", q.receive() is None, "| final total:", total)
 
q.publish("x", 1)
for _ in range(3):
    m = q.receive()
    print("poison attempt:", m["attempts"] if m else None)
    clock.now += 31
print("dead letters:", q.dlq)

Both versions print:

publish a -> ok
publish b -> ok
publish c -> ok
publish d -> rejected (queue full)
processed a total = 10
processed b total = 30
crash while handling c
idle: None
skip duplicate a
processed c total = 35
queue empty: True | final total: 35
poison attempt: 1
poison attempt: 2
poison attempt: None
dead letters: ['x']

Message a was delivered twice but its effect was applied once, because the consumer remembered the id. The total is 35, the sum of the three accepted messages. Message d was rejected up front, which is backpressure in its simplest form: the producer learns immediately and can retry later or shed load.

A poison message that always fails must not loop forever. Here is the path the code's message x takes with a maximum of 2 attempts.

Queuemessage x
Attempt 1fails or times out
Attempt 2fails or times out
Dead-letter queueset aside for inspection
After the maximum attempts, the next receive moves the message to the DLQ instead of delivering it a third time, so healthy messages behind it keep flowing.

Common pitfalls

  • Non-idempotent consumers. With at-least-once delivery, duplicates are certain eventually. Charging a card twice is the classic failure.
  • Unbounded queues. They hide overload until memory or disk runs out and message age becomes unacceptable. Bound them and alert on depth and oldest-message age.
  • Visibility timeout shorter than the work. The message reappears while the first worker is still running, causing concurrent duplicates.
  • No dead-letter queue. One poison message retried forever can starve everything behind it.
  • Assuming global order. Retries and parallel consumers reorder messages. Partition by key if order matters.
  • Using a queue for request-response. If the user needs the answer now, a synchronous call is simpler. Queues suit work that can finish later.

Practice

Make an operation idempotent easy

Hint

What can you store so a repeat is recognized?

Show solution

For "send welcome email", store the message id or a key like welcome:user42 in a table with a unique constraint before sending. A duplicate message hits the constraint and is skipped. For "add 10 to balance", pass a transaction id and apply it only if that id has not been recorded.

Queue or pub/sub? easy

Hint

How many different consumers need each message?

Show solution

Resizing an uploaded image is one job for any one worker: a queue. "User signed up" must reach email, analytics, and fraud checks separately: pub/sub with one subscription per consumer group.

Handle a traffic spike medium

Hint

List what you can do at the producer, the queue, and the consumers.

Show solution

Bound the queue and return a clear rejection or 429 when full; autoscale consumers on queue depth; drop or sample low-value messages; and prioritize critical ones in a separate queue. Monitor oldest-message age because it measures user-visible delay better than depth alone.

Pick a visibility timeout medium

Hint

What happens if it is too short, and what if it is too long?

Show solution

Set it comfortably above the typical worst-case processing time. Too short causes duplicate concurrent processing; too long delays recovery after a crash. For long jobs, have the worker extend the timeout periodically (a heartbeat) instead of picking a huge fixed value.

Interview questions

  1. Why use a message queue? To decouple services, absorb bursts, retry failed work, and let producer and consumer scale independently.
  2. At-least-once or at-most-once? At-least-once with idempotent consumers is the usual choice because losing work is worse than repeating it, as long as repeats are safe.
  3. Can you get exactly-once? Only effectively, by combining at-least-once delivery with idempotent processing or transactional writes of the result and the offset.
  4. What is backpressure? Signaling producers to slow down (or rejecting work) when consumers cannot keep up, so queues do not grow without bound.
  5. How do you handle poison messages? Cap retries and send the message to a dead-letter queue for inspection and replay.

Interviewers look for: immediate mention of duplicates and idempotency, awareness of ordering limits, and a plan for overload and poison messages.

Quick check

A worker processes a message and crashes just before sending the ack. With at-least-once delivery, what happens?

Finished this lesson?

Further resources

Related lessons