Message Queues and Pub/Sub
Decouple services with queues and pub/sub, understand at-least-once delivery, make consumers idempotent, and apply backpressure and dead-letter queues.
intermediate40 min#queues#pub-sub#idempotency#backpressure#delivery-semantics
Prerequisites: A Framework for System Design, Stacks and Queues
Why it matters
When service A calls service B directly, A must wait for B, fails when B is down, and is overwhelmed if B is slow. Putting a message queue between them breaks that coupling: A drops a message and moves on, B processes at its own pace, and a traffic spike becomes a longer queue rather than an outage. Queues power email sending, image processing, payments workflows, and event streams, and interviewers use them to test whether you understand failure.
Intuition
A queue is a post office box. The sender drops a letter and leaves; the receiver collects it later. If the receiver is on holiday the letters pile up and nothing is lost. The interesting questions are all about failure. What if the receiver picks up a letter, starts reading, and falls asleep? Was it handled or not? The answer shapes the whole design.
How it works
Queue (point to point). Producers append messages; each message is delivered to one consumer among a group of competing workers. Use it for work distribution: "resize this image".
Pub/sub (fan-out). A producer publishes to a topic and every subscriber group gets its own copy. Use it for events: "order placed" is interesting to billing, shipping, and analytics independently. Many systems combine both: each subscriber group is itself a queue with competing workers.
Delivery semantics. A message can be lost or duplicated whenever the consumer, broker, or network fails between "work done" and "acknowledged".
- At-most-once: acknowledge first, then work. A crash loses the message. No duplicates.
- At-least-once: work first, then acknowledge. A crash before the ack means the message is delivered again. No loss, but duplicates happen. This is the usual default.
- Exactly-once: in practice, at-least-once delivery plus consumers that make duplicates harmless (called effectively-once). True end-to-end exactly-once across independent systems is not something a broker can promise alone.
Visibility timeout. When a worker receives a message, the broker hides it for a period. If the worker acks in time, the message is deleted. If not, the message reappears and another worker gets it. That is exactly how at-least-once works.
Idempotency. An operation is idempotent if doing it twice has the same effect as doing it once. "Set balance to 50" is idempotent; "add 10 to balance" is not. Make consumers idempotent by recording processed message ids (or a business key) and skipping repeats, ideally in the same transaction as the side effect.
Backpressure. If producers outrun consumers, an unbounded queue grows until memory or disk is exhausted, and latency grows with it. Defenses: bound the queue and reject or slow producers when full, drop low-value messages, scale out consumers, and monitor queue depth and message age.
Dead-letter queue (DLQ). A poison message that always fails would loop forever. After a maximum number of attempts, move it to a DLQ where it can be inspected without blocking healthy traffic.
Ordering. A single queue partition can preserve order, but competing consumers and retries break it. If order matters per entity (all events for one order), route by that key to one partition.
producers --> [ bounded queue ] --> consumer 1
| ^ consumer 2 (competing workers)
full? reject | redeliver after timeout
+-- no ack
too many attempts --> [ dead-letter queue ]
The next diagram follows message a from the code below. The worker does the job but its ack is lost, so the broker redelivers it and the idempotent consumer skips the duplicate.
| Operation | Time | Notes |
|---|---|---|
| Publish | O(1) | Append; rejected when a bounded queue is full |
| Receive | O(1) typical | Hide message for the visibility timeout |
| Ack | O(1) | Delete by id |
| Redelivery | after timeout | Source of at-least-once duplicates |
Code: an in-memory queue with redelivery
This simulation uses a fake clock. It shows backpressure, a lost ack causing a redelivery, a consumer crash, an idempotent handler skipping the duplicate, and a poison message ending in the DLQ.
class Clock:
def __init__(self):
self.now = 0
class Queue:
def __init__(self, capacity, visibility, max_attempts, clock):
self.capacity, self.visibility = capacity, visibility
self.max_attempts, self.clock = max_attempts, clock
self.items, self.dlq = [], []
def publish(self, mid, body):
if len(self.items) >= self.capacity:
return False # backpressure: tell the producer to slow down
self.items.append({"id": mid, "body": body, "attempts": 0, "visible_at": 0})
return True
def receive(self):
for m in list(self.items):
if m["visible_at"] > self.clock.now:
continue
if m["attempts"] >= self.max_attempts:
self.items.remove(m)
self.dlq.append(m["id"])
continue
m["attempts"] += 1
m["visible_at"] = self.clock.now + self.visibility
return m
return None
def ack(self, mid):
self.items = [m for m in self.items if m["id"] != mid]
clock = Clock()
q = Queue(capacity=3, visibility=30, max_attempts=2, clock=clock)
for mid, amount in [("a", 10), ("b", 20), ("c", 5), ("d", 99)]:
print("publish", mid, "->", "ok" if q.publish(mid, amount) else "rejected (queue full)")
total, processed = 0, set()
def handle(msg, ack=True):
global total
if msg["id"] in processed:
print("skip duplicate", msg["id"])
else:
total += msg["body"]
processed.add(msg["id"])
print("processed", msg["id"], "total =", total)
if ack:
q.ack(msg["id"])
handle(q.receive(), ack=False) # a: work done, ack lost
handle(q.receive()) # b: normal
c = q.receive()
print("crash while handling", c["id"]) # c: worker dies, no work, no ack
print("idle:", q.receive())
clock.now = 31
handle(q.receive()) # a redelivered -> duplicate skipped
handle(q.receive()) # c redelivered -> processed
print("queue empty:", q.receive() is None, "| final total:", total)
q.publish("x", 1)
for _ in range(3):
m = q.receive()
print("poison attempt:", m["attempts"] if m else None)
clock.now += 31
print("dead letters:", q.dlq)class Clock { now = 0; }
class Queue {
constructor(capacity, visibility, maxAttempts, clock) {
Object.assign(this, { capacity, visibility, maxAttempts, clock });
this.items = [];
this.dlq = [];
}
publish(id, body) {
if (this.items.length >= this.capacity) return false; // backpressure
this.items.push({ id, body, attempts: 0, visibleAt: 0 });
return true;
}
receive() {
for (const m of [...this.items]) {
if (m.visibleAt > this.clock.now) continue;
if (m.attempts >= this.maxAttempts) {
this.items = this.items.filter((x) => x !== m);
this.dlq.push(m.id);
continue;
}
m.attempts += 1;
m.visibleAt = this.clock.now + this.visibility;
return m;
}
return null;
}
ack(id) { this.items = this.items.filter((m) => m.id !== id); }
}
const clock = new Clock();
const q = new Queue(3, 30, 2, clock);
for (const [id, amount] of [["a", 10], ["b", 20], ["c", 5], ["d", 99]]) {
console.log("publish", id, "->", q.publish(id, amount) ? "ok" : "rejected (queue full)");
}
let total = 0;
const processed = new Set();
function handle(msg, ack = true) {
if (processed.has(msg.id)) {
console.log("skip duplicate", msg.id);
} else {
total += msg.body;
processed.add(msg.id);
console.log("processed", msg.id, "total =", total);
}
if (ack) q.ack(msg.id);
}
handle(q.receive(), false); // a: work done, ack lost
handle(q.receive()); // b: normal
const c = q.receive();
console.log("crash while handling", c.id); // c: worker dies, no work, no ack
console.log("idle:", q.receive() === null ? "None" : "message");
clock.now = 31;
handle(q.receive()); // a redelivered -> duplicate skipped
handle(q.receive()); // c redelivered -> processed
console.log("queue empty:", q.receive() === null ? "True" : "False", "| final total:", total);
q.publish("x", 1);
for (let i = 0; i < 3; i++) {
const m = q.receive();
console.log("poison attempt:", m ? m.attempts : "None");
clock.now += 31;
}
console.log("dead letters:", "['" + q.dlq.join("', '") + "']");Both versions print:
publish a -> ok
publish b -> ok
publish c -> ok
publish d -> rejected (queue full)
processed a total = 10
processed b total = 30
crash while handling c
idle: None
skip duplicate a
processed c total = 35
queue empty: True | final total: 35
poison attempt: 1
poison attempt: 2
poison attempt: None
dead letters: ['x']
Message a was delivered twice but its effect was applied once, because the consumer remembered the id. The total is 35, the sum of the three accepted messages. Message d was rejected up front, which is backpressure in its simplest form: the producer learns immediately and can retry later or shed load.
A poison message that always fails must not loop forever. Here is the path the code's message x takes with a maximum of 2 attempts.
Common pitfalls
- Non-idempotent consumers. With at-least-once delivery, duplicates are certain eventually. Charging a card twice is the classic failure.
- Unbounded queues. They hide overload until memory or disk runs out and message age becomes unacceptable. Bound them and alert on depth and oldest-message age.
- Visibility timeout shorter than the work. The message reappears while the first worker is still running, causing concurrent duplicates.
- No dead-letter queue. One poison message retried forever can starve everything behind it.
- Assuming global order. Retries and parallel consumers reorder messages. Partition by key if order matters.
- Using a queue for request-response. If the user needs the answer now, a synchronous call is simpler. Queues suit work that can finish later.
Practice
Make an operation idempotent easy
Hint
What can you store so a repeat is recognized?
Show solution
For "send welcome email", store the message id or a key like welcome:user42 in a table with a unique constraint before sending. A duplicate message hits the constraint and is skipped. For "add 10 to balance", pass a transaction id and apply it only if that id has not been recorded.
Queue or pub/sub? easy
Hint
How many different consumers need each message?
Show solution
Resizing an uploaded image is one job for any one worker: a queue. "User signed up" must reach email, analytics, and fraud checks separately: pub/sub with one subscription per consumer group.
Handle a traffic spike medium
Hint
List what you can do at the producer, the queue, and the consumers.
Show solution
Bound the queue and return a clear rejection or 429 when full; autoscale consumers on queue depth; drop or sample low-value messages; and prioritize critical ones in a separate queue. Monitor oldest-message age because it measures user-visible delay better than depth alone.
Pick a visibility timeout medium
Hint
What happens if it is too short, and what if it is too long?
Show solution
Set it comfortably above the typical worst-case processing time. Too short causes duplicate concurrent processing; too long delays recovery after a crash. For long jobs, have the worker extend the timeout periodically (a heartbeat) instead of picking a huge fixed value.
Interview questions
- Why use a message queue? To decouple services, absorb bursts, retry failed work, and let producer and consumer scale independently.
- At-least-once or at-most-once? At-least-once with idempotent consumers is the usual choice because losing work is worse than repeating it, as long as repeats are safe.
- Can you get exactly-once? Only effectively, by combining at-least-once delivery with idempotent processing or transactional writes of the result and the offset.
- What is backpressure? Signaling producers to slow down (or rejecting work) when consumers cannot keep up, so queues do not grow without bound.
- How do you handle poison messages? Cap retries and send the message to a dead-letter queue for inspection and replay.
Interviewers look for: immediate mention of duplicates and idempotency, awareness of ordering limits, and a plan for overload and poison messages.
Quick check
A worker processes a message and crashes just before sending the ack. With at-least-once delivery, what happens?
Finished this lesson?
Further resources
- Message queue (Wikipedia) (article)
- Publish-subscribe pattern (Wikipedia) (article)
- Idempotence (Wikipedia) (article)