Message Queue System Design Template

Use SQS when you want a managed service with minimal operational overhead and do not need strict ordering or replay. Use RabbitMQ when you need flexible routing (topic exchanges, headers-based routing) and are comfortable managing infrastructure. Use Kafka when you need high throughput, message replay, and long retention. Kafka is an event log, not a traditional queue. It stores messages permanently and consumers track their own offsets. For most product teams, SQS is the right starting point. Record the technology decision in an [architecture decision record](/templates/architecture-decision-record-template).

Design consumers to be idempotent. The simplest approach: store each `messageId` in a deduplication table (or Redis set with TTL) and skip messages you have already seen. For database writes, use upsert operations with the `messageId` as a unique constraint. The goal is not to prevent duplicate delivery (most brokers cannot guarantee that) but to make duplicate processing harmless.

It depends on your consumer throughput and latency SLA. Calculate: if each consumer processes 100 messages/second and you have 5 consumers, your drain rate is 500 messages/second. If your [SLA](/glossary/service-level-agreement-sla) requires messages to be processed within 60 seconds, your alert threshold is 500 * 60 = 30,000 messages. Set a warning at 50% of that (15,000) and critical at 80% (24,000).

Use FIFO queues only when message order matters for correctness (e.g., processing balance updates for the same account). FIFO queues have lower throughput limits (300-3,000 messages/sec on SQS vs effectively unlimited for standard queues). Most use cases do not require strict ordering. If only a subset of messages needs ordering, use partition keys (message group IDs) to order within a group while allowing parallelism across groups. ---