When building backend systems, I often need to move work from one service to another without making the caller wait for the entire operation. Sending an email, processing an image, running a scraper, generating a report, processing payments, or consuming events can all be handled asynchronously.
That is where queues and message brokers come in. Redis, RabbitMQ, and Kafka can all move data between processes, but they are designed around different models. Understanding those differences is more useful than memorizing which tool is supposed to be the fastest.
What is a queue?
At the simplest level, a queue stores messages until a consumer is ready to process them. A producer creates a message, the queue holds it, and a consumer reads and processes it.
Producer Queue Consumer
| | |
| ---- message -----------> | |
| | |
| | ---- message ---------> |
| | |
| | <---- acknowledgement - |The important part is that the producer and consumer do not have to operate at exactly the same time or speed. The queue provides a buffer between them.
Why use a queue?
Without a queue, a service commonly has to call another service synchronously and wait for the response. That creates tighter coupling and means temporary slowdowns in one component can directly affect another.
A basic asynchronous architecture
+----------------+
| API Server |
+-------+--------+
|
| publish job
v
+----------------+
| Queue / Broker |
+-------+--------+
|
+------------+------------+
| | |
v v v
+---------+ +---------+ +---------+
| Worker 1| | Worker 2| | Worker 3|
+---------+ +---------+ +---------+For example, an API can accept an image upload and immediately create a processing job. Workers can consume those jobs independently. The API does not need to keep the HTTP request open while the image is processed.
Queue semantics matter
The word queue alone does not describe how messages behave. When choosing a system, I look at delivery semantics, ordering, persistence, retries, acknowledgements, retention, consumer behavior, and scaling.
Acknowledgements
A consumer may need to explicitly acknowledge successful processing. If the worker crashes before acknowledging the message, the system can make the message available again depending on the broker and its configuration.
Message available
|
v
Consumer gets message
|
+------ processing succeeds ------> ACK
|
+------ consumer crashes ---------> message can be redeliveredAt-most-once and at-least-once delivery
At-most-once delivery means a message is delivered zero or one time. A message can be lost, but duplicate processing is avoided by the delivery model.
At-least-once delivery means a message is delivered one or more times. This is useful when losing work is worse than processing it twice, but consumers need to handle duplicates safely.
In practice, I design consumers with idempotency in mind. If processing the same message twice can create two payments, two orders, or two side effects, the consumer needs a strategy to detect or safely handle duplicates.
Ordering
Some workloads require messages to be processed in order. Others only care that every message is eventually processed. Ordering requirements can affect partitioning, concurrency, and the choice of messaging system.
Queue versus event stream
One of the most important distinctions is whether I want work to be consumed from a queue or events to remain available as a stream that consumers can read independently.
Traditional work queue
Producer ---> [ A ][ B ][ C ][ D ] ---> Worker
Messages are generally treated as work waiting to be consumed.
Event stream
Producer ---> [ A ][ B ][ C ][ D ][ E ] ---> Consumer A
|
+-------> Consumer B
|
+-------> Consumer C
Multiple consumers can maintain their own position in the stream.This distinction is one reason Kafka is often discussed differently from traditional message queues. Kafka is built around durable append-only logs and consumer positions, while systems such as RabbitMQ are commonly used for message routing and work distribution.
Redis as a queue
Redis is primarily an in-memory data store, but it provides several primitives that can be used for asynchronous processing. Depending on the problem, Redis lists, Pub/Sub, Streams, or other structures can be used.
Redis Lists
A simple list can represent a basic work queue. A producer pushes an item and a worker blocks waiting for an item.
Producer
|
| LPUSH
v
+-------------------+
| job3 | job2 | job1|
+-------------------+
^
|
BRPOP
|
WorkerThis is straightforward and useful for relatively simple background jobs. However, a production queue often needs more than push and pop operations, such as acknowledgements, consumer groups, replay, retention, and stronger delivery controls.
Redis Streams
Redis Streams provide a log-like data structure with message IDs, consumer groups, pending entries, and acknowledgement mechanisms. They are more appropriate than a basic Redis list when I need richer stream-processing behavior while keeping Redis in the architecture.
XADD stream * type email user_id 123
Redis Stream
+------------------------------------------------+
| 171...-0 | 171...-1 | 171...-2 | 171...-3 |
+------------------------------------------------+
| | | |
+----------+----------+----------+
|
Consumer Group
+------+------+
| |
Worker A Worker BRedis Streams can be a good fit when I already depend on Redis and need queue or stream functionality without introducing another infrastructure component.
Redis Pub/Sub is different
Redis Pub/Sub should not be treated as the same thing as a durable queue. A publisher sends a message to a channel and currently connected subscribers receive it. It is useful for transient notifications, but it is not the same model as a durable work queue.
Publisher
|
v
+-----------+
| Channel |
+-----+-----+
|
+---+---+
| |
v v
Sub A Sub B
If a subscriber is not connected when the message is published,
the Pub/Sub model does not provide the same durable replay behavior
as a persistent stream.Kafka
Kafka is designed around durable distributed logs. Producers append records to topics, topics are divided into partitions, and consumers track their position in those partitions.
Kafka Topic
+------------------------------------------------------+
| Partition 0 | A | B | C | D | E | F | G | |
+------------------------------------------------------+
| Partition 1 | H | I | J | K | L | M | N | |
+------------------------------------------------------+
| Partition 2 | O | P | Q | R | S | T | U | |
+------------------------------------------------------+
^ ^
| |
Consumer A Consumer B
offset = 4 offset = 6The important idea is that consuming a Kafka record does not mean the record immediately disappears from the topic. Records are retained according to the configured retention policy, and consumers track their offsets.
Kafka partitions
Partitions provide a way to distribute a topic across brokers and consumers. A consumer group can divide partitions among its consumers so that multiple workers process records concurrently.
Topic: orders
Partition 0 ---> Consumer 1
Partition 1 ---> Consumer 2
Partition 2 ---> Consumer 3
Partition 3 ---> Consumer 4Within a partition, records have an ordered sequence. If I need ordering for related events, I can design the partitioning strategy so those events are placed in the same partition.
RabbitMQ
RabbitMQ is a message broker focused heavily on message routing, queues, acknowledgements, and delivery semantics. Producers publish messages to exchanges, and exchanges route messages to queues.
Producer
|
v
+----------+
| Exchange |
+----+-----+
|
| routing
+--+--------+---------+
| | |
v v v
Queue A Queue B Queue C
| | |
v v v
Worker A Worker B Worker CThe exchange model is useful when I need flexible routing. Messages can be routed based on routing keys, patterns, or other exchange behavior instead of having producers directly choose a particular worker queue.
RabbitMQ acknowledgements
A consumer can acknowledge a message after successful processing. If the consumer fails before acknowledging it, RabbitMQ can requeue the message depending on the configuration and failure mode.
Redis versus RabbitMQ versus Kafka
These technologies overlap, but I would not consider them interchangeable in every architecture.
A practical comparison
Redis RabbitMQ Kafka
Primary model Data store Message broker Distributed log
Simple work queues Yes Yes Possible
Message routing Basic Strong Topic based
Consumer groups Streams Consumers Core concept
Durable event history Streams Queue based Core concept
Replay retained events Streams Limited model Core concept
Partitioned log No No Yes
Very simple setup Yes Moderate More involved
Caching + queue together Yes No NoWhere I would use Redis
For a small or medium-sized application, Redis can be a practical choice when the application already uses Redis for caching, rate limiting, sessions, or other data structures. Adding a simple queue or Redis Stream can avoid introducing another infrastructure component.
Where I would use RabbitMQ
I would consider RabbitMQ when the application is fundamentally about delivering jobs or messages to consumers and I need explicit routing and acknowledgement behavior.
Where I would use Kafka
I would consider Kafka when the problem looks more like event streaming than a simple job queue. The ability to retain records and let consumers independently track their positions changes how I can build the system.
A common mistake: choosing Kafka for every queue
Kafka is powerful, but that does not automatically make it the right choice for every background job. If my requirement is simply to put email jobs into a queue and have workers process them, a simpler queueing system may be easier to operate and reason about.
I try to start with the messaging semantics I need rather than the technology I already know. If I need a durable event log with independent consumers, Kafka becomes interesting. If I need broker-based work distribution and routing, RabbitMQ may fit better. If I already run Redis and need straightforward queue or stream capabilities, Redis may be enough.
Retries and failed messages
A queue becomes much more useful when failure is treated as a normal part of distributed systems. Network failures, temporary database errors, service crashes, malformed messages, and downstream outages can all cause processing failures.
+----------------+
| Queue |
+-------+--------+
|
v
+---------+
| Worker |
+----+----+
|
+------+------+
| |
success failure
| |
v v
ACK retry / retry queue
|
v
dead-letter queueFor retries, I avoid blindly retrying forever. A practical design can use a retry limit, backoff, and a dead-letter or failure-handling path for messages that repeatedly fail.
Idempotent consumers
At-least-once processing means duplicate delivery is possible. Because of that, I prefer consumers that are safe to execute more than once.
func processOrder(ctx context.Context, orderID string) error {
processed, err := store.AlreadyProcessed(ctx, orderID)
if err != nil {
return err
}
if processed {
return nil
}
if err := createOrderSideEffects(ctx, orderID); err != nil {
return err
}
return store.MarkProcessed(ctx, orderID)
}The exact implementation depends on the application. The important principle is that message delivery and business-side effects should be designed together.
Backpressure
Queues also provide a way to absorb differences between production and consumption rates. If producers create work faster than workers can process it, the queue grows.
Producer rate > Consumer rate
Producer ---> ---> ---> --->
|
v
+----------+
| QUEUE |
| growing |
+----------+
|
v
WorkerA growing queue is not automatically a failure. It becomes a capacity problem when the backlog grows continuously or when message age exceeds the application's acceptable processing delay.
Scaling workers
A common way to process more work is to add workers. The queue or broker distributes work according to its delivery model.
Queue
|
+---------+---------+
| | |
v v v
Worker 1 Worker 2 Worker 3
| | |
+---------+---------+
|
More throughputThis works only when the downstream resources can also handle the increased concurrency. Adding workers does not help if the database, external API, CPU, or another bottleneck becomes saturated.
Queue versus synchronous HTTP
Synchronous
Client ---> API ---> Service ---> Database
| |
+----------- waits ---------------+
Asynchronous
Client ---> API ---> Queue ---> Worker ---> Database
|
+------ response returned earlierI use asynchronous processing when the client does not need the final result immediately. If the operation must complete before the response can be returned, introducing a queue may add complexity without solving the actual problem.
A simple Go worker
The basic worker pattern is independent of the specific broker. The worker continuously receives work, processes it, and acknowledges successful completion.
func worker(ctx context.Context, jobs <-chan Job) {
for {
select {
case <-ctx.Done():
return
case job := <-jobs:
if err := process(ctx, job); err != nil {
// retry or move to failure handling
continue
}
}
}
}In a real implementation, the jobs channel would be connected to Redis, RabbitMQ, Kafka, or another messaging system, and the failure path would need explicit retry and acknowledgement behavior.
How I decide which one to use
The mental model I use
I think about Redis, RabbitMQ, and Kafka as tools with overlapping capabilities but different primary abstractions. Redis gives me powerful data structures and can provide queues and streams. RabbitMQ gives me a broker-oriented model centered around routing and message delivery. Kafka gives me a distributed event log with partitions, retention, and consumer offsets.
The right choice depends less on which system has the highest theoretical throughput and more on the semantics the application needs. A simple queue should remain simple. An event platform should provide durable streams and independent consumers. The messaging model should follow the problem.