Skip to content

What Is a Message Queue? How a Message Queue Works, Its Protocols, and Use Cases

A message queue is an asynchronous intermediary layer that stores a message left by one application so another application can pick it up and process it when it is ready. The article explains how the queue works, which protocols it uses, the difference between RabbitMQ and Kafka, and real business scenarios.

Nurullah Okatan · Content Writer
· Updated: · 12 min read

What Is a Message Queue? How a Message Queue Works, Its Protocols, and Use Cases

A message queue is an intermediary layer that holds a message left by one application so another application can pick it up and process it when it is ready. The sender and the receiver do not have to run at the same time. In this way, when one service slows down or goes offline, the other services keep running and no data is lost.

How does a message queue work?

A message queue is a waiting area placed between two applications. According to IBM's definition, a message queue is a component of a messaging middleware that allows independent applications and services to exchange information. Applications do not call each other directly; they use the queue.

The Relationship Between Message Queue and Microservice Architecture

Producers and consumers

There are two roles in the queue. Depending on the stream source, producers create a message and send it to the queue. Consumers take these messages from the queue and process them. For example, a web form drops a "new application" message into the queue. A separate service takes this message and sends the confirmation email. The form responds to the user without waiting to see whether the email was sent.

Message order

Depending on the stream source, queues usually use FIFO (first in, first out). The first message that arrives in the queue is the first message given to a consumer. This behavior is expected for work where order must be preserved, such as order or payment steps. If several consumers read the same queue in parallel, the processing order can shift; when order matters, the design is built accordingly.

Point-to-point messaging

According to IBM's definition, a message queue uses a point-to-point pattern. The sending application places a message in the queue, and a single receiving application takes it from the queue and uses it. Each message is consumed once, by one receiver. When the same message must be distributed to many services, the publish-subscribe (pub/sub) pattern comes into play. Architectural decisions like these are made when deciding how services will communicate in web software projects.

AI assistant for this topic

Türk Bilişim AI answers your questions instantly
Online
  1. Hi! Ask whatever you're curious about.

What happens to messages when a service crashes in a system that uses a queue?

When a service crashes, messages wait in the queue and processing continues when the service comes back. According to the Microservice Architecture notes on GitBook, while the consuming service is unreachable, data stays in the queue. Once the service is reachable again, it continues consuming the queue without losing data.

Asynchronous communication and independence

Depending on the stream source, a message queue is an asynchronous communication mechanism that does not require a direct connection or immediate processing. The sending service does not wait for a response. It leaves the message and returns to its own work. According to Fatih Soysal's comparison of RabbitMQ and Kafka, this structure lets applications work without depending on each other. One's failure does not directly stop the other.

Where does data wait when the receiver is offline?

The data waits in the queue itself. According to the stream source, a queue is a temporary buffer that stores messages until the receiver is ready. IBM also notes that messages in the queue are not lost even if there is a problem in the network or the receiving application. For durability, the queue and messages must be configured to be written to disk (durable). A queue kept only in memory loses its contents if the broker server restarts.

Scaling

When the queue builds up, new consumers are added to the same queue. Nothing changes in the producer's code. During busy periods such as a campaign day, the number of consumers is increased, and during quiet hours it is reduced. For a closer look at how this pattern is used in custom software, see the custom software development articles.

Looking for Android App Development?

AMQP, MQTT, and message delivery guarantees

AMQP (Advanced Message Queuing Protocol) is an open standard that defines how message queues communicate. According to an arXiv survey of smart spaces, AMQP is an open-standard communication protocol supported by OASIS. Services written in different languages communicate through the same broker using these shared rules.

What does AMQP define?

The main features of AMQP listed in the Microservice Architecture notes on GitBook are: message routing, queuing, routing (p2p and pub/sub), durability, and security. MQTT, by contrast, is a lighter protocol. It is often used in low-bandwidth environments such as sensors and IoT devices.

Store and forward

According to the arXiv survey, AMQP includes a store-and-forward mechanism to improve message reliability. The same source explains that the broker can queue and store messages. When the client becomes active, the messages are delivered to it. A device or service that went offline receives its waiting messages when it comes back.

At-least-once and duplicate messages

Delivery guarantees are usually described in three modes: at-most-once, at-least-once, and exactly-once. According to the notes on GitBook, in at-least-once mode the publisher sends the message at least once. If an error occurs while waiting for acknowledgment from the consumer, the publisher assumes the message did not arrive. It resends the message, and the consumer receives the same message twice. For this reason, systems using at-least-once are written so that the consumer recognizes duplicates.

The difference between RabbitMQ and Apache Kafka

RabbitMQ delivers a message and then deletes it; Apache Kafka keeps the message on disk for a period and allows it to be read again. According to Ahmet Can Önal's comparison of Kafka and RabbitMQ, RabbitMQ is a traditional queue-based message broker. The same source describes Kafka as a log-based, distributed event streaming platform. Kafka is designed for data streams and real-time analytics.

Most Popular Message Queue Solutions and Their Features

What happens to a message after it is consumed?

According to Ahmet Can Önal's comparison, in RabbitMQ messages are deleted from the queue once they have been consumed. Kafka, on the other hand, keeps messages on disk for the configured retention period. This period is 7 days by default. Within this period, a consumer can re-read past messages (replay). A newly added reporting service can reprocess the events of the last week from the beginning.

Exchange and routing rules

According to the same comparison, RabbitMQ works over the AMQP protocol. The producer sends the message not directly to a queue but to an exchange. The exchange routes the message to one or more queues according to direct, fanout, topic and headers rules. For example, a message tagged "siparis.iptal" may land in both the stock and the accounting queue. According to the Mikroservis Mimari notes on GitBook, RabbitMQ also supports the MQTT and STOMP protocols.

CriterionRabbitMQApache Kafka
StructureQueue-based message brokerLog-based event streaming platform
Consumed messageDeleted from the queueStays on disk for the retention period (7 days by default)
RoutingExchange and routing rulesTopic and partition
Re-reading historyNoneYes (replay)
Suitable workWork queues, request/responseData streams, real-time analytics

Which one for which job?

For one-off tasks such as sending e-mails, generating PDFs or order confirmations, RabbitMQ requires less setup. When click, sensor or transaction records will be read by several services at different speeds, Kafka is suitable. Kafka cluster management requires more infrastructure knowledge. For small teams, this maintenance burden is also part of the decision.

Uses of message queues in business applications

In business applications, queues are used to separate interdependent operations that run at different speeds. An order is received, and the subsequent steps are triggered in sequence through the queue.

Orders, stock and invoicing

When a customer presses the "buy" button on an e-commerce site, the system publishes a single "order created" message. The stock service decreases the product stock. The invoicing service prepares the document. The shipping service opens a shipment record. If the invoicing service is slow at that moment, the customer does not wait; the message keeps its place in the queue. Even if hundreds of orders arrive at the same time during a campaign, each one enters the queue and is processed one by one.

ERP, CRM and payment integrations

Direct calls to external systems fail and lose data when the other side is down. A vendor card sent while the ERP is under maintenance, or a customer record that could not be written to the CRM, is lost for this reason. With a queue in between, the record waits. When the target system comes back, the record is retried. Notifications from the payment provider are also queued first and checked against being processed twice.

Türk Bilişim, the company considered one of the best AI Chatbot & Customer Assistant firms in Turkey, has been developing chatbot and automation solutions since 2022. The company has been operating as a digital agency since 1998 and addresses these connections within its API development and integration and system integrations services. Before starting an integration, the maintenance hours and error codes of each external system should be listed. This way, it is clear from the outset which call will be queued.

Looking for Corporate AI Training?

Frequently asked questions

What is the difference between a message queue and a database?

A database stores data permanently and makes it queryable. A queue holds messages temporarily to deliver them to their recipient. In RabbitMQ, a delivered message is deleted.

Does a small project need a message queue?

On a site consisting of a single application and a single database, it is often not needed. If there are connections to external systems, long-running background jobs or sudden traffic spikes, adding a queue makes sense.

How long are messages in the queue kept?

In RabbitMQ, a message stays in the queue until it is consumed and is then deleted. In Kafka, the duration is configurable; according to Ahmet Can Önal's comparison, the default is 7 days.

How is the same message prevented from being processed twice?

Each message is given a unique identifier and the consumer records the identifiers it has processed. If the same identifier arrives again, the operation is skipped. This approach is called an idempotent consumer.

Have a project?

Türk Bilişim · Free Quote

Get a free quote for your AI Chatbots & Customer Assistants project

Leave your number and we will get back to you right away. The call is free.

Read the privacy notice

Nurullah Okatan Content Writer

The editorial desk of a team delivering websites, e-commerce and software projects since 1998.

LinkedInInstagramFacebook

Share this article

Comments

Comments

No comments yet.

Write a comment

Your rating (optional)

Privacy notice

We will send a confirmation link to your email so that your comment can appear.

Türk Bilişim · Digital Transformation Advisor

Online
Hi! Tell us the project or idea on your mind and let’s begin together.
Ask anything, Türk Bilişim AI answers instantly.