Messaging
Effective communication between software components, modules, and applications is essential for building robust and efficient systems. This communication allows data exchange and interaction between various parts. There are several methods and needs for communication, both internally within a system and externally with other systems.
Messaging is a powerful approach that decouples components and systems from each other. This decoupling promotes scalability and maintainability. The concept of messaging itself has evolved from the need for asynchronous communication, where components can exchange data without needing to be available at the same time.
Asynchronous Communication
Asynchronous communication allows applications to exchange data without needing to be aware of each other's availability. This decoupled approach offers several benefits for building scalable and maintainable systems.
There are various methods and techniques for asynchronous communication between applications or services. A popular method involves using topics. In this approach, a publisher sends a message to a topic, and subscribers interested in that topic receive the message. This mechanism ensures that the publisher and subscriber don't necessarily need to be aware of each other's existence.
There are several messaging systems, in this post, we will be learning mostly the basics of Apache Kafka.
Pub/Sub
Pub/Sub is an asynchronous messaging service commonly used to broadcast messages. To use Google Cloud Pub/Sub, we must have access to Google Cloud services. Golang has a built-in PubSub package provides an easy way to publish and receive Pub/Sub messages. With this package, we can create a topic to publish messages and create a subscriber to receive notifications.We can achieve similar functionality offered by Pub/Sub using open-source platforms like Apache Kafka and/or RabbitMQ. For more details on Pub/Sub, please refer to the documentation provided by Google Cloud. Since we are not using Google Cloud services in this learning series, we will not go deeply into Pub/Sub in this post.
For more information, visit:
Apache Kafka
Apache Kafka is a messaging and distributed streaming platform that facilitates publish/subscribe communication between applications and services. It is one of the most popular open-source alternatives for messaging platforms.
Kafka operates as a distributed messaging system, utilizing servers and clients to communicate between applications using the TCP protocol. Here are the key components and concepts of Kafka:
Kafka Concepts
- Server: Kafka runs as a cluster on multiple servers. The Kafka cluster is scalable and fault-tolerant, meaning if one server fails, the other servers will take over to ensure continuous operation.
- Client: Kafka clients allow applications to read messages from Kafka topics/servers and process them. Kafka clients are available in several programming languages including Go.
- Topics: The topic is the message category(named) where messages/events will be published, subscribed, and organized.
- Producer: The client application that publishes events/messages to Kafka topics
- Consumer: The client that subscribes and consumes the message/events from Kafka topics
Setting Up Kafka
To use Kafka, you need to set up the Kafka environment, which can be quite complex. You need a cluster management system or platform(Coordination service) to run Kafka clusters. The options include:
- ZooKeeper: Used for managing Kafka clusters in earlier versions. It handles tasks like broker election, configuration management, and partition assignment.
- Kraft: A newer method that eliminates the need for ZooKeeper. Kraft is a built-in alternative that eliminates the need for a separate ZooKeeper service. It replicates metadata across Kafka brokers for fault tolerance.
- Docker: Docker simplifies deployment and management by containerizing Kafka and its dependencies. Docker Compose is a helpful tool for defining and running multi-container applications, including Kafka clusters.
- Kubernetes: Kubernetes is a container orchestration tool that can be used to manage and scale the Kafka cluster.
Exercise: Setting Up Apache Kafka
This exercise will guide you through setting up Apache Kafka and creating a message using Docker.
Step 1: Install Docker
Option 1: Using Homebrew (macOS)
$brew install docker
Option 2: Using Docker Desktop
- Download the Docker Desktop installer from https://www.docker.com/products/docker-desktop/.
- Run the installer and follow the on-screen instructions.
Step 2: Verify Docker Installation
Open a terminal and run:
$docker -v
This command should display information confirming Docker is installed and running.
Step 3: Docker Compose
Docker Compose is a tool for defining and running multi-container applications. It's often bundled with Docker installations. Docker Compose allows you to specify the applications and their configurations in a single file (usually named docker-compose.yml).
Step 4: Start Docker
Verify Docker Status
$docker info
This command will list detailed information about the Docker environment.
List Docker Images
$docker image ls
This command will display a list of Docker images currently available on your machine.
Step 5: Install Kafka
We'll use Docker containers to set up Kafka. Apache Kafka versions 3.7 and later offer an official Docker image. This simplifies the local setup process.
Step 6: Create a Local Kafka Environment
To create a local Kafka environment, we'll utilize a Docker Compose configuration file (docker-compose.yml). This file defines the containers needed for the Kafka cluster to run on your local machine. I'm using the project(learngo) from this series to add my compose file( Part 12 - Program debugging , Profiling and Performance Evaluation)
docker-compose.yml
version: '3.8'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- zookeeper
Step 7: Start the Kafka Cluster
Run the following command to download the Kafka image defined in your docker-compose.yml file and start the Kafka container in the background:
$docker-compose up -d
S*******s-MacBook-Pro:learngo s**********i$ docker compose up -d
WARN[0000] /Users/s*************i/Documents/learngo/docker-compose.yml: `version` is obsolete
[+] Running 8/8
✔ kafka Pulled 25.6s
✔ 42c077c10790 Pull complete 11.5s
✔ 44b062e78fd7 Pull complete 11.9s
✔ b3ba9647f279 Pull complete 11.9s
✔ 10c9a58bd495 Pull complete 16.1s
✔ ed9bd501c190 Pull complete 16.2s
✔ 03346d650161 Pull complete 24.8s
✔ 539ec416bc55 Pull complete 24.8s
[+] Running 2/2
✔ Network learngo_default Created 0.1s
✔ Container learngo-kafka-1 Started
Complete Commands Summary:
- Stop Existing Containers:
$docker-compose down - Start Kafka Service:
$docker-compose up -d - Check Container Status:
$docker-compose ps - View Container Logs:
$docker-compose logs - List Running Containers:
$docker ps
Step 6: Create Topics
We'll use Kafka commands to create topics where messages will be published.
a. List Running Containers:
$docker psS*****s-MacBook-Pro:learngo s*********i$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
385270269b49 wurstmeister/kafka "start-kafka.sh" 24 hours ago Up 24 hours 0.0.0.0:9092->9092/tcp learngo-kafka-1
752d1dcbe150 wurstmeister/zookeeper "/bin/sh -c '/usr/sb…" 24 hours ago Up 24 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp learngo-zookeeper-1b. Access the Kafka Container:
$docker exec -it <container-id> /bin/bashS*****s-MacBook-Pro:learngo s******i$ docker exec -it 385270269b49 /bin/bash root@385270269b49:/#
Here, the 385270269b49 is the container ID of your Kafka container from the previous command.
c. Create a Topic:
Run below command to create topic$kafka-topics.sh --create --topic learngo-person --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
root@385270269b49:/# kafka-topics.sh --create --topic learngo-person --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
Created topic learngo-person.
root@385270269b49:/#
This command creates a topic named learngo-person with one partition and a replication factor of 1 (suitable for a local development environment).
Step 7: View Topic Details
Once you've created a topic, you can view its details using the command:
$kafka-topics.sh --describe --topic learngo-person --bootstrap-server localhost:9092
This command will display information about the newly created topic, including the number of partitions, replication factor, and configuration settings.
root@385270269b49:/# kafka-topics.sh --describe --topic learngo-person --bootstrap-server localhost:9092
Topic: learngo-person TopicId: 3gZMZ-tmTlesfCfmO9Dwng PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: learngo-person Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
root@385270269b49:/#
Step 8: Produce Messages
You can write a producer application to send messages to the Kafka topic(Detail in Part-14 of this series). Alternatively, you can use the kafka-console-producer.sh tool:
$kafka-console-producer.sh --topic learngo-person --bootstrap-server localhost:9092
At the prompt, type your message and press Enter. For example:
>{"person":{"firstName":"learn","lastName":"go"}}Example:
root@385270269b49:/# kafka-console-producer.sh --topic learngo-person --bootstrap-server localhost:9092
>{"person":{"firstName":"learn",lastName:"go"}}
>
Step 9: Consume Messages
Write a consumer application to read messages from the Kafka topic(Detail in Part 14 of this series). Again, you can use thekafka-console-consumer.shtool:
$docker exec -it <container_id> /bin/bash
Replace <container_id> with the actual ID of your Kafka container.
Example:
S*****s-MacBook-Pro:learngo s******i$ docker exec -it 385270269b49 /bin/bash root@385270269b49:/#
Inside the container, run:
$kafka-console-consumer.sh --topic learngo-person --from-beginning --bootstrap-server localhost:9092
This command will start consuming messages from the learngo-person topic, printing them to the console.
Example:
root@385270269b49:/# kafka-console-consumer.sh --topic learngo-person --from-beginning --bootstrap-server localhost:9092
{“person":{"firstName":"learn",lastName:"go"}}
Testing and Verification
You can publish more messages from the producer terminal (Step 8) and observe them being consumed in the consumer terminal (Step 9). Press Ctrl+C in each terminal to stop the producer and consumer.
Docker Desktop Verification
Open Docker Desktop and you should see two running containers: one for ZooKeeper and one for Kafka.
This guide demonstrated how to set up a local Kafka environment using Docker Compose, create topics, and use console tools to produce and consume messages. In the next part, we'll explore how to interact with Kafka using Go programs.
Reference
https://pkg.go.dev/github.com/segmentio/kafka-go
https://kafka.apache.org/intro
https://kafka.apache.org/quickstart
https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac-with-homebrew/
https://www.conduktor.io/kafka/how-to-start-kafka-using-docker/
https://pkg.go.dev/cloud.google.com/go/pubsub
https://cloud.google.com/pubsub/docs/overview
