Getting Started with Kafka: How to Start a Kafka Server and Create a Topic

Apache Kafka is a distributed streaming platform that allows you to publish and subscribe to streams of records. It is widely used for building real-time data pipelines and streaming applications. In this tutorial, we will guide you on how to start a Kafka server and create a topic with an example.

Prerequisites

Before we begin, make sure that you have the following installed on your machine:

• Java 8 or later
• Apache Kafka

Starting a Kafka Server

To start a Kafka server, follow these steps:

1. Open a terminal window and navigate to the Kafka installation directory.
2. Start the ZooKeeper service by running the following command:

bin/zookeeper-server-start.sh config/zookeeper.properties

3. Start the Kafka server by running the following command:

bin/kafka-server-start.sh config/server.properties

Creating a Topic

Once the Kafka server is up and running, you can create a topic using the following steps:

1. Open a new terminal window and navigate to the Kafka installation directory.
2. Create a new topic by running the following command:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic my-topic

In this example, we have created a topic named my-topic with a replication factor of 1 and a single partition.

3. Verify that the topic has been created by running the following command:

bin/kafka-topics.sh --list --zookeeper localhost:2181

This command should display a list of all the topics that have been created.

Example

Now that we have created a topic, let’s see how we can use it in a simple example. In this example, we will create a producer that sends messages to the my-topic topic and a consumer that reads messages from the same topic.

Creating a Producer

1. Open a new terminal window and navigate to the Kafka installation directory.
2. Start the Kafka console producer by running the following command:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

This command starts a console producer that is connected to the my-topic topic.

3. Type a message and press Enter. The message should be sent to the my-topic topic.

Creating a Consumer

1. Open a new terminal window and navigate to the Kafka installation directory.
2. Start the Kafka console consumer by running the following command:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-topic --from-beginning

This command starts a console consumer that is connected to the my-topic topic. The --from-beginning option tells the consumer to read all messages from the beginning of the topic.

3. The console consumer should display the message that was sent by the producer.

Conclusion

Congratulations! You have successfully started a Kafka server, created a topic, and used it in a simple example. You can now start building real-time data pipelines and streaming applications using Kafka.