Scribd
Upload
12 views

How to remove a kafka broker from confluent cluster

Uploaded by

Krishnaraghava
Copyright
© © All Rights Reserved
Available Formats
Download as DOC, PDF, TXT or read online on Scribd
12 views

How to remove a kafka broker from confluent cluster

Uploaded by

Krishnaraghava
Copyright
© © All Rights Reserved
Available Formats
Download as DOC, PDF, TXT or read online on Scribd
You are on page 1/ 3

How to remove a kafka broker from confluent cluster?

ChatGPT said:

ChatGPT

Removing a Kafka broker from a Confluent Kafka cluster involves careful planning to
avoid data loss and service disruption. Here’s a step-by-step guide:

1. Evaluate and Plan

 Review Broker Details: Identify the broker to be removed, its topics, and the
partitions it hosts.
 Assess Cluster Impact: Determine the impact on replication and leader
partitions.
 Backup Data: Ensure you have a backup of critical data.

2. Reassign Partitions

You need to move the partitions hosted on the broker to other brokers.

 Generate a Partition Reassignment Plan: Run the following command to


generate a reassignment JSON plan:

bash
Copy code
kafka-reassign-partitions.sh --zookeeper <zookeeper-host>:<port>
--generate \
--brokers-to-move-json-file brokers-to-move.json

The brokers-to-move.json file should contain the broker ID of the broker you
want to remove:

json
Copy code
{
"version": 1,
"topics": [],
"brokers": [<broker-id>]
}

 Modify the Reassignment Plan: Adjust the generated JSON file to redistribute
the partitions evenly across the remaining brokers.
 Execute the Partition Reassignment: Apply the reassignment using:
bash
Copy code
kafka-reassign-partitions.sh --zookeeper <zookeeper-host>:<port>
--execute \
--reassignment-json-file reassignment.json

 Verify the Reassignment: Check the status:

bash
Copy code
kafka-reassign-partitions.sh --zookeeper <zookeeper-host>:<port>
--verify \
--reassignment-json-file reassignment.json

3. Decommission the Broker

 Stop the Broker: Stop the Kafka broker service:

bash
Copy code
systemctl stop confluent-kafka

or

bash
Copy code
kafka-server-stop.sh

 Remove the Broker from the Cluster: Update the server.properties of the
remaining brokers to remove the broker's ID from their configuration.
 Delete the Broker’s Data: Remove the broker's data directories to free up space:

bash
Copy code
rm -rf /path/to/kafka/logs

4. Update the Cluster Metadata

After the broker is removed, update the cluster metadata.

 Update ZooKeeper (if applicable): If ZooKeeper is in use, clean up references


to the removed broker:

bash
Copy code
zookeeper-shell.sh <zookeeper-host>:<port>

Delete the broker's ZNode:


bash
Copy code
rmr /brokers/ids/<broker-id>

 Verify Cluster Health: Check the cluster health using:

bash
Copy code
kafka-topics.sh --bootstrap-server <broker-list> --describe

5. Monitor the Cluster

 Ensure all partitions have the expected replication factor.


 Monitor the cluster for any unexpected behavior or performance degradation.

Tips

 Rolling Restart: After removing a broker, consider rolling restarts for the
remaining brokers to ensure configurations are applied.
 Test in Staging: Perform the steps in a staging environment before production.
 Monitoring: Use monitoring tools like Confluent Control Center or Prometheus
to monitor cluster health.

Would you like more details on any specific step?

You might also like