kafka2.3.1+zookeeper3.5.6+kafka-manager2.0.0.2集群部署（centos7.7）

一、准备三台服务器，配置好主机名和ip地址

 二、服务器初始化：包括安装常用命令工具，修改系统时区，校对系统时间，关闭selinux，关闭firewalld，修改主机名，修改系统文件描述符，优化内核参数，优化数据盘挂载参数

1、安装常用命令工具

yum install vim net-tools bash-completion wget unzip ntp bzip2 epel-release -y

2、修改系统时区，校对系统时间

timedatectl set-timezone Asia/Shanghai

ntpdate pool.ntp.org

3、关闭selinux

vim /etc/selinux/config

SELINUX=disabled

4、关闭防火墙

systemctl stop firewalld.service

systemctl disable firewalld.service

5、修改主机名

vim /etc/hostname

kafka-broker1

6、修改系统文件描述符大小

vim /etc/security/limits.conf

最后添加：

* soft nofile 655360
* hard nofile 655360
* soft nproc 655360
* hard nproc 655360
* soft memlock unlimited
* hard memlock unlimited

7.优化内核参数

vim /etc/sysctl.conf

最后添加：

vm.max_map_count = 655360
net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_max_syn_backlog = 8192
net.core.netdev_max_backlog = 10000
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

8.优化数据盘挂载参数，我的数据盘挂载在opt下，对应自己数据盘作相应调整

vim /etc/fstab

/dev/mapper/centos-opt  /opt                    xfs     defaults,noatime,largeio        0 0

 9.重启系统使配置生效

init 6

三、安装zookeeper集群

1.因zookeeper和kafka需要java启动

首先安装jdk1.8环境

yum install java-1.8.0-openjdk-devel.x86_64 -y

2.Kakfa集群需要依赖ZooKeeper存储Broker、Topic等信息，所以我们先安装zookeeper集群

cd /opt

wget https://mirrors.tuna./apache/zookeeper/zookeeper-3.5.6/apache-zookeeper-3.5.6-bin.tar.gz

tar -zxvf apache-zookeeper-3.5.6-bin.tar.gz

mv apache-zookeeper-3.5.6 zookeeper-3.5.6

3.修改zookeeper配置文件

cd zookeeper-3.5.6/conf/

cp zoo_sample.cfg zoo.cfg

vim zoo.cfg

修改：

dataDir=/opt/zookeeper-3.5.6/data

末尾添加集群其他节点信息

server.1=192.168.0.13:2888:3888
server.2=192.168.0.14:2888:3888
server.3=192.168.0.15:2888:3888

4.添加zookeeper数据目录

创建/opt/zookeeper-3.5.6/data目录

mkdir /opt/zookeeper-3.5.6/data

在data目录里创建myid文件，写上该节点id

 然后将opt下zookeeper-3.5.6目录上传到其他两个节点上

scp -r zookeeper-3.5.6 192.168.0.14:/opt
scp -r zookeeper-3.5.6 192.168.0.15:/opt

修改其他两个节点data下myid文件内容分别为2和3

5.启动zookeeper服务

三台节点分别启动zookeeper服务

/opt/zookeeper-3.5.6/bin/zkServer.sh start

6.查看节点zookeeper状态

/opt/zookeeper-3.5.6/bin/zkServer.sh status

 四、安装kafka集群

1.官网下载kafka

cd /opt

wget http://mirrors.tuna./apache/kafka/2.3.1/kafka_2.12-2.3.1.tgz

tar -zxvf kafka_2.12-2.3.1.tgz

2.修改kafka配置文件

cd kafka_2.12-2.3.1/config

vim server.properties

修改以下参数

broker.id=1
host.name=192.168.0.13
listeners=PLAINTEXT://192.168.0.13:9092
delete.topic.enable=true
log.cleanup.policy=delete
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
auto.create.topics.enable=true
num.network.threads=24
num.io.threads=48
log.dirs=/opt/kafka_2.12-2.3.1/kafka-logs
num.partitions=3
zookeeper.connect=192.168.0.13:2181,192.168.0.14:2181,192.168.0.15:2181

3.优化调整kafka jvm堆内存大小

 vim /opt/kafka_2.12-2.3.1/bin/kafka-server-start.sh

export KAFKA_HEAP_OPTS="-Xmx5G -Xms5G"

4.开启kafka JMX监控

 vim /opt/kafka_2.12-2.3.1/bin/kafka-server-start.sh

export JMX_PORT="9999"

4.其他两台节点按照上面步骤同步安装kafka

5.三台节点分别启动kafka

/opt/kafka_2.12-2.3.1/bin/kafka-server-start.sh -daemon /opt/kafka_2.12-2.3.1/config/server.properties

五、安装kafka-manager，方便管理kafka

1.下载kafka-manager源码

cd /opt

wget https://github.com/yahoo/kafka-manager/archive/2.0.0.2.zip

mv 2.0.0.2.zip kafka-manager-2.0.0.2.zip

2.解压zip包

unzip kafka-manager-2.0.0.2.zip

cd kafka-manager-2.0.0.2

3.yum安装sbt(因为kafka-manager需要sbt编译)

curl https:///sbt/rpm/rpm > /etc/yum.repos.d/bintray-sbt-rpm.repo
yum install sbt -y

4.编译kafka-manager

./sbt clean dist

可能要等好几个小时。

 

 看到Your package is ready in /opt/kafka-manager-2.0.0.2/target/universal/kafka-manager-2.0.0.2.zip代表编译成功了。

5.然后将编译好的zip包拷贝到/opt下，并删除原来kafka-manager-2.0.0.2文件夹

cp /opt/kafka-manager-2.0.0.2/target/universal/kafka-manager-2.0.0.2.zip /opt

rm -rf kafka-manager-2.0.0.2

unzip kafka-manager-2.0.0.2.zip

cd kafka-manager-2.0.0.2

6. 修改kafka-manager配置

vim conf/application.conf

kafka-manager.zkhosts="192.168.0.13:2181,192.168.0.14:2181,192.168.0.15:2181"

7.启动kafka-manager

nohup /opt/kafka-manager-2.0.0.2/bin/kafka-manager &>>/dev/null &

8.用浏览器访问http://192.168.0.13:9000/，添加集群

 

 9.添加成功