|
版本：v1.0时间：2017-09-28作者：陈二狗
1 准备 1.1 前言 本文档使用二进制部署kubernetes集群，同时开启集群TLS安全认证，搭建整套stable环境，为今后k8s上生产做准备。 1.2 详情
1.3 准备 下载二进制文件地址kubernetes: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.mdetcd: https://github.com/coreos/etcd/releasesflannel: https://github.com/coreos/flannel/releases/ [root@kube-master01 ~]# wget https://dl./v1.8.0/kubernetes-server-linux-amd64.tar.gz [root@kube-master01 ~]# wget https://dl./v1.6.0/kubernetes-client-linux-amd64.tar.gz [root@kube-master01 ~]# wget https://github.com/coreos/etcd/releases/download/v3.2.7/etcd-v3.2.7-linux-amd64.tar.gz [root@kube-master01 ~]# wget https://github.com/coreos/flannel/releases/download/v0.9.0/flannel-v0.9.0-linux-amd64.tar.gz [root@kube-master01 ~]# setenforce 0 [root@kube-master01 ~]# sed -i "s/SELINUX\=.*$/SELINUX\=disabled/g" /etc/selinux/config 把三个压缩包rsync到所有服务器上，并设置所有服务器SElinux disabled 2 CA认证 kubernetes 系统各组件需要使用 TLS 证书对通信进行加密，本文档使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 证书和秘钥文件，CA 是自签名的证书，用来签名后续创建的其它 TLS 证书。 生成的 CA 证书和秘钥文件如下：
使用证书的组件如下：
安装 CFSSL $ wget https://pkg./R1.2/cfssl_linux-amd64 $ chmod +x cfssl_linux-amd64 $ mv cfssl_linux-amd64 /usr/local/bin/cfssl $ wget https://pkg./R1.2/cfssljson_linux-amd64 $ chmod +x cfssljson_linux-amd64 $ mv cfssljson_linux-amd64 /usr/local/bin/cfssljson $ wget https://pkg./R1.2/cfssl-certinfo_linux-amd64 $ chmod +x cfssl-certinfo_linux-amd64 $ mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo 2.1 创建 CA (Certificate Authority) 创建 CA 配置文件 mkdir /opt/ssl cd /opt/ssl cfssl print-defaults config > config.json cfssl print-defaults csr > csr.json 根据config.json文件的格式创建如下的ca-config.json文件过期时间设置成了 87600h cat > ca-config.json <<EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } EOF 字段说明 ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；server auth：表示client可以用该 CA 对server提供的证书进行验证；client auth：表示server可以用该CA对client提供的证书进行验证； 创建 CA 证书签名请求 创建 ca-csr.json 文件，内容如下： { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "TZG" } ] } "CN"：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；"O"：Organization，kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；生成 CA 证书和私钥 $ cfssl gencert -initca ca-csr.json | cfssljson -bare ca $ ls ca* ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem 2.2 创建kube-apiserver证书和私钥 创建kube-apiserver证书签名请求配置apiserver-csr.json： { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.1.210", "192.168.1.211", "192.168.1.212", "192.168.1.213", "10.254.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "TZG" } ] } 注意上面配置hosts字段中指定授权使用该证书的IP和域名列表，因为现在要生成的证书需要被Kubernetes Master集群各个节点使用，所以这里指定了各个节点的IP和hostname。 另外，我们为了实现kube-apiserver的高可用，我们将在其前面部署一个高可用的负载均衡器，Kubernetes的一些核心组件，通过这个负载均衡器和apiserver进行通信，因此这里在证书请求配置的hosts字段中还要加上这个负载均衡器的IP地址，这里是192.168.1.210。 同时还要指定集群内部kube-apiserver的多个域名和IP地址10.254.0.1(后边kube-apiserver-service-cluster-ip-range=10.254.0.0/16参数的指定网段的第一个IP)。 下面生成kube-apiserver的证书和私钥： cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes apiserver-csr.json | cfssljson -bare apiserver ls apiserver* apiserver.csr apiserver-csr.json apiserver-key.pem apiserver.pem 2.3 创建kubernetes-admin客户端证书和私钥 创建admin证书的签名请求配置admin-csr.json： { "CN": "kubernetes-admin", "hosts": [ "192.168.1.211", "192.168.1.212", "192.168.1.213" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:masters", "OU": "TZG" } ] } kube-apiserver将提取CN作为客户端的用户名，这里是kubernetes-admin，将提取O作为用户所属的组，这里是system:master。 kube-apiserver预定义了一些 RBAC使用的ClusterRoleBindings，例如 cluster-admin将组system:masters与 ClusterRole cluster-admin绑定，而cluster-admin拥有访问kube-apiserver的所有权限，因此kubernetes-admin这个用户将作为集群的超级管理员。 下面生成kubernetes-admin的证书和私钥： cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin ls admin* admin.csr admin-csr.json admin-key.pem admin.pem 2.4 创建kube-controller-manager客户端证书和私钥 下面创建kube-controller-manager客户端访问ApiServer所需的证书和私钥。 controller-manager证书的签名请求配置controller-manager-csr.json： { "CN": "system:kube-controller-manager", "hosts": [ "192.168.1.211", "192.168.1.212", "192.168.1.213" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-controller-manager", "OU": "TZG" } ] } kube-apiserver将提取CN作为客户端的用户名，这里是system:kube-controller-manager。 kube-apiserver预定义的 RBAC使用的ClusterRoleBindings system:kube-controller-manager将用户system:kube-controller-manager与ClusterRole system:kube-controller-manager绑定。 下面生成证书和私钥： cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes controller-manager-csr.json | cfssljson -bare controller-manager ls controller-manager* controller-manager.csr controller-manager-csr.json controller-manager-key.pem controller-manager.pem 2.5 创建kube-scheduler客户端证书和私钥 下面创建kube-scheduler客户端访问ApiServer所需的证书和私钥。 scheduler证书的签名请求配置scheduler-csr.json： { "CN": "system:kube-scheduler", "hosts": [ "192.168.1.211", "192.168.1.212", "192.168.1.213" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:kube-scheduler", "OU": "TZG" } ] } kube-scheduler将提取CN作为客户端的用户名，这里是system:kube-scheduler。 kube-apiserver预定义的RBAC使用的ClusterRoleBindings system:kube-scheduler将用户system:kube-scheduler与ClusterRole system:kube-scheduler绑定。 下面生成证书和私钥： cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes scheduler-csr.json | cfssljson -bare scheduler ls scheduler* scheduler.csr scheduler-csr.json scheduler-key.pem scheduler.pem 2.6 创建etcd证书和私钥 创建etcd证书签名请求配置etcd-csr.json： { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.1.211", "192.168.1.212", "192.168.1.213", "kube-master01", "kube-master02", "kube-master03" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "TZG" } ] } 注意上面配置hosts字段中制定授权使用该证书的IP和域名列表，因为现在要生成的证书需要被etcd集群各个节点使用，所以这里指定了各个节点的IP和hostname。 下面生成etcd的证书和私钥： cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd ls etcd* etcd.csr etcd-csr.json etcd-key.pem etcd.pem 3 etcd高可用部署 kuberntes 系统使用 etcd 存储所有数据，本节介绍部署一个三节点高可用 etcd 集群的步骤，这三个节点同 kubernetes master 机器，分别命名为 etcd1：192.168.1.211etcd2：192.168.1.212etcd3：192.168.1.213 LS 认证文件 需要为 etcd 集群创建加密通信的 TLS 证书，这里复用以前创建的 kubernetes 证书 确认三台机器 /etc/kubernetes/ssl 包含 ca.pem kubernetes-key.pem kubernetes.pem [root@kube-master01 ~]# tar -xvf etcd-v3.2.7-linux-amd64.tar.gz [root@kube-master01 ~]# mv etcd-v3.2.7-linux-amd64/etcd* /usr/bin [root@kube-master01 ~]# mkdir -pv /etc/etcd /opt/data/etcd 3.1 配置 etcd 创建 etcd 的 systemd unit 文件 /usr/lib/systemd/system/etcd.service。三台服务器上均创建。 [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify WorkingDirectory=/opt/data/etcd EnvironmentFile=-/etc/etcd/etcd.conf ExecStart=/usr/bin/etcd \ --name ${ETCD_NAME} \ --cert-file=/etc/kubernetes/ssl/etcd.pem \ --key-file=/etc/kubernetes/ssl/etcd-key.pem \ --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \ --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \ --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \ --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \ --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \ --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \ --initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \ --initial-cluster etcd1=https://192.168.1.211:2380,etcd2=https://192.168.1.212:2380,etcd3=https://192.168.1.213:2380 \ --initial-cluster-state new \ --data-dir=${ETCD_DATA_DIR} Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target 创建etcd配置文件 /etc/etcd/etcd.conf # [member] ETCD_NAME=etcd1 ETCD_DATA_DIR="/var/lib/etcd" ETCD_LISTEN_PEER_URLS="https://192.168.1.211:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.1.211:2379" # [cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.211:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.:2379" 这是192.168.1.211节点的配置，其他两个etcd节点只要将上面的IP地址改成相应节点的IP地址即可。ETCD_NAME换成对应节点的etcd1/2/3。 3.2 启动 etcd 服务 [root@kube-master01 ~]# systemctl daemon-reload [root@kube-master01 ~]# systemctl enable etcd [root@kube-master01 ~]# systemctl start etcd 在所有的 kubernetes master 节点重复上面的步骤，直到所有机器的 etcd 服务都已启动。 3.3 验证 etcd 高可用集群 在任一 kubernetes master 机器上执行如下命令： etcdctl \ --ca-file=/etc/kubernetes/ssl/ca.pem \ --cert-file=/etc/kubernetes/ssl/etcd.pem \ --key-file=/etc/kubernetes/ssl/etcd-key.pem \ cluster-health member 9b3a481ed99b49e7 is healthy: got healthy result from http://127.0.0.1:2379 member da08b5dd9b8751ad is healthy: got healthy result from http://127.0.0.1:2379 member f068fe9f74997046 is healthy: got healthy result from http://127.0.0.1:2379 cluster is healthy 结果最后一行为 cluster is healthy 时表示集群服务正常。 4 master高可用部署 部署的Master节点集群由kube-master01, kube-master02, kube-master03三个节点组成，每个节点上部署kube-apiserver,kube-controller-manager,kube-scheduler三个核心组件。 kube-apiserver的3个实例同时提供服务，在其前端部署一个高可用的负载均衡器作为kube-apiserver的地址。 kube-controller-manager和kube-scheduler也是各自3个实例，在同一时刻只能有1个实例工作，这个实例通过选举产生。 将前面生成的 ca.pem, apiserver-key.pem, apiserver.pem, admin.pem, admin-key.pem, controller-manager.pem, controller-manager-key.pem, scheduler-key.pem, scheduler.pem拷贝到各个节点的/etc/kubernetes/ssl目录下： 将二进制文件拷贝到指定路径 cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/local/bin/ 4.1 kube-apiserver部署 创建kube-apiserver的systemd unit文件/usr/lib/systemd/system/kube-apiserver.service，注意替换INTERNAL_IP变量的值： mkdir -p /opt/logs/kubernetes cat > /usr/lib/systemd/system/kube-apiserver.service <<EOF [Unit] Description=kube-apiserver After=network.target After=etcd.service [Service] EnvironmentFile=-/etc/kubernetes/apiserver ExecStart=/usr/local/bin/kube-apiserver \ --logtostderr=true \ --v=0 \ --advertise-address=192.168.1.211 \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --insecure-port=0 \ --allow-privileged=true \ --etcd-servers=https://192.168.1.211:2379,https://192.168.1.212:2379,https://192.168.1.213:2379 \ --etcd-cafile=/etc/kubernetes/ssl/ca.pem \ --etcd-certfile=/etc/kubernetes/ssl/etcd.pem \ --etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem \ --storage-backend=etcd3 \ --service-cluster-ip-range=10.254.0.0/16 \ --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem \ --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \ --experimental-bootstrap-token-auth=true \ --apiserver-count=3 \ --enable-swagger-ui=true \ --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds \ --authorization-mode=RBAC \ --audit-log-maxage=30 \ --audit-log-maxbackup=3 \ --audit-log-maxsize=100 \ --audit-log-path=/opt/logs/kubernetes/audit.log Restart=on-failure Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF --insecure-port禁用了不安全的http端口--secure-port指定https安全端口，kube-scheduler、kube-controller-manager、kubelet、kube-proxy、kubectl等组件都将使用安全端口与ApiServer通信(实际上会由我们在前端部署的负载均衡器代理)。--authorization-mode=RBAC表示在安全端口启用RBAC授权模式，在授权过程会拒绝会授权的请求。kube-scheduler、kube-controller-manager、kubelet、kube-proxy、kubectl等组件都使用各自证书或kubeconfig指定相关的User、Group来通过RBAC授权。--admission-control为准入机制，这里配置了NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds--service-cluster-ip-range指定Service Cluster IP的地址段，注意该地址段是Kubernetes Service使用的，是虚拟IP，从外部不能路由可达。启动kube-apiserver: systemctl daemon-reload systemctl enable kube-apiserver systemctl start kube-apiserver systemctl status kube-apiserver 4.2 配置kubectl访问apiserver 我们已经部署了kube-apiserver，并且前面已经kubernetes-admin的证书和私钥，我们将使用这个用户访问ApiServer。 kubectl --server=https://192.168.1.211:6443 \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --client-certificate=/etc/kubernetes/ssl/admin.pem \ --client-key=/etc/kubernetes/ssl/admin-key.pem \ get componentstatuses NAME STATUS MESSAGE ERROR scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: getsockopt: connection refused etcd-0 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} 上面我们使用kubectl命令打印出了Kubernetes核心组件的状态，因为我们还没有部署kube-scheduler和controller-manager，所以这两个组件当前是不健康的。 前面面使用kubectl时需要指定ApiServer的地址以及客户端的证书，用起来比较繁琐。 接下来我们创建kubernetes-admin的kubeconfig文件 admin.conf。 cd /etc/kubernetes export e="https://192.168.1.211:6443" # set-cluster kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=admin.conf # set-credentials kubectl config set-credentials kubernetes-admin \ --client-certificate=/etc/kubernetes/ssl/admin.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/ssl/admin-key.pem \ --kubeconfig=admin.conf # set-context kubectl config set-context kubernetes-admin@kubernetes \ --cluster=kubernetes \ --user=kubernetes-admin \ --kubeconfig=admin.conf
# set default context kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=admin.conf 为了使用kubectl访问apiserver使用admin.conf，将其拷贝到$HOME/.kube中并重命名成config： cp /etc/kubernetes/admin.conf ~/.kube/config尝试直接使用kubectl命令访问apiserver： kubectl get cs NAME STATUS MESSAGE ERROR scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: getsockopt: connection refused etcd-0 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"} 因为我们已经把kubernetes-admin用户的相关信息和证书保存到了admin.conf中，所以可以删除/etc/kubernetes/pki中的admin.pem和admin-key.pem: cd /etc/kubernetes/ssl rm -f admin.pem admin-key.pem 4.3 kube-apiserver高可用 经过前面的步骤，我们已经在kube-master01,kube-master02,kube-master03上部署了Kubernetes Master节点的kube-apiserver，地址如下： https://192.168.1.211:6443https://192.168.1.212:6443https://192.168.1.213:6443 为了实现高可用，可以在前面放一个负载均衡器来代理访问kube-apiserver的请求，再对负载均衡器实现高可用。 例如，HAProxy+Keepavlided，我们在master01~master03这3个Master节点上运行keepalived，VIP:192.168.1.210。 部署略。 最终kube-apiserver高可用的地址为https://192.168.1.210:6443，修改前面$HOME/.kube/config中的clusters:cluster:server的值为这个地址。 kubectl get cs NAME STATUS MESSAGE ERROR scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: getsockopt: connection refused etcd-0 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"} 4.4 kube-controller-manager部署 前面已经创建了controller-manager.pem和controller-manage-key.pem，下面生成controller-manager的kubeconfig文件controller-manager.conf： cd /etc/kubernetes export KUBE_APISERVER="https://192.168.1.210:6443" # set-cluster kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=controller-manager.conf # set-credentials kubectl config set-credentials system:kube-controller-manager \ --client-certificate=/etc/kubernetes/ssl/controller-manager.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/ssl/controller-manager-key.pem \ --kubeconfig=controller-manager.conf # set-context kubectl config set-context system:kube-controller-manager@kubernetes \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=controller-manager.conf
# set default context kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=controller-manager.conf controller-manager.conf文件生成后将这个文件分发到各个Master节点的/etc/kubernetes目录下。 创建kube-controller-manager的systemd unit文件/usr/lib/systemd/system/kube-controller-manager.service： export KUBE_APISERVER="https://192.168.1.210:6443" cat > /usr/lib/systemd/system/kube-controller-manager.service <<EOF [Unit] Description=kube-controller-manager After=network.target After=kube-apiserver.service [Service] EnvironmentFile=-/etc/kubernetes/controller-manager ExecStart=/usr/local/bin/kube-controller-manager \ --logtostderr=true \ --v=0 \ --master=${KUBE_APISERVER} \ --kubeconfig=/etc/kubernetes/controller-manager.conf \ --cluster-name=kubernetes \ --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \ --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \ --root-ca-file=/etc/kubernetes/ssl/ca.pem \ --insecure-experimental-approve-all-kubelet-csrs-for-group=system:bootstrappers \ --use-service-account-credentials=true \ --service-cluster-ip-range=10.254.0.0/16 \ --cluster-cidr=10.244.0.0/16 \ --allocate-node-cidrs=true \ --leader-elect=true \ --controllers=*,bootstrapsigner,tokencleaner Restart=on-failure Type=simple LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF --service-cluster-ip-range和kube-api-server中指定的参数值一致在各节点上启动: systemctl daemon-reload systemctl enable kube-controller-manager systemctl start kube-controller-manager systemctl status kube-controller-manager 完成部署和启动后，查看一下状态： kubectl get cs NAME STATUS MESSAGE ERROR scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused controller-manager Healthy ok etcd-1 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} 到这一步三个Master节点上的kube-controller-manager部署完成，通过选举出一个leader工作。 分别在三个节点查看状态： #master01 systemctl status -l kube-controller-manager ...... ...... attempting to acquire leader lease... #master02 systemctl status -l kube-controller-manager ...... ...... controllermanager.go:437] Started "endpoint" ...... controllermanager.go:437] Started "replicationcontroller" ...... replication_controller.go:150] Starting RC Manager ...... controllermanager.go:437] Started "replicaset" ...... replica_set.go:155] Starting ReplicaSet controller ...... controllermanager.go:437] Started "disruption" ...... disruption.go:269] Starting disruption controller ...... controllermanager.go:437] Started "statefuleset" ...... stateful_set.go:144] Starting statefulset controller ...... controllermanager.go:437] Started "bootstrapsigner" #master03 systemctl status -l kube-controller-manager ...... ...... attempting to acquire leader lease... 从输出的日志可以看出现在第二个节点上的kube-controller-manager。 4.5 kube-scheduler部署 前面已经创建了scheduler.pem和scheduler-key.pem，下面生成kube-scheduler的kubeconfig文件scheduler.conf： cd /etc/kubernetes export KUBE_APISERVER="https://192.168.1.210:6443" # set-cluster kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/ssl/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=scheduler.conf # set-credentials kubectl config set-credentials system:kube-scheduler \ --client-certificate=/etc/kubernetes/ssl/scheduler.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/ssl/scheduler-key.pem \ --kubeconfig=scheduler.conf # set-context kubectl config set-context system:kube-scheduler@kubernetes \ --cluster=kubernetes \ --user=system:kube-scheduler \ --kubeconfig=scheduler.conf
# set default context kubectl config use-context system:kube-scheduler@kubernetes --kubeconfig=scheduler.conf scheduler.conf文件生成后将这个文件分发到各个Master节点的/etc/kubernetes目录下。 创建kube-scheduler的systemd unit文件/usr/lib/systemd/system/kube-scheduler.service： export KUBE_APISERVER="https://192.168.1.210:6443" cat > /usr/lib/systemd/system/kube-scheduler.service <<EOF [Unit] Description=kube-scheduler After=network.target After=kube-apiserver.service [Service] EnvironmentFile=-/etc/kubernetes/scheduler ExecStart=/usr/local/bin/kube-scheduler \ --logtostderr=true \ --v=0 \ --master=${KUBE_APISERVER} \ --kubeconfig=/etc/kubernetes/scheduler.conf \ --leader-elect=true Restart=on-failure Type=simple LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF 在各节点上启动： systemctl daemon-reload systemctl enable kube-scheduler systemctl start kube-scheduler systemctl status kube-scheduler 到这一步三个Master节点上的kube-scheduler部署完成，通过选举出一个leader工作。 查看Kubernetes Master集群各个核心组件的状态全部正常。 kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-2 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} etcd-1 Healthy {"health": "true"} 5 Kubernetes Node节点部署 Kubernetes的一个Node节点上需要运行如下组件： Dockerkubeletkube-proxy下面我们以在node1上为例部署这些组件： 5.1 CNI安装 wget https://github.com/containernetworking/cni/releases/download/v0.5.2/cni-amd64-v0.5.2.tgz mkdir -p /opt/cni/bin tar -zxvf cni-amd64-v0.5.2.tgz -C /opt/cni/bin ls /opt/cni/bin/ bridge cnitool dhcp flannel host-local ipvlan loopback macvlan noop ptp tuning 5.2 kubelet部署 首先确认将kubelet的二进制文件拷贝到/usr/local/bin下。 创建kubelet的工作目录： mkdir -p /var/lib/kubelet 安装依赖包： yum install ebtables socat util-linux conntrack-tools 接下来创建访问ApiServer的证书和私钥，kubelet-csr.json： { "CN": "system:node:node1", "hosts": [ ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:nodes", "OU": "TZG" } ] } 注意CN为用户名，使用system:node:O为用户组，Kubernetes RBAC定义了ClusterRoleBinding将Group system:nodes和ClusterRole system:node关联。生成证书和私钥： cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=frognew kubelet-csr.json | cfssljson -bare kubelet ls kubelet* kubelet.csr kubelet-csr.json kubelet-key.pem kubelet.pem 生成kubeconfig文件kubelet.conf： cd /etc/kubernetes export KUBE_APISERVER="https://192.168.1.210:6443" # set-cluster kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kubelet.conf # set-credentials kubectl config set-credentials system:node:node1 \ --client-certificate=/etc/kubernetes/pki/kubelet.pem \ --embed-certs=true \ --client-key=/etc/kubernetes/pki/kubelet-key.pem \ --kubeconfig=kubelet.conf # set-context kubectl config set-context system:node:node1@kubernetes \ --cluster=kubernetes \ --user=system:node:node1 \ --kubeconfig=kubelet.conf # set default context kubectl config use-context system:node:node1@kubernetes --kubeconfig=kubelet.conf 创建kubelet的systemd unit service文件，注意替换NodeIP变量： export KUBE_APISERVER="https://192.168.1.210:6443" export NodeIP="192.168.1.214" cat > /usr/lib/systemd/system/kubelet.service <<EOF [Unit] Description=kubelet After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet EnvironmentFile=-/etc/kubernetes/kubelet ExecStart=/usr/local/bin/kubelet \ --logtostderr=true \ --v=0 \ --address=${NodeIP} \ --api-servers==${KUBE_APISERVER} \ --cluster-dns=10.96.0.10 \ --cluster-domain=cluster.local \ --kubeconfig=/etc/kubernetes/kubelet.conf \ --require-kubeconfig=true \ --pod-manifest-path=/etc/kubernetes/manifests \ --allow-privileged=true \ --authorization-mode=AlwaysAllow \ # --authorization-mode=Webhook \ # --client-ca-file=/etc/kubernetes/pki/ca.pem \ --network-plugin=cni \ --cni-conf-dir=/etc/cni/net.d \ --cni-bin-dir=/opt/cni/bin Restart=on-failure [Install] WantedBy=multi-user.target EOF --pod-manifest-path=/etc/kubernetes/manifests指定了静态Pod定义的目录。可以提前创建好这个目录mkdir -p /etc/kubernetes/manifests。--authorization-mode=AlwaysAllow 这里没有启用kubelet API的认证授权功能，先设置AlwaysAllow和我们的线上环境保持一致启动kubelet: systemctl daemon-reload systemctl enable kubelet systemctl start kubelet systemctl status kubelet kubectl get nodes NAME STATUS AGE VERSION node1 NotReady 35s v1.6.2 集群统一日志管理 Dashboard |
|
|
