kubernetes集群管理系列讲座（二）安装etcd

kubernetes集群管理系列讲座（二）安装etcd

课程目标

安装单机版etcd安装etcd集群配置安全的etcd（配置SSL证书）

1. 环境

1.1. 软件版本

1.2. 硬件规划

关于机器选项可以参考这个

Here are a few example hardware setups on AWS and GCE environments. As mentioned before, but must be stressed regardless, administrators should test an etcd deployment with a simulated workload before putting it into production.

Note that these configurations assume these machines are totally dedicated to etcd. Running other applications along with etcd on these machines may cause resource contentions and lead to cluster instability.

Small cluster

A small cluster serves fewer than 100 clients, fewer than 200 of requests per second, and stores no more than 100MB of data.

Example application workload: A 50-node Kubernetes cluster

Medium cluster

A medium cluster serves fewer than 500 clients, fewer than 1,000 of requests per second, and stores no more than 500MB of data.

Example application workload: A 250-node Kubernetes cluster

Large cluster

A large cluster serves fewer than 1,500 clients, fewer than 10,000 of requests per second, and stores no more than 1GB of data.

Example application workload: A 1,000-node Kubernetes cluster

xLarge cluster

An xLarge cluster serves more than 1,500 clients, more than 10,000 of requests per second, and stores more than 1GB data.

Example application workload: A 3,000 node Kubernetes cluster

2. 安装单机版etcd

2.1. 二进制包安装etcd

下载和解压这里有下载的向导，也可以点这里直接下载v3.4.9

#设置要下载的版本ETCD_VER=v3.4.9INSTALL_DIR=/opt# 也可以从google下载，鉴于国内无法访问，就注释掉了# GOOGLE_URL=https://storage.googleapis.com/etcdGITHUB_URL=https://github.com/etcd-io/etcd/releases/downloadDOWNLOAD_URL=${GITHUB_URL}# 清理原来下载过的rm -f ${INSTALL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gzrm -rf ${INSTALL_DIR}/etcd && mkdir -p ${INSTALL_DIR}/etcd# 下载curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o ${INSTALL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gz# 解压tar xzvf ${INSTALL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gz -C ${INSTALL_DIR}/etcd --strip-components=1# 删除压缩包rm -f ${INSTALL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gz# 测试${INSTALL_DIR}/etcd/etcd --version${INSTALL_DIR}/etcd/etcdctl version

启动etcd

./etcd

2.2. 编译安装etcd

官方文档在这里注意：编译安装的话需要安装1.14版本及以上的golang环境，下载golang的二进制包，配置GOROOT和GOPATH注意：官网要求golang版本是1.13以上（也就是1.14版本），目前的镜像只有RHEL8.2和AMAZON linux2的yum源中的golang是1.13版本，换句话说，截止2020年5月27日，如果想编译安装etcd，必须自己手动配置golang环境注意：目前版本上使用编译后的etcd创建集群的时候会出现错误，目前测试在AWS的linux2上会出现这个问题，所以只有二进制方式安装最保险panic: runtime error: invalid memory address or nil pointer dereference

下载golang

# 国内无法访问google，请使用下面的链接下载二进制包wget https://studygolang.com/dl/golang/go1.14.3.linux-amd64.tar.gz# 解压tar xf go1.14.3.linux-amd64.tar.gz# 验证./go/bin/go versiongo version go1.14.3 linux/amd64

配置go环境

cat << EOF > /etc/profile.d/golong.shexport GOROOT=/opt/goexport PATH=$PATH:/opt/go/binEOFsource /etc/profile

编译

$ git clone https://github.com/etcd-io/etcd.git$ cd etcd$ go env -w GOPROXY=https://goproxy.cn,direct$ go mod vendor$ ./build

验证

# 在bin目录下面会多出两个可执行文件ls bin/etcd etcdctl# 查看版本$ ./bin/etcd --versionetcd Version: 3.5.0-preGit SHA: 9b6c3e337Go Version: go1.14.3Go OS/Arch: linux/amd64$ ./bin/etcdctl versionetcdctl version: 3.5.0-preAPI version: 3.5

3. 安装etcd集群

如果是简单的demo，我们可以参考官方文档。这里介绍的是在生产环境上搭建etcd集群。

3.1. 准备环境

配置时间服务，ntpd和chrony都可以为etcd创建独立的文件系统，在公有云环境中，系统基本都是镜像启动的，实例被干掉之后容易丢数据，而且速度不如外挂存储卷，且稳定性好。注意：不管我们的数据放在哪里，都有丢失的危险，一定要记得备份！etcd再轻量也是数据库，数据丢了，什么都没了$ lvcreate -n lv_etcd -L 10G vg_system$ mkfs.xfs /dev/mapper/vg_system-lv_etcd$ mkdir -p /data/etcd权限控制# 创建etcd用户，只用来跑程序$ useradd etcd# 修改etcd的主要组为adm，便于同属于adm组的管理员查看，并且指定不可以登录$ usermod -g adm -s nologin etcd# 修改数据的权限，etcd用户拥有所有的权限# etcd所在的组adm拥有读和执行的权限，方便管理员查看或者备份，也可以给他只读权限# 其他人员没有权限# root拥有所有权限$ chmod 740 /data/etcd$ chown etcd:adm /data/etcd$ mkdir /etc/etcd$ chmod 740 /data/etcd$ chown etcd:adm /etc/etcd

3.2. 二进制包安装etcd

下载和解压#设置要下载的版本ETCD_VER=v3.4.9INSTALL_DIR=/opt# 也可以从google下载，鉴于国内无法访问，就注释掉了# GOOGLE_URL=https://storage.googleapis.com/etcdGITHUB_URL=https://github.com/etcd-io/etcd/releases/downloadDOWNLOAD_URL=${GITHUB_URL}# 清理原来下载过的rm -f ${INSTALL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gzrm -rf ${INSTALL_DIR}/etcd && mkdir -p ${INSTALL_DIR}/etcd# 下载curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o ${INSTALL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gz# 解压tar xzvf ${INSTALL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gz -C ${INSTALL_DIR}/etcd --strip-components=1# 删除压缩包rm -f ${INSTALL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gz# 测试${INSTALL_DIR}/etcd/etcd --version${INSTALL_DIR}/etcd/etcdctl version配置etcd的路径cat << EOF > /etc/profile.d/etcd.shexport PATH=$PATH:/opt/etcdEOFsource /etc/profile生成一个长一点的token保证安全$ echo k8s-cluster|md5sumea8cfe2bfe85b7e6c66fe190f9225838 -配置文件/etc/etcd/etcd.confmaster1DATA_DIR=/data/etcdHOST_NAME=master1HOST_IP=10.0.1.204CLUSTER=master1=http://10.0.1.204:2380,master2=http://10.0.1.67:2380,master3=http://10.0.1.236:2380CLUSTER_STATE=newTOKEN=ea8cfe2bfe85b7e6c66fe190f9225838master2DATA_DIR=/data/etcdHOST_NAME=master2HOST_IP=10.0.1.67CLUSTER=master1=http://10.0.1.204:2380,master2=http://10.0.1.67:2380,master3=http://10.0.1.236:2380CLUSTER_STATE=newTOKEN=ea8cfe2bfe85b7e6c66fe190f9225838master3DATA_DIR=/data/etcdHOST_NAME=master3HOST_IP=10.0.1.236CLUSTER=master1=http://10.0.1.204:2380,master2=http://10.0.1.67:2380,master3=http://10.0.1.236:2380CLUSTER_STATE=newTOKEN=ea8cfe2bfe85b7e6c66fe190f9225838

编辑systemd服务文件 /usr/lib/systemd/etcd.service(rhel系列的)或者/lib/systemd/system/etcd.service(ubuntu系列)[Unit]Description=Etcd ServerAfter=network.targetAfter=network-online.targetWants=network-online.target[Service]Type=notifyWorkingDirectory=/data/etcdEnvironmentFile=-/etc/etcd/etcd.confUser=etcd# set GOMAXPROCS to number of processorsExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/etcd/etcd \ --data-dir ${DATA_DIR} \--name \"${HOST_NAME}\" \ --initial-advertise-peer-urls http://${HOST_IP}:2380 \ --listen-peer-urls http://${HOST_IP}:2380 \ --advertise-client-urls http://${HOST_IP}:2379 \ --listen-client-urls http://${HOST_IP}:2379 \ --initial-cluster ${CLUSTER} \ --initial-cluster-state ${CLUSTER_STATE} \ --initial-cluster-token ${TOKEN}"Restart=on-failureLimitNOFILE=65536[Install]WantedBy=multi-user.target启动systemctl daemon-reloadsystemctl start etcd查看状态etcdctl endpoint health注意：etcdctl endpoint health命令不加参数的话，默认是访问本地的2379端口，也就是127.0.0.1：2379，但是咱们刚才配置集群的时候是没有监听本地端口的，所以要使用--endpoint命令指定端口。否则会报错127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded可悲的是，如果使用endpoint参数，就必须使用https协议，也就是必须使用证书etcdctl --endpoints=https://10.0.1.236:2379 --cacert=/etc/k8s/ssl/etcd-root-ca.pem --key=/etc/k8s/ssl/etcd-key.pem --cert=/etc/k8s/ssl/etcd.pem endpoint health我们目前还没配置证书，只能从日志中查看etcd的状态是否正常journalctl -u etcd而etcd的输出位置是没有参数去指定的，他的默认输出是stdout，会由journald来管理Logging: --logger 'capnslog' Specify 'zap' for structured logging or 'capnslog'. [WARN] 'capnslog' will be deprecated in v3.5. --log-outputs 'default' Specify 'stdout' or 'stderr' to skip journald logging even when running under systemd, or list of comma separated output targets. --log-level 'info' Configures log level. Only supports debug, info, warn, error, panic, or fatal.

4. 配置安全的etcd

4.1. 制作证书

参考github

4.1.1. 准备环境

下载cfssl工具curl -s -L -o /usr/local/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64curl -s -L -o /usr/local/bin/cfssl https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64chmod +x /usr/local/bin/{cfssl,cfssljson}测试一下$ cfsslNo command is given.Usage:Available commands: serve version genkey gencrl ocsprefresh selfsign scan print-defaults revoke bundle sign gencert ocspdump ocspserve info certinfo ocspsignTop-level flags: -allow_verification_with_non_compliant_keys Allow a SignatureVerifier to use keys which are technically non-compliant with RFC6962. -loglevel int Log level (0 = DEBUG, 5 = FATAL) (default 1)创建工作目录$ mkdir -p /etc/kubernetes/pki/etcd

4.1.2. 创建CA相关证书

创建CA配置文件（默认创建）cd /etc/kubernetes/pki/etcdcfssl print-defaults config > ca-config.jsoncfssl print-defaults csr > ca-csr.json修改ca-config.json为{ "signing": { "default": { "expiry": "43800h" }, "profiles": { "server": { "expiry": "43800h", "usages": [ "signing", "key encipherment", "server auth" ] }, "client": { "expiry": "43800h", "usages": [ "signing", "key encipherment", "client auth" ] }, "peer": { "expiry": "43800h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } }}修改ca-csr.json为{ "CN": "My own CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "US", "L": "CA", "O": "My Company Name", "ST": "San Francisco", "OU": "Org Unit 1", "OU": "Org Unit 2" } ]}创建CA的证书cfssl gencert -initca ca-csr.json | cfssljson -bare ca -会得到三个文件ca-key.pemca.csrca.pem

4.1.3. 创建服务器（server）相关证书

生成配置文件cfssl print-defaults csr > server.json修改server.json中CN和host的部分{ "CN": "etcd", "hosts": [ "127.0.0.1", "10.0.1.204", "10.0.1.67", "10.0.1.236" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ]}生成证书cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server同样是三个文件（这个是服务器启动时候的证书）server-key.pemserver.csrserver.pem

4.1.4. 创建服务器互相通讯（peer）的相关证书

生成配置文件cfssl print-defaults csr > members.json修改server.json中CN和host的部分{ "CN": "members", "hosts": [ "127.0.0.1", "10.0.1.204", "10.0.1.67", "10.0.1.236" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ]}生成证书cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer members.json | cfssljson -bare members三个文件members-key.pemmembers.csrmembers.pem

4.1.5. 创建客户端（client）的相关证书

生成配置文件cfssl print-defaults csr > client.json修改server.json中CN和host的部分注意：一般来说，如果etcd需要手动创建的话，架构上会把这三台etcd独立拿出来作为数据库来管理，所以客户端会的hosts是etcd之外的IP地址，但是我们这里实验是使用这三台etcd作为客户端的，所以地址还是这三台机器。如果实在不明白，就把所有的机器ip和DNS名称都写在这个hosts里面，或者让hosts留空(下面的例子)，防止出错，不过这样并不算最安全的选择。{ "CN": "client", "hosts": [""], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", "L": "CA", "ST": "San Francisco" } ]}生成证书cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client又得到一组证书client-key.pemclient.csrclient.pem

4.2. 配置etcd使用ssl证书

把刚才生成的所有证书(在/etc/kubernetes/pki/etcd下的所有文件)都复制到另外的etcd机器上去。修改启动文件

[Unit]Description=Etcd ServerAfter=network.targetAfter=network-online.targetWants=network-online.target[Service]Type=notifyWorkingDirectory=/data/etcdEnvironmentFile=-/etc/etcd/etcd.confUser=etcd# set GOMAXPROCS to number of processorsExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/etcd/etcd \ --data-dir ${DATA_DIR} \ --name ${HOST_NAME} \ --initial-advertise-peer-urls https://${HOST_IP}:2380 \ --listen-peer-urls https://${HOST_IP}:2380 \ --advertise-client-urls https://${HOST_IP}:2379 \ --listen-client-urls https://127.0.0.1:2379,https://${HOST_IP}:2379 \ --listen-metrics-urls=http://127.0.0.1:2381 \ --initial-cluster ${CLUSTER} \ --initial-cluster-state ${CLUSTER_STATE} \ --initial-cluster-token ${TOKEN} \ --client-cert-auth \ --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ --cert-file=/etc/kubernetes/pki/etcd/server.pem \ --key-file=/etc/kubernetes/pki/etcd/server-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.pem \ --peer-cert-file=/etc/kubernetes/pki/etcd/members.pem \ --peer-key-file=/etc/kubernetes/pki/etcd/members-key.pem"Restart=on-failureLimitNOFILE=65536[Install]WantedBy=multi-user.target

修改配置文件，/etc/etcd/etcd.confmaster1DATA_DIR=/data/etcdHOST_NAME=master1HOST_IP=10.0.1.204CLUSTER=master1=https://10.0.1.204:2380,master2=https://10.0.1.67:2380,master3=https://10.0.1.236:2380CLUSTER_STATE=newTOKEN=ea8cfe2bfe85b7e6c66fe190f9225838master2DATA_DIR=/data/etcdHOST_NAME=master2HOST_IP=10.0.1.67CLUSTER=master1=https://10.0.1.204:2380,master2=https://10.0.1.67:2380,master3=https://10.0.1.236:2380CLUSTER_STATE=newTOKEN=ea8cfe2bfe85b7e6c66fe190f9225838master3DATA_DIR=/data/etcdHOST_NAME=master3HOST_IP=10.0.1.236CLUSTER=master1=https://10.0.1.204:2380,master2=https://10.0.1.67:2380,master3=https://10.0.1.236:2380CLUSTER_STATE=newTOKEN=ea8cfe2bfe85b7e6c66fe190f9225838

修改权限chmod 400 /etc/kubernetes/pki/etcd/*chown -R etcd:adm /etc/kubernetes/pki/etcd/

重启启动集群会报错error "tls: first record does not look like a TLS handshake删除数据文件，重新启动就好了