[kubernetes] 跨云厂商使用公网IP搭建k8s v1.20.9集群

[kubernetes] 跨云厂商使用公网IP搭建k8s v1.20.9集群

最近双11，各大云厂家都在搞活动，所以我在腾讯云和阿里云都各买了一台服务器玩玩。之前的实验环境都是在家里的笔记本用虚拟机搭的，想着搭一套云上的玩一下，本文参照https://blog.csdn.net/chen645800876/article/details/105833835进行操作，搭建后暂时没有遇到问题，配置前一定要配置安全组/防火墙放通端口哦!

两台服务器均使用了Centos 7.6的操作系统，环境如下：

0x01 安装docker（所有主机）

1）配置yum源

sudo yum install -y yum-utilssudo yum-config-manager \--add-repo \http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

2）安装docker

sudo yum install -y docker-ce docker-ce-cli containerd.io#以下是在安装k8s的时候使用yum install -y docker-ce-20.10.7 docker-ce-cli-20.10.7 containerd.io-1.4.6

3）启动docker

systemctl enable docker --now

4）配置加速

sudo mkdir -p /etc/dockersudo tee /etc/docker/daemon.json <<-'EOF'{ "registry-mirrors": ["https://82m9ar63.mirror.aliyuncs.com"], "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2"}EOFsudo systemctl daemon-reloadsudo systemctl restart docker

0x02 配置基础环境（所有主机）    1）配置参数

# 将 SELinux 设置为 permissive 模式（相当于将其禁用）sudo setenforce 0sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config#关闭swapswapoff -a sed -ri 's/.*swap.*/#&/' /etc/fstab#允许 iptables 检查桥接流量cat <

2）下载镜像

sudo tee ./images.sh <<-'EOF'#!/bin/bashimages=(kube-apiserver:v1.20.9kube-proxy:v1.20.9kube-controller-manager:v1.20.9kube-scheduler:v1.20.9coredns:1.7.0etcd:3.4.13-0pause:3.2)for imageName in ${images[@]} ; dodocker pull registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/$imageNamedoneEOFchmod +x ./images.sh && ./images.sh

0x03 安装kubernetes

1）安装kubelet、kubeadm、kubectl（所有主机）

cat <

2）建立虚拟机网卡（重要！所有机器都需要做）

# step1 ，注意替换你的公网IP进去cat > /etc/sysconfig/network-scripts/ifcfg-eth0:1 <

3）修改kubelet启动参数文件（所有节点）

# 此文件安装kubeadm后就存在了vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf# 注意，这步很重要，如果不做，节点仍然会使用内网IP注册进集群# 在末尾添加参数 --node-ip=公网IPExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --node-ip=xx.xx.xx.xx

4）添加配置文件（master）

# step1 添加配置文件，注意替换下面的IPcat > kubeadm-config.yaml <

安装成功后会出现以下信息，请保存一下：

5）安装后配置（master）

# 信息1 上面初始化成功后，将会生成kubeconfig文件，用于请求api服务器，请执行下面操作mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config

6）修改kube-apiserver参数（master，只修改注释项，其余的不要动）

# 修改两个信息，添加--bind-address和修改--advertise-addressvim /etc/kubernetes/manifests/kube-apiserver.yamlapiVersion: v1kind: Podmetadata: annotations: kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.0.20.8:6443 creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-systemspec: containers: - command: - kube-apiserver    - --advertise-address=xx.xx.xx.xx # 修改为公网IP - --bind-address=0.0.0.0 # 新增参数 - --allow-privileged=true - --authorization-mode=Node,RBAC - --client-ca-file=/etc/kubernetes/pki/ca.crt - --enable-admission-plugins=NodeRestriction - --enable-bootstrap-token-auth=true - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key - --etcd-servers=https://127.0.0.1:2379 - --insecure-port=0 - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key - --requestheader-allowed-names=front-proxy-client - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt - --requestheader-extra-headers-prefix=X-Remote-Extra- - --requestheader-group-headers=X-Remote-Group - --requestheader-username-headers=X-Remote-User - --secure-port=6443 - --service-account-issuer=https://kubernetes.default.svc.cluster.local - --service-account-key-file=/etc/kubernetes/pki/sa.pub - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key - --service-cluster-ip-range=10.96.0.0/12 - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-apiserver:v1.20.9 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 10.0.20.8 path: /livez port: 6443 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 name: kube-apiserver readinessProbe: failureThreshold: 3 httpGet: host: 10.0.20.8 path: /readyz port: 6443 scheme: HTTPS periodSeconds: 1 timeoutSeconds: 15 resources: requests: cpu: 250m startupProbe: failureThreshold: 24 httpGet: host: 10.0.20.8 path: /livez port: 6443 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 15 volumeMounts: - mountPath: /etc/ssl/certs name: ca-certs readOnly: true - mountPath: /etc/pki name: etc-pki readOnly: true - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true hostNetwork: true priorityClassName: system-node-critical volumes: - hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - hostPath: path: /etc/pki type: DirectoryOrCreate name: etc-pki - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certsstatus: {}

7）保存后kube-apiserver会被自动重启，确认pod被重启

8）node节点加入集群（命令用安装后输出的那一段）

# 信息2 此信息用于后面工作节点加入主节点使用kubeadm join xx.xx.xx.xx:6443 --token sdfs.dsfsdfsdfijdth \ --discovery-token-ca-cert-hash sha256:sdfsdfsdfsdfsdfsdfsdfsdfg9a460f44b118050091245c1d

9）安装flannel网络插件

[root@txy201-51 ~]# cat flannel.yaml ---apiVersion: policy/v1beta1kind: PodSecurityPolicymetadata: name: psp.flannel.unprivileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/defaultspec: privileged: false volumes: - configMap - secret - emptyDir - hostPath allowedHostPaths: - pathPrefix: "/etc/cni/net.d" - pathPrefix: "/etc/kube-flannel" - pathPrefix: "/run/flannel" readOnlyRootFilesystem: false # Users and groups runAsUser: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny # Privilege Escalation allowPrivilegeEscalation: false defaultAllowPrivilegeEscalation: false # Capabilities allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] defaultAddCapabilities: [] requiredDropCapabilities: [] # Host namespaces hostPID: false hostIPC: false hostNetwork: true hostPorts: - min: 0 max: 65535 # SELinux seLinux: # SELinux is unused in CaaSP rule: 'RunAsAny'---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata: name: flannelrules:- apiGroups: ['extensions'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: ['psp.flannel.unprivileged']- apiGroups: - "" resources: - pods verbs: - get- apiGroups: - "" resources: - nodes verbs: - list - watch- apiGroups: - "" resources: - nodes/status verbs: - patch---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata: name: flannelroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannelsubjects:- kind: ServiceAccount name: flannel namespace: kube-system---apiVersion: v1kind: ServiceAccountmetadata: name: flannel namespace: kube-system---kind: ConfigMapapiVersion: v1metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node app: flanneldata: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | {      "Network": "10.244.0.0/16",  # 这里是kubeadm-config.yaml配置的podsnetwork "Backend": { "Type": "vxlan" } }---apiVersion: apps/v1kind: DaemonSetmetadata: name: kube-flannel-ds namespace: kube-system labels: tier: node app: flannelspec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay.io/coreos/flannel:v0.14.0 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.14.0 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr        - --public-ip=$(PUBLIC_IP)  # 新增 - --iface=eth0 # 新增 resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: PUBLIC_IP #新增 valueFrom: #新增            fieldRef: #新增 fieldPath: status.podIP #新增 volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap:          name: kube-flannel-cfg

10）创建

#创建flannelkubectl apply -f flannel.yaml# 查看flannel是否正常运行[root@txy201-51 ~]# kubectl get pods -A| grep flannel

‍

0x04 测试

1）查看nodes

2）查看pods

3）创建测试pod

# 创建deploymentkubectl create deployment nginx --image=nginx# 查看pod信息kubectl get pods -o wide

4）测试网络可达性