k8s部署-43-带你深入学习ingress-nginx（下）

k8s部署-43-带你深入学习ingress-nginx（下）

3TLS和https的配置

很多情况下，nginx会代理https协议，那么我们的ingress-nginx是如何实现这个功能呢，下面看看。

我这里没有https证书，所以需要自己生成https证书；

[root@node1 ~]# cd namespace/[root@node1 namespace]# mkdir tls[root@node1 namespace]# cd tls/[root@node1 tls]# vim gen-secret.sh #!/bin/bashopenssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout yunweijia.key -out yunweijia.crt -subj "/CN=*.yunweijia.com/O=*.yunweijia.com"kubectl create secret tls yunweijia-tls --key yunweijia.key --cert yunweijia.crt[root@node1 tls]# sh gen-secret.sh Generating a 2048 bit RSA private key.......+++.....................................+++writing new private key to 'yunweijia.key'-----secret/yunweijia-tls created[root@node1 tls]#

通过运行上面的脚本之后，生成并导入了k8s中，我们查看下；

[root@node1 tls]# lsgen-secret.sh yunweijia.crt yunweijia.key[root@node1 tls]# [root@node1 tls]# kubectl get secret yunweijia-tls -o yamlapiVersion: v1data: tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURPekNDQWlPZ0F3SUJBZ0lKQU42S1M2MjRCTld4TUEwR0NTcUdTSWIzRFFFQkN3VUFNRFF4R0RBV0JnTlYKQkFNTUR5b3VlWFZ1ZDJWcGFtbGhMbU52YlRFWU1CWUdBMVVFQ2d3UEtpNTVkVzUzWldscWFXRXVZMjl0TUI0WApEVEl5TURReU1URXdNRGN4TUZvWERUSXpNRFF5TVRFd01EY3hNRm93TkRFWU1CWUdBMVVFQXd3UEtpNTVkVzUzClpXbHFhV0V1WTI5dE1SZ3dGZ1lEVlFRS0RBOHFMbmwxYm5kbGFXcHBZUzVqYjIwd2dnRWlNQTBHQ1NxR1NJYjMKRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFDaFozcVFaZWxLTkYvYStOYUNqTGVUS21xWWhDN3Mza0Y4TW5BdApXS0l1Z1llT1hlTmpBdUZ2aFIvRklJU1lhM2F2aG03UGMzNlhNRzBNVDZWRUhjakt3VkxLTWcvZnFyKys5QzB0CjV3Ujd4NngrWFVRLzRDQmRkZDlkeCtJOHN4bUgzLzRxODYxd211YVNGRHFEYWZUSVUyVSs1bHQ2VnZ5V29IRjcKYWFNSGxjZUNMUTFtKzJha1RGNDFPSERUSFhRemh0bUhHck9paHBLYmZWbmJtQmltb1RoU1RwRzJTcWJWZlVHOQphT2Z5UnNBTmlCNmtDeXgrVTNrZWVmc0FUOVJIbmVSZDhrR25lclMxQ1JGdzB4K3JhcVRzQ25Nb3dJdDBHbXpGCi9Md2R0YVVVWUFQeFRnaThNQjhsSjJPNnZFTVRuOVg5SHhrM0NqRHY1eGx6RVlzcEFnTUJBQUdqVURCT01CMEcKQTFVZERnUVdCQlRpTld4Zmhub3dMQTlCZjhxSld3ZlYrV3NER0RBZkJnTlZIU01FR0RBV2dCVGlOV3hmaG5vdwpMQTlCZjhxSld3ZlYrV3NER0RBTUJnTlZIUk1FQlRBREFRSC9NQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUFjCnZNak1VdGZnQ1R0N3BJMG9Vb0tEVU52VEdRY3p0clpCQUYwMzNwR1FNL3htTlIrUFNLbDZGc0FYWjRZd1NkYjYKNUVDMXYwN09zclVWam9xajh6UDNpQXFrVkdISVFFYmpFK1dyaDZsVHpDVkJUbUlXekFKbjRpMTJYcTh3QmljKwpaVzVYN1FRc3NBcGc0SHBVL0owNG9DYklyS1EvNFdkb2IyNzN3STVVYWNHUXFLbVlaZ01jWnY4Vjc0a004ZTFTCnl4OGNPTzMxaGNaTzRLYXY3N1F2L0hjT0g1RnRVY09jbVBiTk5VSEpwT3FJcEIyUHhLUnZibWpQSmE2Zk1xTDcKWDBzV1VTNDFzcitVWW8xcGU2bTJLZndDd2RlaHZVWlRqNG1ubWErZXYvN1pwRnhjdHNicU9sR1BPZFVVZXpDYwp4NkNUa3VXODY2cmdFM1ZzMUh6bQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQ2haM3FRWmVsS05GL2EKK05hQ2pMZVRLbXFZaEM3czNrRjhNbkF0V0tJdWdZZU9YZU5qQXVGdmhSL0ZJSVNZYTNhdmhtN1BjMzZYTUcwTQpUNlZFSGNqS3dWTEtNZy9mcXIrKzlDMHQ1d1I3eDZ4K1hVUS80Q0JkZGQ5ZHgrSThzeG1IMy80cTg2MXdtdWFTCkZEcURhZlRJVTJVKzVsdDZWdnlXb0hGN2FhTUhsY2VDTFExbSsyYWtURjQxT0hEVEhYUXpodG1IR3JPaWhwS2IKZlZuYm1CaW1vVGhTVHBHMlNxYlZmVUc5YU9meVJzQU5pQjZrQ3l4K1Uza2VlZnNBVDlSSG5lUmQ4a0duZXJTMQpDUkZ3MHgrcmFxVHNDbk1vd0l0MEdtekYvTHdkdGFVVVlBUHhUZ2k4TUI4bEoyTzZ2RU1UbjlYOUh4azNDakR2CjV4bHpFWXNwQWdNQkFBRUNnZ0VBSDVya3JCdllsN2d6d093VERSTkM2eVZXSkRGV0F2bnVkc1JscE91RExub2MKc2lyK1dLZjZ2dHloZ1BkQ0g1cURiaEZ6NTFsbFEzT3ZGc2NOeEkzVWdLZGtiOFZueUZObUlwMHJyNTVhQ1dicgpCdjk2V3N1bmFzV25ESFVVZnZCUElvVWcvd1lpUXplQnlMNy9TalpZUVZYYk1IcnBGTWF0eFV6N2k3Lyt0WUEyCkQvTVRjTWdKeFhkY2JWSWN1SFc2clFlVnV6MVlJYWV5ZENra3BFZmdXVXhSaEtXZ0dlRlpGT0FXMUVldEd4KzkKYjBQK21HZmY5YWFQMUNCdWJaT0ZZZHdhNFczQVJWbGlmZit1VWNod1BQUG1ZNmFTSmhCa1VUN0t2QkhlOUM1RwovazNGc0M4T1J0UkFoWXpWNnRPdjlqV0Z5b2liQVNnMHVaZXR2VkNaaFFLQmdRRFY0Yk05UEZ0VjB5ZmVoUlpqCnYwYUZ0Rm95cTlJQ2JndVdFaG9KVXNseFpsc1VzOGd3MUx1Z013VDVtcWpydVRPMm8waEhQVGV1MTRBYUhwWjYKUm1NOEh4c0kwZ3o4Zy9ocW4vN3NhQ253VjJRSzFKOTNSYW9QemI5RVQ3c0ttRWxITVBaM1NNWnYwQXk5MmdtMwpxbmFnOHREZUlLdC8xWE1YclJpMll3cEx5d0tCZ1FEQk1FTXFqbXNjZFBVa20vUkp5aDdpSjBhY0VXSUtHSWt3CkMvV1o5T1h3MFdMeTAyTEJYTHR1cEZ5UEpUR0xkU21iZjVEYTl1ZkFWRW04Z1RYSGxWd0hKUDJSQWduU1pIMEIKbE5pKzQwMVNRQmdyZHpZME9xYTVOK1lQY3Y0djM5aDI3cURKaGJSZmFibE9kMFpndVh2UlhsSDRjRU0yajR3cgpVWVVyU1ZxT1d3S0JnR1VLd0hPd2ZQRzUxTDhDSHNhMnlXbEcvOG1xZElkalY2UHBIMVhDUDVxTUlZRlJRY1VYCjZ4L09tbzRVNjdLWkJ2NUVlMVAwYnNieDFmb0E4MVFHeHNEVGJTRW9vcXYwNkxudXBpOG5NcER4cURpWnBGQmoKbitqaGFYZXJOeERWU3VFUUY0L1kySzVnR25UaWVlN0Q0RkUvQlQrN0xXb3grN0oydXhNSERRa3JBb0dCQUxUQgorNVp0K3pwOUZJSlVpWlloVUJRNnU0NTdsVWZzL1MrL2dPVzBoeEYwV1Nqck1KUEx1SFFseFpVS0wvbFVmc0hICjhqbHVuQmtReEkxa09IV3VBcFdNdnRSWEcrbUhySTgyUGpjZFp0TjJ0U2EyUERsU1IzMDJHRVNNUzlsdmtKSjMKWkdvcWVFSnVJYnlSVFlCakRMOHhpWER6V1hCTGo5TllTMG5kTUtYUkFvR0FER1hOYUhNMW95ZElVd1lnQ3ZmNAo0ZzJlb0dycXBrOGg4aFNJWmVybVltbnptS2V6WHcveGhMd3N1dWVrYi93YUZnVkZ6NXQ5WWFHTnljQ0RaeFJmCm5Ya0xCTW85UE4xMElEQmlzVmR4Y1ZENUE5UlNmZS9PNUlhOTFiUi9qQUtuL3Z4Mk9RL3RxY014ZnZaVGlQZFAKbE5mUXdZR2tubHZhVWxlQlRQSWNkSkU9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0Kkind: Secretmetadata: creationTimestamp: "2022-04-21T10:07:11Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:data: .: {} f:tls.crt: {} f:tls.key: {} f:type: {} manager: kubectl-create operation: Update time: "2022-04-21T10:07:11Z" name: yunweijia-tls namespace: default resourceVersion: "556920" uid: 787c6446-84f5-41ca-aa05-8ca417b4dabdtype: kubernetes.io/tls[root@node1 tls]#

那么我们如何使用这个证书呢？我们可以登录到容器中，看下具体的使用方法，我们来登录到有ingress-nginx的节点上看下；

[root@node2 ~]# crictl ps | grep ingressbe698c18d686e acac7d63e4060 3 hours ago Running nginx-ingress-controller 6 5460063ad17c5[root@node2 ~]# crictl exec -it be698c18d686e bashwww-data@node2:/etc/nginx$ nginx-ingress-controller --help# 省略部分内容 --default-ssl-certificate string Secret containing a SSL certificate to be used by the default HTTPS server (catch-all). Takes the form "namespace/name".# 省略部分内容[root@node2 ~]#

可以看到我们需要新增一个参数来指定，那么我们就来修改下ingress-nginx的配置文件吧；

[root@node1 tls]# vim root/ingress-nginx/mandatory.yaml # 在args下添加一行；- --default-ssl-certificate=default/yunweijia-tls# 变成了如下格式； containers: - name: nginx-ingress-controller image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.20.0 args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io - --default-ssl-certificate=default/yunweijia-tls[root@node1 tls]#

修改完毕之后，我们来让其生效，也就是重新apply一下；

[root@node1 tls]# cd root/ingress-nginx/[root@node1 ingress-nginx]# kubectl apply -f mandatory.yaml[root@node1 ingress-nginx]# cd -/root/namespace/tls[root@node1 tls]#

等容器成功重启完毕之后，我们来使用https的方式来访问一下看看；

[root@node1 tls]# kubectl get pod -n ingress-nginxNAME READY STATUS RESTARTS AGEdefault-http-backend-86dcdf4845-gfwn8 1/1 Running 5 4d20hnginx-ingress-controller-b228z 1/1 Running 0 60snginx-ingress-controller-zlpm4 1/1 Running 0 77s[root@node1 tls]# # 访问一下地址看看https://springboot.yunweijia.com/hello?name=yunweijia

返回状态如下：

可以看到返回了404，为什么会返回这个呢？因为我们的ingress中没有添加，然后我们配置下，把原来的spring-web.yaml文件中的ingress改一下。

[root@node1 tls]# vim ../healthcheck/spring-web.yaml # 省略部分内容apiVersion: extensions/v1beta1kind: Ingressmetadata: name: springboot-web-demospec: rules: - host: springboot.yunweijia.com http: paths: - path: backend: serviceName: springboot-web-demo servicePort: 80 tls: - hosts: - springboot.yunweijia.com secretName: yunweijia-tls[root@node1 tls]# kubectl apply -f ../healthcheck/spring-web.yaml

当新的pod运行起来之后，我们再次访问一下如下地址：

https://springboot.yunweijia.com/hello?name=yunweijia

报错整理

使用ingress-nginx代理https的时候，报错 503 Service Temporarily Unavailable ，我们应该如何排查。

1、首先我们知道，所有的外部调用容器中的服务的时候，经过的都是ingress-nginx，那么首先我们要看下它的日志，使用如下命令进行查看；

[root@node2 ~]# crictl psCONTAINER IMAGE CREATED STATE NAME ATTEMPT POD IDcfb8fd7278d9e 8ad32427177e4 13 minutes ago Running springboot-web-demo 0 d1af2491de6b3ae1e5350fab3f 67da37a9a360e 44 minutes ago Running coredns 38 5387bfd073458752d10d90a4de f2f70adc5d89a 44 minutes ago Running my-nginx 39 e5d7d8dbb13c90b018c1d4a817 b5af743e59849 44 minutes ago Running default-http-backend 1 0a9586855fff64402dda9590f0 7a71aca7b60fc 44 minutes ago Running calico-node 38 428cfeb611fc558e483762b399 acac7d63e4060 44 minutes ago Running nginx-ingress-controller 1 4ecfb510a55777a81629b3b7b2 90f9d984ec9a3 44 minutes ago Running node-cache 38 a78e6521ae606[root@node2 ~]# crictl logs 58e483762b399

其中有类似报错如下：

第一种：E0421第二种：8 controller.go:749] Error obtaining Endpoints for Service "dev/tomcat-demo": no object matching key "dev/tomcat-demo" in local store

排查，登录到master节点上；

[root@node1 healthcheck]# kubectl get svc --all-namespacesNAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEdefault deploy-springboot ClusterIP 10.233.172.184 80/TCP 15ddefault kubernetes ClusterIP 10.233.0.1 443/TCP 33ddefault nginx-ds NodePort 10.233.127.233 80:30323/TCP 32ddefault springboot-web-demo ClusterIP 10.233.109.174 80/TCP 15mingress-nginx default-http-backend ClusterIP 10.233.126.182 80/TCP 152mingress-nginx ingress-nginx ClusterIP 10.233.132.10 80/TCP,443/TCP 37mkube-system coredns ClusterIP 10.233.0.10 53/UDP,53/TCP,9153/TCP 32d[root@node1 healthcheck]#[root@node1 healthcheck]# kubectl get ingress --all-namespacesNAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGEdefault deploy-springboot springboot.yunweijia.com 80 15ddefault springboot-web-demo springboot.yunweijia.com 80 16mdefault springboot.yunweijia.com springboot.yunweijia.com 80, 443 15mdev tomcat-demo-yunweijia2 tomcat.yunweijia.com 80 10d[root@node1 healthcheck]#

从上面可以看到，报错在dev命名空间下的异常找到了，且该服务确实不应该存在，因为我这边就没有启动，其次，也发现了很多本不应该存在的服务，这种时候我采用的是删除方式，如下：

[root@node1 healthcheck]# kubectl delete svc deploy-springboot -n defaultservice "deploy-springboot" deleted[root@node1 healthcheck]# kubectl delete ingress tomcat-demo-yunweijia2 -n devingress.networking.k8s.io "tomcat-demo-yunweijia2" deleted[root@node1 healthcheck]# kubectl delete ingress deploy-springbootingress.networking.k8s.io "deploy-springboot" deleted[root@node1 healthcheck]# kubectl delete ingress springboot-web-demoingress.networking.k8s.io "springboot-web-demo" deleted[root@node1 healthcheck]#

该问题就是，有些不存在的服务还在ingress-nginx的配置中，导致ingress-nginx数据混乱，且同一个域名下，有很多不同名字的映射（这些pod本身也不存在）所导致的，清理完毕之后，再次访问，即可正常。

PS：一定要确保没有删除错误，千万别搞错了，删除的时候一定要谨慎谨慎再谨慎。

至此，本文结束，下一篇咱们继续进行ingress-nginx的深入学习哈。

往期推荐docker下载镜像加速03-docker系列-docker容器的基本操作04-docker系列-commit构建自定义镜像