原来我们使用云厂商的应用程序网关配置HTTPS双向认证，但后来客户提出要求使用证书吊销列表CRL。遗憾的是，云厂商的应用程序网关并不支持通过CRL进行吊销检查，这使得我们不得不将HTTPS双向认证迁移到Kubernetes集群中的ingress-nginx-controller。接下来，我们将详细描述在Kubernetes环境中生成服务端和客户端证书，并配置Ingress进行HTTPS双向认证的步骤。此外，我们还会探讨如何配置客户端证书的证书吊销列表，以满足客户的需求。
生成服务端和客户端证书并配置Ingress
生成服务端用的CA证书和密钥、服务端证书和密钥
mkdir ingress-mtls && ingress-mtls
# Generate the CA Key and Certificate
# 这里openssl生成的证书是pem格式而非der格式，指定证书格式可以使用`-outform arg output format - DER or PEM`参数
openssl req -x509 -sha256 -newkey rsa:4096 -keyout server-ca-key.pem -out server-ca-cert.pem -days 36525 -nodes -subj '/CN=Aispeech Server Cert Authority'
# Generate the Server Key, and Certificate and Sign with the CA Certificate
# 使用X509v3 Subject Alternative Name
echo subjectAltName = DNS:mtls.dev.ityoudao.cn > server-extfile.cnf
openssl req -new -newkey rsa:4096 -keyout server-key.pem -out server.csr -nodes -subj '/CN=mtls.dev.ityoudao.cn'
openssl x509 -req -sha256 -days 365 -in server.csr -CA server-ca-cert.pem -CAkey server-ca-key.pem -extfile server-extfile.cnf -set_serial 01 -out server-cert.pem
# 创建完整的证书链文件
cat server-cert.pem server-ca-cert.pem > server-cert-chain.pem
# ll server*
-rw-r--r--. 1 root root 1846 1月 5 16:39 server-ca-cert.pem
-rw-r--r--. 1 root root 3268 1月 5 16:39 server-ca-key.pem
-rw-r--r--. 1 root root 1704 1月 5 16:39 server-cert.pem
-rw-r--r--. 1 root root 1598 1月 5 16:39 server.csr
-rw-r--r--. 1 root root 3272 1月 5 16:39 server-key.pem
生成客户端用的CA证书和密钥、客户端证书和密钥
# Generate the CA Key and Certificate
openssl req -x509 -sha256 -newkey rsa:4096 -keyout client-ca-key.pem -out client-ca-cert.pem -days 36525 -nodes -subj '/CN=Aispeech Client Cert Authority'
# Generate the Client Key, and Certificate and Sign with the CA Certificate
openssl req -new -newkey rsa:4096 -keyout client-key.pem -out client.csr -nodes -subj '/CN=Client001'
openssl x509 -req -sha256 -days 365 -in client.csr -CA client-ca-cert.pem -CAkey client-ca-key.pem -set_serial 01 -out client-cert.pem
# ll client*
-rw-r--r--. 1 root root 1846 1月 5 15:52 client-ca-cert.pem
-rw-r--r--. 1 root root 3272 1月 5 15:52 client-ca-key.pem
-rw-r--r--. 1 root root 1688 1月 5 15:52 client-cert.pem
-rw-r--r--. 1 root root 1586 1月 5 15:52 client.csr
-rw-r--r--. 1 root root 3268 1月 5 15:52 client-key.pem
创建保存服务端证书和密钥的Secret
kubectl create secret tls -n odcp mtls-server-secret --cert ./server-cert-chain.pem --key ./server-key.pem
创建保存客户端CA证书的Secret
kubectl create secret generic -n odcp mtls-client-ca-secret --from-file=ca.crt=./client-ca-cert.pem
创建HTTPS双向认证的Ingress
kubectl apply -f - < apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: mtls-test-ingress namespace: odcp annotations: # Enable client certificate authentication nginx.ingress.kubernetes.io/auth-tls-verify-client: "on" # Create the secret containing the trusted ca certificates nginx.ingress.kubernetes.io/auth-tls-secret: "odcp/mtls-client-ca-secret" # Specify the verification depth in the client certificates chain nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1" # Specify an error page to be redirected to verification errors #nginx.ingress.kubernetes.io/auth-tls-error-page: "https://www.ityoudao.cn/" # Specify if certificates are passed to upstream server nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "false" spec: ingressClassName: nginx rules: - host: mtls.dev.ityoudao.cn http: paths: - backend: service: name: nginx-test port: number: 80 path: / pathType: ImplementationSpecific tls: - hosts: - mtls.dev.ityoudao.cn secretName: mtls-server-secret EOF Ingress的客户端证书的相关配置可以参考官网User Guide - Annotations - Client Certificate Authentication 使用curl命令测试HTTPS双向认证 不带客户端证书和密钥，返回400 Bad Request错误 # curl -I https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem HTTP/1.1 400 Bad Request Date: Fri, 05 Jan 2024 08:58:47 GMT Content-Type: text/html Content-Length: 230 Connection: close Cache-Control: no-cache 带客户端证书和密钥，返回200 OK正常 # curl -I https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem --cert ./client-cert.pem --key ./client-key.pem HTTP/1.1 200 OK Date: Fri, 05 Jan 2024 08:59:12 GMT Content-Type: text/html Content-Length: 2161 Connection: keep-alive Last-Modified: Tue, 19 Sep 2023 06:28:05 GMT ETag: "65093f75-871" Accept-Ranges: bytes Cache-Control: no-cache 使用带--cacert server-ca-cert.pem参数的curl命令，没有报服务端证书错误，说明服务端证书和密钥配置正确！不带客户端证书和密钥、带客户端证书和密钥的测试结果均符合预期，说明客户端证书和密钥配置正确！ 遇到的问题和解决办法 ingress-nginx-controller报x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn错误 # kubectl logs -n ingress-nginx ingress-nginx-controller-default-7d94c6957d-l6wwr --tail=100 | grep mtls W0105 08:11:38.291923 7 controller.go:1347] Unexpected error validating SSL certificate "odcp/mtls-server-secret" for server "mtls.dev.ityoudao.cn": x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn W0105 08:11:38.291937 7 controller.go:1353] SSL certificate "odcp/mtls-server-secret" does not contain a Common Name or Subject Alternative Name for server "mtls.dev.ityoudao.cn": x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn 后来发现是因为，误用服务端CA证书和私钥文件创建了mtls-server-secret，因此CN和SAN校验都不通过，导致ingress-nginx-controller报x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn错误，更换为正确的服务端证书和密钥即可解决问题！ kubectl delete secret -n odcp mtls-server-secret kubectl create secret tls -n odcp mtls-server-secret --cert ./server-ca-cert.pem --key ./server-ca-key.pem curl命令报curl: (60) Peer's certificate has an invalid signature.错误 # curl https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem curl: (60) Peer's certificate has an invalid signature. More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. 对端证书签名无效，也就是说服务端证书无效，这里的原因是证书链不完整，使用完整的证书链文件即可解决问题： # 错误的做法，证书链不完整 kubectl create secret tls -n odcp mtls-server-secret --cert ./server-cert.pem --key ./server-key.pem # 正确的做法，完整的证书链 cat server-cert.pem server-ca-cert.pem > server-cert-chain.pem kubectl create secret tls -n odcp mtls-server-secret --cert ./server-cert-chain.pem --key ./server-key.pem ingress-nginx-controller报x509: certificate relies on legacy Common Name field, use SANs instead错误 I0105 08:43:27.668845 7 store.go:584] "Secret was added and it is used in ingress annotations. Parsing" secret="odcp/mtls-server-secret" I0105 08:43:27.669756 7 backend_ssl.go:65] "Adding secret to local store" name="odcp/mtls-server-secret" W0105 08:43:27.673743 7 controller.go:1347] Unexpected error validating SSL certificate "odcp/mtls-server-secret" for server "mtls.dev.ityoudao.cn": x509: certificate relies on legacy Common Name field, use SANs instead 在SUSE官网看到一句说明：“While in the past it was sufficient to have the server name as “Common Name(CN)” within the “Subject” of a certificate and additionally within “Subject Alternative Name”, newer implementations may just check for “Subject Alternative Name”.”，翻译为中文意思是：过去，仅在证书的“主题”中设置“通用名称(CN)”，然后在“主题备用名称”中进行附加就足够了，但较新的实现可能只会检查“主题备用名称”。这里是因为服务端证书只使用了Subject，没有使用X509v3 Subject Alternative Name，而ingress-nginx-controller需要检查主题备用名称SAN，导致报x509: certificate relies on legacy Common Name field, use SANs instead错误，在服务端证书使用SAN即可解决问题： # 错误的做法，没有使用X509v3 Subject Alternative Name openssl req -new -newkey rsa:4096 -keyout server-key.pem -out server.csr -nodes -subj '/CN=mtls.dev.ityoudao.cn' openssl x509 -req -sha256 -days 365 -in server.csr -CA server-ca-cert.pem -CAkey server-ca-key.pem -set_serial 01 -out server-cert.pem # 正确的做法，使用了X509v3 Subject Alternative Name echo subjectAltName = DNS:mtls.dev.ityoudao.cn > server-extfile.cnf openssl req -new -newkey rsa:4096 -keyout server-key.pem -out server.csr -nodes -subj '/CN=mtls.dev.ityoudao.cn' openssl x509 -req -sha256 -days 365 -in server.csr -CA server-ca-cert.pem -CAkey server-ca-key.pem -extfile server-extfile.cnf -set_serial 01 -out server-cert.pem 配置客户端证书的证书吊销列表 假设需要被吊销的证书文件为client-cert.pem，使用openssl ca -revoke命令吊销证书 touch /etc/pki/CA/index.txt openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem 如果openssl ca -revoke命令unable to load CA private key错误： # openssl ca -revoke client-cert.pem Using configuration from /etc/pki/tls/openssl.cnf Error opening CA private key /etc/pki/CA/private/cakey.pem 140215560439696:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/pki/CA/private/cakey.pem','r') 140215560439696:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404: unable to load CA private key # 解决办法：将客户端CA证书和密钥文件拷贝到`/etc/pki/CA/`目录下，或者直接用`-cert`和`-keyfile`参数指定客户端CA证书和密钥文件 # openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem 如果openssl ca -revoke命令unable to open '/etc/pki/CA/index.txt'错误： # openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem Using configuration from /etc/pki/tls/openssl.cnf /etc/pki/CA/index.txt: No such file or directory unable to open '/etc/pki/CA/index.txt' 139702194755472:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/pki/CA/index.txt','r') 139702194755472:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404: # 解决办法：创建一个空的`/etc/pki/CA/index.txt`文件即可 # touch /etc/pki/CA/index.txt # openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem Using configuration from /etc/pki/tls/openssl.cnf Adding Entry with serial number 01 to DB for /CN=Client001 Revoking Certificate 01. Data Base Updated # cat /etc/pki/CA/index.txt R 250104075252Z 240105091344Z 01 unknown /CN=Client001 # cat /etc/pki/CA/index.txt.attr unique_subject = yes # cat /etc/pki/CA/index.txt.old 使用openssl ca -gencrl命令生成证书吊销列表文件 echo 01 > /etc/pki/CA/crlnumber openssl ca -gencrl -out client-ca.crl -cert client-ca-cert.pem -keyfile client-ca-key.pem 如果openssl ca -gencrl命令error while loading CRL number错误： # openssl ca -gencrl -out client-ca.crl -cert client-ca-cert.pem -keyfile client-ca-key.pem Using configuration from /etc/pki/tls/openssl.cnf /etc/pki/CA/crlnumber: No such file or directory error while loading CRL number 140466499000208:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/pki/CA/crlnumber','r') 140466499000208:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404: # 解决办法：创建`/etc/pki/CA/crlnumber`文件即可 # echo 01 > /etc/pki/CA/crlnumber # openssl ca -gencrl -out client-ca.crl -cert client-ca-cert.pem -keyfile client-ca-key.pem Using configuration from /etc/pki/tls/openssl.cnf # ll client-ca.crl -rw-r--r--. 1 root root 954 1月 5 17:27 client-ca.crl # cat /etc/pki/CA/crlnumber 02 # cat /etc/pki/CA/crlnumber.old 01 使用openssl crl命令查看证书吊销列表文件 openssl crl -text -noout -in client-ca.crl # 结果如下： # openssl crl -text -noout -in client-ca.crl Certificate Revocation List (CRL): Version 2 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: /CN=Aispeech Client Cert Authority Last Update: Jan 5 09:27:46 2024 GMT Next Update: Feb 4 09:27:46 2024 GMT CRL extensions: X509v3 CRL Number: 1 Revoked Certificates: Serial Number: 01 Revocation Date: Jan 5 09:13:44 2024 GMT Signature Algorithm: sha256WithRSAEncryption 86:b3:a3:1e:31:75:bb:0f:f3:32:af:2b:57:e6:e2:ca:fb:94: 9b:c1:0b:15:39:af:9e:9a:d6:25:9c:3a:69:e0:29:dd:d6:b6: 55:79:2f:24:c3:7f:40:38:b6:77:6e:08:03:bb:23:0c:9c:5f: db:0d:18:4d:c6:d6:01:74:6f:d5:ba:bb:bc:e3:95:96:91:eb: e1:2b:27:3c:62:33:a8:eb:ed:4a:26:5a:a3:27:cd:f8:02:73: 1c:64:c0:60:95:02:ea:b3:63:ba:d4:86:65:83:c7:cf:fb:7e: 9a:71:7e:4f:9d:b2:50:0b:eb:fb:c5:f4:c2:41:8f:57:50:27: 88:a2:19:02:e5:84:6a:f1:30:ea:71:db:23:8a:04:f0:11:75: c5:8c:6d:04:36:99:6d:42:c6:c0:8d:00:0e:3d:b7:dd:0a:28: ed:e2:5e:7c:f4:56:41:35:d3:73:3c:2f:96:86:92:1e:d7:5a: d2:e5:dc:7c:a1:7b:a9:10:ef:5a:7e:30:1d:86:6f:97:c5:58: b8:3b:ee:75:cf:35:ad:7d:fb:47:ac:a8:7a:81:65:56:4d:2e: 15:60:c0:c9:39:ca:b0:cb:21:a1:48:33:11:51:ee:3e:d5:c2: 9b:ba:c5:fc:ac:04:99:87:2c:2b:56:fe:06:62:76:f8:31:df: dd:b8:a1:f4:bc:d6:87:18:79:00:2d:a6:15:cd:c3:88:80:48: 2a:59:70:8b:06:1d:08:11:39:b5:35:7d:58:c3:3b:27:b5:89: 5f:18:fb:5e:4b:48:4c:04:6a:20:08:96:ad:3d:65:23:da:ad: 0c:74:d6:fc:2a:79:8c:41:42:3b:bd:c5:d0:cb:28:3c:f5:68: 9c:e1:d4:7d:28:c1:3b:20:36:90:d0:71:97:3d:54:78:0f:49: 6e:a2:f4:56:7b:4e:64:03:a6:18:6e:8d:d9:a8:28:96:1f:94: 20:35:1d:6d:2c:f4:56:65:4c:0a:07:2c:c8:5f:44:6f:6c:53: e4:13:d8:56:0c:07:79:7a:0d:cb:a6:e1:de:7e:f0:12:aa:9d: f9:5b:59:6b:61:18:21:96:96:75:4c:3f:b0:ab:73:27:d7:41: ac:50:ea:99:56:13:0f:b9:df:4c:6b:0a:e4:5e:df:56:76:9e: 3a:ce:ab:41:1f:2b:96:bc:9f:77:96:0b:c6:fa:a9:7c:ae:94: a9:c7:f2:68:a0:b2:07:82:1e:74:48:b7:68:f7:da:b0:0d:c4: 5c:08:8a:0e:86:14:70:89:6f:25:9b:63:1d:9b:b4:87:28:0e: 15:23:0f:05:51:c9:9e:b7:57:be:06:b8:74:9d:4f:79:d3:49: 91:16:a3:59:d6:54:64:9a 创建保存客户端CA证书和证书吊销列表的Secret kubectl delete secret -n odcp mtls-client-ca-secret kubectl create secret generic -n odcp mtls-client-ca-secret --from-file=ca.crt=./client-ca-cert.pem --from-file=ca.crl=./client-ca.crl 再次使用curl命令测试客户端证书 # curl -I https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem --cert ./client-cert.pem --key ./client-key.pem HTTP/1.1 400 Bad Request Date: Fri, 05 Jan 2024 09:30:37 GMT Content-Type: text/html Content-Length: 208 Connection: close Cache-Control: no-cache 被吊销的客户端证书和密钥，访问服务报400 Bad Request错误，结果符合预期，说明客户端证书吊销成功！ 更新证书吊销列表 如果证书吊销列表发生变更，可以使用如下命令更新证书吊销列表 # 更新证书吊销列表client-ca.crl kubectl create secret generic -n odcp mtls-client-ca-secret --from-file=ca.crt=./client-ca-cert.pem --from-file=ca.crl=./client-ca.crl --dry-run="client" -o yaml | kubectl apply -f - Secret更新后，ingress-nginx-controller会自动reload，reload可能会影响长连接服务： # ingress-nginx-controller的reload日志 I0105 09:49:31.929155 7 store.go:615] "secret was updated and it is used in ingress annotations. Parsing" secret="odcp/mtls-client-ca-secret" I0105 09:49:31.929841 7 backend_ssl.go:57] "Updating secret in local store" name="odcp/mtls-client-ca-secret" I0105 09:49:31.946486 7 controller.go:166] "Configuration changes detected, backend reload required" I0105 09:49:33.286314 7 controller.go:183] "Backend successfully reloaded" I0105 09:49:33.288423 7 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-default-7d94c6957d-l6wwr", UID:"afac1300-97ad-4304-8ff0-95357f4e3144", APIVersion:"v1", ResourceVersion:"783734248", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration 在本文中，我们深入探讨了在Kubernetes环境中配置ingress-nginx-controller的HTTPS双向认证的过程。文章详细介绍了生成服务端和客户端证书的步骤，以及配置Ingress实现HTTPS双向认证的过程。同时，我们解决了在部署过程中遇到的一系列问题，包括证书配置错误、证书链不完整、以及ingress-nginx-controller报错等。最后，我们还介绍了如何配置客户端证书的证书吊销列表，以满足客户对安全性的额外需求。通过本文，希望大家可以获得在Kubernetes环境中进行HTTPS双向认证的全面指南，帮助应对实际部署中可能遇到的各种挑战。 推荐阅读
发表评论