ingress-nginx 安装配置

发表于 2022-09-30 更新于 2023-09-12 上层目录 Kubernetes 阅读次数：

环境信息

Centos7 5.4.212-1
Docker 20.10.18
containerd.io-1.6.8
kubectl-1.25.0
kubeadm-1.25.0
kubelet-1.25.0

安装 ingress-nginx controller

此文档中的配置主要针对基于部署在裸机（安装通用 Linux 发行版的物理机或者云主机系统）上的 Kebernetes 集群

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.1/deploy/static/provider/cloud/deploy.yaml
mv deploy.yaml ingress-nginx-controller-v1.3.1.yaml

基于 hostNetwork 的 ingress-nginx controller

编辑 ingress-nginx-controller 的 Deployment 配置文件，在 Deployment 中的 .spec.template.spec 下添加字段 hostNetwork: true，以使 ingress-nginx-controller 可以使用节点的主机网络提供对外访问

默认情况下，使用了 hostNetwork: true 配置的 Pod，无法使用集群内部的 DNS （如 CoreDNS）进行域名解析，除非 dnsPolicy 字段配置值为 ClusterFirstWithHostNet。

如果 ingress-nginx-controller 需要解析集群内部的域名，需要在 Deployment 中的 .spec.template.spec 修改以下配置：

dnsPolicy: ClusterFirstWithHostNet

此种模式之下，ingress-nginx-controller 的 Pod 会绑定主机端口的 80 和 443 端口。此方式不需要创建暴露 ingress-nginx-controller 的 Service，如果有，建议删除^[1]

ingress-nginx-controller-v1.3.1.yaml

#---
#apiVersion: v1
#kind: Service
#metadata:
#  labels:
#    app.kubernetes.io/component: controller
#    app.kubernetes.io/instance: ingress-nginx
#    app.kubernetes.io/name: ingress-nginx
#    app.kubernetes.io/part-of: ingress-nginx
#    app.kubernetes.io/version: 1.3.1
#  name: ingress-nginx-controller
#  namespace: ingress-nginx
#spec:
#  externalTrafficPolicy: Local
#  ipFamilies:
#  - IPv4
#  ipFamilyPolicy: SingleStack
#  ports:
#  - appProtocol: http
#    name: http
#    port: 80
#    protocol: TCP
#    targetPort: http
#  - appProtocol: https
#    name: https
#    port: 443
#    protocol: TCP
#    targetPort: https
#  selector:
#    app.kubernetes.io/component: controller
#    app.kubernetes.io/instance: ingress-nginx
#    app.kubernetes.io/name: ingress-nginx
#  type: LoadBalancer

在没有 Service 暴露 ingress-nginx-controller 的 Pod 情况下，容器启动参数中的 --publish-service 将会为空（未使用），需要注释或删除此启动参数。部署之后 Ingress 对象中的 ADDRESS 将为空

Deployment.spec.template.spec.containers[*].args

spec:
  hostNetwork: true
  containers:
  - args:
    - /nginx-ingress-controller
    #- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
    - --election-id=ingress-controller-leader
    - --controller-class=k8s.io/ingress-nginx
    - --ingress-class=nginx
    - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
    - --validating-webhook=:8443
    - --validating-webhook-certificate=/usr/local/certificates/cert
    - --validating-webhook-key=/usr/local/certificates/key

部署成功后，查看 Service 对象，结果中不包含 service/ingress-nginx-controller

$ kubectl get services -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
ingress-nginx-controller-admission   ClusterIP   10.103.195.162   <none>        443/TCP   39s

查看 Ingress 对象，ADDRESS 为空

$ kubectl get ingress -A
NAMESPACE   NAME              CLASS   HOSTS          ADDRESS   PORTS   AGE
default     example-ingress   nginx   cs1.k8s.info             80      26h

如果希望显示节点 IP 地址，可以为容器添加启动参数 - --report-node-internal-ip-address

Deployment.spec.template.spec.containers[*].args

spec:
  hostNetwork: true
  containers:
  - args:
    - /nginx-ingress-controller
    #- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
    - --election-id=ingress-controller-leader
    - --controller-class=k8s.io/ingress-nginx
    - --ingress-class=nginx
    - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
    - --validating-webhook=:8443
    - --validating-webhook-certificate=/usr/local/certificates/cert
    - --validating-webhook-key=/usr/local/certificates/key
    - --report-node-internal-ip-address

成功部署后，再次查看 Ingress 资源，ADDRESS 会上报节点的 IP

$ kubectl get ingress -o wide
NAME              CLASS   HOSTS          ADDRESS        PORTS   AGE
example-ingress   nginx   cs1.k8s.info   172.31.0.230   80      27h

此种模式下，集群中的每个节点上，只能有一个 ingress-nginx-controller 的 Pod 存在，因为不可能让同一个端口绑定多次。为了确保每个节点上只能调度一个 ingress-nginx-controller ，可以使用 DaemonSet 代替 Deployment 的方式来部署 ingress-nginx-controller

默认情况下，ingress-nginx-controller 日志输出到 stdout，可以添加启动参数 --log_dir=/var/log/nginx/ 指定日志输出目录

Deployment.spec.template.spec.containers[*].args

spec:
  hostNetwork: true
  containers:
  - args:
    - /nginx-ingress-controller
    #- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
    - --election-id=ingress-controller-leader
    - --controller-class=k8s.io/ingress-nginx
    - --ingress-class=nginx
    - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
    - --validating-webhook=:8443
    - --validating-webhook-certificate=/usr/local/certificates/cert
    - --validating-webhook-key=/usr/local/certificates/key
    - --report-node-internal-ip-address
    
    # controller 日志选项
    - --log_dir=/var/log/nginx/
    - --logtostderr=false

修改配置后，使用以下命令使配置生效：

kubectl apply -f ingress-nginx-controller-v1.3.1.yaml

使用以下命令删除此配置部署得相关资源

kubectl delete -f ingress-nginx-controller-v1.3.1.yaml

ingress-nginx 相关配置

自定义 nginx 配置，可以使用以下 3 种方式

ConfigMap - 使用 Configmap 配置部署 nginx 全局配置
Annotations - 使用 Annotations 为特定的 Ingress rule 部署个性化的配置
Custom template -

Annotations 配置说明

Annotation 配置中，key 和 value 只能是字符串类型的值。

rewrite-target

rewrite-target 示例 ^[3]

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: rewrite
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - path: /something(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: http-svc
            port: 
              number: 80

path 中捕获到的匹配组会被分别放置在 $1,$2,’$3’…$n，可在 rewrite-target 中使用

本示例中，会产生如下效果：

rewrite.bar.com/something rewrites to rewrite.bar.com/
rewrite.bar.com/something/ rewrites to rewrite.bar.com/
rewrite.bar.com/something/new rewrites to rewrite.bar.com/new

app-root

将 path 中的 / 重写为 nginx.ingress.kubernetes.io/app-root 定义的内容 ^[4]

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/app-root: /app1
  name: approot
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: approot.bar.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: http-svc
            port: 
              number: 80

验证结果

$ curl -I -k http://approot.bar.com/
HTTP/1.1 302 Moved Temporarily
Server: nginx/1.11.10
Date: Mon, 13 Mar 2017 14:57:15 GMT
Content-Type: text/html
Content-Length: 162
Location: http://approot.bar.com/app1
Connection: keep-alive

ssl-redirect

默认情况下，如果 Ingress 中配置了 tls ，HTTP 请求会被重定向到 HTTPS （308），要更改此行为，可以配置：

nginx.ingress.kubernetes.io/ssl-redirect: "false"

Redirect from/to www

将 www.domain.com 重定向到 domain.com，或者将 domain.com 重定向到 www.domain.com ^[7]

nginx.ingress.kubernetes.io/from-to-www-redirect: "true"

Session Affinity

nginx.ingress.kubernetes.io/affinity: cookie
会话亲和性，会话保持。ingress-nginx 支持的值唯有 cookie ^[5]
nginx.ingress.kubernetes.io/affinity-mode
定义会话保持的黏合程度。可取以下值：
- balanced - Pod 扩容或缩减后重新分配 sessions
- persistent - Pod 扩容或缩减后依旧保持会话，不重新分配 sessions

CORS

跨域相关配置 ^[6]

key	value	示例
`cors-allow-methods`	默认值 `GET, PUT, POST, DELETE, PATCH, OPTIONS` *当值配置为 `` 时要注意，`` 不代表所有方法，而是代表默认值中的基本方法*
`cors-allow-headers`	Default: `DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization` *当值配置为 `` 时要注意，`` 不代表所有头部，而只表示基本标头，即默认标头。如果要使用自定义标头，需要在这里显示配置，否则会被拦截*
`cors-allow-origin`	Default: `*`	`nginx.ingress.kubernetes.io/cors-allow-origin: "https://origin-site.com:4443, http://origin-site.com, https://example.org:1199"`

Server snippet

通过此配置，可以添加自定义配置到 nginx 的 server 配置中

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
        set $agentflag 0;

        if ($http_user_agent ~* "(Mobile)" ){
          set $agentflag 1;
        }

        if ( $agentflag = 1 ) {
          return 301 https://m.example.com;
        }

Client Body Buffer Size

Example

nginx.ingress.kubernetes.io/client-body-buffer-size: "1000" # 1000 bytes
nginx.ingress.kubernetes.io/client-body-buffer-size: 1k # 1 kilobyte
nginx.ingress.kubernetes.io/client-body-buffer-size: 1K # 1 kilobyte
nginx.ingress.kubernetes.io/client-body-buffer-size: 1m # 1 megabyte
nginx.ingress.kubernetes.io/client-body-buffer-size: 1M # 1 megabyte

Permanent Redirect

返回永久重定向（301）

Example

nginx.ingress.kubernetes.io/permanent-redirect: 'https://www.google.com'

Temporal Redirect

临时重定向（302）

Example

nginx.ingress.kubernetes.io/temporal-redirect: 'https://www.google.com'

Whitelist source range

访问白名单配置 ^[8]

nginx.ingress.kubernetes.io/whitelist-source-range: '10.0.0.0/24,172.10.0.1'

Custom timeouts

自定义超时时间，所有的超时时间单位为秒 ^[9]

nginx.ingress.kubernetes.io/proxy-connect-timeout
nginx.ingress.kubernetes.io/proxy-send-timeout
nginx.ingress.kubernetes.io/proxy-read-timeout
nginx.ingress.kubernetes.io/proxy-next-upstream
nginx.ingress.kubernetes.io/proxy-next-upstream-timeout
nginx.ingress.kubernetes.io/proxy-next-upstream-tries
nginx.ingress.kubernetes.io/proxy-request-buffering

Custom max body size

nginx.ingress.kubernetes.io/proxy-body-size: 8m

ssl 配置示例

首先需要使用生成的证书配置 type: kubernetes.io/tls 类型的资源，产生对应的 Secret，此处假设证书的 Secret 名称为 test-tls-secret，证书中包含多个域名

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 3000m
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  
  name: ingress-api
  namespace: default
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api1.test.com
    - api2.test.com
    - api3.test.com
    
    secretName: test-tls-secret
  rules:
  - host: api1.test.com
    http:
      paths:
      - backend:
          service:
            name: api-svc
            port:
              number: 8080
        path: /
        pathType: Prefix
  - host: api2.test.com
    http:
      paths:
      - backend:
          service:
            name: api-svc
            port:
              number: 8080
        path: /
        pathType: Prefix
  - host: api3.test.com
    http:
      paths:
      - backend:
          service:
            name: api-svc
            port:
              number: 8080
        path: /
        pathType: Prefix
  - host: api4.test.com
    http:
      paths:
      - backend:
          service:
            name: api-svc
            port:
              number: 8080
        path: /
        pathType: Prefix
  - host: api5.test.com
    http:
      paths:
      - backend:
          service:
            name: api-svc
            port:
              number: 8080
        path: /
        pathType: Prefix

ConfigMaps 配置说明

使用 ConfigMaps 方式配置 nginx 参数 ^[10]

The key and values in a ConfigMap can only be strings. This means that we want a value with boolean values we need to quote the values, like “true” or “false”. Same for numbers, like “100”.

Log format

默认情况下，Access 日至输出到 /var/log/nginx/access.log，软链接到了 /dev/stdout ^[11]

默认日志格式及可用变量

要更改默认的日志格式，编辑 ingress-nginx-controller 的部署配置文件，部署 ConfigMap

ingress-nginx-controller-v1.3.1.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: "true"

  log-format-upstream: "{ time: $time_iso8601|http_host:$http_host|cdn_ip:$remote_addr|request:$request|request_method:$request_method|http_user_agent:$http_user_agent|size:$body_bytes_sent|responsetime:$request_time|upstreamtime:$upstream_response_time|upstreamhost:$upstream_addr|upstreamstatus:$upstream_status|url:$http_host$uri|http_x_forwarded_for:$http_x_forwarded_for|referer:$http_referer|server_protocol:$server_protocol|status:$status}"

部署 ingress-nginx-controller 后，检查 nginx 配置中的 log-format

$ kubectl cp -n ingress-nginx ingress-nginx-controller-nxkwq:/etc/nginx/nginx.conf temp.nginx.conf
$ cat temp.nginx.conf
...
log_format upstreaminfo '{ time: $time_iso8601|http_host:$http_host|cdn_ip:$remote_addr|request:$request|reque
st_method:$request_method|http_user_agent:$http_user_agent|size:$body_bytes_sent|responsetime:$request_time|upstreamti
me:$upstream_response_time|upstreamhost:$upstream_addr|upstreamstatus:$upstream_status|url:$http_host$uri|http_x_forwa
rded_for:$http_x_forwarded_for|referer:$http_referer|server_protocol:$server_protocol|status:$status}';
...

在 Ingress 资源中使用此 log_format

kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: example-ingress
  namespace: default

  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      access_log /var/log/nginx/example-ingress.access.log upstreaminfo;
      error_log /var/log/nginx/example-ingress.err.log;

部署成功后，查看 nginx 配置中关于域名的配置，在此 Ingress 资源中配置的每个域名的 server 配置段中会包含此日志配置。

$ kubectl cp -n ingress-nginx ingress-nginx-controller-nxkwq:/etc/nginx/nginx.conf temp.nginx.conf
cat temp.nginx.conf

...
access_log /var/log/nginx/example-ingress.access.log upstreaminfo if=$loggable;
error_log /var/log/nginx/example-ingress.err.log;
...

参考链接

裸机注意事项

环境信息

安装 ingress-nginx controller

基于 hostNetwork 的 ingress-nginx controller

ingress-nginx 相关配置

Annotations 配置说明

rewrite-target

app-root

ssl-redirect

Redirect from/to www

Session Affinity

CORS

Server snippet

Client Body Buffer Size

Permanent Redirect

Temporal Redirect

Whitelist source range

Custom timeouts

Custom max body size

ssl 配置示例

ConfigMaps 配置说明

Log format

参考链接

脚注