UK8S - XXXCloud

Nginx Ingress

This article applies to K8S version 1.19+

What is Ingress

Ingress serves as the entry point for accessing internal cluster services from outside the Kubernetes cluster, providing layer-7 load balancing capabilities for in-cluster Services.

Generally, Services and Pods can only be accessed via IP addresses within the cluster, with all traffic reaching the cluster boundary either discarded or forwarded elsewhere. In the Service chapter, we introduced creating LoadBalancer-type Services: UK8S leverages Kubernetes’ extension interfaces to create a corresponding ULB load balancer for each Service, which receives external traffic and routes it to the cluster. However, in scenarios like microservices, where each Service requires a dedicated load balancer, management costs become prohibitively high—hence the emergence of Ingress.

Think of Ingress as a “Service for Services,” providing proxy-based load balancing for backend Services. It allows configuration of externally accessible URLs, load balancing, SSL, name-based virtual hosting, and more.

I. Deployment of Ingress Controller

For Ingress to function, an Ingress Controller must be deployed in the cluster. Unlike other controllers (e.g., Deployment, which runs as part of the kube-controller-manager binary at cluster startup), Ingress Controller requires manual deployment. The Kubernetes community offers the following options:

Nginx
HAProxy
Envoy
Traefik

Here we choose Nginx as the Ingress Controller. It’s very easy to deploy the Nginx Ingress Controller - just execute the following assignment.


kubectl apply -f https://docs.ucloud-global.com/uk8s/yaml/ingress_nginx/mandatory_1.19.yaml

Notes

The sample Ingress Controller creates an internal ULB. To create an external ULB, download the YAML file and modify service.beta.kubernetes.io/ucloud-load-balancer-type to “outer” in the Service’s metadata.annotations. For more parameters, see ULB Parameter Description.

Containers use UTC by default. To use the host’s time zone, refer to Pod Time Zone Issues.

Explanation

The YAML defines a ReplicaSet of pods using the ingress-nginx-controller image. This Pod acts as a controller that monitors Ingress objects and changes to their backend Services. When a new Ingress is created, the controller generates a corresponding Nginx configuration file (/etc/nginx/nginx.conf) and starts an Nginx service with this config. If the Ingress is updated, the config file is refreshed. Notably, if only the proxied Service is updated, the Nginx service does not need to reload, as ingress-nginx-controller uses Nginx Lua to enable dynamic upstream configuration.

The YAML also defines a ConfigMap, which allows ingress-nginx-controller to customize Nginx configurations. Example:


apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.2.1
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: 'true'
  map-hash-bucket-size: "128"
  ssl-protocols: SSLv2

Note that in ConfigMap, the key and value only support strings, so values like integers need to be wrapped in double quotes, for example “100”. Detailed information can be found at Nginx-Ingress-ConfigMap .

II. External access to nginx ingress

Above, we have deployed a nginx ingress controller in UK8S, and in order to be externally accessible, we have also created an externally accessible LoadBalancer type Service, as shown below.


apiVersion: v1
kind: Service
metadata:
  annotations:
    "service.beta.kubernetes.io/ucloud-load-balancer-type": "inner"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.2.1
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
      appProtocol: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller

The sole job of this Service is to expose the 80 and 433 ports of all Pods with the ingress-nginx label. We can get the external access entry of this Service in the following way.


$ kubectl get svc -n ingress-nginx
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   172.30.48.77   xxx.yy.xxx.yy   80:30052/TCP,443:31285/TCP   14m
......

After deploying the Ingress Controller and its required Service, we can now use it to proxy other Services inside the cluster.

III. Create two applications

In the following YAML, we define 2 applications with the image name echo-nginx, which mainly output some global variables of the nginx application itself.


apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app-1
  labels:
    app:  demo-app-1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo-app-1
  template:
    metadata:
      labels:
        app: demo-app-1
    spec:
      containers:
        - name: demo-app-1
          image: uhub.ucloud-global.com/jenkins_k8s_cicd/echo_nginx:v11
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app-1-svc
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  selector:
    app: demo-app-1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app-2
  labels:
    app:  demo-app-2
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo-app-2
  template:
    metadata:
      labels:
        app: demo-app-2
    spec:
      containers:
        - name: demo-app-2
          image: uhub.ucloud-global.com/jenkins_k8s_cicd/echo_nginx:v11
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: demo-app-2-svc
  labels:
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  selector:
    app: demo-app-2

We will save the above yaml as demo-app.yaml and create an application with the following command.


kubectl apply -f demo-app.yaml

IV. Defining Ingress Objects

We have deployed the nginx ingress controller and exposed it to the outside network, and created two applications. Next, we can define an ingress object to proxy the two applications out of the cluster.


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo-app-ingress
spec:
  ingressClassName: nginx
  defaultBackend:
    service:
      name: demo-app-1-svc
      port:
        number: 80
  rules:
    - host: demo-app.example.com
      http:
        paths:
          - path: /demo-app-1
            pathType: Prefix
            backend:
              service:
                name: demo-app-1-svc
                port:
                  number: 80
          - path: /demo-app-2
            pathType: Prefix
            backend:
              service:
                name: demo-app-2-svc
                port:
                  number: 80

The above yaml file defines an Ingress object, where ingress.spec.rules are the proxy rule sets for ingress.

Explanation of ingressClassName

In the above Ingress configuration, we set ingressClassName: nginx, which specifies which Ingress Controller should handle this Ingress object.

If ingressClassName is not specified, Kubernetes will check if a default IngressClass exists in the cluster.

Setting the Default IngressClass

You can set the default value by adding an annotation to IngressClass. This has been added in mandatory_1.26.yaml:


apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
  annotations:
    ingressclass.kubernetes.io/is-default-class: "true"
spec:
  controller: k8s.io/ingress-nginx

Explanation of defaultBackend

When no host and path in the rules are matched, the default backend defined in ingress.spec.defaultBackend takes effect, forwarding traffic that does not match any host or path to the defaultBackend for processing.

Explanation of rules

In the above Ingress configuration, we set ingressClassName: nginx, which specifies which Ingress Controller should handle this Ingress object.

First, let’s look at the host field. Its value must be a standard domain name format string and cannot be an IP address. The value defined by the host field is the entry of this Ingress. As such, when a user accesses demo-app.example.com, he or she is actually accessing this Ingress object. This enables Kubernetes to use IngressRule to forward your request to the next step.

Next is the path field. You can simply think of each path here as corresponding to a backend Service. So in our example, there are two paths defined, they correspond to demo-app-1 and demo-app-2 respectively, these two Deployment Services (i.e., demo-app-1-svc and demo-app-2-svc).

Each HTTP rule contains the following information: a host configuration item (e.g., demo-app.example.com), a list of paths (e.g., /demo-app-1 and /demo-app-2), each path is associated with a backend (e.g., the 80 port of demo-app-1-svc). Before the LoadBalancer forwards traffic to the backend, all inbound requests must first match the host and path.

When no host and path from a rule match, the default backend defined in ingress.spec.defaultBackend will take effect, redirecting traffic not matching host and path to the defaultBackend for handling.

We save the above yaml as ingress.yaml and create an ingress object directly with the following command:


kubectl apply -f ingress.yaml

Next, we can look at this ingress object


$ kubectl get ingresses.networking.k8s.io
NAME               CLASS    HOSTS                  ADDRESS   PORTS   AGE
demo-app-ingress   <none>   demo-app.example.com             80      5m39s
 
$ kubectl describe ingresses.networking.k8s.io demo-app-ingress
Name:             demo-app-ingress
Labels:           <none>
Namespace:        ingress-nginx
Address:
Default backend: demo-app-1-svc:80 (172.20.145.127:80,172.20.187.251:80)
Rules:
  Host                  Path  Backends
  ----                  ----  --------
  demo-app.example.com
                        /demo-app-1   demo-app-1-svc:80 (172.20.145.127:80,172.20.187.251:80)
                        /demo-app-2   demo-app-2-svc:80 (172.20.40.180:80,172.20.43.118:80,172.20.62.63:80)
Annotations:            <none>
Events:                 <none>

From the rule rules we can see that the Host we defined is demo-app.example.com and it has two forwarding rules (Path), each forwarded to demo-app-1-svc and demo-app-2-svc.

Of course, in the Ingress yaml file, you can also define multiple Hosts to provide load balancing services for more domain names.

Access

Next, we can access the applications we deployed earlier by visiting the address and port of this Ingress. For example, when we visit http://demo-app.example.com/demo-app-2, it should be demo-app-2’s Deployment that responds to my request. When creating LoadBalance, if you choose external network mode and bind EIP, you can directly access through the external network. We can add /etc/hosts locally to directly access via domain name from outside network. If it is in internal network mode, only resources within the VPC can access through Ingress.


$ cat /etc/hosts
......
 
xxx.yy.xxx.yy demo-app.example.com


$ curl http://demo-app.example.com/demo-app-2
Scheme: http
Server address: 172.20.43.118:80
Server name: demo-app-2-5f6c5df698-rvsmc
Date: 12/Jan/2022:02:56:38 +0000
URI: /demo-app-2
Request ID: ba34c07f5cc78e74629041df5568977a

V. TLS Support

In the above ingress object, we did not specify a TLS certificate for the Host. The ingress controller supports encryption of the site by specifying a secret containing the TLS private key and certificate.

First, let’s create a secret containing tls.crt and tls.key. When generating certificates, make sure the CN contains demo-app.example.com and base64 encode the certificate content.


$ HOST=demo-app.example.com
 
$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=${HOST}/O=${HOST}"
Generating a RSA private key
................................+++++
................................+++++
writing new private key to 'tls.key'


$ kubectl create secret tls demo-app-tls --key tls.key --cert tls.crt
secret/demo-app-tls created
 
$ kubectl describe secret demo-app-tls
Name:         demo-app-tls
Namespace:    ingress-nginx
Labels:       <none>
Annotations:  <none>
 
Type:  kubernetes.io/tls
 
Data
====
tls.crt:  1229 bytes
tls.key:  1708 bytes

Then in the ingress object, reference the secret via the ingress.spec.tls field and the ingress controller will encrypt it to protect the communication pipeline between the client and the ingress.


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo-app-ingress
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - demo-app.example.com
      secretName: demo-app-tls
  defaultBackend:
    service:
      name: demo-app-1-svc
      port:
        number: 80
  rules:
    - host: demo-app.example.com
      http:
        paths:
          - path: /demo-app-1
            pathType: Prefix
            backend:
              service:
                name: demo-app-1-svc
                port:
                  number: 80
          - path: /demo-app-2
            pathType: Prefix
            backend:
              service:
                name: demo-app-2-svc
                port:
                  number: 80

At this point, you can use the https protocol to access the ingress, but keep in mind that the certificate for the ingress is self-signed by us, so when using tools like curl to access, you need to skip the certificate check. In a production environment, it is recommended to use a TLS certificate that has been signed by a CA.


$ curl --insecure https://demo-app.example.com/demo-app-2
Scheme: http
Server address: 172.20.40.180:80
Server name: demo-app-2-5f6c5df698-pvr45
Date: 12/Jan/2022:03:19:58 +0000
URI: /demo-app-2
Request ID: 4cc35ddb1a301977f0477ded4c09d5df

VI. Setting Access Whitelist

In some scenarios, our business only allows specified IP addresses to access. This can be achieved by adding annotations, i.e., nginx.ingress.kubernetes.io/whitelist-source-range. The value is a list of CIDRs separated by a comma ”,”. Example as follows:


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo-app-ingress
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 172.16.0.0/16,172.18.0.0/16
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - demo-app.example.com
      secretName: demo-app-tls
  defaultBackend:
    service:
      name: demo-app-1-svc
      port:
        number: 80
  rules:
    - host: demo-app.example.com
      http:
        paths:
          - path: /demo-app-1
            pathType: Prefix
            backend:
              service:
                name: demo-app-1-svc
                port:
                  number: 80
          - path: /demo-app-2
            pathType: Prefix
            backend:
              service:
                name: demo-app-2-svc
                port:
                  number: 80