Fixed IP Usage Method

Fixed IP is suitable for scenarios that are highly dependent on the fixed IP of the container.

In the traditional virtual machine deployment form, some customers rely on the virtual machine IP address for problem troubleshooting, monitoring, traffic allocation, etc. The support for fixed IP can help users to migrate better from virtual machines to containers and improve operation and maintenance efficiency. For businesses that do not restrict IP, it is not recommended to use the fixed IP mode.

Fixed IP only supports StatefulSet form of resource controller.

1. Fixed IP Plugin Installation and Upgrade

Please install the related plugins first through the UK8S console Application Center -> Fixed IP Management feature. Subsequent plugin version updates can also be operated on this page.

2. Create a StatefulSet of Fixed IP Type

Currently, it only supports creating a StatefulSet of Fixed IP type through Yaml. You need to add relevant comments in spec.template.annotations for configuration. Subsequent versions will support creation through console forms:

The following is a YAML template for creating a StatefulSet type Nginx application and exposing it externally.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: nginx
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web-test
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx 
  serviceName: "nginx"
  replicas: 5 
  template:
    metadata:
      annotations:
        # Declare the need to enable the fixed IP function
        network.beta.kubernetes.io/ucloud-statefulset-static-ip: "true"  
        # Set the VpcIP release time to 300 seconds       
        network.beta.kubernetes.io/ucloud-statefulset-ip-claim-policy: "300s"
      labels:
        app: nginx 
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: uhub.ucloud-global.com/ucloud/nginx:1.9.2
        ports:
        - containerPort: 80
          name: web

3. Explanation of Exceptional Situations

1. Node Hangs / Remains

The node hangs or is removed from the cluster, which will trigger the forced migration of the Pod. The VpcIP bound to the Pod will be retained. After the Pod is scheduled to the new Node, the management service will update and record the corresponding mapping relationship.

If there was a fixed IP Pod on this node, after scaling down, this Pod is terminated, and it is not pulled up again or scheduled to other nodes due to scaling up, etc., the VpcIP of this Pod will be kept on the node. When the node is removed, this VpcIP will be unbound from the node and released because it is not occupied by any Pod. If this Pod is pulled up again on other nodes at this time, it may not be able to pull up the Pod because this VpcIP has been occupied by another application. Therefore, it is necessary to confirm that there are no VpcIPs not occupied by Pods on the node before removing the node. For the specific steps, please refer to “4. Node Offline Steps”.

2. Node is Forced to Delete

This refers to the deletion of the resources of the Node’s host on the cloud host page (not through the UK8S cluster management function). In this case, the UK8S management service cannot retain the VpcIP operation, and the VpcIP will be forcibly released by the host service and there is a possibility of being occupied by other resources.

The fixed IP component will try to re-apply for the old VpcIP in the form of a specific IP and bind it to the newly pulled up Pod, but if the corresponding VpcIP is occupied, it will fail to update.

3. Pods in Different VPC Networks Appear in the Same StatefulSet

The fixed IP function does not support cross-VPC subnet. Under the CNI working principle, Pod and its Node are in the same VPC subnet. If the Pod is scheduled to a non-same subnet Node in the situation of StatefulSet expansion, Pod area update, Node node downtime, etc., Pod creation/update failure error will occur.

It is recommended that you reasonably allocate your subnets and network segments, avoid multiple subnets in the cluster, or specify the StatefulSet to be scheduled to the same subnet Node by tags etc.

4. Node Offline Steps

1. Confirm the Mac Information of the Node to be Offline

Login to the node that needs to be deleted, and check the Mac address of the eth0 network card through the ifconfig command.

The fixed IP plugin records the corresponding relationship between VpcIP, Pod, and running nodes through the custom resource object vpcIpClaim. If there is an unevicted fixed IP Pod on the node, you can also check the Node that runs the fixed IP Pod by kubectl get pods -o wide, and then view the CRD information through kubectl describe vpcIpClaims <pod-name> In CRD, Status.Mac can determine the Mac address of the host.

2. Find the Fixed IP Belonging to the Offline Node through the Mac Address

Use the kubectl command to find out the CRD object information of the unused VpcIP on the node.

# In the grep command, replace the mac address after the grep command with the mac address of the node to be offline
kubectl get vpcIpClaims -l attached=false -o=json | jq '.items[]|.metadata.name + " " + .status.mac' | grep 52:54:00:26:6E:DA
"web-test-11 52:54:00:26:6E:DA"
"web-test-13 52:54:00:26:6E:DA"
"web-test-14 52:54:00:26:6E:DA"

This display result indicates that the Pods web-test-11, web-test-13, web-test-14 have been run on the node with Mac address 52:54:00:26:6E:DA.

3. Adjust the Number of StatefulSet Replicas to Ensure that the Fixed IP is Occupied

Execute kubectl patch sts web-test -p '{"spec":{"replicas":15}}', adjust the Sts replicas number to 15, ensure web-test-14 is pulled up and the corresponding fixed IP is occupied.

Execute kubectl drain <node-name> --ignore-daemonsets, empty the node to be offline, if the node has been disabled and emptied, this step can be ignored.

4. Node Offline

Execute kubectl get vpcIpClaims -l attached=false -o=json | jq '.items[]|.metadata.name + " " + .status.mac' | grep 52:54:00:26:6E:DA, make sure that there is no unoccupied VpcIP on the node, remove the node on the UK8S console.

Finally, execute kubectl patch sts web-test -p '{"spec":{"replicas":<new-replicas>}}' again, and adjust the number of Sts replicas to the expected value.