UFS Used in Dynamic PV

Background

Previously, we’ve discussed the use of UFS in UK8S by creating static PVs. This method, however, presents two issues: It requires manual creation of PVs and PVCs, which is quite inconvenient; and it fails to create subdirectories in UFS automatically, hence the need for pre-configuration.

Next, we introduce an open-source project called nfs-subdir-external-provisioner. It can be found at nfs-subdir-external-provisioner. The project enables us to use a StorageClass based on UFS. In instances where UFS storage resources are required, all you need to do is create a PVC. The nfs-client-provisioner will then automatically create the PV and establish a ${namespace}-${pvcName}-${pvName}-named subdirectory under the UFS.

Principle of Operation

We use environment variables to transmit nfs parameters to the nfs-client-provisioner, which is then run by the Deployment controller to manage nfs storage. After server startup, we generate a StorageClass with a provisioner that is identical to the provisioner-name in the nfs-client-provisioner service. The nfs-client-provisioner watches for PVCs in the cluster and offers matching PV services, simultaneously creating corresponding directory under the nfs root directory.

The details of this process are well documented in the official guide, so they will not be further addressed here.

In the next part, we’ll cover how to use this service in the context of UK8S to manage UFS.

Operation Guide

Clone the project

In the deploy/ directory, we focus on three files, namely rbac.yaml, deployment.yaml, class.yaml.

# git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git
# cd nfs-subdir-external-provisioner/deploy/
# ls
class.yaml         kustomization.yaml rbac.yaml          test-claim.yaml
deployment.yaml    objects            test-pod.yaml

Modify the nfs-client-provisioner service namespace

Both the nfs-client-provisioner service and its required RBAC resources will be deployed in the system plugin’s kube-system namespace.

sed -i "s/namespace:.*/namespace: kube-system/g" ./rbac.yaml ./deployment.yaml

Modify deployment.yaml

The nfs-client-provisioner service is launched by mounting UFS. Although the official document does this by declaring spec.volume.nfs, we change it here to the method of declaring PV statically. Here’s why:

When mounting UFS to our host in the cloud, it’s required to assign extra mountOption parameters. However, spec.volume.nfs does not support this parameter. On the other hand, PersistentVolume declaration does support it, hence we choose to mount the static PV method for the first mount.

There are numerous modifications needed in the deployment.yaml file, so direct replacement is preferred:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          # use the uhub provided mirror for faster retrieval speed
          image: uhub.ucloud-global.com/uk8s/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: ucloud/ufs   # modify to UCloud Global/ufs
            - name: NFS_SERVER
              value: 10.9.x.x     # modify to UFS server address here
            - name: NFS_PATH
              value: /            # modify to UFS mount path here
      volumes:
        - name: nfs-client-root
          persistentVolumeClaim:  # change from nfs to persistentVolumeClaim
             claimName: nfs-client-root
---
# create PVC
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-client-root
  namespace: kube-system
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 200Gi
---
# manually create PV and specify mountOption
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-client-root
spec:
  capacity:
    storage: 200Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /
    server: 10.9.x.x # directly put the UFS server address here
  mountOptions:
    - nolock
    - nfsvers=4.0

Modify class.yaml

The final modification is to the StorageClass definition file class.yaml. Mainly, we add the mountOption parameter which will be passed to the nfs-client-provisioner. Failure to include this parameter will cause a failure when mounting the UFS.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: ucloud/ufs
parameters:
  onDelete: "retain"
mountOptions:
  - nolock
  - nfsvers=4.0

Deploy

Execute in order

# kubectl create -f rbac.yaml
# kubectl create -f deployment.yaml
# kubectl create -f class.yaml

Validation

Create test-nfs-sc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
spec:
  storageClassName: nfs-client
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi
---
kind: Pod
apiVersion: v1
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: uhub.ucloud-global.com/uk8s/busybox:1.31.1
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "echo 1 > /mnt/SUCCESS; sleep 10000000"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: test-claim

Create test pod and verify that UFS is readable and writable:

# kubectl create -f test-nfs-sc.yaml
# kubectl exec test-pod -- /bin/sh -c 'ls /mnt/ && cat /mnt/*'
SUCCESS
1
# kubectl delete -f test-nfs-sc.yaml

Upgrade Guide

The upgrade process from the old version, external-storage, to the new version, nfs-subdir-external-provisioner.

Upgrade Background

The old version provisioner supports up to k8s version 1.23, where 1.20 and above requires --feature-gates=RemoveSelfLink=false support on apiserver.
Version 1.24 and new k8s versions must use the new version provisioner (because the apiserver parameter is no longer supported).

Effect on existing PVCs and pods

PVCs requested using the old provisioner (i.e., storage class managed-nfs-storage) can still be mounted and used after the upgrade, and pod mounts are unaffected. Pods can still be mounted after restart. If a pod or PVC is deleted, the corresponding files on ufs will not be cleaned up. If you need to release space, you should manually delete files.

Upgrade Process

Refer to the new provisioner deployment document (see earlier sections of this document) for upgrading process.

Use UFS in UK8S Using US3 in UK8S