Create Custom Image Node

I. Introduction

To meet users’ personalized needs for UK8S nodes, besides standard images, UK8S Node supports custom images as well. However, be sure to use the standard image of UK8S to produce custom images; otherwise, the cluster may not be created or nodes may not be added.

The following will introduce how to create custom images based on standard ones, as well as points to note. The process of creating a custom image introduced in this document is fully automated, requiring no manual intervention during the process. Users need to possess a basic level of shell programming or Ansible experience.

Since the network speed from Hong Kong to mainland China and other global available zones is relatively fast, it can reduce the time consumption when copying images. The method introduced in this article is to construct the image in the Hong Kong region and then copy it to other available zones. Please ensure that there is enough quota for cloud servers in the Hong Kong region.

II. Process of Creating Custom Image

1. Install Packer

Install Packer tool, with this tool, you can easily create and distribute custom images to the available zones you need. The installation method for macOS is introduced below. For other environments, please refer to the Packer Manual.

macOS users can install Packer using the following command:

brew install packer

Packer is only responsible for creating cloud servers. Command-line scripts or Ansible are needed for software installation and configuration in the cloud server. The example given in this document utilizes Ansible, but it can be converted to other equivalent tools. The installation method of Ansible in macOS is introduced below. For other environments, please refer to the Ansible Official Manual.

macOS users can install Ansible using the following command:

brew install ansible

2. Prepare Public Key, Private Key, and Project ID

Please create or use an existing public key and private key in the Account Management -> API Key of the UCloud Global console. Find the project that holds the custom image you are about to create in the Access Control -> Project Management of the UCloud Global console. Set the public key, private key, and project ID into the environment variables. The command example is as follows:

export UCLOUD_PUBLIC_KEY="Public key"
export UCLOUD_PRIVATE_KEY="Private key"
export UCLOUD_PROJECT_ID="Project ID"

It is suggested to set the above commands in the shell’s initialization file, such as .zshrc or .bashrc, etc.

3. Write Packer Configuration File

Assume that the name of this configuration file is custom.json.

{
  "variables": {
    "ucloud_public_key": "{{env `UCLOUD_PUBLIC_KEY`}}",
    "ucloud_private_key": "{{env `UCLOUD_PRIVATE_KEY`}}",
    "ucloud_project_id": "{{env `UCLOUD_PROJECT_ID`}}"
  },

  "builders": [{
    "type": "UCloud Global-uhost",
    "public_key": "{{user `ucloud_public_key`}}",
    "private_key": "{{user `ucloud_private_key`}}",
    "project_id": "{{user `ucloud_project_id`}}",
    "region": "hk",
    "availability_zone": "hk-02",
    "instance_type": "o-standard-2",
    "source_image_id": "<REPLACE_THE_UK8S_BASE_IMAGE_ID_HERE>",
    "ssh_username": "root",
    "image_name": "<YOUR_IMAGE_NAME_GOES_HERE>",
    "image_copy_to_mappings": [
      {
        "project_id": "{{user `ucloud_project_id`}}",
        "region": "<REPLACE_REGION_ID_WHERE_TO_COPY>"
      }
    ]
  }],

  "provisioners": [{
    "type": "ansible",
    "playbook_file": "./playbook.yml"
  }]

}

Please first replace the content in the angle brackets in the above example with actual values. The following table lists the base image IDs corresponding to the supported operating systems and versions under the Hong Kong availability zone for UK8S. Please choose an appropriate image as needed, and replace <REPLACE_THE_UK8S_BASE_IMAGE_ID_HERE> with the value corresponding to the Image ID column:

If you need to copy the finished image to other regions and availability zones, you can set the target availability zone in the image_copy_to_mappings of the file above. Multiple can be specified at the same time. If you don’t need to copy, please delete this attribute.

Next, we need to write a script to install and configure a custom image. This document provides an Ansible example for reference. Packer has other types of provisioners, please refer to the Packer Manual. The corresponding playbook.yml for Ansible is as follows:

- hosts: all
  become: true

  pre_tasks:
    - name: Disable swap
      command: swapoff -a
      when: ansible_swaptotal_mb > 0

  roles:
    - role: custom-setup

The playbook in the above example will disable swap and perform further settings with the role named custom-setup. Please note that this is just a demonstration; the swap has been disabled in the base image of UK8S, so there’s no need to repeat this operation.

4. Run Packer

First time running Packer, please run it in the directory that contains the above custom.json file:

packer init .

If there are problems running the above command, please configure the config.pkr.hcl file in the running path. The file content is as follows:

packer {
  required_plugins {
    ucloud = {
      version = ">= 1.0.8"
      source  = "github.com/ucloud/ucloud"
    }
  }
}

Then run the following command:

packer build custom.json

The process of creating the image is relatively time-consuming, please do not perform any operations in the cloud server during this period, or delete this server, otherwise, the image cannot be created normally. After the creation is complete, packer will display the ID of the image, as demonstrated below:

==> UCloud Global-uhost: Stopping instance "uhost-88888888888"
    UCloud Global-uhost: Stopping instance "uhost-88888888888" complete
==> UCloud Global-uhost: Creating image xxxx-yyyyy-8.5...
    UCloud Global-uhost: Waiting for the created image "uimage-***********" to become available...
    UCloud Global-uhost: Creating image "uimage-***********" complete
==> UCloud Global-uhost: Copying images from "uimage-***********"...
    UCloud Global-uhost: Copying image from org-******:cn-bj2:uimage-*********** to org-******:cn-wlcb:uimage-***********
    UCloud Global-uhost: Copying image from org-******:cn-bj2:uimage-*********** to org-******:hk:uimage-***********
    UCloud Global-uhost: Copying image from org-******:cn-bj2:uimage-*********** to org-******:cn-gd:uimage-***********
    UCloud Global-uhost: Waiting for the copied images to become available...
    UCloud Global-uhost: Copying image complete
==> UCloud Global-uhost: Deleting instance...
    UCloud Global-uhost: Deleting instance "uhost-88888888888" complete
Build 'UCloud Global-uhost' finished after 19 minutes 43 seconds.

After the custom image is created, Packer will automatically delete the cloud server, so there’s no need to worry about unnecessary expenses due to forgetting to delete the server.

III. Points to Note

UK8S’s base image comes with pre-configured dependencies for the deployment of Kubernetes, such as software, file directories, and kernel parameters, etc. When creating a custom image based on UK8S’s base image, please be careful not to modify the related settings as to not cause troubles when creating nodes based on this custom image. The points to note during the process of creating a custom image are briefly explained below.

3.1 System-related

1. Swap is disabled by default, do not enable it;
2. Storage=persistent is configured in journald parameters, not recommended for modification;
3. The directories listed below are created by default, do not delete or modify;

• /etc/kubernetes/ssl
• /etc/etcd/
• /etc/docker/
• /etc/kubelet.d/
• /var/lib/kubelet
• ~/.kube/
• /var/lib/etcd/
• /var/lib/etcd/default.etcd
• /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ucloud~flexv/
• /etc/kubernetes/yaml

4. The ip_conntrack module is loaded, do not modify;
5. IPV6 is disabled by default, do not modify
6. For Anolis (Dragon Lizard) operating system 8.x version, firewalld must be turned off, do not turn it on when creating the custom image

3.2 Software Part

UK8S node initialization depends on the software listed below (part of it), do not uninstall.

• iptables
• ipvsadm
• socat
• nfs-utils (used to mount UFS)
• conntrack
• earlyoom

When initializing the UK8S node, it will copy pre-generated certificate files, configuration files, binary files (kube-proxy, kubelet, scheduler, docker, kubectl, etc.) to the node and start them in sequence. Therefore, when creating a custom image, there is no need to install K8S-related components. Even if they are installed, they will not be put to use, but they might interfere with the UK8S management program and cause the creation of the cluster to fail.