US3 Plugin

The US3 plugin supports backing up Elasticsearch data snapshot to US3 and restores data to Elasticsearch from the snapshots already backed up to US3.

I. Install Plugin

The US3 plugin can be installed via the “Plugin Management” feature in the UES console. The plugin name is elasticsearch-repository-ufile, and for the specific operation method, please refer to the document Plugin Management section.

II. Create US3 Storage

1. Create Storage

Create a storage space for UES data backup in the US3 console. To ensure the quality of network transmission, please create it in the same region as the target UES cluster. Please select “Private Space” for the space type.

2. Create a Token

Create a token in the US3 console for accessing the above storage space. Please note to authorize the previously created storage space and check “Upload”, “Download”, “Delete”, “File List” and so on in the “Token Rights”.

3. Obtain the Private Domain Name and Key Information for the Storage Space

Retrieve and record the following information for future use:

（1） Private domain name for the storage space

Please note that the storage space domain name displayed on the US3 console is the public domain name. You need to view the US3 document to obtain its internal domain name. Please refer to the “Region and Domain” section in the US3 document.

Taking the above example created storage space, the public domain name is: ues-backup.uae-dubai.ufileos.com, and the US3 Dubai’s intranet domain name obtained by inquiry is: www.internal-uae-dubai.ufileos.com, hence the internal domain name for the example storage space is: ues-backup.internal-uae-dubai.ufileos.com

（2） Token Key that has the access rights of the above storage space

View the public key and private key of the token on the “Token Management” page of the US3 console.

III. Create a Repository

Before creating a snapshot or restoring data from a snapshot for ES data, a repository needs to be created first.

PUT /_snapshot/<1>
{
    "type": "ufile",
    "settings": {
        "endpoint": <2>,
        "public_key": <3>, 
        "private_key": <4>, 
        "bucket": <5>,
        "compress": <6>,
        "chunk_size": <7>,
        "base_path": <8>,
        "max_snapshot_bytes_per_sec": <9>,
        "max_restore_bytes_per_sec": <10>
    }
}

<1>: ES backup repository name

<2> endpoint: The internal domain name for the above US3 storage

<3> public_key: The public Key for the above US3 token

<4> private_key: The private Key for the above US3 token

<5> bucket: The name of the above US3 storage space

<6> compress: Whether to compress the backup index (true / false)

<7> chunk_size: The chunk size for file upload, default 64MB

<8> base_path: The path of the backup file in the bucket (prefix name), default to the root path (no prefix)

<9> max_snapshot_bytes_per_sec: Upload speed when creating a snapshot, default 40MB/s

<10> max_restore_bytes_per_sec: Download speed when restoring from a snapshot, default 40MB/s

A typical repository creation request example is as follows:

PUT /_snapshot/ues_snapshot
{
    "type": "ufile",
    "settings": {
        "endpoint": "ues-backup.internal-uae-dubai.ufileos.com",
        "public_key": "TOKEN_XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", 
        "private_key": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", 
        "bucket": "ues-backup",
        "compress": true,
        "chunk_size": "50mb",
        "base_path": "ues",
        "max_snapshot_bytes_per_sec": "20mb",
        "max_restore_bytes_per_sec": "20mb"
    }
}

IV. Instructions

For the specific methods of creating a snapshot, restoring data from a snapshot, querying snapshot information, etc., please refer to the official Elastic documentation:

Snapshot And Restore

V. Common Commands

Retrieve all repository information:

GET /_snapshot?pretty

Retrieve a repository’s information:

GET /_snapshot/<repository>?pretty

Retrieve all snapshots in a repository:

GET /_snapshot/<repository>/_all?pretty

Retrieve a specific snapshot:

GET /_snapshot/<repository>/<snapshot>?pretty

Create a snapshot:

PUT /_snapshot/<repository>/<snapshot>

Create a snapshot for a specific index:

PUT /_snapshot/<repository>/<snapshot>
{
    "indices": "<index1>[,index2]..."
}

Restore a specific snapshot:

POST /_snapshot/<repository>/<snapshot>/_restore

Delete a specific snapshot:

DELETE /_snapshot/<repository>/<snapshot>