Tool Introduction

Overview

US3SYNC is a migration tool designed to synchronize data from different sources to US3. By deploying US3SYNC on local machines or cloud hosts, users can easily migrate data from local environments or other cloud platforms to US3 storage. US3SYNC supports two types of tasks:

• Standard Mode: Synchronizes files from a specific prefix or folder on the source to US3 in bulk.
• Fetch Mode: Synchronizes files by receiving a POST request (specifying the file URL and destination). It supports success and failure callback URLs, sending notifications upon task completion.

How Standard Tasks Work

In standard mode, the workflow is divided into a master node and worker nodes:

• Master Node:

Deployed as a single instance, it manages migration tasks. Its main responsibility is to retrieve file lists from the source and distribute tasks to worker nodes for migration.

• Worker Nodes:

These are scalable nodes responsible for file migration. They download files from the source and upload them to the destination.

The master and worker nodes can be deployed on the same machine or on multiple machines. Users can expand the number of worker nodes as needed. Deployment details:

• Same Machine Deployment:

The master and worker nodes communicate through an internally configured communication address. Ensure the path configured for worker nodes is unique and does not overlap with the master node or other worker nodes.

• Different Machine Deployment:

The master and worker nodes communicate through a configured communication address accessible from the worker machine. Ensure the path configured for each worker node is unique and non-overlapping.

How Fetch Tasks Work

Fetch tasks operate similarly to standard tasks, except that the source for synchronization is specified via a POST request, rather than being automatically retrieved from a pre-configured source.

POST Request Syntax

POST /fetch/ HTTP 1.1
Authorization: Auth
Content-Type: application/json

Request Headers

• Authorization: Required
• Other headers like content-type and date are optional.

Example: Authorization: UCloud Global xxxx-xxxx-xxxx-xxxx-xxxx:xxxxxxxxxxxxx Pseudo-code for generating Authorization:

method := "POST"
md5 := xxxxxx
contentType := xxxxxx
date := xxxxx
privateKey = xxxxx-xxxx-xxxxx-xxxx
publicKey = xxxx-xxxx-xxxx-xxxx
strToSign = method + "\n" + md5 + "\n" + contentType + "\n" + date + "\n"
signature = HmacSHA1(strToSign, privateKey)
signature = Base64(signature)
Authorization: "UCloud Global " + publicKey + ":" + signature

Request Body

The JobId can be obtained from the interface.

Response Body

Example response:

{
"RetCode": 0,
"ErrMsg": "Success",
"TaskId": "d4d62b79-b292-411a-a1f2-47369e2b532f"
}

Example Request

POST /fetch/ HTTP/1.1
Authorization: UCloud Global this-is-my-public-key:AAAArandomsignature=
Content-Type: application/json
Content-Length: 159

{
"Url": "http://xxx.xxx.xxx/xxx/movie.mp4",
"Bucket": "example_bucket",
"Key": "movie.mp4",
"JobId": "xxxxxxxxxxxxxxxxxxxx",
"SuccessCallbackUrl": "http://xxx.xxx.xxx/success",
"FailureCallbackUrl": "http://xxx.xxx.xxx/failure"
}

Example Response

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 122
Connection: keep-alive

{
"RetCode": 0,
"ErrMsg": "Success",
"TaskId": "d4d62b79-b292-411a-a1f2-47369e2b532f"
}

Callback Content

You can specify callback URLs for task completion in the request. US3SYNC will send a POST request to the specified URL based on the task result. Example:

• Failure Callback:

{
"Code": 1,
"TaskId": "d4d62b79-b292-411a-a1f2-47369e2b532f",
"Message": "We encountered an internal error.",
"Resource": "http://xxx.xxx.xxx/xxx/movie.mp4"
}

• Success Callback:

{
"Code": 0,
"TaskId": "d4d62b79-b292-411a-a1f2-47369e2b532f",
"ETag": "xxxxxxxx",
"Key": "xxxxxxxxxxxxxxx.mp4",
"SHA1": "0bc51013e87869137a432200f57daf6affdd3d0c",
"Size": 638304718
}

Key Features

1. Supports data migration from S3, OSS, Qiniu, Youpai, and US3 cloud storage to US3.
2. Supports migrating data from NAS storage or local directories to US3.
3. Allows data migration from specified URL resource lists to US3.
4. Provides web-based management for migration tasks and nodes.
5. Enables source management.
6. Supports scheduling tasks for periodic execution.
7. Allows export of failure file lists.

Note: Migration from source files in archive mode is not supported.

Directory Structure

US3SYNC
├── bin
│   ├── master            # Master executable
│   └── worker            # Worker executable
├── conf
│   └── config.toml       # Configuration file
├── cert                  # HTTPS certificates
├── log                   # Master log file directory
├── pika                  # Dependency package
└── console.sh            # Startup script

Version and Environment

Software Version

Current Version: 1.10.4

Operating Environment

• Linux:
• CentOS 7.0 and above (check with cat /etc/redhat-release)
• Ubuntu 16.04 and above (check with cat /etc/issue)