Quick Start

• Quick Start
  • Interface Overview
    • Task Management Interface
    • Create Task Interface
    • Create Node Interface
    • Create Source Interface
  • Instructions
    • Add Endpoint
    • Create Task
    • Start Migration
    • Pause Migration
    • Retry
    • Export Failed File List
  • Cache Explanation
  • Migration Type Overview
    • S3
    • local
    • http

Interface Overview

Once the service is started, open your browser and access the US3SYNC interface at: https://<web service listening IP>:<web service listening port>. You can perform migration operations directly through the interface.

Task Management Interface

• Task Operations:

  • Start Migration: Begin file migration.
  • Start Validation: Perform file validation by comparing file sizes between source and destination directories, fetching full file lists from both.
  • Manage - Retry: Retry migration for files that failed in previous attempts.
  • Manage - Export: Export a list of failed files to the .US3SYNC/details/[jobid]/error.log location in the destination bucket.
  • Manage - Delete: Delete a task and clear its cache records.
  • Manage - Clone: Open the task creation interface with the current task’s information pre-filled.

• Statistics:

=================================
Total Files:    Total number of files to migrate.
Pending Files:  Number of files waiting to migrate.
In Progress:    Number of files currently being migrated.
Successful:     Number of successfully migrated files.
Failed:         Number of files that failed to migrate.
=================================

• Error Log Export: If migration tasks fail, use the Export button in the task management interface to export error logs. The logs will be uploaded to US3 and a downloadable URL will be provided.
  You can download the logs using wget from the node machine and troubleshoot the issue by referring to the FAQ.

Create Task Interface

• Task Type: Choose the migration task type.
• Regex Match Keys: Use regular expressions to filter the source list.
• Schedule Execution: Configure periodic execution.
• Sync Deletion: Remove files in the destination that do not exist in the source.
• Job ID: Click the expand icon next to the task name to obtain the Job ID.

Create Node Interface

• IP Address: Enter the worker node’s IP address.
• Username: Enter the SSH username for the worker node.
• Password: Enter the SSH password for the worker node (leave blank for password-less login).

Create Source Interface

• Timeout: Specify upload/download timeout (default is 60s).
• Domain: Enter the bucket’s domain suffix, e.g., cn-bj.ufileos.com.
• Prefix: Use PrefixFileList to input multiple prefixes.

Instructions

Add Endpoint

Add a migration source by clicking the Create Endpoint button. Fill in the required information in the dialog box and click OK.

Create Task

Create a migration task by clicking the Create Task button. Fill in the required information in the dialog box and click OK. To create a fetch task, select the Fetch Task option in the interface. To obtain the Job ID, click the expand icon next to the task name.

Migration Process:

• Files under the specified source prefix will be migrated to the specified destination prefix.

Examples:

1. Source Prefix: A/ | Destination Prefix: B/
All files under A/ will be migrated to B/.
Example: Source file A/a.txt will become B/a.txt at the destination.

2. Source Prefix: A/ | Destination Prefix: (empty)
All files under A/ will be migrated to the root directory of the destination.
Example: Source file A/a.txt will become a.txt at the destination root.

Overwrite Options:

• Full Overwrite: Force upload all files from the source to US3.

• Conditional Overwrite: Migrate files based on specified criteria:

  • File type
  • File size
  • File timestamp (migrate files that are newer in the source).

• No Overwrite: Skip files that already exist in the destination.

Scheduled Tasks:

• Configure migration to run periodically, specifying the interval and start time.

Incremental Filters:

• Absolute Time Filter: Specify a start time for file migration.
• Recent Time Filter: Specify files modified in the last few hours. For periodic tasks, the filter adjusts based on the current time.

Regex Matching:

• If enabled, the source list will be filtered using the specified regex.
• The regex does not match the source prefix, allowing finer control within a prefix.

Add Alerts

• Notifications are sent via POST requests for task completions or worker failures.
• Collect error metrics for migration and fetch tasks.
• Configure custom alert rules for task errors using string-based logic. Notifications are sent via POST requests upon rule triggers.

Example Alert Logic:

(RESOURCE_ERR_4XX > 100 || RESOURCE_ERR_5XX > 200) && TARGET_ERR_4XX > 150 && Q_LEN > 150

Supported Variables:

• RESOURCE_ERR_4XX: Number of 4XX errors from the source.
• RESOURCE_ERR_5XX: Number of 5XX errors from the source.
• TARGET_ERR_4XX: Number of 4XX errors from the destination.
• TARGET_ERR_5XX: Number of 5XX errors from the destination.
• LOCAL_ERR: Number of other errors.
• Q_LEN: Number of files in the fetch task queue.

Start Migration

Refer to the Interface Overview and click the Start Migration button.

Pause Migration

Refer to the Interface Overview and click the Pause button.

Retry

Retry migrating failed files by clicking Manage - Retry in the Task Management Interface.

Export Failed File List

Export the list of failed files by clicking Manage - Export in the Task Management Interface.

Cache Explanation

US3SYNC uses Pika to provide caching services. Cached data includes:

• Task details
• Migration statistics
• Migration logs

Migration Type Overview

S3

AWS S3 sources require an additional region field. Refer to: https://docs.aws.amazon.com/zh_cn/general/latest/gr/s3.html.

For third-party S3 implementations, consult the respective documentation.

local

local sources require the directory path to be provided.

http

http sources require a URL list of files to be migrated. Each line in the list should represent a file URL:

<file URL>
<file URL>