Instance Migration

Create an Index

Data migration is essentially the rebuild of indices. The rebuilding of an index doesn’t try to set the target index, it won’t copy the settings of the source index. So, set up the target index before the operation, including setting mappings, number of shards, replicas, etc.

Data Migration

1. Reindex from Remoteedit

Reindex supports rebuilding of index from a remote Elasticsearch cluster:

POST _reindex
{
  "source": {
    "remote": {
      "host": "http://otherhost:9200",
      "username": "user",
      "password": "pass"
    },
    "index": "source",
    "query": {
      "match": {
        "test": "data"
      }
    }
  },
  "dest": {
    "index": "dest"
  }
}

# The host parameter must include scheme, host, and port (for example, https:// otherhost: 9200)
# The username and password parameters are optional

When using, you need to configure the reindex.remote.whitelist property in elasticsearch.yml. Multiple groups can be set (for example, otherhost:9200, another:9200, 127.0.10.:9200, localhost:).

For specific usage, refer to Reindex from Remoteedit

2. Elasticsearch-Dump

Elasticsearch-Dump is an open-source tool for importing and exporting Elasticsearch data. Installation and migration-related executions can be performed on a cloud host in the same availability zone, which is convenient to use.

** Installing **

Node environment is needed, and elasticdump is installed through npm

npm install elasticdump -g
elasticdump

** Use **

# Copy an index from production to staging with analyzer and mapping:
elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=http://staging.es.com:9200/my_index \
  --type=analyzer
elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=http://staging.es.com:9200/my_index \
  --type=mapping
elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=http://staging.es.com:9200/my_index \
  --type=data

# Copy a single shard data:
elasticdump \
  --input=http://es.com:9200/api \
  --output=http://es.com:9200/api2 \
  --params='{"preference" : "_shards:0"}'

For other usage of the elasticdump command, please refer to Elasticdump Options

3. Elasticsearch-Migration

Elasticsearch-Migration is an open-source tool developed in Go. Like Elasticsearch-Dump, related operations can be performed on a cloud host in the same availability zone.

** Download **

Download Link

** Use **

./bin/esm -s http://192.168.1.x:9200/ -d http://192.168.1.y:9200/ -x src_index -y dest_index -w=5 -b=100

# Options
  -s, --source=     source Elasticsearch instance
  -d, --dest=       destination Elasticsearch instance
  -q, --query=      query against source Elasticsearch instance, filter data before migrating, ie: name: medcl
  -m, --source_auth basic auth of source Elasticsearch instance, ie: user:pass
  -n, --dest_auth   basic auth of target Elasticsearch instance, ie: user:pass
  -c, --count=      count of documents at a time: ie "size" in the scroll request (10000)
  --sliced_scroll_size=      size of sliced scroll, to make it work, the size should be > 1, default: "1"
  -t, --time=       scroll time (1m)
      --shards=     set a number of shards on newly created indexes
      --copy_settings, copy index settings from source
      --copy_mappings, copy mappings mappings from source
  -f, --force       delete destination index before copying, default: false
  -x, --src_indexes=    list of indexes to copy, comma-separated (_all), support wildcard match (*)
  -y, --dest_index=    index name to save, allow only one index name, original index name will be used if not specified
  -a, --all         copy indexes starting with . and _ (false)
  -w, --workers=    concurrency number for bulk workers, default is: "1"
  -b  --bulk_size   bulk size in MB" default: 5
  -v  --log         setting log level, options: trace, debug, info, warn, error
  -i  --input_file  indexing from local dump file, file format: {"_id":"xxx","_index":"xxx","_source":{"xxx":"xxx"},"_type":"xxx"  }
  -o  --output_file output documents of source index into local file, file format same as input_file.
  --source_proxy     set proxy to source HTTP connections, ie: http://127.0.0.1:8080
  --dest_proxy       set proxy to destination HTTP connections, ie: http://127.0.0.1:8080
  --refresh          refresh after migration finished

For precautions of Elasticsearch-Migration, please refer to the reference link

Instance operation and maintenance Fault recovery