Snapshot Repository

S3 Repository

Create S3 bucket and required roles

The main steps in the AWS tutorial were already taken for our current organization. We have the es-snapshop-role, and all the users member of the Developers group already have the policy to assume this role.

If you still wanna procceed and register the role and s3 bucket for the repository, please follow this section of the tutorial: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains-snapshots.html#es-managedomains-snapshot-prerequisites.

Add repository to cluster

The script below is needed to add a new repository to an elasticsearch cluster. The user providing the access keys for this script must have the correct policy to take on the es-snapshot-role; otherwise, ES won't allow the repository to be created. In case you want to add the repository to a cluster behind a VPC (like UAT), you need to run this script through a box with access to the VPC directly.

In order to run it, you must save it to a file index.js, install the dependencies npm install, and then run it node index.js.

The repository settings we currently have are:

REPOSITORY-NAME = s3-snapshot
BUCKET-NAME = es-titanhouse-snapshots

Script:

const elasticsearch = require('elasticsearch');
const AWS = require('aws-sdk');
const connectionClass = require('http-aws-es');

AWS.config.update({
  accessKeyId: <YOUR-ACCESS-KEY>,
  secretAccessKey: <YOUR-SECRET-ACCESS-KEY>,
  region: 'us-east-1',
});

const clientConfig = {
  // host: 'https://vpc-elastic-uat-o5sx4clul237c6at4r7k7g7o2m.us-east-1.es.amazonaws.com', // UAT
  host: 'https://search-elastic-77g6cnna6wzwkbx55e2kzbm75y.us-east-1.es.amazonaws.com', // QA
  awsConfig: AWS.config,
  connectionClass,
};

const client = new elasticsearch.Client(clientConfig);

const { snapshot } = client;

new Promise((resolve, reject) => snapshot.createRepository({
  verify: true,
  repository: '<REPOSITORY-NAME>',
  body: {
    type: 's3',
    settings: {
      bucket: '<BUCKET-NAME>',
      endpoint: 's3.amazonaws.com',
      role_arn: 'arn:aws:iam::892443542636:role/es-snapshot-role',
    },
  },
}, (err, data) => {
  if (err) return console.error(err);
  return console.log(data);
}));

Create snapshot

To create a snapshot does not difer from other interactions with the ES cluster. All you need to do is issue an HTTP request to the cluster with a certain body to parameterize the snapshot you are taking. The following body is the default being used when we take any snapshot from our clusters.

PUT /_snapshot/<REPOSITORY-NAME>/<SNAPSHOT-NAME>
{
  "indices: "",
  "ignore_unavailable": true,
  "include_global_state": false,
  "include_aliases": false
}

The indices property is optional. If not present, the snapshot will contain every index present in the cluster. If you do set it, though, make sure to use the index name and not the aliases.

Status

Once you issued the create HTTP request, you can gather its status by issuing another HTTP request to the cluster.

GET /_snapshot/<REPOSITORY-NAME>/<SNAPSHOT-NAME>

Restore snapshot

Once the snapshot is created, and its state is SUCCESS, we can start the restore procedures in any cluster with access to the repository. Like any other action in ES, you can start the restore process through a simple HTTP request

POST /_snapshot/<REPOSITORY-NAME>/<SNAPSHOT-NAME>/_restore
{
  "indices": "",
  "ignore_unavailable": true,
  "include_global_state": false,
  "include_aliases": false
}

The indices property here is also optional. If not present, the process will restore every index in the snapshot; otherwise, only the specified indexes will be restored. In order to see which indexes are present in a snapshot, you can use the status call above GET /_snapshot/<REPOSITORY-NAME>/<SNAPSHOT-NAME>

Status

To retrieve the status of a restore, you will need to do gather it per-index. As always, you will need to issue an HTTP request.

/<INDEX-NAME>/_recovery?human
/<INDEX-NAME-1>,<INDEX-NAME-2>,<INDEX-NAME-3>/_recovery?human

Once you get the response back, you want to look at the records whose type is "type" : "SNAPSHOT". Example:

{
  "titan_20181022065813": {
    "shards": [
      {
        "id": 4,
        "type": "SNAPSHOT",
        "stage": "DONE",
        "primary": true,
        "start_time": "2018-10-23T20:06:15.677Z",
        "start_time_in_millis": 1540325175677,
        "stop_time": "2018-10-23T20:08:51.564Z",
        "stop_time_in_millis": 1540325331564,
      }
    ]
  }
}