ILM & Reindexing Automation Runbooks

Registering an S3 Snapshot Repository

Register an S3-backed snapshot repository so Elasticsearch can write backups and searchable snapshots into an object-storage bucket — the correct order being keystore first, then registration, then verification.

This is the concrete, S3-specific walkthrough of a task that the broader managing snapshot repositories reference frames in the abstract, and it produces the shared dependency that both scheduled backups and the frozen-tier searchable snapshots in the snapshot subsystem depend on. The single idea to hold onto throughout is that an S3 repository has two configuration halves that must never be mixed: the secure half — your access key and secret key — lives in the Elasticsearch keystore on each node, and the insecure half — the bucket name, base_path, and client — lives in the registration request. Reverse that split and you either leak credentials into cluster state or register a repository that no node can authenticate against.

Prerequisites

  • Elasticsearch 8.x, where the repository-s3 module is bundled — no plugin install is needed to register an S3 repository.
  • An existing S3 bucket (es-snapshots-prod here) and an IAM identity granting s3:ListBucket on the bucket plus s3:GetObject, s3:PutObject, and s3:DeleteObject on es-snapshots-prod/prod-cluster/*.
  • Shell access to every master and data node to add keystore entries, since the keystore is per node.
  • manage cluster privilege to register the repository, and elasticsearch-py v8.0+ for the automation step.

Implementation

1. Add the access keys to each node’s keystore

The access key and secret key are secure settings. Add them with the keystore CLI on every node — a node without these entries will authenticate with nothing and fail verification later:

# Run on every master and data node.
bin/elasticsearch-keystore add s3.client.default.access_key
# paste AKIA... when prompted
bin/elasticsearch-keystore add s3.client.default.secret_key
# paste the secret when prompted

# Sanity-check the entries landed on this node:
bin/elasticsearch-keystore list | grep s3.client.default

The default in s3.client.default.access_key is the client name. It is the handle the repository registration will reference; if you later add a second bucket with different credentials, you create a second client name (for example s3.client.backup.access_key) rather than overwriting these.

2. Reload secure settings cluster-wide

Adding keystore entries edits an on-disk file; it does not change what a running node uses to sign requests. Because s3.client.* settings are reloadable, activate them cluster-wide without a restart:

POST _nodes/reload_secure_settings
{
  "secure_settings_password": ""
}

The response lists every node; a node carrying a reload_exception (or one you forgot to run the CLI on) is precisely the node that will be missing from the verification result in step 4. Treat any node not cleanly reloaded as unconfigured.

3. Register the repository

With live credentials on every node, register the pointer. The request carries only the non-secret shape of the repository — never the keys:

PUT _snapshot/prod-s3-repo
{
  "type": "s3",
  "settings": {
    "bucket": "es-snapshots-prod",
    "base_path": "prod-cluster",
    "client": "default"
  }
}

Three settings do the work. bucket names the existing S3 bucket; base_path is the per-cluster prefix under which every object is written, so es-snapshots-prod/prod-cluster/...; and client links the registration to the s3.client.default.* keystore entries from step 1. If your bucket lives in a non-default region or behind a custom endpoint, add region or endpoint alongside these — they are insecure settings and belong here in the request, not in the keystore. A 200 with {"acknowledged": true} confirms the definition is in cluster state, but does not prove any node can reach the bucket; that is step 4’s job.

4. Register and verify with the Python v8 client

For a version-controlled deploy, drive registration and verification through the v8 client. create_repository uses PUT semantics so it is safe to re-run, and any API/HTTP failure — including a wrapped RepositoryVerificationException — arrives as ApiError, which is a sibling of TransportError, not a subclass, so catch ApiError:

import logging
from elasticsearch import Elasticsearch, ApiError, NotFoundError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("s3_repo")


def register_and_verify(es: Elasticsearch, name: str) -> bool:
    try:
        # Insecure settings only — the access/secret keys live in the node keystore.
        es.snapshot.create_repository(
            name=name,
            type="s3",
            settings={
                "bucket": "es-snapshots-prod",
                "base_path": "prod-cluster",
                "client": "default",
            },
        )
        logger.info("Repository '%s' registered.", name)
    except ApiError as exc:
        logger.error("Registration failed (%s): %s", exc.status_code, exc.info)
        return False

    try:
        result = es.snapshot.verify_repository(name=name)
        nodes = result["nodes"]
        logger.info("Verified on %d node(s): %s",
                    len(nodes), ", ".join(n["name"] for n in nodes.values()))
        return True
    except NotFoundError:
        logger.error("Repository '%s' vanished before verification.", name)
        return False
    except ApiError as exc:
        # A node missing its keystore entry lands here as a verification ApiError.
        logger.error("Verification failed (%s): %s", exc.status_code, exc.info)
        return False


if __name__ == "__main__":
    es = Elasticsearch(
        "https://es-cluster-01:9200",
        api_key="YOUR_BASE64_ENCODED_API_KEY",
        verify_certs=True,
    )
    register_and_verify(es, "prod-s3-repo")

Verification

The verify_repository call in step 4 is the authoritative check, but confirm the result two ways. First, over HTTP, verify explicitly and confirm every data node appears:

POST _snapshot/prod-s3-repo/_verify
{
  "nodes": {
    "n1x2...": { "name": "es-01" },
    "n3y4...": { "name": "es-02" },
    "n5z6...": { "name": "es-03" }
  }
}

A missing node means that node cannot read or write the bucket — go back and confirm its keystore entry and reload. Second, read the stored definition back with get_repository (or GET _snapshot/prod-s3-repo) to confirm the settings the deployment actually holds match your intent:

definition = es.snapshot.get_repository(name="prod-s3-repo")
print(definition["prod-s3-repo"]["settings"])
# {'bucket': 'es-snapshots-prod', 'base_path': 'prod-cluster', 'client': 'default'}

Gotchas and edge cases

  • Credentials belong in the keystore, never in the settings. An access_key/secret_key placed in the repository settings block would be stored in cluster state, readable through GET _snapshot, and captured in cluster-state backups. Keep them in s3.client.default.* and reference them only by client name.
  • reload_secure_settings is not optional. Adding a keystore entry does nothing to a running node until you reload. The classic failure is a repository that verifies on two of three nodes because the third node’s keystore was edited but never reloaded — the reload is what makes the credential live.
  • base_path must differ per cluster on a shared bucket. Two clusters writing to es-snapshots-prod are safe only if each uses a distinct base_path. Sharing a base_path corrupts the repository generation metadata, because both clusters believe they own the objects at that prefix.
  • Region and endpoint are insecure settings. A non-default bucket region or an S3-compatible endpoint goes in the registration request as region/endpoint, not the keystore. Omitting a required endpoint for an S3-compatible store is a common cause of a verification that fails with a DNS or connection error rather than a permissions one.

FAQ

Do I need to restart nodes after adding keystore credentials?

No. The s3.client.* access and secret keys are reloadable secure settings, so after elasticsearch-keystore add on each node you activate them with POST _nodes/reload_secure_settings. A restart works too but is unnecessary and disruptive; the reload is the intended path and takes effect immediately across every node.

Where do the bucket region and custom endpoint go?

In the repository registration request, alongside bucket and base_path, as region and endpoint. They are insecure settings that describe where the bucket is, not how to authenticate, so they do not belong in the keystore. Only the access key and secret key are secure and keystore-bound; everything else describing the repository is passed in the PUT _snapshot body.

Can I reuse one bucket for several clusters?

Yes, provided each deployment registers the repository with a distinct base_path. The bucket is shared storage and the base_path is the namespace boundary. Two clusters that share both the bucket and the base_path will corrupt each other’s repository metadata, so give each environment its own prefix such as prod-cluster or staging-cluster.

← Back to Managing Snapshot Repositories · SLM & Searchable Snapshots