Managing Snapshot Repositories #

A snapshot repository is a registered pointer to external storage that every node in the deployment must be able to reach and authenticate against, and it is the single shared dependency sitting underneath everything in the snapshot and searchable-snapshot subsystem. Get it right once and it is invisible; get it wrong and the failure surfaces everywhere at the same time — the nightly SLM backup writes nothing, the ILM searchable_snapshot action stalls in the cold phase, and a disaster-recovery restore has no source to read from. Because the repository is consumed by two independent producers (scheduled SLM policies and the ILM lifecycle) and two independent consumers (point-in-time restores and mounted searchable indices), a misconfiguration does not announce itself as one broken feature. It announces itself as a slow, correlated erosion of backup coverage that nobody notices until the restore that was supposed to save the day finds an empty bucket. This page is about registering that repository correctly, proving every node can reach it, and hardening it so the throughput, isolation, and credential handling hold under production load.

The repository is deliberately simple in concept and unforgiving in detail. Registration is a single API call, but the credentials that call depends on live in the Elasticsearch keystore on each node rather than in the request body, and the difference between a working repository and one that silently fails on a single data node is exactly whether that keystore entry was added and reloaded everywhere. What follows treats the repository as the durable foundation it is: the anatomy of the registration, the credential path that makes it authenticate, and the verification that proves it before you trust a schedule to it.

Prerequisites #

• Elasticsearch 8.x cluster — the repository-s3, repository-gcs, and repository-azure modules are bundled, so no plugin install is required to register an object-storage repository.
• An object-storage bucket (es-snapshots-prod in the examples below) with an IAM identity that grants the deployment ListBucket, GetObject, PutObject, and DeleteObject on that bucket and its objects.
• Keystore access on every master and data node so the secure access key and secret key can be added identically across the deployment — a repository is only as reachable as its least-configured node.
• manage cluster privilege on the service account that registers or removes repositories, kept separate from the day-to-day manage_slm/create_snapshot roles per the governance model in the parent overview.
• elasticsearch-py v8.0+ installed (pip install "elasticsearch>=8,<9") for the automation examples, which use the v8 keyword-argument surface (snapshot.create_repository, snapshot.verify_repository), not the legacy body= pattern.
• Network egress from every node to the object-store endpoint, and clocks synchronised by NTP — request signing (S3 SigV4, for example) rejects requests whose timestamps drift too far.

Architecture: One Repository, Every Node, Two Producers #

The mental model that prevents most repository incidents is this: registration is cluster state, but access is per node. When you call PUT _snapshot/prod-s3-repo, the master node records the repository definition into cluster state and every node inherits it — but each node then reaches the bucket independently, signing its own requests with the credentials in its own keystore. There is no proxy node and no shared connection; a data node whose keystore is missing the access key cannot read or write the repository even though the repository “exists” from the master’s point of view. This is why the verify step, which asks every node to write and read a test blob, is the only honest test of whether a repository works.

Two producers share this one pointer. A scheduled SLM policy references it by name in its repository field, and every ILM searchable_snapshot action names it in snapshot_repository. Neither producer holds credentials or knows anything about the bucket — they hand a repository name to Elasticsearch, which resolves it to the registered definition and the per-node keystore. That indirection is the whole point: rotate the bucket credentials or move to a new endpoint, and you touch the keystore and the registration once, not the dozens of policies that consume the repository. It is also why a single repository typically backs the entire deployment; multiplying repositories multiplies the surface where a credential can go stale.

The base_path is the other load-bearing detail in the diagram. Every object this deployment writes lands under es-snapshots-prod/prod-cluster/..., and that prefix is what makes it safe for a second cluster to share the same bucket under a different base_path. Two clusters writing to the same base_path will corrupt each other’s repository metadata, because both believe they own the index-* and snap-* blobs at that prefix. The bucket is shared storage; the base_path is the namespace that keeps two tenants from colliding.

Configuration Reference #

Registration has two halves that must be kept apart: the insecure settings that go in the API request and describe the repository’s shape, and the secure settings — the access key and secret key — that go in the keystore and never appear in the request body or in cluster state. Putting credentials in the repository settings is the most common and most dangerous mistake, because they would then be readable by anyone with GET _snapshot and would sit in cluster-state backups.

The annotated registration below shows every field an S3 repository commonly sets:

PUT _snapshot/prod-s3-repo
{
  "type": "s3",
  "settings": {
    "bucket": "es-snapshots-prod",       // the object-store bucket, must already exist
    "base_path": "prod-cluster",          // per-cluster prefix — MUST differ across clusters sharing a bucket
    "client": "default",                  // which keystore client block supplies the credentials
    "readonly": false,                    // true = mount/restore only, never write (for a DR mirror)
    "max_snapshot_bytes_per_sec": "80mb", // per-node upload throttle to protect ingest bandwidth
    "max_restore_bytes_per_sec": "160mb"  // per-node download throttle for restores and cold mounts
  }
}

The client value is the link between the registration and the keystore: "client": "default" tells Elasticsearch to sign requests with the s3.client.default.* keystore entries. If you run two buckets with different credentials, you register a second client name (say backup) and add a parallel s3.client.backup.access_key set — the registration stays declarative and credential-free.

The credentials themselves are added with the keystore CLI on each node, then activated cluster-wide with a secure-settings reload. This block runs once per node:

# Run on every master and data node — the keystore is per node, not cluster-wide.
bin/elasticsearch-keystore add s3.client.default.access_key
# (paste the access key when prompted)
bin/elasticsearch-keystore add s3.client.default.secret_key
# (paste the secret key when prompted)

# Confirm the entries exist on this node:
bin/elasticsearch-keystore list | grep s3.client.default

Adding the keystore entries does not, by itself, make them live — s3.client.* settings are reloadable secure settings, so a running node picks them up only after an explicit reload rather than a restart:

POST _nodes/reload_secure_settings
{
  "secure_settings_password": ""
}

The reload is what promotes the on-disk keystore change into the running node’s credential store. Skip it on one node and that node keeps signing with nothing — the repository verifies on every node except that one, which is exactly the partial failure the verify step exists to catch.

Step-by-Step Implementation #

Registering a repository is a four-step sequence: seed credentials, activate them, register the pointer, then prove it. Doing them in that order matters — registering before the credentials are live produces a repository that fails verification, which is noise you do not want while you are still setting things up.

1. Add credentials to the keystore and reload #

On each node, add the access and secret keys as shown above, then reload secure settings across the deployment so every node activates them without a restart:

POST _nodes/reload_secure_settings
{
  "secure_settings_password": ""
}

The response lists every node with a reload_exception field that is absent on success. A node that reports an exception here — or that you forgot to run the keystore CLI on — is the node that will later fail verification.

2. Register the repository #

With credentials live, register the pointer. This is cluster state, so it only runs once against any node:

PUT _snapshot/prod-s3-repo
{
  "type": "s3",
  "settings": {
    "bucket": "es-snapshots-prod",
    "base_path": "prod-cluster",
    "client": "default"
  }
}

A 200 with {"acknowledged": true} means the definition is in cluster state — it does not mean any node can reach the bucket. Registration validates the request shape, not the credentials or the network path.

3. Verify from every node #

The verify call is the real test. It asks each node to write a small blob under the base_path and read it back, returning the nodes that succeeded:

POST _snapshot/prod-s3-repo/_verify

{
  "nodes": {
    "n1x2...": { "name": "es-01" },
    "n3y4...": { "name": "es-02" },
    "n5z6...": { "name": "es-03" }
  }
}

Every data node in the deployment must appear. A node that is missing from this list could not write or read the test blob — almost always a keystore entry that was never added or never reloaded on that specific node. A verification failure raises RepositoryVerificationException, which the deeper repository verification failure playbook walks through cause by cause.

4. Automate registration and verification (Python v8+) #

For repeatable, version-controlled deploys, drive the register-then-verify sequence through the v8 client. create_repository is idempotent (PUT semantics), and API/HTTP failures — including a wrapped RepositoryVerificationException — surface as ApiError, a sibling of TransportError rather than a subclass, so catch ApiError explicitly:

import logging
from elasticsearch import Elasticsearch, ApiError, NotFoundError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("repo_manager")


def register_repository(es: Elasticsearch, name: str, bucket: str, base_path: str) -> bool:
    """Idempotently register an S3 repository. Credentials come from the node keystore,
    never from these settings, so nothing sensitive is passed here."""
    try:
        es.snapshot.create_repository(
            name=name,
            type="s3",
            settings={
                "bucket": bucket,
                "base_path": base_path,
                "client": "default",
            },
            verify=False,  # register first, verify explicitly in the next step
        )
        logger.info("Repository '%s' registered against %s/%s.", name, bucket, base_path)
        return True
    except ApiError as exc:
        logger.error("Registration failed (%s): %s", exc.status_code, exc.info)
        return False


def verify_repository(es: Elasticsearch, name: str) -> bool:
    """Prove every node can read and write the repository before trusting a schedule to it."""
    try:
        result = es.snapshot.verify_repository(name=name)
        nodes = result["nodes"]
        logger.info("Repository '%s' verified on %d node(s): %s",
                    name, len(nodes), ", ".join(n["name"] for n in nodes.values()))
        return True
    except NotFoundError:
        logger.error("Repository '%s' is not registered.", name)
        return False
    except ApiError as exc:
        # RepositoryVerificationException surfaces here as an ApiError.
        logger.error("Verification failed (%s): %s", exc.status_code, exc.info)
        return False


if __name__ == "__main__":
    es = Elasticsearch(
        "https://es-cluster-01:9200",
        api_key="YOUR_BASE64_ENCODED_API_KEY",
        verify_certs=True,
        request_timeout=30,
    )

    if register_repository(es, "prod-s3-repo", "es-snapshots-prod", "prod-cluster"):
        verify_repository(es, "prod-s3-repo")

Passing verify=False on create_repository and then calling verify_repository separately keeps registration and verification as two distinct, individually reportable steps — useful when a pipeline should register even if a subset of nodes is temporarily unreachable, then alert on the verify result rather than fail the whole apply.

Verification #

Beyond the register-time _verify, three read calls confirm a repository is healthy and behaving. First, read the definition back to confirm the settings the deployment actually holds — a mismatch here versus your intended config usually means an old registration was never overwritten:

GET _snapshot/prod-s3-repo

{
  "prod-s3-repo": {
    "type": "s3",
    "settings": {
      "bucket": "es-snapshots-prod",
      "base_path": "prod-cluster",
      "client": "default"
    }
  }
}

Second, re-run verification any time credentials rotate or a node joins the deployment; a newly-added node with no keystore entry is invisible until _verify omits it:

POST _snapshot/prod-s3-repo/_verify

Third, for throughput and integrity concerns, run the repository analysis API, which exercises concurrent reads and writes and reports observed throughput and any consistency problems the object store exhibits under contention:

POST _snapshot/prod-s3-repo/_analyze?blob_count=100&max_blob_size=1gb

A successful _analyze returns per-blob timing and an aggregate throughput figure; a failure means the object store is not providing the read-after-write consistency Elasticsearch requires, which is a hard blocker for using it as a repository at all. Run it once at commissioning and after any change to the storage backend — not on a schedule, since it writes real load.

Threshold Tuning & Performance Guidance #

A repository that verifies can still be a bottleneck. Because both SLM and ILM push snapshots through the same bucket, the repository’s aggregate throughput is a shared resource that a burst of activity can saturate.

• Isolate clusters with base_path, never with shared prefixes. One bucket can safely serve many clusters as long as each writes under a distinct base_path. Two clusters at the same base_path will corrupt each other’s repository metadata — the index-N generation blob is not tenant-aware. Treat base_path as a hard tenancy boundary, and prefer a per-environment prefix (prod-cluster, staging-cluster) that reads clearly in bucket listings.
• Throttle uploads to protect ingest bandwidth. max_snapshot_bytes_per_sec caps per-node upload rate to the object store (unlimited by default since 7.16). On nodes that are simultaneously ingesting at high rate, an unthrottled first snapshot of a large index can starve the ingest path of network. Set it to leave headroom — a common starting point is half the node’s available egress — and raise it only if snapshots fall behind their schedule.
• Budget for incremental cost, not raw index size. Snapshots are incremental at the Lucene segment level: only segments not already in the repository are uploaded. The first snapshot of an index is expensive; subsequent ones are cheap — unless a force_merge or reindex rewrote every segment, in which case the next snapshot re-uploads everything. This is why a force_merge action placed before a searchable_snapshot in the same ILM phase inflates the snapshot it feeds, and why snapshot cost tracks segment churn rather than index size.
• Match max_restore_bytes_per_sec to recovery expectations. Restores and cold-tier mounts read from the repository, and the download throttle bounds how fast a disaster-recovery restore or a cold full_copy mount completes. If your recovery-time objective is tight, this throttle — and the object store’s raw read throughput measured by _analyze — is what you are actually planning against.
• Storage class is a cost-versus-latency trade you own outside Elasticsearch. Object stores offer cheaper, higher-latency storage classes for infrequently-accessed data. Snapshot blobs that back a frozen-tier searchable index are read on cache miss, so a deep-archive storage class can turn a first-touch frozen query into a multi-second (or worse) stall, while a rarely-restored backup tolerates it fine. Set lifecycle rules on the bucket deliberately, and keep any class that has a retrieval delay away from base_paths that back searchable snapshots.

Troubleshooting #

RepositoryVerificationException on a subset of nodes #

Symptom: _verify returns fewer nodes than the deployment has, or fails outright; SLM snapshots complete PARTIAL or fail with a repository error. Resolution:

1. Confirm the keystore entry exists on the missing node with bin/elasticsearch-keystore list — a node added after the initial rollout is the usual culprit.
2. Re-run POST _nodes/reload_secure_settings so a freshly-added key goes live without a restart.
3. If the keys are present and reloaded, check clock skew (NTP) — S3 request signing rejects timestamps that drift too far — and IAM permissions for s3:PutObject/s3:ListBucket scoped to the bucket and base_path. The verification failure playbook covers each cause with its exact error text.

repository_exception: cannot modify or a name conflict #

Symptom: re-registering fails, or two clusters unexpectedly see each other’s snapshots. Resolution:

1. A name conflict where two clusters share bucket and base_path is a data-corruption risk, not a warning — give each deployment a distinct base_path immediately and treat the shared-prefix window as suspect.
2. To change the type or bucket of an existing repository you must delete and re-register (DELETE _snapshot/prod-s3-repo then PUT); the existing snapshots remain in the bucket and are picked up by the new registration if the base_path is unchanged.

Read-only repository rejects writes #

Symptom: SLM or an ILM snapshot fails with a read-only repository error, or you deliberately want a mirror that can never be written. Resolution:

1. If unintended, remove "readonly": true from the settings and re-register — a repository registered read-only accepts mounts and restores but refuses every write.
2. If intended (a DR site that only ever reads the primary’s snapshots), keep readonly: true and register it against the same bucket/base_path from the secondary cluster; it will list and restore snapshots the primary wrote without any risk of the secondary mutating them.

First snapshot never completes / saturates the network #

Symptom: the initial snapshot of a large index runs for hours and degrades ingest latency. Resolution:

1. This is expected cost — the first snapshot uploads every segment. Set max_snapshot_bytes_per_sec to cap per-node upload rate and protect the ingest path.
2. Confirm with GET _snapshot/prod-s3-repo/_status that the snapshot is progressing (bytes climbing) rather than stalled; a stalled snapshot with no byte progress is a reachability problem, not a throughput one, and belongs back at the verify step.

FAQ #

Related #

• Configuring SLM Policies — the scheduled backups that consume the repository you register here.
• Searchable Snapshots & the Frozen Tier — the mounted read-only indices that read from this same repository on the cold and frozen tiers.
• Registering an S3 Snapshot Repository — the end-to-end registration walkthrough for an S3 bucket.
• Troubleshooting Snapshot Repository Verification Failures — diagnosing a failed _verify cause by cause.
• Snapshot Lifecycle Management & Searchable Snapshots — how repositories, SLM, and searchable snapshots fit together.

← Back to SLM & Searchable Snapshots