ILM & Reindexing Automation Runbooks

Troubleshooting Snapshot Repository Verification Failures

When POST _snapshot/<repo>/_verify fails — or a scheduled SLM run fails against the same repository — the fix depends entirely on which cause raised the RepositoryVerificationException, so diagnosis is a matter of reading the error and mapping it to one of a small set of root causes.

Verification failure is the single most common repository problem, and it is almost always one node’s problem masquerading as the whole repository’s. Because access to a repository is per node — each node signs its own requests with its own keystore credentials, as the managing snapshot repositories reference explains — a failure often means the repository is perfectly reachable from every node except one. This page is the diagnostic companion to registering an S3 snapshot repository: once registration succeeds but verification does not, work through the causes below in order of likelihood, and you will resolve nearly every case without guessing. The whole flow feeds the snapshot and searchable-snapshot subsystem, so a repository that verifies is the precondition for every backup and every mount downstream.

Prerequisites

  • A registered S3 (or other object-store) repository whose _verify is failing or returning fewer nodes than the deployment has.
  • Shell access to the nodes to inspect keystores, plus manage cluster privilege to re-verify and reload secure settings.
  • elasticsearch-py v8.0+ for the diagnostic script, which reads structured error detail from the caught exception.

Implementation

Read the verify output and map the error to a cause

_verify returns the nodes that succeeded. The fastest first read is therefore to compare that list against the deployment’s node list — any node present in _cat/nodes but absent from the verify result is the failing one:

POST _snapshot/prod-s3-repo/_verify

When verification fails outright, the response body carries a RepositoryVerificationException whose reason names the cause. The decision flow below maps the reason text to the fix:

Mapping a RepositoryVerificationException reason to its root cause and fixA failed _verify raises RepositoryVerificationException. Branch on the reason: access-denied or signature to a credentials/IAM fix; a failure on only some nodes to reloading secure settings on the missed node; connection or DNS to a network/endpoint fix; request-time-skew to NTP clock sync; path-already-in-use to a base_path collision; read-only to a bucket or repository that refuses writes._verify failsVerification-Exception, read reasonaccess-denied / signatureFix creds: keystore entry + bucket IAM policyfails on some nodes onlyReload secure settings on the missed nodeconnection / DNS / hostFix network: egress + endpoint settingrequest-time-skewSync clocks with NTP on every nodepath-already-in-useGive this deployment a distinct base_pathread-only / write deniedRemove readonly or grant PutObject

The causes, in the order you should suspect them:

  • Missing or wrong credentials — the reason mentions access denied or a signature mismatch on every node. Either the keystore entries were never added, or the wrong access/secret pair was pasted. Fix the s3.client.default.* entries and reload.
  • A node that never reloaded — verification succeeds on most nodes and fails on one. That node’s keystore was edited but POST _nodes/reload_secure_settings never activated it, or the node was added to the deployment after the initial rollout and never had the keys added at all.
  • Network, DNS, or endpoint — the reason is a connection timeout or unknown-host error. The node cannot reach the object-store endpoint; check egress and, for an S3-compatible store, that the endpoint setting is present and correct.
  • IAM permissions — access-denied specifically on write (s3:PutObject) or list (s3:ListBucket). The credentials authenticate but the IAM policy does not grant the actions the verify blob write needs on the bucket and base_path.
  • Clock skew — a request-time-too-skewed error. S3 SigV4 rejects requests whose timestamps drift beyond a tolerance; a node with a wrong clock signs requests the store refuses.
  • base_path collision — a path-already-in-use or unexpected-metadata error, usually because another cluster is writing to the same bucket and base_path.
  • Read-only bucket or repository — writes are refused although reads work, because the repository was registered readonly: true or the bucket policy denies writes.

Diagnose from Python and read the structured detail

The v8 client raises ApiError for the wrapped RepositoryVerificationException; the useful detail is in exc.info, which holds the parsed error body including the per-cause reason. Catch ApiError (a sibling of TransportError, not a subclass) and log the structured info rather than the string:

import logging
from elasticsearch import Elasticsearch, ApiError, NotFoundError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("repo_diag")


def diagnose_verification(es: Elasticsearch, repo: str) -> None:
    try:
        result = es.snapshot.verify_repository(name=repo)
        logger.info("Repository '%s' verified on %d node(s) — no failure to diagnose.",
                    repo, len(result["nodes"]))
    except NotFoundError:
        logger.error("Repository '%s' is not registered — register it before verifying.", repo)
    except ApiError as exc:
        # exc.info carries the parsed RepositoryVerificationException body.
        info = exc.info or {}
        reason = ""
        if isinstance(info, dict):
            reason = info.get("error", {}).get("reason", "") if isinstance(info.get("error"), dict) else ""
        logger.error("Verification failed (%s). reason=%s", exc.status_code, reason)

        text = reason.lower()
        if "access denied" in text or "signature" in text:
            logger.error("→ Credentials/IAM: check s3.client.default.* keystore and bucket IAM policy.")
        elif "unknown host" in text or "connect" in text or "timeout" in text:
            logger.error("→ Network/endpoint: check egress and the endpoint setting on the failing node.")
        elif "skew" in text or "request time" in text:
            logger.error("→ Clock skew: sync NTP across all nodes.")
        elif "already" in text or "metadata" in text:
            logger.error("→ base_path collision: give this cluster a distinct base_path.")
        elif "read" in text and "only" in text:
            logger.error("→ Read-only: remove readonly:true or grant s3:PutObject.")


if __name__ == "__main__":
    es = Elasticsearch(
        "https://es-cluster-01:9200",
        api_key="YOUR_BASE64_ENCODED_API_KEY",
        verify_certs=True,
    )
    diagnose_verification(es, "prod-s3-repo")

For failures that verify hints at but does not fully explain — intermittent write errors, or a store that behaves under load but not at rest — run the repository analysis API, which drives concurrent reads and writes and surfaces consistency problems verification’s single blob cannot:

POST _snapshot/prod-s3-repo/_analyze?blob_count=100&max_blob_size=1gb

Verification

After applying a fix, re-run verification and confirm every data node returns — a partial list means the fix landed on some nodes but not the one that was actually failing:

POST _snapshot/prod-s3-repo/_verify
{
  "nodes": {
    "n1x2...": { "name": "es-01" },
    "n3y4...": { "name": "es-02" },
    "n5z6...": { "name": "es-03" }
  }
}

Cross-check the returned names against GET _cat/nodes?v&h=name,node.role so no data node is silently absent; only when the two lists match is the repository trustworthy for the next scheduled backup.

Gotchas and edge cases

  • A partial pass is still a failure. Verification returning three of four nodes is not “mostly working” — the fourth node will fail every snapshot that touches a shard it hosts, producing PARTIAL snapshots. Always reconcile the verify result against the full node list, not just check that the call returned 200.
  • A node added after rollout has no keys. Autoscaling or a manual node addition brings up a node whose keystore never received s3.client.default.*. It joins the deployment fine, serves queries fine, and fails repository verification silently until someone reads the node list. Bake keystore seeding into node provisioning.
  • Clock skew looks like a credentials problem. A request-time-skew rejection returns an auth-flavoured error even though the credentials are correct, sending people down the wrong path. If credentials are provably right but one node fails, check that node’s clock before re-pasting keys.
  • Read-only can be intentional. A DR-mirror deployment deliberately registers the repository readonly: true so it can restore but never mutate the primary’s snapshots. A write failure on such a mirror is expected — do not “fix” it by making the mirror writable.

FAQ

Verification passes on most nodes but fails on one — what is wrong?

Almost always that one node is missing a live keystore credential. Either its keystore never received the s3.client.default.* entries (common for a node added after the initial rollout), or the entries were added but POST _nodes/reload_secure_settings was never run on it. Add the keys on that node and reload secure settings, then re-verify and confirm the node now appears in the returned list.

Where do I read the actual reason a verification failed?

In the RepositoryVerificationException body — over HTTP it is the error.reason field of the response, and in the Python v8 client it is inside exc.info on the caught ApiError. The reason text distinguishes an access-denied credentials problem from a connection/DNS network problem, a clock-skew rejection, or a base_path collision, which is what tells you which fix to apply.

My credentials are correct but one node still fails — what else?

Check that node’s clock and its network path. S3 request signing rejects requests whose timestamps drift too far, returning an auth-flavoured error that mimics a credentials problem, so a node with a wrong clock fails verification despite correct keys. A node that cannot reach the object-store endpoint — blocked egress, or a missing endpoint setting for an S3-compatible store — fails with a connection or unknown-host error instead.

← Back to Managing Snapshot Repositories · SLM & Searchable Snapshots