Snapshot Lifecycle Management & Searchable Snapshots #

Snapshot Lifecycle Management (SLM) is the declarative counterpart to index lifecycle automation: where a retention policy decides when data moves between tiers and when it is deleted, SLM decides when a durable, point-in-time copy of that data is written to an external repository and how long each copy is kept. Together with searchable snapshots — the mechanism that mounts a snapshot back as a queryable, read-only index — SLM turns object storage into an operational layer of the deployment rather than a cold archive you hope never to touch. For search engineers and DevOps operators this is where backup, retention compliance, and cost optimization converge: the same repository that satisfies a disaster-recovery mandate also backs the frozen tier that keeps a year of logs searchable at a fraction of hot-tier cost. This guide covers repository topology, SLM schedule and retention mechanics, the cold and frozen searchable-snapshot tiers, governance, Python automation, and the failure modes that turn a backup strategy into a false sense of security.

Repository Topology & Node Roles #

Every snapshot — whether a scheduled SLM backup or one created by an ILM searchable_snapshot action — is written to a snapshot repository, a registered pointer to external storage that every node in the deployment can reach. In production this is almost always object storage: s3, gcs, or azure. A shared-filesystem (fs) repository is viable only when every master and data node mounts the same networked volume at the same path, which is why it is common in on-premise clusters and rare in cloud ones. The repository-s3, repository-gcs, and repository-azure modules are bundled in Elasticsearch 8.x, so registering a bucket is a configuration task, not a plugin install; the secure half of that configuration — access keys — lives in the Elasticsearch keystore on every node, not in the repository settings themselves.

The repository is the single most important shared dependency in this whole subsystem, and its throughput and latency bound everything downstream. A snapshot is incremental at the segment level: it copies only the Lucene segments that are not already present in the repository, so the first snapshot of a large index is expensive and subsequent ones are cheap — unless a force_merge or a reindex rewrote every segment, in which case the next snapshot copies everything again. This coupling is why the ILM searchable_snapshot action and a force_merge action in the same phase interact so strongly, and why snapshot cost is a function of segment churn rather than raw index size.

Searchable snapshots add a second dimension: node roles. Mounting a snapshot as a searchable index requires nodes carrying the right data-tier role. The cold tier (data_cold) mounts with full_copy storage, keeping a complete local copy of the index on disk while treating the repository as the source of truth for recovery. The frozen tier (data_frozen) mounts with shared_cache storage, keeping only a bounded local cache — sized by xpack.searchable.snapshot.shared_cache.size — and fetching the rest from the repository on demand. That single setting is what lets a frozen node hold far more index data than it has local disk, and it is the reason a year of rarely-queried logs can stay searchable on a handful of cheap nodes. The tier a snapshot lands on is governed by the same hot-warm-cold architecture that routes live shards, extended one tier colder.

Key settings to internalize:

• PUT _snapshot/<repo> with type and settings.bucket/settings.base_path — registers the repository; multiple clusters can share a bucket only if each uses a distinct base_path.
• xpack.searchable.snapshot.shared_cache.size — the frozen-tier disk cache, set as a node setting (percentage or absolute) on data_frozen nodes; a frozen node with no shared cache configured cannot mount shared_cache indices.
• snapshot.max_restore_bytes_per_sec / snapshot.max_snapshot_bytes_per_sec — per-node repository throughput throttles (unlimited by default since 7.16), the first knob to reach for when snapshots saturate network to object storage.
• index.store.snapshot.* — the mounted-index settings that a searchable snapshot carries; they are read-only and cannot be changed after the mount.

SLM Schedule & Retention Mechanics #

An SLM policy is a compact declarative object with four moving parts: a schedule (a cron expression in Elasticsearch’s 0 sec min hour ... form), a name template (a date-math pattern so each snapshot is uniquely named), a repository, and a config block naming the indices and whether to include the global cluster state. A fifth block, retention, is what makes SLM a lifecycle manager rather than a cron job: it defines expire_after, min_count, and max_count, and a separate scheduled retention run deletes snapshots that have aged out — but only while keeping at least min_count and never more than max_count, so a retention window can never delete your last good backup.

The retention run is scheduled cluster-wide by slm.retention_schedule (default 0 30 1 * * ?, 01:30 daily), independent of any single policy’s snapshot schedule. A subtle consequence trips up operators: taking a snapshot and expiring old ones are two different scheduled events, so a policy that snapshots hourly but relies on the once-daily retention run can accumulate up to a day of extra snapshots before they are cleaned. Sizing max_count to tolerate that gap — rather than to the exact snapshot cadence — avoids surprise repository growth. The annotated policy below shows every field in its production place:

PUT _slm/policy/daily-logs-backup
{
  "schedule": "0 30 1 * * ?",
  "name": "<daily-logs-{now/d}>",
  "repository": "prod-s3-repo",
  "config": {
    "indices": ["logs-*"],
    "include_global_state": false,
    "ignore_unavailable": true
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 7,
    "max_count": 60
  }
}

include_global_state: false keeps cluster-wide settings, templates, and ILM policies out of the data backup so a restore cannot silently overwrite live configuration; back those up in a separate policy that sets it true. ignore_unavailable: true stops a single deleted index from failing the whole snapshot. Designing these schedules to interlock with retention timing rather than fight it is the subject of configuring SLM policies, while the ILM policy design and lifecycle synchronization topic covers keeping the ILM side of that pairing versioned and drift-free.

Searchable Snapshots & the Frozen Tier #

A searchable snapshot is a snapshot mounted back as an index you can query. Nothing is restored to primary storage: the index metadata lives in cluster state, but the Lucene data stays in the repository and is read through a local cache. This is what makes the frozen tier economical — the shared_cache storage option keeps only recently-touched blocks on disk, so a data_frozen node can front an order of magnitude more index data than it physically stores. The trade is latency: a cache miss fetches from object storage, so first-touch queries on frozen data are seconds, not milliseconds.

There are two ways an index becomes a searchable snapshot. The first is automatic, through ILM: a searchable_snapshot action in the cold or frozen phase snapshots the index into a named repository and mounts the result, deleting the original once the mount succeeds. In the cold phase the default storage is full_copy; in the frozen phase it is shared_cache. The second is manual, through searchable_snapshots.mount, which is how you promote an existing snapshot — including one an SLM backup already wrote — into a queryable index without re-snapshotting. The mechanics of both, and when to choose each tier, are detailed under searchable snapshots and the frozen tier; converting already-cold indices in place is covered by converting cold indices to searchable snapshots via ILM.

PUT _ilm/policy/logs-with-frozen
{
  "policy": {
    "phases": {
      "hot":  { "actions": { "rollover": { "max_primary_shard_size": "50gb" } } },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "prod-s3-repo"
          }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "prod-s3-repo",
            "storage": "shared_cache"
          }
        }
      },
      "delete": { "min_age": "365d", "actions": { "delete": {} } }
    }
  }
}

One rule prevents the most common design error: do not run searchable_snapshot in two phases against two different repositories, and be aware that a force_merge in a phase before the snapshot rewrites every segment, inflating the snapshot it feeds. When the cold and frozen phases point at the same repository, ILM is smart enough to reuse the existing snapshot rather than take a second one, so the frozen mount is nearly free. Pairing this with fallback routing for data retention gives a read path that survives the loss of a whole tier, because the repository — not the node — holds the durable copy.

Security & Governance #

Snapshots and repositories are cluster-scoped resources with outsized blast radius: a misconfigured restore can overwrite live indices, and delete access to a repository is delete access to your backups. Managing SLM policies requires cluster privilege manage_slm; observing them needs read_slm. Registering or removing repositories requires cluster-level manage, and taking or restoring snapshots requires create_snapshot plus index-level read/write on the affected indices. A practical governance model mirrors the ILM one from securing ILM policies with RBAC: humans get read_slm by default, a scoped CI/CD service account holds manage_slm, and repository registration is reserved for a platform-admin role whose actions are audited.

Retention compliance is the governance case that matters most. When a regulatory mandate requires that data be preserved for a fixed window, the SLM retention block is the control — but only if min_count is high enough that a burst of failed snapshots cannot let retention delete every valid copy, and only if the repository itself is protected by object-storage bucket policies against out-of-band deletion. Treat the SLM policy JSON as version-controlled configuration reconciled by a pipeline, exactly as you would an ILM policy, so a live edit in Kibana can never silently shorten a legal hold. SLM writes an audit trail to the hidden .slm-history-* indices; ship those to your compliance store so every snapshot and every retention deletion is attributable.

Production Automation with Python v8+ #

The Python v8 client exposes SLM under client.slm, repositories and snapshots under client.snapshot, and mounting under client.searchable_snapshots. The pattern below deploys a policy idempotently, triggers an on-demand snapshot to validate the whole path end to end (the fastest way to catch a broken repository before the first scheduled run), and reports SLM health. As always in v8, API and HTTP failures surface as ApiError — a sibling of TransportError, not a subclass — so catch ApiError explicitly.

import logging
from elasticsearch import Elasticsearch, ApiError, NotFoundError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("slm_automation")


def deploy_slm_policy(es: Elasticsearch, policy_id: str, policy: dict) -> bool:
    """Idempotently create or update an SLM policy (PUT semantics)."""
    try:
        es.slm.put_lifecycle(policy_id=policy_id, body=policy)
        logger.info("SLM policy '%s' applied.", policy_id)
        return True
    except ApiError as exc:
        logger.error("Failed to apply SLM policy: %s", exc.info)
        return False


def validate_repository(es: Elasticsearch, repo: str) -> bool:
    """Verify every node can reach the repository before trusting the schedule."""
    try:
        nodes = es.snapshot.verify_repository(name=repo)
        logger.info("Repository '%s' verified on %d node(s).", repo, len(nodes["nodes"]))
        return True
    except NotFoundError:
        logger.error("Repository '%s' is not registered.", repo)
        return False
    except ApiError as exc:
        logger.error("Repository verification failed: %s", exc.info)
        return False


def execute_and_report(es: Elasticsearch, policy_id: str) -> None:
    """Trigger one snapshot now and surface SLM policy-level failure counters."""
    try:
        result = es.slm.execute_lifecycle(policy_id=policy_id)
        logger.info("On-demand snapshot started: %s", result.get("snapshot_name"))
    except ApiError as exc:
        logger.error("On-demand execution failed: %s", exc.info)

    policy = es.slm.get_lifecycle(policy_id=policy_id)[policy_id]
    stats = policy.get("stats", {})
    logger.info(
        "Policy '%s': taken=%s failed=%s deleted=%s",
        policy_id,
        stats.get("snapshots_taken", 0),
        stats.get("snapshots_failed", 0),
        stats.get("snapshots_deleted", 0),
    )


if __name__ == "__main__":
    es = Elasticsearch(
        "https://es-cluster-01:9200",
        api_key="YOUR_BASE64_ENCODED_API_KEY",
        verify_certs=True,
        request_timeout=30,
    )

    slm_policy = {
        "schedule": "0 30 1 * * ?",
        "name": "<daily-logs-{now/d}>",
        "repository": "prod-s3-repo",
        "config": {"indices": ["logs-*"], "include_global_state": False},
        "retention": {"expire_after": "30d", "min_count": 7, "max_count": 60},
    }

    if validate_repository(es, "prod-s3-repo"):
        deploy_slm_policy(es, "daily-logs-backup", slm_policy)
        execute_and_report(es, "daily-logs-backup")

Verifying the repository before deploying the policy is the load-bearing step: an SLM policy against an unreachable bucket applies cleanly and then fails silently every night until someone reads the stats. Building this into the same version-controlled apply step that owns your ILM automation in Python keeps both lifecycles reconciled from one pipeline.

Monitoring & Observability #

SLM, like ILM, fails silently by default — a snapshot that never runs looks exactly like a deployment with nothing to back up until the day you need a restore. Instrument the policy-level counters and the repository directly:

• GET _slm/stats — cluster-wide totals: total_snapshots_taken, total_snapshots_failed, and per-policy breakdowns; alert on any non-zero snapshots_failed delta.
• GET _slm/policy/<id> — the last_success and last_failure timestamps and messages for one policy; a last_failure newer than last_success is a page.
• GET _cat/snapshots/<repo>?v&s=end_epoch:desc — the actual snapshots in the repository with state (SUCCESS, PARTIAL, FAILED) and duration; PARTIAL means some shards did not snapshot.
• GET _snapshot/<repo>/_status — in-progress snapshot progress by shard, for diagnosing a run that is slow rather than failed.
• GET _cat/nodes?v&h=name,node.role — confirm data_frozen nodes exist and carry a configured shared cache before an ILM frozen phase needs them.

The alerting signals that matter most: any SLM snapshots_failed increment, a policy whose last_failure is newer than last_success, a snapshot in PARTIAL/FAILED state, a repository that fails _verify, and a frozen tier at its shared-cache eviction ceiling. Wire these into Kibana Stack Monitoring or a scheduled job built on the execute_and_report helper above, and treat a failed snapshot with the same urgency as a stuck ILM step — the two subsystems fail for overlapping reasons (disk, allocation, and repository reachability). Deeper ILM-side observability lives in monitoring ILM health with the _cat and explain APIs.

Common Failure Modes #

When SLM and ILM both drive snapshots into one repository, a slow object store becomes a shared bottleneck. The end-to-end mechanics of registering and hardening that repository live in managing snapshot repositories.

Frequently Asked Questions #

Related #

• Configuring SLM Policies — schedules, name templates, config, and retention.
• Searchable Snapshots & the Frozen Tier — mounting snapshots as queryable indices and sizing the shared cache.
• Managing Snapshot Repositories — registering, verifying, and hardening object-storage repositories.
• ILM Policy Design & Lifecycle Synchronization — keeping the ILM policies that drive searchable snapshots versioned and drift-free.
• Elasticsearch ILM Architecture & Fundamentals — the phase state machine whose cold and frozen phases invoke these snapshots.

← Back to index-lifecycle-management.org home