Setting Snapshot Retention with SLM #

Configure the SLM retention block so Elasticsearch deletes aged snapshots on its own — reclaiming repository storage automatically while guaranteeing that a burst of failed backups can never leave you without a single valid copy to restore from.

Retention is the field where the cost and the safety of a backup strategy meet. Set it too loose and the repository grows without bound, quietly inflating the object-storage bill until someone audits it; set it too aggressive, or misunderstand how its three sub-fields interact, and you can find the retention run has deleted the very snapshot you needed. This page is about the retention block specifically: how expire_after, min_count, and max_count combine, when deletions actually happen, and how to make retention safe for compliance workloads. It is one part of the broader task of configuring SLM policies, under Snapshot Lifecycle Management & Searchable Snapshots; the scheduling half — when snapshots are created — is covered in scheduling automated snapshots with SLM.

Prerequisites #

• Elasticsearch 8.x with SLM running and at least one policy already taking snapshots into prod-s3-repo.
• A repository whose backing store permits deletes — retention cannot prune snapshots from a bucket that denies delete operations via its object-storage policy.
• The manage_slm privilege to edit the retention block and to trigger a manual retention run.

How the three fields interact #

The retention block has three fields, and understanding their precedence is the whole game:

"retention": {
  "expire_after": "30d",
  "min_count": 7,
  "max_count": 60
}

• expire_after marks a snapshot as eligible for deletion once it is older than the given age. It is a suggestion, not a command — eligibility does not guarantee deletion.
• min_count is a floor: SLM always keeps at least this many of the most recent successful snapshots, even if they are older than expire_after. min_count wins over expire_after. This is the field that protects your last good backup.
• max_count is a ceiling: SLM never keeps more than this many, deleting the oldest beyond the cap regardless of their age. max_count wins over the desire to keep everything.

Put concretely, the retention run applies them in this order: keep the min_count most recent successful snapshots no matter what; among the rest, delete anything older than expire_after; and if what remains still exceeds max_count, delete the oldest until it fits. The practical consequence is the reassuring one — if your last seven nightly snapshots all failed, min_count: 7 means the seven most recent successful ones are still there, however old they have become. Retention will never trade your safety net for tidiness.

A worked example makes the precedence concrete. Suppose you take a daily snapshot, expire_after is 30d, min_count is 7, and max_count is 60. On a healthy cluster you accumulate roughly 30 snapshots (everything younger than 30 days), comfortably between the floor of 7 and the ceiling of 60. Now suppose a misconfiguration causes 25 straight days of failed snapshots. Those failed runs create nothing to keep, so the newest successful snapshot is 25 days old — but min_count: 7 guarantees the 7 most recent successful snapshots remain, even as they age past 30 days, giving you a restore point throughout the outage. Without min_count, expire_after alone would eventually delete every copy and leave the repository empty exactly when you needed it most.

When retention actually runs #

Deleting expired snapshots is not part of taking a snapshot. It is a separate, cluster-wide job governed by the slm.retention_schedule setting, which defaults to 0 30 1 * * ? — 01:30 daily. One retention run processes every SLM policy in the deployment at once. Two consequences follow directly:

1. Retention lags snapshot expiry. A snapshot that crosses expire_after at noon is not deleted until the next retention run at 01:30. Between those moments it still occupies the repository, so momentary snapshot counts can exceed what expire_after implies.
2. All policies sharing the retention schedule are pruned together. If multiple policies write into prod-s3-repo, one 01:30 run evaluates all of their retention blocks in the same pass; there is no per-policy retention timing.

You can force a retention pass immediately rather than waiting for the schedule — invaluable when validating a new retention block or reclaiming space after lowering max_count:

POST _slm/_execute_retention

This runs the deletion logic for every policy right now and is safe to call: it honours each policy’s min_count, so a manual run cannot delete anything the scheduled run would have protected.

Implementation #

Apply the retention block as part of the policy. The PUT is idempotent, so you can tune the numbers and re-apply without recreating the policy:

PUT _slm/policy/daily-logs-backup
{
  "schedule": "0 30 1 * * ?",
  "name": "<daily-logs-{now/d}>",
  "repository": "prod-s3-repo",
  "config": { "indices": ["logs-*"], "include_global_state": false },
  "retention": { "expire_after": "30d", "min_count": 7, "max_count": 60 }
}

To reclaim space on demand and confirm the run is deleting what you expect, drive execute_retention and read the resulting stats from the elasticsearch-py v8 client. get_stats exposes the total_snapshots_deleted counter cluster-wide, which increments as retention prunes:

import logging
from elasticsearch import Elasticsearch, ApiError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("slm_retention")


def run_retention_and_report(es: Elasticsearch) -> None:
    """Force a retention pass now and report how many snapshots it deleted."""
    try:
        before = es.slm.get_stats().get("total_snapshots_deleted", 0)
        es.slm.execute_retention()          # cluster-wide, honours every min_count
        after = es.slm.get_stats().get("total_snapshots_deleted", 0)
        logger.info("Retention run complete: %d snapshot(s) deleted this pass.", after - before)
    except ApiError as exc:
        # A repository that denies deletes surfaces here as an ApiError.
        logger.error("Retention run failed (%s): %s", exc.status_code, exc.info)


if __name__ == "__main__":
    es = Elasticsearch("https://es-cluster-01:9200", api_key="YOUR_API_KEY", verify_certs=True)
    run_retention_and_report(es)

Verification #

Read the SLM stats to see the retention counters cluster-wide. retention_runs should increment on each pass, and retention_deletion_time shows how long pruning takes:

GET _slm/stats

{
  "retention_runs": 42,
  "retention_failed": 0,
  "retention_deletion_time": "1.2s",
  "total_snapshots_deleted": 118,
  "total_snapshots_taken": 176
}

A non-zero retention_failed almost always means the repository denied a delete — check the bucket’s object-storage permissions first. Then watch the actual snapshot count in the repository fall after a run and hold steady between your floor and ceiling:

GET _cat/snapshots/prod-s3-repo?v&h=id,status,start_epoch&s=start_epoch:desc

The number of rows should sit at roughly the count expire_after implies, never dropping below min_count and never rising above max_count.

Gotchas and edge cases #

• Retention prunes across every policy on one schedule. There is a single cluster-wide slm.retention_schedule; you cannot give one policy a faster prune cadence than another. If a policy needs tighter storage control, adjust its max_count, not its retention timing — the timing is shared.
• min_count keeps snapshots past expire_after. This is intentional, not a bug. A snapshot older than expire_after is still retained if deleting it would drop the count below min_count, so during a run of failed backups your oldest kept snapshot can be far older than the expiry window. That is precisely the protection you want.
• The repository must permit deletes. Retention issues delete calls to object storage. A bucket policy or object-lock configuration that blocks deletes makes retention fail every run — visible as a rising retention_failed. Reconcile the intended retention with any immutability the bucket enforces.
• Compliance and legal hold need a high floor plus bucket protection. When a mandate requires keeping data for a fixed window, min_count set high enough to span that window is the in-cluster control, but pair it with an object-storage retention/immutability policy so an out-of-band deletion — or a live edit that lowers min_count — cannot shorten the hold. Treat the policy JSON as version-controlled and reconciled by a pipeline so no Kibana edit silently weakens it.

FAQ #

Related #

• Scheduling Automated Snapshots with SLM — the cadence at which the snapshots this retention block prunes are created.
• Configuring SLM Policies — the full policy of which retention is one block.
• SLM & Searchable Snapshots — how retention fits alongside repositories, searchable snapshots, and governance.

← Back to Configuring SLM Policies · SLM & Searchable Snapshots