ILM & Reindexing Automation Runbooks

Converting Cold Indices to Searchable Snapshots via ILM

Add a searchable_snapshot action to the cold (and optionally frozen) phase of an existing ILM policy so metrics-* indices that have already aged onto the cold tier convert themselves into repository-backed searchable snapshots — automatically, on the next lifecycle step, with no manual mount.

Doing it by hand with the mount API works for one-off promotion, but it does not scale to a fleet of aging indices or survive the next new index that rolls over. Baking the conversion into the policy makes it a standing rule: every index that reaches the cold phase snapshots into your repository and remounts as a restored- index, shedding its primary-storage footprint while staying fully queryable. This page is the automatic counterpart to mounting a searchable snapshot to the frozen tier; the storage-mode mechanics it relies on are covered under searchable snapshots and the frozen tier, within the wider Snapshot Lifecycle Management & searchable snapshots topic.

Prerequisites

  • Elasticsearch 8.x with data_cold nodes (and data_frozen nodes, if you add a frozen phase) carrying the right roles; frozen nodes additionally need xpack.searchable.snapshot.shared_cache.size set.
  • cold-archive-repo already registered and reachable — verify with POST _snapshot/cold-archive-repo/_verify before editing the policy, or the searchable-snapshot step will fail at runtime.
  • elasticsearch-py v8.0+ for client.ilm.put_lifecycle.
  • manage_ilm cluster privilege on the account applying the policy.

Implementation

The change is a phase-level edit: add a searchable_snapshot action to the cold phase (and, if you want a deeper tier, the frozen phase) of the policy that already manages your metrics-* indices. Because PUT _ilm/policy replaces the whole policy, include the phases you want to keep — this is not a partial patch.

PUT _ilm/policy/metrics-archive
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "50gb" }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "cold-archive-repo",
            "force_merge_index": true,
            "storage": "full_copy"
          }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "cold-archive-repo",
            "storage": "shared_cache"
          }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": { "delete": {} }
      }
    }
  }
}

Two behaviours make this work cleanly. First, once the cold-phase snapshot is taken and mounted, ILM deletes the original index and continues managing the mounted restored-metrics-* copy in its place — you do not end up with two copies of the data. Second, because the cold and frozen phases name the same snapshot_repository, ILM reuses the snapshot the cold phase already wrote when the index later reaches the frozen phase, remounting it as partial-metrics-* with shared_cache storage rather than snapshotting a second time. Pointing the two phases at different repositories throws that saving away.

The same edit from Python v8, applied idempotently:

import logging
from elasticsearch import Elasticsearch, ApiError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("ilm_convert")


def add_searchable_snapshot(es: Elasticsearch, policy_id: str, repo: str) -> bool:
    """Apply a metrics-archive policy whose cold and frozen phases mount searchable snapshots."""
    policy = {
        "phases": {
            "hot": {"actions": {"rollover": {"max_primary_shard_size": "50gb"}}},
            "cold": {
                "min_age": "30d",
                "actions": {
                    "searchable_snapshot": {
                        "snapshot_repository": repo,
                        "force_merge_index": True,
                        "storage": "full_copy",
                    }
                },
            },
            "frozen": {
                "min_age": "90d",
                "actions": {
                    "searchable_snapshot": {
                        "snapshot_repository": repo,
                        "storage": "shared_cache",
                    }
                },
            },
            "delete": {"min_age": "365d", "actions": {"delete": {}}},
        }
    }
    try:
        es.ilm.put_lifecycle(name=policy_id, policy=policy)
        logger.info("Policy '%s' updated with cold + frozen searchable_snapshot.", policy_id)
        return True
    except ApiError as exc:
        logger.error("Failed to update policy (%s): %s", exc.status_code, exc.info)
        return False


if __name__ == "__main__":
    es = Elasticsearch(
        "https://es-cluster-01:9200",
        api_key="YOUR_BASE64_ENCODED_API_KEY",
        verify_certs=True,
    )
    add_searchable_snapshot(es, "metrics-archive", "cold-archive-repo")

A policy update does not retroactively rewind an index that has already moved past the cold phase, and it does not interrupt one mid-step. Each managed index picks up the new phase definition at its next lifecycle step evaluation — so an index currently sitting in the cold phase adopts the new cold actions when it next advances, and indices still in hot simply carry the updated definition forward. There is no need to reattach the policy; managed indices always read the current version.

Verification

Confirm an index is actually running the searchable-snapshot conversion by asking ILM where it is. The explain output names the phase, action, and step:

GET metrics-000031/_ilm/explain
{
  "indices": {
    "metrics-000031": {
      "managed": true,
      "policy": "metrics-archive",
      "phase": "cold",
      "action": "searchable_snapshot",
      "step": "mount-snapshot"
    }
  }
}

A step of mount-snapshot (or the surrounding wait-for-green / copy-execution-state steps) means the conversion is in flight. Once it completes, the original metrics-000031 is gone and a mounted index appears under the tier’s prefix — restored- for the cold full_copy mount, partial- for a frozen shared_cache mount. List them:

GET _cat/indices/restored-metrics-*,partial-metrics-*?v&h=index,health,status,store.size

Seeing restored-metrics-000031 (or partial-metrics-000031 after it reaches the frozen phase) confirms the conversion landed, and the reduced store.size reflects that the data is now served from the repository rather than held on primary storage.

Gotchas and edge cases

  • force_merge_index inflates the first snapshot. With force_merge_index: true (the default) ILM force-merges the index toward a single segment before snapshotting, which speeds later queries but rewrites every segment — so the snapshot copies everything, not just changed segments. If a warm phase already force-merged the index, that cost is spent; if not, budget for a large first write to cold-archive-repo.
  • You cannot change storage after the mount. The storage mode is baked into the mounted index at mount time. Editing the policy from full_copy to shared_cache will not convert an index that has already mounted as full_copy; only indices that reach the phase after the edit use the new mode. To change an existing mount you must remount it.
  • The update applies at the next step, not immediately. A policy change never yanks an index out of its current step. Indices adopt the new phase definitions when they next advance, so do not expect an already-cold index to convert the instant you press save — it converts on its next lifecycle evaluation, bounded by indices.lifecycle.poll_interval.
  • The repository must exist before the step runs. ILM does not validate the repository when you save the policy — it fails at the mount-snapshot step at runtime if cold-archive-repo is unregistered or unreachable, leaving the index stuck in an ERROR step. Verify the repository first, and register it read-write for the ILM account.

FAQ

Will updating the policy convert indices that are already cold?

Yes, but on their own schedule. A managed index reads the current policy version and picks up the new cold-phase searchable_snapshot action at its next lifecycle step evaluation, so an index already sitting in the cold phase converts when it next advances — not the instant you save. Indices still in hot or warm carry the updated definition forward and convert when they reach the cold phase.

Does ILM keep the original index after mounting the snapshot?

No. Once the cold-phase snapshot is taken and successfully mounted, ILM deletes the original index and continues managing the mounted restored- copy in its place. You do not end up storing the data twice. The underlying snapshot in the repository is what backs the mounted index from then on.

Do the cold and frozen phases take two separate snapshots?

Not when they name the same snapshot_repository. ILM reuses the snapshot the cold phase already wrote when the index later reaches the frozen phase, remounting it with shared_cache storage instead of snapshotting again. Pointing the two phases at different repositories forces a second, redundant snapshot, so keep them on one repository.

← Back to Searchable Snapshots & the Frozen Tier · Snapshot Lifecycle Management & Searchable Snapshots