ILM & Reindexing Automation Runbooks

Rolling Back a Broken ILM Policy Version

A bad edit to app-events-ilm — say a delete phase whose min_age was shortened from 30d to 3d — is live and already deleting data early; revert it safely without waiting for a fix to be authored from scratch.

Rolling back an Elasticsearch ILM policy is not a call to some “revert” API, because none exists. A deployment keeps only the current definition, a version counter, and a modified_date; it stores no history of previous bodies. Git is the history. A rollback is therefore a forward apply of an earlier known-good revision — the same put_lifecycle mechanism as any other change, pointed at an older commit. This is the recovery counterpart to promoting ILM policy changes across environments and builds on the field mechanics laid out in versioning ILM policies across environments, all within ILM policy design and lifecycle synchronization. The crucial caveat: re-applying the good definition fixes what future indices and future phase transitions will do, but it cannot rewind an index that already advanced or an index that was already deleted.

Prerequisites

  • Elasticsearch 8.x with the affected policy (app-events-ilm) live, and the previous known-good policy JSON retrievable from source control by commit.
  • elasticsearch-py v8.0+, using the v8 surface (ilm.put_lifecycle, ilm.explain_lifecycle, ilm.move_to_step, ilm.retry).
  • manage_ilm at cluster scope, plus manage on app-events-* to drive per-index recovery, per securing ILM policies with RBAC.

Implementation

Rollback is two moves. First, re-PUT the previous good body so the definition reverts (its version bumps forward — Elasticsearch only ever increments). Second, deal with any index the bad version already harmed or wedged: an index sitting in an ERROR step after you fix the definition needs ilm.retry, and an index that advanced into the wrong phase needs ilm.move_to_step to be nudged back onto a valid step. Recovering a genuinely stuck step in depth is the subject of recovering a stuck ILM step.

import json
import logging
from elasticsearch import Elasticsearch
from elasticsearch import ApiError, BadRequestError, NotFoundError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)

POLICY_ID = "app-events-ilm"


def load_good_revision(path: str, git_sha: str) -> dict:
    """Load the previous known-good policy JSON from git and stamp its source sha."""
    with open(path, encoding="utf-8") as fh:
        artifact = json.load(fh)
    # Record which revision we rolled back TO — the version integer will still bump forward.
    artifact["policy"].setdefault("_meta", {})["git_sha"] = git_sha
    return artifact


def rollback_policy(client: Elasticsearch, artifact: dict) -> dict:
    """Re-PUT the known-good definition. version increments; the actions revert."""
    before = client.ilm.get_lifecycle(name=POLICY_ID)[POLICY_ID]["version"]
    try:
        client.ilm.put_lifecycle(name=POLICY_ID, policy=artifact["policy"])
    except BadRequestError as exc:
        logger.error("Rollback body rejected: %s", exc.info)
        raise
    live = client.ilm.get_lifecycle(name=POLICY_ID)[POLICY_ID]
    logger.info("Rolled back %s: version %s -> %s, restored git_sha=%s",
                POLICY_ID, before, live["version"], live["policy"].get("_meta", {}).get("git_sha"))
    return live


def recover_indices(client: Elasticsearch, index_pattern: str = "app-events-*") -> None:
    """After reverting the definition, unstick any index the bad version left in ERROR."""
    explain = client.ilm.explain_lifecycle(index=index_pattern)
    for idx, state in explain.get("indices", {}).items():
        if state.get("step") != "ERROR":
            continue
        logger.warning("%s in ERROR: %s", idx, state.get("step_info"))
        try:
            # Per-index retry re-runs the failed step under the now-reverted definition.
            client.ilm.retry(index=idx)
            logger.info("Retried %s", idx)
        except ApiError as exc:
            logger.error("Retry failed for %s (%s): %s", idx, exc.status_code, exc.info)


def move_off_bad_step(client: Elasticsearch, index: str) -> None:
    """For an index that advanced into the wrong action, move it back to a valid step."""
    state = client.ilm.explain_lifecycle(index=index)["indices"][index]
    current = {"phase": state["phase"], "action": state["action"], "name": state["step"]}
    # Steer the index back to the hot-phase rollover check under the reverted policy.
    target = {"phase": "hot", "action": "rollover", "name": "check-rollover-ready"}
    client.ilm.move_to_step(index=index, current_step=current, next_step=target)
    logger.info("Moved %s from %s to %s", index, current["name"], target["name"])


if __name__ == "__main__":
    es = Elasticsearch("https://prod-es:9200", api_key="PROD_KEY", verify_certs=True)
    try:
        good = load_good_revision("repo/ilm/app-events-ilm.json", git_sha="9f8e7d6c5b4a")
        rollback_policy(es, good)     # 1. revert the definition
        recover_indices(es)           # 2. unstick any index left in ERROR
    except NotFoundError:
        logger.error("Policy %s not found — nothing to roll back.", POLICY_ID)
        raise

Note what the rollback does and does not touch. Re-PUTting the good body reverts the actions every index will run from its next step evaluation onward. It does not undo a delete that already fired: an app-events-* index the shortened delete phase already removed is gone, and no policy change brings it back — restore it from a snapshot instead. move_to_step and retry operate per index and only help indices that still exist and are still managed.

Verification

Confirm the definition reverted by reading the phase that was broken back to its correct value, and confirm the restored _meta.git_sha:

GET _ilm/policy/app-events-ilm?filter_path=**.version,**._meta,**.policy.phases.delete
{
  "app-events-ilm": {
    "version": 14,
    "policy": {
      "_meta": { "git_sha": "9f8e7d6c5b4a3f2e1d0c9b8a7f6e5d4c3b2a1f0e", "version": "2.3.0" },
      "phases": {
        "delete": { "min_age": "30d", "actions": { "delete": {} } }
      }
    }
  }
}

The delete.min_age is back to 30d and _meta reflects the good revision — even though version has climbed to 14 rather than returning to an earlier number, because it only ever increments. Then check the affected indices are no longer wedged:

GET app-events-*/_ilm/explain?filter_path=**.step,**.phase,**.step_info

Every remaining app-events-* index should report a non-ERROR step (typically hot/rollover/check-rollover-ready), confirming the reverted definition took and the recovery unstuck anything the bad version had halted.

Gotchas and edge cases

  • Re-applying does not rewind indices already advanced or deleted. The rollback changes future behavior only. An index the bad delete phase already removed cannot be recovered by any policy edit — restore it from a snapshot. An index that already force-merged or relocated under the bad version stays in that state; the good definition governs it from its next transition.
  • Version only increments — it never returns to the old number. After a rollback, version is higher than before, not lower, and modified_date reflects the rollback moment. Verify the rollback by inspecting the reverted phases and _meta.git_sha, never by expecting version to match a previous value.
  • Check in_use_by before assuming the rollback protects anything. If in_use_by is empty, the policy is attached to nothing and reverting it changes no index’s behavior — the real problem may be a missing template binding. Confirm in_use_by lists the app-events-* indices you intend to protect before and after the rollback.
  • Test the rollback in staging first. Apply the good revision to staging and run explain_lifecycle there before touching production, so a mistaken “good” revision (or one that references a repository production lacks) fails on staging rather than compounding the incident.

FAQ

Does Elasticsearch keep old policy versions I can roll back to?

No. A deployment stores only the current definition plus a version counter and modified_date; there is no server-side archive of previous bodies. Rollback means re-applying an earlier revision from git with put_lifecycle, which increments version forward while reverting the actual actions. Your source control is the only history that exists.

Will rolling back recover data the bad policy already deleted?

No. Re-applying the good definition only changes what indices do from their next step evaluation onward — it cannot resurrect an index a shortened delete phase already removed. Recover deleted data from a snapshot. For an index that merely advanced into the wrong phase but still exists, use ilm.move_to_step to steer it back to a valid step under the reverted policy.

An index is stuck in ERROR after I reverted the policy — how do I clear it?

Once the definition is reverted and the underlying cause resolved, call ilm.retry(index=...) to re-run the failed step under the good definition. If the index advanced into an invalid action, use ilm.move_to_step to move it to a valid step first. The deeper recovery playbook for a genuinely wedged step is covered in recovering a stuck ILM step.

← Back to Versioning ILM Policies Across Environments · ILM Policy Design & Lifecycle Synchronization