ILM & Reindexing Automation Runbooks

Recovering a Stuck ILM Step

An index has stopped moving through its lifecycle — it either sits in the ERROR step or lingers in one step that never advances — and this runbook walks the diagnosis-to-recovery path that gets it moving again without guessing.

A stalled index is not an ILM outage; the coordinator is still polling, but this one index has hit a precondition it cannot satisfy and has parked itself. The recovery discipline is the same every time: read the authoritative state from the explain API, let step_info name the fault, repair the underlying constraint, and only then resume the state machine. This page is the hands-on continuation of monitoring ILM execution and error states, which is where you learn to detect the stall; here we clear it. Both live under the broader discipline of ILM policy design and lifecycle synchronization, where a declared policy is continuously reconciled against what the deployment is actually running.

Prerequisites

  • Elasticsearch 8.x with the ILM coordinator confirmed RUNNING (GET _ilm/status returns "operation_mode": "RUNNING") — if the coordinator is stopped, no index advances and no amount of retry helps.
  • elasticsearch-py v8.0+ (pip install "elasticsearch>=8,<9"); the code below uses the v8 keyword-argument surface (ilm.explain_lifecycle, ilm.retry, ilm.move_to_step), not the legacy body= pattern.
  • manage_ilm on the account that calls POST <index>/_ilm/retry and POST _ilm/move/<index>, kept separate from read-only monitoring per securing ILM policies with RBAC.
  • Read access to GET _cluster/allocation/explain and GET _cat/aliases, the two tools that confirm the two most common root causes.

The recovery decision path

The single rule that separates a real fix from a retry loop: retry re-runs the failed step exactly as written, so if the world has not changed since it failed, it fails again. Read step_info, change the world, then retry. The flow below is the whole runbook in one picture — explain, read the fault, repair the cause, and choose retry (the common case) over move (the wedged-step exception).

Diagnose with explain, fix the root cause, then retry the failed step or move a wedged oneA left-to-right recovery path. Stage one, GET index/_ilm/explain, feeds stage two, Read step_info, which lists the four usual faults: rollover alias mismatch, allocation impossible from no eligible node or a disk watermark, unmet shrink preconditions, and a missing snapshot repository. An arrow leads to Fix root cause. From there a decision splits: the upper branch, taken when the failed step is correct, goes to POST index/_ilm/retry and is labelled the common path; the lower branch, taken when the step is wedged, goes to POST _ilm/move/index with an exact current_step and next_step. Both branches converge on Step advances off ERROR.Explain_ilm/explainRead step_infoalias mismatch · watermarkno eligible node · shrinkmissing snapshot repoFix root causechange the worldstep OKPOST _ilm/retryre-run failed step · commonstep wedgedPOST _ilm/moveexact current → next stepStep advances off ERROR

Implementation

1. Read the fault from explain

The explain API is the only authoritative source — never infer the problem from the index name or age. When a step fails, the current step becomes the literal string ERROR, failed_step names which step broke, and step_info carries the machine-readable type and a human-readable reason that usually names the fix outright.

GET logs-app-000042/_ilm/explain?human

A rollover-alias mismatch is the single most common stall — ILM cannot roll an index whose rollover_alias no longer resolves to it:

{
  "indices": {
    "logs-app-000042": {
      "managed": true,
      "policy": "logs-app-policy",
      "phase": "hot",
      "action": "rollover",
      "step": "ERROR",
      "failed_step": "check-rollover-ready",
      "step_info": {
        "type": "illegal_argument_exception",
        "reason": "index.lifecycle.rollover_alias [logs-write] does not point to index [logs-app-000042]"
      }
    }
  }
}

The four causes you will see the most: a rollover alias that no longer points at the write index; an allocation the deployment cannot satisfy because no node carries the required data-tier attribute or the destination tier crossed a disk watermark; a shrink whose preconditions are unmet (the source shards are not co-located on one node, or the target name is taken); and a missing snapshot repository for a searchable-snapshot or wait-for-snapshot step. Each has a distinct repair, and step_info.reason tells you which one you have.

2. Repair the root cause

Match the fault to its fix before touching retry. For the alias mismatch above, re-point the write alias:

POST _aliases
{
  "actions": [
    { "add": { "index": "logs-app-000042", "alias": "logs-write", "is_write_index": true } }
  ]
}

For an allocation stall, confirm the real blocker with GET _cluster/allocation/explain — it names the missing node attribute or the crossed watermark — and clear it by tagging a node, freeing disk on the destination tier, or relaxing the watermark. For a shrink, ensure the source index is allocated to a single node and the target name is free. For a snapshot step, register or repair the repository and verify it with POST _snapshot/<repo>/_verify.

3. Resume the lifecycle from Python

Once the cause is fixed, re-run the failed step with POST <index>/_ilm/retry. The v8 client exposes this as client.ilm.retry(index=...) — there is no retry_lifecycle method, and API/HTTP failures raise ApiError, which is a sibling of TransportError and will not be caught by it. The helper below sweeps a pattern, retries only indices actually in ERROR, and reports what it did.

import logging
from elasticsearch import Elasticsearch, ApiError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("ilm_recover")


class StuckStepRecovery:
    def __init__(self, es: Elasticsearch):
        self.es = es

    def find_errored(self, pattern: str = "logs-*") -> dict[str, dict]:
        # only_errors trims the response to indices currently in an ERROR step.
        resp = self.es.ilm.explain_lifecycle(index=pattern, only_errors=True, human=True)
        errored = {}
        for idx, meta in resp.get("indices", {}).items():
            info = meta.get("step_info", {})
            logger.warning(
                "%s stuck at failed_step=%s: %s — %s",
                idx, meta.get("failed_step"),
                info.get("type", "unknown"), info.get("reason", ""),
            )
            errored[idx] = meta
        return errored

    def retry(self, index: str) -> bool:
        # retry ONLY works on an index currently in ERROR; it re-runs the failed
        # step and does not bypass min_age. Fix the cause BEFORE calling this,
        # or the step drops straight back to ERROR on the next poll.
        try:
            self.es.ilm.retry(index=index)
            logger.info("Retry submitted for %s", index)
            return True
        except ApiError as exc:
            # 400 here usually means the index is no longer in ERROR.
            logger.error("Retry failed for %s (%s): %s", index, exc.status_code, exc.info)
            return False

    def move(self, index: str, current_step: dict, next_step: dict) -> bool:
        # Escape hatch for a wedged step: jump to a known-good position.
        # current_step must match the index's CURRENT (phase, action, name) exactly.
        try:
            self.es.ilm.move_to_step(
                index=index, current_step=current_step, next_step=next_step,
            )
            logger.info("Moved %s from %s to %s", index, current_step, next_step)
            return True
        except ApiError as exc:
            logger.error("Move failed for %s (%s): %s", index, exc.status_code, exc.info)
            return False


if __name__ == "__main__":
    es = Elasticsearch(
        "https://es-cluster-01:9200",
        api_key="YOUR_BASE64_ENCODED_API_KEY",
        verify_certs=True,
    )
    recovery = StuckStepRecovery(es)

    stuck = recovery.find_errored("logs-*")
    for name in stuck:
        # Assumes the operator has already repaired the underlying cause.
        recovery.retry(name)

If the step itself is wrong — a corrected policy left the index pointed at a step that no longer applies, or the step is genuinely wedged — use move_to_step (REST POST _ilm/move/<index>) to jump the index to a known-good position. The current_step must match the index’s current (phase, action, name) exactly, or the API rejects the move:

POST _ilm/move/logs-app-000042
{
  "current_step": { "phase": "hot", "action": "rollover", "name": "check-rollover-ready" },
  "next_step":    { "phase": "warm", "action": "allocate", "name": "check-allocation" }
}

Verification

Recovery is proven by explain, not by the absence of an error in the retry response. Poll the same index and confirm step is no longer ERROR and failed_step is gone:

GET logs-app-000042/_ilm/explain?human
{
  "indices": {
    "logs-app-000042": {
      "managed": true,
      "phase": "hot",
      "action": "rollover",
      "step": "check-rollover-ready",
      "step_info": { "message": "Waiting for index to meet rollover conditions" }
    }
  }
}

The step has advanced off ERROR back into a normal waiting state — check-rollover-ready here means ILM is polling the rollover conditions again. For cluster-wide confirmation that nothing is still parked, sweep with the error filter and expect an empty result:

GET _all/_ilm/explain?only_errors=true

An empty indices object means no managed index remains in an error step.

Gotchas and edge cases

  • Retrying without fixing the cause just loops back to ERROR. retry re-runs the step exactly as it failed; if the alias still does not resolve or the watermark is still crossed, the very next poll returns the index to ERROR. Repair the world first — the explain output before and after a fix should differ, or you have not actually changed anything.
  • A poll_interval delay is not a stuck step. The coordinator re-evaluates on indices.lifecycle.poll_interval (default 10m), so an index can wait minutes between a satisfied precondition and the visible advance. Only step == "ERROR", or a step whose step_time_millis has not moved across several intervals, is genuinely stuck — a fresh wait is normal.
  • move requires the exact current step. POST _ilm/move/<index> fails unless current_step matches the index’s live (phase, action, name) tuple. Read it straight from explain immediately before the call; a stale tuple from an earlier poll is the usual reason a move is rejected. Reserve move for genuinely wedged steps — retry is the correct tool whenever the step itself is sound.
  • Allocation and watermarks are the real cause more often than the policy. A stall that presents as check-allocation or a WAITING transition is almost always a node missing its data-tier attribute or a destination tier past cluster.routing.allocation.disk.watermark.high — not a broken policy. Confirm with _cluster/allocation/explain and free capacity before you edit anything in the lifecycle.

FAQ

What is the difference between _ilm/retry and _ilm/move?

POST <index>/_ilm/retry re-runs the step that is currently in ERROR, unchanged — it is the right tool when the step itself is correct and you have just fixed the precondition it was waiting on. POST _ilm/move/<index> deliberately jumps the index from one (phase, action, name) position to a different one, which you only need when the step is genuinely wedged or a corrected policy left the index pointed at a step that no longer applies. Reach for retry first; move is the escape hatch.

Why does my index return to ERROR immediately after I retry it?

Because the underlying cause was never fixed. retry replays the failed step against the current cluster state, so if the rollover alias still does not point to the index, or the destination tier is still over its disk watermark, the step fails again on the next poll. Read step_info.reason, make a concrete change that addresses it, and confirm the fix independently (for example _cat/aliases or _cluster/allocation/explain) before retrying.

Can I call retry on an index that is not in ERROR?

No. retry only applies to an index whose current step is ERROR; calling it on a healthy index returns a 400. If an index is merely slow to advance, it is waiting on the poll interval or a min_age timer, not stuck — and retry does not fast-forward either of those. Use explain to confirm the index is actually in ERROR before you attempt recovery.

← Back to Monitoring ILM Execution & Error States · ILM Policy Design & Lifecycle Synchronization