Monitoring ILM Health with the _cat and Explain APIs #

Index Lifecycle Management does its work quietly and asynchronously, which is exactly why it fails quietly too. A policy that stops advancing does not raise an exception, does not fail a request, and does not appear in any application log — the write path keeps accepting documents while, underneath, an index that should have migrated to warm three days ago sits pinned to the hot tier, or a delete phase that should have reclaimed a terabyte never fires. By the time someone notices, it is usually because a data tier crossed a disk watermark and ingestion started failing cluster-wide. The only defence is to instrument the lifecycle directly: poll _ilm/explain across every managed pattern, classify what each index is actually doing, and correlate the ambiguous cases against allocation and capacity signals so a genuine stall is distinguishable from a step that is simply waiting. This page builds that observability surface on the control plane described in Index Lifecycle Management (ILM), using the running example of an events-* index pattern managed by a policy named events-ilm.

The central fact to internalise is that ILM has no dedicated health endpoint that says “everything is fine.” There is a per-index execution view (_ilm/explain), a global service switch (_ilm/status), and a set of _cat and allocation APIs that describe the deployment the lifecycle runs on — and health is a property you derive by reading all of them together. Kibana Stack Monitoring surfaces the underlying cluster and index metrics but ships no ILM dashboard, so production alerting is always built on explain polling plus _cat thresholds rather than a turnkey rule.

Prerequisites #

• Elasticsearch 8.x with data-tier node roles assigned (data_hot, data_warm, data_cold) so the allocation signals you monitor map onto real tiers.
• A monitoring service account holding cluster privilege read_ilm (to call explain and status) plus monitor (for the _cat and allocation APIs). Deliberately withhold manage_ilm from the poller — observation must not carry mutation rights.
• elasticsearch-py v8.0+ installed — the poller below uses the v8 client surface (ilm.explain_lifecycle, ilm.get_status, keyword arguments, typed exceptions), not the legacy body= pattern.
• An alerting sink your team already runs — a metrics backend, a log pipeline with alert rules, or a scheduler that can page on a boolean. The poller emits signals; the sink decides who gets woken.
• At least one managed index actually flowing through events-ilm, so the explain output has real steps to classify rather than an empty indices map.

Architecture: A Closed Monitoring Loop #

Health monitoring for ILM is a loop, not a snapshot. Each cycle reads the execution state of every managed index, classifies each index into one of three buckets — healthy, in-progress, or halted — and then, for the non-healthy buckets, digs one layer deeper into allocation before deciding whether to alert. The classification step is where most naive monitoring goes wrong: it treats any non-complete step as a problem and pages on the normal check-rollover-ready waiting state, training operators to ignore the alert. A correct loop separates the ERROR step (a true stall) from the ordinary in-progress steps (allocate, shrink, forcemerge, check-rollover-ready) that are simply mid-flight.

The loop has a natural cadence: it should run at roughly the same frequency as indices.lifecycle.poll_interval (default 10m), because polling faster than ILM itself evaluates the state machine only re-reports the same unchanged steps. The three data sources it reads are complementary. _ilm/explain tells you what the lifecycle thinks it is doing; _cat/allocation and _cluster/allocation/explain tell you whether the deployment can actually do it; and _ilm/status tells you whether the lifecycle engine is even running. A stall that shows up as an ERROR step in explain almost always has its real cause in the allocation layer — a tier with no eligible node, or a tier over its high disk watermark — which is why the correlate stage exists rather than alerting straight off the explain output. The tier topology those signals describe is the same one covered in configuring index rollover conditions, where the hot phase hands each rolled index to the later phases this loop watches.

Configuration Reference #

The reference surface here is not a policy to apply but a set of read APIs to interpret. Start with a healthy _ilm/explain response for a write index that is polling its rollover conditions — this is what “everything is fine” looks like, and the classifier must treat it as a non-event:

GET events-000007/_ilm/explain
{
  "indices": {
    "events-000007": {
      "index": "events-000007",
      "managed": true,                       // policy is attached; unmanaged => ignored by ILM
      "policy": "events-ilm",                // which policy governs this index
      "phase": "hot",
      "action": "rollover",
      "step": "check-rollover-ready",         // healthy steady state: polling the conditions
      "age": "16.4h",                        // time since rollover, not since creation
      "phase_execution": {
        "policy": "events-ilm",
        "version": 4,                        // policy version this index is executing
        "modified_date_in_millis": 1719763200000
      }
    }
  }
}

Contrast that with a halted index. The single field that matters is step: "ERROR", and the single field that explains it is step_info, which carries the underlying exception verbatim:

GET events-000005/_ilm/explain
{
  "indices": {
    "events-000005": {
      "index": "events-000005",
      "managed": true,
      "policy": "events-ilm",
      "phase": "warm",
      "action": "allocate",
      "step": "ERROR",                        // the stuck signal — this is what you page on
      "failed_step": "check-allocation",      // the step that threw before ERROR
      "age": "3.2d",
      "step_info": {
        "type": "illegal_state_exception",
        "reason": "Cannot allocate index [events-000005] to tier [data_warm]: no node holds role [data_warm]"
      },
      "phase_execution": { "policy": "events-ilm", "version": 4 }
    }
  }
}

The key _cat and allocation queries — each pinned to an explicit column list with h= so the output is stable enough to parse and threshold:

GET _ilm/status

GET _cat/allocation?v&h=node,shards,disk.used,disk.avail,disk.percent

GET _cat/nodes?v&h=name,heap.percent,disk.used_percent,node.role

GET _cat/shards/events-*?v&h=index,shard,prirep,state,store,node

GET _cluster/allocation/explain
{ "index": "events-000005", "shard": 0, "primary": true }

_ilm/status returns a single field, operation_mode, whose value is RUNNING, STOPPING, or STOPPED; anything other than RUNNING freezes every managed index simultaneously and is a single cluster-level page, not a per-index one. The _cat/allocation disk.percent column is what you threshold against the high watermark; _cluster/allocation/explain is the only API that returns a human-readable explanation string naming why a specific shard cannot move.

Step-by-Step Implementation: A Health Poller #

The poller runs the loop from the architecture diagram in code. Each pass classifies every managed index, checks the global service switch, and samples per-node disk pressure, returning a structured set of signals for the alerting sink to consume.

1. Read the global switch first. If get_status() reports anything but RUNNING, nothing else matters — every index is frozen — so short-circuit to a single cluster-level signal.
2. Scan explain across the managed pattern. Classify each index as error, lagging, or healthy. An index is lagging when its age in a phase exceeds a generous multiple of the expected min_age without an ERROR — a rollover or transition that is late but not yet failed.
3. Sample node disk pressure. Pull _cat/nodes and flag any node whose disk.used_percent has crossed the high-watermark threshold, because that is the most common upstream cause of the allocate stalls the explain scan surfaces.
4. Emit signals. Return the collected signals; the caller forwards them to the sink.

import logging
from elasticsearch import Elasticsearch, ApiError, ConnectionError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("ilm_health")

HIGH_WATERMARK_PCT = 85          # align with cluster.routing.allocation.disk.watermark.high
LAG_MULTIPLE = 3                 # phase age this many times over min_age => lagging


class ILMHealthPoller:
    def __init__(self, es: Elasticsearch, pattern: str = "events-*"):
        self.es = es
        self.pattern = pattern

    def check_service(self) -> dict | None:
        # Global kill-switch: a STOPPED engine halts every managed index at once.
        status = self.es.ilm.get_status()
        mode = status["operation_mode"]
        if mode != "RUNNING":
            return {"signal": "ilm_service_not_running", "operation_mode": mode}
        return None

    def scan_indices(self) -> dict[str, list]:
        errors, lagging = [], []
        try:
            resp = self.es.ilm.explain_lifecycle(index=self.pattern, human=True)
        except ApiError as exc:
            logger.error("explain failed (%s): %s", exc.status_code, exc.info)
            return {"errors": [], "lagging": [], "scan_failed": True}

        for idx, meta in resp["indices"].items():
            if not meta.get("managed"):
                continue                                  # unmanaged indices are out of scope
            step = meta.get("step")
            if step == "ERROR":
                # The one unambiguous stall. step_info carries the real cause.
                errors.append({
                    "index": idx,
                    "phase": meta.get("phase"),
                    "action": meta.get("action"),
                    "failed_step": meta.get("failed_step"),
                    "step_info": meta.get("step_info"),
                })
            elif self._is_lagging(meta):
                lagging.append({"index": idx, "phase": meta.get("phase"),
                                "age_millis": meta.get("age_in_millis")})
        return {"errors": errors, "lagging": lagging, "scan_failed": False}

    def _is_lagging(self, meta: dict) -> bool:
        # A late-but-not-failed transition: still in an in-progress step, but the
        # phase age dwarfs the age we would expect before it should have advanced.
        age = meta.get("age_in_millis")
        expected = meta.get("phase_execution", {}).get("min_age_in_millis")
        if age is None or not expected:
            return False
        return age > expected * LAG_MULTIPLE

    def check_disk(self) -> list[dict]:
        hot = []
        rows = self.es.cat.nodes(format="json",
                                 h="name,disk.used_percent,node.role")
        for row in rows:
            used = row.get("disk.used_percent")
            if used and float(used) >= HIGH_WATERMARK_PCT:
                hot.append({"node": row["name"], "disk_used_percent": float(used),
                            "roles": row.get("node.role")})
        return hot

    def poll(self) -> dict:
        signals = {"service": None, "errors": [], "lagging": [], "disk": []}
        try:
            svc = self.check_service()
            if svc:
                signals["service"] = svc
                return signals               # frozen engine: nothing else is actionable
            scan = self.scan_indices()
            signals["errors"] = scan["errors"]
            signals["lagging"] = scan["lagging"]
            signals["disk"] = self.check_disk()
        except (ApiError, ConnectionError) as exc:
            logger.error("health poll failed: %s", exc)
            signals["poll_failed"] = True
        return signals


if __name__ == "__main__":
    es = Elasticsearch(
        "https://es-monitor-01:9200",
        api_key="YOUR_READ_ILM_MONITOR_API_KEY",
        verify_certs=True,
        request_timeout=30,
    )
    result = ILMHealthPoller(es, "events-*").poll()
    if result["errors"]:
        logger.warning("%d index(es) in ERROR: %s",
                       len(result["errors"]), [e["index"] for e in result["errors"]])
    if result["disk"]:
        logger.warning("%d node(s) over high watermark", len(result["disk"]))

The poller reads only — it never calls move_to_step, retry, or put_lifecycle. That separation is deliberate: monitoring authenticates with a read_ilm + monitor key, and remediation is a distinct, privileged action documented under manually triggering an ILM phase transition.

Verification #

Run the poller against a healthy cluster and it should return empty errors, lagging, and disk lists with service: None. To prove it actually detects a stall, inspect the raw explain output the poller consumes. A healthy scan looks like this:

GET events-*/_ilm/explain?human&filter_path=indices.*.index,indices.*.step,indices.*.phase

{
  "indices": {
    "events-000006": { "index": "events-000006", "phase": "warm", "step": "complete" },
    "events-000007": { "index": "events-000007", "phase": "hot",  "step": "check-rollover-ready" }
  }
}

Every step is either complete or an in-progress step name — no ERROR. Confirm the service switch independently:

GET _ilm/status

{ "operation_mode": "RUNNING" }

Finally, sample disk pressure per node and confirm no tier is near its high watermark:

GET _cat/nodes?v&h=name,heap.percent,disk.used_percent,node.role

name        heap.percent disk.used_percent node.role
es-hot-01             61                72 dhimr
es-warm-01            44                58 dw
es-cold-01            38                49 dc

With disk.used_percent comfortably under 85 on every node, the allocate steps in the warm and cold phases have somewhere to land, and the poller’s disk check stays quiet.

Threshold Tuning & Performance Guidance #

Monitoring ILM is cheap, but it is not free, and the two knobs that matter are poll frequency and query scope.

1. Match your poll interval to indices.lifecycle.poll_interval, do not beat it. ILM evaluates the state machine every 10m by default. Polling explain every thirty seconds returns the same unchanged steps twenty times before the engine even re-evaluates, adding master-node load for no new information. A poll every 5–10 minutes catches a stall within one ILM cycle, which is as fast as the lifecycle itself moves.
2. _ilm/explain runs on the master node. It reads cluster state, so a very wide pattern across thousands of indices makes the master do real work each poll. Scope the pattern to what is actually managed (events-*, not *), and if you monitor many patterns, stagger them rather than firing them all on the same tick.
3. Alert on the derived signals, not on step transitions. The four production-worthy signals are: any index with step == ERROR; _ilm/status other than RUNNING; any tier node over its high disk watermark; and rollover lag — a write index whose age and size exceed its rollover condition without having rolled. Everything else is noise.
4. Set the lag multiple generously. A min_age boundary is a floor, not a schedule, and ILM only advances on its poll tick, so an index legitimately sits past min_age for up to a full poll interval. A lag multiple of 3× the expected phase age keeps the lagging bucket free of false positives while still catching a transition that is genuinely days late.

The heaviest single call in the loop is _cluster/allocation/explain, which is why the poller only reaches for it (in the correlate stage) against indices already classified as ERROR, never on every index every cycle.

Troubleshooting #

An ERROR step versus a normal waiting step #

Symptom: every managed index shows a non-complete step and it is unclear which are problems. Resolution: only step == "ERROR" is a stall. Steps like check-rollover-ready, allocate, check-allocation, shrink, and forcemerge are in-progress states — the index is mid-transition and will advance on the next poll. Read step_info only when step is ERROR; on a healthy step it is either absent or a benign “waiting for conditions” message. Never page on an in-progress step.

ILM service left STOPPED #

Symptom: nothing transitions anywhere, no index shows ERROR, and every index simply sits at its current step indefinitely. Resolution: check GET _ilm/status. ILM is commonly stopped for maintenance (POST _ilm/stop) and never restarted. If operation_mode is STOPPED or stuck STOPPING, run POST _ilm/start and confirm it returns to RUNNING. This pages cluster-wide, not per-index — a stopped engine freezes the entire fleet at once.

Reading _cluster/allocation/explain for a hung transition #

Symptom: an index sits in ERROR with a step_info pointing at allocation, or hangs in allocate/shrink without erroring. Resolution: call GET _cluster/allocation/explain with the specific index/shard/primary and read the explanation string and the per-node deciders. The usual verdicts are NO from the data_tier decider (no node carries the target tier role) or the disk_threshold decider (the tier is over its high watermark). Fix the underlying capacity or role problem — the explain output names it — before you retry the step, because a retry into a still-full tier just returns the index to ERROR.

The poller returns an empty indices map #

Symptom: explain_lifecycle succeeds but classifies nothing. Resolution: the pattern matched no managed indices. Confirm the indices exist and actually carry index.lifecycle.name; an index created without the policy (a common reindex-destination mistake) is managed: false and is invisible to lifecycle monitoring even though it exists.

FAQ #

Related #

• Manually Triggering an ILM Phase Transition — the privileged _ilm/move runbook for the stalls this monitoring surfaces.
• Alerting on ILM Errors with Kibana Stack Monitoring — turning the ERROR-step signal into a page.
• Monitoring ILM Execution and Error States — the policy-design view of diagnosing execution failures.
• Configuring Index Rollover Conditions — the hot-phase boundary that feeds every later phase this loop watches.

← Back to ILM Architecture & Fundamentals