ILM & Reindexing Automation Runbooks

Alerting on ILM Errors with Kibana Stack Monitoring

Turn the ERROR-step signal that _ilm/explain exposes into an actual alert, because Elasticsearch ships no native rule that pages when a lifecycle stalls.

Kibana Stack Monitoring visualises cluster and index metrics ILM depends on, but it has no ILM-aware alerting rule — there is no checkbox that says “page me when an index gets stuck.” So alerting on lifecycle failures always means deriving a signal yourself and feeding it into a rule. This page shows the two ways to do that: a scheduled poller that scans _ilm/explain and emits a metric or log the alerting stack already understands, or a Kibana alerting rule evaluated over that derived signal. It sits alongside the read-only instrumentation in monitoring ILM health with the _cat and explain APIs and the manual remediation in manually triggering an ILM phase transition, all built on Index Lifecycle Management (ILM). The running example is the events-* pattern managed by the events-ilm policy.

What you page on matters as much as how. Three signals are worth waking someone for: any index whose step equals ERROR, the ILM service reporting anything other than RUNNING, and any data tier over its high disk watermark. The trap to avoid is paging on ordinary in-progress steps — an index sitting in check-rollover-ready or allocate is not broken, it is working, and an alert that fires on those trains everyone to ignore it.

Prerequisites

  • Elasticsearch 8.x with the events-ilm policy managing the events-* indices.
  • elasticsearch-py v8.0+ for the poller (ilm.explain_lifecycle, ilm.get_status).
  • A monitoring account with read_ilm and monitor — the poller only reads; it never remediates.
  • A sink the poller can emit to (a metrics backend, a log index Kibana can alert over, or a scheduler webhook) plus a Kibana alerting connector if you route through Kibana rules.

Implementation

1. A scheduled poller that emits a stuck-index signal

The poller scans the managed pattern each run, keeps only the genuinely halted indices, checks the global service switch, and returns a compact payload the sink can threshold. It emits a count and the offending index names, so a rule can fire on stuck_count > 0.

import json
import logging
from elasticsearch import Elasticsearch, ApiError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("ilm_alert")


def collect_ilm_alert_signal(es: Elasticsearch, pattern: str = "events-*") -> dict:
    """Return an alertable payload: stuck indices, service mode, and a page flag."""
    payload = {"stuck_count": 0, "stuck": [], "service_mode": "RUNNING", "should_page": False}

    # Global switch first: a non-RUNNING engine freezes the whole fleet.
    status = es.ilm.get_status()
    payload["service_mode"] = status["operation_mode"]

    try:
        resp = es.ilm.explain_lifecycle(index=pattern, human=True)
    except ApiError as exc:
        logger.error("explain failed (%s): %s", exc.status_code, exc.info)
        payload["scan_failed"] = True
        payload["should_page"] = True          # inability to scan is itself alertable
        return payload

    for idx, meta in resp["indices"].items():
        if not meta.get("managed"):
            continue
        # ONLY the ERROR step is a stall. Waiting steps are healthy and must not page.
        if meta.get("step") == "ERROR":
            payload["stuck"].append({
                "index": idx,
                "phase": meta.get("phase"),
                "failed_step": meta.get("failed_step"),
                "reason": (meta.get("step_info") or {}).get("reason"),
            })

    payload["stuck_count"] = len(payload["stuck"])
    payload["should_page"] = (
        payload["stuck_count"] > 0 or payload["service_mode"] != "RUNNING"
    )
    return payload


def emit(payload: dict) -> None:
    """Sketch of wiring to a sink: write one structured record the alert rule reads.

    Replace the body with your transport — index into a monitoring index, push a
    gauge to a metrics backend, or POST to a scheduler webhook. Keep it one record
    per run so the rule can threshold on stuck_count and dedupe on index name.
    """
    logger.info("ilm_alert_signal %s", json.dumps(payload, sort_keys=True))


if __name__ == "__main__":
    es = Elasticsearch("https://es-monitor-01:9200",
                       api_key="YOUR_READ_ILM_MONITOR_API_KEY", verify_certs=True)
    signal = collect_ilm_alert_signal(es, "events-*")
    emit(signal)
    if signal["should_page"]:
        logger.warning("PAGE: %d stuck, service=%s",
                       signal["stuck_count"], signal["service_mode"])

Run it on a scheduler (cron, a Kubernetes CronJob, a Watcher-adjacent job) at roughly the ILM poll interval. The emit function is deliberately a stub: point it at whatever your team already alerts over. If you index the payload into a monitoring index, a Kibana alerting rule of the form “documents where stuck_count > 0 in the last interval” gives you the page without any custom code — the poller does the ILM-specific classification and Kibana does the notification.

2. Routing through a Kibana alerting rule

If you would rather keep the logic in Kibana, index the poller’s per-index stuck records into a dedicated index (for example ilm-health-signals) and build an alerting rule over it: group by index, condition on the presence of an ERROR record in the last window, and connect it to your notification channel. The poller still does the classification — Kibana Stack Monitoring cannot read the ERROR step on its own — but the rule engine, deduplication, and connectors come for free.

Verification

Prove the alert fires by forcing a benign, reversible ERROR in staging rather than waiting for a real one. The cleanest way is to point an index’s rollover alias at the wrong target, which makes the check-rollover-ready step fail into ERROR:

POST _ilm/move/events-000002
{
  "current_step": { "phase": "hot", "action": "rollover", "name": "check-rollover-ready" },
  "next_step":    { "phase": "hot", "action": "rollover", "name": "check-rollover-ready" }
}

More reliably, temporarily stop the service and confirm the service-mode branch pages:

POST _ilm/stop

Within one poll cycle, collect_ilm_alert_signal should return service_mode: "STOPPED" and should_page: true. Confirm the emitted record shows it:

GET _ilm/status
{ "operation_mode": "STOPPED" }

Once the alert has fired and you have confirmed the page arrived, restore normal operation so nothing stays frozen:

POST _ilm/start

Re-run the poller and confirm should_page returns to false with service_mode: "RUNNING" and an empty stuck list — proving the alert clears as well as fires.

Gotchas and edge cases

  • Poll interval versus poll_interval. ILM only re-evaluates the state machine every indices.lifecycle.poll_interval (default 10m). Scheduling the alert poller far faster does not surface stalls any sooner — it just re-reports identical steps and adds master-node load. Match the poller to the ILM interval; a stall becomes visible one ILM cycle after it happens regardless of how often you poll.
  • Transient waiting steps are not errors — never page on them. check-rollover-ready, allocate, shrink, and forcemerge are in-progress states an index legitimately occupies for minutes to hours. The poller must filter to step == "ERROR" only. Alerting on any non-complete step produces constant false pages and destroys trust in the signal.
  • Dedupe flapping. An index can briefly enter ERROR, get retried by ILM, and recover, and a naive rule pages on every flap. Require the ERROR to persist across two consecutive poll windows before paging, and group notifications by index name so one stuck index is one incident, not one page per poll.
  • Scope by managed pattern. Alert over events-*, not *. A wide pattern drags unmanaged and system indices into the scan, inflates the master-node cost of each _ilm/explain call, and risks paging on indices no one owns. Keep the pattern tight to the lifecycle you actually run.

FAQ

Does Kibana Stack Monitoring alert on stuck ILM indices out of the box?

No. Stack Monitoring charts cluster and index metrics ILM runs on, but it has no rule that reads the lifecycle ERROR step. You derive the signal yourself — a scheduled poller that scans _ilm/explain and emits a metric or log — and then alert on that derived signal, either with a Kibana alerting rule over the emitted records or with your existing metrics-backend alerting.

Which conditions actually deserve a page?

Three: any index whose step equals ERROR (a genuine stall), _ilm/status reporting anything other than RUNNING (the whole engine is frozen), and any data tier over its high disk watermark (the usual upstream cause of allocate stalls). Rollover lag — a write index past its rollover condition that has not rolled — is worth a warning. In-progress steps like check-rollover-ready are never a page.

How do I stop the alert from flapping?

Require the ERROR to persist across two consecutive poll windows before paging, and deduplicate by index name so a single stuck index is one incident rather than one page per poll. ILM retries some failures on its own, so a one-shot ERROR that clears on the next cycle should resolve silently; only a stall that survives multiple polls warrants waking someone.

← Back to Monitoring ILM Health · ILM Architecture & Fundamentals