Detecting ILM Policy Drift #

An ILM policy is only as trustworthy as the guarantee that the definition you reviewed and merged is the definition the deployment is actually enforcing. That guarantee erodes silently. Someone edits a phase in Kibana during an incident, a reindex recreates an index without re-attaching its policy, or a template change lands in one environment and not another — and now the retention contract you believe you are running is fiction. This page is about detecting that divergence before it deletes data early, strands indices on the hot tier, or lets a supposedly-managed index accumulate shards forever. It is the read-side companion to the whole discipline of ILM policy design and lifecycle synchronization: you cannot reconcile drift you cannot see.

Drift is not one condition — it is three, and each breaks retention in a different way. The first is live-versus-canonical drift: the policy body stored in the deployment no longer matches the JSON in your repository. The second is per-index phase execution lag: an index is still running an older snapshot of the policy because it entered a phase before you last edited the definition, so the phase it will execute is not the phase you last wrote. The third, and quietest, is unmanaged drift: an index inherited no index.lifecycle.name at all and will never roll over, migrate, or delete — it simply grows until a disk watermark blocks writes. Elasticsearch ships no _ilm/diff or _ilm/validate endpoint, so detection is a job you build: pull the live state, compare it to the source of truth, and emit a per-index verdict. Throughout, the working example is a policy named metrics-ilm bound to the metrics-* index pattern, whose canonical body lives as metrics-ilm.json in a git repository.

Prerequisites #

• Elasticsearch 8.x cluster with ILM enabled and the coordinator RUNNING (GET _ilm/status).
• The canonical policy body checked into version control as JSON — this repository is the source of truth every comparison runs against.
• elasticsearch-py v8.0+ installed (pip install "elasticsearch>=8,<9"); the scripts use the v8 keyword-argument surface (client.ilm.get_lifecycle, client.ilm.explain_lifecycle, client.indices.get_settings), not the legacy body= pattern.
• A service account with read_ilm and monitor cluster privileges plus view_index_metadata on the metrics-* pattern — detection is read-only and must not carry write privileges.
• Read access to GET _ilm/policy, GET <index>/_ilm/explain, and GET <pattern>/_settings so the job can pull all three drift signals.
• A convention for what “current” means: the version field Elasticsearch stamps on every policy on each put_lifecycle, which is the anchor for detecting phase-execution lag.

Architecture: Three Signals, One Source of Truth #

Detection reads three independent facts from the deployment and compares each against the repository. The canonical body carries no version of its own — Elasticsearch assigns version and modified_date server-side — so the comparison is structural: normalize both phases dictionaries and test for equality. The live policy body answers “is the definition correct?” The per-index explain output answers “is each index running the current definition?” And the settings scan answers “is every index that should be managed actually managed?” Only when all three agree does the deployment match its contract.

The reconcile loop on the right is deliberately out of scope for this page — detection stops at the drift report. What matters here is that each signal is derived from a different API and answers a different question, so a clean report requires all three to be pulled and evaluated independently. A job that checks only the policy body will happily report “in sync” while a dozen indices quietly execute a stale phase or run unmanaged. Fixing what the report finds is covered in reconciling ILM policy drift with Python and, for the unmanaged case, auditing indices with no ILM policy attached.

Configuration Reference #

Two read endpoints supply every signal. Start with GET _ilm/policy/metrics-ilm, which returns the live body plus the server-managed metadata that anchors comparison:

GET _ilm/policy/metrics-ilm

{
  "metrics-ilm": {
    "version": 7,                              // bumped on every put_lifecycle; the "current" anchor
    "modified_date": "2026-07-15T09:22:04.511Z", // when the live body last changed
    "modified_date_millis": 1768469, 
    "policy": {                                 // the body you compare against metrics-ilm.json
      "phases": {
        "hot":  { "min_age": "0ms", "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" }, "set_priority": { "priority": 100 } } },
        "warm": { "min_age": "2d",  "actions": { "forcemerge": { "max_num_segments": 1 }, "set_priority": { "priority": 50 } } },
        "delete": { "min_age": "30d", "actions": { "delete": {} } }
      }
    },
    "in_use_by": {                              // what currently references this policy
      "indices": ["metrics-000012", "metrics-000013"],
      "data_streams": [],
      "composable_templates": ["metrics-template"]
    }
  }
}

Three fields drive detection. version is the integer Elasticsearch increments on each successful put_lifecycle; it is the “current definition” marker every index’s execution version is compared against. policy.phases is the structure you deep-compare against metrics-ilm.json — but note the live body carries server-normalized defaults (an implicit set_priority, canonicalized durations, action key ordering) that the canonical file may omit, so a naive equality check produces false positives unless you normalize first. in_use_by tells you what actually references the policy: an empty indices and data_streams list on a policy you believe is in production is itself a drift signal — nothing is using it.

The second endpoint, GET metrics-*/_ilm/explain, reveals per-index execution state and the version each index is running:

GET metrics-000012/_ilm/explain

{
  "indices": {
    "metrics-000012": {
      "index": "metrics-000012",
      "managed": true,                          // false here is the unmanaged-drift signal
      "policy": "metrics-ilm",
      "phase": "warm",
      "action": "forcemerge",
      "step": "check-rollover-ready",
      "phase_execution": {
        "policy": "metrics-ilm",
        "version": 5,                           // running version 5 while the policy is at 7 → lag
        "modified_date_in_millis": 1768120
      }
    }
  }
}

phase_execution.version is the crux of per-index lag: this index cached the policy definition as it stood at version 5 when it entered its current phase, and it will keep executing that snapshot until it transitions phases or is re-applied. If the live policy is at version 7, metrics-000012 is running a two-revision-old contract. managed: false, meanwhile, is the unmanaged signal — the index carries no index.lifecycle.name and ILM ignores it entirely.

Step-by-Step Implementation #

Detection runs in four stages: fetch and compare the policy body, scan per-index execution versions, list unmanaged indices, then fold all three into a single report.

1. Fetch canonical and live, deep-diff the bodies #

Load metrics-ilm.json from the repository and the live body from the deployment, then compare only the phases dictionary. Never compare version or modified_date — they are server-owned and always differ.

GET _ilm/policy/metrics-ilm

# The canonical source of truth, versioned in git.
cat policies/metrics-ilm.json

The comparison must normalize both sides first, because Elasticsearch echoes defaults the canonical file usually omits. Comparing raw dictionaries reports drift on every run even when nothing changed.

2. Explain all managed indices and flag version lag #

Call the explain API across the whole pattern and read each index’s phase_execution.version. Any value below the live policy’s version is lagging — it will execute a stale phase definition until it moves on.

GET metrics-*/_ilm/explain?only_managed=true

The only_managed=true filter narrows the response to indices ILM actually controls, which keeps the lag scan fast on large clusters. Cross-reference each returned phase_execution.version against the version from step 1.

3. List unmanaged indices #

Any metrics-* index whose settings lack index.lifecycle.name is unmanaged. Pull just the lifecycle settings with filter_path to keep the payload small:

GET metrics-*/_settings?filter_path=**.settings.index.lifecycle.name,**.settings.index.provided_name

Indices present in _cat/indices but absent from this filtered response have no lifecycle name and are the unmanaged set. This is the signal explored in depth in auditing indices with no ILM policy attached.

4. The Python v8 drift-report script #

The script below folds all three signals into one per-index report. It is read-only: it never writes to the deployment.

import json
import logging
from elasticsearch import Elasticsearch, ApiError, NotFoundError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)


def normalize_phases(phases: dict) -> dict:
    """Canonicalize a phases dict so structural comparison ignores key ordering
    and server-echoed defaults. json.dumps with sorted keys gives a stable form."""
    return json.loads(json.dumps(phases, sort_keys=True))


def policy_body_drift(es: Elasticsearch, policy_name: str, canonical_path: str) -> dict:
    """Signal 1: does the live policy body match the canonical JSON in git?"""
    with open(canonical_path) as fh:
        canonical = json.load(fh)  # {"policy": {"phases": {...}}}
    canonical_phases = normalize_phases(canonical["policy"]["phases"])
    try:
        live = es.ilm.get_lifecycle(name=policy_name)
    except NotFoundError:
        return {"policy": policy_name, "state": "MISSING_ON_CLUSTER", "live_version": None}
    live_entry = live[policy_name]
    live_phases = normalize_phases(live_entry["policy"]["phases"])
    return {
        "policy": policy_name,
        "state": "IN_SYNC" if live_phases == canonical_phases else "BODY_DRIFT",
        "live_version": live_entry["version"],
        "in_use_by_indices": live_entry.get("in_use_by", {}).get("indices", []),
    }


def index_execution_drift(es: Elasticsearch, index_pattern: str, current_version: int) -> list[dict]:
    """Signal 2: which managed indices run an older phase_execution.version?"""
    resp = es.ilm.explain_lifecycle(index=index_pattern, only_managed=True)
    lagging = []
    for name, meta in resp.get("indices", {}).items():
        exec_version = meta.get("phase_execution", {}).get("version")
        if exec_version is not None and exec_version < current_version:
            lagging.append({
                "index": name,
                "phase": meta.get("phase"),
                "running_version": exec_version,
                "current_version": current_version,
            })
    return lagging


def unmanaged_indices(es: Elasticsearch, index_pattern: str) -> list[str]:
    """Signal 3: which indices carry no index.lifecycle.name at all?"""
    settings = es.indices.get_settings(
        index=index_pattern,
        filter_path="**.settings.index.lifecycle.name,**.settings.index.provided_name",
        expand_wildcards="open",
    )
    unmanaged = []
    # Re-list the pattern so indices missing entirely from the filtered response surface.
    for name in es.indices.get(index=index_pattern, filter_path="*").keys():
        if name.startswith("."):
            continue  # skip system / hidden indices
        lifecycle = settings.get(name, {}).get("settings", {}).get("index", {}).get("lifecycle", {})
        if not lifecycle.get("name"):
            unmanaged.append(name)
    return unmanaged


def build_drift_report(es: Elasticsearch, policy_name: str, index_pattern: str, canonical_path: str) -> dict:
    try:
        body = policy_body_drift(es, policy_name, canonical_path)
        current_version = body.get("live_version") or 0
        report = {
            "policy_body": body,
            "execution_lag": index_execution_drift(es, index_pattern, current_version),
            "unmanaged": unmanaged_indices(es, index_pattern),
        }
        report["clean"] = (
            body["state"] == "IN_SYNC"
            and not report["execution_lag"]
            and not report["unmanaged"]
        )
        return report
    except ApiError as exc:
        logger.error("Drift scan failed (%s): %s", exc.status_code, exc.info)
        raise


if __name__ == "__main__":
    es = Elasticsearch(
        "https://es-cluster-01:9200",
        api_key="YOUR_BASE64_ENCODED_API_KEY",
        verify_certs=True,
    )
    result = build_drift_report(es, "metrics-ilm", "metrics-*", "policies/metrics-ilm.json")
    logger.info("Drift report: %s", json.dumps(result, indent=2))

The script separates the three signals into pure functions so each can be alerted on independently, and build_drift_report sets a single clean flag only when all three pass. Because it uses get_lifecycle, explain_lifecycle, and get_settings — all read APIs — it is safe to schedule against production with a read_ilm/monitor account.

Verification #

A clean cluster produces a report whose three arms are all empty:

{
  "policy_body": { "policy": "metrics-ilm", "state": "IN_SYNC", "live_version": 7, "in_use_by_indices": ["metrics-000012", "metrics-000013"] },
  "execution_lag": [],
  "unmanaged": [],
  "clean": true
}

state: IN_SYNC confirms the live phases matches metrics-ilm.json; the empty execution_lag array confirms every managed index runs version 7; the empty unmanaged array confirms nothing under metrics-* slipped out of management. After reconciliation, re-run the scan and confirm clean flips to true. When you re-apply a corrected body, GET _ilm/policy/metrics-ilm should show version incremented by exactly one — proof the write landed — and a follow-up explain should show lagging indices adopting the new version as they transition phases (or immediately, if you forced them onto the new step). Confirm a specific index has caught up:

GET metrics-000012/_ilm/explain?only_managed=true

The phase_execution.version in the response should now equal the policy version.

Threshold Tuning & Performance Guidance #

Drift detection is a scheduled read job, not a real-time watcher, and it must be scoped so it stays cheap and its output stays actionable.

1. Scope by pattern, never scan _all. Run one report per policy-and-pattern pair (metrics-*, logs-*, …). Running GET */_ilm/explain cluster-wide on a deployment with tens of thousands of indices returns a large payload and competes with production reads; per-pattern scans keep each run bounded and let you own drift per team.
2. Run on a cadence aligned to change, not to the poll interval. ILM re-evaluates every indices.lifecycle.poll_interval (default 10m), but policy definitions change at the speed of pull requests. A drift scan every 15–30 minutes catches divergence well within any retention SLA without hammering the deployment. Trigger an immediate scan from your deploy pipeline right after a put_lifecycle to confirm adoption.
3. Ignore in-flight execution lag under one revision. An index that entered its phase moments before you edited the policy will legitimately show phase_execution.version one behind until it next transitions. Treat a single-revision lag on a recently-transitioned index as expected; alert only when the gap is two or more revisions or the lag persists past a phase’s min_age window.
4. Cache the canonical body per run. Read metrics-ilm.json once at the top of the job rather than per index. The comparison cost is in normalization, not I/O, but re-reading the file for every index is wasted work on large patterns.

On payload size: always use filter_path on the settings scan and only_managed=true on explain. The unmanaged scan is the one call that must see all matching indices (managed and not), so it is the most expensive — restrict its expand_wildcards to open and exclude dot-prefixed system indices to keep it lean.

Troubleshooting #

The report flags BODY_DRIFT on every run even after re-applying #

Symptom: state is always BODY_DRIFT immediately after a fresh put_lifecycle of the canonical file. Root cause: the live body carries server-normalized defaults the canonical JSON omits — an implicit set_priority, reordered action keys, or a duration written 1d versus 86400000ms. Resolution: normalize both sides before comparing (the normalize_phases helper sorts keys), and either add the omitted defaults to metrics-ilm.json so it round-trips exactly, or compare only the fields you author. A stable canonical file that matches what the deployment echoes back eliminates the false positive permanently.

Persistent execution lag that never clears #

Symptom: an index reports phase_execution.version below the current policy version for far longer than expected. Root cause: phase execution version only advances when the index transitions to a new phase — an index parked in warm with a long min_age before delete will run its cached version for days. It is not broken; it is waiting. Resolution: this is transient lag, not drift, if the index will still execute the correct action. If the edit changed the phase the index is currently in and you need it applied now, force it onto the new definition with POST _ilm/move/metrics-000012 — covered in reconciling ILM policy drift with Python. Otherwise let it clear naturally at the next transition.

in_use_by is empty on a policy you believe is in production #

Symptom: GET _ilm/policy/metrics-ilm returns an in_use_by with empty indices and data_streams. Root cause: the template that should attach the policy was never applied, was overwritten, or names a different policy — so nothing references metrics-ilm even though the policy exists. Resolution: inspect the composable template with GET _index_template/metrics-template and confirm its index.lifecycle.name equals metrics-ilm. Fix the template, then bootstrap or re-attach so new and existing indices pick it up. An empty in_use_by on a production policy is a genuine, high-severity drift finding.

System and hidden indices pollute the unmanaged list #

Symptom: the unmanaged report contains .ds-, .security, or other dot-prefixed indices. Resolution: filter out names starting with . (the script does this) and remember that a data stream’s backing indices are managed through the stream, not by a direct index.lifecycle.name on each backing index — treat them via the stream, not as standalone unmanaged indices.

FAQ #

Related #

• Reconciling ILM policy drift with Python — the fix side: idempotently re-apply the canonical body when it differs from live.
• Auditing indices with no ILM policy attached — finding and attaching a policy to the unmanaged indices this scan surfaces.
• Versioning ILM policies across environments — why cross-cluster version integers diverge and how to keep the canonical body identical everywhere.
• Bootstrapping index templates and data streams — ensuring new indices inherit the policy so they never enter the unmanaged set.
• Monitoring ILM execution and error states — the runtime companion to drift detection, watching step-level errors and stalls.

← Back to ILM Policy Design & Lifecycle Synchronization