Reconciling ILM Policy Drift with Python #

Build an automated job that both detects and repairs live-versus-canonical drift: it loads the source-controlled policy body, compares it structurally against what the deployment is running, and re-applies the canonical definition only when the two genuinely differ.

Detection tells you a policy body has diverged; reconciliation closes the gap. This page is the write-side counterpart to detecting ILM policy drift, and it belongs to the broader practice of treating lifecycle definitions as versioned, idempotent contracts described in ILM policy design and lifecycle synchronization. The whole point is idempotency: running the job against an already-synchronized cluster must be a no-op, and running it against a drifted one must converge the live body back to metrics-ilm.json — without ever bumping the version when nothing changed. A job that blindly re-applies on every run defeats the version integer as a change signal and floods your audit log with phantom edits.

Prerequisites #

• Elasticsearch 8.x with the ILM coordinator running, and the canonical body checked into git as policies/metrics-ilm.json.
• elasticsearch-py v8.0+ — the code uses client.ilm.get_lifecycle and client.ilm.put_lifecycle with keyword arguments.
• A service account with manage_ilm at cluster scope (reconciliation writes, unlike detection, which only needs read_ilm).
• Familiarity with the drift signals: this job handles only signal one (policy body drift). Per-index execution lag and unmanaged indices are handled separately.

Implementation #

The job is four moves: load canonical, GET live, deep-compare the normalized phases, and put_lifecycle only on a real difference. Normalization is the load-bearing step — Elasticsearch echoes defaults and reorders keys, so a raw dict comparison reports drift that is not there.

import json
import logging
from elasticsearch import Elasticsearch, ApiError, NotFoundError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)


def normalize_phases(phases: dict) -> dict:
    """Stable, order-independent form of a phases dict for structural comparison.
    Sorting keys neutralizes action-key ordering differences that are not real drift."""
    return json.loads(json.dumps(phases, sort_keys=True))


def diff_phases(canonical: dict, live: dict) -> list[str]:
    """Human-readable per-phase diff, for logging what actually changed."""
    changes = []
    for phase in sorted(set(canonical) | set(live)):
        c, l = canonical.get(phase), live.get(phase)
        if c is None:
            changes.append(f"phase '{phase}' present in canonical, absent live")
        elif l is None:
            changes.append(f"phase '{phase}' present live, absent in canonical")
        elif normalize_phases({phase: c}) != normalize_phases({phase: l}):
            changes.append(f"phase '{phase}' differs")
    return changes


def reconcile_policy(es: Elasticsearch, policy_name: str, canonical_path: str, dry_run: bool = False) -> dict:
    """Detect and, unless dry_run, repair live-versus-canonical body drift. Idempotent:
    a put_lifecycle is issued ONLY when the normalized phases differ."""
    with open(canonical_path) as fh:
        canonical = json.load(fh)                      # {"policy": {"phases": {...}}}
    canonical_phases = canonical["policy"]["phases"]

    try:
        live_entry = es.ilm.get_lifecycle(name=policy_name)[policy_name]
        live_phases = live_entry["policy"]["phases"]
        live_version = live_entry["version"]
    except NotFoundError:
        # Policy missing entirely: first-time apply is itself the reconciliation.
        live_phases, live_version = None, None

    if live_phases is not None and normalize_phases(canonical_phases) == normalize_phases(live_phases):
        logger.info("Policy '%s' in sync at version %s — no write issued.", policy_name, live_version)
        return {"policy": policy_name, "action": "none", "version": live_version}

    changes = diff_phases(canonical_phases, live_phases or {})
    logger.warning("Drift on '%s': %s", policy_name, "; ".join(changes) or "policy missing on cluster")

    if dry_run:
        return {"policy": policy_name, "action": "would_apply", "changes": changes}

    try:
        # Re-apply the canonical body. This is what bumps the server-side version.
        es.ilm.put_lifecycle(name=policy_name, policy={"phases": canonical_phases})
        new_version = es.ilm.get_lifecycle(name=policy_name)[policy_name]["version"]
        logger.info("Re-applied '%s': version %s -> %s", policy_name, live_version, new_version)
        return {"policy": policy_name, "action": "applied", "version": new_version, "changes": changes}
    except ApiError as exc:
        logger.error("put_lifecycle failed (%s): %s", exc.status_code, exc.info)
        raise


if __name__ == "__main__":
    es = Elasticsearch(
        "https://es-cluster-01:9200",
        api_key="YOUR_BASE64_ENCODED_API_KEY",
        verify_certs=True,
    )
    result = reconcile_policy(es, "metrics-ilm", "policies/metrics-ilm.json")
    logger.info("Reconcile result: %s", json.dumps(result, indent=2))

The guard clause is the whole safety story. get_lifecycle returns the live body; if its normalized phases equals the canonical phases, the function returns action: none and never touches the deployment. Only a genuine structural difference reaches the put_lifecycle call. A NotFoundError — the policy does not exist at all — is treated as maximal drift and falls through to a first-time apply. Catching ApiError (not TransportError, which is a sibling class and will not catch API failures) surfaces a rejected write with its status_code and info intact.

Verification #

Run the job twice. The first run against a drifted cluster should log the per-phase diff and report action: applied with an incremented version:

{ "policy": "metrics-ilm", "action": "applied", "version": 8, "changes": ["phase 'warm' differs"] }

Immediately re-running it must be a clean no-op — proof of idempotency:

{ "policy": "metrics-ilm", "action": "none", "version": 8 }

Confirm from the deployment that the version bumped exactly once for one real change:

GET _ilm/policy/metrics-ilm?filter_path=metrics-ilm.version,metrics-ilm.modified_date

{ "metrics-ilm": { "version": 8, "modified_date": "2026-07-17T11:04:22.318Z" } }

A version that climbs on every scheduled run — even when you changed nothing — means normalization is failing and the job is re-applying needlessly; treat a monotonically-climbing version with no corresponding pull requests as a bug in the comparison, not as real drift.

Gotchas and edge cases #

• Normalize both dicts before comparing. Elasticsearch stores canonical, server-side forms: it may inject an implicit set_priority, canonicalize durations (1d versus 86400000ms), and return action keys in a different order. Comparing raw dicts reports drift on a body that is actually identical. Sort keys and, ideally, make metrics-ilm.json round-trip exactly what the deployment echoes so the comparison is stable.
• Never re-apply when equal. Each put_lifecycle increments version and writes a modified_date, which is an audit event and invalidates the “version equals change count” invariant. The guard clause exists precisely so an unchanged body is never re-written — do not remove it for “simplicity.”
• Phase execution lag is not policy drift. After a real re-apply, existing indices still report their old phase_execution.version until they transition phases — that is expected, not a second drift to fix here. If you need a lagging index onto the new definition immediately, that is a per-index POST _ilm/move concern, described in detecting ILM policy drift, not something reconciliation of the body should trigger automatically.
• Scope by policy id. Reconcile one named policy per invocation against its own canonical file. A loop that re-applies every policy it finds risks clobbering a policy that exists on the deployment but has no canonical file (a legitimately deployment-local policy) — only reconcile the ids you own a source file for.

FAQ #

Related #

• Auditing indices with no ILM policy attached — the other repair job: attaching a policy to indices that carry none.
• Detecting ILM policy drift — the read-side scan that surfaces the body drift this job fixes.
• ILM policy design and lifecycle synchronization — the versioned, idempotent-contract discipline this job implements.

← Back to Detecting ILM Policy Drift · ILM Policy Design & Lifecycle Synchronization