Versioning ILM Policies Across Environments #

Elasticsearch gives you no way to promote a lifecycle policy from one cluster to another. There is no _ilm/export, no environment-aware policy store, and no server that remembers what app-events-ilm looked like yesterday. Each deployment holds exactly one live definition per policy id, and the only history it keeps is a monotonically increasing version integer that tells you that the policy changed, never what changed or why. When the same policy must run identically on a staging cluster and a production cluster, that gap becomes an operational hazard: the two definitions drift, a warm-phase edit that was smoke-tested in staging never reaches production, or a delete phase shortened during an incident lingers on one cluster after being reverted on the other. This page shows how to close that gap by treating the policy JSON as source-controlled configuration, applying it through a pipeline, and reading the three fields — version, modified_date, and in_use_by — that let you prove two clusters agree. It is one discipline inside the broader practice of ILM policy design and lifecycle synchronization.

The worked example throughout is a policy named app-events-ilm, attached to the app-events-* index pattern, running on two clusters we will call staging and production. The mechanics generalize to any number of environments and any policy id.

Prerequisites #

• Elasticsearch 8.x on every target environment, each an independent cluster with its own _ilm state — the version counter is per-cluster and never synchronized.
• The canonical policy JSON stored in source control, with a change flowing through a reviewed pull request rather than a live console edit.
• A CI/CD service account holding manage_ilm at cluster scope on each environment, plus manage on the app-events-* pattern for the bootstrap that attaches the policy, scoped per securing ILM policies with RBAC.
• elasticsearch-py v8.0+ (pip install "elasticsearch>=8,<9") — the code here uses the v8 keyword-argument surface (ilm.get_lifecycle, ilm.put_lifecycle), not the legacy body= pattern.
• Per-environment credentials (separate API keys or basic-auth principals), so a pipeline that promotes to production cannot accidentally target staging and vice versa.

Architecture: One Artifact, Many Clusters #

Because Elasticsearch has no native promotion, the artifact — the exact bytes of the policy JSON — must be the unit that moves between environments, and the pipeline must apply those same bytes to each deployment in turn. Git is the source of truth and the only history that exists; each deployment is a downstream replica whose live definition should converge to whatever the repository declares. The diagram below traces one change from commit to production.

Three properties of this topology drive everything else on the page. First, the artifact never changes between environments — only its target host does; if the bytes staging validated differ by even one whitespace character from the bytes production receives, you no longer have a promotion, you have two edits. Second, every apply is a full replacement: put_lifecycle does not patch or merge, it overwrites the named policy wholesale, which is exactly what makes re-applying an unchanged artifact safe. Third, the version integer is local state — staging and production increment their own counters independently, so a matching version number across clusters means nothing and a mismatched one is expected. The field that should match is the _meta you embed yourself.

Configuration Reference #

The three fields that describe a live policy #

GET _ilm/policy/app-events-ilm returns the policy body plus three envelope fields that a versioning workflow reads constantly:

GET _ilm/policy/app-events-ilm
{
  "app-events-ilm": {
    "version": 7,
    "modified_date": "2026-07-17T09:14:22.481Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" },
            "set_priority": { "priority": 100 }
          }
        },
        "warm": {
          "min_age": "3d",
          "actions": {
            "forcemerge": { "max_num_segments": 1 },
            "set_priority": { "priority": 50 }
          }
        },
        "delete": {
          "min_age": "30d",
          "actions": { "delete": {} }
        }
      },
      "_meta": {
        "git_sha": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0",
        "version": "2.4.0"
      }
    },
    "in_use_by": {
      "indices": ["app-events-000012", "app-events-000013"],
      "data_streams": [],
      "composable_templates": ["app-events-template"]
    }
  }
}

• version is an integer Elasticsearch increments on every successful put_lifecycle, whether or not the body actually changed. It is a change counter scoped to this deployment, not a semantic version and not a hash of the content. Two clusters with byte-identical policies routinely show different version values because they have absorbed a different number of applies over their lifetimes.
• modified_date is the UTC timestamp of the last put_lifecycle. Like version, it moves on every write, so a re-apply of unchanged bytes bumps it. It is useful for correlating a policy change with a deployment log or an audit event, not for comparing two clusters.
• in_use_by lists the indices, data_streams, and composable_templates that currently reference the policy. An empty object under all three keys means the policy exists but is attached to nothing — a strong signal that a template binding is missing or that a reindex recreated an index without re-attaching index.lifecycle.name.

The policy body, with an embedded _meta #

Elasticsearch supports a free-form _meta object inside the policy body and stores it verbatim, round-tripping it on every get_lifecycle. Because it survives writes untouched, _meta is the one place you can stamp the information the deployment otherwise throws away — which commit produced this definition and what human-readable version it corresponds to:

PUT _ilm/policy/app-events-ilm
{
  "policy": {
    "_meta": {
      "git_sha": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0",
      "version": "2.4.0",
      "owner": "search-platform",
      "source": "repo/ilm/app-events-ilm.json"
    },
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

The _meta.version here is a semantic version you control, bumped by the same commit that changes the phases; git_sha ties the live definition back to an exact revision in source control. When the pipeline promotes the artifact, this block travels inside the bytes, so a correctly promoted policy shows the same _meta.git_sha on staging and production even though their version integers diverge. That is the invariant a drift check compares — and the reason detecting ILM policy drift keys on the policy body and _meta, never on version. The rollover action shown here is the entry point covered in depth under configuring index rollover conditions; attaching the policy through a template so in_use_by is ever populated is the subject of bootstrapping index templates and data streams.

Step-by-Step Implementation #

1. Store the policy JSON in git with a _meta version #

The canonical file lives at a stable path — repo/ilm/app-events-ilm.json — and contains exactly the policy object you will PUT, _meta included. A change is a commit: bump _meta.version, update the phases, and let CI stamp _meta.git_sha at apply time (or commit the placeholder and let the pipeline substitute the resolved sha). Because put_lifecycle replaces the whole definition, the file must be complete — there is no partial update, so the repository always holds the entire intended state, not a diff against a live cluster.

2. Apply to staging and smoke-test #

There is no server-side _ilm/validate endpoint. The only way Elasticsearch tells you whether a policy is well-formed is to apply it, so staging is your validator: a malformed phase, an unknown action, or an illegal min_age ordering is rejected by put_lifecycle on staging before it can ever reach production. After the apply, confirm the definition took and that ILM will actually run it against a representative index.

import json
import logging
from elasticsearch import Elasticsearch
from elasticsearch import ApiError, BadRequestError, NotFoundError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)

POLICY_ID = "app-events-ilm"


def load_artifact(path: str, git_sha: str) -> dict:
    """Read the canonical policy JSON and stamp the resolved commit sha into _meta."""
    with open(path, encoding="utf-8") as fh:
        artifact = json.load(fh)
    artifact["policy"].setdefault("_meta", {})["git_sha"] = git_sha
    return artifact


def apply_to_cluster(client: Elasticsearch, artifact: dict) -> int:
    """PUT the policy and return the cluster-local version it now reports."""
    try:
        # put_lifecycle REPLACES the definition in place and bumps version + modified_date.
        client.ilm.put_lifecycle(name=POLICY_ID, policy=artifact["policy"])
    except BadRequestError as exc:
        # A malformed phase/action is rejected here — staging is the validator.
        logger.error("Policy rejected by %s: %s", client, exc.info)
        raise
    live = client.ilm.get_lifecycle(name=POLICY_ID)[POLICY_ID]
    logger.info("Applied %s: version=%s meta=%s",
                POLICY_ID, live["version"], live["policy"].get("_meta"))
    return live["version"]


def smoke_test(client: Elasticsearch, index_pattern: str = "app-events-*") -> None:
    """Confirm ILM can explain a managed index without an ERROR step."""
    try:
        explain = client.ilm.explain_lifecycle(index=index_pattern)
    except NotFoundError:
        logger.warning("No %s indices on this cluster yet — schema apply still validated.",
                       index_pattern)
        return
    for idx, meta in explain.get("indices", {}).items():
        if meta.get("step") == "ERROR":
            raise RuntimeError(f"{idx} is in ERROR after apply: {meta.get('step_info')}")
    logger.info("Smoke test clean for %s", index_pattern)


staging = Elasticsearch("https://staging-es:9200", api_key="STAGING_KEY", verify_certs=True)
artifact = load_artifact("repo/ilm/app-events-ilm.json", git_sha="a1b2c3d4e5f6")
apply_to_cluster(staging, artifact)
smoke_test(staging)

3. Promote the identical bytes to production #

Promotion is the same apply_to_cluster call pointed at a different client. Critically, it re-uses the same artifact object that staging validated — the bytes are not re-read, re-serialized, or re-templated per environment, which is what guarantees production receives what staging approved. The deterministic promotion helper and its guardrails are the focus of promoting ILM policy changes across environments; the authoring surface it builds on is building custom ILM policies via the API.

production = Elasticsearch("https://prod-es:9200", api_key="PROD_KEY", verify_certs=True)
apply_to_cluster(production, artifact)  # same artifact object staging validated

4. Verify version and in_use_by on both clusters #

After promotion, prove the two clusters agree on the definition and disagree only on the local version integer.

def summarize(client: Elasticsearch) -> dict:
    live = client.ilm.get_lifecycle(name=POLICY_ID)[POLICY_ID]
    return {
        "version": live["version"],                       # per-cluster, will differ
        "meta": live["policy"].get("_meta", {}),          # should match across clusters
        "in_use_by": live["in_use_by"],
    }

s, p = summarize(staging), summarize(production)
assert s["meta"].get("git_sha") == p["meta"].get("git_sha"), "policy bodies drifted!"
logger.info("staging v%s / prod v%s / same git_sha=%s",
            s["version"], p["version"], s["meta"].get("git_sha"))

Verification #

The authoritative check is GET _ilm/policy/app-events-ilm on each deployment, read for its three envelope fields:

GET _ilm/policy/app-events-ilm?filter_path=**.version,**._meta,**.in_use_by

{
  "app-events-ilm": {
    "version": 12,
    "policy": { "_meta": { "git_sha": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0", "version": "2.4.0" } },
    "in_use_by": {
      "indices": ["app-events-000031", "app-events-000032"],
      "data_streams": [],
      "composable_templates": ["app-events-template"]
    }
  }
}

Read it as three assertions. The _meta.git_sha must equal the sha the pipeline promoted and must match the other cluster — this is the real equality check. The version integer confirms a write happened but is expected to differ from staging. The in_use_by block must be non-empty on any cluster that actually runs app-events-* traffic; an empty in_use_by on production means the policy is present but attached to nothing, so no index is being managed by it. To confirm a specific in-flight index has picked up the new definition rather than the version it started its current phase under, read the per-index execution snapshot:

GET app-events-000031/_ilm/explain?filter_path=**.phase_execution.version,**.phase,**.step

The phase_execution.version there is the policy version the index cached when it entered its current phase — it can legitimately lag the policy’s live version until the index reaches its next step evaluation, which is exactly the caching behavior described next.

Threshold Tuning, Performance & Governance #

Idempotent apply, not blind re-apply #

put_lifecycle bumps version and modified_date on every call, even when the body is unchanged. On a deployment with alerting on modified_date, a pipeline that re-PUTs on every run generates constant false “policy changed” noise and inflates the version counter without cause. Make the apply conditional: fetch the live policy body, compare it (ignoring _meta.git_sha, which legitimately changes per commit) against the artifact, and issue the write only on a real difference. A redeploy of unchanged bytes then becomes a genuine no-op that leaves version and modified_date untouched.

Phase-execution caching means in-flight indices lag #

When you apply a changed definition, managed indices do not re-read it immediately. Each index caches the policy definition it entered its current phase with — the phase_execution snapshot — and finishes the current action under that cached version. It picks up the new definition only at its next step evaluation, on the next indices.lifecycle.poll_interval. So a warm-phase forcemerge already in progress completes under the old rules even after you PUT new ones; the change applies cleanly to the next phase transition. This is safe by design — it prevents a mid-action edit from corrupting an in-flight migration — but it means “the policy is updated” and “every index is running the update” are two different moments, and phase_execution.version in _ilm/explain is how you tell them apart.

Drift is the failure this whole workflow prevents #

Two clusters silently running different definitions of app-events-ilm is the failure mode versioning exists to eliminate. It happens the instant someone edits a policy through Kibana’s console on one cluster instead of committing a change and letting the pipeline apply it everywhere. Enforce apply-only-from-pipeline through RBAC — application principals get zero manage_ilm — and run a scheduled comparison of each deployment’s live body against the repository. The detection side is covered in detecting ILM policy drift; the point here is that the _meta.git_sha you embedded is the anchor that makes drift a one-line equality check rather than a deep JSON diff.

Troubleshooting #

Version drift between environments #

Symptom: staging and production report different version integers and someone flags it as a problem. Resolution: This is usually a non-issue — version is per-cluster and diverges naturally. Compare _meta.git_sha instead: if the shas match, the definitions are identical and the version gap is expected. Only if the shas differ is there real drift; re-run the pipeline to promote the canonical artifact and bring both clusters to the same git_sha.

_meta not updated after a change #

Symptom: the phases changed but _meta.version / _meta.git_sha on the live policy still show the previous values. Resolution: Either the artifact was applied without re-stamping _meta (the pipeline read a stale sha), or a hand edit bypassed the pipeline entirely. Confirm the applied git_sha matches HEAD of the merged change, and make _meta.git_sha substitution a mandatory step so an apply can never ship phases without a matching stamp.

in_use_by is empty #

Symptom: GET _ilm/policy/app-events-ilm succeeds but in_use_by lists no indices, data streams, or templates. Resolution: The policy exists but nothing references it, so no data is being managed. Check that the app-events-template sets index.lifecycle.name: app-events-ilm, and that backing indices were created after the template binding. A reindex that recreated app-events-* without re-attaching the policy is a common cause — re-attach via the template and confirm in_use_by repopulates.

Production apply rejected after staging accepted it #

Symptom: the identical artifact PUTs cleanly on staging but put_lifecycle returns a 400 on production. Resolution: The clusters differ in a way the policy depends on — a searchable_snapshot action referencing a repository that exists only on staging, or a feature gated by a license tier present on staging but not production. Align the environments’ repositories and licenses; a policy is only truly validated by staging when staging matches production in the capabilities the policy exercises.

FAQ #

Related #

• Promoting ILM policy changes across environments — the deterministic staging-to-production promotion helper.
• Rolling back a broken ILM policy version — reverting to a known-good definition from git.
• Bootstrapping index templates and data streams — attaching a policy so in_use_by is ever populated.
• Detecting ILM policy drift — scheduled comparison of live bodies against the repository.
• Securing ILM policies with RBAC — enforcing apply-only-from-pipeline so no cluster is edited by hand.

← Back to ILM Policy Design & Lifecycle Synchronization