Promoting ILM Policy Changes Across Environments #

Promote a reviewed change to the app-events-ilm policy from staging to production so production ends up running the exact bytes staging validated — no re-editing, no drift, no surprises.

Elasticsearch has no cross-cluster policy promotion, so “promote” here means one thing precisely: apply the same artifact to a second cluster. This is the operational sequel to versioning ILM policies across environments, which establishes why the policy JSON is source-controlled and why _meta.git_sha — not the per-cluster version integer — is the field that proves two clusters agree. It fits inside the wider discipline of ILM policy design and lifecycle synchronization. The danger a deterministic promotion guards against is subtle: any per-environment re-templating, re-serialization, or hand tweak between staging and production turns a promotion back into two independent edits, and two edits drift.

Prerequisites #

• Elasticsearch 8.x staging and production clusters, each with its own independent _ilm state and version counter.
• The canonical policy JSON in source control, already reviewed and merged, carrying a _meta block with git_sha and a semantic version.
• elasticsearch-py v8.0+, and separate per-environment credentials so a promotion targeting production can never be mis-pointed at staging.
• manage_ilm at cluster scope on both service accounts, per securing ILM policies with RBAC.

Implementation #

The promotion reads the canonical JSON once, stamps the resolved commit sha into _meta, applies it to staging, runs a validation against a test index, then applies the identical in-memory object to production. Reading the artifact a single time is the whole trick — it removes any opportunity for the two clusters to receive different bytes. put_lifecycle replaces the policy in place and bumps version and modified_date on each deployment independently, so the helper never compares version numbers across environments; it compares _meta.git_sha.

import json
import logging
from elasticsearch import Elasticsearch
from elasticsearch import ApiError, BadRequestError, NotFoundError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)

POLICY_ID = "app-events-ilm"
TEST_INDEX = "app-events-000001"  # a representative managed index used only to validate


def load_artifact(path: str, git_sha: str) -> dict:
    """Read the canonical policy JSON exactly once and stamp the resolved commit sha."""
    with open(path, encoding="utf-8") as fh:
        artifact = json.load(fh)
    artifact["policy"].setdefault("_meta", {})["git_sha"] = git_sha
    return artifact


def apply_policy(client: Elasticsearch, artifact: dict) -> dict:
    """PUT the policy (a full in-place replacement) and return the live envelope."""
    try:
        client.ilm.put_lifecycle(name=POLICY_ID, policy=artifact["policy"])
    except BadRequestError as exc:
        # Malformed phase/action or illegal min_age ordering is rejected here.
        logger.error("Policy rejected: %s", exc.info)
        raise
    return client.ilm.get_lifecycle(name=POLICY_ID)[POLICY_ID]


def validate_on(client: Elasticsearch) -> None:
    """Confirm ILM can explain a managed index and it is not wedged in ERROR."""
    try:
        explain = client.ilm.explain_lifecycle(index=TEST_INDEX)
    except NotFoundError:
        logger.warning("%s absent on staging; schema apply still validated the body.", TEST_INDEX)
        return
    state = explain["indices"][TEST_INDEX]
    if state.get("step") == "ERROR":
        raise RuntimeError(f"{TEST_INDEX} wedged after apply: {state.get('step_info')}")
    logger.info("Validation clean: %s in phase=%s step=%s",
                TEST_INDEX, state.get("phase"), state.get("step"))


def promote(path: str, git_sha: str, staging: Elasticsearch, production: Elasticsearch) -> None:
    artifact = load_artifact(path, git_sha)          # read ONCE
    logger.info("Promoting git_sha=%s", git_sha)

    staged = apply_policy(staging, artifact)          # 1. apply to staging
    validate_on(staging)                              # 2. validate against a test index
    logger.info("Staging now version=%s meta=%s", staged["version"], staged["policy"].get("_meta"))

    promoted = apply_policy(production, artifact)      # 3. apply the SAME object to production
    logger.info("Production now version=%s meta=%s",
                promoted["version"], promoted["policy"].get("_meta"))

    # 4. Compare _meta, never version — version is per-cluster local state.
    staged_sha = staged["policy"].get("_meta", {}).get("git_sha")
    prod_sha = promoted["policy"].get("_meta", {}).get("git_sha")
    if staged_sha != prod_sha:
        raise RuntimeError(f"promotion drift: staging {staged_sha} != production {prod_sha}")
    logger.info("Promotion verified: identical git_sha=%s (staging v%s, prod v%s)",
                prod_sha, staged["version"], promoted["version"])


if __name__ == "__main__":
    staging = Elasticsearch("https://staging-es:9200", api_key="STAGING_KEY", verify_certs=True)
    production = Elasticsearch("https://prod-es:9200", api_key="PROD_KEY", verify_certs=True)
    try:
        promote("repo/ilm/app-events-ilm.json", "a1b2c3d4e5f6", staging, production)
    except ApiError as exc:
        logger.error("Promotion aborted on API error (%s): %s", exc.status_code, exc.info)
        raise

The apply is idempotent in effect: promoting the same git_sha twice leaves both clusters running the same definition. It is not, however, a no-op on the envelope — each put_lifecycle still bumps that cluster’s version and modified_date, so guard against needless re-applies upstream by only promoting when the merged change actually altered the policy body.

Verification #

Read the envelope on both clusters and assert the definitions match while accepting divergent version integers:

GET _ilm/policy/app-events-ilm?filter_path=**.version,**.modified_date,**._meta

{
  "app-events-ilm": {
    "version": 12,
    "modified_date": "2026-07-17T09:41:03.220Z",
    "policy": { "_meta": { "git_sha": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0", "version": "2.4.0" } }
  }
}

Run the same request against staging: it will report a different version and modified_date but the same _meta.git_sha. Equal shas mean the promotion landed identical bytes; a sha mismatch is real drift and the promotion must be re-run. The modified_date on each side is useful only for correlating the apply with a deployment log, never for cross-cluster comparison.

Gotchas and edge cases #

• There is no atomic multi-cluster apply. The helper writes staging, then production, as two separate calls; a failure between them leaves the clusters temporarily divergent. Make the promotion re-runnable so a retry simply re-applies the same git_sha and converges both clusters — never patch one side by hand to “catch up”.
• Version numbers differ per cluster — do not assert on them. version counts applies absorbed by each deployment independently, so production may sit at 12 while staging sits at 7 for byte-identical policies. Compare _meta.git_sha; asserting equal version values will fail spuriously and mask the check that matters.
• In-flight indices finish under the old definition. Promotion updates the policy, but a managed app-events-* index mid-phase keeps running its cached phase_execution version until its next step evaluation. Do not expect an in-progress forcemerge to change behavior mid-action; the new rules apply at the next phase transition. Confirm adoption per index via phase_execution.version in GET <index>/_ilm/explain.
• Secure per-environment credentials. Use distinct API keys for staging and production and never share a single key across both. A promotion mis-pointed at staging while it believes it is production silently skips the real target and can pass its own verification against the wrong cluster.

FAQ #

Related #

• Rolling back a broken ILM policy version — reverting production to a known-good definition from git.
• Versioning ILM policies across environments — why the artifact is source-controlled and how the version, modified_date and in_use_by fields behave.
• Securing ILM policies with RBAC — scoping the per-environment service accounts a promotion pipeline uses.

← Back to Versioning ILM Policies Across Environments · ILM Policy Design & Lifecycle Synchronization