ILM & Reindexing Automation Runbooks

Understanding Hot-Warm-Cold Architecture

Understanding Hot-Warm-Cold Architecture requires shifting from monolithic index management to tiered, phase-driven data routing. In production Elasticsearch deployments, this model decouples ingestion velocity, query latency, and storage economics by mapping index lifecycle phases to dedicated node tiers. Hot nodes handle high-throughput indexing and low-latency queries, typically backed by NVMe storage and high CPU core counts. Warm nodes transition to read-heavy workloads where indexing ceases, enabling aggressive segment merging and reduced memory overhead. Cold nodes serve archival or compliance-bound data, leveraging high-capacity HDDs or remote object storage via searchable snapshots.

The operational foundation relies on Index Lifecycle Management (ILM) to automate phase transitions without manual intervention. As established in Elasticsearch ILM Architecture & Fundamentals, ILM evaluates index metadata against policy-defined thresholds, triggering actions like rollover, shrink, force_merge, and tier allocation. For search engineers and log analytics teams, this means query routing, shard distribution, and retention SLAs are governed by declarative JSON rather than cron-driven scripts. The architecture inherently supports cost optimization while maintaining searchable compliance footprints, provided node attributes and allocation filters are strictly enforced at the cluster level.

flowchart LR
  ING["Ingestion"] --> HOT["Hot tier (NVMe, write + query)"]
  HOT -->|"min_age 7d"| WARM["Warm tier (shrink, forcemerge)"]
  WARM -->|"min_age 30d"| COLD["Cold tier (read-only, dense)"]
  COLD -->|"min_age 90d"| DEL["Delete"]

Node Attribute Enforcement & Allocation Filters

Configuration begins with explicit node tier labeling and index template alignment. Each data node must declare its role via node.attr.data in elasticsearch.yml. ILM policies then reference these attributes using index.routing.allocation.require.data to guarantee phase-specific placement.

Node Configuration (elasticsearch.yml):

# Hot Tier
node.attr.data: hot
node.roles: [data_hot, master, ingest]

# Warm Tier
node.attr.data: warm
node.roles: [data_warm]

# Cold Tier
node.attr.data: cold
node.roles: [data_cold]

Allocation filters must be applied at the index template level to prevent cross-tier shard leakage. Misconfigured filters result in UNASSIGNED shards or hot-tier saturation. Always validate allocation rules using GET _cluster/allocation/explain before deploying templates to production.

ILM Policy Definition & Threshold Tuning

A production-grade policy must define precise rollover triggers to prevent hot-tier saturation and ensure predictable shard sizing. Rollover thresholds should align with JVM heap constraints and segment merge capabilities. Refer to Configuring Index Rollover Conditions for detailed threshold calibration strategies.

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "7d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "force_merge": { "max_num_segments": 1 },
          "allocate": {
            "require": { "data": "warm" },
            "number_of_replicas": 1
          },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "require": { "data": "cold" },
            "number_of_replicas": 0
          },
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}

Note that max_primary_shard_size should be tuned carefully. Oversized primary shards degrade recovery times and force_merge performance. See How to Configure Rollover Based on Max Primary Shard Size for sizing matrices aligned with disk IOPS and heap limits.

Python v8 Client Orchestration & Reindexing Automation

Manual ILM configuration is error-prone at scale. The following Python v8+ script automates policy deployment, index template application, and idempotent reindexing for legacy indices that bypass ILM.

import logging
from elasticsearch import Elasticsearch, exceptions

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

es = Elasticsearch(
    ["https://es-node-01:9200", "https://es-node-02:9200"],
    ca_certs="/path/to/ca.crt",
    basic_auth=("elastic", "CHANGE_ME_STRONG_PASSWORD"),
    request_timeout=30
)

ILM_POLICY_NAME = "logs-hot-warm-cold"
INDEX_TEMPLATE_NAME = "logs-tiered-template"
ILM_POLICY = {
    "phases": {
        "hot": {"min_age": "0ms", "actions": {"rollover": {"max_primary_shard_size": "50gb", "max_age": "7d"}, "set_priority": {"priority": 100}}},
        "warm": {"min_age": "7d", "actions": {"shrink": {"number_of_shards": 1}, "force_merge": {"max_num_segments": 1}, "allocate": {"require": {"data": "warm"}, "number_of_replicas": 1}, "set_priority": {"priority": 50}}},
        "cold": {"min_age": "30d", "actions": {"allocate": {"require": {"data": "cold"}, "number_of_replicas": 0}, "set_priority": {"priority": 0}}},
        "delete": {"min_age": "90d", "actions": {"delete": {}}}
    }
}

def deploy_ilm_and_template():
    try:
        es.ilm.put_lifecycle(name=ILM_POLICY_NAME, policy=ILM_POLICY)
        logger.info(f"ILM policy '{ILM_POLICY_NAME}' deployed successfully.")
    except exceptions.BadRequestError as e:
        logger.warning(f"ILM policy already exists or invalid: {e}")

    template_body = {
        "index_patterns": ["logs-app-*"],
        "template": {
            "settings": {
                "index.lifecycle.name": ILM_POLICY_NAME,
                "index.lifecycle.rollover_alias": "logs-app-write",
                "index.routing.allocation.require.data": "hot",
                "number_of_shards": 3,
                "number_of_replicas": 1
            },
            "mappings": {"dynamic": "strict", "properties": {"@timestamp": {"type": "date"}}}
        }
    }
    es.indices.put_index_template(name=INDEX_TEMPLATE_NAME, body=template_body)
    logger.info(f"Index template '{INDEX_TEMPLATE_NAME}' applied.")

def reindex_legacy_indices(source_pattern: str, target_alias: str):
    """Idempotent reindex for indices missing ILM metadata."""
    try:
        response = es.reindex(
            body={
                "source": {"index": source_pattern},
                "dest": {"index": target_alias, "op_type": "create"},
                "conflicts": "proceed"
            },
            wait_for_completion=False,
            timeout="1h"
        )
        logger.info(f"Reindex task submitted: {response['task']}")
    except exceptions.ApiError as e:
        logger.error(f"Reindex failed: {e}")

if __name__ == "__main__":
    deploy_ilm_and_template()
    # Bootstrap initial write alias if missing
    if not es.indices.exists_alias(name="logs-app-write"):
        es.indices.create(index="logs-app-000001", body={"aliases": {"logs-app-write": {"is_write_index": True}}})
    reindex_legacy_indices("logs-app-legacy-*", "logs-app-write")

For environments with strict access controls, ensure service accounts possess manage_ilm and manage_index_templates privileges. Review Securing ILM Policies with RBAC for role-scoped permission matrices that prevent unauthorized policy overrides.

Production Debugging & Failure Recovery

ILM execution is asynchronous and subject to cluster routing constraints. When phases stall, immediate triage requires inspecting step metadata rather than guessing at node health.

1. Diagnose Stuck Phases 

GET logs-app-000003/_ilm/explain

Look for the step_info object (present when step is ERROR). Common failures:

  • allocation_failed: Warm nodes lack disk space or cluster.routing.allocation.enable is set to none.
  • shrink_failed: Source index has active replicas or unassigned shards. Run POST /<index>/_shrink/<new-index> manually after setting index.blocks.write: true.
  • force_merge_failed: Insufficient heap on warm nodes. Reduce max_num_segments to 2 or increase indices.memory.index_buffer_size.

2. Force ILM Evaluation

ILM polls every 10 minutes by default. After resolving the root cause, re-run the failed step for the affected index:

POST /logs-app-000003/_ilm/retry

3. Bypass Allocation Deadlocks

If warm/cold nodes reject shards due to disk watermark thresholds (cluster.routing.allocation.disk.watermark), temporarily adjust routing:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%",
    "cluster.routing.allocation.disk.watermark.high": "90%"
  }
}

Revert immediately after shard relocation completes. Persistent watermark overrides risk OOM conditions and data loss.

4. Validate Rollover Triggers

If rollover does not fire despite exceeding max_primary_shard_size, verify the write alias points to the correct index and that index.lifecycle.rollover_alias matches exactly. Use GET /_alias/logs-app-write to confirm.

Understanding Hot-Warm-Cold Architecture succeeds when policy definitions, node attributes, and client orchestration operate as a single control plane. By enforcing strict allocation filters, automating reindexing fallbacks, and monitoring step_info payloads, teams can maintain predictable shard sizing, enforce retention SLAs, and eliminate manual lifecycle interventions.

Subtopic How to Configure Rollover Based on Max Primary Shard Size