ILM & Reindexing Automation Runbooks

Fallback Routing for Data Retention 

flowchart TD
  A["Allocate shard (_tier_preference: data_warm,data_hot)"] --> B{"data_warm node available?"}
  B -->|"yes"| W["Place on warm tier"]
  B -->|"no"| C{"data_hot node available?"}
  C -->|"yes"| H["Fall back to hot tier"]
  C -->|"no"| U["Unassigned: check watermarks and roles"]

Operational Context & Failure Modes

Fallback Routing for Data Retention is an operational resilience pattern that guarantees index continuity when primary Index Lifecycle Management (ILM) transitions fail due to node evictions, storage tier unavailability, or allocation filter mismatches. In production clusters, ILM policies execute sequentially, but real-world infrastructure constraints—such as disk watermark breaches, transient network partitions, or misaligned node attributes—can stall warm, cold, or frozen transitions. Without a deterministic fallback path, indices remain stranded in intermediate states, risking compliance violations, query degradation, and uncontrolled primary shard growth.

Implementing this pattern requires treating ILM not as a rigid pipeline but as a state machine with explicit failure branches. When a primary allocation target becomes unreachable, the cluster must automatically reroute shards to a secondary tier while preserving retention SLAs. This approach aligns with foundational lifecycle design principles documented in Elasticsearch ILM Architecture & Fundamentals, where policy execution is decoupled from physical topology through dynamic routing overrides and tier-aware allocation filters.

Allocation Hierarchy & Policy Design

Elasticsearch supports three allocation filters — require, include, and exclude — none of which provide an ordered fallback on their own (require is a hard constraint: if no matching node exists, the shard stays unallocated). The fallback mechanism is instead the special index.routing.allocation.include._tier_preference setting: an ordered, comma-separated list of data tiers (for example data_warm,data_hot) that the allocator evaluates left to right, falling back to the next tier when the preferred one is saturated or offline.

In a tiered deployment, this preference list must mirror your physical node topology. As outlined in Understanding Hot-Warm-Cold Architecture, each node should carry the appropriate data-tier role (data_hot, data_warm, data_cold) — and optionally custom attributes such as zone — that the allocator evaluates during shard placement. Fallback routing bridges the gap between ideal topology and runtime reality by allowing ILM to degrade gracefully rather than halt.

Index Template & Routing Overrides

Configuration centers on defining explicit routing hierarchies within ILM policies and index templates. The allocate action supports require, include, and exclude filters; fallback routing relies on the ordered index.routing.allocation.include._tier_preference list applied at the template level.

Production-Ready ILM Policy 

PUT _ilm/policy/logs-retention-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "7d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "allocate": {
            "include": { "_tier_preference": "data_warm,data_hot" },
            "number_of_replicas": 1
          },
          "shrink": { "number_of_shards": 1 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": { "include": { "_tier_preference": "data_cold,data_warm" } }
        }
      }
    }
  }
}

Index Template with Fallback Preference 

PUT _index_template/logs-app-fallback
{
  "index_patterns": ["logs-app-*"],
  "template": {
    "settings": {
      "index.routing.allocation.include._tier_preference": "data_warm,data_hot",
      "index.lifecycle.name": "logs-retention-policy",
      "index.lifecycle.rollover_alias": "logs-app"
    }
  }
}

The _tier_preference list instructs the allocator to attempt data_warm first, then gracefully fall back to data_hot if warm nodes are unavailable or at capacity. This prevents allocation deadlocks during rolling upgrades or hardware failures. For precise rollover triggers that interact with this routing logic, reference Configuring Index Rollover Conditions.

Python v8+ Client Orchestration

Deploying and validating fallback routing programmatically eliminates configuration drift. The elasticsearch Python client v8.x provides type-safe wrappers for ILM and allocation APIs.

import logging
from elasticsearch import Elasticsearch, ApiError
from elasticsearch.helpers import scan

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class ILMFallbackManager:
    def __init__(self, hosts: list[str], api_key: str):
        self.es = Elasticsearch(
            hosts=hosts,
            api_key=api_key,
            verify_certs=True,
            request_timeout=30
        )

    def deploy_policy(self, name: str, policy: dict) -> bool:
        try:
            self.es.ilm.put_lifecycle(name=name, policy=policy)
            logger.info(f"ILM policy '{name}' deployed successfully.")
            return True
        except ApiError as e:
            logger.error(f"Failed to deploy policy: {e}")
            return False

    def verify_allocation_routing(self, index_pattern: str) -> dict:
        """Checks current allocation state for indices matching pattern."""
        try:
            # Fetch indices matching pattern
            indices = self.es.cat.indices(index=index_pattern, format="json", h="index,health,status,pri,rep,store.size")
            results = {}
            for idx in indices:
                explain = self.es.cluster.allocation_explain(
                    index=idx["index"],
                    primary=True,
                    current_node=None,
                    include_disk_info=True
                )
                results[idx["index"]] = {
                    "current_node": explain.get("current_node"),
                    "can_allocate": explain.get("can_allocate"),
                    "deciders": explain.get("deciders", [])
                }
            return results
        except ApiError as e:
            logger.error(f"Allocation check failed: {e}")
            return {}

    def force_fallback_reroute(self, index: str, tier_preference: str = "data_warm,data_hot"):
        """Relax the index's tier preference so the allocator can fall back to
        another tier when ILM stalls. This is non-destructive — unlike
        allocate_stale_primary, it never risks data loss."""
        try:
            self.es.indices.put_settings(
                index=index,
                settings={
                    "index.routing.allocation.include._tier_preference": tier_preference
                },
            )
            # Ask the allocator to retry shards that previously failed allocation.
            self.es.cluster.reroute(retry_failed=True)
            logger.info(f"Applied fallback tier preference '{tier_preference}' to {index}")
        except ApiError as e:
            logger.error(f"Fallback reroute failed: {e}")

if __name__ == "__main__":
    manager = ILMFallbackManager(["https://es-cluster:9200"], "YOUR_API_KEY")
    # Deployment and validation flow
    manager.deploy_policy("logs-retention-policy", {
        "phases": {
            "hot": {"min_age": "0ms", "actions": {"rollover": {"max_primary_shard_size": "50gb"}}},
            "warm": {"min_age": "7d", "actions": {"allocate": {"include": {"_tier_preference": "data_warm,data_hot"}}}}
        }
    })
    status = manager.verify_allocation_routing("logs-app-*")
    logger.info(status)

Threshold Tuning & Watermark Alignment

Fallback routing only functions if disk watermarks do not block allocation entirely. Elasticsearch evaluates three thresholds: low, high, and flood_stage. When high is breached, the allocator stops placing new shards. When flood_stage is breached, indices are forced read-only.

Safe Production Tuning:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%",
    "cluster.routing.allocation.disk.threshold_enabled": true
  }
}

Align these thresholds with your storage provisioning. If fallback routing routes to hot nodes, ensure those nodes have sufficient headroom to absorb warm/cold shards without triggering flood_stage. Monitor fs.disk_usage via _nodes/stats and automate alerting at 80% to preemptively scale or purge.

Production Troubleshooting & Debugging Flow

When ILM stalls during tier transitions, follow this deterministic diagnostic sequence. All steps map directly to cluster APIs and require no third-party tooling.

1. Identify Stalled ILM Steps 

GET logs-app-2024.01.01/_ilm/explain

Inspect step_info. If error contains allocation_failed or no_valid_shard_copy, proceed to allocation diagnostics.

2. Diagnose Allocation Decisions 

POST _cluster/allocation/explain
{
  "index": "logs-app-2024.01.01",
  "shard": 0,
  "primary": true,
  "current_node": null
}

Review deciders. Common blockers:

  • disk_threshold_decider: Node exceeds watermark.
  • data_tier_allocation: no node carries a role matching the index’s _tier_preference.
  • same_shard: Replica placement conflicts.

3. Validate Node Roles 

GET _cat/nodes?v&h=name,node.role

Ensure fallback nodes carry the expected data-tier roles (for example data_hot, data_warm). A tier with no live node of that role causes silent allocation failures.

4. Force Tier Transition (Emergency Override)

If ILM remains stuck in the allocate action for >2 hours, bypass the policy temporarily by widening the index’s tier preference:

PUT logs-app-2024.01.01/_settings
{
  "index.routing.allocation.include._tier_preference": "data_hot,data_warm"
}

After shards relocate and stabilize, revert to the original template and reattach the ILM policy:

POST logs-app-2024.01.01/_ilm/retry

5. Verify Policy Execution Resumption 

GET _ilm/status
GET logs-app-2024.01.01/_ilm/explain

Confirm phase transitions to warm and action moves to allocate. Monitor _cat/indices?v&h=index,health,status,pri,rep,store.size&s=store.size:desc to validate shard distribution.

Fallback routing eliminates single-point failures in lifecycle management. By combining tier-aware _tier_preference lists, strict watermark governance, and automated Python orchestration, teams maintain continuous data retention even during infrastructure degradation.