ILM & Reindexing Automation Runbooks

Shrinking an Index Before Moving It to the Warm Tier

To shrink orders-2025 from several primaries down to one before it settles on warm nodes, you co-locate every primary on a single node, make the index read-only, issue POST orders-2025/_shrink/orders-2025-warm, then reset the settings the shrink no longer needs.

Shrink is the cheap path to fewer shards because it hard-links the source’s existing Lucene segments into the new index instead of re-reading a single document — the whole operation is metadata plus filesystem links and usually completes in seconds to minutes. That speed comes with three non-negotiable preconditions this runbook satisfies in order. When the mapping is unchanged and the target count divides the source, shrink beats a full reindex on every axis of cost; deciding between the two is the job of the comparison guide. This page assumes that decision is already made and shows the manual mechanics, as one piece of the larger automated reindexing pipelines and workflows practice.

Prerequisites

  • Elasticsearch 8.x with a named data_warm node (this runbook targets warm-node-01) to receive the shrunk index.
  • elasticsearch-py v8.0+ for the helper (client.indices.shrink, keyword-argument settings).
  • A source orders-2025 whose target shard count is a factor of its current primary count, and enough free disk on the target node for the hard-linked segments.
  • manage on the orders-2025* pattern for the service account issuing the settings changes and the shrink.

Implementation

1. Co-locate every primary on one node

Shrink hard-links segments on a single filesystem, so every primary of orders-2025 must live on the same node before it can run. Force that with an allocation requirement, then wait for the deployment to finish relocating:

PUT orders-2025/_settings
{
  "settings": {
    "index.routing.allocation.require._name": "warm-node-01"
  }
}
GET _cluster/health/orders-2025?wait_for_status=green&wait_for_no_relocating_shards=true&timeout=120s

The index must be green with no relocating shards before you proceed — shrink refuses to start while a copy of any shard is missing from the target node.

2. Make the index read-only

The source segments must be stable while they are hard-linked, so freeze writes. Reads continue uninterrupted, so search availability is unaffected:

PUT orders-2025/_settings
{
  "settings": {
    "index.blocks.write": true
  }
}

3. Shrink into the warm target

Issue the shrink. The request body is the new index’s settings: its number_of_shards (a factor of the source), a replica count, and a reset of the single-node allocation pin so the shrunk index can spread across warm nodes afterward:

POST orders-2025/_shrink/orders-2025-warm
{
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 1,
    "index.routing.allocation.require._name": null,
    "index.blocks.write": null
  }
}

Nulling index.routing.allocation.require._name on the target releases it from the single node so Elasticsearch can balance it; nulling index.blocks.write leaves the target writable. Once the target reports green, clear the block and the allocation pin on the source too (or delete the source after an observation window):

PUT orders-2025/_settings
{
  "settings": {
    "index.blocks.write": null,
    "index.routing.allocation.require._name": null
  }
}

4. Python v8 helper

The same sequence, wrapped for repeatable, unattended runs. API and HTTP failures surface as ApiError (not TransportError, which is a sibling class and will not catch them):

import logging
from elasticsearch import Elasticsearch, ApiError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)


def shrink_before_warm(
    es: Elasticsearch,
    source: str,
    target: str,
    node_name: str,
    target_shards: int = 1,
) -> None:
    # 1. Pull every primary onto one node and freeze writes in a single settings call.
    es.indices.put_settings(
        index=source,
        settings={
            "index.routing.allocation.require._name": node_name,
            "index.blocks.write": True,
        },
    )

    # Block until the source is green on that node — shrink will reject otherwise.
    es.cluster.health(
        index=source,
        wait_for_status="green",
        wait_for_no_relocating_shards=True,
        timeout="120s",
    )
    logger.info("Source %s co-located on %s and write-blocked.", source, node_name)

    # 2. Shrink. target_shards MUST be a factor of the source primary count.
    es.indices.shrink(
        index=source,
        target=target,
        settings={
            "index.number_of_shards": target_shards,
            "index.number_of_replicas": 1,
            "index.routing.allocation.require._name": None,  # unpin the target
        },
    )
    logger.info("Shrunk %s into %s (%d primary).", source, target, target_shards)

    # 3. Reset the source so it is not left pinned and blocked.
    es.indices.put_settings(
        index=source,
        settings={
            "index.blocks.write": None,
            "index.routing.allocation.require._name": None,
        },
    )


es = Elasticsearch(
    "https://es-cluster-01:9200",
    api_key="YOUR_BASE64_ENCODED_API_KEY",
    verify_certs=True,
)

try:
    shrink_before_warm(es, "orders-2025", "orders-2025-warm", "warm-node-01", target_shards=1)
except ApiError as exc:
    # A non-factor target, un-colocated primaries, or a writable source all land here.
    logger.error("Shrink rejected (%s): %s", exc.status_code, exc.info)
    raise

Verification

Confirm the target exists with the reduced primary count and that both source and target sit where you expect:

GET _cat/shards/orders-2025*?v&h=index,shard,prirep,state,node&s=index,shard

orders-2025-warm should show a single primary (p) in state STARTED; the old orders-2025 shards remain until you delete the source. Then confirm the target is fully allocated and healthy:

GET _cluster/health/orders-2025-warm?wait_for_status=green&timeout=60s
{ "status": "green", "active_shards": 2, "relocating_shards": 0, "unassigned_shards": 0 }

A green target with zero unassigned shards means the shrink completed and the new index is ready to serve reads from the warm tier.

Gotchas and edge cases

  • The target shard count must divide the source. Shrink merges whole primaries, so a 6-shard source can shrink to 3, 2, or 1 but never to 4 or 5. Choose number_of_shards as a factor of the source; a non-divisor target is rejected with an illegal_argument_exception and forces a reindex instead.
  • Read-only and co-location are both mandatory. Shrink will not start unless index.blocks.write is true and every primary sits on the target node. Skipping the health wait in step 1 is the most common cause of a shrink that fails immediately because a shard copy is still relocating.
  • Hard-links need one filesystem. The speed of shrink comes from hard-linking segment files, which only works when source and target directories live on the same filesystem on the same node — which is exactly why co-location is required. On a node whose data path spans multiple filesystems, Elasticsearch falls back to copying, which is slower but still correct.
  • ILM automates all of this in the warm phase. For a managed index, a warm-phase "shrink": { "number_of_shards": 1 } action makes ILM co-locate, write-block, shrink, and swap the alias with no manual runbook. Run this procedure by hand only for unmanaged indices or one-off consolidations outside a policy.

FAQ

Does shrinking cause any search downtime?

No search downtime. Making the source read-only stops writes only — reads against orders-2025 continue throughout co-location and the shrink itself. Once orders-2025-warm is green you swap read traffic to it (via an alias) and retire the source. Ingestion is the only thing paused, and only from the moment you set index.blocks.write: true.

How long does a shrink take compared with a reindex?

Shrink usually completes in seconds to a few minutes regardless of index size, because it hard-links existing segments rather than re-reading documents. A reindex of the same data re-indexes every document and can run for minutes to hours depending on throughput. That gap is the entire reason to prefer shrink when the mapping is unchanged and the target count is a divisor.

Can I shrink directly to the number of shards I want on warm?

Only if that number is a factor of the source primary count. Shrink can merge a 6-shard index to 3, 2, or 1, so pick the warm target from that set. If your desired warm shard count is not a divisor — say 4 from a 6-shard source — shrink cannot produce it and you must reindex into a freshly created index with that shape.

← Back to Reindex vs Shrink for Warm-Tier Migration · Automated Reindexing Pipelines & Workflows