Reindex vs Shrink for Warm-Tier Migration #

Moving an index off the hot tier almost always means fewer, larger shards. A hot index is over-sharded on purpose — spreading write load across primaries keeps ingest throughput high — but once it stops taking writes, that same shard count becomes pure overhead: every idle primary still costs heap for its segment metadata, a thread-pool slot, and cluster-state bookkeeping on the master. Consolidating three or six primaries down to one before the index settles on warm nodes reclaims all of that. Elasticsearch gives you two ways to get there, and they cost wildly different amounts. POST <src>/_shrink/<target> re-uses the source’s existing Lucene segments by hard-linking them into a new index with fewer primaries — near-zero data rewrite, seconds to minutes. POST _reindex reads every document out of the source and writes it into a brand-new index, which is far slower and heavier but can change anything: mappings, analyzers, routing, or an arbitrary shard count. This page sits inside the broader work of building automated reindexing pipelines and workflows, and it exists to make one decision precise — which operation to reach for when the only goal is preparing an index for the warm tier.

The short version, worked through in detail below: prefer shrink when you only need to reduce shard count and the mapping is unchanged, because it is cheap and because ILM already runs it for you as a native warm-phase action. Reach for reindex when the mapping, analyzer, or shard target forces a full document copy. Everything else is knowing the constraints that make shrink refuse to run — and the examples throughout use a source set of daily indices, orders-2025-*, consolidated into a single orders-2025-warm target.

Prerequisites #

• Elasticsearch 8.x with data-tier node roles assigned (data_hot, data_warm) so the shrunk or reindexed target can be routed onto warm hardware.
• elasticsearch-py v8.0+ installed — this page uses the v8 client surface (client.indices.shrink, client.reindex, keyword-argument bodies, typed exceptions), not the legacy body= pattern.
• For shrink: the source index made read-only (index.blocks.write: true) and all of its primary shards allocated onto a single node, with cluster health green for that index.
• For shrink: a target number_of_shards that is a factor of the source shard count (e.g. 6 → 1, 2, or 3), and enough free disk on the target node for the hard-linked (or copied) segments.
• For reindex: create_index, write, and manage on the target pattern and read on the source pattern; heap and I/O headroom on a coordinating node to drive the scroll-and-bulk copy.
• Monitoring access to GET _cat/shards, GET _cluster/health, and GET _tasks to confirm the operation before and after it runs.

Architecture: Two Roads to Fewer Shards #

The two operations reach the same destination — fewer primaries on a warm-ready index — through mechanically opposite means, and that difference dictates their cost and their flexibility. Shrink is a metadata operation. It requires every primary of the source to sit on one node so it can hard-link each source segment file into the new index’s directory on the same filesystem, then stitch those segments together under a smaller number of primaries. No document is re-read, re-analyzed, or re-indexed; the byte-for-byte segments are shared. That is why it is fast and why it is rigid: the target shard count must divide the source count evenly (you can only merge whole shards, never split one), and the mapping cannot change because the segments already encode it.

Reindex is a data operation. The coordinating node scrolls the source and issues bulk writes into a fresh target whose mapping, analysis chain, and shard count you defined from scratch. Because every document passes through the indexing pipeline again, you can remap a field, switch an analyzer, drop fields with a _source filter, run an ingest pipeline, or choose any shard count at all — 6 → 4 is fine here even though it is not a divisor. The price is that you pay full indexing cost a second time, and the target transiently doubles the storage of the source until you delete it.

Both roads end on the warm tier, but note where the flexibility lives: shrink trades away every degree of freedom except shard count in exchange for speed, while reindex buys total flexibility at full indexing cost. The rest of this page is about reading that trade-off against a concrete migration. For where the resulting index actually lands and the node roles that route it there, see the hot-warm-cold architecture reference.

Side-by-Side Comparison #

The table makes the decision boundary visible: every row where shrink wins is about cost, and every row where reindex wins is about flexibility. If you need none of the flexibility — same mapping, a shard count that divides evenly — shrink is strictly cheaper and, because it is a first-class ILM action, usually already automated for you.

Configuration Reference #

Shrink is a three-part operation: prepare the source, issue the shrink, then reset the source settings. The _shrink request body carries the target’s settings — critically, its number_of_shards (a factor of the source) and a reset of the write block and allocation requirement that the source needed but the target should not inherit.

// 1. Prepare the source: co-locate primaries and make it read-only.
PUT orders-2025/_settings
{
  "settings": {
    "index.routing.allocation.require._name": "warm-node-01",  // pull all primaries onto one node
    "index.blocks.write": true                                  // freeze writes; reads stay live
  }
}

// 2. Shrink into the warm target. Body = the NEW index's settings.
POST orders-2025/_shrink/orders-2025-warm
{
  "settings": {
    "index.number_of_shards": 1,                    // must divide the source shard count
    "index.number_of_replicas": 1,
    "index.routing.allocation.require._name": null, // clear the single-node pin on the target
    "index.blocks.write": null                       // target must be writable to finish, then read
  }
}

Reindex creates the target first (with the mapping and shard count you actually want), then copies. The request runs asynchronously so a large copy never trips the HTTP timeout:

// 1. Create the warm target with its own mapping and any shard count.
PUT orders-2025-warm
{
  "settings": { "index.number_of_shards": 4, "index.number_of_replicas": 1 },
  "mappings": {
    "properties": {
      "order_id":  { "type": "keyword" },
      "placed_at": { "type": "date" },
      "total":     { "type": "scaled_float", "scaling_factor": 100 }
    }
  }
}

// 2. Copy asynchronously; the response carries a task id, not the result.
POST _reindex?wait_for_completion=false
{
  "source": { "index": "orders-2025-*" },
  "dest":   { "index": "orders-2025-warm", "op_type": "create" },
  "conflicts": "proceed"
}

op_type: "create" makes a restarted reindex skip already-copied documents rather than overwrite them, and conflicts: "proceed" stops a single version conflict from aborting the whole task — the same idempotency contract the parent pipeline relies on.

Step-by-Step Implementation #

Shrink path (mappings unchanged) #

1. Pin the primaries. Set index.routing.allocation.require._name to a single node and wait for the deployment to relocate every primary and replica of orders-2025 there. Shrink refuses to start until they are co-located and the index is green.
2. Freeze writes. Set index.blocks.write: true. Reads continue; only ingestion stops.
3. Shrink. Issue POST orders-2025/_shrink/orders-2025-warm with the target number_of_shards (a factor of the source) and null out the single-node pin so the target can spread across warm nodes.
4. Reset. Once the target is green, remove the source’s write block and allocation requirement (or delete the source after an observation window).

from elasticsearch import Elasticsearch, ApiError

def shrink_for_warm(es: Elasticsearch, source: str, target: str,
                    node_name: str, target_shards: int = 1) -> None:
    # 1 + 2: co-locate every primary on one node and freeze writes.
    es.indices.put_settings(
        index=source,
        settings={
            "index.routing.allocation.require._name": node_name,
            "index.blocks.write": True,
        },
    )
    # Wait until the index is green on that node before shrinking.
    es.cluster.health(index=source, wait_for_status="green", timeout="120s")

    # 3: shrink — target_shards must be a factor of the source shard count.
    es.indices.shrink(
        index=source,
        target=target,
        settings={
            "index.number_of_shards": target_shards,
            "index.number_of_replicas": 1,
            "index.routing.allocation.require._name": None,  # unpin the target
        },
    )

try:
    shrink_for_warm(es_client, "orders-2025", "orders-2025-warm", "warm-node-01", 1)
except ApiError as exc:
    # A non-factor target, un-colocated primaries, or a writable source all surface here.
    raise RuntimeError(f"Shrink rejected: {exc.info}") from exc

Reindex path (mapping or non-divisor shard count changes) #

1. Create the target with the mapping, analyzers, and shard count you want — none of which need any relationship to the source.
2. Submit asynchronously with wait_for_completion=False; keep the returned task id.
3. Poll the task to completion, watching version_conflicts and throughput.
4. Swap or alias the warm index into place and delete the source after verification.

def reindex_for_warm(es: Elasticsearch, source_pattern: str, target: str) -> str:
    # Submit async: the response is a handle, not the finished result.
    response = es.reindex(
        source={"index": source_pattern},
        dest={"index": target, "op_type": "create"},
        conflicts="proceed",
        wait_for_completion=False,
        requests_per_second=2000,  # throttle so the copy never saturates the write pool
    )
    task_id = response["task"]  # v8: the async task id lives in the `task` key
    return task_id

task = reindex_for_warm(es_client, "orders-2025-*", "orders-2025-warm")
# Poll es.tasks.get(task_id=task) until it reports completed:true.

Verification #

For either path, count the shards before and after. Before shrink, orders-2025 should show its full primary count co-located on one node; after, orders-2025-warm should show the target count:

GET _cat/shards/orders-2025*?v&h=index,shard,prirep,state,node&s=index,shard

Confirm the target is green and fully allocated:

GET _cluster/health/orders-2025-warm?wait_for_status=green&timeout=60s

{ "status": "green", "active_shards": 2, "unassigned_shards": 0, "relocating_shards": 0 }

For a reindex, poll the async task to read progress and completion — created should climb toward total, and version_conflicts should stay flat:

GET _tasks/<task_id>

{
  "completed": true,
  "task": {
    "status": { "total": 4200000, "created": 4200000, "updated": 0, "version_conflicts": 0 }
  }
}

A created count equal to total with zero conflicts confirms a clean copy; if created is short of total because some documents already existed, that is op_type: create doing its job on a restarted run.

Decision Guidance: Factor, Mapping, and the ILM Default #

Three questions settle the choice, in order:

1. Does the mapping, analyzer, or routing need to change? If yes, you must reindex — shrink shares the source’s segments, which already encode the mapping, so there is no way to alter it in flight. This is the one hard forcing function.
2. Is the target shard count a factor of the source? Shrink can only merge whole shards, so 6 → 3, 6 → 2, and 6 → 1 are legal, but 6 → 4 is not. If your target count is not a divisor and you cannot pick one that is, reindex is the only option. (The interaction between shard count and per-primary sizing is covered under rollover based on max primary shard size.)
3. If neither forces a reindex — are you doing this by hand at all? For a pure shard-count reduction with an unchanged mapping, ILM’s warm phase already carries a shrink action. Attaching a policy with "shrink": { "number_of_shards": 1 } means the lifecycle engine co-locates the primaries, sets the write block, shrinks, and cleans up automatically when the index ages into warm — no manual runbook required. Manual shrink is for one-off consolidations of indices that were never under a policy.

The default answer for most warm-tier moves is therefore shrink, and let ILM do it. Reindex is the deliberate exception you take only when a schema change or a non-divisor shard count leaves you no cheaper road.

Troubleshooting #

Shrink rejected: primaries not co-located #

Symptom: the _shrink call fails with a message that a copy of every shard is not available on one node. Resolution:

1. Confirm the allocation pin took: GET _cat/shards/orders-2025?v&h=shard,prirep,state,node — every row must name the same node.
2. Wait for green: GET _cluster/health/orders-2025?wait_for_status=green. Relocation of large shards can take minutes.
3. Only then re-issue the shrink.

Shrink rejected: target shard count is not a factor #

Symptom: illegal_argument_exception — the requested number_of_shards does not divide the source count. Resolution: pick a divisor (for a 6-shard source: 1, 2, or 3), or switch to reindex if you genuinely need a non-divisor count like 4. There is no shrink path to an arbitrary shard count.

Shrink rejected: source is still writable #

Symptom: the operation refuses to run because index.blocks.write is not true. Resolution: set PUT orders-2025/_settings {"index.blocks.write": true} and retry. The block must be in place so the source segments are stable while they are hard-linked.

Reindex: version conflicts or write-pool rejections #

Symptom: version_conflicts climbs, or _cat/thread_pool shows rising rejected on the write pool. Resolution: ensure the source is write-locked so no concurrent writes race the copy, rely on conflicts: "proceed", and re-throttle live with es.reindex_rethrottle(task_id=..., requests_per_second=500) instead of cancelling. The full throttling and conflict-handling playbook lives in the parent pipeline reference.

FAQ #

Related #

• Choosing reindex or shrink for tier migration — the constraint-by-constraint decision guide distilled to a checklist.
• Shrinking an index before moving it to the warm tier — the full manual shrink runbook, step by step.
• Designing batch reindex workflows — partitioning a large reindex across queued, independent tasks.
• Tracking reindex progress & performance — reading the _tasks API and wiring alerts for the reindex path.
• Understanding hot-warm-cold architecture — where the consolidated index lands and the node roles that route it there.

← Back to Automated Reindexing Pipelines & Workflows