ILM & Reindexing Automation Runbooks

Safely Aborting a Running Reindex Task

A _reindex is saturating the deployment or was pointed at the wrong destination, and you need to stop it cleanly — this runbook covers finding the task, deciding between throttling and killing it, and dealing with the partial data an aborted copy always leaves behind.

The trap is treating cancel like an undo button. A reindex is not transactional: every document it has already written stays written, cancellation lands at a batch boundary rather than instantly, and killing the task is often a heavier response than the situation needs. This page sits inside tracking reindex progress and performance, where the same _tasks telemetry that shows a copy running hot is what you use to slow it, stop it, and confirm it stopped. It is one operation within the wider set of automated reindexing pipelines and workflows, and the safe path almost always starts with a throttle, not a kill.

Prerequisites

  • Elasticsearch 8.x with the _tasks, _reindex/<id>/_rethrottle, and _tasks/<id>/_cancel endpoints reachable from the automation host.
  • elasticsearch-py v8.0+ (pip install "elasticsearch>=8,<9"); the helper below uses client.tasks.list, client.tasks.get, client.reindex_rethrottle, and client.tasks.cancel — the tasks client exposes only cancel, get, and list.
  • The task id (<node>:<id>) the reindex returned when submitted with wait_for_completion=false, or the ability to look it up via _tasks.
  • manage on the destination index pattern, plus cluster monitor to read _tasks, scoped per securing ILM policies with RBAC.

The abort decision path

Before cancelling anything, decide whether cancelling is even the right move. If the problem is load, a rethrottle fixes it without discarding the work already done; if the problem is correctness — wrong destination, wrong query — then cancel, but plan for the partial destination the copy leaves behind. The flow below is the whole runbook: choose throttle over cancel where you can, and when you do cancel, wait for the cooperative stop and then clean up.

Throttle when the copy is only heavy; cancel when it is wrong, then wait for the stop and clean the partial destinationA decision path. The opening question, load or correctness, splits two ways. The upper branch, labelled too heavy, leads to POST _reindex/task/_rethrottle, which lowers requests_per_second and keeps completed work; it is marked the preferred path. The lower branch, labelled mis-targeted, leads to POST _tasks/task/_cancel. From cancel, an arrow leads to Wait for stop, noting cancellation is cooperative and lands at the next batch boundary. A final arrow leads to Clean partial destination, which says delete the destination index if new or reconcile if pre-existing.load orwrong?too heavyPOST _rethrottlelower rps · keep work · preferredmis-targetedPOST _tasks/../_cancelstop the copyWait for stopnext batch boundaryClean destdelete if newreconcile if not

Implementation

1. Find the task id

If the reindex was submitted with wait_for_completion=false, it returned a { "task": "<node>:<id>" } handle. If that id was lost — a crashed automation host, a copy someone kicked off by hand — recover it from the tasks API by filtering on the reindex action:

GET _tasks?actions=*reindex&detailed=true&human

The detailed=true flag includes the source and destination in each task’s description, which is how you tell a mis-targeted copy apart from a legitimate one when several reindex tasks are in flight:

{
  "nodes": {
    "aH3xQ2": {
      "tasks": {
        "aH3xQ2:918273": {
          "action": "indices:data/write/reindex",
          "description": "reindex from [logs-2023.01] to [logs-2023.01-v2]",
          "running_time_in_nanos": 742000000000,
          "cancellable": true,
          "status": { "total": 4200000, "created": 1830000, "batches": 366 }
        }
      }
    }
  }
}

Note "cancellable": true — reindex tasks are cancellable, and created already shows 1.83 million documents written, the partial destination you will have to account for.

2. Prefer a rethrottle over a kill

If the copy is only too heavy — the write thread pool is backing up, queries are slowing — you do not need to abort it at all. Slowing it preserves every document already copied and lets the migration finish on its own. POST _reindex/<task_id>/_rethrottle?requests_per_second=N retunes the live task; set a low positive number to throttle hard, or -1 to remove throttling. In the v8 client this is client.reindex_rethrottle:

# Slow a saturating copy to 50 req/s instead of killing it.
client.reindex_rethrottle(task_id="aH3xQ2:918273", requests_per_second=50)

Only escalate to cancellation when the copy is wrong — pointed at the wrong destination, running the wrong query, or otherwise producing data you do not want — because that is the case where finishing the copy is worse than stopping it.

3. Cancel cooperatively, then reconcile

Cancellation is cooperative: POST _tasks/<task_id>/_cancel marks the task cancelled, but it stops at the next batch boundary, not the instant you call it, so a task mid-batch keeps writing until that batch finishes. And because reindex is not transactional, every document already written stays in the destination — cancelling leaves a partial index, never a clean rollback. The helper below throttles or cancels, waits for the cooperative stop, and reports the destination count so you can plan the cleanup.

import time
import logging
from elasticsearch import Elasticsearch, ApiError, NotFoundError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("reindex_abort")


class ReindexAbort:
    def __init__(self, es: Elasticsearch):
        self.es = es

    def find_reindex_tasks(self) -> dict[str, str]:
        # Recover live reindex task ids and their source→dest description.
        resp = self.es.tasks.list(actions="*reindex", detailed=True)
        found = {}
        for node in resp.get("nodes", {}).values():
            for tid, task in node.get("tasks", {}).items():
                found[tid] = task.get("description", "")
                logger.info("Reindex task %s: %s", tid, task.get("description"))
        return found

    def throttle(self, task_id: str, rps: int) -> None:
        # Preferred first response: slow, don't kill. rps=-1 removes throttling.
        self.es.reindex_rethrottle(task_id=task_id, requests_per_second=rps)
        logger.info("Rethrottled %s to %s req/s", task_id, rps)

    def cancel_and_wait(self, task_id: str, timeout_s: int = 120) -> dict:
        # Cancellation is COOPERATIVE — the task stops at the next batch boundary.
        try:
            self.es.tasks.cancel(task_id=task_id)
        except ApiError as exc:
            logger.error("Cancel failed for %s (%s): %s", task_id, exc.status_code, exc.info)
            raise
        logger.info("Cancel requested for %s — waiting for it to stop", task_id)

        deadline = time.monotonic() + timeout_s
        while time.monotonic() < deadline:
            try:
                task = self.es.tasks.get(task_id=task_id)
            except NotFoundError:
                # Task object gone means it has fully stopped and been reaped.
                logger.info("Task %s no longer present — stopped", task_id)
                return {"stopped": True}
            if task.get("completed"):
                status = task.get("task", {}).get("status", {})
                written = status.get("created", 0) + status.get("updated", 0)
                logger.info("Task %s stopped after writing ~%s docs", task_id, written)
                return {"stopped": True, "written": written}
            time.sleep(2)
        logger.warning("Task %s still running after %ss", task_id, timeout_s)
        return {"stopped": False}

    def dest_doc_count(self, dest_index: str) -> int:
        self.es.indices.refresh(index=dest_index)
        count = self.es.count(index=dest_index)["count"]
        logger.info("Destination %s holds %s docs (partial)", dest_index, count)
        return count


if __name__ == "__main__":
    es = Elasticsearch(
        "https://es-cluster-01:9200",
        api_key="YOUR_BASE64_ENCODED_API_KEY",
        verify_certs=True,
    )
    aborter = ReindexAbort(es)

    aborter.find_reindex_tasks()
    result = aborter.cancel_and_wait("aH3xQ2:918273")
    if result["stopped"]:
        # The destination was newly created for this copy — safe to drop and rebuild.
        aborter.dest_doc_count("logs-2023.01-v2")
        es.indices.delete(index="logs-2023.01-v2")
        logger.info("Deleted partial destination for a clean re-run.")

If the destination was created fresh for this copy, deleting it is the clean reset — a re-run starts from an empty index. If the destination pre-existed (you were merging into a live index), you cannot simply drop it; reconcile instead by identifying which _id values the aborted copy wrote and deciding whether to keep or remove them, ideally with op_type=create on the re-run so existing documents are never silently overwritten.

Verification

Confirm the task actually stopped — a cancel request is not proof it has landed. Fetch the task and expect completed: true with the cancellation recorded:

GET _tasks/aH3xQ2:918273
{
  "completed": true,
  "task": {
    "action": "indices:data/write/reindex",
    "status": { "total": 4200000, "created": 1830000, "batches": 366 },
    "description": "reindex from [logs-2023.01] to [logs-2023.01-v2]"
  },
  "response": {
    "canceled": "by user request",
    "created": 1830000
  }
}

The response.canceled string confirms a clean, user-requested stop, and created is the exact size of the partial destination. Cross-check that against the destination itself so you know precisely what to clean up:

GET _cat/count/logs-2023.01-v2?v

The count should match response.created (plus any updated). A destination that was newly created can now be deleted; a pre-existing one needs the reconciliation described above before it is trustworthy again.

Gotchas and edge cases

  • Cancel is not instant. _tasks/<id>/_cancel is cooperative — the task finishes its current batch before stopping, so on a large size a few thousand more documents land after you call it. Always verify with GET _tasks/<id> that completed is true before you touch the destination; acting on the assumption that the copy stopped the moment you hit cancel is how you race a still-writing task.
  • Partial writes stay — there is no rollback. Reindex is not transactional, so every document written before the abort remains in the destination. Plan for it: delete and rebuild a newly created destination, or reconcile a pre-existing one against response.created. Never assume a cancelled copy left the destination untouched.
  • Make the re-run safe by construction. Submit reindex jobs with op_type=create (which refuses to overwrite an existing _id) or always copy into a fresh destination index. Either choice makes an abort-and-retry idempotent: the re-run either skips the documents the aborted attempt already wrote or starts from a clean slate, instead of double-writing or clobbering.
  • Throttle before you kill. If the only problem is load, reindex_rethrottle down to a low requests_per_second (or -1 to pause pressure) relieves the deployment while keeping the work already done — cancelling instead throws away that work and forces a full re-copy. Reserve cancellation for a copy that is genuinely wrong, not merely heavy. The throttle-vs-load calibration behind this lives in optimizing reindex thresholds and bulk sizes.

FAQ

Does cancelling a reindex roll back the documents it already copied?

No. _reindex is not transactional — there is no rollback. Every document written before the cancel lands stays in the destination, so an aborted copy always leaves a partial index. The response.created count on the cancelled task tells you exactly how many documents are there. Clean up by deleting the destination if it was created just for this copy, or by reconciling the written _id values if it pre-existed.

Should I cancel a reindex that is overloading the deployment, or throttle it?

Throttle it first. If the copy is correct and merely too heavy, POST _reindex/<task_id>/_rethrottle?requests_per_second=N (or client.reindex_rethrottle) lowers the rate without discarding the documents already written, and the migration finishes on its own once load subsides. Only cancel when the copy is actually wrong — mis-targeted or running the wrong query — because cancelling throws away the work done so far and forces a full re-copy.

Why is my reindex still writing after I cancelled it?

Because cancellation is cooperative. _tasks/<id>/_cancel marks the task cancelled, but it only stops at the next scroll batch boundary, so a task in the middle of a batch keeps writing until that batch completes. On a large size that can be several thousand more documents. Poll GET _tasks/<id> and wait for completed: true before you delete or reconcile the destination.

← Back to Tracking Reindex Progress & Performance · Automated Reindexing Pipelines & Workflows