ILM & Reindexing Automation Runbooks

Resolving Document Conflicts During Reindex: Operational Playbook for ILM & Automation Pipelines

Document conflicts during index lifecycle transitions represent a primary failure vector in high-throughput Elasticsearch environments. When executing mapping migrations, ILM rollovers, or architecture refactors, the _reindex API frequently encounters version_conflict_engine_exception errors. These stem from concurrent ingestion pipelines, external versioning drift, or misaligned seq_no/primary_term states. Left unmanaged, conflicts trigger pipeline aborts, shard hot-spotting, and silent data loss. Establishing a deterministic strategy for Resolving Document Conflicts During Reindex is a prerequisite for maintaining data integrity and SLA compliance across automated Automated Reindexing Pipelines & Workflows.

flowchart TD
  A["Reindex into an existing target?"] --> B{"Conflict strategy"}
  B -->|"skip docs that exist"| C["op_type: create"]
  B -->|"keep newest by version"| D["version_type: external / external_gte"]
  B -->|"continue past clashes"| E["conflicts: proceed"]
  C --> F["Reconcile version_conflicts after run"]
  D --> F
  E --> F

Configuration & Policy Alignment

Elasticsearch defaults to conflicts=abort, which halts execution on the first mismatch. For production log analytics and search migrations, operators must explicitly configure conflicts=proceed in the reindex payload. This allows the background task to continue while aggregating conflict counts in the task metadata, enabling post-migration reconciliation rather than mid-flight pipeline stalls.

When decoupling from internal versioning—for instance, when migrating from Kafka-backed CDC streams or application-level sequence IDs—you must configure version_type=external and ensure the destination index mapping accepts the external payload. Shard allocation awareness during bulk transfers prevents I/O bottlenecks; explicitly define index.routing.allocation.include._tier_preference and pre-warm indices before traffic cutover. Properly tuning slices=auto alongside max_docs and scroll parameters ensures throughput remains predictable, a critical factor when Designing Batch Reindex Workflows that operate within strict maintenance windows.

Python v8+ Orchestration & Execution

Modern orchestration requires programmatic control over task submission, polling, and error categorization. The Elasticsearch Python client v8+ provides native async execution and structured task management. Below is a production-ready implementation that submits a reindex task, polls for completion with exponential backoff, and extracts conflict metrics:

import asyncio
import time
from elasticsearch import AsyncElasticsearch
from elasticsearch.exceptions import ApiError

async def execute_and_monitor_reindex(es: AsyncElasticsearch, source_index: str, dest_index: str):
    # Pick ONE conflict strategy: `op_type: create` (skip docs that already exist)
    # OR external versioning (`version_type: external`). They are mutually exclusive —
    # see the external-versioning sub-guide for that alternative.
    body = {
        "source": {
            "index": source_index,
            "size": 5000,
            "query": {"match_all": {}}
        },
        "dest": {
            "index": dest_index,
            "op_type": "create"
        },
        "conflicts": "proceed",
        "slices": "auto",
        "requests_per_second": 1000
    }

    try:
        response = await es.reindex(body=body, wait_for_completion=False)
        task_id = response["task"]
        print(f"Reindex task submitted: {task_id}")
    except ApiError as e:
        print(f"Failed to submit reindex task: {e.info}")
        raise

    # Poll task status with exponential backoff
    backoff = 2.0
    while True:
        task_status = await es.tasks.get(task_id=task_id)
        status = task_status["task"]["status"]
        completed = task_status["completed"]
        
        if completed:
            conflicts = status.get("version_conflicts", 0)
            print(f"Task complete. Version conflicts: {conflicts}")
            return task_status
        
        await asyncio.sleep(backoff)
        backoff = min(backoff * 1.5, 30.0)

For detailed telemetry integration and metric extraction patterns, refer to Tracking Reindex Progress & Performance. The official Elasticsearch Python Client Documentation provides additional async context manager patterns for connection pooling and timeout handling.

Threshold Tuning & Bulk Optimization

Conflict resolution overhead scales non-linearly with bulk size. The default _reindex batch size (1000) often causes thread pool exhaustion on nodes with heavy write loads. Adjust size in the source payload to 2000–5000 documents, but continuously monitor thread_pool.write.queue and jvm.mem.heap_used_percent via _nodes/stats. If version_conflict_engine_exception rates exceed 5%, reduce requests_per_second to throttle ingestion and allow the destination cluster to flush segments.

Apply index.refresh_interval: -1 on the destination index during migration to minimize segment creation I/O, then restore to 1s post-cutover. When targeting petabyte-scale clusters, partition reindex operations by time range or hash routing to isolate conflict domains. Refer to the official Elasticsearch Reindex API documentation for parameter precedence, cluster-level defaults, and memory footprint calculations.

Real-World Debugging & Troubleshooting Flow

When conflicts persist despite conflicts=proceed, systematic debugging is required. Follow this operational flow to isolate and remediate failure modes:

  1. Identify Conflict Topology: Run GET _tasks/<task_id> and inspect response.failures. If failures show version_conflict_engine_exception, extract the _id and _version from the source document. Cross-reference with GET /<index>/_doc/<id> to verify if an active ingestion pipeline is writing concurrently.
  2. Validate Sequence Numbers: Check seq_no and primary_term alignment using GET /<index>/_doc/<id>?routing=<routing_key>. Mismatches indicate concurrent writes bypassing the reindex pipeline or a split-brain scenario during primary shard relocation.
  3. External Versioning Drift: If using application-level sequence IDs, verify that the destination mapping explicitly allows version_type=external. Misconfigured mappings will reject payloads with version <= existing_version. Detailed resolution strategies for this scenario are covered in Handling Version Conflicts with External Versioning.
  4. Routing & Shard Allocation Mismatch: Conflicts often manifest when routing keys differ between source and destination. Ensure dest.routing matches the source, or explicitly set "dest": {"routing": ""} in the reindex body to preserve shard locality.
  5. Partial Failure Recovery: A _reindex task cannot be resumed from a scroll cursor — the tasks API does not expose a resumable scroll_id for reindex. Instead, re-run _reindex with a bounding range query on @timestamp (or an _id range) that covers only the unprocessed window, combined with op_type: "create" so already-copied documents are skipped rather than overwritten. This makes re-runs idempotent without duplicate processing.

Resolving Document Conflicts During Reindex requires a shift from reactive error handling to proactive pipeline design. By aligning conflicts=proceed with external versioning controls, tuning bulk thresholds against cluster capacity, and implementing structured Python v8+ orchestration, operators can guarantee deterministic migrations. Automated conflict resolution is no longer optional; it is the operational baseline for scaling Elasticsearch across petabyte-class data architectures.

Subtopic Handling Version Conflicts with External Versioning