ILM & Reindexing Automation Runbooks

Handling Version Conflicts with External Versioning

When orchestrating Automated Reindexing Pipelines & Workflows, external versioning (version_type=external or external_gte) is mandated to preserve strict monotonic sequence numbers from upstream telemetry, Kafka offsets, or CDC streams. During ILM-driven rollovers or zero-downtime migrations, these version vectors routinely collide with Elasticsearch’s internal _version tracking. The result is a deterministic cascade of version_conflict_engine_exception errors that stall bulk ingestion, fragment shard allocation, and corrupt immutable audit trails. This guide provides the definitive diagnostic and remediation protocol for rapid cluster restoration and compliance enforcement.

flowchart TD
  A["version_conflict_engine_exception"] --> B{"incoming vs stored version"}
  B -->|"incoming lower"| C["Stale write: drop or fix upstream offset"]
  B -->|"equal"| D["Replay with external_gte"]
  B -->|"out of order arrival"| E["Pause producers, reorder, retry"]

Root-Cause Analysis & Reproducible Triggers

External versioning enforces absolute monotonicity. Elasticsearch rejects any write where incoming_version <= existing_version. Unlike internal versioning, which auto-increments on every mutation, external versioning treats the supplied integer as ground truth. Conflicts manifest under three operational conditions:

  1. Out-of-Order Bulk Dispatch: Network partitions, thread-pool saturation, or uneven shard routing cause non-sequential arrival. A document with version=104 lands before 105. Retries without version adjustment trigger immediate rejection and queue backpressure.
  2. Dual-Write Overlap During Migration: Live application traffic continues writing to the source index while the _reindex job executes. Clock skew, offset resets, or partition rebalancing yield newer documents carrying lower external version numbers than already-reindexed records.
  3. ILM State Leakage & Metadata Inheritance: When ILM transitions an index to shrink or rollover, the target index inherits document metadata. Pipelines that blindly copy _version without explicitly declaring version_type default to internal versioning. Subsequent external-versioned updates attempting to overwrite inherited state are immediately rejected.

Diagnostic Workflow & Cluster State Validation

Execute the following sequence to isolate stuck tasks and collision vectors. Do not proceed until cluster state is fully mapped and shard health is verified.

# 1. Identify failed reindex tasks and extract conflict payloads
GET _tasks?actions=*reindex*&detailed=true&error_trace=true

Expected Output Analysis: Filter for "type": "version_conflict_engine_exception". Note the current version [X] vs provided [Y] delta. A positive delta indicates out-of-order arrival; a zero/negative delta indicates duplicate or stale writes.

# 2. Verify cluster health and shard allocation state
GET _cluster/health?pretty&level=shards

Expected Output:

{
  "status": "yellow",
  "unassigned_shards": 12,
  "indices": {
    "logs-app-000001": {
      "status": "yellow",
      "shards": { "total": 5, "unassigned": 2 }
    }
  }
}

Version conflicts frequently trigger shard-level throttling or delayed allocation. Cross-reference unassigned shards with the target index pattern.

# 3. Isolate ILM policy execution state
GET <target-index-pattern>/_ilm/explain?pretty

Expected Output:

{
  "indices": {
    "logs-app-000001": {
      "index": "logs-app-000001",
      "managed": true,
      "policy": "daily-rollover",
      "phase": "hot",
      "action": "rollover",
      "step": "check-rollover-ready",
      "step_info": {
        "message": "rollover condition not met, waiting..."
      }
    }
  }
}

If step is shrink or rollover and phase is stuck while conflicts accumulate in _reindex, metadata inheritance is actively blocking writes. Review baseline conflict resolution thresholds at Resolving Document Conflicts During Reindex before proceeding.

# 4. Segment-level version fragmentation check
GET _cat/segments/<target-index>?v&h=index,shard,segment,version&sort=index,shard

High variance in version across segments indicates fragmented write paths. Consolidate diagnostics before initiating recovery.

Resolution Protocol & Safe Manual Reroutes

Once diagnostics confirm external version collisions, execute controlled remediation. Do not force-merge or manually delete shards.

Step 1: Halt Conflicting Ingestion Pause upstream producers or throttle the _reindex task. Use POST _tasks/<task_id>/_cancel to stop the bleeding pipeline. Verify cancellation via _tasks?actions=*reindex*.

Step 2: Safe Manual Reroute If shards are unassigned due to conflict-induced allocation failures, execute a controlled reroute. Never use allocate_empty_primary. Use allocate_stale_primary only after verifying data consistency:

POST _cluster/reroute
{
  "commands": [
    {
      "allocate_stale_primary": {
        "index": "logs-app-000001",
        "shard": 2,
        "node": "data-node-03",
        "accept_data_loss": false
      }
    }
  ]
}

Monitor allocation state with GET _cluster/allocation/explain.

Step 3: Conflict Resolution Strategy If conflicts=proceed was used, conflicting documents were skipped (counted under version_conflicts), not dropped — the existing destination document, which already held an equal or higher version, was preserved. To re-apply updates idempotently, replay with version_type=external_gte, which accepts an incoming version greater than or equal to the stored version while still rejecting strictly lower versions (monotonicity is preserved, not relaxed). Refer to the official concurrency control documentation for exact parameter behavior: Optimistic Concurrency Control.

Automated Python v8+ Recovery Pattern

Deploy the following script to scan, reconcile, and replay failed documents using the official Elasticsearch Python client (v8+). This pattern enforces compliance by logging all overrides and preserving audit trails.

import elasticsearch
import logging
from elasticsearch.helpers import bulk

# Configure client with strict timeout and retry logic (v8+ syntax)
client = elasticsearch.Elasticsearch(
    ["https://cluster-node-01:9200"],
    api_key="YOUR_API_KEY",
    request_timeout=30,
    max_retries=3,
    retry_on_timeout=True,
    verify_certs=True
)

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("version_conflict_recovery")

def recover_external_version_conflicts(target_index, failed_docs_batch):
    """
    Replays failed documents using external_gte to enforce idempotent upserts.
    Logs all version overrides for audit compliance.
    """
    actions = []
    for doc in failed_docs_batch:
        doc_id = doc["_id"]
        doc_body = doc["_source"]
        ext_version = doc_body.get("external_version", doc.get("_version"))

        actions.append({
            "_index": target_index,
            "_id": doc_id,
            "_op_type": "index",
            # helpers.bulk action metadata uses underscore-prefixed keys; plain
            # "version"/"version_type" would be ignored, silently disabling versioning.
            "_version": ext_version,
            "_version_type": "external_gte",  # accepts >= instead of strict >
            "_source": doc_body
        })

    try:
        success, errors = bulk(
            client, 
            actions, 
            raise_on_error=False, 
            chunk_size=500,
            stats_only=True
        )
        logger.info(f"Recovery complete: {success} documents upserted.")
        if errors:
            logger.warning(f"{errors} documents failed. Review cluster logs for root cause.")
    except Exception as e:
        logger.critical(f"Bulk recovery failed: {e}")
        raise

# Reference official client documentation for bulk API parameters:
# https://elasticsearch-py.readthedocs.io/en/v8.15.0/helpers.html

Validate monotonic progression post-recovery:

GET <target-index>/_search
{
  "size": 0,
  "aggs": {
    "max_version": { "max": { "field": "external_version" } },
    "min_version": { "min": { "field": "external_version" } }
  }
}

Resume ILM only after confirming conflicts: 0 in subsequent _reindex runs.

Escalation Paths & Compliance Enforcement

  • Tier 1 (Automated Circuit Breaker): If version_conflict_engine_exception rate exceeds 5% of bulk throughput, trigger immediate producer backpressure and switch ingestion to external_gte.
  • Tier 2 (Manual Intervention): If shard allocation fails post-recovery, execute _cluster/reroute with allocate_stale_primary only after verifying data consistency. Never bypass accept_data_loss: false without explicit engineering sign-off.
  • Tier 3 (Architecture Review): Persistent collisions indicate upstream sequence generation flaws. Mandate strict offset tracking via centralized version arbitration or implement exactly-once delivery guarantees at the transport layer.
  • Audit Compliance: All version overrides must be logged with user, timestamp, original_version, and resolution_method fields. Immutable audit indices must be explicitly excluded from external versioning to prevent recursive conflict loops. Maintain strict separation between operational telemetry and compliance records.