ILM Rollover vs Curator-Based Rotation #

For most of a decade, the standard way to keep an Elasticsearch time-series pipeline from filling its disks was elasticsearch-curator: an external Python tool, invoked from cron, that issued a scripted sequence of delete, rollover, forcemerge, and snapshot calls against the API on a fixed clock. Index Lifecycle Management (ILM) replaced that model with a declarative state machine that lives inside the deployment and reconciles each managed index against a policy. Teams migrating off Curator cron scripts need a precise answer to two questions: which Curator actions map one-to-one onto ILM, and where the two models diverge in ways that change operational behaviour. This page draws that map for a syslog rotation pipeline, weighing the trade-offs fairly rather than pitching one tool as universally superior.

The comparison only makes sense against the architecture ILM plugs into. Curator treats the deployment as an opaque endpoint it fires commands at; ILM treats it as a set of data tiers whose allocation state it is continuously aware of. That difference — external imperative scripting versus internal declarative reconciliation — is the root of almost every behavioural gap discussed below, and it is grounded in the Index Lifecycle Management (ILM) control plane. What follows assumes you have an existing rotation to migrate and want to understand exactly what you gain, what you give up, and what stays the same.

Prerequisites #

• Elasticsearch 8.x with data-tier node roles assigned (data_hot, data_warm, data_cold), since ILM routes phase transitions through those roles where Curator issued flat allocation filters.
• An existing Curator config to migrate (optional) — an action_file.yml and its cron entry — so you can map each action to an equivalent phase rather than re-deriving retention from scratch.
• elasticsearch-py v8.0+ installed for the automation snippets; this page uses the v8 client surface (ilm.put_lifecycle, keyword-argument bodies, typed exceptions), not the legacy body= pattern.
• manage_ilm and manage_index_templates cluster privileges on the account that applies the policy and template.
• Monitoring access to GET _ilm/explain and the cat APIs to confirm indices are managed after the cutover, and shell access to the host running the Curator cron so you can disable it.
• A baseline of the current rotation cadence (rollover conditions and deletion age) so the ILM policy reproduces the same retention window, not a new one.

Architecture: External Cron vs Internal Reconciliation #

Curator and ILM produce overlapping outcomes — both roll over, force-merge, and delete indices — but they arrive there through opposite mechanics. Curator is a batch program: cron wakes it on a schedule, it evaluates a filter chain against the current index list, issues the matching imperative API calls, and exits. Between invocations nothing watches the deployment. If a delete call lands while a shard is mid-relocation, or fires a rollover into a tier that is over its disk watermark, Curator has no way to know — it has already returned control to cron. ILM, by contrast, runs as a coordinator on the elected master, polling every indices.lifecycle.poll_interval (default 10m) and advancing each managed index one step at a time, only when the deployment’s allocation state allows it.

The consequence is that the two tools fail differently. A Curator run that hits a transient error typically aborts the action and waits for the next cron tick, potentially skipping a rotation entirely; nothing records that the index is now behind. ILM records the failure as an ERROR step on the specific index, holds it there without touching its neighbours, and retries the same step on the next poll once the underlying condition clears. Neither is strictly better in the abstract, but ILM’s per-index, retry-until-ready model is far easier to reason about at fleet scale, because the state of every index is queryable at any instant through GET <index>/_ilm/explain rather than reconstructable only from cron logs.

Curator also has no concept of a phase clock tied to the rollover event. It filters on creation_date or name, computing age from when an index was created. ILM’s min_age in warm, cold, and delete phases is measured from the moment the index rolled over, not when it was created. That single semantic shift is the most common source of surprise during a migration: a retention window that read “delete indices older than 90 days” under Curator is not automatically the same 90-day window under ILM, because the clock now starts later.

Capability Comparison #

The table below maps the operations a rotation pipeline actually performs to how each tool handles them. It is deliberately even-handed: Curator retains genuine advantages for a handful of tasks that fall outside ILM’s declarative model.

Two rows deserve emphasis. Curator’s delete-by-age against arbitrary index patterns with no policy attached is something ILM does not do natively — ILM only deletes indices it manages, through their own delete phase. And Curator’s snapshot scheduling has a direct successor not in ILM but in Snapshot Lifecycle Management (SLM), which is the correct target for the snapshot half of a Curator config. Rollover, delete, forcemerge, and shrink all map cleanly onto ILM phases; snapshot scheduling maps onto SLM; and a small residue of ad-hoc maintenance stays with Curator or a bespoke script.

Configuration Reference #

The clearest way to see what maps and what changes is to place an illustrative Curator action file beside the ILM policy that reproduces its behaviour. The Curator YAML below is a representative rotation for a syslog-* pipeline — rollover the write alias, force-merge indices older than two days, delete indices older than ninety:

# /etc/curator/actions/syslog.yml — illustrative legacy config
actions:
  1:
    action: rollover
    options:
      name: syslog-write
      conditions:
        max_age: 1d
        max_primary_shard_size: 50gb
  2:
    action: forcemerge
    options:
      max_num_segments: 1
    filters:
      - filtertype: age
        source: creation_date        # age from creation, not rollover
        direction: older
        unit: days
        unit_count: 2
  3:
    action: delete_indices
    filters:
      - filtertype: pattern
        kind: prefix
        value: syslog-
      - filtertype: age
        source: creation_date
        direction: older
        unit: days
        unit_count: 90

The equivalent ILM policy folds all three actions into one declarative document. Each Curator action becomes a phase; the filter chains become min_age boundaries; and the rollover conditions move verbatim into the hot phase:

PUT _ilm/policy/syslog-ilm
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "50gb"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "2d",
        "actions": {
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}

Two differences are load-bearing, not cosmetic. First, the Curator forcemerge and delete_indices filters compute age from creation_date, whereas the ILM warm and delete phases compute min_age from the rollover event. If a syslog-* index takes three days to fill and roll, its ILM delete clock starts three days later than the Curator equivalent would have. Reproduce the intended retention, and validate it, rather than copying the number blindly. Second, ILM runs actions within a phase in a fixed internal order — allocation and priority before destructive actions — so you cannot reorder them by rearranging the JSON the way you sequenced numbered Curator actions.

Step-by-Step Migration Outline #

Migrating a running pipeline is a sequencing problem: the ILM policy must take ownership of the indices before the Curator cron is retired, but never in a window where both tools could act on the same index. The outline below is the high-level shape; the full, race-safe runbook — including how to attach the policy to indices that already exist — lives in migrating from Curator to ILM policies.

1. Map each Curator action to an ILM phase. Rollover conditions go to the hot phase unchanged; forcemerge and shrink filters become warm-phase actions with a min_age; the delete_indices age filter becomes the delete phase min_age. Snapshot actions move to an SLM policy, not ILM.
2. Create the policy. Apply syslog-ilm with PUT _ilm/policy/syslog-ilm (or ilm.put_lifecycle). Creating the policy is inert — it manages nothing until an index references it, so this step is safe to run while Curator is still live.
3. Attach via a composable template. Add index.lifecycle.name: syslog-ilm and index.lifecycle.rollover_alias: syslog-write to the index template so every new backing index is managed from creation. A template alone does not touch existing indices.
4. Adopt existing indices, then retire the cron. Attach the policy to indices that already exist with PUT <index>/_settings, confirm they report managed: true, and only then disable the Curator cron entry. Retiring the cron before adoption leaves a gap where nothing rotates; retiring it after adoption without care risks a window where both act.
5. Verify and decommission. Confirm every syslog-* index is managed and progressing, watch one full rollover happen under ILM, then remove the Curator package and its config from the host.

Verification #

Confirm ILM has genuinely taken ownership before you trust it. First, check that the indices Curator used to manage are now managed by the policy:

GET syslog-*/_ilm/explain

{
  "indices": {
    "syslog-000007": {
      "index": "syslog-000007",
      "managed": true,
      "policy": "syslog-ilm",
      "phase": "hot",
      "action": "rollover",
      "step": "check-rollover-ready"
    }
  }
}

"managed": true with "policy": "syslog-ilm" confirms adoption; an index reporting "managed": false is still orphaned and will be rotated by neither tool once the cron is gone. Next, confirm the ILM service itself is running, since a stopped coordinator silently freezes every managed index:

GET _ilm/status

A response of { "operation_mode": "RUNNING" } means the state machine is live. Finally — and this is the step migrations skip at their peril — confirm the Curator cron is actually disabled on the host, so it cannot fire a second delete against indices ILM now owns:

# The syslog Curator entry must be gone (or commented out).
crontab -l | grep -i curator
systemctl list-timers | grep -i curator

Both commands should return nothing. If either still shows an active schedule, you are running both systems against the same indices — the dangerous state described next.

Decision Guidance: When Each Tool Wins #

Migration is not an unconditional verdict against Curator. For the recurring lifecycle of managed time-series data, ILM wins decisively; for a narrow band of one-off and out-of-band tasks, Curator (or a small bespoke script) remains the pragmatic choice.

ILM wins when the work is the ongoing lifecycle of indices that share a policy: rolling over on size or age, migrating between data tiers, force-merging and shrinking on a schedule tied to the rollover clock, and deleting at end of retention. Here ILM’s allocation awareness, restart durability, per-index failure isolation, and native data-tier and searchable-snapshot integration are advantages Curator structurally cannot match, because Curator runs outside the master and cannot see allocation state. Anything that should “just keep happening” to every index in a stream belongs in ILM.

Curator (or an equivalent script) still earns its place when the task is genuinely ad-hoc or falls outside the managed-index model. Deleting a set of indices by an arbitrary name pattern that was never attached to a policy — cleaning up an abandoned experiment, purging a one-off backfill, dropping indices from a decommissioned tenant — is something ILM does not do, because ILM only deletes what it manages. Curator’s rich filter chains (by pattern, by space, by count, by creation_date) are still the fastest way to express a one-shot bulk operation. Some teams keep a minimal Curator install purely for these out-of-band cleanups long after ILM owns the routine rotation. That is a legitimate division of labour, not a migration failure. The point is to stop using Curator for recurring lifecycle work — where a missed cron tick means silent retention drift — and reserve it for deliberate, human-invoked maintenance.

Troubleshooting #

Double-management: both Curator and ILM acting on the same indices #

Symptom: indices roll over or delete at unexpected times; ILM _ilm/explain shows an index that vanished mid-step, or an index rolls twice in quick succession. The root cause is that the Curator cron was never disabled after the ILM policy was attached, so two independent controllers are mutating the same indices. Resolution:

1. Disable the Curator cron immediately — crontab -e and remove the entry, or systemctl disable --now the timer.
2. Re-run GET syslog-*/_ilm/explain and repair any index left in an ERROR step with POST /<index>/_ilm/retry after confirming the cause is cleared.
3. Audit the Curator logs for the overlap window to confirm no index was deleted prematurely; restore from snapshot if a retention breach occurred.

Overlapping deletes race and delete twice (or a shard mid-move) #

Symptom: a delete lands while ILM is relocating the index to a colder tier, producing index_not_found_exception in ILM’s step logs, or an index Curator expected to delete is already gone. Resolution:

1. Enforce the ordering rule: only one system may hold the delete phase at a time. Stop Curator’s delete_indices action before ILM’s delete phase can reach the same age boundary.
2. If an index was deleted during a relocation, ignore the transient index_not_found_exception — the index is gone, which was the intent — but confirm no in-retention index was caught by the wider Curator pattern.
3. Narrow the Curator delete filter to a pattern that cannot overlap managed indices (for example, only unmanaged legacy names) if you intend to keep Curator for cleanup.

Existing indices ignore the new policy #

Symptom: new syslog-* indices are managed, but older ones created before the template change report "managed": false and never transition. Resolution:

1. A composable template only affects indices created after it is applied. Attach the policy to existing indices explicitly with PUT <index>/_settings setting index.lifecycle.name (and the rollover alias on the current write index).
2. Confirm each adopted index reports "managed": true in _ilm/explain.
3. The full adoption procedure, including the ordering that prevents a double-delete race, is covered in migrating from Curator to ILM policies.

FAQ #

Related #

• Migrating from Curator to ILM Policies — the race-safe runbook for handing existing indices to ILM without a double-delete.
• Configuring Index Rollover Conditions — the hot-phase triggers that replace a Curator rollover action.
• Monitoring ILM Health with cat and explain APIs — confirming managed indices progress after the cutover, where Curator left only cron logs.
• Index Lifecycle Management (ILM) — the control plane whose reconciliation model underlies every difference above.

← Back to ILM Architecture & Fundamentals