Bootstrapping a Data Stream for ILM #

Stand up an append-only Elasticsearch data stream that is managed by ILM from its very first backing index — no write alias to bootstrap, no per-index settings to remember.

A data stream is the ergonomic default for timestamped telemetry precisely because it collapses three chores into one object: it is the write target, the rollover pointer, and the retention unit all at once. You index into the name logs-nginx, and Elasticsearch quietly maintains a chain of hidden backing indices behind it, cutting a new one whenever the hot-phase rollover action fires and handing the sealed ones to later phases. The catch is a strict ordering requirement — the template that defines the stream must exist before the first document lands — and this page walks that ordering end to end. It is one path through bootstrapping index templates and data streams, the broader area under ILM Policy Design & Lifecycle Synchronization.

Prerequisites #

• Elasticsearch 8.x with data-tier node roles assigned so the stream’s backing indices can migrate off the hot tier.
• An existing ILM policy logs-nginx-ilm whose hot phase carries a rollover action.
• elasticsearch-py v8.0+ — the code uses cluster.put_component_template, indices.put_index_template, and indices.create_data_stream.
• manage_index_templates and manage_ilm cluster privileges on the applying account.

Implementation #

Three objects, applied in order, then the stream itself. Reverse the order and the first write creates an ordinary, unmanaged index that will shadow the stream forever.

1. Component templates (settings + mappings) #

A data stream requires a @timestamp field in its mappings, so the mappings fragment is not optional. Split settings from mappings so each can be reused and versioned independently:

PUT _component_template/logs-nginx-settings
{
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-nginx-ilm",
      "index.number_of_shards": 3,
      "index.number_of_replicas": 1
    }
  }
}

PUT _component_template/logs-nginx-mappings
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp":  { "type": "date" },
        "url.path":    { "type": "keyword" },
        "http.status": { "type": "short" }
      }
    }
  }
}

2. Composable index template with data_stream: {} #

The marker data_stream: {} is what turns a matching template into a stream definition. No rollover_alias appears anywhere — the stream is its own write pointer:

PUT _index_template/logs-nginx-template
{
  "index_patterns": ["logs-nginx*"],
  "data_stream": {},
  "priority": 200,
  "composed_of": ["logs-nginx-settings", "logs-nginx-mappings"]
}

3. Create the stream #

With the template in place, create the stream explicitly so the first backing index exists before any traffic — this lets verification run against a real index instead of racing the first document:

PUT _data_stream/logs-nginx

Indexing the first document would create the stream implicitly with identical results, but an explicit PUT makes the bootstrap deterministic. Either way, the first backing index is born as .ds-logs-nginx-2026.07.17-000001, hidden, and already carrying logs-nginx-ilm.

How the backing indices work #

The physical storage behind logs-nginx is a series of indices named .ds-<stream>-<yyyy.MM.dd>-NNNNNN, where the date is the stream’s creation-or-rollover date and NNNNNN is a zero-padded generation counter. These indices are hidden (they do not appear in a bare GET _cat/indices without ?expand_wildcards=hidden) and append-only: you index through the stream name, never into a backing index directly. Reads fan out across every backing index; writes route only to the current write index, which is always the highest-numbered one.

Driving the bootstrap from Python (v8) #

The v8 client applies the same sequence idempotently. Note that create_data_stream lives on the indices namespace and takes name=:

from elasticsearch import Elasticsearch, ApiError

es = Elasticsearch("https://es-cluster-01:9200", api_key="YOUR_KEY", verify_certs=True)


def bootstrap_stream(policy: str = "logs-nginx-ilm") -> None:
    es.cluster.put_component_template(
        name="logs-nginx-settings",
        template={"settings": {"index.lifecycle.name": policy,
                               "index.number_of_shards": 3,
                               "index.number_of_replicas": 1}},
    )
    es.cluster.put_component_template(
        name="logs-nginx-mappings",
        template={"mappings": {"properties": {"@timestamp": {"type": "date"}}}},
    )
    es.indices.put_index_template(
        name="logs-nginx-template",
        index_patterns=["logs-nginx*"],
        data_stream={},                    # no rollover_alias — the stream owns the write pointer
        priority=200,
        composed_of=["logs-nginx-settings", "logs-nginx-mappings"],
    )
    try:
        es.indices.create_data_stream(name="logs-nginx")
    except ApiError as exc:
        # resource_already_exists on a re-run is expected — the bootstrap is idempotent.
        if "resource_already_exists" not in str(exc.info):
            raise


bootstrap_stream()

Verification #

Confirm the stream reports the policy and its generation, then confirm the write backing index is actually managed. Start at the stream level:

GET _data_stream/logs-nginx

{
  "data_streams": [
    {
      "name": "logs-nginx",
      "generation": 1,
      "template": "logs-nginx-template",
      "ilm_policy": "logs-nginx-ilm",
      "indices": [
        { "index_name": ".ds-logs-nginx-2026.07.17-000001", "index_uuid": "..." }
      ]
    }
  ]
}

"ilm_policy": "logs-nginx-ilm" proves the stream picked up the policy from the template, and "generation": 1 is the rollover counter — it increments by one every time the write index rolls, so a stream that has rolled twice reports generation: 3. Then confirm the backing index is managed and polling:

GET .ds-logs-nginx-2026.07.17-000001/_ilm/explain

{
  "indices": {
    ".ds-logs-nginx-2026.07.17-000001": {
      "managed": true,
      "policy": "logs-nginx-ilm",
      "phase": "hot",
      "action": "rollover",
      "step": "check-rollover-ready"
    }
  }
}

"managed": true in the check-rollover-ready step is the healthy steady state — ILM is evaluating the rollover conditions each indices.lifecycle.poll_interval, exactly as it would for the alias-based path described under configuring index rollover conditions.

Gotchas and edge cases #

• The template must exist before the first write. A document indexed into logs-nginx before the data_stream: {} template is applied creates a plain, unmanaged index named logs-nginx — not a stream — and that ordinary index then blocks stream creation entirely. Apply the template first; if you already tripped this, delete the stray index and re-create the stream.
• Never set rollover_alias on a stream. The stream owns the write pointer, so index.lifecycle.rollover_alias has nothing to bind to and pushes rollover into an ERROR step. A data-stream template carries data_stream: {} and the lifecycle name, and no alias.
• You cannot write directly to a backing index. Attempting to index into .ds-logs-nginx-2026.07.17-000001 fails, because backing indices are append-closed to direct writes — all writes must go through the stream name so the routing and rollover invariants hold. Read from backing indices freely; write only to the stream.
• Generation is a rollover counter, not a document count. generation in GET _data_stream increments by one on each rollover and never decrements, even after old backing indices are deleted by the delete phase. Use it to confirm rollover is happening, not to gauge data volume.

FAQ #

Related #

• Attaching an ILM policy in a composable index template — the exact setting that binds the policy this stream inherits.
• Bootstrapping index templates and data streams — the full template-and-stream anatomy and priority rules.
• Understanding hot-warm-cold architecture — where the stream’s sealed backing indices migrate after they roll off hot.

← Back to Bootstrapping Index Templates & Data Streams · ILM Policy Design & Lifecycle Synchronization