Bootstrapping Index Templates & Data Streams #

An Elasticsearch index adopts a lifecycle policy exactly once — at the moment it is created — and never again on its own. If the index that Elasticsearch cuts on first ingest has no index.lifecycle.name in its effective settings, it is unmanaged for the rest of its existence: it will never roll over, never migrate off the hot tier, and never be deleted by any retention window, no matter how carefully the policy behind it is authored. The only durable way to guarantee that inheritance across hundreds of indices you never create by hand is to make a template carry the policy, and to make every write land on an index that a template governs. That is what this page builds: a composable template that references reusable component templates and a data stream whose backing indices are born managed. It is the bootstrapping layer beneath everything in ILM Policy Design & Lifecycle Synchronization.

The distinction that trips up most first deployments is timing. A policy applied to a deployment does nothing until an index points at it, and an index can only point at it if the pointer is present when the index is born. Templates close that gap by injecting the pointer at creation; data streams close it further by making the write target itself a template-managed object that rolls its own backing indices. Get the ordering wrong — ingest before the template exists — and Elasticsearch happily creates an ordinary, unmanaged index that will shadow your intent forever.

Prerequisites #

• Elasticsearch 8.x with data-tier node roles (data_hot, data_warm, data_cold) assigned so managed indices can migrate after the hot phase.
• An existing ILM policy — this page assumes logs-nginx-ilm is already applied via PUT _ilm/policy/logs-nginx-ilm; authoring it is covered under building custom ILM policies via the API.
• manage_index_templates and manage_ilm cluster privileges on the service account that applies templates, scoped per securing ILM policies with RBAC.
• elasticsearch-py v8.0+ (pip install "elasticsearch>=8,<9") — the code here uses the v8 keyword-argument surface: cluster.put_component_template, indices.put_index_template, indices.create_data_stream, not the legacy body= pattern.
• A decision made in advance: data stream (append-only, automatic rollover, no alias) or alias-based rollover (needs index.lifecycle.rollover_alias and a bootstrap write index). This page centres on data streams and contrasts the alias path where it matters.

Architecture: How Inheritance Flows From Template to Backing Index #

Composable index templates are built from two kinds of parts. Component templates are named, reusable fragments — one holding settings, another holding mappings — that can be shared across many index templates. A composable index template (PUT _index_template/<name>) stitches component templates together with its own inline template block, claims a set of index_patterns, sets a numeric priority to break ties when patterns overlap, and — for a data stream — carries the empty marker object data_stream: {}. When Elasticsearch needs to create an index whose name matches the pattern, it merges the referenced component templates in listed order, overlays the inline block, and stamps the result onto the new index. The lifecycle pointer rides in template.settings.index.lifecycle.name, so it lands in the effective settings automatically.

A data stream adds one more layer. Rather than exposing a write alias you have to bootstrap and maintain, the stream is a single named write target whose physical storage is a series of hidden, append-only backing indices named .ds-<stream>-<yyyy.MM.dd>-NNNNNN. Writes always go to the stream, which routes them to the current write index; reads fan out across all backing indices. When the hot-phase rollover action fires, the stream mints the next backing index, flips the write pointer to it, and leaves the previous one immutable — and because the write index inherited logs-nginx-ilm from the template, the freshly minted -000002 inherits it too. No rollover_alias is ever set, because the stream is the write pointer.

The contrast with alias-based rollover is worth stating plainly, because the two mechanisms answer the same need in incompatible ways. An alias-managed index template names an alias in index.lifecycle.rollover_alias, and you must bootstrap the first backing index yourself with is_write_index: true before any write lands — the same pattern used for configuring index rollover conditions. A data-stream template sets data_stream: {} instead, sets no rollover alias, and needs no bootstrap index: the first document (or an explicit PUT _data_stream) creates the stream and its first backing index atomically. Setting rollover_alias on a data-stream template is not merely redundant — it produces an index that ILM cannot roll, because the stream already owns the write pointer the alias is trying to claim.

Configuration Reference #

The three objects below form one deployable unit: two component templates, then the composable template that binds them and attaches the policy. Comments annotate what each field controls.

PUT _component_template/logs-nginx-settings
{
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-nginx-ilm",   // the pointer every backing index inherits
      "index.number_of_shards": 3,                 // primary count per backing index
      "index.number_of_replicas": 1,
      "index.refresh_interval": "5s"               // relax for heavier ingest throughput
    }
  },
  "_meta": { "owner": "observability-team", "managed_by": "gitops" }
}

PUT _component_template/logs-nginx-mappings
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp":   { "type": "date" },        // data streams require a @timestamp field
        "url.path":     { "type": "keyword" },
        "http.status":  { "type": "short" },
        "message":      { "type": "text" }
      }
    }
  },
  "_meta": { "schema_version": 3 }
}

PUT _index_template/logs-nginx-template
{
  "index_patterns": ["logs-nginx*"],               // matches the stream name and its backing indices
  "data_stream": {},                               // makes this a data-stream template — no rollover_alias
  "priority": 200,                                 // higher than the managed defaults (100) so it wins
  "composed_of": [                                  // component templates merged in this order
    "logs-nginx-settings",
    "logs-nginx-mappings"
  ],
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-nginx-ilm"     // belt-and-braces: also set inline so it always resolves
    }
  },
  "_meta": { "description": "nginx access logs, ILM-managed data stream" }
}

Two rules govern the merge. First, composed_of fragments are applied in listed order, and the inline template block wins over all of them — which is why setting index.lifecycle.name inline as well as in the component template is a safe redundancy, not a conflict. Second, priority resolves overlapping index_patterns: when two templates both match a new index name, the higher priority wins outright and the lower one is ignored entirely (there is no field-level blending across templates). Elasticsearch ships built-in templates for logs-*, metrics-*, and similar at priority 100–200, so a custom pattern that could overlap them must claim a higher number to take effect.

Creating the stream explicitly (rather than waiting for the first document) is a one-line call once the template exists:

PUT _data_stream/logs-nginx

Step-by-Step Implementation #

The sequence is strict: component templates first, composable template second, stream third. Reverse any two and you either reference a template that does not yet exist or ingest into an unmanaged index.

1. Apply the two component templates. Each PUT _component_template/... is independently versioned and reusable; a settings fragment can back several index templates while a mappings fragment stays field-specific.
2. Apply the composable index template with data_stream: {}, composed_of, a winning priority, and index.lifecycle.name. Until this exists, no logs-nginx* write can create a managed index.
3. Create the data stream with PUT _data_stream/logs-nginx, or let the first indexed document create it implicitly. Either way, .ds-logs-nginx-<date>-000001 is born carrying logs-nginx-ilm.
4. Confirm inheritance before trusting the pipeline in production.

The Python v8 helper below drives steps 1–3 idempotently and then reads back the effective lifecycle setting on the write index:

import logging
from elasticsearch import Elasticsearch, ApiError, NotFoundError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)


def bootstrap_managed_stream(es: Elasticsearch, policy_name: str = "logs-nginx-ilm") -> None:
    # 1. Component templates — settings (carries the lifecycle pointer) and mappings.
    es.cluster.put_component_template(
        name="logs-nginx-settings",
        template={
            "settings": {
                "index.lifecycle.name": policy_name,
                "index.number_of_shards": 3,
                "index.number_of_replicas": 1,
                "index.refresh_interval": "5s",
            }
        },
        meta={"owner": "observability-team"},
    )
    es.cluster.put_component_template(
        name="logs-nginx-mappings",
        template={
            "mappings": {
                "properties": {
                    "@timestamp": {"type": "date"},
                    "url.path": {"type": "keyword"},
                    "http.status": {"type": "short"},
                    "message": {"type": "text"},
                }
            }
        },
    )

    # 2. Composable index template — data_stream:{} and a winning priority.
    es.indices.put_index_template(
        name="logs-nginx-template",
        index_patterns=["logs-nginx*"],
        data_stream={},
        priority=200,
        composed_of=["logs-nginx-settings", "logs-nginx-mappings"],
        template={"settings": {"index.lifecycle.name": policy_name}},
        meta={"description": "nginx access logs, ILM-managed data stream"},
    )
    logger.info("Template 'logs-nginx-template' applied.")

    # 3. Create the stream explicitly so the first backing index exists immediately.
    try:
        es.indices.create_data_stream(name="logs-nginx")
        logger.info("Data stream 'logs-nginx' created.")
    except ApiError as exc:
        # resource_already_exists_exception is fine on a re-run — bootstrap is idempotent.
        if getattr(exc, "status_code", None) == 400 and "resource_already_exists" in str(exc.info):
            logger.info("Data stream 'logs-nginx' already exists — skipping.")
        else:
            raise


def confirm_inheritance(es: Elasticsearch) -> None:
    # Read the effective lifecycle setting off the write backing index.
    ds = es.indices.get_data_stream(name="logs-nginx")
    write_index = ds["data_streams"][0]["indices"][-1]["index_name"]
    settings = es.indices.get_settings(
        index=write_index, filter_path="**.lifecycle"
    )
    lifecycle = settings[write_index]["settings"]["index"]["lifecycle"]
    logger.info("Write index %s inherits policy: %s", write_index, lifecycle["name"])


if __name__ == "__main__":
    es = Elasticsearch(
        "https://es-cluster-01:9200",
        api_key="YOUR_BASE64_ENCODED_API_KEY",
        verify_certs=True,
    )
    bootstrap_managed_stream(es)
    confirm_inheritance(es)

The idempotency here is deliberate: put_component_template and put_index_template are upserts, so re-running the bootstrap converges the deployment to the declared state without side effects, and only the stream-creation call needs the already-exists guard. That property is what lets the same script run from CI on every merge without special-casing first deploy versus redeploy.

Verification #

Do not assume inheritance — prove it. First, confirm the template registered and carries the data-stream marker:

GET _index_template/logs-nginx-template

The response index_template object must show "data_stream": {}, your composed_of list, and priority: 200. Next, confirm the stream exists and which policy it reports:

GET _data_stream/logs-nginx

{
  "data_streams": [
    {
      "name": "logs-nginx",
      "generation": 1,
      "template": "logs-nginx-template",
      "ilm_policy": "logs-nginx-ilm",
      "indices": [
        { "index_name": ".ds-logs-nginx-2026.07.17-000001", "index_uuid": "..." }
      ]
    }
  ]
}

"ilm_policy": "logs-nginx-ilm" is the stream-level confirmation; generation starts at 1 and increments by one on every rollover. Now check the effective setting on the backing index itself — the single most decisive test, because it reads the merged result rather than the template’s intent:

GET .ds-logs-nginx-2026.07.17-000001/_settings?filter_path=**.lifecycle

{
  ".ds-logs-nginx-2026.07.17-000001": {
    "settings": { "index": { "lifecycle": { "name": "logs-nginx-ilm" } } }
  }
}

An empty response here is the failure signal: the index was created without a matching template, so it is unmanaged. Finally, confirm ILM has actually adopted it and reports a phase:

GET .ds-logs-nginx-2026.07.17-000001/_ilm/explain

"managed": true with a phase of hot and a step of check-rollover-ready means the write index is in the healthy steady state, polling its rollover conditions each indices.lifecycle.poll_interval. This is the same signal the rest of the lifecycle relies on across the hot-warm-cold architecture.

Threshold Tuning & Priority Guidance #

Templates fail quietly, so the tuning here is mostly about avoiding silent collisions rather than sizing a number.

1. Set priority above every template that could match the same pattern. Elasticsearch’s managed logs-*/metrics-* templates occupy 100–200; a custom logs-nginx* template at priority: 200 or higher wins the overlap. Two custom templates at the same priority for overlapping patterns is a hard error at PUT time — Elasticsearch refuses the ambiguity rather than guessing.
2. Keep the pattern tight. logs-nginx* matches the stream and its .ds-logs-nginx-... backing indices without swallowing unrelated logs-* streams. A pattern like logs-* on a custom template is an invitation to shadow the built-ins and every other team’s stream on the deployment.
3. Choose data stream versus alias by write shape, not habit. Data streams are for append-only, timestamped telemetry where documents are never updated in place — logs, metrics, traces. If the workload updates or deletes individual documents by ID, a data stream is the wrong tool; use an alias-based rollover template with rollover_alias instead, and bootstrap the first write index yourself.
4. Order composed_of from generic to specific. Later fragments and the inline template override earlier ones, so a shared org-wide settings fragment should come first and a stream-specific mappings fragment later, letting the specific win where they touch the same key.

On template ordering during a migration: because a template only affects indices created after it is applied, changing number_of_shards or the policy name in a template does nothing to indices already open. The change takes effect at the next rollover, when a fresh backing index is minted under the new template. Plan shard-count changes to coincide with a rollover boundary rather than expecting them to reshape live data.

Troubleshooting #

Newly created index is unmanaged (managed: false) #

Symptom: GET <index>/_ilm/explain shows "managed": false, and _settings?filter_path=**.lifecycle is empty. Resolution:

1. Confirm a template actually matches the index name: GET _index_template/logs-nginx-template and check index_patterns covers it.
2. If a higher-priority template is winning, it is overriding your settings — list candidates with GET _index_template and compare priorities; raise yours or narrow the competing pattern.
3. Templates do not retro-apply. For an already-open unmanaged index, attach the policy directly with PUT <index>/_settings {"index.lifecycle.name":"logs-nginx-ilm"}, or force a rollover so the next backing index inherits correctly.

Data stream creation fails with no matching index template #

Symptom: PUT _data_stream/logs-nginx returns unable to create data stream because no matching index template with a data_stream definition found. Resolution:

1. Apply the composable template with data_stream: {} before the stream — a template without that marker cannot back a stream.
2. Verify the stream name matches index_patterns; logs-nginx matches logs-nginx* but not nginx-logs*.
3. Re-run PUT _data_stream/logs-nginx after the template exists.

rollover_alias set on a data-stream template #

Symptom: the write index never rolls; _ilm/explain shows an ERROR step referencing the rollover alias, or ILM reports the alias does not point at the index. Resolution:

1. Remove index.lifecycle.rollover_alias from the component/index template entirely — data streams own the write pointer and reject an external alias.
2. Re-apply the corrected template and force a rollover: POST logs-nginx/_rollover.
3. Confirm GET _data_stream/logs-nginx now shows an incremented generation.

Two templates collide on the same pattern and priority #

Symptom: PUT _index_template fails with index template ... has index patterns ... matching patterns from existing templates ... with patterns ... that have the same priority. Resolution:

1. Give the templates distinct priorities so one wins deterministically, or narrow one pattern so they no longer overlap.
2. If a legacy _template (the old non-composable API) is colliding, migrate it — composable templates always win over legacy ones regardless of priority, but a legacy match still muddies diagnosis.

FAQ #

Related #

• Attaching an ILM policy in a composable index template — the alias and data-stream variants of setting the lifecycle pointer.
• Bootstrapping a data stream for ILM — standing up a managed stream from scratch and its .ds- backing indices.
• Versioning ILM policies across environments — promoting the same template-and-policy artifact from staging to production.
• Detecting ILM policy drift — reconciling the template you declared with what the deployment is actually running.
• Configuring index rollover conditions — the alias-based rollover mechanism this page contrasts with data streams.

← Back to ILM Policy Design & Lifecycle Synchronization